Essay Abstract

I give a case that, as a public good, societies and their governments should support and invest in scientific research on crucial phenomena, empirical features of the world that figure strongly in how humanity's choices influence the size of its future. In particular, I give reasons for thinking that (1) humanity's vulnerability or robustness to accidents arising from biological engineering, and (2) the future rates of improvement of artificial intelligence and its susceptibility to misuse, are phenomena that call strongly for our systematic attention.

Author Bio

Daniel Dewey is a Research Fellow at the Oxford Martin Programme on the Impacts of Future Technology and the Future of Humanity Institute. His research centres on high-impact, understudied features of the long-term future of artificial intelligence. Topics of particular interest include intelligence explosion, machine superintelligence, and AI ethics. Daniel was previously a software engineer at Google, Intel Labs Pittsburgh, and Carnegie Mellon University.

Download Essay PDF File

6 days later

Nice essay Daniel.

I am currently working on an anthology on Machine Ethics for IGI press due out next year. Would you have any interest in submitting a chapter?

Best of luck in the contest,

Rick Searle

    Hi Rick. Glad you liked the essay, thanks!

    I might be interested in submitting to a machine ethics anthology, yes. Could you email me with details? You can find my address at my homepage.

    Dear Mr. Dewey,

    I regret that I could not understand most of your essay due possibly to the fact that I have a poor grasp of abstractions. On the one hand, sober scientists have assured me that it has taken thousands of years for the human brain to evolve. You now claim to be able to build an artificial brain that is far superior to any human brain. The only thing I notice about any artificial brain is that it never seems to be attached to a black body, or to ever communicate in Spanish. Billions of dollars have been spent by the predominantly white male government of the United States so that a few predominantly white males can explore space, listen to messages from outer-space, or build brand new never been used artificial brains. Nine billion dollars have been cut from the food stamp program that aids the poor.

      Dear Mr. Fisher,

      Thanks for reading anyway; I'll have to see what I can do to make my writing maximally accessible.

      My essay focuses on reasons for thinking that the size of humanity's future is a very important consideration, but that doesn't mean that I oppose efforts to address present-day wrongs like poverty and animal suffering. I think we should address both.

      9 days later

      Hi Aaron,

      Thanks for writing! I've rated and left a comment on your essay.

      Best of luck,

      Daniel

      An interesting classification of types of risk.You have chosen two interesting technologies to pick out in particular.

      I think it would have been good if you had talked about actual accidents that have occurred with GM crops.There is a lot of information available on the web.

      Contamination of food with crops engineered to produce medicines, contamination of organic crops with GM genes, replacement of traditional local crops by licenced vulnerable terminator gene carrying monocultres; associated with farmer suicides in India due to failure of such crops that leaves the farmer with nothing not even seed or money to buy seed for the following year. There is the risk of transfer of terminator genes to other plants.There is the problem of greater usage of pesticides, particularly glyphosate that has been linked to liver damage, and other health problems in some research. Also the risk to other animals such as aquatic organisms.freecymru.org US Regulation of GM crops: USDA slammed in Congress hearing

      The risks are certainly not only from novel pathogens although the escape of such organisms from secure facilities is shocking. That was the cause of the foot and mouth out break in the UK in 2007 Foot and mouth outbreak caused by petty government dispute over leaky drain.

      I think that would have made the threat seem far more real and urgent.I like that you have compared the two kinds of risk and identified AI as less urgent but still with very high potential risks. I don't yet know of problems caused by AI but having to navigate an automated telephone system can be frustrating. It cam only get worse if the AI is able to argue as well: )

        Hi Daniel,

        I'm sorry I accidentally submitted my post before I was ready. Then I lost my internet connection.

        Firstly I meant to say Hi Daniel.

        "The UK foot and mouth outbreak 2007 was from a "secure" laboratory. The FMD virus strain by which the outbreak was caused was found to be most similar to strains used in international diagnostic laboratories and in vaccine production",..." this strain is a 01 BFS67 - like virus, isolated in the 1967 Foot and Mouth Disease outbreak in Great Britain".From European community, food, animal diseases control

        I think this evidence, regarding GM crops and the escape of organisms from secure facilities, indicates that the dangers are not from lack of research but inaction in the face of known risks.

        I meant to say It can only get worse if the AI is able to argue as well: )

        Good luck. Georgina

        Hi Georgina,

        Thanks for pointing out the GM crops case, and for giving those examples; it's certainly an area I'm interested in, and building more detailed cases with more examples makes sense.

        Best,

        Daniel

        Daniel,

        I didn't realize till I read your essay that the comment you left on my essay on Biology and AI were the focus of your essay.

        I agree with you that "one of the most important tasks facing us today is the scientific investigation of certain Crucial Phenomena." You didn't, however, say why biological stability and AI are crucial, but just selected them to exemplify how they are crucial. Did I miss something? For me, these are just two out of a long laundry list.

        I wish you had elaborated on the game between Humanity and Nature a lot more!

        Also, you talk about funding support and "have the particular skills and resources" as the only important issues; Does researcher interest matter or is funding, resources and skills the only criteria you deem important for scientific research?

        - Ajay

          Hi Ajay,

          Yes, it seems that our essay topics are relevant to one another!

          I did, on pages 8 and 9, explain why I think AI and biological instability are crucial--- they both have the potential to render us extinct in the relatively near term. I expect that there are other crucial phenomena, but I don't have a "long laundry list"--- do you have any examples ready to hand?

          I'm glad you liked the "game between Humanity and Nature", maybe I'll expand on that in a later piece of writing :)

          While researcher interest is clearly important, I do think that a certain amount of societal resources should be devoted to phenomena that are crucial for the long-term future of humanity *whether or not* anyone finds them particularly interesting. This isn't to say that everyone has to work on these things, just that it would be good if enough people work on them for us to reap the considerable benefits. That said, I think that skills and resources are the relevant bottlenecks--- I do regularly encounter many people who are interested in i.e. biological instability or AI safety, if only they had the skills and resources to pursue them!

          Thanks for your thoughtful comment,

          Best,

          Daniel

          Daniel,

          Quite important ideas are simply proposed. Many of us say the same, but not as purposely and emphatically. I like "aiming for a large future" and pursuing "crucial phenomena." Your anecdote at the beginning represents choices we make or don't make during our careers, settling for mundane but insignificant studies rather than pushing for "crucial phenomena."

          My essay is similar in my prospect of looking beyond the mundane and within the microcosm of the universe, our brain.

          Good job,

          Jim

            Excellent essay, Daniel. In my opinion, one of the best. You frame the issues extremely well. I'm almost in complete agreement with you, although I would add that we need need to improve not just our knowledge, but also what we collectively do with our knowledge; that we need better institutions as well as better science.

            Thanks for your comment on my essay, by the way. I responded on my own page, but the short version is that I think I did make a mistake. If I could go back I would change or get rid of that sentence. I really appreciate your pointing it out to me.

            Good luck in the contest--your essay deserves to do well!

            Best,

            Robert

              Daniel,

              Great job, I think your article perfectly matches the theme of this competition! Although science (like other cultural achievements) mostly progresses through small, incremental steps, it is crucial to keep an eye on the long-term consequences and risks. You very clearly identify two potentially critical fields of research.

              Would you agree that the notion of biological instability must be extended to encompass our biological ecosystem? One can argue that major disruption of our natural environment caused by artificial agents would be just as harmful as one that only affects humans themselves.

              Good luck!

              Jens

                Dan,

                Nice essay identifying the importance of scientific advancement and the prioritisation of attention and funding. I consider it very well written, argued and organised, and should be better placed. I also agree your two identified areas, but believe there's also a strong case for targeting a great leap in understanding of nature by unification of classical and quantum physics and demystification of QM. You may feel that's already being done, but my essay shows current views have the opposite effect, keeping us in a deepening 'rut'. I show that QM an be classically derived, comprehensibly.

                My previous successful essays showed how the same mechanism allows SR to converge. But of course nobody is looking and journals won't will risk suggesting such advancement! I agree 'big' and don't think any other single success could give such broad advancement. I subtly suggest that we need to improve our way of thinking to enable the right focus. I look forward to any views on mine.

                Very well done for yours, and best of luck in the results.

                Best wishes

                Peter

                  Dear Daniel Dewey,

                  I very much like your idea of "crucial phenomena". In particular you choose bio-engineering hazards and misuse of AI. I agree with the first, and, while I do not foresee AI becoming conscious, I see its use by Google or by the NSA as potentially destroying privacy, a very negative outcome. And I can envision other serious misuse that does not require super intelligence.

                  I would suggest that another crucial phenomena is growth of government based on AI and communication techniques. It may be a less forgiving disaster than some physical disasters. I analyze this problem in my essay which I hope you will read and comment on.

                  I enjoyed your Hamming anecdotes. I talked with him a few times in the 80s and found him full of interesting opinions.

                  Best regards,

                  Edwin Eugene Klingman

                    Thanks, Robert! Best of luck to you as well, I enjoyed your essay.

                    I agree that we should improve our institutions, especially in linking knowledge to action. I think some folks around FHI are interested in institution design; I should ask them what they've been thinking about.

                    Best,

                    Daniel

                    Thanks, Jens! I'm glad you liked it.

                    I would definitely agree that stability of the ecosystem as a whole in the face of biological engineering / synthetic biology should be included in the "biological instability" category; it might turn out to be the case that evolutionarily difficult steps are easy for biological engineers, and that the biosphere won't have the appropriate defense mechanisms, or that the equilibrium that's eventually reached will be unsuitable to human life.

                    Good luck to you as well!

                    Cheers,

                    Daniel

                    Thanks, Peter!

                    Maybe you can help me understand--- why would a leap forward in quantum mechanics make a significant difference to humanity's future? Is it more, less, or equally important relative to other major scientific questions, in your opinion?

                    Best,

                    Daniel