Superintelligence: Paths, Dangers, Strategies

Last updated

Superintelligence:
Paths, Dangers, Strategies
Superintelligence.jpg
First edition
Author Nick Bostrom
Country United Kingdom
Language English
Subject Artificial intelligence
Genre Philosophy, popular science
Publisher Oxford University Press [1]
Publication date
July 3, 2014 (UK)
September 1, 2014 (US)
Media typePrint, e-book, audiobook
Pages352 pp.
ISBN 978-0199678112
Preceded by Global Catastrophic Risks  

Superintelligence: Paths, Dangers, Strategies is a 2014 book by the philosopher Nick Bostrom. It explores how superintelligence could be created and what its features and motivations might be. [2] It argues that superintelligence, if created, would be difficult to control, and that it could take over the world in order to accomplish its goals. The book also presents strategies to help make superintelligences whose goals benefit humanity. [3] It was particularly influential for raising concerns about existential risk from artificial intelligence. [4]

Contents

Synopsis

It is unknown whether human-level artificial intelligence will arrive in a matter of years, later this century, or not until future centuries. Regardless of the initial timescale, once human-level machine intelligence is developed, a "superintelligent" system that "greatly exceeds the cognitive performance of humans in virtually all domains of interest" would most likely follow surprisingly quickly. Such a superintelligence would be very difficult to control.

While the ultimate goals of superintelligences could vary greatly, a functional superintelligence will spontaneously generate, as natural subgoals, "instrumental goals" such as self-preservation and goal-content integrity, cognitive enhancement, and resource acquisition. For example, an agent whose sole final goal is to solve the Riemann hypothesis (a famous unsolved mathematical conjecture) could create and act upon a subgoal of transforming the entire Earth into some form of computronium (hypothetical material optimized for computation) to assist in the calculation. The superintelligence would proactively resist any outside attempts to turn the superintelligence off or otherwise prevent its subgoal completion. In order to prevent such an existential catastrophe, it is necessary to successfully solve the "AI control problem" for the first superintelligence. The solution might involve instilling the superintelligence with goals that are compatible with human survival and well-being. Solving the control problem is surprisingly difficult because most goals, when translated into machine-implementable code, lead to unforeseen and undesirable consequences.

The owl on the book cover alludes to an analogy which Bostrom calls the "Unfinished Fable of the Sparrows". [5] A group of sparrows decide to find an owl chick and raise it as their servant. [6] They eagerly imagine "how easy life would be" if they had an owl to help build their nests, to defend the sparrows and to free them for a life of leisure. The sparrows start the difficult search for an owl egg; only "Scronkfinkle", a "one-eyed sparrow with a fretful temperament", suggests thinking about the complicated question of how to tame the owl before bringing it "into our midst". The other sparrows demur; the search for an owl egg will already be hard enough on its own: "Why not get the owl first and work out the fine details later?" Bostrom states that "It is not known how the story ends", but he dedicates his book to Scronkfinkle. [5] [4]

Reception

The book ranked #17 on The New York Times list of best selling science books for August 2014. [7] In the same month, business magnate Elon Musk made headlines by agreeing with the book that artificial intelligence is potentially more dangerous than nuclear weapons. [8] [9] [10] Bostrom's work on superintelligence has also influenced Bill Gates’s concern for the existential risks facing humanity over the coming century. [11] [12] In a March 2015 interview by Baidu's CEO, Robin Li, Gates said that he would "highly recommend" Superintelligence. [13] According to the New Yorker , philosophers Peter Singer and Derek Parfit have "received it as a work of importance". [4] Sam Altman wrote in 2015 that the book is the best thing he has ever read on AI risks. [14]

The science editor of the Financial Times found that Bostrom's writing "sometimes veers into opaque language that betrays his background as a philosophy professor" but convincingly demonstrates that the risk from superintelligence is large enough that society should start thinking now about ways to endow future machine intelligence with positive values. [15] A review in The Guardian pointed out that "even the most sophisticated machines created so far are intelligent in only a limited sense" and that "expectations that AI would soon overtake human intelligence were first dashed in the 1960s", but the review finds common ground with Bostrom in advising that "one would be ill-advised to dismiss the possibility altogether". [3]

Some of Bostrom's colleagues suggest that nuclear war presents a greater threat to humanity than superintelligence, as does the future prospect of the weaponisation of nanotechnology and biotechnology. [3] The Economist stated that "Bostrom is forced to spend much of the book discussing speculations built upon plausible conjecture... but the book is nonetheless valuable. The implications of introducing a second intelligent species onto Earth are far-reaching enough to deserve hard thinking, even if the prospect of actually doing so seems remote." [2] Ronald Bailey wrote in the libertarian Reason that Bostrom makes a strong case that solving the AI control problem is the "essential task of our age". [16] According to Tom Chivers of The Daily Telegraph , the book is difficult to read but nonetheless rewarding. [6] A reviewer in the Journal of Experimental & Theoretical Artificial Intelligence broke with others by stating the book's "writing style is clear" and praised the book for avoiding "overly technical jargon". [17] A reviewer in Philosophy judged Superintelligence to be "more realistic" than Ray Kurzweil's The Singularity Is Near . [18]

See also

Related Research Articles

<span class="mw-page-title-main">Eliezer Yudkowsky</span> American AI researcher and writer (born 1979)

Eliezer S. Yudkowsky is an American artificial intelligence researcher and writer on decision theory and ethics, best known for popularizing ideas related to friendly artificial intelligence, including the idea of a "fire alarm" for AI. He is a co-founder and research fellow at the Machine Intelligence Research Institute (MIRI), a private research nonprofit based in Berkeley, California. His work on the prospect of a runaway intelligence explosion influenced philosopher Nick Bostrom's 2014 book Superintelligence: Paths, Dangers, Strategies.

<span class="mw-page-title-main">Friendly artificial intelligence</span> AI to benefit humanity

Friendly artificial intelligence is hypothetical artificial general intelligence (AGI) that would have a positive (benign) effect on humanity or at least align with human interests or contribute to fostering the improvement of the human species. It is a part of the ethics of artificial intelligence and is closely related to machine ethics. While machine ethics is concerned with how an artificially intelligent agent should behave, friendly artificial intelligence research is focused on how to practically bring about this behavior and ensuring it is adequately constrained.

<span class="mw-page-title-main">Nick Bostrom</span> Swedish philosopher and writer (born 1973)

Nick Bostrom is a Swedish philosopher at the University of Oxford known for his work on existential risk, the anthropic principle, human enhancement ethics, whole brain emulation, superintelligence risks, and the reversal test. He is the founding director of the Future of Humanity Institute at Oxford University.

A superintelligence is a hypothetical agent that possesses intelligence far surpassing that of the brightest and most gifted human minds. "Superintelligence" may also refer to a property of problem-solving systems whether or not these high-level intellectual competencies are embodied in agents that act in the world. A superintelligence may or may not be created by an intelligence explosion and associated with a technological singularity.

The Machine Intelligence Research Institute (MIRI), formerly the Singularity Institute for Artificial Intelligence (SIAI), is a non-profit research institute focused since 2005 on identifying and managing potential existential risks from artificial general intelligence. MIRI's work has focused on a friendly AI approach to system design and on predicting the rate of technology development.

<span class="mw-page-title-main">AI takeover</span> Hypothetical artificial intelligence scenario

An AI takeover is a hypothetical scenario in which artificial intelligence (AI) becomes the dominant form of intelligence on Earth, as computer programs or robots effectively take control of the planet away from the human species. Possible scenarios include replacement of the entire human workforce, takeover by a superintelligent AI, and the popular notion of a robot uprising. Stories of AI takeovers are very popular throughout science fiction. Some public figures, such as Stephen Hawking and Elon Musk, have advocated research into precautionary measures to ensure future superintelligent machines remain under human control.

Differential technological development is a strategy of technology governance aiming to decrease risks from emerging technologies by influencing the sequence in which they are developed. On this strategy, societies would strive to delay the development of harmful technologies and their applications, while accelerating the development of beneficial technologies, especially those that offer protection against the harmful ones.

<span class="mw-page-title-main">Future of Humanity Institute</span> Oxford interdisciplinary research centre

The Future of Humanity Institute (FHI) is an interdisciplinary research centre at the University of Oxford investigating big-picture questions about humanity and its prospects. It was founded in 2005 as part of the Faculty of Philosophy and the Oxford Martin School. Its director is philosopher Nick Bostrom, and its research staff include futurist Anders Sandberg and Giving What We Can founder Toby Ord.

<span class="mw-page-title-main">Global catastrophic risk</span> Potentially harmful worldwide events

A global catastrophic risk or a doomsday scenario is a hypothetical future event that could damage human well-being on a global scale, even endangering or destroying modern civilization. An event that could cause human extinction or permanently and drastically curtail humanity's existence or potential is known as an "existential risk."

In futurology, a singleton is a hypothetical world order in which there is a single decision-making agency at the highest level, capable of exerting effective control over its domain, and permanently preventing both internal and external threats to its supremacy. The term was first defined by Nick Bostrom.

In the field of artificial intelligence (AI) design, AI capability control proposals, also referred to as AI confinement, aim to increase our ability to monitor and control the behavior of AI systems, including proposed artificial general intelligences (AGIs), in order to reduce the danger they might pose if misaligned. However, capability control becomes less effective as agents become more intelligent and their ability to exploit flaws in human control systems increases, potentially resulting in an existential risk from AGI. Therefore, the Oxford philosopher Nick Bostrom and others recommend capability control methods only as a supplement to alignment methods.

<span class="mw-page-title-main">Future of Life Institute</span> International nonprofit research institute

Instrumental convergence is the hypothetical tendency for most sufficiently intelligent beings to pursue similar sub-goals, even if their ultimate goals are pretty different. More precisely, agents may pursue instrumental goals—goals which are made in pursuit of some particular end, but are not the end goals themselves—without ceasing, provided that their ultimate (intrinsic) goals may never be fully satisfied.

In January 2015, Stephen Hawking, Elon Musk, and dozens of artificial intelligence experts signed an open letter on artificial intelligence calling for research on the societal impacts of AI. The letter affirmed that society can reap great potential benefits from artificial intelligence, but called for concrete research on how to prevent certain potential "pitfalls": artificial intelligence has the potential to eradicate disease and poverty, but researchers must not create something which is unsafe or uncontrollable. The four-paragraph letter, titled "Research Priorities for Robust and Beneficial Artificial Intelligence: An Open Letter", lays out detailed research priorities in an accompanying twelve-page document.

<span class="mw-page-title-main">Existential risk from artificial general intelligence</span> Hypothesized risk to human existence

Existential risk from artificial general intelligence is the hypothesis that substantial progress in artificial general intelligence (AGI) could result in human extinction or another irreversible global catastrophe.

<i>Life 3.0</i> 2017 book by Max Tegmark on Artificial Intelligence

Life 3.0: Being Human in the Age of Artificial Intelligence is a 2017 book by Swedish-American cosmologist Max Tegmark . Life 3.0 discusses Artificial Intelligence (AI) and its impact on the future of life on Earth and beyond. The book discusses a variety of societal implications, what can be done to maximize the chances of a positive outcome, and potential futures for humanity, technology and combinations thereof.

<span class="mw-page-title-main">AI aftermath scenarios</span> Overview of AIs possible effects on the human state

Many scholars believe that advances in artificial intelligence, or AI, will eventually lead to a semi-apocalyptic post-scarcity economy where intelligent machines can outperform humans in nearly, if not every, domain. The questions of what such a world might look like, and whether specific scenarios constitute utopias or dystopias, are the subject of active debate.

Do You Trust This Computer? is a 2018 American documentary film directed by Chris Paine that outlines the benefits and especially the dangers of artificial intelligence. It features interviews with a range of prominent individuals relevant to AI, such as Ray Kurzweil, Elon Musk, Michal Kosinski, D. Scott Phoenix, and Jonathan Nolan. The film was directed by Chris Paine, known for Who Killed the Electric Car? (2006) and the subsequent followup, Revenge of the Electric Car (2011).

<i>Human Compatible</i> 2019 book by Stuart J. Russell

Human Compatible: Artificial Intelligence and the Problem of Control is a 2019 non-fiction book by computer scientist Stuart J. Russell. It asserts that the risk to humanity from advanced artificial intelligence (AI) is a serious concern despite the uncertainty surrounding future progress in AI. It also proposes an approach to the AI control problem.

Roko's basilisk is a thought experiment which states that an otherwise benevolent artificial superintelligence (AI) in the future would be incentivized to create a virtual reality simulation to torture anyone who knew of its potential existence but did not directly contribute to its advancement or development, in order to incentivise said advancement. It originated in a 2010 post at discussion board LessWrong, a technical forum focused on analytical rational enquiry. The thought experiment's name derives from the poster of the article (Roko) and the basilisk, a mythical creature capable of destroying enemies with its stare.

References

  1. ↑ "Superintelligent Swede snapped up by OUP". The Bookseller. 21 November 2013.
  2. 1 2 "Clever cogs". The Economist. 9 August 2014. Retrieved 9 August 2014.
  3. 1 2 3 Henderson, Caspar (17 July 2014). "Superintelligence by Nick Bostrom and A Rough Ride to the Future by James Lovelock – review". The Guardian. Retrieved 30 July 2014.
  4. 1 2 3 Khatchadourian, Raffi (2015). "The Doomsday Invention". The New Yorker. Retrieved 29 March 2020.
  5. 1 2 Adams, Tim (12 June 2016). "Nick Bostrom: 'We are like small children playing with a bomb'". The Observer . Retrieved 29 March 2020.
  6. 1 2 Chivers, Tom (10 August 2014). "Superintelligence by Nick Bostrom, review: 'a hard read'". The Telegraph. Retrieved 16 August 2014.
  7. ↑ "Best Selling Science Books". The New York Times. 8 September 2014. Retrieved 9 November 2014.
  8. ↑ Dean, James (5 August 2014). "Artificial intelligence 'may wipe out the human race'". The Times. Retrieved 5 August 2014.
  9. ↑ Augenbraun, Eliene (4 August 2014). "Elon Musk tweets Artificial Intelligence may be "more dangerous than nukes"". CBC News. Retrieved 5 August 2014.
  10. ↑ Bratton, Benjamin H. (23 February 2015). "Outing A.I.: Beyond the Turing Test". The New York Times. Retrieved 4 March 2015.
  11. ↑ Mack, Eric (28 January 2015). "Bill Gates Says You Should Worry About Artificial Intelligence". Forbes. Retrieved 19 February 2015.
  12. ↑ Lumby, Andrew (28 January 2015). "Bill Gates Is Worried About the Rise of the Machines". The Fiscal Times. Retrieved 19 February 2015.
  13. ↑ Kaiser Kuo (31 March 2015). "Baidu CEO Robin Li interviews Bill Gates and Elon Musk at the Boao Forum, March 29 2015". YouTube. Retrieved 8 April 2015.
  14. ↑ Black, Melia Russell, Julia. "He's played chess with Peter Thiel, sparred with Elon Musk and once, supposedly, stopped a plane crash: Inside Sam Altman's world, where truth is stranger than fiction". Business Insider. Retrieved 15 August 2023.{{cite web}}: CS1 maint: multiple names: authors list (link)
  15. ↑ Cookson, Clive (13 July 2014). "Superintelligence: Paths, Dangers, Strategies, by Nick Bostrom" . The Financial Times. Archived from the original on 6 August 2014. Retrieved 30 July 2014.
  16. ↑ Bailey, Ronald (12 September 2014). "Will Superintelligent Machines Destroy Humanity?". Reason . Retrieved 16 September 2014.
  17. ↑ Thomas, Joel (July 2015). "In defense of philosophy: a review of Nick Bostrom". Journal of Experimental & Theoretical Artificial Intelligence . 28 (6): 1089–1094. doi: 10.1080/0952813X.2015.1055829 .
  18. ↑ Richmond, Sheldon (8 July 2015). "Superintelligence: Paths, Dangers, Strategies". Philosophy . 91 (1): 125–130. doi:10.1017/S0031819115000340. S2CID   171005535.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.