Pascal's mugging

Last updated

In philosophy, Pascal's mugging is a thought experiment demonstrating a problem in expected utility maximization. A rational agent should choose actions whose outcomes, when weighted by their probability, have higher utility. But some very unlikely outcomes may have very great utilities, and these utilities can grow faster than the probability diminishes. Hence the agent should focus more on vastly improbable cases with implausibly high rewards; this leads first to counter-intuitive choices, and then to incoherence as the utility of every choice becomes unbounded.

Contents

The name refers to Pascal's Wager, but unlike the wager, it does not require infinite rewards. [1] This sidesteps many objections to the Pascal's Wager dilemma that are based on the nature of infinity. [2]

Problem statement

The term "Pascal's mugging" to refer to this problem was originally coined by Eliezer Yudkowsky in the LessWrong forum. [3] [2] Philosopher Nick Bostrom later elaborated the thought experiment in the form of a fictional dialogue. [2] Subsequently, other authors published their own sequels to the events of this first dialogue, adopting the same literary style. [4] [5]

In Bostrom's description, [2] Blaise Pascal is accosted by a mugger who has forgotten their weapon. However, the mugger proposes a deal: the philosopher gives them his wallet, and in exchange the mugger will return twice the amount of money tomorrow. Pascal declines, pointing out that it is unlikely the deal will be honoured. The mugger then continues naming higher rewards, pointing out that even if it is just one chance in 1000 that they will be honourable, it would make sense for Pascal to make a deal for a 2000 times return. Pascal responds that the probability of that high return is even lower than one in 1000. The mugger argues back that for any low but strictly greater than 0 probability of being able to pay back a large amount of money (or pure utility) there exists a finite amount that makes it rational to take the bet. In one example, the mugger succeeds by promising Pascal 1,000 quadrillion happy days of life. Convinced by the argument, Pascal gives the mugger the wallet.

In one of Yudkowsky's examples, the mugger succeeds by saying "give me five dollars, or I'll use my magic powers from outside the Matrix to run a Turing machine that simulates and kills people". Here, the number uses Knuth's up-arrow notation; writing the number out in base 10 would require enormously more writing material than there are atoms in the known universe. [3]

The supposed paradox results from two inconsistent views. On the one side, by multiplying an expected utility calculation, assuming loss of five dollars to be valued at , loss of a life to be valued at , and probability that the mugger is telling the truth at , the solution is to give the money if and only if . Assuming that is higher than , so long as is higher than , which is assumed to be true, [note 1] it is considered rational to pay the mugger. On the other side of the argument, paying the mugger is intuitively irrational due to its exploitability. If the person being mugged agrees to this sequence of logic, then they can be exploited repeatedly for all of their money, resulting in a Dutch-book, which is typically considered irrational. Views on which of these arguments is logically correct differ. [3]

Moreover, in many reasonable-seeming decision systems, Pascal's mugging causes the expected utility of any action to fail to converge, as an unlimited chain of successively dire scenarios similar to Pascal's mugging would need to be factored in. [7] [8]

Some of the arguments concerning this paradox affect not only the expected utility maximization theory, but may also apply to other theoretical systems, such as consequentialist ethics, for example. [note 2]

Consequences and remedies

Philosopher Nick Bostrom argues that Pascal's mugging, like Pascal's wager, suggests that giving a superintelligent artificial intelligence a flawed decision theory could be disastrous. [10] Pascal's mugging may also be relevant when considering low-probability, high-stakes events such as existential risk or charitable interventions with a low probability of success but extremely high rewards. Common sense seems to suggest that spending effort on too unlikely scenarios is irrational.

One advocated remedy might be to only use bounded utility functions: rewards cannot be arbitrarily large. [7] [11] Another approach is to use Bayesian reasoning to (qualitatively) judge the quality of evidence and probability estimates rather than naively calculate expectations. [6] Other approaches are to penalize the prior probability of hypotheses that argue that we are in a surprisingly unique position to affect large numbers of other people who cannot symmetrically affect us, [note 3] reject providing the probability of a payout first, [15] or abandon quantitative decision procedures in the presence of extremely large risks. [8]

See also

Notes

  1. While it may seem very intuitive that the probability cannot be as small as , this is not necessarily true. As a notable example, it is false in Bayesian models for this problem: [6] There, the prior probability of the mugger’s claim being correct decreases with the extraordinariness of his claim. If his claim is as extraordinary as in this case, its probability will also be extraordinarily small in a Bayesian model. Furthermore, a frequentist may estimate the probability of the mugger's threat being realized as equal to 0, since no realization of such an extraordinary threat has ever been observed.
  2. A ‘consequentialist version’ of Pascal’s mugging was proposed by Bradley Monton as follows: A strange person hands you a baby and asks you to torture it, telling you that the torture will prevent significant unjust suffering of a large number of sentient creatures in some distant galaxy. While the probability is very low that torturing the baby will prevent the suffering, as long as the strange person makes the number of claimed distant-galaxy creatures high enough, according to consequentialism equipped with probabilistic models in which the probability that the torturing prevents the suffering does not decrease if the number of galaxies is increased (notably Bayesian models do have a decreasing probability for increasing numbers of galaxies because the prior will necessarily concentrate around some fixed number of galaxies), you should torture the baby. [9]
  3. This 'leverage penalty' was first proposed by Robin Hanson in a comment on Yudkowsky's original statement of the problem; [12] [13] Yudkowsky noted that this would imply refusing to believe in theories implying we can affect vastly many others, even in the face of what might otherwise look like overwhelming observational evidence for the theory. [14]

Related Research Articles

Bayesian probability is an interpretation of the concept of probability, in which, instead of frequency or propensity of some phenomenon, probability is interpreted as reasonable expectation representing a state of knowledge or as quantification of a personal belief.

<span class="mw-page-title-main">Eliezer Yudkowsky</span> American AI researcher and writer (born 1979)

Eliezer S. Yudkowsky is an American artificial intelligence researcher and writer on decision theory and ethics, best known for popularizing ideas related to friendly artificial intelligence. He is the founder of and a research fellow at the Machine Intelligence Research Institute (MIRI), a private research nonprofit based in Berkeley, California. His work on the prospect of a runaway intelligence explosion influenced philosopher Nick Bostrom's 2014 book Superintelligence: Paths, Dangers, Strategies.

<span class="mw-page-title-main">Prospect theory</span> Theory of behavioral economics

Prospect theory is a theory of behavioral economics, judgment and decision making that was developed by Daniel Kahneman and Amos Tversky in 1979. The theory was cited in the decision to award Kahneman the 2002 Nobel Memorial Prize in Economics.

<span class="mw-page-title-main">Pascal's wager</span> Argument that posits human beings bet with their lives that God either exists or does not

Pascal's wager is a philosophical argument advanced by Blaise Pascal (1623–1662), seventeenth-century French mathematician, philosopher, physicist, and theologian. This argument posits that individuals essentially engage in a life-defining gamble regarding the belief in the existence of God.

<span class="mw-page-title-main">Decision theory</span> Branch of applied probability theory

Decision theory or the theory of rational choice is a branch of probability, economics, and analytic philosophy that uses the tools of expected utility and probability to model how individuals should behave rationally under uncertainty. It differs from the cognitive and behavioral sciences in that it is prescriptive and concerned with identifying optimal decisions for a rational agent, rather than describing how people really do make decisions. Despite this, the field is extremely important to the study of real human behavior by social scientists, as it lays the foundations for the rational agent models used to mathematically model and analyze individuals in fields such as sociology, economics, criminology, cognitive science, and political science.

<span class="mw-page-title-main">St. Petersburg paradox</span> Paradox involving a game with repeated coin flipping

The St. Petersburg paradox or St. Petersburg lottery is a paradox involving the game of flipping a coin where the expected payoff of the lottery game is infinite but nevertheless seems to be worth only a very small amount to the participants. The St. Petersburg paradox is a situation where a naïve decision criterion that takes only the expected value into account predicts a course of action that presumably no actual person would be willing to take. Several resolutions to the paradox have been proposed, including the impossible amount of money a casino would need to continue the game indefinitely.

The expected utility hypothesis is a foundational assumption in mathematical economics concerning decision making under uncertainty. It postulates that rational agents maximize utility, meaning the subjective desirability of their actions. Rational choice theory, a cornerstone of microeconomics, builds this postulate to model aggregate social behaviour.

In decision theory, subjective expected utility is the attractiveness of an economic opportunity as perceived by a decision-maker in the presence of risk. Characterizing the behavior of decision-makers as using subjective expected utility was promoted and axiomatized by L. J. Savage in 1954 following previous work by Ramsey and von Neumann. The theory of subjective expected utility combines two subjective concepts: first, a personal utility function, and second a personal probability distribution.

In decision theory, economics, and probability theory, the Dutch book arguments are a set of results showing that agents must satisfy the axioms of rational choice to avoid a kind of self-contradiction called a Dutch book. A Dutch book or money pump is a set of bets that ensures a guaranteed loss, i.e. the gambler will lose money no matter what happens. A set of beliefs and preferences is called coherent if it cannot result in a Dutch book.

In statistical decision theory, an admissible decision rule is a rule for making a decision such that there is no other rule that is always "better" than it, in the precise sense of "better" defined below. This concept is analogous to Pareto efficiency.

The self-indication assumption doomsday argument rebuttal is an objection to the doomsday argument by arguing that the chance of being born is not one, but is an increasing function of the number of people who will be born.

In game theory, a Bayesian game is a strategic decision-making model which assumes players have incomplete information. Players hold private information relevant to the game, meaning that the payoffs are not common knowledge. Bayesian games model the outcome of player interactions using aspects of Bayesian probability. They are notable because they allowed, for the first time in game theory, for the specification of the solutions to games with incomplete information.

In decision theory, the Ellsberg paradox is a paradox in which people's decisions are inconsistent with subjective expected utility theory. John Maynard Keynes published a version of the paradox in 1921. Daniel Ellsberg popularized the paradox in his 1961 paper, "Risk, Ambiguity, and the Savage Axioms". It is generally taken to be evidence of ambiguity aversion, in which a person tends to prefer choices with quantifiable risks over those with unknown, incalculable risks.

<span class="mw-page-title-main">Two envelopes problem</span> Puzzle in logic and mathematics

The two envelopes problem, also known as the exchange paradox, is a paradox in probability theory. It is of special interest in decision theory and for the Bayesian interpretation of probability theory. It is a variant of an older problem known as the necktie paradox. The problem is typically introduced by formulating a hypothetical challenge like the following example:

Imagine you are given two identical envelopes, each containing money. One contains twice as much as the other. You may pick one envelope and keep the money it contains. Having chosen an envelope at will, but before inspecting it, you are given the chance to switch envelopes. Should you switch?

The Allais paradox is a choice problem designed by Maurice Allais to show an inconsistency of actual observed choices with the predictions of expected utility theory. The Allais paradox demonstrates that individuals rarely make rational decisions consistently when required to do so immediately. The independence axiom of expected utility theory, which requires that the preferences of an individual should not change when altering two lotteries by equal proportions, was proven to be violated by the paradox.

The rank-dependent expected utility model is a generalized expected utility model of choice under uncertainty, designed to explain the behaviour observed in the Allais paradox, as well as for the observation that many people both purchase lottery tickets and insure against losses.

Evidential decision theory (EDT) is a school of thought within decision theory which states that, when a rational agent is confronted with a set of possible actions, one should select the action with the highest news value, that is, the action which would be indicative of the best outcome in expectation if one received the "news" that it had been taken. In other words, it recommends to "do what you most want to learn that you will do."

In expected utility theory, a lottery is a discrete distribution of probability on a set of states of nature. The elements of a lottery correspond to the probabilities that each of the states of nature will occur,. Much of the theoretical analysis of choice under uncertainty involves characterizing the available choices in terms of lotteries.

Bayesian epistemology is a formal approach to various topics in epistemology that has its roots in Thomas Bayes' work in the field of probability theory. One advantage of its formal method in contrast to traditional epistemology is that its concepts and theorems can be defined with a high degree of precision. It is based on the idea that beliefs can be interpreted as subjective probabilities. As such, they are subject to the laws of probability theory, which act as the norms of rationality. These norms can be divided into static constraints, governing the rationality of beliefs at any moment, and dynamic constraints, governing how rational agents should change their beliefs upon receiving new evidence. The most characteristic Bayesian expression of these principles is found in the form of Dutch books, which illustrate irrationality in agents through a series of bets that lead to a loss for the agent no matter which of the probabilistic events occurs. Bayesians have applied these fundamental principles to various epistemological topics but Bayesianism does not cover all topics of traditional epistemology. The problem of confirmation in the philosophy of science, for example, can be approached through the Bayesian principle of conditionalization by holding that a piece of evidence confirms a theory if it raises the likelihood that this theory is true. Various proposals have been made to define the concept of coherence in terms of probability, usually in the sense that two propositions cohere if the probability of their conjunction is higher than if they were neutrally related to each other. The Bayesian approach has also been fruitful in the field of social epistemology, for example, concerning the problem of testimony or the problem of group belief. Bayesianism still faces various theoretical objections that have not been fully solved.

Roko's basilisk is a thought experiment which states that an otherwise benevolent artificial superintelligence (AI) in the future would be incentivized to create a virtual reality simulation to torture anyone who knew of its potential existence but did not directly contribute to its advancement or development, in order to incentivize said advancement. It originated in a 2010 post at discussion board LessWrong, a technical forum focused on analytical rational enquiry. The thought experiment's name derives from the poster of the article (Roko) and the basilisk, a mythical creature capable of destroying enemies with its stare.

References

Citations

  1. Häggström 2016, p. 82.
  2. 1 2 3 4 Bostrom 2009.
  3. 1 2 3 Yudkowsky 2007.
  4. Balfour 2021.
  5. Russell 2022.
  6. 1 2 Holden Karnofsky, Why We Can’t Take Expected Value Estimates Literally (Even When They’re Unbiased). GiveWell Blog August 18, 2011 http://blog.givewell.org/2011/08/18/why-we-cant-take-expected-value-estimates-literally-even-when-theyre-unbiased/
  7. 1 2 De Blanc, Peter. Convergence of Expected Utilities with Algorithmic Probability Distributions (2007), arXiv : 0712.4318
  8. 1 2 Kieran Marray, Dealing With Uncertainty in Ethical Calculations of Existential Risk, Presented at The Economic and Social Research Council Climate Ethics and Climate Economics Workshop Series: Workshop Five - Risk and the Culture of Science, May 2016 http://www.nottingham.ac.uk/climateethicseconomics/documents/papers-workshop-5/marray.pdf
  9. Monton, Bradley (2019). "How to Avoid Maximizing Expected Utility". Philosophers' Imprint. 19 (18): 1–25. hdl:2027/spo.3521354.0019.018.
  10. Bostrom, Nick (2014). "Choosing the Criteria for Choosing". Superintelligence: Paths, Dangers, Strategies . Oxford: Oxford University Press. ISBN   978-0199678112. "Decision Theory" section.
  11. Cowen, Tyler; High, Jack (1988). "Time, Bounded Utility, and the St. Petersburg Paradox". Theory and Decision. 25 (3): 219–223. doi:10.1007/BF00133163. S2CID   120584258.
  12. Robin Hanson (21 October 2007), comment on Eliezer Yudkowsky's "Pascal's Mugging: Tiny Probabilities of Vast Utilities", LessWrong: "People have been talking about assuming that states with many people hurt have a low (prior) probability. It might be more promising to assume that states with many people hurt have a low correlation with what any random person claims to be able to effect."
  13. Tomasik, Brian (June 2016). "How the Simulation Argument Dampens Future Fanaticism" (PDF). Center on Long-Term Risk. pp. 3–4. Archived (PDF) from the original on 2021-11-23.
  14. Yudkowsky, Eliezer (2013-05-08). "Pascal's Muggle: Infinitesimal Priors and Strong Evidence". LessWrong . Archived from the original on 2016-06-27.
  15. Baumann, Peter (2009). "Counting on numbers" (PDF). Analysis. 69 (3): 446–448. doi:10.1093/analys/anp061. JSTOR   40607656. Archived (PDF) from the original on 2019-11-21.

Sources