Bayesian probability

Last updated

Bayesian probability is an interpretation of the concept of probability, in which, instead of frequency or propensity of some phenomenon, probability is interpreted as reasonable expectation [1] representing a state of knowledge [2] or as quantification of a personal belief. [3]


The Bayesian interpretation of probability can be seen as an extension of propositional logic that enables reasoning with hypotheses; [4] that is, with propositions whose truth or falsity is unknown. In the Bayesian view, a probability is assigned to a hypothesis, whereas under frequentist inference, a hypothesis is typically tested without being assigned a probability.

Bayesian probability belongs to the category of evidential probabilities; to evaluate the probability of a hypothesis, the Bayesian probabilist specifies a prior probability. This, in turn, is then updated to a posterior probability in the light of new, relevant data (evidence). [5] The Bayesian interpretation provides a standard set of procedures and formulae to perform this calculation.

The term Bayesian derives from the 18th-century mathematician and theologian Thomas Bayes, who provided the first mathematical treatment of a non-trivial problem of statistical data analysis using what is now known as Bayesian inference. [6] :131 Mathematician Pierre-Simon Laplace pioneered and popularized what is now called Bayesian probability. [6] :97–98

Bayesian methodology

Bayesian methods are characterized by concepts and procedures as follows:

Objective and subjective Bayesian probabilities

Broadly speaking, there are two interpretations of Bayesian probability. For objectivists, who interpret probability as an extension of logic, probability quantifies the reasonable expectation that everyone (even a "robot") who shares the same knowledge should share in accordance with the rules of Bayesian statistics, which can be justified by Cox's theorem. [2] [8] For subjectivists, probability corresponds to a personal belief. [3] Rationality and coherence allow for substantial variation within the constraints they pose; the constraints are justified by the Dutch book argument or by decision theory and de Finetti's theorem. [3] The objective and subjective variants of Bayesian probability differ mainly in their interpretation and construction of the prior probability.


The term Bayesian derives from Thomas Bayes (1702–1761), who proved a special case of what is now called Bayes' theorem in a paper titled "An Essay towards solving a Problem in the Doctrine of Chances". [9] In that special case, the prior and posterior distributions were beta distributions and the data came from Bernoulli trials. It was Pierre-Simon Laplace (1749–1827) who introduced a general version of the theorem and used it to approach problems in celestial mechanics, medical statistics, reliability, and jurisprudence. [10] Early Bayesian inference, which used uniform priors following Laplace's principle of insufficient reason, was called "inverse probability" (because it infers backwards from observations to parameters, or from effects to causes). [11] After the 1920s, "inverse probability" was largely supplanted by a collection of methods that came to be called frequentist statistics. [11]

In the 20th century, the ideas of Laplace developed in two directions, giving rise to objective and subjective currents in Bayesian practice. Harold Jeffreys' Theory of Probability (first published in 1939) played an important role in the revival of the Bayesian view of probability, followed by works by Abraham Wald (1950) and Leonard J. Savage (1954). The adjective Bayesian itself dates to the 1950s; the derived Bayesianism, neo-Bayesianism is of 1960s coinage. [12] [13] [14] In the objectivist stream, the statistical analysis depends on only the model assumed and the data analysed. [15] No subjective decisions need to be involved. In contrast, "subjectivist" statisticians deny the possibility of fully objective analysis for the general case.

In the 1980s, there was a dramatic growth in research and applications of Bayesian methods, mostly attributed to the discovery of Markov chain Monte Carlo methods and the consequent removal of many of the computational problems, and to an increasing interest in nonstandard, complex applications. [16] While frequentist statistics remains strong (as demonstrated by the fact that much of undergraduate teaching is based on it [17] ), Bayesian methods are widely accepted and used, e.g., in the field of machine learning. [18]

Justification of Bayesian probabilities

The use of Bayesian probabilities as the basis of Bayesian inference has been supported by several arguments, such as Cox axioms, the Dutch book argument, arguments based on decision theory and de Finetti's theorem.

Axiomatic approach

Richard T. Cox showed that Bayesian updating follows from several axioms, including two functional equations and a hypothesis of differentiability. [8] The assumption of differentiability or even continuity is controversial; Halpern found a counterexample based on his observation that the Boolean algebra of statements may be finite. [19] Other axiomatizations have been suggested by various authors with the purpose of making the theory more rigorous. [7]

Dutch book approach

The Dutch book argument was proposed by de Finetti; it is based on betting. A Dutch book is made when a clever gambler places a set of bets that guarantee a profit, no matter what the outcome of the bets. If a bookmaker follows the rules of the Bayesian calculus in the construction of his odds, a Dutch book cannot be made.

However, Ian Hacking noted that traditional Dutch book arguments did not specify Bayesian updating: they left open the possibility that non-Bayesian updating rules could avoid Dutch books. For example, Hacking writes [20] [21] "And neither the Dutch book argument, nor any other in the personalist arsenal of proofs of the probability axioms, entails the dynamic assumption. Not one entails Bayesianism. So the personalist requires the dynamic assumption to be Bayesian. It is true that in consistency a personalist could abandon the Bayesian model of learning from experience. Salt could lose its savour."

In fact, there are non-Bayesian updating rules that also avoid Dutch books (as discussed in the literature on "probability kinematics" [22] following the publication of Richard C. Jeffrey's rule, which is itself regarded as Bayesian [23] ). The additional hypotheses sufficient to (uniquely) specify Bayesian updating are substantial [24] and not universally seen as satisfactory. [25]

Decision theory approach

A decision-theoretic justification of the use of Bayesian inference (and hence of Bayesian probabilities) was given by Abraham Wald, who proved that every admissible statistical procedure is either a Bayesian procedure or a limit of Bayesian procedures. [26] Conversely, every Bayesian procedure is admissible. [27]

Personal probabilities and objective methods for constructing priors

Following the work on expected utility theory of Ramsey and von Neumann, decision-theorists have accounted for rational behavior using a probability distribution for the agent. Johann Pfanzagl completed the Theory of Games and Economic Behavior by providing an axiomatization of subjective probability and utility, a task left uncompleted by von Neumann and Oskar Morgenstern: their original theory supposed that all the agents had the same probability distribution, as a convenience. [28] Pfanzagl's axiomatization was endorsed by Oskar Morgenstern: "Von Neumann and I have anticipated ... [the question whether probabilities] might, perhaps more typically, be subjective and have stated specifically that in the latter case axioms could be found from which could derive the desired numerical utility together with a number for the probabilities (cf. p. 19 of The Theory of Games and Economic Behavior). We did not carry this out; it was demonstrated by Pfanzagl ... with all the necessary rigor". [29]

Ramsey and Savage noted that the individual agent's probability distribution could be objectively studied in experiments. Procedures for testing hypotheses about probabilities (using finite samples) are due to Ramsey (1931) and de Finetti (1931, 1937, 1964, 1970). Both Bruno de Finetti [30] [31] and Frank P. Ramsey [31] [32] acknowledge their debts to pragmatic philosophy, particularly (for Ramsey) to Charles S. Peirce. [31] [32]

The "Ramsey test" for evaluating probability distributions is implementable in theory, and has kept experimental psychologists occupied for a half century. [33] This work demonstrates that Bayesian-probability propositions can be falsified, and so meet an empirical criterion of Charles S. Peirce, whose work inspired Ramsey. (This falsifiability-criterion was popularized by Karl Popper. [34] [35] )

Modern work on the experimental evaluation of personal probabilities uses the randomization, blinding, and Boolean-decision procedures of the Peirce-Jastrow experiment. [36] Since individuals act according to different probability judgments, these agents' probabilities are "personal" (but amenable to objective study).

Personal probabilities are problematic for science and for some applications where decision-makers lack the knowledge or time to specify an informed probability-distribution (on which they are prepared to act). To meet the needs of science and of human limitations, Bayesian statisticians have developed "objective" methods for specifying prior probabilities.

Indeed, some Bayesians have argued the prior state of knowledge defines the (unique) prior probability-distribution for "regular" statistical problems; cf. well-posed problems. Finding the right method for constructing such "objective" priors (for appropriate classes of regular problems) has been the quest of statistical theorists from Laplace to John Maynard Keynes, Harold Jeffreys, and Edwin Thompson Jaynes. These theorists and their successors have suggested several methods for constructing "objective" priors (Unfortunately, it is not clear how to assess the relative "objectivity" of the priors proposed under these methods):

Each of these methods contributes useful priors for "regular" one-parameter problems, and each prior can handle some challenging statistical models (with "irregularity" or several parameters). Each of these methods has been useful in Bayesian practice. Indeed, methods for constructing "objective" (alternatively, "default" or "ignorance") priors have been developed by avowed subjective (or "personal") Bayesians like James Berger (Duke University) and José-Miguel Bernardo (Universitat de València), simply because such priors are needed for Bayesian practice, particularly in science. [37] The quest for "the universal method for constructing priors" continues to attract statistical theorists. [37]

Thus, the Bayesian statistician needs either to use informed priors (using relevant expertise or previous data) or to choose among the competing methods for constructing "objective" priors.

See also

Related Research Articles

Frequentist probability

Frequentist probability or frequentism is an interpretation of probability; it defines an event's probability as the limit of its relative frequency in many trials. Probabilities can be found by a repeatable objective process. The continued use of frequentist methods in scientific inference, however, has been called into question.

The word probability has been used in a variety of ways since it was first applied to the mathematical study of games of chance. Does probability measure the real, physical, tendency of something to occur, or is it a measure of how strongly one believes it will occur, or does it draw on both these elements? In answering such questions, mathematicians interpret the probability values of probability theory.

Statistical inference Process of using data analysis

Statistical inference is the process of using data analysis to infer properties of an underlying distribution of probability. Inferential statistical analysis infers properties of a population, for example by testing hypotheses and deriving estimates. It is assumed that the observed data set is sampled from a larger population.

Bayesian inference is a method of statistical inference in which Bayes' theorem is used to update the probability for a hypothesis as more evidence or information becomes available. Bayesian inference is an important technique in statistics, and especially in mathematical statistics. Bayesian updating is particularly important in the dynamic analysis of a sequence of data. Bayesian inference has found application in a wide range of activities, including science, engineering, philosophy, medicine, sport, and law. In the philosophy of decision theory, Bayesian inference is closely related to subjective probability, often called "Bayesian probability".

Bruno de Finetti Italian mathematician (1906–1985)

Bruno de Finetti was an Italian probabilist statistician and actuary, noted for the "operational subjective" conception of probability. The classic exposition of his distinctive theory is the 1937 "La prévision: ses lois logiques, ses sources subjectives," which discussed probability founded on the coherence of betting odds and the consequences of exchangeability.

Bayesian statistics is a theory in the field of statistics based on the Bayesian interpretation of probability where probability expresses a degree of belief in an event. The degree of belief may be based on prior knowledge about the event, such as the results of previous experiments, or on personal beliefs about the event. This differs from a number of other interpretations of probability, such as the frequentist interpretation that views probability as the limit of the relative frequency of an event after many trials.

In mathematical optimization and decision theory, a loss function or cost function is a function that maps an event or values of one or more variables onto a real number intuitively representing some "cost" associated with the event. An optimization problem seeks to minimize a loss function. An objective function is either a loss function or its opposite, in which case it is to be maximized. The loss function could include terms from several levels of the hierarchy.

Decision theory is a branch of applied probability theory concerned with the theory of making decisions based on assigning probabilities to various factors and assigning numerical consequences to the outcome.

In Bayesian statistical inference, a prior probability distribution, often simply called the prior, of an uncertain quantity is the probability distribution that would express one's beliefs about this quantity before some evidence is taken into account. For example, the prior could be the probability distribution representing the relative proportions of voters who will vote for a particular politician in a future election. The unknown quantity may be a parameter of the model or a latent variable rather than an observable variable.

In decision theory, subjective expected utility is the attractiveness of an economic opportunity as perceived by a decision-maker in the presence of risk. Characterizing the behavior of decision-makers as using subjective expected utility was promoted and axiomatized by L. J. Savage in 1954 following previous work by Ramsey and von Neumann. The theory of subjective expected utility combines two subjective concepts: first, a personal utility function, and second a personal probability distribution.

Imprecise probability generalizes probability theory to allow for partial probability specifications, and is applicable when information is scarce, vague, or conflicting, in which case a unique probability distribution may be hard to identify. Thereby, the theory aims to represent the available knowledge more accurately. Imprecision is useful for dealing with expert elicitation, because:

In Bayesian statistics, a credible interval is an interval within which an unobserved parameter value falls with a particular probability. It is an interval in the domain of a posterior probability distribution or a predictive distribution. The generalisation to multivariate problems is the credible region. Credible intervals are analogous to confidence intervals in frequentist statistics, although they differ on a philosophical basis: Bayesian intervals treat their bounds as fixed and the estimated parameter as a random variable, whereas frequentist confidence intervals treat their bounds as random variables and the parameter as a fixed value. Also, Bayesian credible intervals use knowledge of the situation-specific prior distribution, while the frequentist confidence intervals do not.

Fiducial inference is one of a number of different types of statistical inference. These are rules, intended for general application, by which conclusions can be drawn from samples of data. In modern statistical practice, attempts to work with fiducial inference have fallen out of fashion in favour of frequentist inference, Bayesian inference and decision theory. However, fiducial inference is important in the history of statistics since its development led to the parallel development of concepts and tools in theoretical statistics that are widely used. Some current research in statistical methodology is either explicitly linked to fiducial inference or is closely connected to it.

Bayes linear statistics is a subjectivist statistical methodology and framework. Traditional subjective Bayesian analysis is based upon fully specified probability distributions, which are very difficult to specify at the necessary level of detail. Bayes linear analysis attempts to solve this problem by developing theory and practise for using partially specified probability models. Bayes linear in its current form has been primarily developed by Michael Goldstein. Mathematically and philosophically it extends Bruno de Finetti's Operational Subjective approach to probability and statistics.

Statistics, in the modern sense of the word, began evolving in the 18th century in response to the novel needs of industrializing sovereign states. The evolution of statistics was, in particular, intimately connected with the development of European states following the peace of Westphalia (1648), and with the development of probability theory, which put statistics on a firm theoretical basis.

The foundations of statistics concern the epistemological debate in statistics over how one should conduct inductive inference from data. Among the issues considered in statistical inference are the question of Bayesian inference versus frequentist inference, the distinction between Fisher's "significance testing" and Neyman–Pearson "hypothesis testing", and whether the likelihood principle should be followed. Some of these issues have been debated for up to 200 years without resolution.

Frequentist inference is a type of statistical inference based in frequentist probability, which treats “probability” in equivalent terms to “frequency” and draws conclusions from sample-data by means of emphasizing the frequency or proportion of findings in the data. Frequentist-inference underlies frequentist statistics, in which the well-established methodologies of statistical hypothesis testing and confidence intervals are founded.

Bayesian econometrics is a branch of econometrics which applies Bayesian principles to economic modelling. Bayesianism is based on a degree-of-belief interpretation of probability, as opposed to a relative-frequency interpretation.

Bayesian inference in marketing

In marketing, Bayesian inference allows for decision making and market research evaluation under uncertainty and with limited data.


  1. Cox, R.T. (1946). "Probability, Frequency, and Reasonable Expectation". American Journal of Physics. 14 (1): 1–10. Bibcode:1946AmJPh..14....1C. doi:10.1119/1.1990764.
  2. 1 2 Jaynes, E.T. (1986). "Bayesian Methods: General Background". In Justice, J. H. (ed.). Maximum-Entropy and Bayesian Methods in Applied Statistics. Cambridge: Cambridge University Press. CiteSeerX .
  3. 1 2 3 de Finetti, Bruno (2017). Theory of Probability: A critical introductory treatment. Chichester: John Wiley & Sons Ltd. ISBN   9781119286370.
  4. Hailperin, Theodore (1996). Sentential Probability Logic: Origins, Development, Current Status, and Technical Applications. London: Associated University Presses. ISBN   0934223459.
  5. Paulos, John Allen (5 August 2011). "The Mathematics of Changing Your Mind [by Sharon Bertsch McGrayne]". Book Review. New York Times. Archived from the original on 2022-01-01. Retrieved 2011-08-06.
  6. 1 2 Stigler, Stephen M. (March 1990). The history of statistics. Harvard University Press. ISBN   9780674403413.
  7. 1 2 Dupré, Maurice J.; Tipler, Frank J. (2009). "New axioms for rigorous Bayesian probability". Bayesian Analysis. 4 (3): 599–606. CiteSeerX . doi:10.1214/09-BA422.
  8. 1 2 Cox, Richard T. (1961). The algebra of probable inference (Reprint ed.). Baltimore, MD; London, UK: Johns Hopkins Press; Oxford University Press [distributor]. ISBN   9780801869822.
  9. McGrayne, Sharon Bertsch (2011). The Theory that Would not Die . [ 10  ], p. 10, at Google Books.
  10. Stigler, Stephen M. (1986). "Chapter 3" . The History of Statistics. Harvard University Press.
  11. 1 2 Fienberg, Stephen. E. (2006). "When did Bayesian Inference become "Bayesian"?" (PDF). Bayesian Analysis. 1 (1): 5, 1–40. doi: 10.1214/06-BA101 . Archived from the original (PDF) on 10 September 2014.
  12. Harris, Marshall Dees (1959). "Recent developments of the so-called Bayesian approach to statistics". Agricultural Law Center. Legal-Economic Research. University of Iowa: 125 (fn. #52), 126. The works of Wald, Statistical Decision Functions (1950) and Savage, The Foundation of Statistics (1954) are commonly regarded starting points for current Bayesian approaches
  13. Annals of the Computation Laboratory of Harvard University. Vol. 31. 1962. p. 180. This revolution, which may or may not succeed, is neo-Bayesianism. Jeffreys tried to introduce this approach, but did not succeed at the time in giving it general appeal.
  14. Kempthorne, Oscar (1967). The Classical Problem of Inference—Goodness of Fit. Fifth Berkeley Symposium on Mathematical Statistics and Probability. p. 235. It is curious that even in its activities unrelated to ethics, humanity searches for a religion. At the present time, the religion being 'pushed' the hardest is Bayesianism.
  15. Bernardo, J.M. (2005). "Reference analysis". Bayesian Thinking - Modeling and Computation. Handbook of Statistics. Vol. 25. pp. 17–90. doi:10.1016/S0169-7161(05)25002-2. ISBN   9780444515391.
  16. Wolpert, R.L. (2004). "A conversation with James O. Berger". Statistical Science. 9: 205–218. doi: 10.1214/088342304000000053 .
  17. Bernardo, José M. (2006). A Bayesian mathematical statistics primer (PDF). ICOTS-7. Bern.
  18. Bishop, C.M. (2007). Pattern Recognition and Machine Learning. Springer.
  19. Halpern, J. (1999). "A counterexample to theorems of Cox and Fine" (PDF). Journal of Artificial Intelligence Research. 10: 67–85. doi: 10.1613/jair.536 . S2CID   1538503.
  20. Hacking (1967), Section 3, page 316
  21. Hacking (1988, page 124)
  22. Skyrms, Brian (1 January 1987). "Dynamic Coherence and Probability Kinematics". Philosophy of Science. 54 (1): 1–20. CiteSeerX . doi:10.1086/289350. JSTOR   187470.
  23. Joyce, James (30 September 2003). "Bayes' Theorem". The Stanford Encyclopedia of Philosophy.
  24. Fuchs, Christopher A.; Schack, Rüdiger (1 January 2012). Ben-Menahem, Yemima; Hemmo, Meir (eds.). Probability in Physics . The Frontiers Collection. Springer Berlin Heidelberg. pp.  233–247. arXiv: 1103.5950 . doi:10.1007/978-3-642-21329-8_15. ISBN   9783642213281. S2CID   119215115.
  25. van Frassen, Bas (1989). Laws and Symmetry. Oxford University Press. ISBN   0-19-824860-1.
  26. Wald, Abraham (1950). Statistical Decision Functions. Wiley.
  27. Bernardo, José M.; Smith, Adrian F.M. (1994). Bayesian Theory. John Wiley. ISBN   0-471-92416-4.
  28. Pfanzagl (1967, 1968)
  29. Morgenstern (1976, page 65)
  30. Galavotti, Maria Carla (1 January 1989). "Anti-Realism in the Philosophy of Probability: Bruno de Finetti's Subjectivism". Erkenntnis. 31 (2/3): 239–261. doi:10.1007/bf01236565. JSTOR   20012239. S2CID   170802937.
  31. 1 2 3 Galavotti, Maria Carla (1 December 1991). "The notion of subjective probability in the work of Ramsey and de Finetti". Theoria. 57 (3): 239–259. doi:10.1111/j.1755-2567.1991.tb00839.x. ISSN   1755-2567.
  32. 1 2 Dokic, Jérôme; Engel, Pascal (2003). Frank Ramsey: Truth and Success. Routledge. ISBN   9781134445936.
  33. Davidson et al. (1957)
  34. Thornton, Stephen (7 August 2018). "Karl Popper". Stanford Encyclopedia of Philosophy.
  35. Popper, Karl (2002) [1959]. The Logic of Scientific Discovery (2nd ed.). Routledge. p. 57. ISBN   0-415-27843-0 via Google Books. (translation of 1935 original, in German).
  36. Peirce & Jastrow (1885)
  37. 1 2 Bernardo, J. M. (2005). "Reference Analysis". In Dey, D.K.; Rao, C. R. (eds.). Handbook of Statistics (PDF). Vol. 25. Amsterdam: Elsevier. pp. 17–90.