Belief aggregation

Last updated

Belief aggregation, [1] also called risk aggregation, [2] opinion aggregation [3] or probabilistic opinion pooling, [4] is a process in which different probability distributions, produced by different experts, are combined to yield a single probability distribution.

Contents

Background

Expert opinions are often uncertain. Rather than saying e.g. "it will rain tomorrow", a weather expert may say "it will rain with probability 70% and be sunny with probability 30%". Such a statement is called a belief. Different experts may have different beliefs; for example, a different weather expert may say "it will rain with probability 60% and be sunny with probability 40%". In other words, each expert has a subjeciive probability distribution over a given set of outcomes.

A belief aggregation rule is a function that takes as input two or more probability distributions over the same set of outcomes, and returns a single probability distribution over the same space.

Applications

Documented applications of belief aggregation include:

During COVID-19, the European Academy of Neurology developed an ad-hoc three-round voting method to aggregate expert opinions and reach a consensus. [9]

Common rules

Common belief aggregation rules include:

Dietrich and List [4] present axiomatic characterizations of each class of rules. They argue that that linear aggregation can be justified “procedurally” but not “epistemically”, while the other two rules can be justified epistemically. Geometric aggregation is justified when the experts' beliefs are based on the same information, and multiplicative aggregation is justified when the experts' beliefs are based on private information.

Properties of belief aggregation rules

A belief aggregation rule should arguably satisfy some desirable properties, or axioms:

Truthful aggregation rules with money

Most literature on belief aggregation assumes that the experts report their beliefs honestly, as their main goal is to help the decision-maker get to the truth. In practice, experts may have strategic incentives. For example, the FDA uses advisory committees, and there have been controversies due to conflicts of interests within these committees. [11] Therefore, a truthful mechanism for belief aggregation could be useful.

In some settings, it is possible to pay the experts a certain sum of money, depending both on their expressed belief and on the realized outcome. Careful design of the payment function (often called a "scoring rule") can lead to a truthful mechanism. Various truthful scoring rules exist. [12] [13] [14] [15]

Truthful aggregation rules without money

In some settings, monetary transfers are not possible. For example, the realized outcome may happen in the far future, or a wrong decision may be catastrophic.

To develop truthful mechanisms, one must make assumptions about the experts' preferences over the set of accepted probability-distributions. If the space of possible preferences is too rich, then strong impossibility results imply that the only truthful mechanism is the dictatorship mechanism (see Gibbard–Satterthwaite theorem).

Single-peaked preferences

A useful domain restriction is that the experts have single-peaked preferences. An aggregation rule is called one-dimensional strategyproof(1D-SP) if whenever all experts have single-peaked preferences, and submit their peaks to the aggregation rule, no expert can impose a strictly better aggregated distribution by reporting a false peak. An equivalent property is called uncompromisingness: [16] it says that, if the belief of expert i is smaller than the aggregate distribution, and i changes his report, then the aggregate distribution will be weakly larger; and vice-versa.

Moulin [17] proved a characterization of all 1D-SP rules, as well as the following two characterizations:

Jennings, Laraki, Puppe and Varloot [18] present new characterizations of strategyproof mechanisms with single-peaked preferences.

Single-peaked preferences of the pdf

A further restriction of the single-peaked domain is that agents have single-peaked preferences with L1 metric on the probability density function. That is: for each agent i, there is an "ideal" probability distribution pi, and his utility from a selected probability distribution p* is minus the L1 distance between pi and p*. An aggregation rule is called L1-metric-strategyproof(L1-metric-SP) if whenever all experts have single-peaked preferences with L1 metric, and submit their peaks to the aggregation rule, no expert can impose a strictly better aggregated distribution by reporting a false peak. Several L1-metric-SP aggregation rules were suggested in the context of budget-proposal aggregation:

However, such preferences may not be a good fit for belief aggregation, as they are neutral - they do not distinguish between different outcomes. For example, suppose there are three outcomes, and the expert's belief pi assigns 100% to outcome 1. Then, the L1 metric between pi and "100% outcome 2" is 2, and the L1 metric between pi and "100% outcome 3" is 2 too. The same is true for any neutral metric. This makes sense when 1,2,3 are budget items. However, if these outcomes describe the potential strength of an earthquake in the Richter scale, then the distance between pi to "100% outcome 2" should be much smaller than the distance to "100% outcome 3".

Single-peaked preferences on the cdf

Varloot and Laraki [1] study a different preference domain, in which the outcomes are linearly ordered, and the preferences are single-peaked in the space of cumulative distribution function (cdf). That is: each agent i has an ideal cumulative distribution function ci, and his utility depends negatively on the distance between ci and the accepted distribution c*. They define a new concept called level-strategyproofness (Level-SP), which is relevant when society's decision is based on the question of whether the probability of some event is above or below a given threshold. Level-SP provably implies strategyproofness for a rich class of cdf-single-peaked preferences. They characterize two new aggregation rules:

Other results include:

Software

ANDURIL [21] is a MATLAB toolbox for belief aggregation.

See also

Further reading

Several books on related topics are available. [22] [23] [3]

Related Research Articles

<span class="mw-page-title-main">Probability distribution</span> Mathematical function for the probability a given outcome occurs in an experiment

In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomenon in terms of its sample space and the probabilities of events.

The expected utility hypothesis is a foundational assumption in mathematical economics concerning decision making under uncertainty. It postulates that rational agents maximize utility, meaning the subjective desirability of their actions. Rational choice theory, a cornerstone of microeconomics, builds this postulate to model aggregate social behaviour.

In gambling, economics, and the philosophy of probability, a Dutch book or lock is a set of odds and bets that ensures a guaranteed profit. It is generally used as a thought experiment to motivate Von Neumann–Morgenstern axioms or the axioms of probability by showing they are equivalent to philosophical coherence or Pareto efficiency.

In mechanism design, a strategyproof (SP) mechanism is a game in which each player has a weakly-dominant strategy, so that no player can gain by "spying" over the other players to know what they are going to play. When the players have private information, and the strategy space of each player consists of the possible information values, a truthful mechanism is a game in which revealing the true information is a weakly-dominant strategy for each player. An SP mechanism is also called dominant-strategy-incentive-compatible (DSIC), to distinguish it from other kinds of incentive compatibility.

In social choice theory, a dictatorship mechanism is a rule by which, among all possible alternatives, the results of voting mirror a single pre-determined person's preferences, without consideration of the other voters. Dictatorship by itself is not considered a good mechanism in practice, but it is theoretically important: by Arrow's impossibility theorem, when there are at least three alternatives, dictatorship is the only ranked voting electoral system that satisfies unrestricted domain, Pareto efficiency, and independence of irrelevant alternatives. Similarly, by Gibbard's theorem, when there are at least three alternatives, dictatorship is the only strategyproof rule.

Single-peaked preferences are a class of preference relations. A group of agents is said to have single-peaked preferences over a set of possible outcomes if the outcomes can be ordered along a line such that:

  1. Each agent has a "best outcome" in the set, and -
  2. For each agent, outcomes that are further from his or her best outcome are preferred less.

Majority judgment (MJ) is a single-winner voting system proposed in 2010 by Michel Balinski and Rida Laraki. It is a kind of highest median rule, a cardinal voting system that elects the candidate with the highest median rating.

Maximal lotteries refers to a probabilistic voting system first considered by the French mathematician and social scientist Germain Kreweras in 1965. The method uses preferential ballots and returns so-called maximal lotteries, i.e., probability distributions over the alternatives that are weakly preferred to any other probability distribution. Maximal lotteries satisfy the Condorcet criterion, the Smith criterion, polynomial runtime, and probabilistic versions of reinforcement, participation, and independence of clones.

In economics, dichotomous preferences (DP) are preference relations that divide the set of alternatives to two subsets: "Good" versus "Bad".

Random priority (RP), also called Random serial dictatorship (RSD), is a procedure for fair random assignment - dividing indivisible items fairly among people.

A simultaneous eating algorithm(SE) is an algorithm for allocating divisible objects among agents with ordinal preferences. "Ordinal preferences" means that each agent can rank the items from best to worst, but cannot (or does not want to) specify a numeric value for each item. The SE allocation satisfies SD-efficiency - a weak ordinal variant of Pareto-efficiency (it means that the allocation is Pareto-efficient for at least one vector of additive utility functions consistent with the agents' item rankings).

Multiwinner approval voting, also called approval-based committee (ABC) voting, is a multi-winner electoral system that uses approval ballots. Each voter may select ("approve") any number of candidates, and multiple candidates are elected. The number of elected candidates is usually fixed in advance. For example, it can be the number of seats in a country's parliament, or the required number of members in a committee.

Fractional social choice is a branch of social choice theory in which the collective decision is not a single alternative, but rather a weighted sum of two or more alternatives. For example, if society has to choose between three candidates: A B or C, then in standard social choice, exactly one of these candidates is chosen, while in fractional social choice, it is possible to choose "2/3 of A and 1/3 of B". A common interpretation of the weighted sum is as a lottery, in which candidate A is chosen with probability 2/3 and candidate B is chosen with probability 1/3. Due to this interpretation, fractional social choice is also called random social choice, probabilistic social choice, or stochastic social choice. But it can also be interpreted as a recipe for sharing, for example:

Fractional approval voting is an electoral system using approval ballots, in which the outcome is fractional: for each alternative j there is a fraction pj between 0 and 1, such that the sum of pj is 1. It can be seen as a generalization of approval voting: in the latter, one candidate wins and the other candidates lose. The fractions pj can be interpreted in various ways, depending on the setting. Examples are:

A median mechanism is a voting rule that allows people to decide on a value in a one-dimensional domain. Each person votes by writing down his/her ideal value, and the rule selects a single value which is the median of all votes.

Lexicographic dominance is a total order between random variables. It is a form of stochastic ordering. It is defined as follows. Random variable A has lexicographic dominance over random variable B if one of the following holds:

Budget-proposal aggregation (BPA) is a problem in social choice theory. A group has to decide on how to distribute its budget among several issues. Each group-member has a different idea about what the ideal budget-distribution should be. The problem is how to aggregate the different opinions into a single budget-distribution program.

Participatory budgeting experiments are experiments done in the laboratory and in computerized simulations, in order to check various ethical and practical aspects of participatory budgeting. These experiments aim to decide on two main issues:

  1. Front-end: which ballot type to use as an input? See Participatory budgeting ballot types for common types of ballots.
  2. Back-end: Which rule to use for aggregating the voters' preferences? See combinatorial participatory budgeting for detailed descriptions of various aggregation rules.

Belief merging, also called belief fusion or propositional belief merging, is a process in which an individual agent aggregates possibly conflicting pieces of information, expressed in logical formulae, into a consistent knowledge-base. Applications include combining conflicting sensor information received by the same agent and combining multiple databases to build an expert system. It also has applications in multi-agent systems.

The average voting rule is a rule for group decision-making when the decision is a distribution, and each of the voters reports his ideal distribution. This is a special case of budget-proposal aggregation. It is a simple aggregation rule, that returns the arithmetic mean of all individual ideal distributions. The average rule was first studied formally by Michael Intrilligator. This rule and its variants are commonly used in economics and sports.

References

  1. 1 2 3 Varloot, Estelle Marine; Laraki, Rida (2022-07-13). "Level-strategyproof Belief Aggregation Mechanisms". Proceedings of the 23rd ACM Conference on Economics and Computation. EC '22. New York, NY, USA: Association for Computing Machinery. pp. 335–369. doi:10.1145/3490486.3538309. ISBN   978-1-4503-9150-4.
  2. ↑ Boyer-Kassem, Thomas (January 2019). "Scientific Expertise and Risk Aggregation". Philosophy of Science. 86 (1): 124–144. doi:10.1086/701071. ISSN   0031-8248.
  3. 1 2 3 Martini, Carlo; Sprenger, Jan (2017-01-01), Boyer-Kassem, Thomas; Mayo-Wilson, Conor; Weisberg, Michael (eds.), "Opinion Aggregation and Individual Expertise", Scientific Collaboration and Collective Knowledge, Oxford University Press USA, pp. 180–201, ISBN   978-0-19-068053-4 , retrieved 2023-11-09
  4. 1 2 Dietrich, Franz; List, Christian (March 2014). Probabilistic Opinion Pooling (Report). University Library of Munich, Germany.
  5. ↑ Aspinall, William P. (2006), "Structured elicitation of expert judgement for probabilistic hazard and risk assessment in volcanic eruptions", Statistics in Volcanology, The Geological Society of London on behalf of The International Association of Volcanology and Chemistry of the Earth's Interior, pp. 15–30, doi:10.1144/iavcei001.2, ISBN   978-1-86239-623-4 , retrieved 2023-11-09
  6. ↑ Christophersen, Annemarie; Deligne, Natalia I.; Hanea, Anca M.; Chardot, Lauriane; Fournier, Nicolas; Aspinall, Willy P. (2018). "Bayesian Network Modeling and Expert Elicitation for Probabilistic Eruption Forecasting: Pilot Study for Whakaari/White Island, New Zealand". Frontiers in Earth Science. 6. doi: 10.3389/feart.2018.00211 . hdl: 10356/85752 . ISSN   2296-6463.
  7. ↑ Arnell, Nigel W.; Tompkins, Emma L.; Adger, W. Neil (December 2005). "Eliciting Information from Experts on the Likelihood of Rapid Climate Change". Risk Analysis. 25 (6): 1419–1431. doi:10.1111/j.1539-6924.2005.00689.x. ISSN   0272-4332.
  8. ↑ O’Neill, Saffron J.; Osborn, Tim J.; Hulme, Mike; Lorenzoni, Irene; Watkinson, Andrew R. (2008-10-21). "Using expert knowledge to assess uncertainties in future polar bear populations under climate change". Journal of Applied Ecology. 45 (6): 1649–1659. doi:10.1111/j.1365-2664.2008.01552.x. ISSN   0021-8901.
  9. ↑ von Oertzen, T. J.; Macerollo, A.; Leone, M. A.; Beghi, E.; Crean, M.; Oztuk, S.; Bassetti, C.; Twardzik, A.; Bereczki, D.; Di Liberto, G.; Helbok, R.; Oreja‐ Guevara, C.; Pisani, A.; Sauerbier, A.; Sellner, J. (January 2021). "EAN consensus statement for management of patients with neurological diseases during the COVID‐19 pandemic". European Journal of Neurology. 28 (1): 7–14. doi:10.1111/ene.14521. ISSN   1351-5101. PMC   7675361 . PMID   33058321.
  10. ↑ Dietrich, Franz; List, Christian (2017-04-01). "Probabilistic opinion pooling generalized. Part one: general agendas". Social Choice and Welfare. 48 (4): 747–786. doi: 10.1007/s00355-017-1034-z . ISSN   1432-217X.
  11. ↑ Elster, Jon (2015-06-26). Secrecy and Publicity in Votes and Debates. Cambridge University Press. ISBN   978-1-107-08336-3.
  12. ↑ Good, I. J. (January 1952). "Rational Decisions". Journal of the Royal Statistical Society, Series B (Methodological). 14 (1): 107–114. doi:10.1111/j.2517-6161.1952.tb00104.x. ISSN   0035-9246.
  13. ↑ McCarthy, John (September 1956). "Measures of the Value of Information". Proceedings of the National Academy of Sciences. 42 (9): 654–655. doi: 10.1073/pnas.42.9.654 . ISSN   0027-8424. PMC   534271 . PMID   16589926.
  14. ↑ Winkler, Robert L. (September 1969). "Scoring Rules and the Evaluation of Probability Assessors". Journal of the American Statistical Association. 64 (327): 1073–1078. doi:10.1080/01621459.1969.10501037. ISSN   0162-1459.
  15. ↑ Friedman, Daniel (April 1983). "Effective Scoring Rules for Probabilistic Forecasts". Management Science. 29 (4): 447–454. doi:10.1287/mnsc.29.4.447. ISSN   0025-1909.
  16. ↑ Border, Kim C.; Jordan, J. S. (1983). "Straightforward Elections, Unanimity and Phantom Voters". The Review of Economic Studies. 50 (1): 153–170. doi:10.2307/2296962. ISSN   0034-6527. JSTOR   2296962.
  17. ↑ Moulin, H. (1980-01-01). "On strategy-proofness and single peakedness". Public Choice. 35 (4): 437–455. doi:10.1007/BF00128122. ISSN   1573-7101. S2CID   154508892.
  18. ↑ Jennings, Andrew B.; Laraki, Rida; Puppe, Clemens; Varloot, Estelle M. (2023-08-28). "New characterizations of strategy-proofness under single-peakedness". Mathematical Programming. arXiv: 2102.11686 . doi: 10.1007/s10107-023-02010-x . ISSN   1436-4646.
  19. ↑ Goel, Ashish; Krishnaswamy, Anilesh K.; Sakshuwong, Sukolsak; Aitamurto, Tanja (2019-07-29). "Knapsack Voting for Participatory Budgeting". ACM Transactions on Economics and Computation. 7 (2): 8:1–8:27. arXiv: 2009.06856 . doi: 10.1145/3340230 . ISSN   2167-8375.
  20. ↑ Freeman, Rupert; Pennock, David M.; Peters, Dominik; Wortman Vaughan, Jennifer (2019-06-17). "Truthful Aggregation of Budget Proposals". Proceedings of the 2019 ACM Conference on Economics and Computation. EC '19. New York, NY, USA: Association for Computing Machinery. pp. 751–752. arXiv: 1905.00457 . doi:10.1145/3328526.3329557. ISBN   978-1-4503-6792-9.
  21. ↑ Leontaris, Georgios; Morales-Nápoles, Oswaldo (2018-01-01). "ANDURIL — A MATLAB toolbox for ANalysis and Decisions with UnceRtaInty: Learning from expert judgments". SoftwareX. 7: 313–317. doi: 10.1016/j.softx.2018.07.001 . ISSN   2352-7110.
  22. ↑ Cooke, Roger M. (1991-10-24). Experts in Uncertainty: Opinion and Subjective Probability in Science. Oxford University Press. ISBN   978-0-19-536237-4.
  23. ↑ Armstrong, J. Scott, ed. (2001). "Principles of Forecasting". International Series in Operations Research & Management Science. 30. doi:10.1007/978-0-306-47630-3. ISBN   978-0-7923-7401-5. ISSN   0884-8289.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.