The Good Judgment Project

Last updated

The Good Judgment Project (GJP) is an organization dedicated to "harnessing the wisdom of the crowd to forecast world events". It was co-created by Philip E. Tetlock (author of Superforecasting and Expert Political Judgment ), decision scientist Barbara Mellers, and Don Moore, all professors at the University of Pennsylvania. [1] [2] [3]

Contents

The project began as a participant in the Aggregative Contingent Estimation (ACE) program of the Intelligence Advanced Research Projects Activity (IARPA). [4] [5] It then extended its crowd wisdom to commercial activities, recruiting forecasters and aggregating the predictions of the most historically accurate among them to forecast future events. [6] [7] Predictions are scored using Brier scores. [8] The top forecasters in GJP are "reportedly 30% better than intelligence officers with access to actual classified information." [9]

History

The Good Judgment Project began in July 2011 in collaboration with the Aggregative Contingent Estimation (ACE) Program at IARPA (IARPA-ACE). [10] The first contest began in September 2011. [11] GJP was one of many entrants in the IARPA-ACE tournament, which posed around 100 to 150 questions each year on geopolitical events. The GJP research team gathered a large number of talented amateurs (rather than geopolitical subject matter experts), gave them basic tutorials on forecasting best practice and overcoming cognitive biases, and created an aggregation algorithm to combine the individual predictions of the forecasters. [5] [12] GJP won both seasons of the contest, and were 35% to 72% more accurate than any other research team. [13] Starting with the summer of 2013, GJP were the only research team IARPA-ACE was still funding, and GJP participants had access to the Integrated Conflict Early Warning System. [8]

People

The co-leaders of the GJP include Philip Tetlock, Barbara Mellers and Don Moore. [1] The website lists a total of about 30 team members, including the co-leaders as well as David Budescu, Lyle Ungar, Jonathan Baron, and prediction-markets entrepreneur Emile Servan-Schreiber. [14] The advisory board included Daniel Kahneman, Robert Jervis, J. Scott Armstrong, Michael Mauboussin, Carl Spetzler and Justin Wolfers. [15] The study employed several thousand people as volunteer forecasters. [12] Using personality-trait tests, training methods and strategies the researchers at GJP were able to select forecasting participants with less cognitive bias than the average person; as the forecasting contest continued the researchers were able to further down select these individuals in groups of so-called superforecasters. The last season of the GJP enlisted a total of 260 superforecasters.[ citation needed ]

Research

A significant amount of research has been conducted based on the Good Judgment Project by the people involved with it. [16] The results show that harnessing a blend of statistics, psychology, training and various levels of interaction between individual forecasters, consistently produced the best forecast for several years in a row. [12]

Good Judgment Inc.

A commercial spin-off of the Good Judgment Project started to operate on the web in July 2015 under the name Good Judgment Inc. Their services include forecasts on questions of general interest, custom forecasts, and training in Good Judgment's forecasting techniques. [17] Starting in September 2015, Good Judgment Inc has been running a public forecasting tournament at the Good Judgment Open site. Like the Good Judgment Project, Good Judgment Open has questions about geopolitical and financial events, although it also has questions about US politics, entertainment, and sports. [18] [19]

Media coverage

GJP has repeatedly been discussed in The Economist . [11] [20] [21] [22] GJP has also been covered in The New York Times , [3] The Washington Post , [5] [23] [24] and Co.Exist. [25] NPR aired a segment on The Good Judgment Project by the title "So You Think You're Smarter Than a CIA Agent", on April 2, 2014. [9] The Financial Times published an article on the GJP on September 5, 2014. [26] Washingtonian published an article that mentioned the GJP on January 8, 2015. [27] The BBC and The Washington Post published articles on the GJP respectively on January 20, 21, and 29, 2015. [28] [29] [30]

The Almanac of Menlo Park published a story on the GJP on January 29, 2015. [31] An article on the GJP appeared on the portal of the Philadelphia Inquirer, Philly.com, on February 4, 2015. [32] The book Wiser: Getting Beyond Groupthink to Make Groups Smarter has a section detailing the involvement of the GJP in the tournament run by IARPA. [33] Psychology Today published online a short article summarizing the paper by Mellers, et al., that wraps up the main findings of the GJP. [34] [35]

The project spawned a 2015 book by Tetlock and coauthored by Dan Gardner, Superforecasting - The Art and Science of Prediction , that divulges the main findings of the research conducted with the data from the GJP. [36] Co-author Gardner had already published a book in 2010, that quoted previous research by Tetlock that seeded the GJP effort. [37] A book review in the September 26, 2015, print edition of the Economist discusses the main concepts. [38] A Wall Street Journal article depicts it as: "The most important book on decision making since Daniel Kahneman’s Thinking, Fast and Slow ." [39] The Harvard Business Review paired it to the book How Not to Be Wrong: The Power of Mathematical Thinking by Jordan Ellenberg. [40] On September 30, 2015, NPR aired an episode of the Colin McEnroe Show centering on the GJP and the book Superforecasting; guests on the show were Tetlock, IARPA Director Jason Matheny, and superforecaster Elaine Rich. [41]

See also

Related Research Articles

Forecasting is the process of making predictions based on past and present data. Later these can be compared (resolved) against what happens. For example, a company might estimate their revenue in the next year, then compare it against the actual results. Prediction is a similar but more general term. Forecasting might refer to specific formal statistical methods employing time series, cross-sectional or longitudinal data, or alternatively to less formal judgmental methods or the process of prediction and resolution itself. Usage can vary between areas of application: for example, in hydrology the terms "forecast" and "forecasting" are sometimes reserved for estimates of values at certain specific future times, while the term "prediction" is used for more general estimates, such as the number of times floods will occur over a long period.

Prediction markets are open markets where specific outcomes can be predicted using financial incentives. Essentially, they are exchange-traded markets created for the purpose of trading the outcome of events. The market prices can indicate what the crowd thinks the probability of the event is. A prediction market contract trades between 0 and 100%. The most common form of a prediction market is a binary option market, which will expire at the price of 0 or 100%. Prediction markets can be thought of as belonging to the more general concept of crowdsourcing which is specially designed to aggregate information on particular topics of interest. The main purposes of prediction markets are eliciting aggregating beliefs over an unknown future outcome. Traders with different beliefs trade on contracts whose payoffs are related to the unknown future outcome and the market prices of the contracts are considered as the aggregated belief.

There are two main uses of the term calibration in statistics that denote special types of statistical inference problems. "Calibration" can mean

<i>The Wisdom of Crowds</i> 2004 book by James Surowiecki

The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations, published in 2004, is a book written by James Surowiecki about the aggregation of information in groups, resulting in decisions that, he argues, are often better than could have been made by any single member of the group. The book presents numerous case studies and anecdotes to illustrate its argument, and touches on several fields, primarily economics and psychology.

<i>The Hedgehog and the Fox</i> 1953 book by Isaiah Berlin

The Hedgehog and the Fox is an essay by philosopher Isaiah Berlin that was published as a book in 1953. It was one of his most popular essays with the general public. However, Berlin said, "I meant it as a kind of enjoyable intellectual game, but it was taken seriously. Every classification throws light on something". It has been compared to "an intellectual's cocktail-party game".

<span class="mw-page-title-main">Intelligence Advanced Research Projects Activity</span> American government agency

The Intelligence Advanced Research Projects Activity (IARPA) is an organization within the Office of the Director of National Intelligence responsible for leading research to overcome difficult challenges relevant to the United States Intelligence Community. IARPA characterizes its mission as follows: "To envision and lead high-risk, high-payoff research that delivers innovative technology for future overwhelming intelligence advantage."

The wisdom of the crowd is the collective opinion of a diverse independent group of individuals rather than that of a single expert. This process, while not new to the Information Age, has been pushed into the mainstream spotlight by social information sites such as Quora, Reddit, Stack Exchange, Wikipedia, Yahoo! Answers, and other web resources which rely on collective human knowledge. An explanation for this phenomenon is that there is idiosyncratic noise associated with each individual judgment, and taking the average over a large number of responses will go some way toward canceling the effect of this noise.

<span class="mw-page-title-main">Philip E. Tetlock</span>

Philip E. Tetlock is a Canadian-American political science writer, and is currently the Annenberg University Professor at the University of Pennsylvania, where he is cross-appointed at the Wharton School and the School of Arts and Sciences. He was elected a Member of the American Philosophical Society in 2019.

The Program on Vehicle and Mobility Innovation (PVMI) is the oldest and largest international research consortium aimed at analyzing the global automotive industry.

Illusion of validity is a cognitive bias in which a person overestimates their ability to interpret and predict accurately the outcome when analyzing a set of data, in particular when the data analyzed show a very consistent pattern—that is, when the data "tell" a coherent story.

The Integrated Crisis Early Warning System (ICEWS) combines a database of political events and a system using these to provide conflict early warnings. It is supported by the Defense Advanced Research Projects Agency in the United States. The database as well as the model used by Lockheed Martin Advanced Technology Laboratories are currently undergoing operational test and evaluation by the United States Southern Command and United States Pacific Command.

Lyle H. Ungar is a machine learning researcher and professor of Computer and Information Science at the University of Pennsylvania. He is also affiliated with the psychology department at the university.

Aggregative Contingent Estimation (ACE) was a program of the Office of Incisive Analysis (OIA) at the Intelligence Advanced Research Projects Activity (IARPA). The program ran from June 2010 until June 2015.

An information technology generalist is a technology professional proficient in many facets of information technology without any specific specialty. Furthermore, an IT generalist is generally considered to possess general business knowledge and soft skills allowing them to be adaptable in a wide array of work environments. The IT Generalist is often able to fulfill many different roles within a company depending on specific technology needs. In a small business environment, budgets often delegate many different facets of technology to a single individual, especially considering a small business will often require an individual proficient in desktop support, web page design, databases, phone systems, and even server administration. The role of the IT Generalist within a larger company, however, often becomes more of a project leader or integrations specialist due to a project team consisting of a varying degree of IT specialists and interfacing with end-users requiring soft-skills.

SciCast is a collaborative platform for science and technology forecasting created by George Mason University with the help of a grant from the Intelligence Advanced Research Projects Activity (IARPA) as part of its ForeST program. SciCast is currently on hiatus, after losing its main IARPA funding. It was expected to re-open in the fall of 2015 with the support of a major Science & Technology sponsor, but this had not occurred by January 2016.

<i>Superforecasting: The Art and Science of Prediction</i> Book by Philip E. Tetlock and Dan Gardner released in 2015

Superforecasting: The Art and Science of Prediction is a book by Philip E. Tetlock and Dan Gardner released in 2015. It details findings from The Good Judgment Project.

Expert Judgment (EJ) denotes a wide variety of techniques ranging from a single undocumented opinion, through preference surveys, to formal elicitation with external validation of expert probability assessments. Recent books are . In the nuclear safety area, Rasmussen formalized EJ by documenting all steps in the expert elicitation process for scientific review. This made visible wide spreads in expert assessments and teed up questions regarding the validation and synthesis of expert judgments. The nuclear safety community later took onboard expert judgment techniques underpinned by external validation . Empirical validation is the hallmark of science, and forms the centerpiece of the classical model of probabilistic forecasting . A European Network coordinates workshops. Application areas include nuclear safety, investment banking, volcanology, public health, ecology, engineering, climate change and aeronautics/aerospace. For a survey of applications through 2006 see and give exhortatory overviews. A recent large scale implementation by the World Health Organization is described in . A long running application at the Montserrat Volcano Observatory is described in . The classical model scores expert performance in terms of statistical accuracy and informativeness . These terms should not be confused with “accuracy and precision”. Accuracy “is a description of systematic errors” while precision “is a description of random errors”. In the classical model statistical accuracy is measured as the p-value or probability with which one would falsely reject the hypotheses that an expert's probability assessments were statistically accurate. A low value means it is very unlikely that the discrepancy between an expert's probability statements and observed outcomes should arise by chance. Informativeness is measured as Shannon relative information with respect to an analyst-supplied background measure. Shannon relative information is used because it is scale invariant, tail insensitive, slow, and familiar. Parenthetically, measures with physical dimensions, such as the standard deviation, or the width of prediction intervals, raise serious problems, as a change of units would affect some variables but not others. The product of statistical accuracy and informativeness for each expert is their combined score. With an optimal choice of a statistical accuracy threshold beneath which experts are unweighted, the combined score is a long run “strictly proper scoring rule”: an expert achieves his long run maximal expected score by and only by stating his true beliefs. The classical model derives Performance Weighted (PW) combinations. These are compared with Equally Weighted (EW) combinations, and recently with Harmonically Weighted (HW) combinations, as well as with individual expert assessments.

Barbara Ann Mellers is I. George Heyman University Professor in the psychology department at the University of Pennsylvania. Her research focuses on decision processes.

Don Andrew Moore is an author, academic, and professor. He is the Lorraine Tyson Mitchell Chair I of Leadership and Communication at UC Berkeley's Haas School of Business where he teaches classes on leadership, negotiation, and decision making.

A superforecaster is a person who makes forecasts that can be shown by statistical means to have been consistently more accurate than the general public or experts. Superforecasters sometimes use modern analytical and statistical methodologies to augment estimates of base rates of events; research finds that such forecasters are typically more accurate than experts in the field who do not use analytical and statistical techniques. The term "superforecaster" is a trademark of Good Judgment Inc.

References

  1. 1 2 "Welcome to the Good Judgment Project". The Good Judgment Project. Retrieved May 5, 2014.
  2. ↑ "Who's who in the Good Judgment Project". The Good Judgment Project. July 27, 2011. Retrieved May 5, 2014.
  3. 1 2 Brooks, David (March 21, 2013). "Forecasting Fox". New York Times . Retrieved May 5, 2014.
  4. ↑ "The Project". The Good Judgment Project. Archived from the original on May 6, 2014. Retrieved May 5, 2014.
  5. 1 2 3 Horowitz, Michael (November 26, 2013). "Good judgment in forecasting international affairs (and an invitation for season 3)". Washington Post . Retrieved May 5, 2014.
  6. ↑ "About Superforecasting | Unprecedented Accurate & Precise Forecasting". Good Judgment. Retrieved 2022-02-17.
  7. ↑ Matthews, Dylan (2022-02-16). "How can we prevent major conflicts like a Russia-Ukraine war?". Vox. Retrieved 2022-02-17.
  8. 1 2 Dickenson, Matt (November 12, 2013). "Prediction and Good Judgment: Can ICEWS Inform Forecasts?". Predictive Heuristics. Retrieved May 24, 2014.
  9. 1 2 Spiegel, Alix. "So You Think You're Smarter Than A CIA Agent". NPR.org. Retrieved 2014-08-18.
  10. ↑ "The idea behind the Good Judgment Project". The Good Judgment Project. July 27, 2011. Archived from the original on May 6, 2014. Retrieved May 5, 2014.
  11. 1 2 "The perils of prediction: Adventures in punditry". The Economist . September 2, 2011. Retrieved May 6, 2014.
  12. 1 2 3 Mellers, Barbara; Ungar, Lyle; Baron, Jonathan; Ramos, Jaime; Gurcay, Burcu; Fincher, Katrina; Scott, Sydney E.; Moore, Don; Atanasov, Pavel; Swift, Samuel A.; Murray, Terry; Stone, Eric; Tetlock, Philip E. (2014-05-01). "Psychological strategies for winning a geopolitical forecasting tournament". Psychological Science. 25 (5): 1106–1115. doi:10.1177/0956797614524255. ISSN   1467-9280. PMID   24659192. S2CID   42143367.
  13. ↑ "The first championship season". Good Judgment. Retrieved 2022-02-17.
  14. ↑ "Team". The Good Judgment Project. Retrieved May 5, 2014.
  15. ↑ "Freakonomics". Sign Up for a Prediction Tournament. 2011-08-04.
  16. ↑ Multiple Sources:
  17. ↑ Brody, Liz (January 1, 2022). "Meet the Elite Team of Superforecasters Who Have Turned Future-Gazing Into a Science". Entrepreneur.
  18. ↑ Gossett, Stephen (August 6, 2020). "How the Good Judgment Project's Superforecasters Use Data to Make Predictions". builtin.com. Retrieved 2022-06-07.
  19. ↑ "Good Judgment® Open". www.gjopen.com. Retrieved 2022-02-17.
  20. ↑ "Monetary policy: How likely is deflation?". The Economist . September 13, 2011. Retrieved May 6, 2014.
  21. ↑ "International: Who's good at forecasts? How to sort the best from the rest". The Economist . November 18, 2013. Retrieved May 6, 2014.
  22. ↑ "The experts' best bets". The Economist . November 10, 2021. Retrieved June 7, 2022.
  23. ↑ Ignatius, David (November 1, 2013). "More chatter than needed". Washington Post . Retrieved May 6, 2014.
  24. ↑ Bender, Jeremy (April 3, 2014). "Huge Experiment Finds Regular Folks Predict World Events Better Than CIA Agents". Business Insider . Retrieved May 6, 2014.
  25. ↑ "The Surprising Accuracy Of Crowdsourced Predictions About The Future. Do you know whether Turkey will get a new constitution? It turns out you do: A group of well-informed citizens can predict future events more often than any foreign policy expert or CIA analyst". Co.exist. April 21, 2014. Retrieved May 6, 2014.
  26. ↑ Harford, Tim (2014-09-05). "How to see into the future". Financial Times. ISSN   0307-1766 . Retrieved 2014-09-05.
  27. ↑ Hamilton, Keegan (8 January 2015). "How US Agencies Are Using the Web to Pick Our Brains". Washingtonian. Retrieved 2015-01-24.
  28. ↑ Burton, Tara (2015-01-20). "Could you be a 'super-forecaster'?". BBC Future. Retrieved 2015-01-21.
  29. ↑ Jensen, Nathan (2015-01-21). "Experts see a Republican Senate and fast-track authority for Obama as keys to new trade agreements". The Washington Post. ISSN   0190-8286 . Retrieved 2015-01-21.
  30. ↑ Mellers, Barbara; Michael C. Horowitz (2015-01-29). "Does anyone make accurate geopolitical predictions?". The Washington Post. ISSN   0190-8286 . Retrieved 2015-01-30.
  31. ↑ "Feature story: Bob Sawyer of Woodside discovers his latent talent in forecasting" . Retrieved 2015-03-17.
  32. ↑ Dribben, Melissa; Inquirer Staff Writer (2015-02-04). "Fortune telling: Crowds surpass pundits". Philly.com. Retrieved 2015-02-06.
  33. ↑ Sunstein, Cass R.; Hastie, Reid (2014-12-23). Wiser: Getting Beyond Groupthink to Make Groups Smarter. Harvard Business Review Press. ISBN   978-1-4221-2299-0.
  34. ↑ "Who's Best at Predicting the Future? (and How to Get Better)". Psychology Today. Retrieved 2015-07-11.
  35. ↑ Mellers, Barbara; Stone, Eric; Murray, Terry; Minster, Angela; Rohrbaugh, Nick; Bishop, Michael; Chen, Eva; Baker, Joshua; Hou, Yuan; Horowitz, Michael; Ungar, Lyle; Tetlock, Philip (2015-05-01). "Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions". Perspectives on Psychological Science. 10 (3): 267–281. doi:10.1177/1745691615577794. ISSN   1745-6916. PMID   25987508. S2CID   3118872.
  36. ↑ Tetlock, Philip E.; Gardner, Dan (2015-09-29). Superforecasting - The Art and Science of Prediction. New York: Crown. ISBN   978-0-8041-3669-3.
  37. ↑ Gardner, Dan (2010-10-12). Future Babble: Why Expert Predictions Fail - and Why We Believe Them Anyway. McClelland & Stewart.
  38. ↑ "Unclouded vision". The Economist. 2015-09-26. ISSN   0013-0613 . Retrieved 2015-09-24.
  39. ↑ Zweig, Jason. "Can You See the Future? Probably Better Than Professional Forecasters". The Wall Street Journal. Retrieved September 25, 2015. I think Philip Tetlock's "Superforecasting: The Art and Science of Prediction," ..., is the most important book on decision making since Daniel Kahneman's "Thinking, Fast and Slow."
  40. ↑ Frick, Walter. "Question Certainty". Harvard Business Review. Retrieved 2015-09-26.
  41. ↑ McEnroe, Colin; Wolf, Chion. "The Colin McEnroe Show". WNPR. National Public Radio . Retrieved October 1, 2015.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.