Words of estimative probability

Last updated June 03, 2024

Words of estimative probability (WEP or WEPs) are terms used by intelligence analysts in the production of analytic reports to convey the likelihood of a future event occurring. A well-chosen WEP gives a decision maker a clear and unambiguous estimate upon which to base a decision. Ineffective WEPs are vague or misleading about the likelihood of an event. An ineffective WEP places the decision maker in the role of the analyst, increasing the likelihood of poor or snap decision making. Some intelligence and policy failures appear to be related to the imprecise use of estimative words.^{[ citation needed ]}

History

Intelligence

In 1964 Sherman Kent, one of the first contributors to a formal discipline of intelligence analysis addressed the problem of misleading expressions of odds in National Intelligence Estimates (NIE). In Words of Estimative Probability, Kent distinguished between "poets" (those preferring wordy probabilistic statements) from "mathematicians" (those preferring quantitative odds). To bridge the gap between them and decision makers, Kent developed a paradigm relating estimative terms to odds. His goal was to "... set forth the community's findings in such a way as to make clear to the reader what is certain knowledge and what is reasoned judgment, and within this large realm of judgment what varying degrees of certitude lie behind each key judgment."^[1] Kent's initiative was not adopted although the idea was well received and remains compelling today.^{[ citation needed ]}

Table 1: Kent's "words of estimative probability"^[2]
Certain	100%	Give or take 0%
The general area of possibility
Almost certain	93%	Give or take about 6%
Probable	75%	Give or take about 12%
Chances about even	50%	Give or take about 10%
Probably not	30%	Give or take about 10%
Almost certainly not	7%	Give or take about 5%
Impossible	0	Give or take 0%

Policy and intelligence failures related to WEPs

An example of the damage that missing or vague WEPs can do is to be found in the President's Daily Brief (PDB), entitled Bin Laden Determined to Strike in US. The President's Daily Brief is arguably the pinnacle of concise, relevant, actionable analytic writing in the U.S. Intelligence Community (IC). The PDB is intended to keep the president informed on a wide range of issues, the best analysts write it and senior leaders review it. This, the "August 6 PDB", is at the center of much controversy for the USIC. The August 6 PDB began with not only a vague warning in the title, but also continued with vague warnings:

"Bin Ladin since 1997 has wanted to conduct terrorist attacks in the US" (CIA, 2001, para. 1)
"Bin Ladin implied...that his followers would 'bring the fighting to America'" (CIA, 2001, para. 1)
Bin Ladin’s “attacks against...US embassies...in 1998 demonstrate that he prepares operations years in advance and is not deterred by setbacks" (CIA, 2001, para. 6)
"FBI information...indicates patterns of suspicious activity in this country consistent with preparations for hijackings or other types of attacks" (CIA, 2001, para. 10)
"a call to [the US] Embassy in the UAE in May [said] that a group of Bin Ladin supporters was in the US planning attacks with explosives" (CIA, 2001, para. 11)^[3]

The PDB described Bin Laden's previous activities. It did not present the president with a critically important clear estimate of Bin Laden's likely activities in the coming months:

Bush had specifically asked for an intelligence analysis of possible al Qaeda attacks within the United States, because most of the information presented to him over the summer about al Qaeda focused on threats against U.S. targets overseas, sources said. But one source said the White House was disappointed because the analysis lacked focus and did not present fresh intelligence.^[4]

The lack of appropriate WEPs would lead to confusion about the likelihood of an attack and to guessing about the period in which it was likely to occur.

The language used in the memo lacks words of estimative probability that reduce uncertainty, thus preventing the President and his decisionmakers from implementing measures directed at stopping al Qaeda’s actions^[5]

The consequences of the 9/11 and the Iraq/WMD intelligence failures, the 9/11 Commission and the Iraq Intelligence Commission, were the movers of structural reform of the intelligence community. Although these reforms intended to improve the functioning of the IC, particularly concerning inter-agency cooperation and information sharing, "they paid scant attention to improving the quality of analyses and intelligence writing". There is a pervasive feeling that this improvement is needed.^{[ opinion ]} However, there seems to be resistance in the IC, due in part to habit and in part to the reality of politics and the understandable preference for the "plausible deniability" that less precise jargon offers.^[5]

Medicine

Physicians and clinical scientists face a very similar problem in obtaining informed consent for patients, where words such as "rare" or "infrequent" do have actual probabilities defined. Numerical information of this type, however, is rarely presented to patients.

A representative guide for obtaining informed consent from people participating in social science or behavioural research, or of the potential risks of a medical procedure, suggests giving typical numerical chances of an adverse event when words of estimative probability first are used.

Word	Probability
Likely	Expected to happen to more than 50% of subjects
Frequent	Will probably happen to 10-50% of subjects
Occasional	Will happen to 1-10% of subjects
Rare	Will happen to less than 1% of subjects

The guideline continues,

For studies involving investigational agents, or experimental doses or combinations of drugs and/or treatments, subjects should be warned that there may be as yet unknown risks associated with the drug/treatment but that they will be advised if any new information becomes available that may affect their desire to participate in the study.^[6]

Reforms to methodology

Estimative statements can be improved in four ways;^{[ opinion ]} either by:

Adding quantitative source reliability and confidence measures to estimative statements^{[ citation needed ]}
Complementing estimative statements with stochastic analyses^{[ citation needed ]}
Standardizing WEPs^{[ citation needed ]}
Standardizing WEPs and complementing estimative statements with ratings of source reliability and analytic confidence ^{[ citation needed ]}

Quantification of source reliability and analytic confidence

Michael Schrage,^[7] an advisor to the Massachusetts Institute of Technology's (MIT) Security Studies Program, wrote in a Washington Post editorial that requiring analysts to produce and include quantitative measures of source reliability and confidence along with their findings would reduce ambiguity.^[8] Yet Schrage also reported that former interim Director of Central Intelligence John E. McLaughlin's attempted to enact this at the CIA, but, like Kent's initiative, it was not adopted.

Former acting CIA director and longtime analyst John McLaughlin tried to promote greater internal efforts at assigning probabilities to intelligence assessments during the 1990s, but they never took. Intelligence analysts "would rather use words than numbers to describe how confident we are in our analysis," a senior CIA officer who's served for more than 20 years told me. Moreover, "most consumers of intelligence aren't particularly sophisticated when it comes to probabilistic analysis. They like words and pictures, too. My experience is that [they] prefer briefings that don't center on numerical calculation. That's not to say we can't do it, but there's not that much demand for it."^[9]

Stochastic analysis

Since combining quantitative, probabilistic information with estimates is successful in business forecasting, marketing, medicine, and epidemiology it should be implemented by the intelligence community as well. These fields have used probability theory and Bayesian analysis as forecasting tools. Using probability theory and other stochastic methods are appealing because they rely on rationality and mathematical rigor, are less subject to analytical bias, and such findings appear to be unambiguous. As an opposing argument, it is indisputable that few analysts or intelligence consumers have the stomach for numerical calculation. Additionally, Bruce Blair, Director of the Center for Defense Information, a proponent of quantitative methods for the IC, points out; intelligence information from secret sources is often murky, and the application of advanced math is not sufficient to make it more reliable. However, he sees a place for stochastic analyses over a very long period, it "points to a fairly slow learning curve that also challenges the wisdom of making preemption a cornerstone of U.S. security strategy."^[10] The reservations stated are significant: Mathematical and statistical analyses require a lot of work without rapid and necessarily commensurate gains in accuracy, speed or comprehension.

Standardization

UK intelligence assessments use the PHIA "probability yardstick" for communicating probability:

The National Intelligence Council's recommendations described the use of a WEP paradigm (table 2) in combination with an assessment of confidence levels ("high, moderate, low") based on the scope and quality supporting information:

Table 2: National Intelligence Council WEPs^[11]
Almost certainly
Probably/likely
Even chance
Unlikely
Remote

However, the NIC's discussion of this paradigm seems to undercut its chances of being effective:

Intelligence judgments about likelihood are intended to reflect the Community's sense of the probability of a development or event. [...] We do not intend the term "unlikely" to imply an event will not happen. We use "probably" and "likely" to indicate there is a greater than even chance. We use words such as "we cannot dismiss", "we cannot rule out", and "we cannot discount" to reflect an unlikely—or even remote—event whose consequences are such it warrants mentioning. Words such as "may be" and "suggest" are used to reflect situations in which we are unable to assess the likelihood generally because relevant information is nonexistent, sketchy, or fragmented.^[12]

This explanation is 'a half-step forward, half-step backward'.^{[ citation needed ]} An agency-sponsored WEP paradigm is progress.^{[ opinion ]} However—an estimative statement that uses "maybe" , "suggest" , or other weasel words is vague and symptomatic of the problem at hand—not its solution.^{[ opinion ]} In 1964 Kent railed against the "restort to expressions of avoidance...which convey a definite meaning but at the same time either absolves us completely of the responsibility or makes the estimate enough removed ... as not to implicate ourselves."^[13]

Mercyhurst experience with standardized WEPs

The Mercyhurst College Institute for Intelligence Studies conducted several experiments investigating the IC's interpretation of WEPs^[14] (results varied)^[15]^[16] their use of WEPs in NIEs over the past three decades] to determine the significant changes in the ways the NIC has been articulating its intelligence judgments over time.^[16] See Analysis of Competing Hypotheses

Mercyhurst's WEP paradigm reduces Kent's schema to its least ambiguous words:

Table 3: Mercyhurst WEPs ^[5]
Almost certain
Highly likely
Likely/probable
Unlikely
Almost certainly not

Analytic confidence and source reliability are expressed on a 1 to 10 scale, with 10 high.

IPCC

The Intergovernmental Panel on Climate Change has recommended^[17] some language for communicating likelihood.

>99%	Virtually certain
>90%	Very likely
>66%	Likely
33 to 66%	About as likely as not
<33%	Unlikely
<10%	Very unlikely
<1%	Exceptionally unlikely

Related Research Articles

Statistical inference is the process of using data analysis to infer properties of an underlying distribution of probability. Inferential statistical analysis infers properties of a population, for example by testing hypotheses and deriving estimates. It is assumed that the observed data set is sampled from a larger population.

The President's Daily Brief, sometimes referred to as the President's Daily Briefing or the President's Daily Bulletin, is a top-secret document produced and given each morning to the president of the United States; it is also distributed to a small number of top-level US officials who are approved by the president. It includes highly classified intelligence analysis, information about covert operations, and reports from the most sensitive US sources or those shared by allied intelligence agencies. At the discretion of the president, the PDB may also be provided to the president-elect of the United States, between election day and inauguration, and to former presidents on request.

"Bin Ladin Determined To Strike in US" was a President's Daily Brief prepared by the Central Intelligence Agency and given to U.S. President George W. Bush on Monday, August 6, 2001. The brief warned, 36 days before the September 11 attacks, of terrorism threats from Osama bin Laden and al-Qaeda, including "patterns of suspicious activity in this country consistent with preparations for a hijacking" of U.S. aircraft.

The Akaike information criterion (AIC) is an estimator of prediction error and thereby relative quality of statistical models for a given set of data. Given a collection of models for the data, AIC estimates the quality of each model, relative to each of the other models. Thus, AIC provides a means for model selection.

The Bureau of Intelligence and Research (INR) is an intelligence agency in the United States Department of State. Its central mission is to provide all-source intelligence and analysis in support of U.S. diplomacy and foreign policy. INR is the oldest civilian element of the U.S. Intelligence Community and among the smallest, with roughly 300 personnel. Though lacking the resources and technology of other U.S. intelligence agencies, it is "one of the most highly regarded" for the quality of its work.

The Saddam–al-Qaeda conspiracy theory was based on false claims by the United States government alleging that a secretive relationship existed between Iraqi president Saddam Hussein and the Sunni pan-Islamist militant organization al-Qaeda between 1992 and 2003. U.S. president George W. Bush used it as a main reason for invading Iraq in 2003.

Sherman Kent, was a Yale University history professor who, during World War II and through 17 years of Cold War-era service in the Central Intelligence Agency, pioneered many of the methods of intelligence analysis. He is often described as "the father of intelligence analysis".

Intelligence analysis is the application of individual and collective cognitive methods to weigh data and test hypotheses within a secret socio-cultural context. The descriptions are drawn from what may only be available in the form of deliberately deceptive information; the analyst must correlate the similarities among deceptions and extract a common truth. Although its practice is found in its purest form inside national intelligence agencies, its methods are also applicable in fields such as business intelligence or competitive intelligence.

The analysis of competing hypotheses (ACH) is a methodology for evaluating multiple competing hypotheses for observed data. It was developed by Richards (Dick) J. Heuer, Jr., a 45-year veteran of the Central Intelligence Agency, in the 1970s for use by the Agency. ACH is used by analysts in various fields who make judgments that entail a high risk of error in reasoning. ACH aims to help an analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult to achieve.

Intellipedia is an online system for collaborative data sharing used by the United States Intelligence Community (IC). It was established as a pilot project in late 2005 and formally announced in April 2006. Intellipedia consists of three wikis running on the separate JWICS (Intellipedia-TS), SIPRNet (Intellipedia-S), and DNI-U (Intellipedia-U) networks. The levels of classification allowed for information on the three wikis are Top Secret Sensitive Compartmented Information, Secret (S), and Sensitive But Unclassified information, respectively. Each of the wikis is used by individuals with appropriate clearances from the 18 agencies of the US intelligence community and other national-security related organizations, including Combatant Commands and other federal departments. The wikis are not open to the public.

The Ridge College of Intelligence Studies and Applied Sciences at Mercyhurst University (RIAP), located on the campus of Mercyhurst University in Erie, Pennsylvania, offers undergraduate and graduate studies programs in intelligence analysis. The program also offers graduate certificates in Applied Intelligence, Counterintelligence, and Law Enforcement Intelligence. The Intelligence Studies program "promotes the study of Intelligence in higher academic settings, while seeking to identify, promote, and employ best practices in the study and application of intelligence studies throughout its various disciplines (national security, law enforcement, business intelligence and academia)."

<span class="mw-page-title-main">Bin Laden Issue Station</span> CIA unit tracking Osama bin Laden (1996–2005)

The Bin Laden Issue Station, also known as Alec Station, was a standalone unit of the Central Intelligence Agency in operation from 1996 to 2005 dedicated to tracking Osama bin Laden and his associates, both before and after the 9/11 attacks. It was headed initially by CIA analyst Michael Scheuer and later by Richard Blee and others.

Intelligence Analysis Management is the process of managing and organizing the analytical processing of raw intelligence information. The terms "analysis", "production", and "processing" denote the organization and evaluation of raw information used in a phase informally called "connecting the dots", thus creating an "intelligence mosaic". The information may result in multiple analytic products, each with different security classifications, time scales, and levels of detail. Intelligence analysis goes back to the beginning of history. Sherman Kent is often considered the father of modern intelligence analysis. His writings include a 1947 book, Strategic Intelligence for American World Policy.

The United States Intelligence Community A-Space, or Analytic Space, is a project started in 2007 from the Office of the Director of National Intelligence's (ODNI) Office of Analytic Transformation and Technology to develop a common collaborative workspace for all analysts from the USIC. It is accessible from common workstations and provides unprecedented access to interagency databases, a capability to search classified and unclassified sources simultaneously, web-based messaging, and collaboration tools. The Defense Intelligence Agency (DIA) is the executive agent for building the first phase of A-Space. Initial operational capability was scheduled for December 2007. A-Space went live on the government's classified Joint Worldwide Intelligence Communications System 22 September 2008. A-Space is built on Jive Software's Clearspace application.

This article deals with activities of the U.S. Central Intelligence Agency, specifically dealing with arms control, weapons of mass destruction (WMD) and weapons proliferation. It attempts to look at the process of tasking and analyzing, rather than the problem itself, other than whether the CIA's efforts match its legal mandate or assists in treaty compliance. In some cases, the details of a country's programs are introduced because they present a problem in analysis. For example, if Country X's policymakers truly believe in certain history that may not actually be factual, an analyst trying to understand Country X's policymakers needs to be able to understand their approach to an issue.

Analytic confidence is a rating employed by intelligence analysts to convey doubt to decision makers about a statement of estimative probability. The need for analytic confidence ratings arise from analysts' imperfect knowledge of a conceptual model. An analytic confidence rating pairs with a statement using a word of estimative probability to form a complete analytic statement. Scientific methods for determining analytic confidence remain in infancy.

Richards "Dick" J. Heuer, Jr. was a CIA veteran of 45 years and most known for his work on analysis of competing hypotheses and his book, Psychology of Intelligence Analysis. The former provides a methodology for overcoming intelligence biases while the latter outlines how mental models and natural biases impede clear thinking and analysis. Throughout his career, he worked in collection operations, counterintelligence, intelligence analysis and personnel security. In 2010 he co-authored a book with Randolph (Randy) H. Pherson titled Structured Analytic Techniques for Intelligence Analysis.

The Central Intelligence Agency (CIA), a United States intelligence agency that "provides objective intelligence on foreign countries", also informally referred to as the Agency. The CIA is part of the United States Intelligence Community, is organized into numerus divisions. The divisions include directors, deputy directors, and offices. The CIA board is made up of five distinct entitles called Directorates. The CIA is overseen by the Director of Central Intelligence. Under the Director of Central Intelligence is the Deputy Director of Central Intelligence. Under this the CIA is divided into four directorates. These directorates are as follows:

In simple terms, risk is the possibility of something bad happening. Risk involves uncertainty about the effects/implications of an activity with respect to something that humans value, often focusing on negative, undesirable consequences. Many different definitions have been proposed. The international standard definition of risk for common understanding in different applications is "effect of uncertainty on objectives".

Geographic information systems (GIS) play a constantly evolving role in geospatial intelligence (GEOINT) and United States national security. These technologies allow a user to efficiently manage, analyze, and produce geospatial data, to combine GEOINT with other forms of intelligence collection, and to perform highly developed analysis and visual production of geospatial data. Therefore, GIS produces up-to-date and more reliable GEOINT to reduce uncertainty for a decisionmaker. Since GIS programs are Web-enabled, a user can constantly work with a decision maker to solve their GEOINT and national security related problems from anywhere in the world. There are many types of GIS software used in GEOINT and national security, such as Google Earth, ERDAS IMAGINE, GeoNetwork opensource, and Esri ArcGIS.

References

↑ Sherman Kent and the Profession of Intelligence Analysis, Center for the Study of Intelligence, Central Intelligence Agency, November 2002, p. 50, archived from the original on June 12, 2007
↑ Sherman Kent and the Profession of Intelligence Analysis, Center for the Study of Intelligence, Central Intelligence Agency, November 2002, p. 55, archived from the original on June 12, 2007
↑ Wozny, J.L. (2005), Optimizing Communication Between Decisionmakers And Intelligence Analysts: Stopping "Slam Dunks" And Avoiding "Dead Wrongs" (MS Thesis), Erie, PA: Mercyhurst College, p. 192
↑ Woodward, B.; Eggen, D. (2002-05-18), "August Memo Focused On Attacks in U.S", Washington Post, retrieved 2008-04-21– via Common Dreams^{[ dead link ]}
1 2 3 Wheaton, K.; Chido, Diane (2008), "Words of Estimative Probability", Competitive Intelligence Magazine, Alexandria, VA: Society of Competitive Intelligence Professionals, ISSN 1521-5881 ^{[ dead link ]}
↑ University of Tennessee at Chattanooga, Institutional Review Board (2008), Institutional Review Board Frequently Asked Questions
↑ Schrage, Michael, Michael Schrage homepage at the Massachusetts Institute of Technology, archived from the original on April 19, 2008
↑ Schrage, Michael (20 February 2005), "What Percent Is 'Slam Dunk'? Give Us Odds on Those Estimates", Washington Post, p. B01
↑ Schrage, 2003
↑ Blair, Bruce (2004), The Logic of Intelligence Failure (PDF), Center for Defense Information, p. 11, retrieved 2008-04-21
↑ "Prospects for Iraq's Stability: A Challenging Road Ahead" (PDF), What We Mean When We Say: An Explanation of Estimative Language, National Intelligence Council, p. 5, 2007, retrieved 2008-04-21
↑ National Intelligence Council, 2007, p5
↑ Sherman Kent and the Profession of Intelligence Analysis, Center for the Study of Intelligence, Central Intelligence Agency, November 2002, p. 63, archived from the original on June 12, 2007
↑ "What do Words of Estimative Probability Mean? (Final Version with Abstract)". 24 March 2008.
↑ Wheaton, K. (24 March 2008), What Do Words Of Estimative Probability Mean? An Exercise In Analyst Education , retrieved 2008-04-24
1 2 Kesselman, Rachel (26 March 2008), "Estimative Words of Probability Trends in National Intelligence Estimates", International Studies Association 49th Annual Convention: Bridging Multiple Divides
↑ IPCC (2005), Guidance Notes for Lead Authors of the IPCC Fourth Assessment Report on Addressing Uncertainties, July 2005. PDF version.