Reforms to methodology
Estimative statements can be improved in four ways;[ opinion ] either by:
- Adding quantitative source reliability and confidence measures to estimative statements[ citation needed ]
- Complementing estimative statements with stochastic analyses[ citation needed ]
- Standardizing WEPs[ citation needed ]
- Standardizing WEPs and complementing estimative statements with ratings of source reliability and analytic confidence [ citation needed ]
Quantification of source reliability and analytic confidence
Michael Schrage, [7] an advisor to the Massachusetts Institute of Technology's (MIT) Security Studies Program, wrote in a Washington Post editorial that requiring analysts to produce and include quantitative measures of source reliability and confidence along with their findings would reduce ambiguity. [8] Yet Schrage also reported that former Interim Director of Central Intelligence John E. McLaughlin’s attempted to enact this at the CIA, but, like Kent’s initiative, it was not adopted.
Former acting CIA director and longtime analyst John McLaughlin tried to promote greater internal efforts at assigning probabilities to intelligence assessments during the 1990s, but they never took. Intelligence analysts "would rather use words than numbers to describe how confident we are in our analysis," a senior CIA officer who's served for more than 20 years told me. Moreover, "most consumers of intelligence aren't particularly sophisticated when it comes to probabilistic analysis. They like words and pictures, too. My experience is that [they] prefer briefings that don't center on numerical calculation. That's not to say we can't do it, but there's not that much demand for it." [9]
Stochastic analysis
Since combining quantitative, probabilistic information with estimates is successful in business forecasting, marketing, medicine, and epidemiology it should be implemented by the intelligence community as well. These fields have used probability theory and Bayesian analysis as forecasting tools. Using probability theory and other stochastic methods are appealing because they rely on rationality and mathematical rigor, are less subject to analytical bias, and such findings appear to be unambiguous. As an opposing argument, it is indisputable that few analysts or intelligence consumers have the stomach for numerical calculation. Additionally, Bruce Blair, Director of the Center for Defense Information, a proponent of quantitative methods for the IC, points out; intelligence information from secret sources is often murky, and the application of advanced math is not sufficient to make it more reliable. However, he sees a place for stochastic analyses over a very long period, it “points to a fairly slow learning curve that also challenges the wisdom of making preemption a cornerstone of U.S. security strategy.” [10] The reservations stated are significant: Mathematical and statistical analyses require a lot of work without rapid and necessarily commensurate gains in accuracy, speed or comprehension.
Standardization
UK intelligence assessments use the PHIA "Probability Yardstick" for communicating probability:
The National Intelligence Council's recommendations described the use of a WEP paradigm (Table 2) in combination with an assessment of confidence levels (“High, moderate, low”) based on the scope and quality supporting information:
| Table 2: National Intelligence Council WEPs [11] |
|---|
| Almost Certainly |
| Probably/Likely |
| Even Chance |
| Unlikely |
| Remote |
However, the NIC's discussion of this paradigm seems to undercut its chances of being effective:
Intelligence judgments about likelihood are intended to reflect the Community’s sense of the probability of a development or event. [...] We do not intend the term “unlikely” to imply an event will not happen. We use “probably” and “likely” to indicate there is a greater than even chance. We use words such as “we cannot dismiss,” “we cannot rule out,” and “we cannot discount” to reflect an unlikely—or even remote—event whose consequences are such it warrants mentioning. Words such as “may be” and “suggest” are used to reflect situations in which we are unable to assess the likelihood generally because relevant information is nonexistent, sketchy, or fragmented. [12]
This explanation is ‘a half-step forward, half-step backward.’[ citation needed ] An agency-sponsored WEP paradigm is progress.[ opinion ] However—an estimative statement that uses “maybe” , “suggest” , or other weasel words is vague and symptomatic of the problem at hand—not its solution.[ opinion ] In 1964 Kent railed against the “restort to expressions of avoidance...which convey a definite meaning but at the same time either absolves us completely of the responsibility or makes the estimate enough removed ... as not to implicate ourselves.” [13]
Mercyhurst experience with standardized WEPs
The Mercyhurst College Institute for Intelligence Studies conducted several experiments investigating the IC's interpretation of WEPs [14] (results varied) [15] [16] their use of WEPs in NIEs over the past three decades] to determine the significant changes in the ways the NIC has been articulating its intelligence judgments over time. [16] See Analysis of Competing Hypotheses
Mercyhurst's WEP paradigm reduces Kent's schema to its least ambiguous words:
| Table 3: Mercyhurst WEPs [5] |
|---|
| Almost Certain |
| Highly Likely |
| Likely/Probable |
| Unlikely |
| Almost Certainly Not |
Analytic Confidence and Source Reliability are expressed on a 1 to 10 scale, with 10 high.
IPCC
The Intergovernmental Panel on Climate Change has recommended [17] some language for communicating likelihood.
| >99% | Virtually certain |
| >90% | Very likely |
| >66% | Likely |
| 33 to 66% | About as likely as not |
| <33% | Unlikely |
| <10% | Very unlikely |
| <1% | Exceptionally unlikely |
Weasel words
Table 4 contains a non-exhaustive list of common terms that are especially vague, known pejoratively as Weasel Words.[ opinion ] Their use in estimative statements is almost certain to confuse; they should be avoided at all costs.[ citation needed ]
| Table 4: Misleading WEPs: Avoid![ opinion ] | |
|---|---|
| ... | |
| ... | |
| ... | ... |
