Expert elicitation

Last updated

In science, engineering, and research, expert elicitation is the synthesis of opinions of authorities of a subject where there is uncertainty due to insufficient data or when such data is unattainable because of physical constraints or lack of resources. [1] Expert elicitation is essentially a scientific consensus methodology. It is often used in the study of rare events. [2] Expert elicitation allows for parametrization, an "educated guess", for the respective topic under study. Expert elicitation generally quantifies uncertainty.

Contents

Expert elicitation tends to be multidisciplinary as well as interdisciplinary, with practically universal applicability, and is used in a broad range of fields. Prominent recent expert elicitation applications include climate change, modeling seismic hazard and damage, association of tornado damage to wind speed in developing the Enhanced Fujita scale, risk analysis for nuclear waste storage.

In performing expert elicitation certain factors need to be taken into consideration. The topic must be one for which there are people who have predictive expertise. Furthermore, the objective should be to obtain an experts' carefully considered judgment based on a systematic consideration of all relevant evidence. For this reason one should take care to adopt strategies designed to help the expert being interviewed to avoid overlooking relevant evidence. Additionally, vocabulary used should face intense scrutiny; qualitative uncertainty words such as "likely" and "unlikely" are not sufficient and can lead to confusion. Such words can mean very different things to different people, or to the same people in different situations. [3]

See also

Related Research Articles

Broadly speaking, a risk assessment is the combined effort of:

  1. identifying and analyzing potential (future) events that may negatively impact individuals, assets, and/or the environment ; and
  2. making judgments "on the tolerability of the risk on the basis of a risk analysis" while considering influencing factors.
<span class="mw-page-title-main">Seismic hazard</span> Probability that an earthquake will occur in a given geographic area, within a given window of time

A seismic hazard is the probability that an earthquake will occur in a given geographic area, within a given window of time, and with ground motion intensity exceeding a given threshold. With a hazard thus estimated, risk can be assessed and included in such areas as building codes for standard buildings, designing larger buildings and infrastructure projects, land use planning and determining insurance rates. The seismic hazard studies also may generate two standard measures of anticipated ground motion, both confusingly abbreviated MCE; the simpler probabilistic Maximum Considered Earthquake, used in standard building codes, and the more detailed and deterministic Maximum Credible Earthquake incorporated in the design of larger buildings and civil infrastructure like dams or bridges. It is important to clarify which MCE is being discussed.

Induced seismicity is typically earthquakes and tremors that are caused by human activity that alters the stresses and strains on Earth's crust. Most induced seismicity is of a low magnitude. A few sites regularly have larger quakes, such as The Geysers geothermal plant in California which averaged two M4 events and 15 M3 events every year from 2004 to 2009. The Human-Induced Earthquake Database (HiQuake) documents all reported cases of induced seismicity proposed on scientific grounds and is the most complete compilation of its kind.

Decision analysis (DA) is the discipline comprising the philosophy, methodology, and professional practice necessary to address important decisions in a formal manner. Decision analysis includes many procedures, methods, and tools for identifying, clearly representing, and formally assessing important aspects of a decision; for prescribing a recommended course of action by applying the maximum expected-utility axiom to a well-formed representation of the decision; and for translating the formal representation of a decision and its corresponding recommendation into insight for the decision maker, and other corporate and non-corporate stakeholders.

Probabilistic risk assessment (PRA) is a systematic and comprehensive methodology to evaluate risks associated with a complex engineered technological entity or the effects of stressors on the environment.

WASH-1400, 'The Reactor Safety Study', was a report produced in 1975 for the Nuclear Regulatory Commission by a committee of specialists under Professor Norman Rasmussen. It "generated a storm of criticism in the years following its release". In the years immediately after its release, WASH-1400 was followed by a number of reports that either peer reviewed its methodology or offered their own judgments about probabilities and consequences of various events at commercial reactors. In at least a few instances, some offered critiques of the study's assumptions, methodology, calculations, peer review procedures, and objectivity. A succession of reports, including NUREG-1150, the State-of-the-Art Reactor Consequence Analyses and others, have carried-on the tradition of PRA and its application to commercial power plants.

Probabilistic forecasting summarizes what is known about, or opinions about, future events. In contrast to single-valued forecasts, probabilistic forecasts assign a probability to each of a number of different outcomes, and the complete set of probabilities represents a probability forecast. Thus, probabilistic forecasting is a type of probabilistic classification.

Uncertainty quantification (UQ) is the science of quantitative characterization and reduction of uncertainties in both computational and real world applications. It tries to determine how likely certain outcomes are if some aspects of the system are not exactly known. An example would be to predict the acceleration of a human body in a head-on crash with another car: even if the speed was exactly known, small differences in the manufacturing of individual cars, how tightly every bolt has been tightened, etc., will lead to different results that can only be predicted in a statistical sense.

<span class="mw-page-title-main">Event chain methodology</span> Network analysis technique

Event chain methodology is a network analysis technique that is focused on identifying and managing events and relationship between them that affect project schedules. It is an uncertainty modeling schedule technique. Event chain methodology is an extension of quantitative project risk analysis with Monte Carlo simulations. It is the next advance beyond critical path method and critical chain project management. Event chain methodology tries to mitigate the effect of motivational and cognitive biases in estimating and scheduling. It improves accuracy of risk assessment and helps to generate more realistic risk adjusted project schedules.

Earthquake forecasting is a branch of the science of seismology concerned with the probabilistic assessment of general earthquake seismic hazard, including the frequency and magnitude of damaging earthquakes in a given area over years or decades. While forecasting is usually considered to be a type of prediction, earthquake forecasting is often differentiated from earthquake prediction, whose goal is the specification of the time, location, and magnitude of future earthquakes with sufficient precision that a warning can be issued. Both forecasting and prediction of earthquakes are distinguished from earthquake warning systems, which, upon detection of an earthquake, provide a real-time warning to regions that might be affected.

A natural hazard is a natural phenomenon that might have a negative effect on humans and other animals, or the environment. Natural hazard events can be classified into two broad categories: geophysical and biological.

Quantification of Margins and Uncertainty (QMU) is a decision support methodology for complex technical decisions. QMU focuses on the identification, characterization, and analysis of performance thresholds and their associated margins for engineering systems that are evaluated under conditions of uncertainty, particularly when portions of those results are generated using computational modeling and simulation. QMU has traditionally been applied to complex systems where comprehensive experimental test data is not readily available and cannot be easily generated for either end-to-end system execution or for specific subsystems of interest. Examples of systems where QMU has been applied include nuclear weapons performance, qualification, and stockpile assessment. QMU focuses on characterizing in detail the various sources of uncertainty that exist in a model, thus allowing the uncertainty in the system response output variables to be well quantified. These sources are frequently described in terms of probability distributions to account for the stochastic nature of complex engineering systems. The characterization of uncertainty supports comparisons of design margins for key system performance metrics to the uncertainty associated with their calculation by the model. QMU supports risk-informed decision-making processes where computational simulation results provide one of several inputs to the decision-making authority. There is currently no standardized methodology across the simulation community for conducting QMU; the term is applied to a variety of different modeling and simulation techniques that focus on rigorously quantifying model uncertainty in order to support comparison to design margins.

The technique for human error-rate prediction (THERP) is a technique used in the field of human reliability assessment (HRA), for the purposes of evaluating the probability of a human error occurring throughout the completion of a specific task. From such analyses measures can then be taken to reduce the likelihood of errors occurring within a system and therefore lead to an improvement in the overall levels of safety. There exist three primary reasons for conducting an HRA: error identification, error quantification and error reduction. As there exist a number of techniques used for such purposes, they can be split into one of two classifications: first-generation techniques and second-generation techniques. First-generation techniques work on the basis of the simple dichotomy of ‘fits/doesn’t fit’ in matching an error situation in context with related error identification and quantification. Second generation techniques are more theory-based in their assessment and quantification of errors. ‘HRA techniques have been utilised for various applications in a range of disciplines and industries including healthcare, engineering, nuclear, transportation and business.

In simple terms, risk is the possibility of something bad happening. Risk involves uncertainty about the effects/implications of an activity with respect to something that humans value, often focusing on negative, undesirable consequences. Many different definitions have been proposed. The international standard definition of risk for common understanding in different applications is “effect of uncertainty on objectives”.

Risk management tools allow the uncertainty to be addressed by identifying and generating metrics, parameterizing, prioritizing, and developing responses, and tracking risk. These activities may be difficult to track without tools and techniques, documentation and information systems.

<span class="mw-page-title-main">Probability box</span> Characterization of uncertain numbers consisting of both aleatoric and epistemic uncertainties

A probability box is a characterization of uncertain numbers consisting of both aleatoric and epistemic uncertainties that is often used in risk analysis or quantitative uncertainty modeling where numerical calculations must be performed. Probability bounds analysis is used to make arithmetic and logical calculations with p-boxes.

Probability bounds analysis (PBA) is a collection of methods of uncertainty propagation for making qualitative and quantitative calculations in the face of uncertainties of various kinds. It is used to project partial information about random variables and other quantities through mathematical expressions. For instance, it computes sure bounds on the distribution of a sum, product, or more complex function, given only sure bounds on the distributions of the inputs. Such bounds are called probability boxes, and constrain cumulative probability distributions.

P-boxes and probability bounds analysis have been used in many applications spanning many disciplines in engineering and environmental science, including:

Expert Judgment (EJ) denotes a wide variety of techniques ranging from a single undocumented opinion, through preference surveys, to formal elicitation with external validation of expert probability assessments. Recent books are . In the nuclear safety area, Rasmussen formalized EJ by documenting all steps in the expert elicitation process for scientific review. This made visible wide spreads in expert assessments and teed up questions regarding the validation and synthesis of expert judgments. The nuclear safety community later took onboard expert judgment techniques underpinned by external validation . Empirical validation is the hallmark of science, and forms the centerpiece of the classical model of probabilistic forecasting . A European Network coordinates workshops. Application areas include nuclear safety, investment banking, volcanology, public health, ecology, engineering, climate change and aeronautics/aerospace. For a survey of applications through 2006 see and give exhortatory overviews. A recent large scale implementation by the World Health Organization is described in . A long running application at the Montserrat Volcano Observatory is described in . The classical model scores expert performance in terms of statistical accuracy and informativeness . These terms should not be confused with “accuracy and precision”. Accuracy “is a description of systematic errors” while precision “is a description of random errors”. In the classical model statistical accuracy is measured as the p-value or probability with which one would falsely reject the hypotheses that an expert's probability assessments were statistically accurate. A low value means it is very unlikely that the discrepancy between an expert's probability statements and observed outcomes should arise by chance. Informativeness is measured as Shannon relative information with respect to an analyst-supplied background measure. Shannon relative information is used because it is scale invariant, tail insensitive, slow, and familiar. Parenthetically, measures with physical dimensions, such as the standard deviation, or the width of prediction intervals, raise serious problems, as a change of units would affect some variables but not others. The product of statistical accuracy and informativeness for each expert is their combined score. With an optimal choice of a statistical accuracy threshold beneath which experts are unweighted, the combined score is a long run “strictly proper scoring rule”: an expert achieves his long run maximal expected score by and only by stating his true beliefs. The classical model derives Performance Weighted (PW) combinations. These are compared with Equally Weighted (EW) combinations, and recently with Harmonically Weighted (HW) combinations, as well as with individual expert assessments.

<span class="mw-page-title-main">Omar-Darío Cardona Arboleda</span> Columbian author & academic

Omar-Darío Cardona Arboleda is a civil engineer, academic, and author. He is a Titular Professor of integrated disaster risk management and climate change adaptation in the Institute of Environment Studies at the National University of Colombia, Co-founder, and CEO of Ingeniar: Risk Intelligence.

References

  1. van der Sluijs, Jeroen P.; et al. (2008). "Expert Elicitation: Methodological suggestions for its use in environmental health impact assessments" (PDF). NUSAP. Retrieved 25 November 2015.
  2. Schwarzenegger, Rafael; Quigley, John; Walls, Lesley (23 November 2021). "Is eliciting dependency worth the effort? A study for the multivariate Poisson-Gamma probability model". Proceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability: 1748006X2110594. doi: 10.1177/1748006X211059417 . S2CID   244549831.
  3. Tversky, Amos; Kahneman, Daniel (27 September 1974). Judgments under uncertainty: Heuristics and biases (PDF) (Vol. 185, No. 4157 ed.). Science. pp. 1124–1131. Archived from the original (PDF) on 28 May 2019. Retrieved 25 November 2015.

Bibliography