Expert elicitation

Last updated

In science, engineering, and research, expert elicitation is the synthesis of opinions of authorities of a subject where there is uncertainty due to insufficient data or when such data is unattainable because of physical constraints or lack of resources. [1] Expert elicitation is essentially a scientific consensus methodology. It is often used in the study of rare events. [2] Expert elicitation allows for parametrization, an "educated guess", for the respective topic under study. Expert elicitation generally quantifies uncertainty.

Contents

Expert elicitation tends to be multidisciplinary as well as interdisciplinary, with practically universal applicability, and is used in a broad range of fields. Prominent recent expert elicitation applications include climate change, modeling seismic hazard and damage, association of tornado damage to wind speed in developing the Enhanced Fujita scale, risk analysis for nuclear waste storage.

In performing expert elicitation certain factors need to be taken into consideration. The topic must be one for which there are people who have predictive expertise. Furthermore, the objective should be to obtain an experts' carefully considered judgment based on a systematic consideration of all relevant evidence. For this reason one should take care to adopt strategies designed to help the expert being interviewed to avoid overlooking relevant evidence. Additionally, vocabulary used should face intense scrutiny; qualitative uncertainty words such as "likely" and "unlikely" are not sufficient and can lead to confusion. Such words can mean very different things to different people, or to the same people in different situations. [3]

See also

Related Research Articles

Risk assessment determines possible mishaps, their likelihood and consequences, and the tolerances for such events. The results of this process may be expressed in a quantitative or qualitative fashion. Risk assessment is an inherent part of a broader risk management strategy to help reduce any potential risk-related consequences.

Induced seismicity is typically earthquakes and tremors that are caused by human activity that alters the stresses and strains on Earth's crust. Most induced seismicity is of a low magnitude. A few sites regularly have larger quakes, such as The Geysers geothermal plant in California which averaged two M4 events and 15 M3 events every year from 2004 to 2009. The Human-Induced Earthquake Database (HiQuake) documents all reported cases of induced seismicity proposed on scientific grounds and is the most complete compilation of its kind.

Decision analysis (DA) is the discipline comprising the philosophy, methodology, and professional practice necessary to address important decisions in a formal manner. Decision analysis includes many procedures, methods, and tools for identifying, clearly representing, and formally assessing important aspects of a decision; for prescribing a recommended course of action by applying the maximum expected-utility axiom to a well-formed representation of the decision; and for translating the formal representation of a decision and its corresponding recommendation into insight for the decision maker, and other corporate and non-corporate stakeholders.

Probabilistic risk assessment (PRA) is a systematic and comprehensive methodology to evaluate risks associated with a complex engineered technological entity or the effects of stressors on the environment.

WASH-1400, 'The Reactor Safety Study was a report produced in 1975 for the Nuclear Regulatory Commission by a committee of specialists under Professor Norman Rasmussen. It "generated a storm of criticism in the years following its release". In the years immediately after its release, WASH-1400 was followed by a number of reports that either peer reviewed its methodology or offered their own judgments about probabilities and consequences of various events at commercial reactors. In at least a few instances, some offered critiques of the study's assumptions, methodology, calculations, peer review procedures, and objectivity. A succession of reports, including NUREG-1150, the State-of-the-Art Reactor Consequence Analyses and others, have carried-on the tradition of PRA and its application to commercial power plants.

<span class="mw-page-title-main">Risk perception</span>

Risk perception is the subjective judgement that people make about the characteristics and severity of a risk. Risk perceptions often differ from statistical assessments of risk since they are affected by a wide range of affective, cognitive, contextual, and individual factors. Several theories have been proposed to explain why different people make different estimates of the dangerousness of risks. Three major families of theory have been developed: psychology approaches, anthropology/sociology approaches and interdisciplinary approaches.

Probabilistic forecasting summarizes what is known about, or opinions about, future events. In contrast to single-valued forecasts, probabilistic forecasts assign a probability to each of a number of different outcomes, and the complete set of probabilities represents a probability forecast. Thus, probabilistic forecasting is a type of probabilistic classification.

Uncertainty quantification (UQ) is the science of quantitative characterization and estimation of uncertainties in both computational and real world applications. It tries to determine how likely certain outcomes are if some aspects of the system are not exactly known. An example would be to predict the acceleration of a human body in a head-on crash with another car: even if the speed was exactly known, small differences in the manufacturing of individual cars, how tightly every bolt has been tightened, etc., will lead to different results that can only be predicted in a statistical sense.

Earthquake forecasting is a branch of the science of seismology concerned with the probabilistic assessment of general earthquake seismic hazard, including the frequency and magnitude of damaging earthquakes in a given area over years or decades. While forecasting is usually considered to be a type of prediction, earthquake forecasting is often differentiated from earthquake prediction, Earthquake forecasting estimates the likelihood of earthquakes in a specific timeframe and region, while earthquake prediction attempts to pinpoint the exact time, location, and magnitude of an impending quake, which is currently not reliably achievable.Wood & Gutenberg (1935). Kagan says: "This definition has several defects which contribute to confusion and difficulty in prediction research." In addition to specification of time, location, and magnitude, Allen suggested three other requirements: 4) indication of the author's confidence in the prediction, 5) the chance of an earthquake occurring anyway as a random event, and 6) publication in a form that gives failures the same visibility as successes. Kagan & Knopoff define prediction "to be a formal rule where by the available space-time-seismic moment manifold of earthquake occurrence is significantly contracted ...."</ref> Both forecasting and prediction of earthquakes are distinguished from earthquake warning systems, which, upon detection of an earthquake, provide a real-time warning to regions that might be affected.

Quantification of Margins and Uncertainty (QMU) is a decision support methodology for complex technical decisions. QMU focuses on the identification, characterization, and analysis of performance thresholds and their associated margins for engineering systems that are evaluated under conditions of uncertainty, particularly when portions of those results are generated using computational modeling and simulation. QMU has traditionally been applied to complex systems where comprehensive experimental test data is not readily available and cannot be easily generated for either end-to-end system execution or for specific subsystems of interest. Examples of systems where QMU has been applied include nuclear weapons performance, qualification, and stockpile assessment. QMU focuses on characterizing in detail the various sources of uncertainty that exist in a model, thus allowing the uncertainty in the system response output variables to be well quantified. These sources are frequently described in terms of probability distributions to account for the stochastic nature of complex engineering systems. The characterization of uncertainty supports comparisons of design margins for key system performance metrics to the uncertainty associated with their calculation by the model. QMU supports risk-informed decision-making processes where computational simulation results provide one of several inputs to the decision-making authority. There is currently no standardized methodology across the simulation community for conducting QMU; the term is applied to a variety of different modeling and simulation techniques that focus on rigorously quantifying model uncertainty in order to support comparison to design margins.

The Technique for human error-rate prediction (THERP) is a technique that is used in the field of Human Reliability Assessment (HRA) to evaluate the probability of human error occurring throughout the completion of a task. From such an analysis, some corrective measures could be taken to reduce the likelihood of errors occurring within a system. The overall goal of THERP is to apply and document probabilistic methodological analyses to increase safety during a given process. THERP is used in fields such as error identification, error quantification and error reduction.

<span class="mw-page-title-main">Risk</span> The possibility of something bad happening

In simple terms, risk is the possibility of something bad happening. Risk involves uncertainty about the effects/implications of an activity with respect to something that humans value, often focusing on negative, undesirable consequences. Many different definitions have been proposed. The international standard definition of risk for common understanding in different applications is "effect of uncertainty on objectives".

<span class="mw-page-title-main">C. Allin Cornell</span> American engineer

Carl Allin Cornell was an American civil engineer, researcher, and professor who made important contributions to reliability theory and earthquake engineering and, along with Luis Esteva, developed the field of probabilistic seismic hazard analysis by publishing the seminal document of the field in 1968.

Risk management tools allow the uncertainty to be addressed by identifying and generating metrics, parameterizing, prioritizing, and developing responses, and tracking risk. These activities may be difficult to track without tools and techniques, documentation and information systems.

<span class="mw-page-title-main">Probability box</span> Characterization of uncertain numbers consisting of both aleatoric and epistemic uncertainties

A probability box is a characterization of uncertain numbers consisting of both aleatoric and epistemic uncertainties that is often used in risk analysis or quantitative uncertainty modeling where numerical calculations must be performed. Probability bounds analysis is used to make arithmetic and logical calculations with p-boxes.

Probability bounds analysis (PBA) is a collection of methods of uncertainty propagation for making qualitative and quantitative calculations in the face of uncertainties of various kinds. It is used to project partial information about random variables and other quantities through mathematical expressions. For instance, it computes sure bounds on the distribution of a sum, product, or more complex function, given only sure bounds on the distributions of the inputs. Such bounds are called probability boxes, and constrain cumulative probability distributions.

P-boxes and probability bounds analysis have been used in many applications spanning many disciplines in engineering and environmental science, including:

Expert Judgment (EJ) denotes a wide variety of techniques ranging from a single undocumented opinion, through preference surveys, to formal elicitation with external validation of expert probability assessments. Recent books are . In the nuclear safety area, Rasmussen formalized EJ by documenting all steps in the expert elicitation process for scientific review. This made visible wide spreads in expert assessments and teed up questions regarding the validation and synthesis of expert judgments. The nuclear safety community later took onboard expert judgment techniques underpinned by external validation . Empirical validation is the hallmark of science, and forms the centerpiece of the classical model of probabilistic forecasting . A European Network coordinates workshops. Application areas include nuclear safety, investment banking, volcanology, public health, ecology, engineering, climate change and aeronautics/aerospace. For a survey of applications through 2006 see and give exhortatory overviews. A recent large scale implementation by the World Health Organization is described in . A long running application at the Montserrat Volcano Observatory is described in . The classical model scores expert performance in terms of statistical accuracy and informativeness . These terms should not be confused with “accuracy and precision”. Accuracy “is a description of systematic errors” while precision “is a description of random errors”. In the classical model statistical accuracy is measured as the p-value or probability with which one would falsely reject the hypotheses that an expert's probability assessments were statistically accurate. A low value means it is very unlikely that the discrepancy between an expert's probability statements and observed outcomes should arise by chance. Informativeness is measured as Shannon relative information with respect to an analyst-supplied background measure. Shannon relative information is used because it is scale invariant, tail insensitive, slow, and familiar. Parenthetically, measures with physical dimensions, such as the standard deviation, or the width of prediction intervals, raise serious problems, as a change of units would affect some variables but not others. The product of statistical accuracy and informativeness for each expert is their combined score. With an optimal choice of a statistical accuracy threshold beneath which experts are unweighted, the combined score is a long run “strictly proper scoring rule”: an expert achieves his long run maximal expected score by and only by stating his true beliefs. The classical model derives Performance Weighted (PW) combinations. These are compared with Equally Weighted (EW) combinations, and recently with Harmonically Weighted (HW) combinations, as well as with individual expert assessments.

<span class="mw-page-title-main">Omar-Darío Cardona Arboleda</span> Columbian author & academic

Omar-Darío Cardona Arboleda is a civil engineer, academic, and author. He is a Titular Professor of integrated disaster risk management and climate change adaptation in the Institute of Environment Studies at the National University of Colombia, Co-founder, and CEO of Ingeniar: Risk Intelligence.

Belief aggregation, also called risk aggregation,opinion aggregation or probabilistic opinion pooling, is a process in which different probability distributions, produced by different experts, are combined to yield a single probability distribution.

References

  1. van der Sluijs, Jeroen P.; et al. (2008). "Expert Elicitation: Methodological suggestions for its use in environmental health impact assessments" (PDF). NUSAP. Retrieved 25 November 2015.
  2. Schwarzenegger, Rafael; Quigley, John; Walls, Lesley (23 November 2021). "Is eliciting dependency worth the effort? A study for the multivariate Poisson-Gamma probability model". Proceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability. 237 (5): 858–867. doi: 10.1177/1748006X211059417 . S2CID   244549831.
  3. Tversky, Amos; Kahneman, Daniel (27 September 1974). Judgments under uncertainty: Heuristics and biases (PDF) (Vol. 185, No. 4157 ed.). Science. pp. 1124–1131. Archived from the original (PDF) on 28 May 2019. Retrieved 25 November 2015.

Bibliography