Coverage probability

Last updated

In statistical estimation theory, the coverage probability, or coverage for short, is the probability that a confidence interval or confidence region will include the true value (parameter) of interest. It can be defined as the proportion of instances where the interval surrounds the true value as assessed by long-run frequency. [1]

Contents

In statistical prediction, the coverage probability is the probability that a prediction interval will include an out-of-sample value of the random variable. The coverage probability can be defined as the proportion of instances where the interval surrounds an out-of-sample value as assessed by long-run frequency. [2]

Concept

The fixed degree of certainty pre-specified by the analyst, referred to as the confidence level or confidence coefficient of the constructed interval, is effectively the nominal coverage probability of the procedure for constructing confidence intervals. Hence, referring to a "nominal confidence level" or "nominal confidence coefficient" (e.g., as a synonym for nominal coverage probability) generally has to be considered tautological and misleading, as the notion of confidence level itself inherently implies nominality already. [lower-alpha 1] The nominal coverage probability is often set at 0.95. By contrast, the (true) coverage probability is the actual probability that the interval contains the parameter.

If all assumptions used in deriving a confidence interval are met, the nominal coverage probability will equal the coverage probability (termed "true" or "actual" coverage probability for emphasis). If any assumptions are not met, the actual coverage probability could either be less than or greater than the nominal coverage probability. When the actual coverage probability is greater than the nominal coverage probability, the interval is termed a conservative (confidence) interval; if it is less than the nominal coverage probability, the interval is termed anti-conservative, or permissive. For example, suppose the interest is in the mean number of months that people with a particular type of cancer remain in remission following successful treatment with chemotherapy. The confidence interval aims to contain the unknown mean remission duration with a given probability. In this example, the coverage probability would be the real probability that the interval actually contains the true mean remission duration.

A discrepancy between the coverage probability and the nominal coverage probability frequently occurs when approximating a discrete distribution with a continuous one. The construction of binomial confidence intervals is a classic example where coverage probabilities rarely equal nominal levels. [3] [4] [5] For the binomial case, several techniques for constructing intervals have been created. The Wilson score interval is one well-known construction based on the normal distribution. Other constructions include the Wald, exact, Agresti-Coull, and likelihood intervals. While the Wilson score interval may not be the most conservative estimate, it produces average coverage probabilities that are equal to nominal levels while still producing a comparatively narrow confidence interval.

The "probability" in coverage probability is interpreted with respect to a set of hypothetical repetitions of the entire data collection and analysis procedure. In these hypothetical repetitions, independent data sets following the same probability distribution as the actual data are considered, and a confidence interval is computed from each of these data sets; see Neyman construction. The coverage probability is the fraction of these computed confidence intervals that include the desired but unobservable parameter value.

Probability Matching

In estimation, when the coverage probability is equal to the nominal coverage probability, that is known as probability matching. [6]

In prediction, when the coverage probability is equal to the nominal coverage probability, that is known as predictive probability matching. [2]

Formula

The construction of the confidence interval ensures that the probability of finding the true parameter in the sample-dependent interval is (at least) :

See also

Notes

  1. However, some textbooks use the terms nominal confidence level or nominal confidence coefficient, and actual confidence level or actual confidence coefficient in the sense of "nominal" and "actual coverage probability"; cf., for instance, Wackerly, Dennis; Mendenhall, William; Schaeffer, Richard L. (2008), Mathematical Statistics with Applications (7th ed.), Cengage Learning, p. 437, ISBN   978-1-111-79878-9 .

Related Research Articles

<span class="mw-page-title-main">Binomial distribution</span> Probability distribution

In probability theory and statistics, the binomial distribution with parameters n and p is the discrete probability distribution of the number of successes in a sequence of n independent experiments, each asking a yes–no question, and each with its own Boolean-valued outcome: success or failure. A single success/failure experiment is also called a Bernoulli trial or Bernoulli experiment, and a sequence of outcomes is called a Bernoulli process; for a single trial, i.e., n = 1, the binomial distribution is a Bernoulli distribution. The binomial distribution is the basis for the popular binomial test of statistical significance.

<span class="mw-page-title-main">Estimator</span> Rule for calculating an estimate of a given quantity based on observed data

In statistics, an estimator is a rule for calculating an estimate of a given quantity based on observed data: thus the rule, the quantity of interest and its result are distinguished. For example, the sample mean is a commonly used estimator of the population mean.

The likelihood function is the joint probability of observed data viewed as a function of the parameters of a statistical model.

In statistics, point estimation involves the use of sample data to calculate a single value which is to serve as a "best guess" or "best estimate" of an unknown population parameter. More formally, it is the application of a point estimator to the data to obtain a point estimate.

In statistics, interval estimation is the use of sample data to estimate an interval of possible values of a parameter of interest. This is in contrast to point estimation, which gives a single value.

<span class="mw-page-title-main">Confidence interval</span> Range to estimate an unknown parameter

Informally, in frequentist statistics, a confidence interval (CI) is an interval which is expected to typically contain the parameter being estimated. More specifically, given a confidence level , a CI is a random interval which contains the parameter being estimated % of the time. The confidence level, degree of confidence or confidence coefficient represents the long-run proportion of CIs that theoretically contain the true value of the parameter; this is tantamount to the nominal coverage probability. For example, out of all intervals computed at the 95% level, 95% of them should contain the parameter's true value.

A tolerance interval (TI) is a statistical interval within which, with some confidence level, a specified sampled proportion of a population falls. "More specifically, a 100×p%/100×(1−α) tolerance interval provides limits within which at least a certain proportion (p) of the population falls with a given level of confidence (1−α)." "A (p, 1−α) tolerance interval (TI) based on a sample is constructed so that it would include at least a proportion p of the sampled population with confidence 1−α; such a TI is usually referred to as p-content − (1−α) coverage TI." "A (p, 1−α) upper tolerance limit (TL) is simply a 1−α upper confidence limit for the 100 p percentile of the population."

This glossary of statistics and probability is a list of definitions of terms and concepts used in the mathematical sciences of statistics and probability, their sub-disciplines, and related fields. For additional related terms, see Glossary of mathematics and Glossary of experimental design.

In statistics, a binomial proportion confidence interval is a confidence interval for the probability of success calculated from the outcome of a series of success–failure experiments. In other words, a binomial proportion confidence interval is an interval estimate of a success probability p when only the number of experiments n and the number of successes nS are known.

Bootstrapping is any test or metric that uses random sampling with replacement, and falls under the broader class of resampling methods. Bootstrapping assigns measures of accuracy to sample estimates. This technique allows estimation of the sampling distribution of almost any statistic using random sampling methods.

Neyman construction, named after Jerzy Neyman, is a frequentist method to construct an interval at a confidence level such that if we repeat the experiment many times the interval will contain the true value of some parameter a fraction of the time.

Frequentist inference is a type of statistical inference based in frequentist probability, which treats “probability” in equivalent terms to “frequency” and draws conclusions from sample-data by means of emphasizing the frequency or proportion of findings in the data. Frequentist inference underlies frequentist statistics, in which the well-established methodologies of statistical hypothesis testing and confidence intervals are founded.

In statistics, additive smoothing, also called Laplace smoothing or Lidstone smoothing, is a technique used to smooth count data, eliminating issues caused by certain values having 0 occurrences. Given a set of observation counts from a -dimensional multinomial distribution with trials, a "smoothed" version of the counts gives the estimator:

In statistical inference, the concept of a confidence distribution (CD) has often been loosely referred to as a distribution function on the parameter space that can represent confidence intervals of all levels for a parameter of interest. Historically, it has typically been constructed by inverting the upper limits of lower sided confidence intervals of all levels, and it was also commonly associated with a fiducial interpretation, although it is a purely frequentist concept. A confidence distribution is NOT a probability distribution function of the parameter of interest, but may still be a function useful for making inferences.

In particle physics, CLs represents a statistical method for setting upper limits on model parameters, a particular form of interval estimation used for parameters that can take only non-negative values. Although CLs are said to refer to Confidence Levels, "The method's name is ... misleading, as the CLs exclusion region is not a confidence interval." It was first introduced by physicists working at the LEP experiment at CERN and has since been used by many high energy physics experiments. It is a frequentist method in the sense that the properties of the limit are defined by means of error probabilities, however it differs from standard confidence intervals in that the stated confidence level of the interval is not equal to its coverage probability. The reason for this deviation is that standard upper limits based on a most powerful test necessarily produce empty intervals with some fixed probability when the parameter value is zero, and this property is considered undesirable by most physicists and statisticians.

Probability distribution fitting or simply distribution fitting is the fitting of a probability distribution to a series of data concerning the repeated measurement of a variable phenomenon. The aim of distribution fitting is to predict the probability or to forecast the frequency of occurrence of the magnitude of the phenomenon in a certain interval.

In statistics, a false coverage rate (FCR) is the average rate of false coverage, i.e. not covering the true parameters, among the selected intervals.

In statistics, a population proportion, generally denoted by or the Greek letter , is a parameter that describes a percentage value associated with a population. For example, the 2010 United States Census showed that 83.7% of the American population was identified as not being Hispanic or Latino; the value of .837 is a population proportion. In general, the population proportion and other population parameters are unknown. A census can be conducted in order to determine the actual value of a population parameter, but often a census is not practical due to its costs and time consumption.

In statistics, when selecting a statistical model for given data, the relative likelihood compares the relative plausibilities of different candidate models or of different values of a parameter of a single model.

Brent Andrew Coull is an American statistician and Professor of Biostatistics at Harvard University.

References

  1. Dodge, Y. (2003). The Oxford Dictionary of Statistical Terms. OUP, ISBN   0-19-920613-9, p. 93.
  2. 1 2 Severini, T; Mukerjee, R; Ghosh, M (2002). "On an exact probability matching property of right-invariant priors". Biometrika. 89 (4): 952–957. JSTOR   4140551.
  3. Agresti, Alan; Coull, Brent (1998). "Approximate Is Better than "Exact" for Interval Estimation of Binomial Proportions". The American Statistician. 52 (2): 119–126. doi:10.2307/2685469. JSTOR   2685469.
  4. Brown, Lawrence; Cai, T. Tony; DasGupta, Anirban (2001). "Interval Estimation for a binomial proportion" (PDF). Statistical Science. 16 (2): 101–117. doi: 10.1214/ss/1009213286 . Archived (PDF) from the original on 23 June 2010. Retrieved 17 July 2009.
  5. Newcombe, Robert (1998). "Two-sided confidence intervals for the single proportion: Comparison of seven methods". Statistics in Medicine. 17 (2, issue 8): 857–872. doi:10.1002/(SICI)1097-0258(19980430)17:8<857::AID-SIM777>3.0.CO;2-E. PMID   9595616. Archived from the original on 5 January 2013.
  6. Ghosh, M; Mukerjee, R (1998). Recent developments on probability matching priors. New York Science Publishers. pp. 227–252.