Frequentist inference

Last updated

Frequentist inference is a type of statistical inference based in frequentist probability, which treats “probability” in equivalent terms to “frequency” and draws conclusions from sample-data by means of emphasizing the frequency or proportion of findings in the data. Frequentist-inference underlies frequentist statistics, in which the well-established methodologies of statistical hypothesis testing and confidence intervals are founded.


History of frequentist statistics

The history of frequentist statistics is more recent than its prevailing philosophical rival, Bayesian statistics. Frequentist statistics were largely developed in the early 20th century and have recently developed to become the dominant paradigm in inferential statistics, while Bayesian statistics were invented in the 19th century. Despite this dominance, there is no agreement as to whether frequentism is better than Bayesian statistics, with a vocal minority of professionals studying statistical inference decrying frequentist inference for being internally-inconsistent. For the purposes of this article, frequentist methodology will be discussed as summarily as possible but it is worth noting that this subject remains controversial even into the modern day.

The primary formulation of frequentism stems from the presumption that statistics could be perceived to have been a probabilistic frequency. This view was primarily developed by Ronald Fisher and the team of Jerzy Neyman and Egon Pearson. Ronald Fisher's contributed to frequentist statistics by developing the frequentist concept of "significance testing", which is the study of the significance of a measure of a statistic when compared to the hypothesis. Neyman-Pearson extended Fisher's ideas to multiple hypotheses by conjecturing that the ratio of probabilities of hypotheses when maximizing the difference between the two hypotheses leads to a maximization of exceeding a given p-value, and also provides the basis of type I and type II errors. For more, see the foundations of statistics page.


For statistical inference, the relevant statistic about which we want to make inferences is , where the random vector is a function of an unknown parameter, . The parameter is further partitioned into (), where is the parameter of interest, and is the nuisance parameter. For concreteness, in one area might be the population mean, , and the nuisance parameter would then be the standard deviation of the population mean, . [1]

Thus, statistical inference is concerned with the expectation of random vector , namely .

To construct areas of uncertainty in frequentist inference, a pivot is used which defines the area around that can be used to provide an interval to estimate uncertainty. The pivot is a probability such that for a pivot, , which is a function, that is strictly increasing in , where is a random vector. This allows that, for some 0 < < 1, we can define , which is the probability that the pivot function is less than some well-defined value. This implies , where is a upper limit for . Note that is a range of outcomes that define a one-sided limit for , and that is a two-sided limit for , when we want to estimate a range of outcomes where may occur. This rigorously defines the confidence interval, which is the range of outcomes about which we can make statistical inferences.

Fisherian reduction and Neyman-Pearson operational criteria

Two complementary concepts in frequentist inference are the Fisherian reduction and the Neyman-Pearson operational criteria. Together these concepts illustrate a way of constructing frequentist intervals that define the limits for . The Fisherian reduction is a method of determining the interval within which the true value of may lie, while the Neyman-Pearson operational criteria is a decision rule about making a priori probability assumptions.

The Fisherian reduction is defined as follows:

Essentially, the Fisherian reduction is design to find where the sufficient statistic can be used to determine the range of outcomes where may occur on a probability distribution that defines all the potential values of . This is necessary to formulating confidence intervals, where we can find a range of outcomes over which is likely to occur in the long-run.

The Neyman-Pearon operational criteria is an even more specific understanding of the range of outcomes where the relevant statistic, , can be said to occur in the long run. The Neyman-Pearson operational criteria defines the likelihood of that range actually being adequate or of the range being inadequate. The Neyman-Pearson criteria defines the range of the probability distribution that, if exists in this range, is still below the true population statistic. For example, if the distribution from the Fisherian reduction exceeds a threshold that we consider to be a priori implausible, then the Neyman-Pearson reduction's evaluation of that distribution can be used to infer where looking purely at the Fisherian reduction's distributions can give us inaccurate results. Thus, the Neyman-Pearson reduction is used to find the probability of type I and type II errors. [3] As a point of reference, the complement to this in Bayesian statistics is the minimum Bayes risk criterion.

Because of the reliance of the Neyman-Pearson criteria on our ability to find a range of outcomes where is likely to occur, the Neyman-Pearson approach is only possible where a Fisherian reduction can be achieved. [4]

Experimental design and methodology

Frequentist inferences are associated with the application frequentist probability to experimental design and interpretation, and specifically with the view that any given experiment can be considered one of an infinite sequence of possible repetitions of the same experiment, each capable of producing statistically independent results. [5] In this view, the frequentist inference approach to drawing conclusions from data is effectively to require that the correct conclusion should be drawn with a given (high) probability, among this notional set of repetitions.

However, exactly the same procedures can be developed under a subtly different formulation. This is one where a pre-experiment point of view is taken. It can be argued that the design of an experiment should include, before undertaking the experiment, decisions about exactly what steps will be taken to reach a conclusion from the data yet to be obtained. These steps can be specified by the scientist so that there is a high probability of reaching a correct decision where, in this case, the probability relates to a yet to occur set of random events and hence does not rely on the frequency interpretation of probability. This formulation has been discussed by Neyman, [6] among others. This is especially pertinent because the significance of a frequentist test can vary under model selection, a violation of the likelihood principle.

The statistical philosophy of frequentism

Frequentism is the study of probability with the assumption that results occur with a given frequency over some period of time or with repeated sampling. As such, frequentist analysis must be formulated with consideration to the assumptions of the problem frequentism attempts to analyze. This requires looking into whether the question at hand is concerned with understanding variety of a statistic or locating the true value of a statistic. The difference between these assumptions is critical for interpreting a hypothesis test. The next paragraph elaborates on this.

There are broadly two camps of statistical inference, the epistemic approach and the epidemiological approach. The epistemic approach is the study of variability; namely, how often do we expect a statistic to deviate from some observed value. The epidemiological approach is concerned with the study of uncertainty; in this approach, the value of the statistic is fixed but our understanding of that statistic is incomplete. [7] For concreteness, imagine trying to measure the stock market quote versus evaluating an asset's price. The stock market fluctuates so greatly that trying to find exactly where a stock price is going to be is not useful: the stock market is better understood using the epistemic approach, where we can try to quantify its fickle movements. Conversely, the price of an asset might not change that much from day to day: it is better to locate the true value of the asset rather than find a range of prices and thus the epidemiological approach is better. The difference between these approaches is non-trivial for the purposes of inference.

For the epistemic approach, we formulate the problem as if we want to attribute probability to a hypothesis. Unfortunately, this kind of approach is (for highly rigorous reasons) best answered with Bayesian statistics, where the interpretation of probability is straightforward because Bayesian statistics is conditional on the entire sample space, whereas frequentist data is inherently conditional on unobserved and unquantifiable data. The reason for this is inherent to frequentist design. Frequentist statistics is unfortunately conditioned not on solely the data but also on the experimental design. [8] In frequentist statistics, the cutoff for understanding the frequency occurrence is derived from the family distribution used in the experiment design. For example, a binomial distribution and a negative binomial distribution can be used to analyze exactly the same data, but because their tail ends are different the frequentist analysis will realize different levels of statistical significance for the same data that assumes different probability distributions. This difference does not occur in Bayesian inference. For more, see the likelihood principle, which frequentist statistics inherently violates. [9]

For the epidemiological approach, the central idea behind frequentist statistics must be discussed. Frequentist statistics is designed so that, in the long-run, the frequency of a statistic may be understood, and in the long-run the range of the true mean of a statistic can be inferred. This leads to the Fisherian reduction and the Neyman-Pearson operational criteria, discussed above. When we define the Fisherian reduction and the Neyman-Pearson operational criteria for any statistic, we are assessing, according to these authors, the likelihood that the true value of the statistic will occur within a given range of outcomes assuming a number of repetitions of our sampling method. [8] This allows for inference where, in the long-run, we can define that the combined results of multiple frequentist inferences to mean that a 95% confidence interval literally means the true mean lies in the confidence interval 95% of the time, but not that the mean is in a particular confidence interval with 95% certainty. This is a popular misconception.

Very commonly the epistemic view and the epidemiological view are regarded as interconvertible. This is demonstrably false. First, the epistemic view is centered around Fisherian significance tests that are designed to provide inductive evidence against the null hypothesis, , in a single experiment, and is defined by the Fisherian p-value. Conversely, the epidemiological view, conducted with Neyman-Pearson hypothesis testing, is designed to minimize the Type II false acceptance errors in the long-run by providing error minimizations that work in the long-run. The difference between the two is critical because the epistemic view stresses the conditions under which we might find one value to be statistically significant; meanwhile, the epidemiological view defines the conditions under which long-run results present valid results. These are extremely different inferences, because one-time, epistemic conclusions do not inform long-run errors, and long-run errors cannot be used to certify whether one-time experiments are sensical. The assumption of one-time experiments to long-run occurrences is a misattribution, and the assumption of long run trends to individuals experiments is an example of the ecological fallacy. [10]

Relationship with other approaches

Frequentist inferences stand in contrast to other types of statistical inferences, such as Bayesian inferences and fiducial inferences. While the "Bayesian inference" is sometimes held to include the approach to inferences leading to optimal decisions, a more restricted view is taken here for simplicity.

Bayesian inference

Bayesian inference is based in Bayesian probability, which treats “probability” as equivalent with “certainty”, and thus that the essential difference between the frequentist inference and the Bayesian inference is the same as the difference between the two interpretations of what a “probability” means. However, where appropriate, Bayesian inferences (meaning in this case an application of Bayes' theorem) are used by those employing frequency probability.

There are two major differences in the frequentist and Bayesian approaches to inference that are not included in the above consideration of the interpretation of probability:

  1. In a frequentist approach to inference, unknown parameters are often, but not always, treated as having fixed but unknown values that are not capable of being treated as random variates in any sense, and hence there is no way that probabilities can be associated with them. In contrast, a Bayesian approach to inference does allow probabilities to be associated with unknown parameters, where these probabilities can sometimes have a frequency probability interpretation as well as a Bayesian one. The Bayesian approach allows these probabilities to have an interpretation as representing the scientist's belief that given values of the parameter are true (see Bayesian probability - Personal probabilities and objective methods for constructing priors).
  2. While "probabilities" are involved in both approaches to inference, the probabilities are associated with different types of things. The result of a Bayesian approach can be a probability distribution for what is known about the parameters given the results of the experiment or study. The result of a frequentist approach is either a "true or false" conclusion from a significance test or a conclusion in the form that a given sample-derived confidence interval covers the true value: either of these conclusions has a given probability of being correct, where this probability has either a frequency probability interpretation or a pre-experiment interpretation.

See also

Related Research Articles

Frequentist probability

Frequentist probability or frequentism is an interpretation of probability; it defines an event's probability as the limit of its relative frequency in many trials. Probabilities can be found by a repeatable objective process. The continued use of frequentist methods in scientific inference, however, has been called into question.

The word probability has been used in a variety of ways since it was first applied to the mathematical study of games of chance. Does probability measure the real, physical, tendency of something to occur, or is it a measure of how strongly one believes it will occur, or does it draw on both these elements? In answering such questions, mathematicians interpret the probability values of probability theory.

Statistical inference Process of using data analysis

Statistical inference is the process of using data analysis to infer properties of an underlying distribution of probability. Inferential statistical analysis infers properties of a population, for example by testing hypotheses and deriving estimates. It is assumed that the observed data set is sampled from a larger population.

A statistical hypothesis test is a method of statistical inference used to decide whether the data at hand sufficiently support a particular hypothesis.

In statistics, interval estimation is the use of sample data to estimate an interval of plausible values of a parameter of interest. This is in contrast to point estimation, which gives a single value.

Confidence interval Range of estimates for an unknown parameter

In frequentist statistics, a confidence interval (CI) is a range of estimates for an unknown parameter. A confidence interval is computed at a designated confidence level (CL); the 95% CL is most common, but other levels, such as 90% or 99%, are sometimes used. The CL represents the long-run proportion of correspondingly CI that end up containing the true value of the parameter. For example, out of all intervals computed at the 95% level, 95% of them should contain the parameter's true value.

Bayesian statistics is a theory in the field of statistics based on the Bayesian interpretation of probability where probability expresses a degree of belief in an event. The degree of belief may be based on prior knowledge about the event, such as the results of previous experiments, or on personal beliefs about the event. This differs from a number of other interpretations of probability, such as the frequentist interpretation that views probability as the limit of the relative frequency of an event after many trials.

In statistical inference, specifically predictive inference, a prediction interval is an estimate of an interval in which a future observation will fall, with a certain probability, given what has already been observed. Prediction intervals are often used in regression analysis.

Debabrata Basu Indian statistician

Debabrata Basu was an Indian statistician who made fundamental contributions to the foundations of statistics. Basu invented simple examples that displayed some difficulties of likelihood-based statistics and frequentist statistics; Basu's paradoxes were especially important in the development of survey sampling. In statistical theory, Basu's theorem established the independence of a complete sufficient statistic and an ancillary statistic.

This glossary of statistics and probability is a list of definitions of terms and concepts used in the mathematical sciences of statistics and probability, their sub-disciplines, and related fields. For additional related terms, see Glossary of mathematics.

In Bayesian statistics, a credible interval is an interval within which an unobserved parameter value falls with a particular probability. It is an interval in the domain of a posterior probability distribution or a predictive distribution. The generalisation to multivariate problems is the credible region. Credible intervals are analogous to confidence intervals in frequentist statistics, although they differ on a philosophical basis: Bayesian intervals treat their bounds as fixed and the estimated parameter as a random variable, whereas frequentist confidence intervals treat their bounds as random variables and the parameter as a fixed value. Also, Bayesian credible intervals use knowledge of the situation-specific prior distribution, while the frequentist confidence intervals do not.

Frequency (statistics) Number of occurrences in an experiment or study

In statistics, the frequency of an event is the number of times the observation occurred/recorded in an experiment or study. These frequencies are often graphically represented in histograms.

Uncertainty quantification (UQ) is the science of quantitative characterization and reduction of uncertainties in both computational and real world applications. It tries to determine how likely certain outcomes are if some aspects of the system are not exactly known. An example would be to predict the acceleration of a human body in a head-on crash with another car: even if the speed was exactly known, small differences in the manufacturing of individual cars, how tightly every bolt has been tightened, etc., will lead to different results that can only be predicted in a statistical sense.

Fiducial inference is one of a number of different types of statistical inference. These are rules, intended for general application, by which conclusions can be drawn from samples of data. In modern statistical practice, attempts to work with fiducial inference have fallen out of fashion in favour of frequentist inference, Bayesian inference and decision theory. However, fiducial inference is important in the history of statistics since its development led to the parallel development of concepts and tools in theoretical statistics that are widely used. Some current research in statistical methodology is either explicitly linked to fiducial inference or is closely connected to it.

Neyman construction, named after Jerzy Neyman, is a frequentist method to construct an interval at a confidence level such that if we repeat the experiment many times the interval will contain the true value of some parameter a fraction of the time.

Henry E. Kyburg Jr. (1928–2007) was Gideon Burbank Professor of Moral Philosophy and Professor of Computer Science at the University of Rochester, New York, and Pace Eminent Scholar at the Institute for Human and Machine Cognition, Pensacola, Florida. His first faculty posts were at Rockefeller Institute, University of Denver, Wesleyan College, and Wayne State University.

Statistics, in the modern sense of the word, began evolving in the 18th century in response to the novel needs of industrializing sovereign states. The evolution of statistics was, in particular, intimately connected with the development of European states following the peace of Westphalia (1648), and with the development of probability theory, which put statistics on a firm theoretical basis.

The foundations of statistics concern the epistemological debate in statistics over how one should conduct inductive inference from data. Among the issues considered in statistical inference are the question of Bayesian inference versus frequentist inference, the distinction between Fisher's "significance testing" and Neyman–Pearson "hypothesis testing", and whether the likelihood principle should be followed. Some of these issues have been debated for up to 200 years without resolution.

In statistical inference, the concept of a confidence distribution (CD) has often been loosely referred to as a distribution function on the parameter space that can represent confidence intervals of all levels for a parameter of interest. Historically, it has typically been constructed by inverting the upper limits of lower sided confidence intervals of all levels, and it was also commonly associated with a fiducial interpretation, although it is a purely frequentist concept. A confidence distribution is NOT a probability distribution function of the parameter of interest, but may still be a function useful for making inferences.

In statistics, suppose that we have been given some data, and we are selecting a statistical model for that data. The relative likelihood compares the relative plausibilities of different candidate models or of different values of a parameter of a single model.


  1. Cox (2006), pp. 1–2.
  2. Cox (2006), pp. 24, 47.
  3. "OpenStax CNX". Retrieved 2021-09-14.
  4. Cox (2006), p. 24.
  5. Everitt (2002).
  6. Jerzy (1937), pp. 236, 333–380.
  7. Romeijn, Jan-Willem (2017), Zalta, Edward N. (ed.), "Philosophy of Statistics", The Stanford Encyclopedia of Philosophy (Spring 2017 ed.), Metaphysics Research Lab, Stanford University, retrieved 2021-09-14
  8. 1 2 Wagenmakers et al. (2008).
  9. Vidakovic, Brani. "The Likelihood Principle" (PDF).{{cite web}}: CS1 maint: url-status (link)
  10. Hubbard, R.; Bayarri, M.J. (2003). "Confusion over measures of evidence (p's) versus errors (α's) in classical statistical testing" (PDF). The American Statistician. 57: 171–182.