Likelihood-ratio test

Last updated

In statistics, the likelihood-ratio test assesses the goodness of fit of two competing statistical models, specifically one found by maximization over the entire parameter space and another found after imposing some constraint, based on the ratio of their likelihoods. If the constraint (i.e., the null hypothesis) is supported by the observed data, the two likelihoods should not differ by more than sampling error. [1] Thus the likelihood-ratio test tests whether this ratio is significantly different from one, or equivalently whether its natural logarithm is significantly different from zero.

Contents

The likelihood-ratio test, also known as Wilks test, [2] is the oldest of the three classical approaches to hypothesis testing, together with the Lagrange multiplier test and the Wald test. [3] In fact, the latter two can be conceptualized as approximations to the likelihood-ratio test, and are asymptotically equivalent. [4] [5] [6] In the case of comparing two models each of which has no unknown parameters, use of the likelihood-ratio test can be justified by the Neyman–Pearson lemma. The lemma demonstrates that the test has the highest power among all competitors. [7]

Definition

General

Suppose that we have a statistical model with parameter space . A null hypothesis is often stated by saying that the parameter lies in a specified subset of . The alternative hypothesis is thus that lies in the complement of , i.e. in , which is denoted by . The likelihood ratio test statistic for the null hypothesis is given by: [8]

where the quantity inside the brackets is called the likelihood ratio. Here, the notation refers to the supremum. As all likelihoods are positive, and as the constrained maximum cannot exceed the unconstrained maximum, the likelihood ratio is bounded between zero and one.

Often the likelihood-ratio test statistic is expressed as a difference between the log-likelihoods

where

is the logarithm of the maximized likelihood function , and is the maximal value in the special case that the null hypothesis is true (but not necessarily a value that maximizes for the sampled data) and

denote the respective arguments of the maxima and the allowed ranges they're embedded in. Multiplying by 2 ensures mathematically that (by Wilks' theorem) converges asymptotically to being χ²-distributed if the null hypothesis happens to be true. [9] The finite-sample distributions of likelihood-ratio statistics are generally unknown. [10]

The likelihood-ratio test requires that the models be nested i.e. the more complex model can be transformed into the simpler model by imposing constraints on the former's parameters. Many common test statistics are tests for nested models and can be phrased as log-likelihood ratios or approximations thereof: e.g. the Z-test, the F-test, the G-test, and Pearson's chi-squared test; for an illustration with the one-sample t-test, see below.

If the models are not nested, then instead of the likelihood-ratio test, there is a generalization of the test that can usually be used: for details, see relative likelihood .

Case of simple hypotheses

A simple-vs.-simple hypothesis test has completely specified models under both the null hypothesis and the alternative hypothesis, which for convenience are written in terms of fixed values of a notional parameter :

In this case, under either hypothesis, the distribution of the data is fully specified: there are no unknown parameters to estimate. For this case, a variant of the likelihood-ratio test is available: [11] [12]

Some older references may use the reciprocal of the function above as the definition. [13] Thus, the likelihood ratio is small if the alternative model is better than the null model.

The likelihood-ratio test provides the decision rule as follows:

If , do not reject ;
If , reject ;
If , reject with probability .

The values and are usually chosen to obtain a specified significance level , via the relation

The Neyman–Pearson lemma states that this likelihood-ratio test is the most powerful among all level tests for this case. [7] [12]

Interpretation

The likelihood ratio is a function of the data ; therefore, it is a statistic, although unusual in that the statistic's value depends on a parameter, . The likelihood-ratio test rejects the null hypothesis if the value of this statistic is too small. How small is too small depends on the significance level of the test, i.e. on what probability of Type I error is considered tolerable (Type I errors consist of the rejection of a null hypothesis that is true).

The numerator corresponds to the likelihood of an observed outcome under the null hypothesis. The denominator corresponds to the maximum likelihood of an observed outcome, varying parameters over the whole parameter space. The numerator of this ratio is less than the denominator; so, the likelihood ratio is between 0 and 1. Low values of the likelihood ratio mean that the observed result was much less likely to occur under the null hypothesis as compared to the alternative. High values of the statistic mean that the observed outcome was nearly as likely to occur under the null hypothesis as the alternative, and so the null hypothesis cannot be rejected.

An example

The following example is adapted and abridged from Stuart, Ord & Arnold (1999 , §22.2).

Suppose that we have a random sample, of size n, from a population that is normally-distributed. Both the mean, μ, and the standard deviation, σ, of the population are unknown. We want to test whether the mean is equal to a given value, μ0.

Thus, our null hypothesis is H0: μ = μ0  and our alternative hypothesis is H1: μμ0 . The likelihood function is

With some calculation (omitted here), it can then be shown that

where t is the t-statistic with n1 degrees of freedom. Hence we may use the known exact distribution of tn1 to draw inferences.

Asymptotic distribution: Wilks’ theorem

If the distribution of the likelihood ratio corresponding to a particular null and alternative hypothesis can be explicitly determined then it can directly be used to form decision regions (to sustain or reject the null hypothesis). In most cases, however, the exact distribution of the likelihood ratio corresponding to specific hypotheses is very difficult to determine.[ citation needed ]

Assuming H0 is true, there is a fundamental result by Samuel S. Wilks: As the sample size approaches , and if the null hypothesis lies strictly within the interior of the parameter space, the test statistic defined above will be asymptotically chi-squared distributed () with degrees of freedom equal to the difference in dimensionality of and . [14] This implies that for a great variety of hypotheses, we can calculate the likelihood ratio for the data and then compare the observed to the value corresponding to a desired statistical significance as an approximate statistical test. Other extensions exist.[ which? ]

See also

Related Research Articles

In statistics, the likelihood principle is the proposition that, given a statistical model, all the evidence in a sample relevant to model parameters is contained in the likelihood function.

The likelihood function is the joint probability mass of observed data viewed as a function of the parameters of a statistical model. Intuitively, the likelihood function is the probability of observing data assuming is the actual parameter.

In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statistical model, the observed data is most probable. The point in the parameter space that maximizes the likelihood function is called the maximum likelihood estimate. The logic of maximum likelihood is both intuitive and flexible, and as such the method has become a dominant means of statistical inference.

In statistics, the score is the gradient of the log-likelihood function with respect to the parameter vector. Evaluated at a particular point of the parameter vector, the score indicates the steepness of the log-likelihood function and thereby the sensitivity to infinitesimal changes to the parameter values. If the log-likelihood function is continuous over the parameter space, the score will vanish at a local maximum or minimum; this fact is used in maximum likelihood estimation to find the parameter values that maximize the likelihood function.

In statistics, the Neyman–Pearson lemma describes the existence and uniqueness of the likelihood ratio as a uniformly most powerful test in certain contexts. It was introduced by Jerzy Neyman and Egon Pearson in a paper in 1933. The Neyman–Pearson lemma is part of the Neyman–Pearson theory of statistical testing, which introduced concepts like errors of the second kind, power function, and inductive behavior. The previous Fisherian theory of significance testing postulated only one hypothesis. By introducing a competing hypothesis, the Neyman–Pearsonian flavor of statistical testing allows investigating the two types of errors. The trivial cases where one always rejects or accepts the null hypothesis are of little interest but it does prove that one must not relinquish control over one type of error while calibrating the other. Neyman and Pearson accordingly proceeded to restrict their attention to the class of all level tests while subsequently minimizing type II error, traditionally denoted by . Their seminal paper of 1933, including the Neyman–Pearson lemma, comes at the end of this endeavor, not only showing the existence of tests with the most power that retain a prespecified level of type I error, but also providing a way to construct such tests. The Karlin-Rubin theorem extends the Neyman–Pearson lemma to settings involving composite hypotheses with monotone likelihood ratios.

In statistics, G-tests are likelihood-ratio or maximum likelihood statistical significance tests that are increasingly being used in situations where chi-squared tests were previously recommended.

A marginal likelihood is a likelihood function that has been integrated over the parameter space. In Bayesian statistics, it represents the probability of generating the observed sample for all possible values of the parameters; it can be understood as the probability of the model itself and is therefore often referred to as model evidence or simply evidence.

In statistics, the score test assesses constraints on statistical parameters based on the gradient of the likelihood function—known as the score—evaluated at the hypothesized parameter value under the null hypothesis. Intuitively, if the restricted estimator is near the maximum of the likelihood function, the score should not differ from zero by more than sampling error. While the finite sample distributions of score tests are generally unknown, they have an asymptotic χ2-distribution under the null hypothesis as first proved by C. R. Rao in 1948, a fact that can be used to determine statistical significance.

In statistics, the Wald test assesses constraints on statistical parameters based on the weighted distance between the unrestricted estimate and its hypothesized value under the null hypothesis, where the weight is the precision of the estimate. Intuitively, the larger this weighted distance, the less likely it is that the constraint is true. While the finite sample distributions of Wald tests are generally unknown, it has an asymptotic χ2-distribution under the null hypothesis, a fact that can be used to determine statistical significance.

In statistics, the Vuong closeness test is a likelihood-ratio-based test for model selection using the Kullback–Leibler information criterion. This statistic makes probabilistic statements about two models. They can be nested, strictly non-nested or partially non-nested. The statistic tests the null hypothesis that the two models are equally close to the true data generating process, against the alternative that one model is closer. It cannot make any decision whether the "closer" model is the true model.

Proportional hazards models are a class of survival models in statistics. Survival models relate the time that passes, before some event occurs, to one or more covariates that may be associated with that quantity of time. In a proportional hazards model, the unique effect of a unit increase in a covariate is multiplicative with respect to the hazard rate. For example, taking a drug may halve one's hazard rate for a stroke occurring, or, changing the material from which a manufactured component is constructed may double its hazard rate for failure. Other types of survival models such as accelerated failure time models do not exhibit proportional hazards. The accelerated failure time model describes a situation where the biological or mechanical life history of an event is accelerated.

Omnibus tests are a kind of statistical test. They test whether the explained variance in a set of data is significantly greater than the unexplained variance, overall. One example is the F-test in the analysis of variance. There can be legitimate significant effects within a model even if the omnibus test is not significant. For instance, in a model with two independent variables, if only one variable exerts a significant effect on the dependent variable and the other does not, then the omnibus test may be non-significant. This fact does not affect the conclusions that may be drawn from the one significant variable. In order to test effects within an omnibus test, researchers often use contrasts.

The sequential probability ratio test (SPRT) is a specific sequential hypothesis test, developed by Abraham Wald and later proven to be optimal by Wald and Jacob Wolfowitz. Neyman and Pearson's 1933 result inspired Wald to reformulate it as a sequential analysis problem. The Neyman-Pearson lemma, by contrast, offers a rule of thumb for when all the data is collected.

In statistical hypothesis testing, a uniformly most powerful (UMP) test is a hypothesis test which has the greatest power among all possible tests of a given size α. For example, according to the Neyman–Pearson lemma, the likelihood-ratio test is UMP for testing simple (point) hypotheses.

Frequentist inference is a type of statistical inference based in frequentist probability, which treats “probability” in equivalent terms to “frequency” and draws conclusions from sample-data by means of emphasizing the frequency or proportion of findings in the data. Frequentist inference underlies frequentist statistics, in which the well-established methodologies of statistical hypothesis testing and confidence intervals are founded.

In statistics Wilks' theorem states that the log-likelihood ratio is asymptotically normal. This can be used to produce confidence intervals for maximum-likelihood estimates or as a test statistic for performing the likelihood-ratio test.

In particle physics, CLs represents a statistical method for setting upper limits on model parameters, a particular form of interval estimation used for parameters that can take only non-negative values. Although CLs are said to refer to Confidence Levels, "The method's name is ... misleading, as the CLs exclusion region is not a confidence interval." It was first introduced by physicists working at the LEP experiment at CERN and has since been used by many high energy physics experiments. It is a frequentist method in the sense that the properties of the limit are defined by means of error probabilities, however it differs from standard confidence intervals in that the stated confidence level of the interval is not equal to its coverage probability. The reason for this deviation is that standard upper limits based on a most powerful test necessarily produce empty intervals with some fixed probability when the parameter value is zero, and this property is considered undesirable by most physicists and statisticians.

In statistical decision theory, a randomised decision rule or mixed decision rule is a decision rule that associates probabilities with deterministic decision rules. In finite decision problems, randomised decision rules define a risk set which is the convex hull of the risk points of the nonrandomised decision rules.

In statistics, when selecting a statistical model for given data, the relative likelihood compares the relative plausibilities of different candidate models or of different values of a parameter of a single model.

In statistical hypothesis testing, e-values quantify the evidence in the data against a null hypothesis. They serve as a more robust alternative to p-values, addressing some shortcomings of the latter.

References

  1. King, Gary (1989). Unifying Political Methodology : The Likelihood Theory of Statistical Inference. New York: Cambridge University Press. p. 84. ISBN   0-521-36697-6.
  2. Li, Bing; Babu, G. Jogesh (2019). A Graduate Course on Statistical Inference. Springer. p. 331. ISBN   978-1-4939-9759-6.
  3. Maddala, G. S.; Lahiri, Kajal (2010). Introduction to Econometrics (Fourth ed.). New York: Wiley. p. 200.
  4. Buse, A. (1982). "The Likelihood Ratio, Wald, and Lagrange Multiplier Tests: An Expository Note". The American Statistician . 36 (3a): 153–157. doi:10.1080/00031305.1982.10482817.
  5. Pickles, Andrew (1985). An Introduction to Likelihood Analysis. Norwich: W. H. Hutchins & Sons. pp.  24–27. ISBN   0-86094-190-6.
  6. Severini, Thomas A. (2000). Likelihood Methods in Statistics. New York: Oxford University Press. pp. 120–121. ISBN   0-19-850650-3.
  7. 1 2 Neyman, J.; Pearson, E. S. (1933), "On the problem of the most efficient tests of statistical hypotheses" (PDF), Philosophical Transactions of the Royal Society of London A , 231 (694–706): 289–337, Bibcode:1933RSPTA.231..289N, doi: 10.1098/rsta.1933.0009 , JSTOR   91247
  8. Koch, Karl-Rudolf (1988). Parameter Estimation and Hypothesis Testing in Linear Models . New York: Springer. p.  306. ISBN   0-387-18840-1.
  9. Silvey, S.D. (1970). Statistical Inference. London: Chapman & Hall. pp. 112–114. ISBN   0-412-13820-4.
  10. Mittelhammer, Ron C.; Judge, George G.; Miller, Douglas J. (2000). Econometric Foundations. New York: Cambridge University Press. p. 66. ISBN   0-521-62394-4.
  11. Mood, A.M.; Graybill, F.A.; Boes, D.C. (1974). Introduction to the Theory of Statistics (3rd ed.). McGraw-Hill. §9.2.
  12. 1 2 Stuart, A.; Ord, K.; Arnold, S. (1999), Kendall's Advanced Theory of Statistics, vol. 2A, Arnold, §§20.10–20.13
  13. Cox, D. R.; Hinkley, D. V. (1974), Theoretical Statistics, Chapman & Hall, p. 92, ISBN   0-412-12420-3
  14. Wilks, S.S. (1938). "The large-sample distribution of the likelihood ratio for testing composite hypotheses". Annals of Mathematical Statistics . 9 (1): 60–62. doi: 10.1214/aoms/1177732360 .

Further reading