Sufficient statistic

Last updated

In statistics, a statistic is sufficient with respect to a statistical model and its associated unknown parameter if "no other statistic that can be calculated from the same sample provides any additional information as to the value of the parameter". [1] In particular, a statistic is sufficient for a family of probability distributions if the sample from which it is calculated gives no additional information than the statistic, as to which of those probability distributions is the sampling distribution.

Contents

A related concept is that of linear sufficiency, which is weaker than sufficiency but can be applied in some cases where there is no sufficient statistic, although it is restricted to linear estimators. [2] The Kolmogorov structure function deals with individual finite data; the related notion there is the algorithmic sufficient statistic.

The concept is due to Sir Ronald Fisher in 1920. Stephen Stigler noted in 1973 that the concept of sufficiency had fallen out of favor in descriptive statistics because of the strong dependence on an assumption of the distributional form (see Pitman–Koopman–Darmois theorem below), but remained very important in theoretical work. [3]

Background

Roughly, given a set of independent identically distributed data conditioned on an unknown parameter , a sufficient statistic is a function whose value contains all the information needed to compute any estimate of the parameter (e.g. a maximum likelihood estimate). Due to the factorization theorem (see below), for a sufficient statistic , the probability density can be written as . From this factorization, it can easily be seen that the maximum likelihood estimate of will interact with only through . Typically, the sufficient statistic is a simple function of the data, e.g. the sum of all the data points.

More generally, the "unknown parameter" may represent a vector of unknown quantities or may represent everything about the model that is unknown or not fully specified. In such a case, the sufficient statistic may be a set of functions, called a jointly sufficient statistic. Typically, there are as many functions as there are parameters. For example, for a Gaussian distribution with unknown mean and variance, the jointly sufficient statistic, from which maximum likelihood estimates of both parameters can be estimated, consists of two functions, the sum of all data points and the sum of all squared data points (or equivalently, the sample mean and sample variance).

The concept is equivalent to the statement that, conditional on the value of a sufficient statistic for a parameter, the joint probability distribution of the data does not depend on that parameter. Both the statistic and the underlying parameter can be vectors.

Mathematical definition

A statistic t = T(X) is sufficient for underlying parameter θ precisely if the conditional probability distribution of the data X, given the statistic t = T(X), does not depend on the parameter θ. [4]

Alternatively, one can say the statistic T(X) is sufficient for θ if its mutual information with θ equals the mutual information between X and θ. [5] In other words, the data processing inequality becomes an equality:

Example

As an example, the sample mean is sufficient for the mean (μ) of a normal distribution with known variance. Once the sample mean is known, no further information about μ can be obtained from the sample itself. On the other hand, for an arbitrary distribution the median is not sufficient for the mean: even if the median of the sample is known, knowing the sample itself would provide further information about the population mean. For example, if the observations that are less than the median are only slightly less, but observations exceeding the median exceed it by a large amount, then this would have a bearing on one's inference about the population mean.

Fisher–Neyman factorization theorem

Fisher's factorization theorem or factorization criterion provides a convenient characterization of a sufficient statistic. If the probability density function is ƒθ(x), then T is sufficient for θ if and only if nonnegative functions g and h can be found such that

i.e. the density ƒ can be factored into a product such that one factor, h, does not depend on θ and the other factor, which does depend on θ, depends on x only through T(x).

It is easy to see that if F(t) is a one-to-one function and T is a sufficient statistic, then F(T) is a sufficient statistic. In particular we can multiply a sufficient statistic by a nonzero constant and get another sufficient statistic.

Likelihood principle interpretation

An implication of the theorem is that when using likelihood-based inference, two sets of data yielding the same value for the sufficient statistic T(X) will always yield the same inferences about θ. By the factorization criterion, the likelihood's dependence on θ is only in conjunction with T(X). As this is the same in both cases, the dependence on θ will be the same as well, leading to identical inferences.

Proof

Due to Hogg and Craig. [6] Let , denote a random sample from a distribution having the pdf f(x, θ) for ι < θ < δ. Let Y1 = u1(X1, X2, ..., Xn) be a statistic whose pdf is g1(y1; θ). What we want to prove is that Y1 = u1(X1, X2, ..., Xn) is a sufficient statistic for θ if and only if, for some function H,

First, suppose that

We shall make the transformation yi = ui(x1, x2, ..., xn), for i = 1, ..., n, having inverse functions xi = wi(y1, y2, ..., yn), for i = 1, ..., n, and Jacobian . Thus,

The left-hand member is the joint pdf g(y1, y2, ..., yn; θ) of Y1 = u1(X1, ..., Xn), ..., Yn = un(X1, ..., Xn). In the right-hand member, is the pdf of , so that is the quotient of and ; that is, it is the conditional pdf of given .

But , and thus , was given not to depend upon . Since was not introduced in the transformation and accordingly not in the Jacobian , it follows that does not depend upon and that is a sufficient statistics for .

The converse is proven by taking:

where does not depend upon because depend only upon , which are independent on when conditioned by , a sufficient statistics by hypothesis. Now divide both members by the absolute value of the non-vanishing Jacobian , and replace by the functions in . This yields

where is the Jacobian with replaced by their value in terms . The left-hand member is necessarily the joint pdf of . Since , and thus , does not depend upon , then

is a function that does not depend upon .

Another proof

A simpler more illustrative proof is as follows, although it applies only in the discrete case.

We use the shorthand notation to denote the joint probability density of by . Since is a function of , we have , as long as and zero otherwise. Therefore:

with the last equality being true by the definition of sufficient statistics. Thus with and .

Conversely, if , we have

With the first equality by the definition of pdf for multiple variables, the second by the remark above, the third by hypothesis, and the fourth because the summation is not over .

Let denote the conditional probability density of given . Then we can derive an explicit expression for this:

With the first equality by definition of conditional probability density, the second by the remark above, the third by the equality proven above, and the fourth by simplification. This expression does not depend on and thus is a sufficient statistic. [7]

Minimal sufficiency

A sufficient statistic is minimal sufficient if it can be represented as a function of any other sufficient statistic. In other words, S(X) is minimal sufficient if and only if [8]

  1. S(X) is sufficient, and
  2. if T(X) is sufficient, then there exists a function f such that S(X) = f(T(X)).

Intuitively, a minimal sufficient statistic most efficiently captures all possible information about the parameter θ.

A useful characterization of minimal sufficiency is that when the density fθ exists, S(X) is minimal sufficient if and only if

is independent of θ :S(x) = S(y)

This follows as a consequence from Fisher's factorization theorem stated above.

A case in which there is no minimal sufficient statistic was shown by Bahadur, 1954. [9] However, under mild conditions, a minimal sufficient statistic does always exist. In particular, in Euclidean space, these conditions always hold if the random variables (associated with ) are all discrete or are all continuous.

If there exists a minimal sufficient statistic, and this is usually the case, then every complete sufficient statistic is necessarily minimal sufficient [10] (note that this statement does not exclude a pathological case in which a complete sufficient exists while there is no minimal sufficient statistic). While it is hard to find cases in which a minimal sufficient statistic does not exist, it is not so hard to find cases in which there is no complete statistic.

The collection of likelihood ratios for , is a minimal sufficient statistic if the parameter space is discrete .

Examples

Bernoulli distribution

If X1, ...., Xn are independent Bernoulli-distributed random variables with expected value p, then the sum T(X) = X1 + ... + Xn is a sufficient statistic for p (here 'success' corresponds to Xi = 1 and 'failure' to Xi = 0; so T is the total number of successes)

This is seen by considering the joint probability distribution:

Because the observations are independent, this can be written as

and, collecting powers of p and 1  p, gives

which satisfies the factorization criterion, with h(x) = 1 being just a constant.

Note the crucial feature: the unknown parameter p interacts with the data x only via the statistic T(x) = Σ xi.

As a concrete application, this gives a procedure for distinguishing a fair coin from a biased coin.

Uniform distribution

If X1, ...., Xn are independent and uniformly distributed on the interval [0,θ], then T(X) = max(X1, ..., Xn) is sufficient for θ — the sample maximum is a sufficient statistic for the population maximum.

To see this, consider the joint probability density function of X  (X1,...,Xn). Because the observations are independent, the pdf can be written as a product of individual densities

where 1{...} is the indicator function. Thus the density takes form required by the Fisher–Neyman factorization theorem, where h(x) = 1{min{xi}≥0}, and the rest of the expression is a function of only θ and T(x) = max{xi}.

In fact, the minimum-variance unbiased estimator (MVUE) for θ is

This is the sample maximum, scaled to correct for the bias, and is MVUE by the Lehmann–Scheffé theorem. Unscaled sample maximum T(X) is the maximum likelihood estimator for θ.

Uniform distribution (with two parameters)

If are independent and uniformly distributed on the interval (where and are unknown parameters), then is a two-dimensional sufficient statistic for .

To see this, consider the joint probability density function of . Because the observations are independent, the pdf can be written as a product of individual densities, i.e.

The joint density of the sample takes the form required by the Fisher–Neyman factorization theorem, by letting

Since does not depend on the parameter and depends only on through the function

the Fisher–Neyman factorization theorem implies is a sufficient statistic for .

Poisson distribution

If X1, ...., Xn are independent and have a Poisson distribution with parameter λ, then the sum T(X) = X1 + ... + Xn is a sufficient statistic for λ.

To see this, consider the joint probability distribution:

Because the observations are independent, this can be written as

which may be written as

which shows that the factorization criterion is satisfied, where h(x) is the reciprocal of the product of the factorials. Note the parameter λ interacts with the data only through its sum T(X).

Normal distribution

If are independent and normally distributed with expected value (a parameter) and known finite variance then

is a sufficient statistic for

To see this, consider the joint probability density function of . Because the observations are independent, the pdf can be written as a product of individual densities, i.e.

The joint density of the sample takes the form required by the Fisher–Neyman factorization theorem, by letting

Since does not depend on the parameter and depends only on through the function

the Fisher–Neyman factorization theorem implies is a sufficient statistic for .

If is unknown and since , the above likelihood can be rewritten as

The Fisher–Neyman factorization theorem still holds and implies that is a joint sufficient statistic for .

Exponential distribution

If are independent and exponentially distributed with expected value θ (an unknown real-valued positive parameter), then is a sufficient statistic for θ.

To see this, consider the joint probability density function of . Because the observations are independent, the pdf can be written as a product of individual densities, i.e.

The joint density of the sample takes the form required by the Fisher–Neyman factorization theorem, by letting

Since does not depend on the parameter and depends only on through the function

the Fisher–Neyman factorization theorem implies is a sufficient statistic for .

Gamma distribution

If are independent and distributed as a , where and are unknown parameters of a Gamma distribution, then is a two-dimensional sufficient statistic for .

To see this, consider the joint probability density function of . Because the observations are independent, the pdf can be written as a product of individual densities, i.e.

The joint density of the sample takes the form required by the Fisher–Neyman factorization theorem, by letting

Since does not depend on the parameter and depends only on through the function

the Fisher–Neyman factorization theorem implies is a sufficient statistic for

Rao–Blackwell theorem

Sufficiency finds a useful application in the Rao–Blackwell theorem, which states that if g(X) is any kind of estimator of θ, then typically the conditional expectation of g(X) given sufficient statistic T(X) is a better (in the sense of having lower variance) estimator of θ, and is never worse. Sometimes one can very easily construct a very crude estimator g(X), and then evaluate that conditional expected value to get an estimator that is in various senses optimal.

Exponential family

According to the Pitman–Koopman–Darmois theorem, among families of probability distributions whose domain does not vary with the parameter being estimated, only in exponential families is there a sufficient statistic whose dimension remains bounded as sample size increases.

Less tersely, suppose are independent identically distributed real random variables whose distribution is known to be in some family of probability distributions, parametrized by , such that for all , the distribution of is . Only if that family is an exponential family there is a vector-valued sufficient statistic whose number of scalar components does not increase as the sample size n increases. [11]

This theorem shows that the existence of a real-vector-valued sufficient statistics sharply restricts the possible forms of the distribution.

When the parameters or the random variables are no longer real-valued, the situation is more complex. [12]

Other types of sufficiency

Bayesian sufficiency

An alternative formulation of the condition that a statistic be sufficient, set in a Bayesian context, involves the posterior distributions obtained by using the full data-set and by using only a statistic. Thus the requirement is that, for almost every x,

More generally, without assuming a parametric model, we can say that the statistics T is predictive sufficient if

It turns out that this "Bayesian sufficiency" is a consequence of the formulation above, [13] however they are not directly equivalent in the infinite-dimensional case. [14] A range of theoretical results for sufficiency in a Bayesian context is available. [15]

Linear sufficiency

A concept called "linear sufficiency" can be formulated in a Bayesian context, [16] and more generally. [17] First define the best linear predictor of a vector Y based on X as . Then a linear statistic T(x) is linear sufficient [18] if

See also

Notes

  1. Fisher, R.A. (1922). "On the mathematical foundations of theoretical statistics". Philosophical Transactions of the Royal Society A. 222 (594–604): 309–368. Bibcode:1922RSPTA.222..309F. doi: 10.1098/rsta.1922.0009 . JFM   48.1280.02. JSTOR   91208.
  2. Dodge, Y. (2003) — entry for linear sufficiency
  3. Stigler, Stephen (December 1973). "Studies in the History of Probability and Statistics. XXXII: Laplace, Fisher and the Discovery of the Concept of Sufficiency". Biometrika. 60 (3): 439–445. doi:10.1093/biomet/60.3.439. JSTOR   2334992. MR   0326872.
  4. Casella, George; Berger, Roger L. (2002). Statistical Inference, 2nd ed. Duxbury Press.
  5. Cover, Thomas M. (2006). Elements of Information Theory. Joy A. Thomas (2nd ed.). Hoboken, N.J.: Wiley-Interscience. p. 36. ISBN   0-471-24195-4. OCLC   59879802.
  6. Hogg, Robert V.; Craig, Allen T. (1995). Introduction to Mathematical Statistics. Prentice Hall. ISBN   978-0-02-355722-4.
  7. "The Fisher–Neyman Factorization Theorem".. Webpage at Connexions (cnx.org)
  8. Dodge (2003) — entry for minimal sufficient statistics
  9. Lehmann and Casella (1998), Theory of Point Estimation, 2nd Edition, Springer, p 37
  10. Lehmann and Casella (1998), Theory of Point Estimation, 2nd Edition, Springer, page 42
  11. Tikochinsky, Y.; Tishby, N. Z.; Levine, R. D. (1984-11-01). "Alternative approach to maximum-entropy inference". Physical Review A. 30 (5): 2638–2644. doi:10.1103/physreva.30.2638. ISSN   0556-2791.
  12. Andersen, Erling Bernhard (September 1970). "Sufficiency and Exponential Families for Discrete Sample Spaces". Journal of the American Statistical Association. 65 (331): 1248–1255. doi:10.1080/01621459.1970.10481160. ISSN   0162-1459.
  13. Bernardo, J.M.; Smith, A.F.M. (1994). "Section 5.1.4". Bayesian Theory. Wiley. ISBN   0-471-92416-4.
  14. Blackwell, D.; Ramamoorthi, R. V. (1982). "A Bayes but not classically sufficient statistic". Annals of Statistics . 10 (3): 1025–1026. doi: 10.1214/aos/1176345895 . MR   0663456. Zbl   0485.62004.
  15. Nogales, A.G.; Oyola, J.A.; Perez, P. (2000). "On conditional independence and the relationship between sufficiency and invariance under the Bayesian point of view". Statistics & Probability Letters. 46 (1): 75–84. doi:10.1016/S0167-7152(99)00089-9. MR   1731351. Zbl   0964.62003.
  16. Goldstein, M.; O'Hagan, A. (1996). "Bayes Linear Sufficiency and Systems of Expert Posterior Assessments". Journal of the Royal Statistical Society . Series B. 58 (2): 301–316. JSTOR   2345978.
  17. Godambe, V. P. (1966). "A New Approach to Sampling from Finite Populations. II Distribution-Free Sufficiency". Journal of the Royal Statistical Society . Series B. 28 (2): 320–328. JSTOR   2984375.
  18. Witting, T. (1987). "The linear Markov property in credibility theory". ASTIN Bulletin. 17 (1): 71–84. doi: 10.2143/ast.17.1.2014984 .

Related Research Articles

Exponential distribution Probability distribution

In probability theory and statistics, the exponential distribution is the probability distribution of the time between events in a Poisson point process, i.e., a process in which events occur continuously and independently at a constant average rate. It is a particular case of the gamma distribution. It is the continuous analogue of the geometric distribution, and it has the key property of being memoryless. In addition to being used for the analysis of Poisson point processes it is found in various other contexts.

In mechanics and geometry, the 3D rotation group, often denoted SO(3), is the group of all rotations about the origin of three-dimensional Euclidean space under the operation of composition.

Gamma distribution Probability distribution

In probability theory and statistics, the gamma distribution is a two-parameter family of continuous probability distributions. The exponential distribution, Erlang distribution, and chi-square distribution are special cases of the gamma distribution. There are two different parameterizations in common use:

  1. With a shape parameter k and a scale parameter θ.
  2. With a shape parameter α = k and an inverse scale parameter β = 1/θ, called a rate parameter.

In probability and statistics, an exponential family is a parametric set of probability distributions of a certain form, specified below. This special form is chosen for mathematical convenience, based on some useful algebraic properties, as well as for generality, as exponential families are in a sense very natural sets of distributions to consider. The term exponential class is sometimes used in place of "exponential family", or the older term Koopman–Darmois family. The terms "distribution" and "family" are often used loosely: specifically, an exponential family is a set of distributions, where the specific distribution varies with the parameter; however, a parametric family of distributions is often referred to as "a distribution", and the set of all exponential families is sometimes loosely referred to as "the" exponential family. They are distinct because they possess a variety of desirable properties, most importantly the existence of a sufficient statistic.

In information geometry, the Fisher information metric is a particular Riemannian metric which can be defined on a smooth statistical manifold, i.e., a smooth manifold whose points are probability measures defined on a common probability space. It can be used to calculate the informational difference between measurements.

In statistics, the Neyman–Pearson lemma was introduced by Jerzy Neyman and Egon Pearson in a paper in 1933. The Neyman-Pearson lemma is part of the Neyman-Pearson theory of statistical testing, which introduced concepts like errors of the second kind, power function, and inductive behavior. The previous Fisherian theory of significance testing postulated only one hypothesis. By introducing a competing hypothesis, the Neyman-Pearsonian flavor of statistical testing allows investigating the two types of errors. The trivial cases where one always rejects or accepts the null hypothesis are of little interest but it does prove that one must not relinquish control over one type of error while calibrating the other. Neyman and Pearson accordingly proceeded to restrict their attention to the class of all level tests while subsequently minimizing type II error, traditionally denoted by . Their seminal paper of 1933, including the Neyman-Pearson lemma, comes at the end of this endeavor, not only showing the existence of tests with the most power that retain a prespecified level of type I error, but also providing a way to construct such tests. The Karlin-Rubin theorem extends the Neyman-Pearson lemma to settings involving composite hypotheses with monotone likelihood ratios.

Bloch sphere Geometrical representation of the pure state space of a two-level quantum mechanical system

In quantum mechanics and computing, the Bloch sphere is a geometrical representation of the pure state space of a two-level quantum mechanical system (qubit), named after the physicist Felix Bloch.

Simple linear regression Linear regression model with a single explanatory variable

In statistics, simple linear regression is a linear regression model with a single explanatory variable. That is, it concerns two-dimensional sample points with one independent variable and one dependent variable and finds a linear function that, as accurately as possible, predicts the dependent variable values as a function of the independent variable. The adjective simple refers to the fact that the outcome variable is related to a single predictor.

In mathematics, the Bessel polynomials are an orthogonal sequence of polynomials. There are a number of different but closely related definitions. The definition favored by mathematicians is given by the series

A ratio distribution is a probability distribution constructed as the distribution of the ratio of random variables having two other known distributions. Given two random variables X and Y, the distribution of the random variable Z that is formed as the ratio Z = X/Y is a ratio distribution.

In mathematics and statistics, a circular mean or angular mean is a mean designed for angles and similar cyclic quantities, such as daytimes, and fractional parts of real numbers. This is necessary since most of the usual means may not be appropriate on angle-like quantities. For example, the arithmetic mean of 0° and 360° is 180°, which is misleading because 360° equals 0° modulo a full cycle. As another example, the "average time" between 11 PM and 1 AM is either midnight or noon, depending on whether the two times are part of a single night or part of a single calendar day. The circular mean is one of the simplest examples of circular statistics and of statistics of non-Euclidean spaces.

In statistical hypothesis testing, a uniformly most powerful (UMP) test is a hypothesis test which has the greatest power among all possible tests of a given size α. For example, according to the Neyman–Pearson lemma, the likelihood-ratio test is UMP for testing simple (point) hypotheses.

Half-normal distribution Probability distribution

In probability theory and statistics, the half-normal distribution is a special case of the folded normal distribution.

Normal-inverse-gamma distribution

In probability theory and statistics, the normal-inverse-gamma distribution is a four-parameter family of multivariate continuous probability distributions. It is the conjugate prior of a normal distribution with unknown mean and variance.

Poisson distribution Discrete probability distribution

In probability theory and statistics, the Poisson distribution is a discrete probability distribution that expresses the probability of a given number of events occurring in a fixed interval of time or space if these events occur with a known constant mean rate and independently of the time since the last event. It is named after French mathematician Siméon Denis Poisson. The Poisson distribution can also be used for the number of events in other specified interval types such as distance, area or volume.

A product distribution is a probability distribution constructed as the distribution of the product of random variables having two other known distributions. Given two statistically independent random variables X and Y, the distribution of the random variable Z that is formed as the product

For certain applications in linear algebra, it is useful to know properties of the probability distribution of the largest eigenvalue of a finite sum of random matrices. Suppose is a finite sequence of random matrices. Analogous to the well-known Chernoff bound for sums of scalars, a bound on the following is sought for a given parameter t:

In mathematics, the oscillator representation is a projective unitary representation of the symplectic group, first investigated by Irving Segal, David Shale, and André Weil. A natural extension of the representation leads to a semigroup of contraction operators, introduced as the oscillator semigroup by Roger Howe in 1988. The semigroup had previously been studied by other mathematicians and physicists, most notably Felix Berezin in the 1960s. The simplest example in one dimension is given by SU(1,1). It acts as Möbius transformations on the extended complex plane, leaving the unit circle invariant. In that case the oscillator representation is a unitary representation of a double cover of SU(1,1) and the oscillator semigroup corresponds to a representation by contraction operators of the semigroup in SL(2,C) corresponding to Möbius transformations that take the unit disk into itself.

In statistics and probability theory, the nonparametric skew is a statistic occasionally used with random variables that take real values. It is a measure of the skewness of a random variable's distribution—that is, the distribution's tendency to "lean" to one side or the other of the mean. Its calculation does not require any knowledge of the form of the underlying distribution—hence the name nonparametric. It has some desirable properties: it is zero for any symmetric distribution; it is unaffected by a scale shift; and it reveals either left- or right-skewness equally well. In some statistical samples it has been shown to be less powerful than the usual measures of skewness in detecting departures of the population from normality.

In mathematics, van der Corput's method generates estimates for exponential sums. The method applies two processes, the van der Corput processes A and B which relate the sums into simpler sums which are easier to estimate.

References