Law of total variance

Last updated May 30, 2024

In probability theory, the law of total variance^[1] or variance decomposition formula or conditional variance formulas or law of iterated variances also known as Eve's law,^[2] states that if $X$ and $Y$ are random variables on the same probability space, and the variance of $Y$ is finite, then

In language perhaps better known to statisticians than to probability theorists, the two terms are the "unexplained" and the "explained" components of the variance respectively (cf. fraction of variance unexplained, explained variation). In actuarial science, specifically credibility theory, the first component is called the expected value of the process variance (EVPV) and the second is called the variance of the hypothetical means (VHM).^[3] These two components are also the source of the term "Eve's law", from the initials EV VE for "expectation of variance" and "variance of expectation".

Explanation

To understand the formula above, we need to comprehend the random variables $\operatorname {E} [Y|X]$ and $\operatorname {Var} (Y|X)$ . These variables depend on the value of $X$ : for a given $x$ , $\operatorname {E} [Y|X=x]$ and $\operatorname {Var} (Y|X=x)$ are constant numbers. Essentially, we use the possible values of $X$ to group the outcomes and then compute the expected values and variances for each group.

The "unexplained" component $\operatorname {E} (\operatorname {Var} [Y|X])$ is simply the average of all the variances of $Y$ within each group. The "explained" component $\operatorname {Var} (\operatorname {E} [Y|X])$ is the variance of the expected values, i.e., it represents the part of the variance that is explained by the variation of the average value of $Y$ for each group.

For an illustration, consider the example of a dog show (a selected excerpt of Analysis_of_variance#Example). Let the random variable $Y$ correspond to the dog weight and $X$ correspond to the breed. In this situation, it is reasonable to expect that the breed explains a major portion of the variance in weight since there is a big variance in the breeds' average weights. Of course, there is still some variance in weight for each breed, which is taken into account in the "unexplained" term.

Note that the "explained" term actually means "explained by the averages." If variances for each fixed $X$ (e.g., for each breed in the example above) are very distinct, those variances are still combined in the "unexplained" term.

Examples

Example 1

Five graduate students take an exam that is graded from 0 to 100. Let $Y$ denote the student's grade and $X$ indicate whether the student is international or domestic. The data is summarized as follows:

Student	$Y$	$X$
1	20	International
2	30	International
3	100	International
4	40	Domestic
5	60	Domestic

Among international students, the mean is $\operatorname {E} [Y|X={\text{International}}]=50$ and the variance is $\operatorname {Var} (Y|X={\text{International}})={\frac {3800}{3}}=1266.{\overline {6}}$ .

Among domestic students, the mean is $\operatorname {E} [Y|X={\text{Domestic}}]=50$ and the variance is $\operatorname {Var} (Y|X={\text{Domestic}})=100$ .

$X$	$P(X)$	$\operatorname {E} [Y\|X]$	$\operatorname {Var} (Y\|X)$
International	3/5	50	1266.6
Domestic	2/5	50	100

The part of the variance of $Y$ "unexplained" by $X$ is the mean of the variances for each group. In this case, it is $\left({\frac {3}{5}}\right)\left({\frac {3800}{3}}\right)+\left({\frac {2}{5}}\right)(100)=800$ . The part of the variance of $Y$ "explained" by $X$ is the variance of the means of $Y$ inside each group defined by the values of the $X$ . In this case, it is zero, since the mean is the same for each group. So the total variation is

\operatorname {Var} (Y)=\operatorname {E} [\operatorname {Var} (Y|X)]+\operatorname {Var} (\operatorname {E} [Y|X])=800+0=800.

Example 2

Suppose $X$ is a coin flip with the probability of heads being $h$ . Suppose that when $X = heads$ then $Y$ is drawn from a normal distribution with mean $μ h$ and standard deviation $σ h$ , and that when $X = tails$ then $Y$ is drawn from normal distribution with mean $μ t$ and standard deviation $σ t$ . Then the first, "unexplained" term on the right-hand side of the above formula is the weighted average of the variances, $hσ h 2 + (1 - h) σ t 2$ , and the second, "explained" term is the variance of the distribution that gives $μ h$ with probability $h$ and gives $μ t$ with probability $1 - h$ .

Formulation

There is a general variance decomposition formula for $c\geq 2$ components (see below).^[4] For example, with two conditioning random variables:

\operatorname {Var} [Y]=\operatorname {E} \left[\operatorname {Var} \left(Y\mid X_{1},X_{2}\right)\right]+\operatorname {E} [\operatorname {Var} (\operatorname {E} \left[Y\mid X_{1},X_{2}\right]\mid X_{1})]+\operatorname {Var} (\operatorname {E} \left[Y\mid X_{1}\right]),

which follows from the law of total conditional variance:^[4]

\operatorname {Var} (Y\mid X_{1})=\operatorname {E} \left[\operatorname {Var} (Y\mid X_{1},X_{2})\mid X_{1}\right]+\operatorname {Var} \left(\operatorname {E} \left[Y\mid X_{1},X_{2}\right]\mid X_{1}\right).

Note that the conditional expected value $\operatorname {E} (Y\mid X)$ is a random variable in its own right, whose value depends on the value of $X.$ Notice that the conditional expected value of $Y$ given the event $X=x$ is a function of $x$ (this is where adherence to the conventional and rigidly case-sensitive notation of probability theory becomes important!). If we write $\operatorname {E} (Y\mid X=x)=g(x)$ then the random variable $\operatorname {E} (Y\mid X)$ is just $g(X).$ Similar comments apply to the conditional variance.

One special case, (similar to the law of total expectation) states that if $A_{1},\ldots ,A_{n}$ is a partition of the whole outcome space, that is, these events are mutually exclusive and exhaustive, then

{\begin{aligned}\operatorname {Var} (X)={}&\sum _{i=1}^{n}\operatorname {Var} (X\mid A_{i})\Pr(A_{i})+\sum _{i=1}^{n}\operatorname {E} [X\mid A_{i}]^{2}(1-\Pr(A_{i}))\Pr(A_{i})\\[4pt]&{}-2\sum _{i=2}^{n}\sum _{j=1}^{i-1}\operatorname {E} [X\mid A_{i}]\Pr(A_{i})\operatorname {E} [X\mid A_{j}]\Pr(A_{j}).\end{aligned}}

In this formula, the first component is the expectation of the conditional variance; the other two components are the variance of the conditional expectation.

Proof

Finite Case

Let $(x_{1},y_{1}),\ldots ,(x_{n},y_{n})$ be observed values of $(X,Y)$ , with repetitions.

Set ${\bar {y}}=\operatorname {E} [Y]$ and, for each possible value $x$ of $X$ , set ${\bar {y}}_{x_{i}}=\operatorname {E} [Y|X=x_{i}]$ .

Note that

(y_{i}-{\bar {y}})^{2}=\left(y_{i}-{\bar {y}}_{x_{i}}+{\bar {y}}_{x_{i}}-{\bar {y}}\right)^{2}=(y_{i}-{\bar {y}}_{x_{i}})^{2}+({\bar {y}}_{x_{i}}-{\bar {y}})^{2}+2(y_{i}-{\bar {y}}_{x_{i}})({\bar {y}}_{x_{i}}-{\bar {y}}).

Summing these for $1\leq i\leq n$ , the last parcel becomes

\sum _{i=1}^{n}2(y_{i}-{\bar {y}}_{x_{i}})({\bar {y}}_{x_{i}}-{\bar {y}})=2\sum _{x}\left(\sum _{\{1\leq i\leq n|x_{i}=x\}}(y_{i}-{\bar {y}}_{x})\right)({\bar {y}}_{x}-{\bar {y}})=2\sum _{x}0\cdot ({\bar {y}}_{x}-{\bar {y}})=0.

Hence,

\operatorname {Var} (Y)={\frac {1}{n}}\sum _{i=1}^{n}(y_{i}-{\bar {y}})^{2}={\frac {1}{n}}\sum _{i=1}^{n}(y_{i}-{\bar {y}}_{x_{i}})^{2}+{\frac {1}{n}}\sum _{i=1}^{n}({\bar {y}}_{x_{i}}-{\bar {y}})^{2}=\operatorname {E} [\operatorname {Var} (Y\mid X)]+\operatorname {Var} (\operatorname {E} [Y\mid X]).

General Case

The law of total variance can be proved using the law of total expectation.^[5] First,

\operatorname {Var} (Y)=\operatorname {E} \left[Y^{2}\right]-\operatorname {E} [Y]^{2}

from the definition of variance. Again, from the definition of variance, and applying the law of total expectation, we have

\operatorname {E} \left[Y^{2}\right]=\operatorname {E} \left[\operatorname {E} [Y^{2}\mid X]\right]=\operatorname {E} \left[\operatorname {Var} (Y\mid X)+\operatorname {E} [Y\mid X]^{2}\right].

Now we rewrite the conditional second moment of $Y$ in terms of its variance and first moment, and apply the law of total expectation on the right hand side:

\operatorname {E} \left[Y^{2}\right]-\operatorname {E} [Y]^{2}=\operatorname {E} \left[\operatorname {Var} (Y\mid X)+\operatorname {E} [Y\mid X]^{2}\right]-\operatorname {E} [\operatorname {E} [Y\mid X]]^{2}.

Since the expectation of a sum is the sum of expectations, the terms can now be regrouped:

=\left(\operatorname {E} [\operatorname {Var} (Y\mid X)]\right)+\left(\operatorname {E} \left[\operatorname {E} [Y\mid X]^{2}\right]-\operatorname {E} [\operatorname {E} [Y\mid X]]^{2}\right).

Finally, we recognize the terms in the second set of parentheses as the variance of the conditional expectation $\operatorname {E} [Y\mid X]$ :

=\operatorname {E} [\operatorname {Var} (Y\mid X)]+\operatorname {Var} (\operatorname {E} [Y\mid X]).

General variance decomposition applicable to dynamic systems

The following formula shows how to apply the general, measure theoretic variance decomposition formula ^[4] to stochastic dynamic systems. Let $Y(t)$ be the value of a system variable at time $t.$ Suppose we have the internal histories (natural filtrations) $H_{1t},H_{2t},\ldots ,H_{c-1,t}$ , each one corresponding to the history (trajectory) of a different collection of system variables. The collections need not be disjoint. The variance of $Y(t)$ can be decomposed, for all times $t,$ into $c\geq 2$ components as follows:

{\begin{aligned}\operatorname {Var} [Y(t)]={}&\operatorname {E} (\operatorname {Var} [Y(t)\mid H_{1t},H_{2t},\ldots ,H_{c-1,t}])\\[4pt]&{}+\sum _{j=2}^{c-1}\operatorname {E} (\operatorname {Var} [\operatorname {E} [Y(t)\mid H_{1t},H_{2t},\ldots ,H_{jt}]\mid H_{1t},H_{2t},\ldots ,H_{j-1,t}])\\[4pt]&{}+\operatorname {Var} (\operatorname {E} [Y(t)\mid H_{1t}]).\end{aligned}}

The decomposition is not unique. It depends on the order of the conditioning in the sequential decomposition.

The square of the correlation and explained (or informational) variation

In cases where $(Y,X)$ are such that the conditional expected value is linear; that is, in cases where

\operatorname {E} (Y\mid X)=aX+b,

it follows from the bilinearity of covariance that

a={\operatorname {Cov} (Y,X) \over \operatorname {Var} (X)}

and

b=\operatorname {E} (Y)-{\operatorname {Cov} (Y,X) \over \operatorname {Var} (X)}\operatorname {E} (X)

and the explained component of the variance divided by the total variance is just the square of the correlation between $Y$ and $X;$ that is, in such cases,

{\operatorname {Var} (\operatorname {E} (Y\mid X)) \over \operatorname {Var} (Y)}=\operatorname {Corr} (X,Y)^{2}.

One example of this situation is when $(X,Y)$ have a bivariate normal (Gaussian) distribution.

More generally, when the conditional expectation $\operatorname {E} (Y\mid X)$ is a non-linear function of $X$ ^[4]

\iota _{Y\mid X}={\operatorname {Var} (\operatorname {E} (Y\mid X)) \over \operatorname {Var} (Y)}=\operatorname {Corr} (\operatorname {E} (Y\mid X),Y)^{2},

which can be estimated as the $R$ squared from a non-linear regression of $Y$ on $X,$ using data drawn from the joint distribution of $(X,Y).$ When $\operatorname {E} (Y\mid X)$ has a Gaussian distribution (and is an invertible function of $X$ ), or $Y$ itself has a (marginal) Gaussian distribution, this explained component of variation sets a lower bound on the mutual information:^[4]

\operatorname {I} (Y;X)\geq \ln \left([1-\iota _{Y\mid X}]^{-1/2}\right).

Higher moments

A similar law for the third central moment $\mu _{3}$ says

\mu _{3}(Y)=\operatorname {E} \left(\mu _{3}(Y\mid X)\right)+\mu _{3}(\operatorname {E} (Y\mid X))+3\operatorname {cov} (\operatorname {E} (Y\mid X),\operatorname {var} (Y\mid X)).

For higher cumulants, a generalization exists. See law of total cumulance.

Related Research Articles

In statistics, the standard deviation is a measure of the amount of variation of a random variable expected about its mean. A low standard deviation indicates that the values tend to be close to the mean of the set, while a high standard deviation indicates that the values are spread out over a wider range. The standard deviation is commonly used in the determination of what constitutes an outlier and what does not.

In probability theory and statistics, variance is the expected value of the squared deviation from the mean of a random variable. The standard deviation (SD) is obtained as the square root of the variance. Variance is a measure of dispersion, meaning it is a measure of how far a set of numbers is spread out from their average value. It is the second central moment of a distribution, and the covariance of the random variable with itself, and it is often represented by $,,,, or .$

In probability theory, the central limit theorem (CLT) states that, under appropriate conditions, the distribution of a normalized version of the sample mean converges to a standard normal distribution. This holds even if the original variables themselves are not normally distributed. There are several versions of the CLT, each applying in the context of different conditions.

In probability, and statistics, a multivariate random variable or random vector is a list or vector of mathematical variables each of whose value is unknown, either because the value has not yet occurred or because there is imperfect knowledge of its value. The individual variables in a random vector are grouped together because they are all part of a single mathematical system — often they represent different properties of an individual statistical unit. For example, while a given person has a specific age, height and weight, the representation of these features of an unspecified person from within a group would be a random vector. Normally each element of a random vector is a real number.

Covariance in probability theory and statistics is a measure of the joint variability of two random variables.

<span class="mw-page-title-main">Covariance matrix</span> Measure of covariance of components of a random vector

In probability theory and statistics, a covariance matrix is a square matrix giving the covariance between each pair of elements of a given random vector.

In probability theory and statistics, the beta distribution is a family of continuous probability distributions defined on the interval [0, 1] or in terms of two positive parameters, denoted by alpha (α) and beta (β), that appear as exponents of the variable and its complement to 1, respectively, and control the shape of the distribution.

In statistics, the Pearson correlation coefficient (PCC) is a correlation coefficient that measures linear correlation between two sets of data. It is the ratio between the covariance of two variables and the product of their standard deviations; thus, it is essentially a normalized measurement of the covariance, such that the result always has a value between −1 and 1. As with covariance itself, the measure can only reflect a linear correlation of variables, and ignores many other types of relationships or correlations. As a simple example, one would expect the age and height of a sample of teenagers from a high school to have a Pearson correlation coefficient significantly greater than 0, but less than 1.

In mathematics, the moments of a function are certain quantitative measures related to the shape of the function's graph. If the function represents mass density, then the zeroth moment is the total mass, the first moment is the center of mass, and the second moment is the moment of inertia. If the function is a probability distribution, then the first moment is the expected value, the second central moment is the variance, the third standardized moment is the skewness, and the fourth standardized moment is the kurtosis. The mathematical concept is closely related to the concept of moment in physics.

In statistics, the Rao–Blackwell theorem, sometimes referred to as the Rao–Blackwell–Kolmogorov theorem, is a result that characterizes the transformation of an arbitrarily crude estimator into an estimator that is optimal by the mean-squared-error criterion or any of a variety of similar criteria.

In probability theory, the conditional expectation, conditional expected value, or conditional mean of a random variable is its expected value evaluated with respect to the conditional probability distribution. If the random variable can take on only a finite number of values, the "conditions" are that the variable can only take on a subset of those values. More formally, in the case when the random variable is defined over a discrete probability space, the "conditions" are a partition of this probability space.

In statistics, originally in geostatistics, kriging or Kriging, also known as Gaussian process regression, is a method of interpolation based on Gaussian process governed by prior covariances. Under suitable assumptions of the prior, kriging gives the best linear unbiased prediction (BLUP) at unsampled locations. Interpolating methods based on other criteria such as smoothness may not yield the BLUP. The method is widely used in the domain of spatial analysis and computer experiments. The technique is also known as Wiener–Kolmogorov prediction, after Norbert Wiener and Andrey Kolmogorov.

In probability theory, a compound Poisson distribution is the probability distribution of the sum of a number of independent identically-distributed random variables, where the number of terms to be added is itself a Poisson-distributed variable. The result can be either a continuous or a discrete distribution.

In statistics, the bias of an estimator is the difference between this estimator's expected value and the true value of the parameter being estimated. An estimator or decision rule with zero bias is called unbiased. In statistics, "bias" is an objective property of an estimator. Bias is a distinct concept from consistency: consistent estimators converge in probability to the true value of the parameter, but may be biased or unbiased; see bias versus consistency for more.

In probability theory and statistics, partial correlation measures the degree of association between two random variables, with the effect of a set of controlling random variables removed. When determining the numerical relationship between two variables of interest, using their correlation coefficient will give misleading results if there is another confounding variable that is numerically related to both variables of interest. This misleading information can be avoided by controlling for the confounding variable, which is done by computing the partial correlation coefficient. This is precisely the motivation for including other right-side variables in a multiple regression; but while multiple regression gives unbiased results for the effect size, it does not give a numerical value of a measure of the strength of the relationship between the two variables of interest.

In probability theory and statistics, a conditional variance is the variance of a random variable given the value(s) of one or more other variables. Particularly in econometrics, the conditional variance is also known as the scedastic function or skedastic function. Conditional variances are important parts of autoregressive conditional heteroskedasticity (ARCH) models.

In regression, mean response and predicted response, also known as mean outcome and predicted outcome, are values of the dependent variable calculated from the regression parameters and a given value of the independent variable. The values of these two responses are the same, but their calculated variances are different. The concept is a generalization of the distinction between the standard error of the mean and the sample standard deviation.

In probability theory, the law of total covariance, covariance decomposition formula, or conditional covariance formula states that if X, Y, and Z are random variables on the same probability space, and the covariance of X and Y is finite, then

In statistics and in probability theory, distance correlation or distance covariance is a measure of dependence between two paired random vectors of arbitrary, not necessarily equal, dimension. The population distance correlation coefficient is zero if and only if the random vectors are independent. Thus, distance correlation measures both linear and nonlinear association between two random variables or random vectors. This is in contrast to Pearson's correlation, which can only detect linear association between two random variables.

In probability theory and statistics, complex random variables are a generalization of real-valued random variables to complex numbers, i.e. the possible values a complex random variable may take are complex numbers. Complex random variables can always be considered as pairs of real random variables: their real and imaginary parts. Therefore, the distribution of one complex random variable may be interpreted as the joint distribution of two real random variables.

References

↑ Neil A. Weiss, A Course in Probability, Addison–Wesley, 2005, pages 385–386.
↑ Joseph K. Blitzstein and Jessica Hwang: "Introduction to Probability"
↑ Mahler, Howard C.; Dean, Curtis Gary (2001). "Chapter 8: Credibility" (PDF). In Casualty Actuarial Society (ed.). Foundations of Casualty Actuarial Science (4th ed.). Casualty Actuarial Society. pp. 525–526. ISBN 978-0-96247-622-8 . Retrieved June 25, 2015.
1 2 3 4 5 Bowsher, C.G. and P.S. Swain, Identifying sources of variation and the flow of information in biochemical networks, PNAS May 15, 2012 109 (20) E1320-E1328.
↑ Neil A. Weiss, A Course in Probability, Addison–Wesley, 2005, pages 380–383.

Blitzstein, Joe. "Stat 110 Final Review (Eve's Law)" (PDF). stat110.net. Harvard University, Department of Statistics. Retrieved 9 July 2014.
Billingsley, Patrick (1995). Probability and Measure. New York, NY: John Wiley & Sons, Inc. ISBN 0-471-00710-2. (Problem 34.10(b))

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.

[1] Neil A. Weiss, A Course in Probability, Addison–Wesley, 2005, pages 385–386.

[2] Joseph K. Blitzstein and Jessica Hwang: "Introduction to Probability"

[FCAS4ed-3] Mahler, Howard C.; Dean, Curtis Gary (2001). "Chapter 8: Credibility" (PDF). In Casualty Actuarial Society (ed.). Foundations of Casualty Actuarial Science (4th ed.). Casualty Actuarial Society. pp. 525–526. ISBN 978-0-96247-622-8 . Retrieved June 25, 2015.

[bs-4] 1 2 3 4 5 Bowsher, C.G. and P.S. Swain, Identifying sources of variation and the flow of information in biochemical networks, PNAS May 15, 2012 109 (20) E1320-E1328.

[5] Neil A. Weiss, A Course in Probability, Addison–Wesley, 2005, pages 380–383.

[1]

[2]

[3]

[4]

[5]