Moment-generating function

Last updated May 25, 2024

In probability theory and statistics, the moment-generating function of a real-valued random variable is an alternative specification of its probability distribution. Thus, it provides the basis of an alternative route to analytical results compared with working directly with probability density functions or cumulative distribution functions. There are particularly simple results for the moment-generating functions of distributions defined by the weighted sums of random variables. However, not all random variables have moment-generating functions.

Definition

Let $X$ be a random variable with CDF $F_{X}$ . The moment generating function (mgf) of $X$ (or $F_{X}$ ), denoted by $M_{X}(t)$ , is

M_{X}(t)=\operatorname {E} \left[e^{tX}\right]

provided this expectation exists for $t$ in some open neighborhood of 0. That is, there is an $h>0$ such that for all $t$ in $-h<t<h$ , $\operatorname {E} \left[e^{tX}\right]$ exists. If the expectation does not exist in an open neighborhood of 0, we say that the moment generating function does not exist.^[1]

In other words, the moment-generating function of X is the expectation of the random variable $e^{tX}$ . More generally, when $\mathbf {X} =(X_{1},\ldots ,X_{n})^{\mathrm {T} }$ , an $n$ -dimensional random vector, and $\mathbf {t}$ is a fixed vector, one uses $\mathbf {t} \cdot \mathbf {X} =\mathbf {t} ^{\mathrm {T} }\mathbf {X}$ instead of $tX$ :

M_{\mathbf {X} }(\mathbf {t} ):=\operatorname {E} \left(e^{\mathbf {t} ^{\mathrm {T} }\mathbf {X} }\right).

$M_{X}(0)$ always exists and is equal to 1. However, a key problem with moment-generating functions is that moments and the moment-generating function may not exist, as the integrals need not converge absolutely. By contrast, the characteristic function or Fourier transform always exists (because it is the integral of a bounded function on a space of finite measure), and for some purposes may be used instead.

The moment-generating function is so named because it can be used to find the moments of the distribution.^[2] The series expansion of $e^{tX}$ is

e^{t\,X}=1+t\,X+{\frac {t^{2}\,X^{2}}{2!}}+{\frac {t^{3}\,X^{3}}{3!}}+\cdots +{\frac {t^{n}\,X^{n}}{n!}}+\cdots .

Hence

{\begin{aligned}M_{X}(t)=\operatorname {E} (e^{t\,X})&=1+t\operatorname {E} (X)+{\frac {t^{2}\operatorname {E} (X^{2})}{2!}}+{\frac {t^{3}\operatorname {E} (X^{3})}{3!}}+\cdots +{\frac {t^{n}\operatorname {E} (X^{n})}{n!}}+\cdots \\&=1+tm_{1}+{\frac {t^{2}m_{2}}{2!}}+{\frac {t^{3}m_{3}}{3!}}+\cdots +{\frac {t^{n}m_{n}}{n!}}+\cdots ,\end{aligned}}

where $m_{n}$ is the $n$ th moment. Differentiating $M_{X}(t)$ $i$ times with respect to $t$ and setting $t=0$ , we obtain the $i$ th moment about the origin, $m_{i}$ ; see Calculations of moments below.

If $X$ is a continuous random variable, the following relation between its moment-generating function $M_{X}(t)$ and the two-sided Laplace transform of its probability density function $f_{X}(x)$ holds:

M_{X}(t)={\mathcal {L}}\{f_{X}\}(-t),

since the PDF's two-sided Laplace transform is given as

{\mathcal {L}}\{f_{X}\}(s)=\int _{-\infty }^{\infty }e^{-sx}f_{X}(x)\,dx,

and the moment-generating function's definition expands (by the law of the unconscious statistician) to

M_{X}(t)=\operatorname {E} \left[e^{tX}\right]=\int _{-\infty }^{\infty }e^{tx}f_{X}(x)\,dx.

This is consistent with the characteristic function of $X$ being a Wick rotation of $M_{X}(t)$ when the moment generating function exists, as the characteristic function of a continuous random variable $X$ is the Fourier transform of its probability density function $f_{X}(x)$ , and in general when a function $f(x)$ is of exponential order, the Fourier transform of $f$ is a Wick rotation of its two-sided Laplace transform in the region of convergence. See the relation of the Fourier and Laplace transforms for further information.

Examples

Here are some examples of the moment-generating function and the characteristic function for comparison. It can be seen that the characteristic function is a Wick rotation of the moment-generating function $M_{X}(t)$ when the latter exists.

Distribution	Moment-generating function $M_{X}(t)$	Characteristic function $\varphi (t)$
Degenerate $\delta _{a}$	$e^{ta}$	$e^{ita}$
Bernoulli $P(X=1)=p$	$1-p+pe^{t}$	$1-p+pe^{it}$
Geometric $(1-p)^{k-1}\,p$	${\frac {pe^{t}}{1-(1-p)e^{t}}},~t<-\ln(1-p)$	${\frac {pe^{it}}{1-(1-p)\,e^{it}}}$
Binomial $B(n,p)$	$\left(1-p+pe^{t}\right)^{n}$	$\left(1-p+pe^{it}\right)^{n}$
Negative binomial $\operatorname {NB} (r,p)$	$\left({\frac {p}{1-e^{t}+pe^{t}}}\right)^{r},~t<-\ln(1-p)$	$\left({\frac {p}{1-e^{it}+pe^{it}}}\right)^{r}$
Poisson $\operatorname {Pois} (\lambda )$	$e^{\lambda (e^{t}-1)}$	$e^{\lambda (e^{it}-1)}$
Uniform (continuous) $\operatorname {U} (a,b)$	${\frac {e^{tb}-e^{ta}}{t(b-a)}}$	${\frac {e^{itb}-e^{ita}}{it(b-a)}}$
Uniform (discrete) $\operatorname {DU} (a,b)$	${\frac {e^{at}-e^{(b+1)t}}{(b-a+1)(1-e^{t})}}$	${\frac {e^{ait}-e^{(b+1)it}}{(b-a+1)(1-e^{it})}}$
Laplace $L(\mu ,b)$	${\frac {e^{t\mu }}{1-b^{2}t^{2}}},~\|t\|<1/b$	${\frac {e^{it\mu }}{1+b^{2}t^{2}}}$
Normal $N(\mu ,\sigma ^{2})$	$e^{t\mu +{\frac {1}{2}}\sigma ^{2}t^{2}}$	$e^{it\mu -{\frac {1}{2}}\sigma ^{2}t^{2}}$
Chi-squared $\chi _{k}^{2}$	$(1-2t)^{-{\frac {k}{2}}},~t<1/2$	$(1-2it)^{-{\frac {k}{2}}}$
Noncentral chi-squared $\chi _{k}^{2}(\lambda )$	$e^{\lambda t/(1-2t)}(1-2t)^{-{\frac {k}{2}}}$	$e^{i\lambda t/(1-2it)}(1-2it)^{-{\frac {k}{2}}}$
Gamma $\Gamma (k,\theta )$	$(1-t\theta )^{-k},~t<{\tfrac {1}{\theta }}$	$(1-it\theta )^{-k}$
Exponential $\operatorname {Exp} (\lambda )$	$\left(1-t\lambda ^{-1}\right)^{-1},~t<\lambda$	$\left(1-it\lambda ^{-1}\right)^{-1}$
Beta	$1+\sum _{k=1}^{\infty }\left(\prod _{r=0}^{k-1}{\frac {\alpha +r}{\alpha +\beta +r}}\right){\frac {t^{k}}{k!}}$	${\displaystyle {}_{1}F_{1}(\alpha$ (see Confluent hypergeometric function)
Multivariate normal $N(\mathbf {\mu } ,\mathbf {\Sigma } )$	$e^{\mathbf {t} ^{\mathrm {T} }\left({\boldsymbol {\mu }}+{\frac {1}{2}}\mathbf {\Sigma t} \right)}$	$e^{\mathbf {t} ^{\mathrm {T} }\left(i{\boldsymbol {\mu }}-{\frac {1}{2}}{\boldsymbol {\Sigma }}\mathbf {t} \right)}$
Cauchy $\operatorname {Cauchy} (\mu ,\theta )$	Does not exist	$e^{it\mu -\theta \|t\|}$
Multivariate Cauchy $\operatorname {MultiCauchy} (\mu ,\Sigma )$ ^[3]	Does not exist	$\!\,e^{i\mathbf {t} ^{\mathrm {T} }{\boldsymbol {\mu }}-{\sqrt {\mathbf {t} ^{\mathrm {T} }{\boldsymbol {\Sigma }}\mathbf {t} }}}$

Calculation

The moment-generating function is the expectation of a function of the random variable, it can be written as:

For a discrete probability mass function, $M_{X}(t)=\sum _{i=0}^{\infty }e^{tx_{i}}\,p_{i}$
For a continuous probability density function, $M_{X}(t)=\int _{-\infty }^{\infty }e^{tx}f(x)\,dx$
In the general case: $M_{X}(t)=\int _{-\infty }^{\infty }e^{tx}\,dF(x)$ , using the Riemann–Stieltjes integral, and where $F$ is the cumulative distribution function. This is simply the Laplace-Stieltjes transform of $F$ , but with the sign of the argument reversed.

Note that for the case where $X$ has a continuous probability density function $f(x)$ , $M_{X}(-t)$ is the two-sided Laplace transform of $f(x)$ .

{\begin{aligned}M_{X}(t)&=\int _{-\infty }^{\infty }e^{tx}f(x)\,dx\\&=\int _{-\infty }^{\infty }\left(1+tx+{\frac {t^{2}x^{2}}{2!}}+\cdots +{\frac {t^{n}x^{n}}{n!}}+\cdots \right)f(x)\,dx\\&=1+tm_{1}+{\frac {t^{2}m_{2}}{2!}}+\cdots +{\frac {t^{n}m_{n}}{n!}}+\cdots ,\end{aligned}}

where $m_{n}$ is the $n$ th moment.

Linear transformations of random variables

If random variable $X$ has moment generating function $M_{X}(t)$ , then $\alpha X+\beta$ has moment generating function $M_{\alpha X+\beta }(t)=e^{\beta t}M_{X}(\alpha t)$

M_{\alpha X+\beta }(t)=E[e^{(\alpha X+\beta )t}]=e^{\beta t}E[e^{\alpha Xt}]=e^{\beta t}M_{X}(\alpha t)

Linear combination of independent random variables

If $S_{n}=\sum _{i=1}^{n}a_{i}X_{i}$ , where the X_i are independent random variables and the a_i are constants, then the probability density function for S_n is the convolution of the probability density functions of each of the X_i, and the moment-generating function for S_n is given by

M_{S_{n}}(t)=M_{X_{1}}(a_{1}t)M_{X_{2}}(a_{2}t)\cdots M_{X_{n}}(a_{n}t)\,.

Vector-valued random variables

For vector-valued random variables

\mathbf {X}

with real components, the moment-generating function is given by

M_{X}(\mathbf {t} )=E\left(e^{\langle \mathbf {t} ,\mathbf {X} \rangle }\right)

where $\mathbf {t}$ is a vector and $\langle \cdot ,\cdot \rangle$ is the dot product.

Important properties

Moment generating functions are positive and log-convex,^{[ citation needed ]} with M(0) = 1.

An important property of the moment-generating function is that it uniquely determines the distribution. In other words, if $X$ and $Y$ are two random variables and for all values of t,

M_{X}(t)=M_{Y}(t),\,

then

F_{X}(x)=F_{Y}(x)\,

for all values of x (or equivalently X and Y have the same distribution). This statement is not equivalent to the statement "if two distributions have the same moments, then they are identical at all points." This is because in some cases, the moments exist and yet the moment-generating function does not, because the limit

\lim _{n\rightarrow \infty }\sum _{i=0}^{n}{\frac {t^{i}m_{i}}{i!}}

may not exist. The log-normal distribution is an example of when this occurs.

Calculations of moments

The moment-generating function is so called because if it exists on an open interval around t = 0, then it is the exponential generating function of the moments of the probability distribution:

m_{n}=E\left(X^{n}\right)=M_{X}^{(n)}(0)=\left.{\frac {d^{n}M_{X}}{dt^{n}}}\right|_{t=0}.

That is, with n being a nonnegative integer, the nth moment about 0 is the nth derivative of the moment generating function, evaluated at t = 0.

Other properties

Jensen's inequality provides a simple lower bound on the moment-generating function:

M_{X}(t)\geq e^{\mu t},

where $\mu$ is the mean of X.

The moment-generating function can be used in conjunction with Markov's inequality to bound the upper tail of a real random variable X. This statement is also called the Chernoff bound. Since $x\mapsto e^{xt}$ is monotonically increasing for $t>0$ , we have

P(X\geq a)=P(e^{tX}\geq e^{ta})\leq e^{-at}E[e^{tX}]=e^{-at}M_{X}(t)

for any $t>0$ and any a, provided $M_{X}(t)$ exists. For example, when X is a standard normal distribution and $a>0$ , we can choose $t=a$ and recall that $M_{X}(t)=e^{t^{2}/2}$ . This gives $P(X\geq a)\leq e^{-a^{2}/2}$ , which is within a factor of 1+a of the exact value.

Various lemmas, such as Hoeffding's lemma or Bennett's inequality provide bounds on the moment-generating function in the case of a zero-mean, bounded random variable.

When $X$ is non-negative, the moment generating function gives a simple, useful bound on the moments:

E[X^{m}]\leq \left({\frac {m}{te}}\right)^{m}M_{X}(t),

For any $X,m\geq 0$ and $t>0$ .

This follows from the inequality $1+x\leq e^{x}$ into which we can substitute $x'=tx/m-1$ implies $tx/m\leq e^{tx/m-1}$ for any $x,t,m\in \mathbb {R}$ . Now, if $t>0$ and $x,m\geq 0$ , this can be rearranged to $x^{m}\leq (m/(te))^{m}e^{tx}$ . Taking the expectation on both sides gives the bound on $E[X^{m}]$ in terms of $E[e^{tX}]$ .

As an example, consider $X\sim {\text{Chi-Squared}}$ with $k$ degrees of freedom. Then from the examples $M_{X}(t)=(1-2t)^{-k/2}$ . Picking $t=m/(2m+k)$ and substituting into the bound:

E[X^{m}]\leq (1+2m/k)^{k/2}e^{-m}(k+2m)^{m}.

We know that in this case the correct bound is $E[X^{m}]\leq 2^{m}\Gamma (m+k/2)/\Gamma (k/2)$ . To compare the bounds, we can consider the asymptotics for large $k$ . Here the moment-generating function bound is $k^{m}(1+m^{2}/k+O(1/k^{2}))$ , where the real bound is $k^{m}(1+(m^{2}-m)/k+O(1/k^{2}))$ . The moment-generating function bound is thus very strong in this case.

Relation to other functions

Related to the moment-generating function are a number of other transforms that are common in probability theory:

Characteristic function: The characteristic function $\varphi _{X}(t)$ is related to the moment-generating function via $\varphi _{X}(t)=M_{iX}(t)=M_{X}(it):$ the characteristic function is the moment-generating function of iX or the moment generating function of X evaluated on the imaginary axis. This function can also be viewed as the Fourier transform of the probability density function, which can therefore be deduced from it by inverse Fourier transform.
Cumulant-generating function: The cumulant-generating function is defined as the logarithm of the moment-generating function; some instead define the cumulant-generating function as the logarithm of the characteristic function, while others call this latter the second cumulant-generating function.
Probability-generating function: The probability-generating function is defined as $G(z)=E\left[z^{X}\right].\,$ This immediately implies that $G(e^{t})=E\left[e^{tX}\right]=M_{X}(t).\,$

Related Research Articles

In probability theory and statistics, the cumulative distribution function (CDF) of a real-valued random variable $, or just distribution function of, evaluated at, is the probability that will take a value less than or equal to .$

The Cauchy distribution, named after Augustin Cauchy, is a continuous probability distribution. It is also known, especially among physicists, as the Lorentz distribution, Cauchy–Lorentz distribution, Lorentz(ian) function, or Breit–Wigner distribution. The Cauchy distribution $is the distribution of the x -intercept of a ray issuing from with a uniformly distributed angle. It is also the distribution of the ratio of two independent normally distributed random variables with mean zero.$

In probability theory, the expected value is a generalization of the weighted average. Informally, the expected value is the arithmetic mean of the possible values a random variable can take, weighted by the probability of those outcomes. Since it is obtained through arithmetic, the expected value sometimes may not even be included in the sample data set; it is not the value you would "expect" to get in reality.

In probability theory and statistics, a normal distribution or Gaussian distribution is a type of continuous probability distribution for a real-valued random variable. The general form of its probability density function is

Independence is a fundamental notion in probability theory, as in statistics and the theory of stochastic processes. Two events are independent, statistically independent, or stochastically independent if, informally speaking, the occurrence of one does not affect the probability of occurrence of the other or, equivalently, does not affect the odds. Similarly, two random variables are independent if the realization of one does not affect the probability distribution of the other.

In probability theory, the central limit theorem (CLT) states that, under appropriate conditions, the distribution of a normalized version of the sample mean converges to a standard normal distribution. This holds even if the original variables themselves are not normally distributed. There are several versions of the CLT, each applying in the context of different conditions.

In probability theory, a probability density function (PDF), density function, or density of an absolutely continuous random variable, is a function whose value at any given sample in the sample space can be interpreted as providing a relative likelihood that the value of the random variable would be equal to that sample. Probability density is the probability per unit length, in other words, while the absolute likelihood for a continuous random variable to take on any particular value is 0, the value of the PDF at two different samples can be used to infer, in any particular draw of the random variable, how much more likely it is that the random variable would be close to one sample compared to the other sample.

In probability, and statistics, a multivariate random variable or random vector is a list or vector of mathematical variables each of whose value is unknown, either because the value has not yet occurred or because there is imperfect knowledge of its value. The individual variables in a random vector are grouped together because they are all part of a single mathematical system — often they represent different properties of an individual statistical unit. For example, while a given person has a specific age, height and weight, the representation of these features of an unspecified person from within a group would be a random vector. Normally each element of a random vector is a real number.

<span class="mw-page-title-main">Multivariate normal distribution</span> Generalization of the one-dimensional normal distribution to higher dimensions

In probability theory and statistics, the multivariate normal distribution, multivariate Gaussian distribution, or joint normal distribution is a generalization of the one-dimensional (univariate) normal distribution to higher dimensions. One definition is that a random vector is said to be k-variate normally distributed if every linear combination of its k components has a univariate normal distribution. Its importance derives mainly from the multivariate central limit theorem. The multivariate normal distribution is often used to describe, at least approximately, any set of (possibly) correlated real-valued random variables, each of which clusters around a mean value.

In probability theory, the probability generating function of a discrete random variable is a power series representation (the generating function) of the probability mass function of the random variable. Probability generating functions are often employed for their succinct description of the sequence of probabilities Pr(X = i) in the probability mass function for a random variable X, and to make available the well-developed theory of power series with non-negative coefficients.

In probability theory and statistics, the cumulants $κ n$ of a probability distribution are a set of quantities that provide an alternative to the moments of the distribution. Any two probability distributions whose moments are identical will have identical cumulants as well, and vice versa.

In mathematics, the moments of a function are certain quantitative measures related to the shape of the function's graph. If the function represents mass density, then the zeroth moment is the total mass, the first moment is the center of mass, and the second moment is the moment of inertia. If the function is a probability distribution, then the first moment is the expected value, the second central moment is the variance, the third standardized moment is the skewness, and the fourth standardized moment is the kurtosis. The mathematical concept is closely related to the concept of moment in physics.

In signal processing, cross-correlation is a measure of similarity of two series as a function of the displacement of one relative to the other. This is also known as a sliding dot product or sliding inner-product. It is commonly used for searching a long signal for a shorter, known feature. It has applications in pattern recognition, single particle analysis, electron tomography, averaging, cryptanalysis, and neurophysiology. The cross-correlation is similar in nature to the convolution of two functions. In an autocorrelation, which is the cross-correlation of a signal with itself, there will always be a peak at a lag of zero, and its size will be the signal energy.

In probability theory and statistics, the Lévy distribution, named after Paul Lévy, is a continuous probability distribution for a non-negative random variable. In spectroscopy, this distribution, with frequency as the dependent variable, is known as a van der Waals profile. It is a special case of the inverse-gamma distribution. It is a stable distribution.

In probability theory and statistics, the continuous uniform distributions or rectangular distributions are a family of symmetric probability distributions. Such a distribution describes an experiment where there is an arbitrary outcome that lies between certain bounds. The bounds are defined by the parameters, $and which are the minimum and maximum values. The interval can either be closed or open. Therefore, the distribution is often abbreviated where stands for uniform distribution. The difference between the bounds defines the interval length; all intervals of the same length on the distribution's support are equally probable. It is the maximum entropy probability distribution for a random variable under no constraint other than that it is contained in the distribution's support.$

In probability theory and statistics, the characteristic function of any real-valued random variable completely defines its probability distribution. If a random variable admits a probability density function, then the characteristic function is the Fourier transform of the probability density function. Thus it provides an alternative route to analytical results compared with working directly with probability density functions or cumulative distribution functions. There are particularly simple results for the characteristic functions of distributions defined by the weighted sums of random variables.

In probability theory and statistics, the factorial moment generating function (FMGF) of the probability distribution of a real-valued random variable X is defined as

In statistics, the multivariate t-distribution is a multivariate probability distribution. It is a generalization to random vectors of the Student's t-distribution, which is a distribution applicable to univariate random variables. While the case of a random matrix could be treated within this structure, the matrix t-distribution is distinct and makes particular use of the matrix structure.

In probability theory and statistics, the Poisson distribution is a discrete probability distribution that expresses the probability of a given number of events occurring in a fixed interval of time if these events occur with a known constant mean rate and independently of the time since the last event. It can also be used for the number of events in other types of intervals than time, and in dimension greater than 1.

In probability theory and statistics, the Conway–Maxwell–binomial (CMB) distribution is a three parameter discrete probability distribution that generalises the binomial distribution in an analogous manner to the way that the Conway–Maxwell–Poisson distribution generalises the Poisson distribution. The CMB distribution can be used to model both positive and negative association among the Bernoulli summands,.

References

Citations

↑ Casella, George; Berger, Roger L. (1990). Statistical Inference. Wadsworth & Brooks/Cole. p. 61. ISBN 0-534-11958-1.
↑ Bulmer, M. G. (1979). Principles of Statistics. Dover. pp. 75–79. ISBN 0-486-63760-3.
↑ Kotz et al.^{[ full citation needed ]} p. 37 using 1 as the number of degree of freedom to recover the Cauchy distribution

{{Ref

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.

[1] Casella, George; Berger, Roger L. (1990). Statistical Inference. Wadsworth & Brooks/Cole. p. 61. ISBN 0-534-11958-1.

[2] Bulmer, M. G. (1979). Principles of Statistics. Dover. pp. 75–79. ISBN 0-486-63760-3.

[3] Kotz et al.^{[ full citation needed ]} p. 37 using 1 as the number of degree of freedom to recover the Cauchy distribution

[1]

[2]

[3]

v t e Theory of probability distributions
probability mass function (pmf) probability density function (pdf) cumulative distribution function (cdf) quantile function
raw moment central moment mean variance standard deviation skewness kurtosis L-moment
moment-generating function (mgf) characteristic function probability-generating function (pgf) cumulant combinant