# Exponential family

Last updated

In probability and statistics, an exponential family is a parametric set of probability distributions of a certain form, specified below. This special form is chosen for mathematical convenience, based on some useful algebraic properties, as well as for generality, as exponential families are in a sense very natural sets of distributions to consider. The term exponential class is sometimes used in place of "exponential family", [1] or the older term Koopman–Darmois family. The terms "distribution" and "family" are often used loosely: specifically, an exponential family is a set of distributions, where the specific distribution varies with the parameter; [lower-alpha 1] however, a parametric family of distributions is often referred to as "a distribution" (like "the normal distribution", meaning "the family of normal distributions"), and the set of all exponential families is sometimes loosely referred to as "the" exponential family. They are distinct because they possess a variety of desirable properties, most importantly the existence of a sufficient statistic.

## Contents

The concept of exponential families is credited to [2] E. J. G. Pitman, [3] G. Darmois, [4] and B. O. Koopman [5] in 19351936. Exponential families of distributions provides a general framework for selecting a possible alternative parameterisation of a parametric family of distributions, in terms of natural parameters, and for defining useful sample statistics, called the natural sufficient statistics of the family.

## Definition

Most of the commonly used distributions form an exponential family or subset of an exponential family, listed in the subsection below. The subsections following it are a sequence of increasingly more general mathematical definitions of an exponential family. A casual reader may wish to restrict attention to the first and simplest definition, which corresponds to a single-parameter family of discrete or continuous probability distributions.

### Examples of exponential family distributions

Exponential families include many of the most common distributions. Among many others, exponential families includes the following:

A number of common distributions are exponential families, but only when certain parameters are fixed and known. For example:

Notice that in each case, the parameters which must be fixed determine a limit on the size of observation values.

Examples of common distributions that are not exponential families are Student's t, most mixture distributions, and even the family of uniform distributions when the bounds are not fixed. See the section below on examples for more discussion.

### Scalar parameter

A single-parameter exponential family is a set of probability distributions whose probability density function (or probability mass function, for the case of a discrete distribution) can be expressed in the form

${\displaystyle f_{X}(x\mid \theta )=h(x)\,\exp \!{\bigl [}\,\eta (\theta )\cdot T(x)-A(\theta )\,{\bigr ]}}$

where T(x), h(x), η(θ), and A(θ) are known functions. The function h(x) must of course be non-negative.

An alternative, equivalent form often given is

${\displaystyle f_{X}(x\mid \theta )=h(x)\,g(\theta )\,\exp \!{\bigl [}\,\eta (\theta )\cdot T(x)\,{\bigr ]}}$

or equivalently

${\displaystyle f_{X}(x\mid \theta )=\exp \!{\bigl [}\,\eta (\theta )\cdot T(x)-A(\theta )+B(x)\,{\bigr ]}}$

The value θ is called the parameter of the family.

In addition, the support of ${\displaystyle f_{X}\!\left(x\mid \theta \right)}$ (i.e. the set of all ${\displaystyle x}$ for which ${\displaystyle f_{X}\!\left(x\mid \theta \right)}$ is greater than 0) does not depend on ${\displaystyle \theta }$. [6] This can be used to exclude a parametric family distribution from being an exponential family. For example, the Pareto distribution has a pdf which is defined for ${\displaystyle x\geq x_{m}}$ (${\displaystyle x_{m}}$ being the scale parameter) and its support, therefore, has a lower limit of ${\displaystyle x_{m}}$. Since the support of ${\displaystyle f_{\alpha ,x_{m}}\!(x)}$ is dependent on the value of the parameter, the family of Pareto distributions does not form an exponential family of distributions (at least when ${\displaystyle x_{m}}$ is unknown).

Often x is a vector of measurements, in which case T(x) may be a function from the space of possible values of x to the real numbers. More generally, η(θ) and T(x) can each be vector-valued such that ${\displaystyle \eta '(\theta )\cdot T(x)}$ is real-valued. However, see the discussion below on vector parameters, regarding the curved exponential family.

If η(θ) = θ, then the exponential family is said to be in canonical form . By defining a transformed parameter η = η(θ), it is always possible to convert an exponential family to canonical form. The canonical form is non-unique, since η(θ) can be multiplied by any nonzero constant, provided that T(x) is multiplied by that constant's reciprocal, or a constant c can be added to η(θ) and h(x) multiplied by ${\displaystyle \exp \!{\bigl [}-c\cdot T(x)\,{\bigr ]}}$ to offset it. In the special case that η(θ) = θ and T(x) = x then the family is called a natural exponential family.

Even when x is a scalar, and there is only a single parameter, the functions η(θ) and T(x) can still be vectors, as described below.

The function A(θ), or equivalently g(θ), is automatically determined once the other functions have been chosen, since it must assume a form that causes the distribution to be normalized (sum or integrate to one over the entire domain). Furthermore, both of these functions can always be written as functions of η, even when η(θ) is not a one-to-one function, i.e. two or more different values of θ map to the same value of η(θ), and hence η(θ) cannot be inverted. In such a case, all values of θ mapping to the same η(θ) will also have the same value for A(θ) and g(θ).

### Factorization of the variables involved

What is important to note, and what characterizes all exponential family variants, is that the parameter(s) and the observation variable(s) must factorize (can be separated into products each of which involves only one type of variable), either directly or within either part (the base or exponent) of an exponentiation operation. Generally, this means that all of the factors constituting the density or mass function must be of one of the following forms:

${\displaystyle f(x),g(\theta ),c^{f(x)},c^{g(\theta )},{[f(x)]}^{c},{[g(\theta )]}^{c},{[f(x)]}^{g(\theta )},{[g(\theta )]}^{f(x)},{[f(x)]}^{h(x)g(\theta )},{\text{ or }}{[g(\theta )]}^{h(x)j(\theta )},}$

where f and h are arbitrary functions of x; g and j are arbitrary functions of θ; and c is an arbitrary "constant" expression (i.e. an expression not involving x or θ).

There are further restrictions on how many such factors can occur. For example, the two expressions:

${\displaystyle {[f(x)g(\theta )]}^{h(x)j(\theta )},\qquad {[f(x)]}^{h(x)j(\theta )}[g(\theta )]^{h(x)j(\theta )},}$

are the same, i.e. a product of two "allowed" factors. However, when rewritten into the factorized form,

${\displaystyle {[f(x)g(\theta )]}^{h(x)j(\theta )}={[f(x)]}^{h(x)j(\theta )}[g(\theta )]^{h(x)j(\theta )}=e^{[h(x)\log f(x)]j(\theta )+h(x)[j(\theta )\log g(\theta )]},}$

it can be seen that it cannot be expressed in the required form. (However, a form of this sort is a member of a curved exponential family, which allows multiple factorized terms in the exponent.[ citation needed ])

To see why an expression of the form

${\displaystyle {[f(x)]}^{g(\theta )}}$

qualifies,

${\displaystyle {[f(x)]}^{g(\theta )}=e^{g(\theta )\log f(x)}}$

and hence factorizes inside of the exponent. Similarly,

${\displaystyle {[f(x)]}^{h(x)g(\theta )}=e^{h(x)g(\theta )\log f(x)}=e^{[h(x)\log f(x)]g(\theta )}}$

and again factorizes inside of the exponent.

A factor consisting of a sum where both types of variables are involved (e.g. a factor of the form ${\displaystyle 1+f(x)g(\theta )}$) cannot be factorized in this fashion (except in some cases where occurring directly in an exponent); this is why, for example, the Cauchy distribution and Student's t distribution are not exponential families.

### Vector parameter

The definition in terms of one real-number parameter can be extended to one real-vector parameter

${\displaystyle {\boldsymbol {\theta }}\equiv \left[\,\theta _{1},\,\theta _{2},\,\ldots ,\,\theta _{s}\,\right]^{\mathsf {T}}~.}$

A family of distributions is said to belong to a vector exponential family if the probability density function (or probability mass function, for discrete distributions) can be written as

${\displaystyle f_{X}(x\mid {\boldsymbol {\theta }})=h(x)\,\exp \left(\sum _{i=1}^{s}\eta _{i}({\boldsymbol {\theta }})T_{i}(x)-A({\boldsymbol {\theta }})\right)~,}$

or in a more compact form,

${\displaystyle f_{X}(x\mid {\boldsymbol {\theta }})=h(x)\,\exp {\Big (}{\boldsymbol {\eta }}({\boldsymbol {\theta }})\cdot \mathbf {T} (x)-A({\boldsymbol {\theta }}){\Big )}}$

This form writes the sum as a dot product of vector-valued functions ${\displaystyle {\boldsymbol {\eta }}({\boldsymbol {\theta }})}$ and ${\displaystyle \mathbf {T} (x)\,}$.

An alternative, equivalent form often seen is

${\displaystyle f_{X}(x\mid {\boldsymbol {\theta }})=h(x)\,g({\boldsymbol {\theta }})\,\exp {\Big (}{\boldsymbol {\eta }}({\boldsymbol {\theta }})\cdot \mathbf {T} (x){\Big )}}$

As in the scalar valued case, the exponential family is said to be in canonical form if

${\displaystyle \quad \eta _{i}({\boldsymbol {\theta }})=\theta _{i}\quad \forall i\,.}$

A vector exponential family is said to be curved if the dimension of

${\displaystyle {\boldsymbol {\theta }}\equiv \left[\,\theta _{1},\,\theta _{2},\,\ldots ,\,\theta _{d}\,\,\right]^{\mathsf {T}}}$

is less than the dimension of the vector

${\displaystyle {\boldsymbol {\eta }}({\boldsymbol {\theta }})\equiv \left[\,\eta _{1}({\boldsymbol {\theta }}),\,\eta _{2}({\boldsymbol {\theta }}),\,\ldots ,\,\eta _{s}({\boldsymbol {\theta }})\,\right]^{\mathsf {T}}~.}$

That is, if the dimension, d, of the parameter vector is less than the number of functions, s, of the parameter vector in the above representation of the probability density function. Most common distributions in the exponential family are not curved, and many algorithms designed to work with any exponential family implicitly or explicitly assume that the distribution is not curved.

As in the above case of a scalar-valued parameter, the function ${\displaystyle A({\boldsymbol {\theta }})}$ or equivalently ${\displaystyle g({\boldsymbol {\theta }})}$ is automatically determined once the other functions have been chosen, so that the entire distribution is normalized. In addition, as above, both of these functions can always be written as functions of ${\displaystyle {\boldsymbol {\eta }}}$, regardless of the form of the transformation that generates ${\displaystyle {\boldsymbol {\eta }}}$ from ${\displaystyle {\boldsymbol {\theta }}\,}$. Hence an exponential family in its "natural form" (parametrized by its natural parameter) looks like

${\displaystyle f_{X}(x\mid {\boldsymbol {\eta }})=h(x)\,\exp {\Big (}{\boldsymbol {\eta }}\cdot \mathbf {T} (x)-A({\boldsymbol {\eta }}){\Big )}}$

or equivalently

${\displaystyle f_{X}(x\mid {\boldsymbol {\eta }})=h(x)\,g({\boldsymbol {\eta }})\,\exp {\Big (}{\boldsymbol {\eta }}\cdot \mathbf {T} (x){\Big )}}$

The above forms may sometimes be seen with ${\displaystyle {\boldsymbol {\eta }}^{\mathsf {T}}\mathbf {T} (x)}$ in place of ${\displaystyle {\boldsymbol {\eta }}\cdot \mathbf {T} (x)\,}$. These are exactly equivalent formulations, merely using different notation for the dot product.

### Vector parameter, vector variable

The vector-parameter form over a single scalar-valued random variable can be trivially expanded to cover a joint distribution over a vector of random variables. The resulting distribution is simply the same as the above distribution for a scalar-valued random variable with each occurrence of the scalar x replaced by the vector

${\displaystyle \mathbf {x} =\left(x_{1},x_{2},\cdots ,x_{k}\right)^{\mathsf {T}}~.}$

The dimensions k of the random variable need not match the dimension d of the parameter vector, nor (in the case of a curved exponential function) the dimension s of the natural parameter ${\displaystyle {\boldsymbol {\eta }}}$ and sufficient statistic T(x) .

The distribution in this case is written as

${\displaystyle f_{X}\!\left(\mathbf {x} \mid {\boldsymbol {\theta }}\right)=h(\mathbf {x} )\,\exp \!\left(\,\sum _{i=1}^{s}\eta _{i}({\boldsymbol {\theta }})T_{i}(\mathbf {x} )-A({\boldsymbol {\theta }})\,\right)}$

Or more compactly as

${\displaystyle f_{X}\!\left(\,\mathbf {x} \mid {\boldsymbol {\theta }}\,\right)=h(\mathbf {x} )\,\exp \!{\Big (}\,{\boldsymbol {\eta }}({\boldsymbol {\theta }})\cdot \mathbf {T} (\mathbf {x} )-A({\boldsymbol {\theta }})\,{\Big )}}$

Or alternatively as

${\displaystyle f_{X}\!\left(\,\mathbf {x} \mid {\boldsymbol {\theta }}\,\right)=g({\boldsymbol {\theta }})\;h(\mathbf {x} )\,\exp \!{\Big (}\,{\boldsymbol {\eta }}({\boldsymbol {\theta }})\cdot \mathbf {T} (\mathbf {x} )\,{\Big )}}$

### Measure-theoretic formulation

We use cumulative distribution functions (CDF) in order to encompass both discrete and continuous distributions.

Suppose H is a non-decreasing function of a real variable. Then Lebesgue–Stieltjes integrals with respect to ${\displaystyle {\rm {d\,}}H(\mathbf {x} )}$ are integrals with respect to the reference measure of the exponential family generated by H .

Any member of that exponential family has cumulative distribution function

${\displaystyle {\rm {d\,}}F\left(\,\mathbf {x} \mid {\boldsymbol {\theta }}\,\right)=\exp {\bigl (}\,{\boldsymbol {\eta }}(\theta )\cdot \mathbf {T} (\mathbf {x} )\,-\,A({\boldsymbol {\theta }})\,{\bigr )}~{\rm {d\,}}H(\mathbf {x} )~.}$

H(x) is a Lebesgue–Stieltjes integrator for the reference measure. When the reference measure is finite, it can be normalized and H is actually the cumulative distribution function of a probability distribution. If F is absolutely continuous with a density ${\displaystyle f(x)}$ with respect to a reference measure ${\displaystyle \,{\rm {d\,}}x\,}$ (typically Lebesgue measure), one can write ${\displaystyle \,{\rm {d\,}}F(x)=f(x)~{\rm {d\,}}x\,}$. In this case, H is also absolutely continuous and can be written ${\displaystyle \,{\rm {d\,}}H(x)=h(x)\,{\rm {d\,}}x\,}$ so the formulas reduce to that of the previous paragraphs. If F is discrete, then H is a step function (with steps on the support of F).

Alternatively, we can write the probability measure directly as

${\displaystyle P\left(\,{\rm {d\,}}\mathbf {x} \mid {\boldsymbol {\theta }}\,\right)=\exp {\bigl (}\,{\boldsymbol {\eta }}(\theta )\cdot \mathbf {T} (\mathbf {x} )-A({\boldsymbol {\theta }})\,{\bigr )}~\mu ({\rm {d\,}}\mathbf {x} )~.}$

for some reference measure ${\displaystyle \mu \,}$.

## Interpretation

In the definitions above, the functions T(x), η(θ), and A(η) were apparently arbitrarily defined. However, these functions play a significant role in the resulting probability distribution.

• T(x) is a sufficient statistic of the distribution. For exponential families, the sufficient statistic is a function of the data that holds all information the data x provides with regard to the unknown parameter values. This means that, for any data sets ${\displaystyle x}$ and ${\displaystyle y}$, the likelihood ratio is the same, that is ${\displaystyle {\frac {f(x;\theta _{1})}{f(x;\theta _{2})}}={\frac {f(y;\theta _{1})}{f(y;\theta _{2})}}}$ if  T(x) = T(y) . This is true even if x and y are quite distinct – that is, even if ${\displaystyle d(x,y)>0\,}$. The dimension of T(x) equals the number of parameters of θ and encompasses all of the information regarding the data related to the parameter θ. The sufficient statistic of a set of independent identically distributed data observations is simply the sum of individual sufficient statistics, and encapsulates all the information needed to describe the posterior distribution of the parameters, given the data (and hence to derive any desired estimate of the parameters). (This important property is discussed further below.)
• η is called the natural parameter. The set of values of η for which the function ${\displaystyle f_{X}(x;\eta )}$ is integrable is called the natural parameter space. It can be shown that the natural parameter space is always convex.
• A(η) is called the log-partition function [lower-alpha 2] because it is the logarithm of a normalization factor, without which ${\displaystyle f_{X}(x;\theta )}$ would not be a probability distribution:
${\displaystyle A(\eta )=\log \left(\int _{X}h(x)\,\exp(\eta (\theta )\cdot T(x))\,\mathrm {d} x\right)}$

The function A important in its own right, because the mean, variance and other moments of the sufficient statistic T(x) can be derived simply by differentiating A(η). For example, because log(x) is one of the components of the sufficient statistic of the gamma distribution, ${\displaystyle \operatorname {\mathcal {E}} [\log x]}$ can be easily determined for this distribution using A(η). Technically, this is true because

${\displaystyle K\left(u\mid \eta \right)=A(\eta +u)-A(\eta )\,,}$

is the cumulant generating function of the sufficient statistic.

## Properties

Exponential families have a large number of properties that make them extremely useful for statistical analysis. In many cases, it can be shown that only exponential families have these properties. Examples:

Given an exponential family defined by ${\displaystyle f_{X}(x\mid \theta )=h(x)\,\exp \!{\bigl [}\,\theta \cdot T(x)-A(\theta )\,{\bigr ]}}$, where ${\displaystyle \Theta }$ is the parameter space, such that ${\displaystyle \theta \in \Theta \subset \mathbb {R} ^{k}}$. Then

• If ${\displaystyle \Theta }$ has nonempty interior in ${\displaystyle \mathbb {R} ^{k}}$, then given any IID samples ${\displaystyle X_{1},...,X_{n}\sim f_{X}}$, the statistic ${\displaystyle T(X_{1},...,X_{n}):=\sum _{i=1}^{n}T(X_{i})}$ is a complete statistic for ${\displaystyle \theta }$. [8] [9]
• ${\displaystyle T}$ is a minimal statistic for ${\displaystyle \theta }$ iff for all ${\displaystyle \theta _{1},\theta _{2}\in \Theta }$, and ${\displaystyle x_{1},x_{2}}$ in the support of ${\displaystyle X}$, if ${\displaystyle (\theta _{1}-\theta _{2})\cdot (T(x_{1})-T(x_{2}))=0}$, then ${\displaystyle \theta _{1}=\theta _{2}}$ or ${\displaystyle x_{1}=x_{2}}$. [10]

## Examples

It is critical, when considering the examples in this section, to remember the discussion above about what it means to say that a "distribution" is an exponential family, and in particular to keep in mind that the set of parameters that are allowed to vary is critical in determining whether a "distribution" is or is not an exponential family.

The normal, exponential, log-normal, gamma, chi-squared, beta, Dirichlet, Bernoulli, categorical, Poisson, geometric, inverse Gaussian, von Mises and von Mises-Fisher distributions are all exponential families.

Some distributions are exponential families only if some of their parameters are held fixed. The family of Pareto distributions with a fixed minimum bound xm form an exponential family. The families of binomial and multinomial distributions with fixed number of trials n but unknown probability parameter(s) are exponential families. The family of negative binomial distributions with fixed number of failures (a.k.a. stopping-time parameter) r is an exponential family. However, when any of the above-mentioned fixed parameters are allowed to vary, the resulting family is not an exponential family.

As mentioned above, as a general rule, the support of an exponential family must remain the same across all parameter settings in the family. This is why the above cases (e.g. binomial with varying number of trials, Pareto with varying minimum bound) are not exponential families — in all of the cases, the parameter in question affects the support (particularly, changing the minimum or maximum possible value). For similar reasons, neither the discrete uniform distribution nor continuous uniform distribution are exponential families as one or both bounds vary.

The Weibull distribution with fixed shape parameter k is an exponential family. Unlike in the previous examples, the shape parameter does not affect the support; the fact that allowing it to vary makes the Weibull non-exponential is due rather to the particular form of the Weibull's probability density function (k appears in the exponent of an exponent).

In general, distributions that result from a finite or infinite mixture of other distributions, e.g. mixture model densities and compound probability distributions, are not exponential families. Examples are typical Gaussian mixture models as well as many heavy-tailed distributions that result from compounding (i.e. infinitely mixing) a distribution with a prior distribution over one of its parameters, e.g. the Student's t-distribution (compounding a normal distribution over a gamma-distributed precision prior), and the beta-binomial and Dirichlet-multinomial distributions. Other examples of distributions that are not exponential families are the F-distribution, Cauchy distribution, hypergeometric distribution and logistic distribution.

Following are some detailed examples of the representation of some useful distribution as exponential families.

### Normal distribution: unknown mean, known variance

As a first example, consider a random variable distributed normally with unknown mean μ and known variance σ2. The probability density function is then

${\displaystyle f_{\sigma }(x;\mu )={\frac {1}{\sqrt {2\pi \sigma ^{2}}}}e^{-(x-\mu )^{2}/(2\sigma ^{2})}.}$

This is a single-parameter exponential family, as can be seen by setting

{\displaystyle {\begin{aligned}h_{\sigma }(x)&={\frac {1}{\sqrt {2\pi \sigma ^{2}}}}e^{-x^{2}/(2\sigma ^{2})}\\[4pt]T_{\sigma }(x)&={\frac {x}{\sigma }}\\[4pt]A_{\sigma }(\mu )&={\frac {\mu ^{2}}{2\sigma ^{2}}}\\[4pt]\eta _{\sigma }(\mu )&={\frac {\mu }{\sigma }}.\end{aligned}}}

If σ = 1 this is in canonical form, as then η(μ) = μ.

### Normal distribution: unknown mean and unknown variance

Next, consider the case of a normal distribution with unknown mean and unknown variance. The probability density function is then

${\displaystyle f(y;\mu ,\sigma )={\frac {1}{\sqrt {2\pi \sigma ^{2}}}}e^{-(y-\mu )^{2}/2\sigma ^{2}}.}$

This is an exponential family which can be written in canonical form by defining

{\displaystyle {\begin{aligned}{\boldsymbol {\eta }}&=\left[\,{\frac {\mu }{\sigma ^{2}}},~-{\frac {1}{2\sigma ^{2}}}\,\right]\\h(y)&={\frac {1}{\sqrt {2\pi }}}\\T(y)&=\left(y,y^{2}\right)^{\rm {T}}\\A({\boldsymbol {\eta }})&={\frac {\mu ^{2}}{2\sigma ^{2}}}+\log |\sigma |=-{\frac {\eta _{1}^{2}}{4\eta _{2}}}+{\frac {1}{2}}\log \left|{\frac {1}{2\eta _{2}}}\right|\end{aligned}}}

### Binomial distribution

As an example of a discrete exponential family, consider the binomial distribution with known number of trials n. The probability mass function for this distribution is

${\displaystyle f(x)={n \choose x}p^{x}(1-p)^{n-x},\quad x\in \{0,1,2,\ldots ,n\}.}$

This can equivalently be written as

${\displaystyle f(x)={n \choose x}\exp \left(x\log \left({\frac {p}{1-p}}\right)+n\log(1-p)\right),}$

which shows that the binomial distribution is an exponential family, whose natural parameter is

${\displaystyle \eta =\log {\frac {p}{1-p}}.}$

This function of p is known as logit.

## Table of distributions

The following table shows how to rewrite a number of common distributions as exponential-family distributions with natural parameters. Refer to the flashcards [11] for main exponential families.

For a scalar variable and scalar parameter, the form is as follows:

${\displaystyle f_{X}(x\mid \theta )=h(x)\exp {\Big (}\eta ({\theta })T(x)-A({\eta }){\Big )}}$

For a scalar variable and vector parameter:

${\displaystyle f_{X}(x\mid {\boldsymbol {\theta }})=h(x)\exp {\Big (}{\boldsymbol {\eta }}({\boldsymbol {\theta }})\cdot \mathbf {T} (x)-A({\boldsymbol {\eta }}){\Big )}}$
${\displaystyle f_{X}(x\mid {\boldsymbol {\theta }})=h(x)g({\boldsymbol {\theta }})\exp {\Big (}{\boldsymbol {\eta }}({\boldsymbol {\theta }})\cdot \mathbf {T} (x){\Big )}}$

For a vector variable and vector parameter:

${\displaystyle f_{X}(\mathbf {x} \mid {\boldsymbol {\theta }})=h(\mathbf {x} )\exp {\Big (}{\boldsymbol {\eta }}({\boldsymbol {\theta }})\cdot \mathbf {T} (\mathbf {x} )-A({\boldsymbol {\eta }}){\Big )}}$

The above formulas choose the functional form of the exponential-family with a log-partition function ${\displaystyle A({\boldsymbol {\eta }})}$. The reason for this is so that the moments of the sufficient statistics can be calculated easily, simply by differentiating this function. Alternative forms involve either parameterizing this function in terms of the normal parameter ${\displaystyle {\boldsymbol {\theta }}}$ instead of the natural parameter, and/or using a factor ${\displaystyle g({\boldsymbol {\eta }})}$ outside of the exponential. The relation between the latter and the former is:

${\displaystyle A({\boldsymbol {\eta }})=-\log g({\boldsymbol {\eta }})}$
${\displaystyle g({\boldsymbol {\eta }})=e^{-A({\boldsymbol {\eta }})}}$

To convert between the representations involving the two types of parameter, use the formulas below for writing one type of parameter in terms of the other.

DistributionParameter(s) ${\displaystyle {\boldsymbol {\theta }}}$Natural parameter(s) ${\displaystyle {\boldsymbol {\eta }}}$Inverse parameter mappingBase measure ${\displaystyle h(x)}$Sufficient statistic ${\displaystyle T(x)}$Log-partition ${\displaystyle A({\boldsymbol {\eta }})}$Log-partition ${\displaystyle A({\boldsymbol {\theta }})}$
Bernoulli distribution ${\displaystyle p}$${\displaystyle \log {\frac {p}{1-p}}}$
${\displaystyle {\frac {1}{1+e^{-\eta }}}={\frac {e^{\eta }}{1+e^{\eta }}}}$
${\displaystyle 1}$${\displaystyle x}$${\displaystyle \log(1+e^{\eta })}$${\displaystyle -\log(1-p)}$
binomial distribution
with known number of trials ${\displaystyle n}$
${\displaystyle p}$${\displaystyle \log {\frac {p}{1-p}}}$${\displaystyle {\frac {1}{1+e^{-\eta }}}={\frac {e^{\eta }}{1+e^{\eta }}}}$${\displaystyle {n \choose x}}$${\displaystyle x}$${\displaystyle n\log(1+e^{\eta })}$${\displaystyle -n\log(1-p)}$
Poisson distribution ${\displaystyle \lambda }$${\displaystyle \log \lambda }$${\displaystyle e^{\eta }}$${\displaystyle {\frac {1}{x!}}}$${\displaystyle x}$${\displaystyle e^{\eta }}$${\displaystyle \lambda }$
negative binomial distribution
with known number of failures ${\displaystyle r}$
${\displaystyle p}$${\displaystyle \log p}$${\displaystyle e^{\eta }}$${\displaystyle {x+r-1 \choose x}}$${\displaystyle x}$${\displaystyle -r\log(1-e^{\eta })}$${\displaystyle -r\log(1-p)}$
exponential distribution ${\displaystyle \lambda }$${\displaystyle -\lambda }$${\displaystyle -\eta }$${\displaystyle 1}$${\displaystyle x}$${\displaystyle -\log(-\eta )}$${\displaystyle -\log \lambda }$
Pareto distribution
with known minimum value ${\displaystyle x_{m}}$
${\displaystyle \alpha }$${\displaystyle -\alpha -1}$${\displaystyle -1-\eta }$${\displaystyle 1}$${\displaystyle \log x}$${\displaystyle -\log(-1-\eta )+(1+\eta )\log x_{\mathrm {m} }}$${\displaystyle -\log \alpha -\alpha \log x_{\mathrm {m} }}$
Weibull distribution
with known shape k
${\displaystyle \lambda }$${\displaystyle -{\frac {1}{\lambda ^{k}}}}$${\displaystyle (-\eta )^{-{\frac {1}{k}}}}$${\displaystyle x^{k-1}}$${\displaystyle x^{k}}$${\displaystyle -\log(-\eta )-\log k}$${\displaystyle k\log \lambda -\log k}$
Laplace distribution
with known mean ${\displaystyle \mu }$
${\displaystyle b}$${\displaystyle -{\frac {1}{b}}}$${\displaystyle -{\frac {1}{\eta }}}$${\displaystyle 1}$${\displaystyle |x-\mu |}$${\displaystyle \log \left(-{\frac {2}{\eta }}\right)}$${\displaystyle \log 2b}$
chi-squared distribution ${\displaystyle \nu }$${\displaystyle {\frac {\nu }{2}}-1}$${\displaystyle 2(\eta +1)}$${\displaystyle e^{-{\frac {x}{2}}}}$${\displaystyle \log x}$${\displaystyle \log \Gamma (\eta +1)+(\eta +1)\log 2}$${\displaystyle \log \Gamma \left({\frac {\nu }{2}}\right)+{\frac {\nu }{2}}\log 2}$
normal distribution
known variance
${\displaystyle \mu }$${\displaystyle {\frac {\mu }{\sigma }}}$${\displaystyle \sigma \eta }$${\displaystyle {\frac {e^{-{\frac {x^{2}}{2\sigma ^{2}}}}}{{\sqrt {2\pi }}\sigma }}}$${\displaystyle {\frac {x}{\sigma }}}$${\displaystyle {\frac {\eta ^{2}}{2}}}$${\displaystyle {\frac {\mu ^{2}}{2\sigma ^{2}}}}$
continuous Bernoulli distribution ${\displaystyle \lambda }$${\displaystyle \log {\frac {\lambda }{1-\lambda }}}$${\displaystyle {\frac {e^{\eta }}{1+e^{\eta }}}}$${\displaystyle 1}$${\displaystyle x}$${\displaystyle \log {\frac {e^{\eta }-1}{\eta }}}$${\displaystyle \log \left({\frac {1-2\lambda }{(1-\lambda )\log \left({\frac {1-\lambda }{\lambda }}\right)}}\right)}$
normal distribution ${\displaystyle \mu ,\ \sigma ^{2}}$${\displaystyle {\begin{bmatrix}{\dfrac {\mu }{\sigma ^{2}}}\\[10pt]-{\dfrac {1}{2\sigma ^{2}}}\end{bmatrix}}}$${\displaystyle {\begin{bmatrix}-{\dfrac {\eta _{1}}{2\eta _{2}}}\\[15pt]-{\dfrac {1}{2\eta _{2}}}\end{bmatrix}}}$${\displaystyle {\frac {1}{\sqrt {2\pi }}}}$${\displaystyle {\begin{bmatrix}x\\x^{2}\end{bmatrix}}}$${\displaystyle -{\frac {\eta _{1}^{2}}{4\eta _{2}}}-{\frac {1}{2}}\log(-2\eta _{2})}$${\displaystyle {\frac {\mu ^{2}}{2\sigma ^{2}}}+\log \sigma }$
log-normal distribution ${\displaystyle \mu ,\ \sigma ^{2}}$${\displaystyle {\begin{bmatrix}{\dfrac {\mu }{\sigma ^{2}}}\\[10pt]-{\dfrac {1}{2\sigma ^{2}}}\end{bmatrix}}}$${\displaystyle {\begin{bmatrix}-{\dfrac {\eta _{1}}{2\eta _{2}}}\\[15pt]-{\dfrac {1}{2\eta _{2}}}\end{bmatrix}}}$${\displaystyle {\frac {1}{{\sqrt {2\pi }}x}}}$${\displaystyle {\begin{bmatrix}\log x\\(\log x)^{2}\end{bmatrix}}}$${\displaystyle -{\frac {\eta _{1}^{2}}{4\eta _{2}}}-{\frac {1}{2}}\log(-2\eta _{2})}$${\displaystyle {\frac {\mu ^{2}}{2\sigma ^{2}}}+\log \sigma }$
inverse Gaussian distribution ${\displaystyle \mu ,\ \lambda }$${\displaystyle {\begin{bmatrix}-{\dfrac {\lambda }{2\mu ^{2}}}\\[15pt]-{\dfrac {\lambda }{2}}\end{bmatrix}}}$${\displaystyle {\begin{bmatrix}{\sqrt {\dfrac {\eta _{2}}{\eta _{1}}}}\\[15pt]-2\eta _{2}\end{bmatrix}}}$${\displaystyle {\frac {1}{{\sqrt {2\pi }}x^{\frac {3}{2}}}}}$${\displaystyle {\begin{bmatrix}x\\[5pt]{\dfrac {1}{x}}\end{bmatrix}}}$${\displaystyle 2{\sqrt {\eta _{1}\eta _{2}}}-{\frac {1}{2}}\log(-2\eta _{2})}$${\displaystyle -{\frac {\lambda }{\mu }}-{\frac {1}{2}}\log \lambda }$
gamma distribution ${\displaystyle \alpha ,\ \beta }$${\displaystyle {\begin{bmatrix}\alpha -1\\-\beta \end{bmatrix}}}$${\displaystyle {\begin{bmatrix}\eta _{1}+1\\-\eta _{2}\end{bmatrix}}}$${\displaystyle 1}$${\displaystyle {\begin{bmatrix}\log x\\x\end{bmatrix}}}$${\displaystyle \log \Gamma (\eta _{1}+1)-(\eta _{1}+1)\log(-\eta _{2})}$${\displaystyle \log \Gamma (\alpha )-\alpha \log \beta }$
${\displaystyle k,\ \theta }$${\displaystyle {\begin{bmatrix}k-1\\[5pt]-{\dfrac {1}{\theta }}\end{bmatrix}}}$${\displaystyle {\begin{bmatrix}\eta _{1}+1\\[5pt]-{\dfrac {1}{\eta _{2}}}\end{bmatrix}}}$${\displaystyle \log \Gamma (k)+k\log \theta }$
inverse gamma distribution ${\displaystyle \alpha ,\ \beta }$${\displaystyle {\begin{bmatrix}-\alpha -1\\-\beta \end{bmatrix}}}$${\displaystyle {\begin{bmatrix}-\eta _{1}-1\\-\eta _{2}\end{bmatrix}}}$${\displaystyle 1}$${\displaystyle {\begin{bmatrix}\log x\\{\frac {1}{x}}\end{bmatrix}}}$${\displaystyle \log \Gamma (-\eta _{1}-1)-(-\eta _{1}-1)\log(-\eta _{2})}$${\displaystyle \log \Gamma (\alpha )-\alpha \log \beta }$
generalized inverse Gaussian distribution ${\displaystyle p,\ a,\ b}$${\displaystyle {\begin{bmatrix}p-1\\-a/2\\-b/2\end{bmatrix}}}$${\displaystyle {\begin{bmatrix}\eta _{1}+1\\-2\eta _{2}\\-2\eta _{3}\end{bmatrix}}}$${\displaystyle 1}$${\displaystyle {\begin{bmatrix}\log x\\x\\{\frac {1}{x}}\end{bmatrix}}}$${\displaystyle \log 2K_{\eta _{1}+1}({\sqrt {4\eta _{2}\eta _{3}}})-{\frac {\eta _{1}+1}{2}}\log {\frac {\eta _{2}}{\eta _{3}}}}$${\displaystyle \log 2K_{p}({\sqrt {ab}})-{\frac {p}{2}}\log {\frac {a}{b}}}$
scaled inverse chi-squared distribution ${\displaystyle \nu ,\ \sigma ^{2}}$${\displaystyle {\begin{bmatrix}-{\dfrac {\nu }{2}}-1\\[10pt]-{\dfrac {\nu \sigma ^{2}}{2}}\end{bmatrix}}}$${\displaystyle {\begin{bmatrix}-2(\eta _{1}+1)\\[10pt]{\dfrac {\eta _{2}}{\eta _{1}+1}}\end{bmatrix}}}$${\displaystyle 1}$${\displaystyle {\begin{bmatrix}\log x\\{\frac {1}{x}}\end{bmatrix}}}$${\displaystyle \log \Gamma (-\eta _{1}-1)-(-\eta _{1}-1)\log(-\eta _{2})}$${\displaystyle \log \Gamma \left({\frac {\nu }{2}}\right)-{\frac {\nu }{2}}\log {\frac {\nu \sigma ^{2}}{2}}}$
beta distribution

(variant 1)
${\displaystyle \alpha ,\ \beta }$${\displaystyle {\begin{bmatrix}\alpha \\\beta \end{bmatrix}}}$${\displaystyle {\begin{bmatrix}\eta _{1}\\\eta _{2}\end{bmatrix}}}$${\displaystyle {\frac {1}{x(1-x)}}}$${\displaystyle {\begin{bmatrix}\log x\\\log(1-x)\end{bmatrix}}}$${\displaystyle \log \Gamma (\eta _{1})+\log \Gamma (\eta _{2})-\log \Gamma (\eta _{1}+\eta _{2})}$${\displaystyle \log \Gamma (\alpha )+\log \Gamma (\beta )-\log \Gamma (\alpha +\beta )}$
beta distribution

(variant 2)
${\displaystyle \alpha ,\ \beta }$${\displaystyle {\begin{bmatrix}\alpha -1\\\beta -1\end{bmatrix}}}$${\displaystyle {\begin{bmatrix}\eta _{1}+1\\\eta _{2}+1\end{bmatrix}}}$${\displaystyle 1}$${\displaystyle {\begin{bmatrix}\log x\\\log(1-x)\end{bmatrix}}}$${\displaystyle \log \Gamma (\eta _{1}+1)+\log \Gamma (\eta _{2}+1)-\log \Gamma (\eta _{1}+\eta _{2}+2)}$${\displaystyle \log \Gamma (\alpha )+\log \Gamma (\beta )-\log \Gamma (\alpha +\beta )}$
multivariate normal distribution ${\displaystyle {\boldsymbol {\mu }},\ {\boldsymbol {\Sigma }}}$${\displaystyle {\begin{bmatrix}{\boldsymbol {\Sigma }}^{-1}{\boldsymbol {\mu }}\\[5pt]-{\frac {1}{2}}{\boldsymbol {\Sigma }}^{-1}\end{bmatrix}}}$${\displaystyle {\begin{bmatrix}-{\frac {1}{2}}{\boldsymbol {\eta }}_{2}^{-1}{\boldsymbol {\eta }}_{1}\\[5pt]-{\frac {1}{2}}{\boldsymbol {\eta }}_{2}^{-1}\end{bmatrix}}}$${\displaystyle (2\pi )^{-{\frac {k}{2}}}}$${\displaystyle {\begin{bmatrix}\mathbf {x} \\[5pt]\mathbf {x} \mathbf {x} ^{\mathsf {T}}\end{bmatrix}}}$${\displaystyle -{\frac {1}{4}}{\boldsymbol {\eta }}_{1}^{\mathsf {T}}{\boldsymbol {\eta }}_{2}^{-1}{\boldsymbol {\eta }}_{1}-{\frac {1}{2}}\log \left|-2{\boldsymbol {\eta }}_{2}\right|}$${\displaystyle {\frac {1}{2}}{\boldsymbol {\mu }}^{\mathsf {T}}{\boldsymbol {\Sigma }}^{-1}{\boldsymbol {\mu }}+{\frac {1}{2}}\log |{\boldsymbol {\Sigma }}|}$
categorical distribution

(variant 1)
${\displaystyle p_{1},\ \ldots ,\,p_{k}}$

where ${\displaystyle \textstyle \sum _{i=1}^{k}p_{i}=1}$
${\displaystyle {\begin{bmatrix}\log p_{1}\\\vdots \\\log p_{k}\end{bmatrix}}}$${\displaystyle {\begin{bmatrix}e^{\eta _{1}}\\\vdots \\e^{\eta _{k}}\end{bmatrix}}}$

where ${\displaystyle \textstyle \sum _{i=1}^{k}e^{\eta _{i}}=1}$
${\displaystyle 1}$${\displaystyle {\begin{bmatrix}[x=1]\\\vdots \\{[x=k]}\end{bmatrix}}}$
• ${\displaystyle [x=i]}$ is the Iverson bracket*
${\displaystyle 0}$${\displaystyle 0}$
categorical distribution

(variant 2)
${\displaystyle p_{1},\ \ldots ,\,p_{k}}$

where ${\displaystyle \textstyle \sum _{i=1}^{k}p_{i}=1}$
${\displaystyle {\begin{bmatrix}\log p_{1}+C\\\vdots \\\log p_{k}+C\end{bmatrix}}}$${\displaystyle {\begin{bmatrix}{\dfrac {1}{C}}e^{\eta _{1}}\\\vdots \\{\dfrac {1}{C}}e^{\eta _{k}}\end{bmatrix}}=}$

${\displaystyle {\begin{bmatrix}{\dfrac {e^{\eta _{1}}}{\sum _{i=1}^{k}e^{\eta _{i}}}}\\[10pt]\vdots \\[5pt]{\dfrac {e^{\eta _{k}}}{\sum _{i=1}^{k}e^{\eta _{i}}}}\end{bmatrix}}}$

where ${\displaystyle \textstyle \sum _{i=1}^{k}e^{\eta _{i}}=C}$

${\displaystyle 1}$${\displaystyle {\begin{bmatrix}[x=1]\\\vdots \\{[x=k]}\end{bmatrix}}}$
• ${\displaystyle [x=i]}$ is the Iverson bracket*
${\displaystyle 0}$${\displaystyle 0}$
categorical distribution

(variant 3)
${\displaystyle p_{1},\ \ldots ,\,p_{k}}$

where ${\displaystyle p_{k}=1-\textstyle \sum _{i=1}^{k-1}p_{i}}$
${\displaystyle {\begin{bmatrix}\log {\dfrac {p_{1}}{p_{k}}}\\[10pt]\vdots \\[5pt]\log {\dfrac {p_{k-1}}{p_{k}}}\\[15pt]0\end{bmatrix}}=}$

${\displaystyle {\begin{bmatrix}\log {\dfrac {p_{1}}{1-\sum _{i=1}^{k-1}p_{i}}}\\[10pt]\vdots \\[5pt]\log {\dfrac {p_{k-1}}{1-\sum _{i=1}^{k-1}p_{i}}}\\[15pt]0\end{bmatrix}}}$
${\displaystyle {\begin{bmatrix}{\dfrac {e^{\eta _{1}}}{\sum _{i=1}^{k}e^{\eta _{i}}}}\\[10pt]\vdots \\[5pt]{\dfrac {e^{\eta _{k}}}{\sum _{i=1}^{k}e^{\eta _{i}}}}\end{bmatrix}}=}$

${\displaystyle {\begin{bmatrix}{\dfrac {e^{\eta _{1}}}{1+\sum _{i=1}^{k-1}e^{\eta _{i}}}}\\[10pt]\vdots \\[5pt]{\dfrac {e^{\eta _{k-1}}}{1+\sum _{i=1}^{k-1}e^{\eta _{i}}}}\\[15pt]{\dfrac {1}{1+\sum _{i=1}^{k-1}e^{\eta _{i}}}}\end{bmatrix}}}$

${\displaystyle 1}$${\displaystyle {\begin{bmatrix}[x=1]\\\vdots \\{[x=k]}\end{bmatrix}}}$
• ${\displaystyle [x=i]}$ is the Iverson bracket*
${\displaystyle \log \left(\sum _{i=1}^{k}e^{\eta _{i}}\right)=\log \left(1+\sum _{i=1}^{k-1}e^{\eta _{i}}\right)}$${\displaystyle -\log p_{k}=-\log \left(1-\sum _{i=1}^{k-1}p_{i}\right)}$
multinomial distribution

(variant 1)
with known number of trials ${\displaystyle n}$
${\displaystyle p_{1},\ \ldots ,\,p_{k}}$

where ${\displaystyle \textstyle \sum _{i=1}^{k}p_{i}=1}$
${\displaystyle {\begin{bmatrix}\log p_{1}\\\vdots \\\log p_{k}\end{bmatrix}}}$${\displaystyle {\begin{bmatrix}e^{\eta _{1}}\\\vdots \\e^{\eta _{k}}\end{bmatrix}}}$

where ${\displaystyle \textstyle \sum _{i=1}^{k}e^{\eta _{i}}=1}$
${\displaystyle {\frac {n!}{\prod _{i=1}^{k}x_{i}!}}}$${\displaystyle {\begin{bmatrix}x_{1}\\\vdots \\x_{k}\end{bmatrix}}}$${\displaystyle 0}$${\displaystyle 0}$
multinomial distribution

(variant 2)
with known number of trials ${\displaystyle n}$
${\displaystyle p_{1},\ \ldots ,\,p_{k}}$

where ${\displaystyle \textstyle \sum _{i=1}^{k}p_{i}=1}$
${\displaystyle {\begin{bmatrix}\log p_{1}+C\\\vdots \\\log p_{k}+C\end{bmatrix}}}$${\displaystyle {\begin{bmatrix}{\dfrac {1}{C}}e^{\eta _{1}}\\\vdots \\{\dfrac {1}{C}}e^{\eta _{k}}\end{bmatrix}}=}$

${\displaystyle {\begin{bmatrix}{\dfrac {e^{\eta _{1}}}{\sum _{i=1}^{k}e^{\eta _{i}}}}\\[10pt]\vdots \\[5pt]{\dfrac {e^{\eta _{k}}}{\sum _{i=1}^{k}e^{\eta _{i}}}}\end{bmatrix}}}$

where ${\displaystyle \textstyle \sum _{i=1}^{k}e^{\eta _{i}}=C}$

${\displaystyle {\frac {n!}{\prod _{i=1}^{k}x_{i}!}}}$${\displaystyle {\begin{bmatrix}x_{1}\\\vdots \\x_{k}\end{bmatrix}}}$${\displaystyle 0}$${\displaystyle 0}$
multinomial distribution

(variant 3)
with known number of trials ${\displaystyle n}$
${\displaystyle p_{1},\ \ldots ,\,p_{k}}$

where ${\displaystyle p_{k}=1-\textstyle \sum _{i=1}^{k-1}p_{i}}$
${\displaystyle {\begin{bmatrix}\log {\dfrac {p_{1}}{p_{k}}}\\[10pt]\vdots \\[5pt]\log {\dfrac {p_{k-1}}{p_{k}}}\\[15pt]0\end{bmatrix}}=}$

${\displaystyle {\begin{bmatrix}\log {\dfrac {p_{1}}{1-\sum _{i=1}^{k-1}p_{i}}}\\[10pt]\vdots \\[5pt]\log {\dfrac {p_{k-1}}{1-\sum _{i=1}^{k-1}p_{i}}}\\[15pt]0\end{bmatrix}}}$
${\displaystyle {\begin{bmatrix}{\dfrac {e^{\eta _{1}}}{\sum _{i=1}^{k}e^{\eta _{i}}}}\\[10pt]\vdots \\[5pt]{\dfrac {e^{\eta _{k}}}{\sum _{i=1}^{k}e^{\eta _{i}}}}\end{bmatrix}}=}$

${\displaystyle {\begin{bmatrix}{\dfrac {e^{\eta _{1}}}{1+\sum _{i=1}^{k-1}e^{\eta _{i}}}}\\[10pt]\vdots \\[5pt]{\dfrac {e^{\eta _{k-1}}}{1+\sum _{i=1}^{k-1}e^{\eta _{i}}}}\\[15pt]{\dfrac {1}{1+\sum _{i=1}^{k-1}e^{\eta _{i}}}}\end{bmatrix}}}$

${\displaystyle {\frac {n!}{\prod _{i=1}^{k}x_{i}!}}}$${\displaystyle {\begin{bmatrix}x_{1}\\\vdots \\x_{k}\end{bmatrix}}}$${\displaystyle n\log \left(\sum _{i=1}^{k}e^{\eta _{i}}\right)=n\log \left(1+\sum _{i=1}^{k-1}e^{\eta _{i}}\right)}$${\displaystyle -n\log p_{k}=-n\log \left(1-\sum _{i=1}^{k-1}p_{i}\right)}$
Dirichlet distribution

(variant 1)
${\displaystyle \alpha _{1},\ \ldots ,\,\alpha _{k}}$${\displaystyle {\begin{bmatrix}\alpha _{1}\\\vdots \\\alpha _{k}\end{bmatrix}}}$${\displaystyle {\begin{bmatrix}\eta _{1}\\\vdots \\\eta _{k}\end{bmatrix}}}$${\displaystyle {\frac {1}{\prod _{i=1}^{k}x_{i}}}}$${\displaystyle {\begin{bmatrix}\log x_{1}\\\vdots \\\log x_{k}\end{bmatrix}}}$${\displaystyle \sum _{i=1}^{k}\log \Gamma (\eta _{i})-\log \Gamma \left(\sum _{i=1}^{k}\eta _{i}\right)}$${\displaystyle \sum _{i=1}^{k}\log \Gamma (\alpha _{i})-\log \Gamma \left(\sum _{i=1}^{k}\alpha _{i}\right)}$
Dirichlet distribution

(variant 2)
${\displaystyle \alpha _{1},\ \ldots ,\,\alpha _{k}}$${\displaystyle {\begin{bmatrix}\alpha _{1}-1\\\vdots \\\alpha _{k}-1\end{bmatrix}}}$${\displaystyle {\begin{bmatrix}\eta _{1}+1\\\vdots \\\eta _{k}+1\end{bmatrix}}}$${\displaystyle 1}$${\displaystyle {\begin{bmatrix}\log x_{1}\\\vdots \\\log x_{k}\end{bmatrix}}}$${\displaystyle \sum _{i=1}^{k}\log \Gamma (\eta _{i}+1)-\log \Gamma \left(\sum _{i=1}^{k}(\eta _{i}+1)\right)}$${\displaystyle \sum _{i=1}^{k}\log \Gamma (\alpha _{i})-\log \Gamma \left(\sum _{i=1}^{k}\alpha _{i}\right)}$
Wishart distribution ${\displaystyle \mathbf {V} ,\ n}$${\displaystyle {\begin{bmatrix}-{\frac {1}{2}}\mathbf {V} ^{-1}\\[5pt]{\dfrac {n-p-1}{2}}\end{bmatrix}}}$${\displaystyle {\begin{bmatrix}-{\frac {1}{2}}{{\boldsymbol {\eta }}_{1}}^{-1}\\[5pt]2\eta _{2}+p+1\end{bmatrix}}}$${\displaystyle 1}$${\displaystyle {\begin{bmatrix}\mathbf {X} \\\log |\mathbf {X} |\end{bmatrix}}}$${\displaystyle -\left(\eta _{2}+{\frac {p+1}{2}}\right)\log |-{\boldsymbol {\eta }}_{1}|}$

${\displaystyle +\log \Gamma _{p}\left(\eta _{2}+{\frac {p+1}{2}}\right)=}$
${\displaystyle -{\frac {n}{2}}\log |-{\boldsymbol {\eta }}_{1}|+\log \Gamma _{p}\left({\frac {n}{2}}\right)=}$
${\displaystyle \left(\eta _{2}+{\frac {p+1}{2}}\right)(p\log 2+\log |\mathbf {V} |)}$
${\displaystyle +\log \Gamma _{p}\left(\eta _{2}+{\frac {p+1}{2}}\right)}$

• Three variants with different parameterizations are given, to facilitate computing moments of the sufficient statistics.
${\displaystyle {\frac {n}{2}}(p\log 2+\log |\mathbf {V} |)+\log \Gamma _{p}\left({\frac {n}{2}}\right)}$
Note: Uses the fact that ${\displaystyle {\rm {tr}}(\mathbf {A} ^{\mathsf {T}}\mathbf {B} )=\operatorname {vec} (\mathbf {A} )\cdot \operatorname {vec} (\mathbf {B} ),}$ i.e. the trace of a matrix product is much like a dot product. The matrix parameters are assumed to be vectorized (laid out in a vector) when inserted into the exponential form. Also, ${\displaystyle \mathbf {V} }$ and ${\displaystyle \mathbf {X} }$ are symmetric, so e.g. ${\displaystyle \mathbf {V} ^{\mathsf {T}}=\mathbf {V} \ .}$
inverse Wishart distribution ${\displaystyle \mathbf {\Psi } ,\,m}$${\displaystyle {\begin{bmatrix}-{\frac {1}{2}}{\boldsymbol {\Psi }}\\[5pt]-{\dfrac {m+p+1}{2}}\end{bmatrix}}}$${\displaystyle {\begin{bmatrix}-2{\boldsymbol {\eta }}_{1}\\[5pt]-(2\eta _{2}+p+1)\end{bmatrix}}}$${\displaystyle 1}$${\displaystyle {\begin{bmatrix}\mathbf {X} ^{-1}\\\log |\mathbf {X} |\end{bmatrix}}}$${\displaystyle \left(\eta _{2}+{\frac {p+1}{2}}\right)\log |-{\boldsymbol {\eta }}_{1}|}$

${\displaystyle +\log \Gamma _{p}\left(-{\Big (}\eta _{2}+{\frac {p+1}{2}}{\Big )}\right)=}$
${\displaystyle -{\frac {m}{2}}\log |-{\boldsymbol {\eta }}_{1}|+\log \Gamma _{p}\left({\frac {m}{2}}\right)=}$
${\displaystyle -\left(\eta _{2}+{\frac {p+1}{2}}\right)(p\log 2-\log |{\boldsymbol {\Psi }}|)}$
${\displaystyle +\log \Gamma _{p}\left(-{\Big (}\eta _{2}+{\frac {p+1}{2}}{\Big )}\right)}$

${\displaystyle {\frac {m}{2}}(p\log 2-\log |{\boldsymbol {\Psi }}|)+\log \Gamma _{p}\left({\frac {m}{2}}\right)}$
normal-gamma distribution ${\displaystyle \alpha ,\ \beta ,\ \mu ,\ \lambda }$${\displaystyle {\begin{bmatrix}\alpha -{\frac {1}{2}}\\-\beta -{\dfrac {\lambda \mu ^{2}}{2}}\\\lambda \mu \\-{\dfrac {\lambda }{2}}\end{bmatrix}}}$${\displaystyle {\begin{bmatrix}\eta _{1}+{\frac {1}{2}}\\-\eta _{2}+{\dfrac {\eta _{3}^{2}}{4\eta _{4}}}\\-{\dfrac {\eta _{3}}{2\eta _{4}}}\\-2\eta _{4}\end{bmatrix}}}$${\displaystyle {\dfrac {1}{\sqrt {2\pi }}}}$${\displaystyle {\begin{bmatrix}\log \tau \\\tau \\\tau x\\\tau x^{2}\end{bmatrix}}}$${\displaystyle \log \Gamma \left(\eta _{1}+{\frac {1}{2}}\right)-{\frac {1}{2}}\log \left(-2\eta _{4}\right)}$

${\displaystyle -\left(\eta _{1}+{\frac {1}{2}}\right)\log \left(-\eta _{2}+{\dfrac {\eta _{3}^{2}}{4\eta _{4}}}\right)}$

${\displaystyle \log \Gamma \left(\alpha \right)-\alpha \log \beta -{\frac {1}{2}}\log \lambda }$
* The Iverson bracket is a generalization of the discrete delta-function: If the bracketed expression is true, the bracket has value 1; if the enclosed statement is false, the Iverson bracket is zero. There are many variant notations, e.g. wavey brackets: a=b is equivalent to the [a=b] notation used above.

The three variants of the categorical distribution and multinomial distribution are due to the fact that the parameters ${\displaystyle p_{i}}$ are constrained, such that

${\displaystyle \sum _{i=1}^{k}p_{i}=1~.}$

Thus, there are only ${\displaystyle k-1}$ independent parameters.

• Variant 1 uses ${\displaystyle k}$ natural parameters with a simple relation between the standard and natural parameters; however, only ${\displaystyle k-1}$ of the natural parameters are independent, and the set of ${\displaystyle k}$ natural parameters is nonidentifiable. The constraint on the usual parameters translates to a similar constraint on the natural parameters.
• Variant 2 demonstrates the fact that the entire set of natural parameters is nonidentifiable: Adding any constant value to the natural parameters has no effect on the resulting distribution. However, by using the constraint on the natural parameters, the formula for the normal parameters in terms of the natural parameters can be written in a way that is independent on the constant that is added.
• Variant 3 shows how to make the parameters identifiable in a convenient way by setting ${\displaystyle C=-\log p_{k}\ .}$ This effectively "pivots" around ${\displaystyle p_{k}}$ and causes the last natural parameter to have the constant value of 0. All the remaining formulas are written in a way that does not access ${\displaystyle p_{k}\ }$, so that effectively the model has only ${\displaystyle k-1}$ parameters, both of the usual and natural kind.

Variants 1 and 2 are not actually standard exponential families at all. Rather they are curved exponential families, i.e. there are ${\displaystyle k-1}$ independent parameters embedded in a ${\displaystyle k}$-dimensional parameter space. [12] Many of the standard results for exponential families do not apply to curved exponential families. An example is the log-partition function ${\displaystyle A(x)\ }$, which has the value of 0 in the curved cases. In standard exponential families, the derivatives of this function correspond to the moments (more technically, the cumulants) of the sufficient statistics, e.g. the mean and variance. However, a value of 0 suggests that the mean and variance of all the sufficient statistics are uniformly 0, whereas in fact the mean of the ${\displaystyle i}$th sufficient statistic should be ${\displaystyle p_{i}\ }$. (This does emerge correctly when using the form of ${\displaystyle A(x)\ }$ shown in variant 3.)

## Moments and cumulants of the sufficient statistic

### Normalization of the distribution

We start with the normalization of the probability distribution. In general, any non-negative function f(x) that serves as the kernel of a probability distribution (the part encoding all dependence on x) can be made into a proper distribution by normalizing: i.e.

${\displaystyle p(x)={\frac {1}{Z}}f(x)}$

where

${\displaystyle Z=\int _{x}f(x)\,dx.}$

The factor Z is sometimes termed the normalizer or partition function , based on an analogy to statistical physics.

In the case of an exponential family where

${\displaystyle p(x;{\boldsymbol {\eta }})=g({\boldsymbol {\eta }})h(x)e^{{\boldsymbol {\eta }}\cdot \mathbf {T} (x)},}$

the kernel is

${\displaystyle K(x)=h(x)e^{{\boldsymbol {\eta }}\cdot \mathbf {T} (x)}}$

and the partition function is

${\displaystyle Z=\int _{x}h(x)e^{{\boldsymbol {\eta }}\cdot \mathbf {T} (x)}\,dx.}$

Since the distribution must be normalized, we have

${\displaystyle 1=\int _{x}g({\boldsymbol {\eta }})h(x)e^{{\boldsymbol {\eta }}\cdot \mathbf {T} (x)}\,dx=g({\boldsymbol {\eta }})\int _{x}h(x)e^{{\boldsymbol {\eta }}\cdot \mathbf {T} (x)}\,dx=g({\boldsymbol {\eta }})Z.}$

In other words,

${\displaystyle g({\boldsymbol {\eta }})={\frac {1}{Z}}}$

or equivalently

${\displaystyle A({\boldsymbol {\eta }})=-\log g({\boldsymbol {\eta }})=\log Z.}$

This justifies calling A the log-normalizer or log-partition function.

### Moment-generating function of the sufficient statistic

Now, the moment-generating function of T(x) is

${\displaystyle M_{T}(u)\equiv E[e^{u^{\top }T(x)}\mid \eta ]=\int _{x}h(x)e^{(\eta +u)^{\top }T(x)-A(\eta )}\,dx=e^{A(\eta +u)-A(\eta )}}$

proving the earlier statement that

${\displaystyle K(u\mid \eta )=A(\eta +u)-A(\eta )}$

is the cumulant generating function for T.

An important subclass of exponential families are the natural exponential families, which have a similar form for the moment-generating function for the distribution of x.

#### Differential identities for cumulants

In particular, using the properties of the cumulant generating function,

${\displaystyle \operatorname {E} (T_{j})={\frac {\partial A(\eta )}{\partial \eta _{j}}}}$

and

${\displaystyle \operatorname {cov} \left(T_{i},\ T_{j}\right)={\frac {\partial ^{2}A(\eta )}{\partial \eta _{i}\,\partial \eta _{j}}}.}$

The first two raw moments and all mixed second moments can be recovered from these two identities. Higher-order moments and cumulants are obtained by higher derivatives. This technique is often useful when T is a complicated function of the data, whose moments are difficult to calculate by integration.

Another way to see this that does not rely on the theory of cumulants is to begin from the fact that the distribution of an exponential family must be normalized, and differentiate. We illustrate using the simple case of a one-dimensional parameter, but an analogous derivation holds more generally.

In the one-dimensional case, we have

${\displaystyle p(x)=g(\eta )h(x)e^{\eta T(x)}.}$

This must be normalized, so

${\displaystyle 1=\int _{x}p(x)\,dx=\int _{x}g(\eta )h(x)e^{\eta T(x)}\,dx=g(\eta )\int _{x}h(x)e^{\eta T(x)}\,dx.}$

Take the derivative of both sides with respect to η:

{\displaystyle {\begin{aligned}0&=g(\eta ){\frac {d}{d\eta }}\int _{x}h(x)e^{\eta T(x)}\,dx+g'(\eta )\int _{x}h(x)e^{\eta T(x)}\,dx\\&=g(\eta )\int _{x}h(x)\left({\frac {d}{d\eta }}e^{\eta T(x)}\right)\,dx+g'(\eta )\int _{x}h(x)e^{\eta T(x)}\,dx\\&=g(\eta )\int _{x}h(x)e^{\eta T(x)}T(x)\,dx+g'(\eta )\int _{x}h(x)e^{\eta T(x)}\,dx\\&=\int _{x}T(x)g(\eta )h(x)e^{\eta T(x)}\,dx+{\frac {g'(\eta )}{g(\eta )}}\int _{x}g(\eta )h(x)e^{\eta T(x)}\,dx\\&=\int _{x}T(x)p(x)\,dx+{\frac {g'(\eta )}{g(\eta )}}\int _{x}p(x)\,dx\\&=\operatorname {E} [T(x)]+{\frac {g'(\eta )}{g(\eta )}}\\&=\operatorname {E} [T(x)]+{\frac {d}{d\eta }}\log g(\eta )\end{aligned}}}

Therefore,

${\displaystyle \operatorname {E} [T(x)]=-{\frac {d}{d\eta }}\log g(\eta )={\frac {d}{d\eta }}A(\eta ).}$

#### Example 1

As an introductory example, consider the gamma distribution, whose distribution is defined by

${\displaystyle p(x)={\frac {\beta ^{\alpha }}{\Gamma (\alpha )}}x^{\alpha -1}e^{-\beta x}.}$

Referring to the above table, we can see that the natural parameter is given by

${\displaystyle \eta _{1}=\alpha -1,}$
${\displaystyle \eta _{2}=-\beta ,}$

the reverse substitutions are

${\displaystyle \alpha =\eta _{1}+1,}$
${\displaystyle \beta =-\eta _{2},}$

the sufficient statistics are ${\displaystyle (\log x,x),}$ and the log-partition function is

${\displaystyle A(\eta _{1},\eta _{2})=\log \Gamma (\eta _{1}+1)-(\eta _{1}+1)\log(-\eta _{2}).}$

We can find the mean of the sufficient statistics as follows. First, for η1:

{\displaystyle {\begin{aligned}\operatorname {E} [\log x]&={\frac {\partial A(\eta _{1},\eta _{2})}{\partial \eta _{1}}}={\frac {\partial }{\partial \eta _{1}}}\left(\log \Gamma (\eta _{1}+1)-(\eta _{1}+1)\log(-\eta _{2})\right)\\&=\psi (\eta _{1}+1)-\log(-\eta _{2})\\&=\psi (\alpha )-\log \beta ,\end{aligned}}}

Where ${\displaystyle \psi (x)}$ is the digamma function (derivative of log gamma), and we used the reverse substitutions in the last step.

Now, for η2:

{\displaystyle {\begin{aligned}\operatorname {E} [x]&={\frac {\partial A(\eta _{1},\eta _{2})}{\partial \eta _{2}}}={\frac {\partial }{\partial \eta _{2}}}\left(\log \Gamma (\eta _{1}+1)-(\eta _{1}+1)\log(-\eta _{2})\right)\\&=-(\eta _{1}+1){\frac {1}{-\eta _{2}}}(-1)={\frac {\eta _{1}+1}{-\eta _{2}}}\\&={\frac {\alpha }{\beta }},\end{aligned}}}

again making the reverse substitution in the last step.

To compute the variance of x, we just differentiate again:

{\displaystyle {\begin{aligned}\operatorname {Var} (x)&={\frac {\partial ^{2}A\left(\eta _{1},\eta _{2}\right)}{\partial \eta _{2}^{2}}}={\frac {\partial }{\partial \eta _{2}}}{\frac {\eta _{1}+1}{-\eta _{2}}}\\&={\frac {\eta _{1}+1}{\eta _{2}^{2}}}\\&={\frac {\alpha }{\beta ^{2}}}.\end{aligned}}}

All of these calculations can be done using integration, making use of various properties of the gamma function, but this requires significantly more work.

#### Example 2

As another example consider a real valued random variable X with density

${\displaystyle p_{\theta }(x)={\frac {\theta e^{-x}}{\left(1+e^{-x}\right)^{\theta +1}}}}$

indexed by shape parameter ${\displaystyle \theta \in (0,\infty )}$ (this is called the skew-logistic distribution). The density can be rewritten as

${\displaystyle {\frac {e^{-x}}{1+e^{-x}}}\exp \left(-\theta \log \left(1+e^{-x}\right)+\log(\theta )\right)}$

Notice this is an exponential family with natural parameter

${\displaystyle \eta =-\theta ,}$

sufficient statistic

${\displaystyle T=\log \left(1+e^{-x}\right),}$

and log-partition function

${\displaystyle A(\eta )=-\log(\theta )=-\log(-\eta )}$

So using the first identity,

${\displaystyle \operatorname {E} (\log(1+e^{-X}))=\operatorname {E} (T)={\frac {\partial A(\eta )}{\partial \eta }}={\frac {\partial }{\partial \eta }}[-\log(-\eta )]={\frac {1}{-\eta }}={\frac {1}{\theta }},}$

and using the second identity

${\displaystyle \operatorname {var} (\log \left(1+e^{-X}\right))={\frac {\partial ^{2}A(\eta )}{\partial \eta ^{2}}}={\frac {\partial }{\partial \eta }}\left[{\frac {1}{-\eta }}\right]={\frac {1}{(-\eta )^{2}}}={\frac {1}{\theta ^{2}}}.}$

This example illustrates a case where using this method is very simple, but the direct calculation would be nearly impossible.

#### Example 3

The final example is one where integration would be extremely difficult. This is the case of the Wishart distribution, which is defined over matrices. Even taking derivatives is a bit tricky, as it involves matrix calculus, but the respective identities are listed in that article.

From the above table, we can see that the natural parameter is given by

${\displaystyle {\boldsymbol {\eta }}_{1}=-{\frac {1}{2}}\mathbf {V} ^{-1},}$
${\displaystyle \eta _{2}={\frac {n-p-1}{2}},}$

the reverse substitutions are

${\displaystyle \mathbf {V} =-{\frac {1}{2}}{{\boldsymbol {\eta }}_{1}}^{-1},}$
${\displaystyle n=2\eta _{2}+p+1,}$

and the sufficient statistics are ${\displaystyle (\mathbf {X} ,\log |\mathbf {X} |).}$

The log-partition function is written in various forms in the table, to facilitate differentiation and back-substitution. We use the following forms:

${\displaystyle A({\boldsymbol {\eta }}_{1},n)=-{\frac {n}{2}}\log |-{\boldsymbol {\eta }}_{1}|+\log \Gamma _{p}\left({\frac {n}{2}}\right),}$
${\displaystyle A(\mathbf {V} ,\eta _{2})=\left(\eta _{2}+{\frac {p+1}{2}}\right)(p\log 2+\log |\mathbf {V} |)+\log \Gamma _{p}\left(\eta _{2}+{\frac {p+1}{2}}\right).}$
Expectation of X (associated with η1)

To differentiate with respect to η1, we need the following matrix calculus identity:

${\displaystyle {\frac {\partial \log |a\mathbf {X} |}{\partial \mathbf {X} }}=(\mathbf {X} ^{-1})^{\rm {T}}}$

Then:

{\displaystyle {\begin{aligned}\operatorname {E} [\mathbf {X} ]&={\frac {\partial A\left({\boldsymbol {\eta }}_{1},\cdots \right)}{\partial {\boldsymbol {\eta }}_{1}}}\\&={\frac {\partial }{\partial {\boldsymbol {\eta }}_{1}}}\left[-{\frac {n}{2}}\log |-{\boldsymbol {\eta }}_{1}|+\log \Gamma _{p}\left({\frac {n}{2}}\right)\right]\\&=-{\frac {n}{2}}({\boldsymbol {\eta }}_{1}^{-1})^{\rm {T}}\\&={\frac {n}{2}}(-{\boldsymbol {\eta }}_{1}^{-1})^{\rm {T}}\\&=n(\mathbf {V} )^{\rm {T}}\\&=n\mathbf {V} \end{aligned}}}

The last line uses the fact that V is symmetric, and therefore it is the same when transposed.

Expectation of log |X| (associated with η2)

Now, for η2, we first need to expand the part of the log-partition function that involves the multivariate gamma function:

${\displaystyle \log \Gamma _{p}(a)=\log \left(\pi ^{\frac {p(p-1)}{4}}\prod _{j=1}^{p}\Gamma \left(a+{\frac {1-j}{2}}\right)\right)={\frac {p(p-1)}{4}}\log \pi +\sum _{j=1}^{p}\log \Gamma \left[a+{\frac {1-j}{2}}\right]}$

We also need the digamma function:

${\displaystyle \psi (x)={\frac {d}{dx}}\log \Gamma (x).}$

Then:

{\displaystyle {\begin{aligned}\operatorname {E} [\log |\mathbf {X} |]&={\frac {\partial A\left(\ldots ,\eta _{2}\right)}{\partial \eta _{2}}}\\&={\frac {\partial }{\partial \eta _{2}}}\left[-\left(\eta _{2}+{\frac {p+1}{2}}\right)(p\log 2+\log |\mathbf {V} |)+\log \Gamma _{p}\left(\eta _{2}+{\frac {p+1}{2}}\right)\right]\\&={\frac {\partial }{\partial \eta _{2}}}\left[\left(\eta _{2}+{\frac {p+1}{2}}\right)(p\log 2+\log |\mathbf {V} |)+{\frac {p(p-1)}{4}}\log \pi +\sum _{j=1}^{p}\log \Gamma \left(\eta _{2}+{\frac {p+1}{2}}+{\frac {1-j}{2}}\right)\right]\\&=p\log 2+\log |\mathbf {V} |+\sum _{j=1}^{p}\psi \left(\eta _{2}+{\frac {p+1}{2}}+{\frac {1-j}{2}}\right)\\&=p\log 2+\log |\mathbf {V} |+\sum _{j=1}^{p}\psi \left({\frac {n-p-1}{2}}+{\frac {p+1}{2}}+{\frac {1-j}{2}}\right)\\&=p\log 2+\log |\mathbf {V} |+\sum _{j=1}^{p}\psi \left({\frac {n+1-j}{2}}\right)\end{aligned}}}

This latter formula is listed in the Wishart distribution article. Both of these expectations are needed when deriving the variational Bayes update equations in a Bayes network involving a Wishart distribution (which is the conjugate prior of the multivariate normal distribution).

Computing these formulas using integration would be much more difficult. The first one, for example, would require matrix integration.

## Entropy

### Relative entropy

The relative entropy (Kullback–Leibler divergence, KL divergence) of two distributions in an exponential family has a simple expression as the Bregman divergence between the natural parameters with respect to the log-normalizer. [13] The relative entropy is defined in terms of an integral, while the Bregman divergence is defined in terms of a derivative and inner product, and thus is easier to calculate and has a closed-form expression (assuming the derivative has a closed-form expression). Further, the Bregman divergence in terms of the natural parameters and the log-normalizer equals the Bregman divergence of the dual parameters (expectation parameters), in the opposite order, for the convex conjugate function. [14]

Fixing an exponential family with log-normalizer ${\displaystyle A}$ (with convex conjugate ${\displaystyle A^{*}}$), writing ${\displaystyle P_{A,\theta }}$ for the distribution in this family corresponding a fixed value of the natural parameter ${\displaystyle \theta }$ (writing ${\displaystyle \theta '}$ for another value, and with ${\displaystyle \eta ,\eta '}$ for the corresponding dual expectation/moment parameters), writing KL for the KL divergence, and ${\displaystyle B_{A}}$ for the Bregman divergence, the divergences are related as:

${\displaystyle {\rm {{KL}(P_{A,\theta }\parallel P_{A,\theta '})=B_{A}(\theta '\parallel \theta )=B_{A^{*}}(\eta \parallel \eta ').}}}$

The KL divergence is conventionally written with respect to the first parameter, while the Bregman divergence is conventionally written with respect to the second parameter, and thus this can be read as "the relative entropy is equal to the Bregman divergence defined by the log-normalizer on the swapped natural parameters", or equivalently as "equal to the Bregman divergence defined by the dual to the log-normalizer on the expectation parameters".

### Maximum entropy derivation

Exponential families arise naturally as the answer to the following question: what is the maximum-entropy distribution consistent with given constraints on expected values?

The information entropy of a probability distribution dF(x) can only be computed with respect to some other probability distribution (or, more generally, a positive measure), and both measures must be mutually absolutely continuous. Accordingly, we need to pick a reference measuredH(x) with the same support as dF(x).

The entropy of dF(x) relative to dH(x) is

${\displaystyle S[dF\mid dH]=-\int {\frac {dF}{dH}}\log {\frac {dF}{dH}}\,dH}$

or

${\displaystyle S[dF\mid dH]=\int \log {\frac {dH}{dF}}\,dF}$

where dF/dH and dH/dF are Radon–Nikodym derivatives. The ordinary definition of entropy for a discrete distribution supported on a set I, namely

${\displaystyle S=-\sum _{i\in I}p_{i}\log p_{i}}$

assumes, though this is seldom pointed out, that dH is chosen to be the counting measure on I.

Consider now a collection of observable quantities (random variables) Ti. The probability distribution dF whose entropy with respect to dH is greatest, subject to the conditions that the expected value of Ti be equal to ti, is an exponential family with dH as reference measure and (T1, ..., Tn) as sufficient statistic.

The derivation is a simple variational calculation using Lagrange multipliers. Normalization is imposed by letting T0 = 1 be one of the constraints. The natural parameters of the distribution are the Lagrange multipliers, and the normalization factor is the Lagrange multiplier associated to T0.

For examples of such derivations, see Maximum entropy probability distribution.

## Role in statistics

### Classical estimation: sufficiency

According to the Pitman Koopman Darmois theorem, among families of probability distributions whose domain does not vary with the parameter being estimated, only in exponential families is there a sufficient statistic whose dimension remains bounded as sample size increases.

Less tersely, suppose Xk, (where k = 1, 2, 3, ... n) are independent, identically distributed random variables. Only if their distribution is one of the exponential family of distributions is there a sufficient statistic T(X1, ..., Xn) whose number of scalar components does not increase as the sample size n increases; the statistic T may be a vector or a single scalar number, but whatever it is, its size will neither grow nor shrink when more data are obtained.

As a counterexample if these conditions are relaxed, the family of uniform distributions (either discrete or continuous, with either or both bounds unknown) has a sufficient statistic, namely the sample maximum, sample minimum, and sample size, but does not form an exponential family, as the domain varies with the parameters.

### Bayesian estimation: conjugate distributions

Exponential families are also important in Bayesian statistics. In Bayesian statistics a prior distribution is multiplied by a likelihood function and then normalised to produce a posterior distribution. In the case of a likelihood which belongs to an exponential family there exists a conjugate prior, which is often also in an exponential family. A conjugate prior π for the parameter ${\displaystyle {\boldsymbol {\eta }}}$ of an exponential family

${\displaystyle f(x\mid {\boldsymbol {\eta }})=h(x)\exp \left({\boldsymbol {\eta }}^{\rm {T}}\mathbf {T} (x)-A({\boldsymbol {\eta }})\right)}$

is given by

${\displaystyle p_{\pi }({\boldsymbol {\eta }}\mid {\boldsymbol {\chi }},\nu )=f({\boldsymbol {\chi }},\nu )\exp \left({\boldsymbol {\eta }}^{\rm {T}}{\boldsymbol {\chi }}-\nu A({\boldsymbol {\eta }})\right),}$

or equivalently

${\displaystyle p_{\pi }({\boldsymbol {\eta }}\mid {\boldsymbol {\chi }},\nu )=f({\boldsymbol {\chi }},\nu )g({\boldsymbol {\eta }})^{\nu }\exp \left({\boldsymbol {\eta }}^{\rm {T}}{\boldsymbol {\chi }}\right),\qquad {\boldsymbol {\chi }}\in \mathbb {R} ^{s}}$

where s is the dimension of ${\displaystyle {\boldsymbol {\eta }}}$ and ${\displaystyle \nu >0}$ and ${\displaystyle {\boldsymbol {\chi }}}$ are hyperparameters (parameters controlling parameters). ${\displaystyle \nu }$ corresponds to the effective number of observations that the prior distribution contributes, and ${\displaystyle {\boldsymbol {\chi }}}$ corresponds to the total amount that these pseudo-observations contribute to the sufficient statistic over all observations and pseudo-observations. ${\displaystyle f({\boldsymbol {\chi }},\nu )}$ is a normalization constant that is automatically determined by the remaining functions and serves to ensure that the given function is a probability density function (i.e. it is normalized). ${\displaystyle A({\boldsymbol {\eta }})}$ and equivalently ${\displaystyle g({\boldsymbol {\eta }})}$ are the same functions as in the definition of the distribution over which π is the conjugate prior.

A conjugate prior is one which, when combined with the likelihood and normalised, produces a posterior distribution which is of the same type as the prior. For example, if one is estimating the success probability of a binomial distribution, then if one chooses to use a beta distribution as one's prior, the posterior is another beta distribution. This makes the computation of the posterior particularly simple. Similarly, if one is estimating the parameter of a Poisson distribution the use of a gamma prior will lead to another gamma posterior. Conjugate priors are often very flexible and can be very convenient. However, if one's belief about the likely value of the theta parameter of a binomial is represented by (say) a bimodal (two-humped) prior distribution, then this cannot be represented by a beta distribution. It can however be represented by using a mixture density as the prior, here a combination of two beta distributions; this is a form of hyperprior.

An arbitrary likelihood will not belong to an exponential family, and thus in general no conjugate prior exists. The posterior will then have to be computed by numerical methods.

To show that the above prior distribution is a conjugate prior, we can derive the posterior.

First, assume that the probability of a single observation follows an exponential family, parameterized using its natural parameter:

${\displaystyle p_{F}(x\mid {\boldsymbol {\eta }})=h(x)g({\boldsymbol {\eta }})\exp \left({\boldsymbol {\eta }}^{\rm {T}}\mathbf {T} (x)\right)}$

Then, for data ${\displaystyle \mathbf {X} =(x_{1},\ldots ,x_{n})}$, the likelihood is computed as follows:

${\displaystyle p(\mathbf {X} \mid {\boldsymbol {\eta }})=\left(\prod _{i=1}^{n}h(x_{i})\right)g({\boldsymbol {\eta }})^{n}\exp \left({\boldsymbol {\eta }}^{\rm {T}}\sum _{i=1}^{n}\mathbf {T} (x_{i})\right)}$

Then, for the above conjugate prior:

{\displaystyle {\begin{aligned}p_{\pi }({\boldsymbol {\eta }}\mid {\boldsymbol {\chi }},\nu )&=f({\boldsymbol {\chi }},\nu )g({\boldsymbol {\eta }})^{\nu }\exp({\boldsymbol {\eta }}^{\rm {T}}{\boldsymbol {\chi }})\propto g({\boldsymbol {\eta }})^{\nu }\exp({\boldsymbol {\eta }}^{\rm {T}}{\boldsymbol {\chi }})\end{aligned}}}

We can then compute the posterior as follows:

{\displaystyle {\begin{aligned}p({\boldsymbol {\eta }}\mid \mathbf {X} ,{\boldsymbol {\chi }},\nu )&\propto p(\mathbf {X} \mid {\boldsymbol {\eta }})p_{\pi }({\boldsymbol {\eta }}\mid {\boldsymbol {\chi }},\nu )\\&=\left(\prod _{i=1}^{n}h(x_{i})\right)g({\boldsymbol {\eta }})^{n}\exp \left({\boldsymbol {\eta }}^{\rm {T}}\sum _{i=1}^{n}\mathbf {T} (x_{i})\right)f({\boldsymbol {\chi }},\nu )g({\boldsymbol {\eta }})^{\nu }\exp({\boldsymbol {\eta }}^{\rm {T}}{\boldsymbol {\chi }})\\&\propto g({\boldsymbol {\eta }})^{n}\exp \left({\boldsymbol {\eta }}^{\rm {T}}\sum _{i=1}^{n}\mathbf {T} (x_{i})\right)g({\boldsymbol {\eta }})^{\nu }\exp({\boldsymbol {\eta }}^{\rm {T}}{\boldsymbol {\chi }})\\&\propto g({\boldsymbol {\eta }})^{\nu +n}\exp \left({\boldsymbol {\eta }}^{\rm {T}}\left({\boldsymbol {\chi }}+\sum _{i=1}^{n}\mathbf {T} (x_{i})\right)\right)\end{aligned}}}

The last line is the kernel of the posterior distribution, i.e.

${\displaystyle p({\boldsymbol {\eta }}\mid \mathbf {X} ,{\boldsymbol {\chi }},\nu )=p_{\pi }\left({\boldsymbol {\eta }}\left|~{\boldsymbol {\chi }}+\sum _{i=1}^{n}\mathbf {T} (x_{i}),\nu +n\right.\right)}$

This shows that the posterior has the same form as the prior.

The data X enters into this equation only in the expression

${\displaystyle \mathbf {T} (\mathbf {X} )=\sum _{i=1}^{n}\mathbf {T} (x_{i}),}$

which is termed the sufficient statistic of the data. That is, the value of the sufficient statistic is sufficient to completely determine the posterior distribution. The actual data points themselves are not needed, and all sets of data points with the same sufficient statistic will have the same distribution. This is important because the dimension of the sufficient statistic does not grow with the data size — it has only as many components as the components of ${\displaystyle {\boldsymbol {\eta }}}$ (equivalently, the number of parameters of the distribution of a single data point).

The update equations are as follows:

{\displaystyle {\begin{aligned}{\boldsymbol {\chi }}'&={\boldsymbol {\chi }}+\mathbf {T} (\mathbf {X} )\\&={\boldsymbol {\chi }}+\sum _{i=1}^{n}\mathbf {T} (x_{i})\\\nu '&=\nu +n\end{aligned}}}

This shows that the update equations can be written simply in terms of the number of data points and the sufficient statistic of the data. This can be seen clearly in the various examples of update equations shown in the conjugate prior page. Because of the way that the sufficient statistic is computed, it necessarily involves sums of components of the data (in some cases disguised as products or other forms — a product can be written in terms of a sum of logarithms). The cases where the update equations for particular distributions don't exactly match the above forms are cases where the conjugate prior has been expressed using a different parameterization than the one that produces a conjugate prior of the above form — often specifically because the above form is defined over the natural parameter ${\displaystyle {\boldsymbol {\eta }}}$ while conjugate priors are usually defined over the actual parameter ${\displaystyle {\boldsymbol {\theta }}.}$

### Hypothesis testing: uniformly most powerful tests

A one-parameter exponential family has a monotone non-decreasing likelihood ratio in the sufficient statistic T(x), provided that η(θ) is non-decreasing. As a consequence, there exists a uniformly most powerful test for testing the hypothesis H0: θθ0vs. H1: θ < θ0.

### Generalized linear models

Exponential families form the basis for the distribution functions used in generalized linear models, a class of model that encompass many of the commonly used regression models in statistics.

## Footnotes

1. For example, the family of normal distributions includes the standard normal distribution N(0, 1) with mean 0 and variance 1, as well as other normal distributions with different mean and variance.
2. "Partition function" is often used in statistics as a synonym of "normalization factor".
3. These distributions are often not themselves exponential families. Common examples of non-exponential families arising from exponential ones are the Student's t-distribution, beta-binomial distribution and Dirichlet-multinomial distribution.

## Related Research Articles

The likelihood function describes the joint probability of the observed data as a function of the parameters of the chosen statistical model. For each specific parameter value in the parameter space, the likelihood function therefore assigns a probabilistic prediction to the observed data . Since it is essentially the product of sampling densities, the likelihood generally encapsulates both the data-generating process as well as the missing-data mechanism that produced the observed sample.

In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statistical model, the observed data is most probable. The point in the parameter space that maximizes the likelihood function is called the maximum likelihood estimate. The logic of maximum likelihood is both intuitive and flexible, and as such the method has become a dominant means of statistical inference.

In mechanics and geometry, the 3D rotation group, often denoted SO(3), is the group of all rotations about the origin of three-dimensional Euclidean space under the operation of composition.

In statistics, an expectation–maximization (EM) algorithm is an iterative method to find (local) maximum likelihood or maximum a posteriori (MAP) estimates of parameters in statistical models, where the model depends on unobserved latent variables. The EM iteration alternates between performing an expectation (E) step, which creates a function for the expectation of the log-likelihood evaluated using the current estimate for the parameters, and a maximization (M) step, which computes parameters maximizing the expected log-likelihood found on the E step. These parameter-estimates are then used to determine the distribution of the latent variables in the next E step.

In numerical analysis and computational statistics, rejection sampling is a basic technique used to generate observations from a distribution. It is also commonly called the acceptance-rejection method or "accept-reject algorithm" and is a type of exact simulation method. The method works for any distribution in with a density.

In mathematical statistics, the Fisher information is a way of measuring the amount of information that an observable random variable X carries about an unknown parameter θ of a distribution that models X. Formally, it is the variance of the score, or the expected value of the observed information. In Bayesian statistics, the asymptotic distribution of the posterior mode depends on the Fisher information and not on the prior. The role of the Fisher information in the asymptotic theory of maximum-likelihood estimation was emphasized by the statistician Ronald Fisher. The Fisher information is also used in the calculation of the Jeffreys prior, which is used in Bayesian statistics.

In statistics, the Neyman–Pearson lemma was introduced by Jerzy Neyman and Egon Pearson in a paper in 1933. The Neyman-Pearson lemma is part of the Neyman-Pearson theory of statistical testing, which introduced concepts like errors of the second kind, power function, and inductive behavior. The previous Fisherian theory of significance testing postulated only one hypothesis. By introducing a competing hypothesis, the Neyman-Pearsonian flavor of statistical testing allows investigating the two types of errors. The trivial cases where one always rejects or accepts the null hypothesis are of little interest but it does prove that one must not relinquish control over one type of error while calibrating the other. Neyman and Pearson accordingly proceeded to restrict their attention to the class of all level tests while subsequently minimizing type II error, traditionally denoted by . Their seminal paper of 1933, including the Neyman-Pearson lemma, comes at the end of this endeavor, not only showing the existence of tests with the most power that retain a prespecified level of type I error, but also providing a way to construct such tests. The Karlin-Rubin theorem extends the Neyman-Pearson lemma to settings involving composite hypotheses with monotone likelihood ratios.

In statistics, a generalized linear model (GLM) is a flexible generalization of ordinary linear regression. The GLM generalizes linear regression by allowing the linear model to be related to the response variable via a link function and by allowing the magnitude of the variance of each measurement to be a function of its predicted value.

In statistics and information theory, a maximum entropy probability distribution has entropy that is at least as great as that of all other members of a specified class of probability distributions. According to the principle of maximum entropy, if nothing is known about a distribution except that it belongs to a certain class, then the distribution with the largest entropy should be chosen as the least-informative default. The motivation is twofold: first, maximizing entropy minimizes the amount of prior information built into the distribution; second, many physical systems tend to move towards maximal entropy configurations over time.

In directional statistics, the von Mises–Fisher distribution, is a probability distribution on the -sphere in . If the distribution reduces to the von Mises distribution on the circle.

The cross-entropy (CE) method is a Monte Carlo method for importance sampling and optimization. It is applicable to both combinatorial and continuous problems, with either a static or noisy objective.

In mathematics, the axis–angle representation of a rotation parameterizes a rotation in a three-dimensional Euclidean space by two quantities: a unit vector e indicating the direction of an axis of rotation, and an angle θ describing the magnitude of the rotation about the axis. Only two numbers, not three, are needed to define the direction of a unit vector e rooted at the origin because the magnitude of e is constrained. For example, the elevation and azimuth angles of e suffice to locate it in any particular Cartesian coordinate frame.

In probability and statistics, a natural exponential family (NEF) is a class of probability distributions that is a special case of an exponential family (EF).

In probability and statistics, the class of exponential dispersion models (EDM) is a set of probability distributions that represents a generalisation of the natural exponential family. Exponential dispersion models play an important role in statistical theory, in particular in generalized linear models because they have a special structure which enables deductions to be made about appropriate statistical inference.

Natural evolution strategies (NES) are a family of numerical optimization algorithms for black box problems. Similar in spirit to evolution strategies, they iteratively update the (continuous) parameters of a search distribution by following the natural gradient towards higher expected fitness.

In Bayesian statistics, the posterior predictive distribution is the distribution of possible unobserved values conditional on the observed values.

In statistics, the variance function is a smooth function which depicts the variance of a random quantity as a function of its mean. The variance function is a measure of heteroscedasticity and plays a large role in many settings of statistical modelling. It is a main ingredient in the generalized linear model framework and a tool used in non-parametric regression, semiparametric regression and functional data analysis. In parametric modeling, variance functions take on a parametric form and explicitly describe the relationship between the variance and the mean of a random quantity. In a non-parametric setting, the variance function is assumed to be a smooth function.

In statistics, the class of vector generalized linear models (VGLMs) was proposed to enlarge the scope of models catered for by generalized linear models (GLMs). In particular, VGLMs allow for response variables outside the classical exponential family and for more than one parameter. Each parameter can be transformed by a link function. The VGLM framework is also large enough to naturally accommodate multiple responses; these are several independent responses each coming from a particular statistical distribution with possibly different parameter values.

Exponential Tilting (ET), Exponential Twisting, or Exponential Change of Measure (ECM) is a distribution shifting technique used in many parts of mathematics. The different exponential tiltings of a random variable is known as the natural exponential family of .

The hyperbolastic functions, also known as hyperbolastic growth models, are mathematical functions that are used in medical statistical modeling. These models were originally developed to capture the growth dynamics of multicellular tumor spheres, and were introduced in 2005 by Mohammad Tabatabai, David Williams, and Zoran Bursac. The precision of hyperbolastic functions in modeling real world problems is somewhat due to their flexibility in their point of inflection. These functions can be used in a wide variety of modeling problems such as tumor growth, stem cell proliferation, pharma kinetics, cancer growth, sigmoid activation function in neural networks, and epidemiological disease progression or regression.

## References

### Citations

1. Kupperman, M. (1958). "Probabilities of hypotheses and information-statistics in sampling from exponential-class populations". Annals of Mathematical Statistics . 9 (2): 571–575. doi:. JSTOR   2237349.
2. Andersen, Erling (September 1970). "Sufficiency and Exponential Families for Discrete Sample Spaces". Journal of the American Statistical Association . Journal of the American Statistical Association. 65 (331): 1248–1255. doi:10.2307/2284291. JSTOR   2284291. MR   0268992.
3. Pitman, E.; Wishart, J. (1936). "Sufficient statistics and intrinsic accuracy". Mathematical Proceedings of the Cambridge Philosophical Society . 32 (4): 567–579. Bibcode:1936PCPS...32..567P. doi:10.1017/S0305004100019307.
4. Darmois, G. (1935). "Sur les lois de probabilites a estimation exhaustive". C. R. Acad. Sci. Paris (in French). 200: 1265–1266.
5. Koopman, B. (1936). "On distribution admitting a sufficient statistic". Transactions of the American Mathematical Society . American Mathematical Society. 39 (3): 399–409. doi:. JSTOR   1989758. MR   1501854.
6. Abramovich & Ritov (2013). Statistical Theory: A Concise Introduction. Chapman & Hall. ISBN   978-1439851845.
7. Blei, David. "Variational Inference" (PDF). Princeton U.
8. Casella, George (2002). Statistical inference. Roger L. Berger (2nd ed.). Australia: Thomson Learning. Theorem 6.2.25. ISBN   0-534-24312-6. OCLC   46538638.
9. Brown, Lawrence D. (1986). Fundamentals of statistical exponential families : with applications in statistical decision theory. Hayward, Calif.: Institute of Mathematical Statistics. Theorem 2.12. ISBN   0-940600-10-2. OCLC   15986663.
10. Keener, Robert W. (2010). Theoretical statistics : topics for a core course. New York. pp. 47, Example 3.12. ISBN   978-0-387-93839-4. OCLC   676700036.
11. Nielsen, Frank; Garcia, Vincent (2009). "Statistical exponential families: A digest with flash cards". arXiv: [cs.LG].
12. van Garderen, Kees Jan (1997). "Curved Exponential Models in Econometrics". Econometric Theory . 13 (6): 771–790. doi:10.1017/S0266466600006253.
13. Nielsen & Nock 2010, 4. Bregman Divergences and Relative Entropy of Exponential Families.
14. Barndorff-Nielsen 1978, 9.1 Convex duality and exponential families.

### Sources

• Fahrmeir, Ludwig; Tutz, G. (1994). Multivariate Statistical Modelling based on Generalized Linear Models. Springer. pp. 18–22, 345–349. ISBN   0-387-94233-5.
• Keener, Robert W. (2006). Theoretical Statistics: Topics for a Core Course. Springer. pp. 27–28, 32–33. ISBN   978-0-387-93838-7.
• Lehmann, E. L.; Casella, G. (1998). Theory of Point Estimation (2nd ed.). sec. 1.5. ISBN   0-387-98502-6.