Cross-entropy

Last updated May 13, 2024

In information theory, the cross-entropy between two probability distributions $p$ and $q$ , over the same underlying set of events, measures the average number of bits needed to identify an event drawn from the set when the coding scheme used for the set is optimized for an estimated probability distribution $q$ , rather than the true distribution $p$ .

Definition

The cross-entropy of the distribution $q$ relative to a distribution $p$ over a given set is defined as follows:

H(p,q)=-\operatorname {E} _{p}[\log q]

,

where $E_{p}[\cdot ]$ is the expected value operator with respect to the distribution $p$ .

The definition may be formulated using the Kullback–Leibler divergence $D_{\mathrm {KL} }(p\parallel q)$ , divergence of $p$ from $q$ (also known as the relative entropy of $p$ with respect to $q$ ).

H(p,q)=H(p)+D_{\mathrm {KL} }(p\parallel q),

where $H(p)$ is the entropy of $p$ .

For discrete probability distributions $p$ and $q$ with the same support ${\mathcal {X}}$ , this means

H(p,q)=-\sum _{x\in {\mathcal {X}}}p(x)\,\log q(x)

.

(Eq.1)

The situation for continuous distributions is analogous. We have to assume that $p$ and $q$ are absolutely continuous with respect to some reference measure $r$ (usually $r$ is a Lebesgue measure on a Borel σ-algebra). Let $P$ and $Q$ be probability density functions of $p$ and $q$ with respect to $r$ . Then

-\int _{\mathcal {X}}P(x)\,\log Q(x)\,\mathrm {d} \ \!x=\operatorname {E} _{p}[-\log Q],

and therefore

H(p,q)=-\int _{\mathcal {X}}P(x)\,\log Q(x)\,\mathrm {d} \ \!x

.

(Eq.2)

NB: The notation $H(p,q)$ is also used for a different concept, the joint entropy of $p$ and $q$ .

Motivation

In information theory, the Kraft–McMillan theorem establishes that any directly decodable coding scheme for coding a message to identify one value $x_{i}$ out of a set of possibilities $\{x_{1},\ldots ,x_{n}\}$ can be seen as representing an implicit probability distribution $q(x_{i})=\left({\frac {1}{2}}\right)^{\ell _{i}}$ over $\{x_{1},\ldots ,x_{n}\}$ , where $\ell _{i}$ is the length of the code for $x_{i}$ in bits. Therefore, cross-entropy can be interpreted as the expected message-length per datum when a wrong distribution $q$ is assumed while the data actually follows a distribution $p$ . That is why the expectation is taken over the true probability distribution $p$ and not $q$ . Indeed the expected message-length under the true distribution $p$ is

\operatorname {E} _{p}[\ell ]=-\operatorname {E} _{p}\left[{\frac {\ln {q(x)}}{\ln(2)}}\right]=-\operatorname {E} _{p}\left[\log _{2}{q(x)}\right]=-\sum _{x_{i}}p(x_{i})\,\log _{2}q(x_{i})=-\sum _{x}p(x)\,\log _{2}q(x)=H(p,q).

Estimation

There are many situations where cross-entropy needs to be measured but the distribution of $p$ is unknown. An example is language modeling, where a model is created based on a training set $T$ , and then its cross-entropy is measured on a test set to assess how accurate the model is in predicting the test data. In this example, $p$ is the true distribution of words in any corpus, and $q$ is the distribution of words as predicted by the model. Since the true distribution is unknown, cross-entropy cannot be directly calculated. In these cases, an estimate of cross-entropy is calculated using the following formula:

H(T,q)=-\sum _{i=1}^{N}{\frac {1}{N}}\log _{2}q(x_{i})

where $N$ is the size of the test set, and $q(x)$ is the probability of event $x$ estimated from the training set. In other words, $q(x_{i})$ is the probability estimate of the model that the i-th word of the text is $x_{i}$ . The sum is averaged over the $N$ words of the test. This is a Monte Carlo estimate of the true cross-entropy, where the test set is treated as samples from $p(x)$ ^{[ citation needed ]}.

Relation to maximum likelihood

The cross entropy arises in classification problems when introducing a logarithm in the guise of the log-likelihood function.

The section is concerned with the subject of estimation of the probability of different possible discrete outcomes. To this end, denote a parametrized family of distributions by $q_{\theta }$ , with $\theta$ subject to the optimization effort. Consider a given finite sequence of $N$ values $x_{i}$ from a training set, obtained from conditionally independent sampling. The likelihood assigned to any considered parameter $\theta$ of the model is then given by the product over all probabilities $q_{\theta }(X=x_{i})$ . Repeated occurrences are possible, leading to equal factors in the product. If the count of occurrences of the value equal to $x_{i}$ (for some index $i$ ) is denoted by $\#x_{i}$ , then the frequency of that value equals $\#x_{i}/N$ . Denote the latter by $p(X=x_{i})$ , as it may be understood as empirical approximation to the probability distribution underlying the scenario. Further denote by $PP:={\mathrm {e} }^{H(p,q_{\theta })}$ the perplexity, which can be seen to equal ${\textstyle \prod _{x_{i}}}q_{\theta }(X=x_{i})^{-p(X=x_{i})}$ by the calculation rules for the logarithm, and where the product is over the values without double counting. So

{\displaystyle {\mathcal {L}}(\theta

or

{\displaystyle \log {\mathcal {L}}(\theta

Since the logarithm is a monotonically increasing function, it does not affect extremization. So observe that the likelihood maximization amounts to minimization of the cross-entropy.

Cross-entropy minimization

Cross-entropy minimization is frequently used in optimization and rare-event probability estimation. When comparing a distribution $q$ against a fixed reference distribution $p$ , cross-entropy and KL divergence are identical up to an additive constant (since $p$ is fixed): According to the Gibbs' inequality, both take on their minimal values when $p=q$ , which is $0$ for KL divergence, and $\mathrm {H} (p)$ for cross-entropy. In the engineering literature, the principle of minimizing KL divergence (Kullback's "Principle of Minimum Discrimination Information") is often called the Principle of Minimum Cross-Entropy (MCE), or Minxent.

However, as discussed in the article Kullback–Leibler divergence , sometimes the distribution $q$ is the fixed prior reference distribution, and the distribution $p$ is optimized to be as close to $q$ as possible, subject to some constraint. In this case the two minimizations are not equivalent. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by restating cross-entropy to be $D_{\mathrm {KL} }(p\parallel q)$ , rather than $H(p,q)$ . In fact, cross-entropy is another name for relative entropy; see Cover and Thomas^[1] and Good.^[2] On the other hand, $H(p,q)$ does not agree with the literature and can be misleading.

Cross-entropy loss function and logistic regression

Cross-entropy can be used to define a loss function in machine learning and optimization. The true probability $p_{i}$ is the true label, and the given distribution $q_{i}$ is the predicted value of the current model. This is also known as the log loss (or logarithmic loss^[3] or logistic loss );^[4] the terms "log loss" and "cross-entropy loss" are used interchangeably.^[5]

More specifically, consider a binary regression model which can be used to classify observations into two possible classes (often simply labelled $0$ and $1$ ). The output of the model for a given observation, given a vector of input features $x$ , can be interpreted as a probability, which serves as the basis for classifying the observation. In logistic regression, the probability is modeled using the logistic function $g(z)=1/(1+e^{-z})$ where $z$ is some function of the input vector $x$ , commonly just a linear function. The probability of the output $y=1$ is given by

q_{y=1}={\hat {y}}\equiv g(\mathbf {w} \cdot \mathbf {x} )={\frac {1}{1+e^{-\mathbf {w} \cdot \mathbf {x} }}},

where the vector of weights $\mathbf {w}$ is optimized through some appropriate algorithm such as gradient descent. Similarly, the complementary probability of finding the output $y=0$ is simply given by

q_{y=0}=1-{\hat {y}}.

Having set up our notation, $p\in \{y,1-y\}$ and $q\in \{{\hat {y}},1-{\hat {y}}\}$ , we can use cross-entropy to get a measure of dissimilarity between $p$ and $q$ :

H(p,q)\ =\ -\sum _{i}p_{i}\log q_{i}\ =\ -y\log {\hat {y}}-(1-y)\log(1-{\hat {y}}).

Logistic regression typically optimizes the log loss for all the observations on which it is trained, which is the same as optimizing the average cross-entropy in the sample. Other loss functions that penalize errors differently can be also used for training, resulting in models with different final test accuracy.^[6] For example, suppose we have $N$ samples with each sample indexed by $n=1,\dots ,N$ . The average of the loss function is then given by:

{\begin{aligned}J(\mathbf {w} )\ &=\ {\frac {1}{N}}\sum _{n=1}^{N}H(p_{n},q_{n})\ =\ -{\frac {1}{N}}\sum _{n=1}^{N}\ {\bigg [}y_{n}\log {\hat {y}}_{n}+(1-y_{n})\log(1-{\hat {y}}_{n}){\bigg ]}\,,\end{aligned}}

where ${\hat {y}}_{n}\equiv g(\mathbf {w} \cdot \mathbf {x} _{n})=1/(1+e^{-\mathbf {w} \cdot \mathbf {x} _{n}})$ , with $g(z)$ the logistic function as before.

The logistic loss is sometimes called cross-entropy loss. It is also known as log loss.^{[ duplication? ]} (In this case, the binary label is often denoted by {−1,+1}.^[7])

Remark: The gradient of the cross-entropy loss for logistic regression is the same as the gradient of the squared-error loss for linear regression. That is, define

X^{T}={\begin{pmatrix}1&x_{11}&\dots &x_{1p}\\1&x_{21}&\cdots &x_{2p}\\\vdots &\vdots &&\vdots \\1&x_{n1}&\cdots &x_{np}\\\end{pmatrix}}\in \mathbb {R} ^{n\times (p+1)},

{\hat {y_{i}}}={\hat {f}}(x_{i1},\dots ,x_{ip})={\frac {1}{1+\exp(-\beta _{0}-\beta _{1}x_{i1}-\dots -\beta _{p}x_{ip})}},

L({\boldsymbol {\beta }})=-\sum _{i=1}^{N}\left[y_{i}\log {\hat {y}}_{i}+(1-y_{i})\log(1-{\hat {y}}_{i})\right].

Then we have the result

{\frac {\partial }{\partial {\boldsymbol {\beta }}}}L({\boldsymbol {\beta }})=X^{T}({\hat {Y}}-Y).

The proof is as follows. For any ${\hat {y}}_{i}$ , we have

{\frac {\partial }{\partial \beta _{0}}}\ln {\frac {1}{1+e^{-\beta _{0}+k_{0}}}}={\frac {e^{-\beta _{0}+k_{0}}}{1+e^{-\beta _{0}+k_{0}}}},

{\frac {\partial }{\partial \beta _{0}}}\ln \left(1-{\frac {1}{1+e^{-\beta _{0}+k_{0}}}}\right)={\frac {-1}{1+e^{-\beta _{0}+k_{0}}}},

{\begin{aligned}{\frac {\partial }{\partial \beta _{0}}}L({\boldsymbol {\beta }})&=-\sum _{i=1}^{N}\left[{\frac {y_{i}\cdot e^{-\beta _{0}+k_{0}}}{1+e^{-\beta _{0}+k_{0}}}}-(1-y_{i}){\frac {1}{1+e^{-\beta _{0}+k_{0}}}}\right]\\&=-\sum _{i=1}^{N}\left[y_{i}-{\hat {y}}_{i}\right]=\sum _{i=1}^{N}({\hat {y}}_{i}-y_{i}),\end{aligned}}

{\frac {\partial }{\partial \beta _{1}}}\ln {\frac {1}{1+e^{-\beta _{1}x_{i1}+k_{1}}}}={\frac {x_{i1}e^{k_{1}}}{e^{\beta _{1}x_{i1}}+e^{k_{1}}}},

{\frac {\partial }{\partial \beta _{1}}}\ln \left[1-{\frac {1}{1+e^{-\beta _{1}x_{i1}+k_{1}}}}\right]={\frac {-x_{i1}e^{\beta _{1}x_{i1}}}{e^{\beta _{1}x_{i1}}+e^{k_{1}}}},

{\frac {\partial }{\partial \beta _{1}}}L({\boldsymbol {\beta }})=-\sum _{i=1}^{N}x_{i1}(y_{i}-{\hat {y}}_{i})=\sum _{i=1}^{N}x_{i1}({\hat {y}}_{i}-y_{i}).

In a similar way, we eventually obtain the desired result.

Amended Cross-Entropy Cost: An Approach for Encouraging Diversity in Classification Ensemble

In some cases one would like to train an ensemble of models that have diversity so when we combine them it would provide best results.^[8]^[9] Assuming we use a simple ensemble of averaging $K$ classifiers. Then the Amended Cross-Entropy Cost is

e^{k}=H(p,q^{k})-{\frac {\lambda }{K}}\sum _{j\neq k}H(q^{j},q^{k})

where $e^{k}$ is the cost function of the $k^{th}$ classifier, $q^{k}$ is the probability of the $k^{th}$ classifier, $p$ is the true probability that we need to estimate and $\lambda$ is a parameter between 0 and 1 that define the diversity that we would like to establish. When $\lambda =0$ we want each classifier to do its best regardless of the ensemble and when $\lambda =1$ we would like the classifier to be as diverse as possible.

Related Research Articles

The likelihood function is the joint probability mass of observed data viewed as a function of the parameters of a statistical model. Intuitively, the likelihood function $is the probability of observing data assuming is the actual parameter.$

In probability theory and statistics, the exponential distribution or negative exponential distribution is the probability distribution of the distance between events in a Poisson point process, i.e., a process in which events occur continuously and independently at a constant average rate; the distance parameter could be any meaningful mono-dimensional measure of the process, such as time between production errors, or length along a roll of fabric in the weaving manufacturing process. It is a particular case of the gamma distribution. It is the continuous analogue of the geometric distribution, and it has the key property of being memoryless. In addition to being used for the analysis of Poisson point processes it is found in various other contexts.

In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statistical model, the observed data is most probable. The point in the parameter space that maximizes the likelihood function is called the maximum likelihood estimate. The logic of maximum likelihood is both intuitive and flexible, and as such the method has become a dominant means of statistical inference.

In probability theory and statistics, the Weibull distribution is a continuous probability distribution. It models a broad range of random variables, largely in the nature of a time to failure or time between events. Examples are maximum one-day rainfalls and the time a user spends on a web page.

In probability theory and statistics, the beta distribution is a family of continuous probability distributions defined on the interval [0, 1] or in terms of two positive parameters, denoted by alpha (α) and beta (β), that appear as exponents of the variable and its complement to 1, respectively, and control the shape of the distribution.

In probability theory and statistics, the gamma distribution is a versatile two-parameter family of continuous probability distributions. The exponential distribution, Erlang distribution, and chi-squared distribution are special cases of the gamma distribution. There are two equivalent parameterizations in common use:

With a shape parameter $k$ and a scale parameter $θ$
With a shape parameter $and an inverse scale parameter, called a rate parameter.$

In statistics, the logistic model is a statistical model that models the log-odds of an event as a linear combination of one or more independent variables. In regression analysis, logistic regression is estimating the parameters of a logistic model. Formally, in binary logistic regression there is a single binary dependent variable, coded by an indicator variable, where the two values are labeled "0" and "1", while the independent variables can each be a binary variable or a continuous variable. The corresponding probability of the value labeled "1" can vary between 0 and 1, hence the labeling; the function that converts log-odds to probability is the logistic function, hence the name. The unit of measurement for the log-odds scale is called a logit, from logistic unit, hence the alternative names. See § Background and § Definition for formal mathematics, and § Example for a worked example.

<span class="mw-page-title-main">Partition function (statistical mechanics)</span> Function in thermodynamics and statistical physics

In physics, a partition function describes the statistical properties of a system in thermodynamic equilibrium. Partition functions are functions of the thermodynamic state variables, such as the temperature and volume. Most of the aggregate thermodynamic variables of the system, such as the total energy, free energy, entropy, and pressure, can be expressed in terms of the partition function or its derivatives. The partition function is dimensionless.

In probability and statistics, an exponential family is a parametric set of probability distributions of a certain form, specified below. This special form is chosen for mathematical convenience, including the enabling of the user to calculate expectations, covariances using differentiation based on some useful algebraic properties, as well as for generality, as exponential families are in a sense very natural sets of distributions to consider. The term exponential class is sometimes used in place of "exponential family", or the older term Koopman–Darmois family. Sometimes loosely referred to as "the" exponential family, this class of distributions is distinct because they all possess a variety of desirable properties, most importantly the existence of a sufficient statistic.

In mathematical statistics, the Kullback–Leibler (KL) divergence, denoted $, is a type of statistical distance: a measure of how one probability distribution P is different from a second, reference probability distribution Q . A simple interpretation of the KL divergence of P from Q is the expected excess surprise from using Q as a model instead of P when the actual distribution is P . While it is a measure of how different two distributions are, and in some sense is thus a "distance", it is not actually a metric, which is the most familiar and formal type of distance. In particular, it is not symmetric in the two distributions, and does not satisfy the triangle inequality. Instead, in terms of information geometry, it is a type of divergence, a generalization of squared distance, and for certain classes of distributions, it satisfies a generalized Pythagorean theorem.$

Variational Bayesian methods are a family of techniques for approximating intractable integrals arising in Bayesian inference and machine learning. They are typically used in complex statistical models consisting of observed variables as well as unknown parameters and latent variables, with various sorts of relationships among the three types of random variables, as might be described by a graphical model. As typical in Bayesian inference, the parameters and latent variables are grouped together as "unobserved variables". Variational Bayesian methods are primarily used for two purposes:

To provide an analytical approximation to the posterior probability of the unobserved variables, in order to do statistical inference over these variables.
To derive a lower bound for the marginal likelihood of the observed data. This is typically used for performing model selection, the general idea being that a higher marginal likelihood for a given model indicates a better fit of the data by that model and hence a greater probability that the model in question was the one that generated the data.

In statistics, ordinary least squares (OLS) is a type of linear least squares method for choosing the unknown parameters in a linear regression model by the principle of least squares: minimizing the sum of the squares of the differences between the observed dependent variable in the input dataset and the output of the (linear) function of the independent variable.

In probability theory and statistics, the inverse gamma distribution is a two-parameter family of continuous probability distributions on the positive real line, which is the distribution of the reciprocal of a variable distributed according to the gamma distribution.

In statistics and information theory, a maximum entropy probability distribution has entropy that is at least as great as that of all other members of a specified class of probability distributions. According to the principle of maximum entropy, if nothing is known about a distribution except that it belongs to a certain class, then the distribution with the largest entropy should be chosen as the least-informative default. The motivation is twofold: first, maximizing entropy minimizes the amount of prior information built into the distribution; second, many physical systems tend to move towards maximal entropy configurations over time.

In statistics, Poisson regression is a generalized linear model form of regression analysis used to model count data and contingency tables. Poisson regression assumes the response variable Y has a Poisson distribution, and assumes the logarithm of its expected value can be modeled by a linear combination of unknown parameters. A Poisson regression model is sometimes known as a log-linear model, especially when used to model contingency tables.

In directional statistics, the von Mises–Fisher distribution, is a probability distribution on the $-sphere in . If the distribution reduces to the von Mises distribution on the circle.$

A ratio distribution is a probability distribution constructed as the distribution of the ratio of random variables having two other known distributions. Given two random variables X and Y, the distribution of the random variable Z that is formed as the ratio Z = X/Y is a ratio distribution.

In probability theory and statistics, the Poisson distribution is a discrete probability distribution that expresses the probability of a given number of events occurring in a fixed interval of time if these events occur with a known constant mean rate and independently of the time since the last event. It can also be used for the number of events in other types of intervals than time, and in dimension greater than 1.

The hyperbolastic functions, also known as hyperbolastic growth models, are mathematical functions that are used in medical statistical modeling. These models were originally developed to capture the growth dynamics of multicellular tumor spheres, and were introduced in 2005 by Mohammad Tabatabai, David Williams, and Zoran Bursac. The precision of hyperbolastic functions in modeling real world problems is somewhat due to their flexibility in their point of inflection. These functions can be used in a wide variety of modeling problems such as tumor growth, stem cell proliferation, pharma kinetics, cancer growth, sigmoid activation function in neural networks, and epidemiological disease progression or regression.

In machine learning, diffusion models, also known as diffusion probabilistic models or score-based generative models, are a class of latent variable generative models. A diffusion model consists of three major components: the forward process, the reverse process, and the sampling procedure. The goal of diffusion models is to learn a diffusion process that generates a probability distribution for a given dataset from which we can then sample new images. They learn the latent structure of a dataset by modeling the way in which data points diffuse through their latent space.

References

↑ Thomas M. Cover, Joy A. Thomas, Elements of Information Theory, 2nd Edition, Wiley, p. 80
↑ I. J. Good, Maximum entropy for hypothesis formulation, especially for multidimensional contingency tables, Ann. of Math. Statistics, 1963
↑ The Mathematics of Information Coding, Extraction and Distribution, by George Cybenko, Dianne P. O'Leary, Jorma Rissanen, 1999, p. 82
↑ Probability for Machine Learning: Discover How To Harness Uncertainty With Python, Jason Brownlee, 2019, p. 220: "Logistic loss refers to the loss function commonly used to optimize a logistic regression model. It may also be referred to as logarithmic loss (which is confusing) or simply log loss."
↑ sklearn.metrics.log_loss
↑ Noel, Mathew; Banerjee, Arindam; D, Geraldine Bessie Amali; Muthiah-Nakarajan, Venkataraman (March 17, 2023). "Alternate loss functions for classification and robust regression can improve the accuracy of artificial neural networks". arXiv: 2303.09935 .{{cite journal}}: Cite journal requires |journal= (help)
↑ Murphy, Kevin (2012). Machine Learning: A Probabilistic Perspective. MIT. ISBN 978-0262018029.
↑ Shoham, Ron; Permuter, Haim (2019). Amended Cross-Entropy Cost: An Approach for Encouraging Diversity in Classification Ensemble (Brief Announcement). Lecture Notes in Computer Science. Vol. 11527. pp. 202–207. doi:10.1007/978-3-030-20951-3_18. ISBN 978-3-030-20950-6.{{cite book}}: |journal= ignored (help)
↑ Shoham, Haim; Permuter (2020). "Amended Cross Entropy Cost: Framework For Explicit Diversity Encouragement". arXiv: 2007.08140 [cs.LG].{{cite arXiv}}: Unknown parameter |DUPLICATE_first1= ignored (help)