Cross-entropy

Last updated April 18, 2024

In information theory, the cross-entropy between two probability distributions $p$ and $q$ over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set if a coding scheme used for the set is optimized for an estimated probability distribution $q$ , rather than the true distribution $p$ .

Definition

The cross-entropy of the distribution $q$ relative to a distribution $p$ over a given set is defined as follows:

H(p,q)=-\operatorname {E} _{p}[\log q]

,

where $E_{p}[\cdot ]$ is the expected value operator with respect to the distribution $p$ .

The definition may be formulated using the Kullback–Leibler divergence $D_{\mathrm {KL} }(p\parallel q)$ , divergence of $p$ from $q$ (also known as the relative entropy of $p$ with respect to $q$ ).

H(p,q)=H(p)+D_{\mathrm {KL} }(p\parallel q),

where $H(p)$ is the entropy of $p$ .

For discrete probability distributions $p$ and $q$ with the same support ${\mathcal {X}}$ , this means

H(p,q)=-\sum _{x\in {\mathcal {X}}}p(x)\,\log q(x)

.

(Eq.1)

The situation for continuous distributions is analogous. We have to assume that $p$ and $q$ are absolutely continuous with respect to some reference measure $r$ (usually $r$ is a Lebesgue measure on a Borel σ-algebra). Let $P$ and $Q$ be probability density functions of $p$ and $q$ with respect to $r$ . Then

-\int _{\mathcal {X}}P(x)\,\log Q(x)\,\mathrm {d} \ \!x=\operatorname {E} _{p}[-\log Q],

and therefore

H(p,q)=-\int _{\mathcal {X}}P(x)\,\log Q(x)\,\mathrm {d} \ \!x

.

(Eq.2)

NB: The notation $H(p,q)$ is also used for a different concept, the joint entropy of $p$ and $q$ .

Motivation

In information theory, the Kraft–McMillan theorem establishes that any directly decodable coding scheme for coding a message to identify one value $x_{i}$ out of a set of possibilities $\{x_{1},\ldots ,x_{n}\}$ can be seen as representing an implicit probability distribution $q(x_{i})=\left({\frac {1}{2}}\right)^{\ell _{i}}$ over $\{x_{1},\ldots ,x_{n}\}$ , where $\ell _{i}$ is the length of the code for $x_{i}$ in bits. Therefore, cross-entropy can be interpreted as the expected message-length per datum when a wrong distribution $q$ is assumed while the data actually follows a distribution $p$ . That is why the expectation is taken over the true probability distribution $p$ and not $q$ . Indeed the expected message-length under the true distribution $p$ is

\operatorname {E} _{p}[\ell ]=-\operatorname {E} _{p}\left[{\frac {\ln {q(x)}}{\ln(2)}}\right]=-\operatorname {E} _{p}\left[\log _{2}{q(x)}\right]=-\sum _{x_{i}}p(x_{i})\,\log _{2}q(x_{i})=-\sum _{x}p(x)\,\log _{2}q(x)=H(p,q).

Estimation

There are many situations where cross-entropy needs to be measured but the distribution of $p$ is unknown. An example is language modeling, where a model is created based on a training set $T$ , and then its cross-entropy is measured on a test set to assess how accurate the model is in predicting the test data. In this example, $p$ is the true distribution of words in any corpus, and $q$ is the distribution of words as predicted by the model. Since the true distribution is unknown, cross-entropy cannot be directly calculated. In these cases, an estimate of cross-entropy is calculated using the following formula:

H(T,q)=-\sum _{i=1}^{N}{\frac {1}{N}}\log _{2}q(x_{i})

where $N$ is the size of the test set, and $q(x)$ is the probability of event $x$ estimated from the training set. In other words, $q(x_{i})$ is the probability estimate of the model that the i-th word of the text is $x_{i}$ . The sum is averaged over the $N$ words of the test. This is a Monte Carlo estimate of the true cross-entropy, where the test set is treated as samples from $p(x)$ ^{[ citation needed ]}.

Relation to maximum likelihood

The cross entropy arises in classification problems when introducing a logarithm in the guise of the log-likelihood function.

The section is concerned with the subject of estimation of the probability of different possible discrete outcomes. To this end, denote a parametrized family of distributions by $q_{\theta }$ , with $\theta$ subject to the optimization effort. Consider a given finite sequence of $N$ values $x_{i}$ from a training set, obtained from conditionally independent sampling. The likelihood assigned to any considered parameter $\theta$ of the model is then given by the product over all probabilities $q_{\theta }(X=x_{i})$ . Repeated occurrences are possible, leading to equal factors in the product. If the count of occurrences of the value equal to $x_{i}$ (for some index $i$ ) is denoted by $\#x_{i}$ , then the frequency of that value equals $\#x_{i}/N$ . Denote the latter by $p(X=x_{i})$ , as it may be understood as empirical approximation to the probability distribution underlying the scenario. Further denote by $PP:={\mathrm {e} }^{H(p,q_{\theta })}$ the perplexity, which can be seen to equal ${\textstyle \prod _{x_{i}}}q_{\theta }(X=x_{i})^{-p(X=x_{i})}$ by the calculation rules for the logarithm, and where the product is over the values without double counting. So

{\displaystyle {\mathcal {L}}(\theta

or

{\displaystyle \log {\mathcal {L}}(\theta

Since the logarithm is a monotonically increasing function, it does not affect extremization. So observe that the likelihood maximization amounts to minimization of the cross-entropy.

Cross-entropy minimization

Cross-entropy minimization is frequently used in optimization and rare-event probability estimation. When comparing a distribution $q$ against a fixed reference distribution $p$ , cross-entropy and KL divergence are identical up to an additive constant (since $p$ is fixed): According to the Gibbs' inequality, both take on their minimal values when $p=q$ , which is $0$ for KL divergence, and $\mathrm {H} (p)$ for cross-entropy. In the engineering literature, the principle of minimizing KL divergence (Kullback's "Principle of Minimum Discrimination Information") is often called the Principle of Minimum Cross-Entropy (MCE), or Minxent.

However, as discussed in the article Kullback–Leibler divergence , sometimes the distribution $q$ is the fixed prior reference distribution, and the distribution $p$ is optimized to be as close to $q$ as possible, subject to some constraint. In this case the two minimizations are not equivalent. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by restating cross-entropy to be $D_{\mathrm {KL} }(p\parallel q)$ , rather than $H(p,q)$ . In fact, cross-entropy is another name for relative entropy; see Cover and Thomas^[1] and Good.^[2] On the other hand, $H(p,q)$ does not agree with the literature and can be misleading.

Cross-entropy loss function and logistic regression

Cross-entropy can be used to define a loss function in machine learning and optimization. The true probability $p_{i}$ is the true label, and the given distribution $q_{i}$ is the predicted value of the current model. This is also known as the log loss (or logarithmic loss^[3] or logistic loss );^[4] the terms "log loss" and "cross-entropy loss" are used interchangeably.^[5]

More specifically, consider a binary regression model which can be used to classify observations into two possible classes (often simply labelled $0$ and $1$ ). The output of the model for a given observation, given a vector of input features $x$ , can be interpreted as a probability, which serves as the basis for classifying the observation. In logistic regression, the probability is modeled using the logistic function $g(z)=1/(1+e^{-z})$ where $z$ is some function of the input vector $x$ , commonly just a linear function. The probability of the output $y=1$ is given by

q_{y=1}={\hat {y}}\equiv g(\mathbf {w} \cdot \mathbf {x} )={\frac {1}{1+e^{-\mathbf {w} \cdot \mathbf {x} }}},

where the vector of weights $\mathbf {w}$ is optimized through some appropriate algorithm such as gradient descent. Similarly, the complementary probability of finding the output $y=0$ is simply given by

q_{y=0}=1-{\hat {y}}.

Having set up our notation, $p\in \{y,1-y\}$ and $q\in \{{\hat {y}},1-{\hat {y}}\}$ , we can use cross-entropy to get a measure of dissimilarity between $p$ and $q$ :

H(p,q)\ =\ -\sum _{i}p_{i}\log q_{i}\ =\ -y\log {\hat {y}}-(1-y)\log(1-{\hat {y}}).

Logistic regression typically optimizes the log loss for all the observations on which it is trained, which is the same as optimizing the average cross-entropy in the sample. Other loss functions that penalize errors differently can be also used for training, resulting in models with different final test accuracy.^[6] For example, suppose we have $N$ samples with each sample indexed by $n=1,\dots ,N$ . The average of the loss function is then given by:

{\begin{aligned}J(\mathbf {w} )\ &=\ {\frac {1}{N}}\sum _{n=1}^{N}H(p_{n},q_{n})\ =\ -{\frac {1}{N}}\sum _{n=1}^{N}\ {\bigg [}y_{n}\log {\hat {y}}_{n}+(1-y_{n})\log(1-{\hat {y}}_{n}){\bigg ]}\,,\end{aligned}}

where ${\hat {y}}_{n}\equiv g(\mathbf {w} \cdot \mathbf {x} _{n})=1/(1+e^{-\mathbf {w} \cdot \mathbf {x} _{n}})$ , with $g(z)$ the logistic function as before.

The logistic loss is sometimes called cross-entropy loss. It is also known as log loss.^{[ duplication? ]} (In this case, the binary label is often denoted by {−1,+1}.^[7])

Remark: The gradient of the cross-entropy loss for logistic regression is the same as the gradient of the squared-error loss for linear regression. That is, define

X^{T}={\begin{pmatrix}1&x_{11}&\dots &x_{1p}\\1&x_{21}&\cdots &x_{2p}\\\vdots &\vdots &&\vdots \\1&x_{n1}&\cdots &x_{np}\\\end{pmatrix}}\in \mathbb {R} ^{n\times (p+1)},

{\hat {y_{i}}}={\hat {f}}(x_{i1},\dots ,x_{ip})={\frac {1}{1+\exp(-\beta _{0}-\beta _{1}x_{i1}-\dots -\beta _{p}x_{ip})}},

L({\boldsymbol {\beta }})=-\sum _{i=1}^{N}\left[y_{i}\log {\hat {y}}_{i}+(1-y_{i})\log(1-{\hat {y}}_{i})\right].

Then we have the result

{\frac {\partial }{\partial {\boldsymbol {\beta }}}}L({\boldsymbol {\beta }})=X^{T}({\hat {Y}}-Y).

The proof is as follows. For any ${\hat {y}}_{i}$ , we have

{\frac {\partial }{\partial \beta _{0}}}\ln {\frac {1}{1+e^{-\beta _{0}+k_{0}}}}={\frac {e^{-\beta _{0}+k_{0}}}{1+e^{-\beta _{0}+k_{0}}}},

{\frac {\partial }{\partial \beta _{0}}}\ln \left(1-{\frac {1}{1+e^{-\beta _{0}+k_{0}}}}\right)={\frac {-1}{1+e^{-\beta _{0}+k_{0}}}},

{\begin{aligned}{\frac {\partial }{\partial \beta _{0}}}L({\boldsymbol {\beta }})&=-\sum _{i=1}^{N}\left[{\frac {y_{i}\cdot e^{-\beta _{0}+k_{0}}}{1+e^{-\beta _{0}+k_{0}}}}-(1-y_{i}){\frac {1}{1+e^{-\beta _{0}+k_{0}}}}\right]\\&=-\sum _{i=1}^{N}\left[y_{i}-{\hat {y}}_{i}\right]=\sum _{i=1}^{N}({\hat {y}}_{i}-y_{i}),\end{aligned}}

{\frac {\partial }{\partial \beta _{1}}}\ln {\frac {1}{1+e^{-\beta _{1}x_{i1}+k_{1}}}}={\frac {x_{i1}e^{k_{1}}}{e^{\beta _{1}x_{i1}}+e^{k_{1}}}},

{\frac {\partial }{\partial \beta _{1}}}\ln \left[1-{\frac {1}{1+e^{-\beta _{1}x_{i1}+k_{1}}}}\right]={\frac {-x_{i1}e^{\beta _{1}x_{i1}}}{e^{\beta _{1}x_{i1}}+e^{k_{1}}}},

{\frac {\partial }{\partial \beta _{1}}}L({\boldsymbol {\beta }})=-\sum _{i=1}^{N}x_{i1}(y_{i}-{\hat {y}}_{i})=\sum _{i=1}^{N}x_{i1}({\hat {y}}_{i}-y_{i}).

In a similar way, we eventually obtain the desired result.

Amended Cross-Entropy Cost: An Approach for Encouraging Diversity in Classification Ensemble

In some cases one would like to train an ensemble of models that have diversity so when we combine them it would provide best results^[8].^[9] Assuming we use a simple ensemble of averaging $K$ classifiers. Then the Amended Cross-Entropy Cost is

e^{k}=H(p,q^{k})-{\frac {\lambda }{K}}\sum _{j\neq k}H(q^{j},q^{k})

where $e^{k}$ is the cost function of the $k^{th}$ classifier, $q^{k}$ is the probability of the $k^{th}$ classifier, $p$ is the true probability that we need to estimate and $\lambda$ is a parameter between 0 and 1 that define the diversity that we would like to establish. When $\lambda =0$ we want each classifier to do its best regardless of the ensemble and when $\lambda =1$ we would like the classifier to be as diverse as possible.

Related Research Articles

The likelihood function is the joint probability mass of observed data viewed as a function of the parameters of a statistical model. Intuitively, the likelihood function $is the probability of observing data assuming is the actual parameter.$

In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statistical model, the observed data is most probable. The point in the parameter space that maximizes the likelihood function is called the maximum likelihood estimate. The logic of maximum likelihood is both intuitive and flexible, and as such the method has become a dominant means of statistical inference.

In statistics, a statistic is sufficient with respect to a statistical model and its associated unknown parameter if "no other statistic that can be calculated from the same sample provides any additional information as to the value of the parameter". In particular, a statistic is sufficient for a family of probability distributions if the sample from which it is calculated gives no additional information than the statistic, as to which of those probability distributions is the sampling distribution.

In probability theory and statistics, the gamma distribution is a versatile two-parameter family of continuous probability distributions. The exponential distribution, Erlang distribution, and chi-squared distribution are special cases of the gamma distribution. There are two equivalent parameterizations in common use:

With a shape parameter $k$ and a scale parameter $θ$
With a shape parameter $and an inverse scale parameter, called a rate parameter.$

In statistics, the logistic model is a statistical model that models the log-odds of an event as a linear combination of one or more independent variables. In regression analysis, logistic regression is estimating the parameters of a logistic model. Formally, in binary logistic regression there is a single binary dependent variable, coded by an indicator variable, where the two values are labeled "0" and "1", while the independent variables can each be a binary variable or a continuous variable. The corresponding probability of the value labeled "1" can vary between 0 and 1, hence the labeling; the function that converts log-odds to probability is the logistic function, hence the name. The unit of measurement for the log-odds scale is called a logit, from logistic unit, hence the alternative names. See § Background and § Definition for formal mathematics, and § Example for a worked example.

In probability and statistics, an exponential family is a parametric set of probability distributions of a certain form, specified below. This special form is chosen for mathematical convenience, including the enabling of the user to calculate expectations, covariances using differentiation based on some useful algebraic properties, as well as for generality, as exponential families are in a sense very natural sets of distributions to consider. The term exponential class is sometimes used in place of "exponential family", or the older term Koopman–Darmois family. Sometimes loosely referred to as "the" exponential family, this class of distributions is distinct because they all possess a variety of desirable properties, most importantly the existence of a sufficient statistic.

In mathematical statistics, the Fisher information is a way of measuring the amount of information that an observable random variable X carries about an unknown parameter θ of a distribution that models X. Formally, it is the variance of the score, or the expected value of the observed information.

In statistical modeling, regression analysis is a set of statistical processes for estimating the relationships between a dependent variable and one or more independent variables. The most common form of regression analysis is linear regression, in which one finds the line that most closely fits the data according to a specific mathematical criterion. For example, the method of ordinary least squares computes the unique line that minimizes the sum of squared differences between the true data and that line. For specific mathematical reasons, this allows the researcher to estimate the conditional expectation of the dependent variable when the independent variables take on a given set of values. Less common forms of regression use slightly different procedures to estimate alternative location parameters or estimate the conditional expectation across a broader collection of non-linear models.

In mathematical statistics, the Kullback–Leibler (KL) divergence, denoted $, is a type of statistical distance: a measure of how one probability distribution P is different from a second, reference probability distribution Q . A simple interpretation of the KL divergence of P from Q is the expected excess surprise from using Q as a model when the actual distribution is P . While it is a measure of how different two distributions are, and in some sense is thus a "distance", it is not actually a metric, which is the most familiar and formal type of distance. In particular, it is not symmetric in the two distributions, and does not satisfy the triangle inequality. Instead, in terms of information geometry, it is a type of divergence, a generalization of squared distance, and for certain classes of distributions, it satisfies a generalized Pythagorean theorem.$

In statistics, a probit model is a type of regression where the dependent variable can take only two values, for example married or not married. The word is a portmanteau, coming from probability + unit. The purpose of the model is to estimate the probability that an observation with particular characteristics will fall into a specific one of the categories; moreover, classifying observations based on their predicted probabilities is a type of binary classification model.

In statistics, ordinary least squares (OLS) is a type of linear least squares method for choosing the unknown parameters in a linear regression model by the principle of least squares: minimizing the sum of the squares of the differences between the observed dependent variable in the input dataset and the output of the (linear) function of the independent variable.

Differential entropy is a concept in information theory that began as an attempt by Claude Shannon to extend the idea of (Shannon) entropy, a measure of average (surprisal) of a random variable, to continuous probability distributions. Unfortunately, Shannon did not derive this formula, and rather just assumed it was the correct continuous analogue of discrete entropy, but it is not. The actual continuous version of discrete entropy is the limiting density of discrete points (LDDP). Differential entropy is commonly encountered in the literature, but it is a limiting case of the LDDP, and one that loses its fundamental association with discrete entropy.

In statistics, multinomial logistic regression is a classification method that generalizes logistic regression to multiclass problems, i.e. with more than two possible discrete outcomes. That is, it is a model that is used to predict the probabilities of the different possible outcomes of a categorically distributed dependent variable, given a set of independent variables.

In directional statistics, the von Mises–Fisher distribution, is a probability distribution on the $-sphere in . If the distribution reduces to the von Mises distribution on the circle.$

The cross-entropy (CE) method is a Monte Carlo method for importance sampling and optimization. It is applicable to both combinatorial and continuous problems, with either a static or noisy objective.

A ratio distribution is a probability distribution constructed as the distribution of the ratio of random variables having two other known distributions. Given two random variables X and Y, the distribution of the random variable Z that is formed as the ratio Z = X/Y is a ratio distribution.

In probability theory and statistics, the half-normal distribution is a special case of the folded normal distribution.

In probability theory and statistics, the Poisson distribution is a discrete probability distribution that expresses the probability of a given number of events occurring in a fixed interval of time if these events occur with a known constant mean rate and independently of the time since the last event. It can also be used for the number of events in other types of intervals than time, and in dimension greater than 1.

In statistics, the class of vector generalized linear models (VGLMs) was proposed to enlarge the scope of models catered for by generalized linear models (GLMs). In particular, VGLMs allow for response variables outside the classical exponential family and for more than one parameter. Each parameter can be transformed by a link function. The VGLM framework is also large enough to naturally accommodate multiple responses; these are several independent responses each coming from a particular statistical distribution with possibly different parameter values.

The hyperbolastic functions, also known as hyperbolastic growth models, are mathematical functions that are used in medical statistical modeling. These models were originally developed to capture the growth dynamics of multicellular tumor spheres, and were introduced in 2005 by Mohammad Tabatabai, David Williams, and Zoran Bursac. The precision of hyperbolastic functions in modeling real world problems is somewhat due to their flexibility in their point of inflection. These functions can be used in a wide variety of modeling problems such as tumor growth, stem cell proliferation, pharma kinetics, cancer growth, sigmoid activation function in neural networks, and epidemiological disease progression or regression.

References

↑ Thomas M. Cover, Joy A. Thomas, Elements of Information Theory, 2nd Edition, Wiley, p. 80
↑ I. J. Good, Maximum entropy for hypothesis formulation, especially for multidimensional contingency tables, Ann. of Math. Statistics, 1963
↑ The Mathematics of Information Coding, Extraction and Distribution, by George Cybenko, Dianne P. O'Leary, Jorma Rissanen, 1999, p. 82
↑ Probability for Machine Learning: Discover How To Harness Uncertainty With Python, Jason Brownlee, 2019, p. 220: "Logistic loss refers to the loss function commonly used to optimize a logistic regression model. It may also be referred to as logarithmic loss (which is confusing) or simply log loss."
↑ sklearn.metrics.log_loss
↑ Noel, Mathew; Banerjee, Arindam; D, Geraldine Bessie Amali; Muthiah-Nakarajan, Venkataraman (March 17, 2023). "Alternate loss functions for classification and robust regression can improve the accuracy of artificial neural networks". arXiv: 2303.09935 .{{cite journal}}: Cite journal requires |journal= (help)
↑ Murphy, Kevin (2012). Machine Learning: A Probabilistic Perspective. MIT. ISBN 978-0262018029.
↑ Shoham, Ron; Permuter, Haim (2019). Amended Cross-Entropy Cost: An Approach for Encouraging Diversity in Classification Ensemble (Brief Announcement). Lecture Notes in Computer Science. Vol. 11527. pp. 202–207. doi:10.1007/978-3-030-20951-3_18. ISBN 978-3-030-20950-6.{{cite book}}: |journal= ignored (help)
↑ Shoham, Haim; Permuter (2020). "Amended Cross Entropy Cost: Framework For Explicit Diversity Encouragement". arXiv: 2007.08140 [cs.LG].{{cite arXiv}}: Unknown parameter |DUPLICATE_first1= ignored (help)