![]() | This article may require cleanup to meet Wikipedia's quality standards. The specific problem is: unclear terminology.(June 2017) |
In information theory, the information content, self-information, surprisal, or Shannon information is a basic quantity derived from the probability of a particular event occurring from a random variable. It can be thought of as an alternative way of expressing probability, much like odds or log-odds, but which has particular mathematical advantages in the setting of information theory.
The Shannon information can be interpreted as quantifying the level of "surprise" of a particular outcome. As it is such a basic quantity, it also appears in several other settings, such as the length of a message needed to transmit the event given an optimal source coding of the random variable.
The Shannon information is closely related to entropy , which is the expected value of the self-information of a random variable, quantifying how surprising the random variable is "on average". This is the average amount of self-information an observer would expect to gain about a random variable when measuring it. [1]
The information content can be expressed in various units of information, of which the most common is the "bit" (more formally called the shannon), as explained below.
The term 'perplexity' has been used in language modelling to quantify the uncertainty inherent in a set of prospective events.
Claude Shannon's definition of self-information was chosen to meet several axioms:
The detailed derivation is below, but it can be shown that there is a unique function of probability that meets these three axioms, up to a multiplicative scaling factor. Broadly, given a real number and an event with probability , the information content is defined as follows:
The base b corresponds to the scaling factor above. Different choices of b correspond to different units of information: when b = 2, the unit is the shannon (symbol Sh), often called a 'bit'; when b = e , the unit is the natural unit of information (symbol nat); and when b = 10, the unit is the hartley (symbol Hart).
Formally, given a discrete random variable with probability mass function , the self-information of measuring as outcome is defined as [2]
The use of the notation for self-information above is not universal. Since the notation is also often used for the related quantity of mutual information, many authors use a lowercase for self-entropy instead, mirroring the use of the capital for the entropy.
![]() | This section needs expansion. You can help by adding to it. (October 2018) |
For a given probability space, the measurement of rarer events are intuitively more "surprising", and yield more information content, than more common values. Thus, self-information is a strictly decreasing monotonic function of the probability, or sometimes called an "antitonic" function.
While standard probabilities are represented by real numbers in the interval , self-informations are represented by extended real numbers in the interval . In particular, we have the following, for any choice of logarithmic base:
From this, we can get a few general properties:
The Shannon information is closely related to the log-odds. In particular, given some event , suppose that is the probability of occurring, and that is the probability of not occurring. Then we have the following definition of the log-odds:
This can be expressed as a difference of two Shannon informations:
In other words, the log-odds can be interpreted as the level of surprise when the event doesn't happen, minus the level of surprise when the event does happen.
The information content of two independent events is the sum of each event's information content. This property is known as additivity in mathematics, and sigma additivity in particular in measure and probability theory. Consider two independent random variables with probability mass functions and respectively. The joint probability mass function is
because and are independent. The information content of the outcome is
See § Two independent, identically distributed dice below for an example.
The corresponding property for likelihoods is that the log-likelihood of independent events is the sum of the log-likelihoods of each event. Interpreting log-likelihood as "support" or negative surprisal (the degree to which an event supports a given model: a model is supported by an event to the extent that the event is unsurprising, given the model), this states that independent events add support: the information that the two events together provide for statistical inference is the sum of their independent information.
The Shannon entropy of the random variable above is defined as
by definition equal to the expected information content of measurement of . [3] : 11 [4] : 19–20 The expectation is taken over the discrete values over its support.
Sometimes, the entropy itself is called the "self-information" of the random variable, possibly because the entropy satisfies , where is the mutual information of with itself. [5]
For continuous random variables the corresponding concept is differential entropy.
This measure has also been called surprisal, as it represents the "surprise" of seeing the outcome (a highly improbable outcome is very surprising). This term (as a log-probability measure) was coined by Myron Tribus in his 1961 book Thermostatics and Thermodynamics. [6] [7]
When the event is a random realization (of a variable) the self-information of the variable is defined as the expected value of the self-information of the realization.
Self-information is an example of a proper scoring rule.[ clarification needed ]
Consider the Bernoulli trial of tossing a fair coin . The probabilities of the events of the coin landing as heads and tails (see fair coin and obverse and reverse) are one half each, . Upon measuring the variable as heads, the associated information gain is
so the information gain of a fair coin landing as heads is 1 shannon. [2] Likewise, the information gain of measuring tails is
Suppose we have a fair six-sided die. The value of a dice roll is a discrete uniform random variable with probability mass function
The probability of rolling a 4 is , as for any other valid roll. The information content of rolling a 4 is thus
of information.
Suppose we have two independent, identically distributed random variables each corresponding to an independent fair 6-sided dice roll. The joint distribution of and is
The information content of the random variate is
and can also be calculated by additivity of events
If we receive information about the value of the dice without knowledge of which die had which value, we can formalize the approach with so-called counting variables
for , then and the counts have the multinomial distribution
To verify this, the 6 outcomes correspond to the event and a total probability of 1/6. These are the only events that are faithfully preserved with identity of which dice rolled which outcome because the outcomes are the same. Without knowledge to distinguish the dice rolling the other numbers, the other combinations correspond to one die rolling one number and the other die rolling a different number, each having probability 1/18. Indeed, , as required.
Unsurprisingly, the information content of learning that both dice were rolled as the same particular number is more than the information content of learning that one dice was one number and the other was a different number. Take for examples the events and for . For example, and .
The information contents are
Let be the event that both dice rolled the same value and be the event that the dice differed. Then and . The information contents of the events are
The probability mass or density function (collectively probability measure) of the sum of two independent random variables is the convolution of each probability measure. In the case of independent fair 6-sided dice rolls, the random variable has probability mass function , where represents the discrete convolution. The outcome has probability . Therefore, the information asserted is
Generalizing the § Fair dice roll example above, consider a general discrete uniform random variable (DURV) For convenience, define . The probability mass function is
In general, the values of the DURV need not be integers, or for the purposes of information theory even uniformly spaced; they need only be equiprobable. [2] The information gain of any observation is
If above, degenerates to a constant random variable with probability distribution deterministically given by and probability measure the Dirac measure . The only value can take is deterministically , so the information content of any measurement of is
In general, there is no information gained from measuring a known value. [2]
Generalizing all of the above cases, consider a categorical discrete random variable with support and probability mass function given by
For the purposes of information theory, the values do not have to be numbers; they can be any mutually exclusive events on a measure space of finite measure that has been normalized to a probability measure . Without loss of generality, we can assume the categorical distribution is supported on the set ; the mathematical structure is isomorphic in terms of probability theory and therefore information theory as well.
The information of the outcome is given
From these examples, it is possible to calculate the information of any set of independent DRVs with known distributions by additivity.
By definition, information is transferred from an originating entity possessing the information to a receiving entity only when the receiver had not known the information a priori. If the receiving entity had previously known the content of a message with certainty before receiving the message, the amount of information of the message received is zero. Only when the advance knowledge of the content of the message by the receiver is less than 100% certain does the message actually convey information.
For example, quoting a character (the Hippy Dippy Weatherman) of comedian George Carlin:
Weather forecast for tonight: dark.Continued dark overnight, with widely scattered light by morning. [8]
Assuming that one does not reside near the polar regions, the amount of information conveyed in that forecast is zero because it is known, in advance of receiving the forecast, that darkness always comes with the night.
Accordingly, the amount of self-information contained in a message conveying content informing an occurrence of event, , depends only on the probability of that event.
for some function to be determined below. If , then . If , then .
Further, by definition, the measure of self-information is nonnegative and additive. If a message informing of event is the intersection of two independent events and , then the information of event occurring is that of the compound message of both independent events and occurring. The quantity of information of compound message would be expected to equal the sum of the amounts of information of the individual component messages and respectively:
Because of the independence of events and , the probability of event is
However, applying function results in
Thanks to work on Cauchy's functional equation, the only monotone functions having the property such that
are the logarithm functions . The only operational difference between logarithms of different bases is that of different scaling constants, so we may assume
where is the natural logarithm. Since the probabilities of events are always between 0 and 1 and the information associated with these events must be nonnegative, that requires that .
Taking into account these properties, the self-information associated with outcome with probability is defined as:
The smaller the probability of event , the larger the quantity of self-information associated with the message that the event indeed occurred. If the above logarithm is base 2, the unit of is shannon. This is the most common practice. When using the natural logarithm of base , the unit will be the nat. For the base 10 logarithm, the unit of information is the hartley.
As a quick illustration, the information content associated with an outcome of 4 heads (or any specific outcome) in 4 consecutive tosses of a coin would be 4 shannons (probability 1/16), and the information content associated with getting a result other than the one specified would be ~0.09 shannons (probability 15/16). See above for detailed examples.
In probability theory, the expected value is a generalization of the weighted average. Informally, the expected value is the arithmetic mean of the possible values a random variable can take, weighted by the probability of those outcomes. Since it is obtained through arithmetic, the expected value sometimes may not even be included in the sample data set; it is not the value you would "expect" to get in reality.
In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomenon in terms of its sample space and the probabilities of events.
A random variable is a mathematical formalization of a quantity or object which depends on random events. The term 'random variable' in its mathematical definition refers to neither randomness nor variability but instead is a mathematical function in which
Independence is a fundamental notion in probability theory, as in statistics and the theory of stochastic processes. Two events are independent, statistically independent, or stochastically independent if, informally speaking, the occurrence of one does not affect the probability of occurrence of the other or, equivalently, does not affect the odds. Similarly, two random variables are independent if the realization of one does not affect the probability distribution of the other.
In probability theory, the central limit theorem (CLT) states that, under appropriate conditions, the distribution of a normalized version of the sample mean converges to a standard normal distribution. This holds even if the original variables themselves are not normally distributed. There are several versions of the CLT, each applying in the context of different conditions.
A binary symmetric channel is a common communications channel model used in coding theory and information theory. In this model, a transmitter wishes to send a bit, and the receiver will receive a bit. The bit will be "flipped" with a "crossover probability" of p, and otherwise is received correctly. This model can be applied to varied communication channels such as telephone lines or disk drive storage.
In probability theory, the law of large numbers (LLN) is a mathematical theorem that states that the average of the results obtained from a large number of independent and identical random samples converges to the true value, if it exists. More formally, the LLN states that given a sample of independent and identically distributed values, the sample mean converges to the true mean.
In probability theory and statistics, the beta distribution is a family of continuous probability distributions defined on the interval [0, 1] or in terms of two positive parameters, denoted by alpha (α) and beta (β), that appear as exponents of the variable and its complement to 1, respectively, and control the shape of the distribution.
In mathematics, an indicator function or a characteristic function of a subset of a set is a function that maps elements of the subset to one, and all other elements to zero. That is, if A is a subset of some set X, then if and otherwise, where is a common notation for the indicator function. Other common notations are and
In information theory, the asymptotic equipartition property (AEP) is a general property of the output samples of a stochastic source. It is fundamental to the concept of typical set used in theories of data compression.
In mathematics, the moments of a function are certain quantitative measures related to the shape of the function's graph. If the function represents mass density, then the zeroth moment is the total mass, the first moment is the center of mass, and the second moment is the moment of inertia. If the function is a probability distribution, then the first moment is the expected value, the second central moment is the variance, the third standardized moment is the skewness, and the fourth standardized moment is the kurtosis. The mathematical concept is closely related to the concept of moment in physics.
In probability theory, the conditional expectation, conditional expected value, or conditional mean of a random variable is its expected value evaluated with respect to the conditional probability distribution. If the random variable can take on only a finite number of values, the "conditions" are that the variable can only take on a subset of those values. More formally, in the case when the random variable is defined over a discrete probability space, the "conditions" are a partition of this probability space.
Probability theory and statistics have some commonly used conventions, in addition to standard mathematical notation and mathematical symbols.
Lottery mathematics is used to calculate probabilities of winning or losing a lottery game. It is based primarily on combinatorics, particularly the twelvefold way and combinations without replacement.
In mathematics, a π-system on a set is a collection of certain subsets of such that
The Nakagami distribution or the Nakagami-m distribution is a probability distribution related to the gamma distribution. It is used to model physical phenomena, such as those found in medical ultrasound imaging, communications engineering, meteorology, hydrology, multimedia, and seismology.
In probability theory, a standard probability space, also called Lebesgue–Rokhlin probability space or just Lebesgue space is a probability space satisfying certain assumptions introduced by Vladimir Rokhlin in 1940. Informally, it is a probability space consisting of an interval and/or a finite or countable number of atoms.
In mathematics – specifically, in the theory of stochastic processes – Doob's martingale convergence theorems are a collection of results on the limits of supermartingales, named after the American mathematician Joseph L. Doob. Informally, the martingale convergence theorem typically refers to the result that any supermartingale satisfying a certain boundedness condition must converge. One may think of supermartingales as the random variable analogues of non-increasing sequences; from this perspective, the martingale convergence theorem is a random variable analogue of the monotone convergence theorem, which states that any bounded monotone sequence converges. There are symmetric results for submartingales, which are analogous to non-decreasing sequences.
In probability theory, conditional probability is a measure of the probability of an event occurring, given that another event (by assumption, presumption, assertion or evidence) is already known to have occurred. This particular method relies on event A occurring with some sort of relationship with another event B. In this situation, the event A can be analyzed by a conditional probability with respect to B. If the event of interest is A and the event B is known or assumed to have occurred, "the conditional probability of A given B", or "the probability of A under the condition B", is usually written as P(A|B) or occasionally PB(A). This can also be understood as the fraction of probability B that intersects with A, or the ratio of the probabilities of both events happening to the "given" one happening (how many times A occurs rather than not assuming B has occurred): .
A geometric stable distribution or geo-stable distribution is a type of leptokurtic probability distribution. Geometric stable distributions were introduced in Klebanov, L. B., Maniya, G. M., and Melamed, I. A. (1985). A problem of Zolotarev and analogs of infinitely divisible and stable distributions in a scheme for summing a random number of random variables. These distributions are analogues for stable distributions for the case when the number of summands is random, independent of the distribution of summand, and having geometric distribution. The geometric stable distribution may be symmetric or asymmetric. A symmetric geometric stable distribution is also referred to as a Linnik distribution. The Laplace distribution and asymmetric Laplace distribution are special cases of the geometric stable distribution. The Mittag-Leffler distribution is also a special case of a geometric stable distribution.