Independence (probability theory)

Last updated

Independence is a fundamental notion in probability theory, as in statistics and the theory of stochastic processes. Two events are independent, statistically independent, or stochastically independent [1] if, informally speaking, the occurrence of one does not affect the probability of occurrence of the other or, equivalently, does not affect the odds. Similarly, two random variables are independent if the realization of one does not affect the probability distribution of the other.

Contents

When dealing with collections of more than two events, two notions of independence need to be distinguished. The events are called pairwise independent if any two events in the collection are independent of each other, while mutual independence (or collective independence) of events means, informally speaking, that each event is independent of any combination of other events in the collection. A similar notion exists for collections of random variables. Mutual independence implies pairwise independence, but not the other way around. In the standard literature of probability theory, statistics, and stochastic processes, independence without further qualification usually refers to mutual independence.

Definition

For events

Two events

Two events and are independent (often written as or , where the latter symbol often is also used for conditional independence) if and only if their joint probability equals the product of their probabilities: [2] :p. 29 [3] :p. 10

 

 

 

 

(Eq.1)

indicates that two independent events and have common elements in their sample space so that they are not mutually exclusive (mutually exclusive iff ). Why this defines independence is made clear by rewriting with conditional probabilities as the probability at which the event occurs provided that the event has or is assumed to have occurred:

and similarly

Thus, the occurrence of does not affect the probability of , and vice versa. In other words, and are independent to each other. Although the derived expressions may seem more intuitive, they are not the preferred definition, as the conditional probabilities may be undefined if or are 0. Furthermore, the preferred definition makes clear by symmetry that when is independent of , is also independent of .

Odds

Stated in terms of odds, two events are independent if and only if the odds ratio of and is unity (1). Analogously with probability, this is equivalent to the conditional odds being equal to the unconditional odds:

or to the odds of one event, given the other event, being the same as the odds of the event, given the other event not occurring:

The odds ratio can be defined as

or symmetrically for odds of given , and thus is 1 if and only if the events are independent.

More than two events

A finite set of events is pairwise independent if every pair of events is independent [4] that is, if and only if for all distinct pairs of indices ,

 

 

 

 

(Eq.2)

A finite set of events is mutually independent if every event is independent of any intersection of the other events [4] [3] :p. 11that is, if and only if for every and for every k indices ,

 

 

 

 

(Eq.3)

This is called the multiplication rule for independent events. It is not a single condition involving only the product of all the probabilities of all single events; it must hold true for all subsets of events.

For more than two events, a mutually independent set of events is (by definition) pairwise independent; but the converse is not necessarily true. [2] :p. 30

Log probability and information content

Stated in terms of log probability, two events are independent if and only if the log probability of the joint event is the sum of the log probability of the individual events:

In information theory, negative log probability is interpreted as information content, and thus two events are independent if and only if the information content of the combined event equals the sum of information content of the individual events:

See Information content § Additivity of independent events for details.

For real valued random variables

Two random variables

Two random variables and are independent if and only if (iff) the elements of the π-system generated by them are independent; that is to say, for every and , the events and are independent events (as defined above in Eq.1 ). That is, and with cumulative distribution functions and , are independent iff the combined random variable has a joint cumulative distribution function [3] :p. 15

 

 

 

 

(Eq.4)

or equivalently, if the probability densities and and the joint probability density exist,

More than two random variables

A finite set of random variables is pairwise independent if and only if every pair of random variables is independent. Even if the set of random variables is pairwise independent, it is not necessarily mutually independent as defined next.

A finite set of random variables is mutually independent if and only if for any sequence of numbers , the events are mutually independent events (as defined above in Eq.3 ). This is equivalent to the following condition on the joint cumulative distribution function . A finite set of random variables is mutually independent if and only if [3] :p. 16

 

 

 

 

(Eq.5)

Notice that it is not necessary here to require that the probability distribution factorizes for all possible -element subsets as in the case for events. This is not required because e.g. implies .

The measure-theoretically inclined may prefer to substitute events for events in the above definition, where is any Borel set. That definition is exactly equivalent to the one above when the values of the random variables are real numbers. It has the advantage of working also for complex-valued random variables or for random variables taking values in any measurable space (which includes topological spaces endowed by appropriate σ-algebras).

For real valued random vectors

Two random vectors and are called independent if [5] :p. 187

 

 

 

 

(Eq.6)

where and denote the cumulative distribution functions of and and denotes their joint cumulative distribution function. Independence of and is often denoted by . Written component-wise, and are called independent if

For stochastic processes

For one stochastic process

The definition of independence may be extended from random vectors to a stochastic process. Therefore, it is required for an independent stochastic process that the random variables obtained by sampling the process at any times are independent random variables for any . [6] :p. 163

Formally, a stochastic process is called independent, if and only if for all and for all

 

 

 

 

(Eq.7)

where . Independence of a stochastic process is a property within a stochastic process, not between two stochastic processes.

For two stochastic processes

Independence of two stochastic processes is a property between two stochastic processes and that are defined on the same probability space . Formally, two stochastic processes and are said to be independent if for all and for all , the random vectors and are independent, [7] :p. 515 i.e. if

 

 

 

 

(Eq.8)

Independent σ-algebras

The definitions above ( Eq.1 and Eq.2 ) are both generalized by the following definition of independence for σ-algebras. Let be a probability space and let and be two sub-σ-algebras of . and are said to be independent if, whenever and ,

Likewise, a finite family of σ-algebras , where is an index set, is said to be independent if and only if

and an infinite family of σ-algebras is said to be independent if all its finite subfamilies are independent.

The new definition relates to the previous ones very directly:

Using this definition, it is easy to show that if and are random variables and is constant, then and are independent, since the σ-algebra generated by a constant random variable is the trivial σ-algebra . Probability zero events cannot affect independence so independence also holds if is only Pr-almost surely constant.

Properties

Self-independence

Note that an event is independent of itself if and only if

Thus an event is independent of itself if and only if it almost surely occurs or its complement almost surely occurs; this fact is useful when proving zero–one laws. [8]

Expectation and covariance

If and are statistically independent random variables, then the expectation operator has the property

[9] :p. 10

and the covariance is zero, as follows from

The converse does not hold: if two random variables have a covariance of 0 they still may be not independent.

Similarly for two stochastic processes and : If they are independent, then they are uncorrelated. [10] :p. 151

Characteristic function

Two random variables and are independent if and only if the characteristic function of the random vector satisfies

In particular the characteristic function of their sum is the product of their marginal characteristic functions:

though the reverse implication is not true. Random variables that satisfy the latter condition are called subindependent.

Examples

Rolling dice

The event of getting a 6 the first time a die is rolled and the event of getting a 6 the second time are independent. By contrast, the event of getting a 6 the first time a die is rolled and the event that the sum of the numbers seen on the first and second trial is 8 are not independent.

Drawing cards

If two cards are drawn with replacement from a deck of cards, the event of drawing a red card on the first trial and that of drawing a red card on the second trial are independent. By contrast, if two cards are drawn without replacement from a deck of cards, the event of drawing a red card on the first trial and that of drawing a red card on the second trial are not independent, because a deck that has had a red card removed has proportionately fewer red cards.

Pairwise and mutual independence

Pairwise independent, but not mutually independent, events Pairwise independent.svg
Pairwise independent, but not mutually independent, events
Mutually independent events Mutually independent.svg
Mutually independent events

Consider the two probability spaces shown. In both cases, and . The random variables in the first space are pairwise independent because , , and ; but the three random variables are not mutually independent. The random variables in the second space are both pairwise independent and mutually independent. To illustrate the difference, consider conditioning on two events. In the pairwise independent case, although any one event is independent of each of the other two individually, it is not independent of the intersection of the other two:

In the mutually independent case, however,

Triple-independence but no pairwise-independence

It is possible to create a three-event example in which

and yet no two of the three events are pairwise independent (and hence the set of events are not mutually independent). [11] This example shows that mutual independence involves requirements on the products of probabilities of all combinations of events, not just the single events as in this example.

Conditional independence

For events

The events and are conditionally independent given an event when

.

For random variables

Intuitively, two random variables and are conditionally independent given if, once is known, the value of does not add any additional information about . For instance, two measurements and of the same underlying quantity are not independent, but they are conditionally independent given (unless the errors in the two measurements are somehow connected).

The formal definition of conditional independence is based on the idea of conditional distributions. If , , and are discrete random variables, then we define and to be conditionally independent given if

for all , and such that . On the other hand, if the random variables are continuous and have a joint probability density function , then and are conditionally independent given if

for all real numbers , and such that .

If discrete and are conditionally independent given , then

for any , and with . That is, the conditional distribution for given and is the same as that given alone. A similar equation holds for the conditional probability density functions in the continuous case.

Independence can be seen as a special kind of conditional independence, since probability can be seen as a kind of conditional probability given no events.

History

Before 1933, independence, in probability theory, was defined in a verbal manner. For example, de Moivre gave the following definition: “Two events are independent, when they have no connexion one with the other, and that the happening of one neither forwards nor obstructs the happening of the other”. [12] If there are n independent events, the probability of the event, that all of them happen was computed as the product of the probabilities of these n events. Apparently, there was the conviction, that this formula was a consequence of the above definition. (Sometimes this was called the Multiplication Theorem.), Of course, a proof of his assertion cannot work without further more formal tacit assumptions.

The definition of independence, given in this article, became the standard definition (now used in all books) after it appeared in 1933 as part of Kolmogorovs axiomatization of probability. [13] Kolmogorov credited it to S.N. Bernstein, and quoted a publication which had appeared in Russian in 1927. [14]

Unfortunately, both Bernstein and Kolmogorov had not been aware of the work of the Georg Bohlmann. Bohlmann had given the same definition for two events in 1901 [15] and for n events in 1908 [16] In the latter paper, he studied his notion in detail. For example, he gave the first example showing that pairwise independence does not imply imply mutual independence. Even today, Bohlmann is rarely quoted. More about his work can be found in On the contributions of Georg Bohlmann to probability theory from de:Ulrich Krengel. [17]

See also

Related Research Articles

<span class="mw-page-title-main">Entropy (information theory)</span> Expected amount of information needed to specify the output of a stochastic data source

In information theory, the entropy of a random variable is the average level of "information", "surprise", or "uncertainty" inherent to the variable's possible outcomes. Given a discrete random variable , which takes values in the alphabet and is distributed according to :

<span class="mw-page-title-main">Normal distribution</span> Probability distribution

In statistics, a normal distribution or Gaussian distribution is a type of continuous probability distribution for a real-valued random variable. The general form of its probability density function is

In probability theory, the central limit theorem (CLT) states that, under appropriate conditions, the distribution of a normalized version of the sample mean converges to a standard normal distribution. This holds even if the original variables themselves are not normally distributed. There are several versions of the CLT, each applying in the context of different conditions.

<span class="mw-page-title-main">Probability density function</span> Function whose integral over a region describes the probability of an event occurring in that region

In probability theory, a probability density function (PDF), density function, or density of an absolutely continuous random variable, is a function whose value at any given sample in the sample space can be interpreted as providing a relative likelihood that the value of the random variable would be equal to that sample. Probability density is the probability per unit length, in other words, while the absolute likelihood for a continuous random variable to take on any particular value is 0, the value of the PDF at two different samples can be used to infer, in any particular draw of the random variable, how much more likely it is that the random variable would be close to one sample compared to the other sample.

<span class="mw-page-title-main">Multivariate random variable</span> Random variable with multiple component dimensions

In probability, and statistics, a multivariate random variable or random vector is a list or vector of mathematical variables each of whose value is unknown, either because the value has not yet occurred or because there is imperfect knowledge of its value. The individual variables in a random vector are grouped together because they are all part of a single mathematical system — often they represent different properties of an individual statistical unit. For example, while a given person has a specific age, height and weight, the representation of these features of an unspecified person from within a group would be a random vector. Normally each element of a random vector is a real number.

<span class="mw-page-title-main">Multivariate normal distribution</span> Generalization of the one-dimensional normal distribution to higher dimensions

In probability theory and statistics, the multivariate normal distribution, multivariate Gaussian distribution, or joint normal distribution is a generalization of the one-dimensional (univariate) normal distribution to higher dimensions. One definition is that a random vector is said to be k-variate normally distributed if every linear combination of its k components has a univariate normal distribution. Its importance derives mainly from the multivariate central limit theorem. The multivariate normal distribution is often used to describe, at least approximately, any set of (possibly) correlated real-valued random variables each of which clusters around a mean value.

Covariance in probability theory and statistics is a measure of the joint variability of two random variables.

In probability theory and statistics, two real-valued random variables, , , are said to be uncorrelated if their covariance, , is zero. If two variables are uncorrelated, there is no linear relationship between them.

<span class="mw-page-title-main">Covariance matrix</span> Measure of covariance of components of a random vector

In probability theory and statistics, a covariance matrix is a square matrix giving the covariance between each pair of elements of a given random vector.

<span class="mw-page-title-main">Mutual information</span> Measure of dependence between two variables

In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables. More specifically, it quantifies the "amount of information" obtained about one random variable by observing the other random variable. The concept of mutual information is intimately linked to that of entropy of a random variable, a fundamental notion in information theory that quantifies the expected "amount of information" held in a random variable.

In probability theory, the conditional expectation, conditional expected value, or conditional mean of a random variable is its expected value evaluated with respect to the conditional probability distribution. If the random variable can take on only a finite number of values, the "conditions" are that the variable can only take on a subset of those values. More formally, in the case when the random variable is defined over a discrete probability space, the "conditions" are a partition of this probability space.

In probability theory and statistics, the conditional probability distribution is a probability distribution that describes the probability of an outcome given the occurrence of a particular event. Given two jointly distributed random variables and , the conditional probability distribution of given is the probability distribution of when is known to be a particular value; in some cases the conditional probabilities may be expressed as functions containing the unspecified value of as a parameter. When both and are categorical variables, a conditional probability table is typically used to represent the conditional probability. The conditional distribution contrasts with the marginal distribution of a random variable, which is its distribution without reference to the value of the other variable.

<span class="mw-page-title-main">Conditional independence</span> Probability theory concept

In probability theory, conditional independence describes situations wherein an observation is irrelevant or redundant when evaluating the certainty of a hypothesis. Conditional independence is usually formulated in terms of conditional probability, as a special case where the probability of the hypothesis given the uninformative observation is equal to the probability without. If is the hypothesis, and and are observations, conditional independence can be stated as an equality:

<span class="mw-page-title-main">Joint probability distribution</span> Type of probability distribution

Given two random variables that are defined on the same probability space, the joint probability distribution is the corresponding probability distribution on all possible pairs of outputs. The joint distribution can just as well be considered for any given number of random variables. The joint distribution encodes the marginal distributions, i.e. the distributions of each of the individual random variables and the conditional probability distributions, which deal with how the outputs of one random variable are distributed when given information on the outputs of the other random variable(s).

Probability theory and statistics have some commonly used conventions, in addition to standard mathematical notation and mathematical symbols.

In mathematics, a π-system on a set is a collection of certain subsets of such that

In the theory of stochastic processes, filtering describes the problem of determining the state of a system from an incomplete and potentially noisy set of observations. While originally motivated by problems in engineering, filtering found applications in many fields from signal processing to finance.

<span class="mw-page-title-main">Conditional mutual information</span> Information theory

In probability theory, particularly information theory, the conditional mutual information is, in its most basic form, the expected value of the mutual information of two random variables given the value of a third.

In probability theory, a Markov kernel is a map that in the general theory of Markov processes plays the role that the transition matrix does in the theory of Markov processes with a finite state space.

In probability theory, the family of complex normal distributions, denoted or , characterizes complex random variables whose real and imaginary parts are jointly normal. The complex normal family has three parameters: location parameter μ, covariance matrix , and the relation matrix . The standard complex normal is the univariate distribution with , , and .

References

  1. Russell, Stuart; Norvig, Peter (2002). Artificial Intelligence: A Modern Approach . Prentice Hall. p.  478. ISBN   0-13-790395-2.
  2. 1 2 Florescu, Ionut (2014). Probability and Stochastic Processes. Wiley. ISBN   978-0-470-62455-5.
  3. 1 2 3 4 Gallager, Robert G. (2013). Stochastic Processes Theory for Applications. Cambridge University Press. ISBN   978-1-107-03975-9.
  4. 1 2 Feller, W (1971). "Stochastic Independence". An Introduction to Probability Theory and Its Applications. Wiley.
  5. Papoulis, Athanasios (1991). Probability, Random Variables and Stochastic Processes. MCGraw Hill. ISBN   0-07-048477-5.
  6. Hwei, Piao (1997). Theory and Problems of Probability, Random Variables, and Random Processes . McGraw-Hill. ISBN   0-07-030644-3.
  7. Amos Lapidoth (8 February 2017). A Foundation in Digital Communication. Cambridge University Press. ISBN   978-1-107-17732-1.
  8. Durrett, Richard (1996). Probability: theory and examples (Second ed.). page 62
  9. E Jakeman. MODELING FLUCTUATIONS IN SCATTERED WAVES. ISBN   978-0-7503-1005-5.
  10. Park, Kun Il (2018). Fundamentals of Probability and Stochastic Processes with Applications to Communications. Springer. ISBN   978-3-319-68074-3.
  11. George, Glyn, "Testing for the independence of three events," Mathematical Gazette 88, November 2004, 568. PDF
  12. Cited according to: Grinstead and Snell’s Introduction to Probability. In: The CHANCE Project. Version of July 4, 2006.
  13. Kolmogorov, Andrey (1933). Grundbegriffe der Wahrscheinlichkeitsrechnung (in German). Berlin: Julius SpringerTranslation: Kolmogorov, Andrey (1956). Translation:Foundations of the Theory of Probability (2nd ed.). New York: Chelsea. ISBN 978-0-8284-0023-7.
  14. S.N. Bernstein, Probability Theory (Russian), Moscow, 1927 (4 editions, latest 1946)
  15. Georg Bohlmann: Lebensversicherungsmathematik, Encyklop¨adie der mathematischen Wissenschaften, Bd I, Teil 2, Artikel I D 4b (1901), 852–917
  16. Georg Bohlmann: Die Grundbegriffe der Wahrscheinlichkeitsrechnung in ihrer Anwendung auf die Lebensversichrung, Atti del IV. Congr. Int. dei Matem. Rom, Bd. III (1908), 244–278.
  17. de:Ulrich Krengel: On the contributions of Georg Bohlmann to probability theory (PDF; 6,4 MB), Electronic Journal for History of Probability and Statistics, 2011.