Interaction information

Last updated May 29, 2024

Venn diagram of information theoretic measures for three variables x, y, and z, represented by the lower left, lower right, and upper circles, respectively. The interaction information is represented by gray region, and it is the only one that can be negative. VennInfo3Var.svg — Venn diagram of information theoretic measures for three variables x, y, and z, represented by the lower left, lower right, and upper circles, respectively. The interaction information is represented by gray region, and it is the only one that can be negative.

The interaction information is a generalization of the mutual information for more than two variables.

There are many names for interaction information, including amount of information,^[1]information correlation,^[2]co-information,^[3] and simply mutual information.^[4] Interaction information expresses the amount of information (redundancy or synergy) bound up in a set of variables, beyond that which is present in any subset of those variables. Unlike the mutual information, the interaction information can be either positive or negative. These functions, their negativity and minima have a direct interpretation in algebraic topology.^[5]

Definition

The conditional mutual information can be used to inductively define the interaction information for any finite number of variables as follows:

I(X_{1};\ldots ;X_{n+1})=I(X_{1};\ldots ;X_{n})-I(X_{1};\ldots ;X_{n}\mid X_{n+1}),

where

I(X_{1};\ldots ;X_{n}\mid X_{n+1})=\mathbb {E} _{X_{n+1}}{\big (}I(X_{1};\ldots ;X_{n})\mid X_{n+1}{\big )}.

Some authors^[6] define the interaction information differently, by swapping the two terms being subtracted in the preceding equation. This has the effect of reversing the sign for an odd number of variables.

For three variables $\{X,Y,Z\}$ , the interaction information $I(X;Y;Z)$ is given by

I(X;Y;Z)=I(X;Y)-I(X;Y\mid Z)

where $I(X;Y)$ is the mutual information between variables $X$ and $Y$ , and $I(X;Y\mid Z)$ is the conditional mutual information between variables $X$ and $Y$ given $Z$ . The interaction information is symmetric, so it does not matter which variable is conditioned on. This is easy to see when the interaction information is written in terms of entropy and joint entropy, as follows:

{\begin{alignedat}{3}I(X;Y;Z)&=&&\;{\bigl (}H(X)+H(Y)+H(Z){\bigr )}\\&&&-{\bigl (}H(X,Y)+H(X,Z)+H(Y,Z){\bigr )}\\&&&+H(X,Y,Z)\end{alignedat}}

In general, for the set of variables ${\mathcal {V}}=\{X_{1},X_{2},\ldots ,X_{n}\}$ , the interaction information can be written in the following form (compare with Kirkwood approximation):

I({\mathcal {V}})=\sum _{{\mathcal {T}}\subseteq {\mathcal {V}}}(-1)^{\left\vert {\mathcal {T}}\right\vert -1}H({\mathcal {T}})

For three variables, the interaction information measures the influence of a variable $Z$ on the amount of information shared between $X$ and $Y$ . Because the term $I(X;Y\mid Z)$ can be larger than $I(X;Y)$ , the interaction information can be negative as well as positive. This will happen, for example, when $X$ and $Y$ are independent but not conditionally independent given $Z$ . Positive interaction information indicates that variable $Z$ inhibits (i.e., accounts for or explains some of) the correlation between $X$ and $Y$ , whereas negative interaction information indicates that variable $Z$ facilitates or enhances the correlation.

Properties

Interaction information is bounded. In the three variable case, it is bounded by^[4]

-\min\{I(X;Y\mid Z),I(Y;Z\mid X),I(X;Z\mid Y)\}\leq I(X;Y;Z)\leq \min\{I(X;Y),I(Y;Z),I(X;Z)\}

If three variables form a Markov chain $X\to Y\to Z$ , then $I(X;Z\mid Y)=0$ , but $I(X;Z)\geq 0$ . Therefore

I(X;Y;Z)=I(X;Z)-I(X;Z\mid Y)=I(X;Z)\geq 0.

Examples

Positive interaction information

Positive interaction information seems much more natural than negative interaction information in the sense that such explanatory effects are typical of common-cause structures. For example, clouds cause rain and also block the sun; therefore, the correlation between rain and darkness is partly accounted for by the presence of clouds, $I({\text{rain}};{\text{dark}}\mid {\text{cloud}})<I({\text{rain}};{\text{dark}})$ . The result is positive interaction information $I({\text{rain}};{\text{dark}};{\text{cloud}})$ .

Negative interaction information

A car's engine can fail to start due to either a dead battery or a blocked fuel pump. Ordinarily, we assume that battery death and fuel pump blockage are independent events, $I({\text{blocked fuel}};{\text{dead battery}})=0$ . But knowing that the car fails to start, if an inspection shows the battery to be in good health, we can conclude that the fuel pump must be blocked. Therefore $I({\text{blocked fuel}};{\text{dead battery}}\mid {\text{engine fails}})>0$ , and the result is negative interaction information.

Difficulty of interpretation

The possible negativity of interaction information can be the source of some confusion.^[3] Many authors have taken zero interaction information as a sign that three or more random variables do not interact, but this interpretation is wrong.^[7]

To see how difficult interpretation can be, consider a set of eight independent binary variables $\{X_{1},X_{2},X_{3},X_{4},X_{5},X_{6},X_{7},X_{8}\}$ . Agglomerate these variables as follows:

{\begin{aligned}Y_{1}&=\{X_{1},X_{2},X_{3},X_{4},X_{5},X_{6},X_{7}\}\\Y_{2}&=\{X_{4},X_{5},X_{6},X_{7}\}\\Y_{3}&=\{X_{5},X_{6},X_{7},X_{8}\}\end{aligned}}

Because the $Y_{i}$ 's overlap each other (are redundant) on the three binary variables $\{X_{5},X_{6},X_{7}\}$ , we would expect the interaction information $I(Y_{1};Y_{2};Y_{3})$ to equal $3$ bits, which it does. However, consider now the agglomerated variables

{\begin{aligned}Y_{1}&=\{X_{1},X_{2},X_{3},X_{4},X_{5},X_{6},X_{7}\}\\Y_{2}&=\{X_{4},X_{5},X_{6},X_{7}\}\\Y_{3}&=\{X_{5},X_{6},X_{7},X_{8}\}\\Y_{4}&=\{X_{7},X_{8}\}\end{aligned}}

These are the same variables as before with the addition of $Y_{4}=\{X_{7},X_{8}\}$ . However, $I(Y_{1};Y_{2};Y_{3};Y_{4})$ in this case is actually equal to $+1$ bit, indicating less redundancy. This is correct in the sense that

{\begin{aligned}I(Y_{1};Y_{2};Y_{3};Y_{4})&=I(Y_{1};Y_{2};Y_{3})-I(Y_{1};Y_{2};Y_{3}|Y_{4})\\&=3-2\\&=1\end{aligned}}

but it remains difficult to interpret.

Uses

Jakulin and Bratko (2003b) provide a machine learning algorithm which uses interaction information.
Killian, Kravitz and Gilson (2007) use mutual information expansion to extract entropy estimates from molecular simulations.^[8]
LeVine and Weinstein (2014) use interaction information and other N-body information measures to quantify allosteric couplings in molecular simulations.^[9]
Moore et al. (2006), Chanda P, Zhang A, Brazeau D, Sucheston L, Freudenheim JL, Ambrosone C, Ramanathan M. (2007) and Chanda P, Sucheston L, Zhang A, Brazeau D, Freudenheim JL, Ambrosone C, Ramanathan M. (2008) demonstrate the use of interaction information for analyzing gene-gene and gene-environmental interactions associated with complex diseases.
Pandey and Sarkar (2017) use interaction information in Cosmology to study the influence of large-scale environments on galaxy properties.
A python package for computing all multivariate interaction or mutual informations, conditional mutual information, joint entropies, total correlations, information distance in a dataset of n variables is available .^[10]

Related Research Articles

In information theory, the entropy of a random variable is the average level of "information", "surprise", or "uncertainty" inherent to the variable's possible outcomes. Given a discrete random variable $, which takes values in the set and is distributed according to, the entropy is$

Independence is a fundamental notion in probability theory, as in statistics and the theory of stochastic processes. Two events are independent, statistically independent, or stochastically independent if, informally speaking, the occurrence of one does not affect the probability of occurrence of the other or, equivalently, does not affect the odds. Similarly, two random variables are independent if the realization of one does not affect the probability distribution of the other.

<span class="mw-page-title-main">Multivariate normal distribution</span> Generalization of the one-dimensional normal distribution to higher dimensions

In probability theory and statistics, the multivariate normal distribution, multivariate Gaussian distribution, or joint normal distribution is a generalization of the one-dimensional (univariate) normal distribution to higher dimensions. One definition is that a random vector is said to be k-variate normally distributed if every linear combination of its k components has a univariate normal distribution. Its importance derives mainly from the multivariate central limit theorem. The multivariate normal distribution is often used to describe, at least approximately, any set of (possibly) correlated real-valued random variables, each of which clusters around a mean value.

In statistics, correlation or dependence is any statistical relationship, whether causal or not, between two random variables or bivariate data. Although in the broadest sense, "correlation" may indicate any type of association, in statistics it usually refers to the degree to which a pair of variables are linearly related. Familiar examples of dependent phenomena include the correlation between the height of parents and their offspring, and the correlation between the price of a good and the quantity the consumers are willing to purchase, as it is depicted in the so-called demand curve.

In mathematics, a measure-preserving dynamical system is an object of study in the abstract formulation of dynamical systems, and ergodic theory in particular. Measure-preserving systems obey the Poincaré recurrence theorem, and are a special case of conservative systems. They provide the formal, mathematical basis for a broad range of physical systems, and, in particular, many systems from classical mechanics as well as systems in thermodynamic equilibrium.

<span class="mw-page-title-main">Mutual information</span> Measure of dependence between two variables

In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables. More specifically, it quantifies the "amount of information" obtained about one random variable by observing the other random variable. The concept of mutual information is intimately linked to that of entropy of a random variable, a fundamental notion in information theory that quantifies the expected "amount of information" held in a random variable.

<span class="mw-page-title-main">Conditional entropy</span> Measure of relative information in probability theory

In information theory, the conditional entropy quantifies the amount of information needed to describe the outcome of a random variable $given that the value of another random variable is known. Here, information is measured in shannons, nats, or hartleys. The entropy of conditioned on is written as .$

<span class="mw-page-title-main">Joint entropy</span> Measure of information in probability and information theory

In information theory, joint entropy is a measure of the uncertainty associated with a set of variables.

In mathematical statistics, the Kullback–Leibler (KL) divergence, denoted $, is a type of statistical distance: a measure of how one probability distribution P is different from a second, reference probability distribution Q . A simple interpretation of the KL divergence of P from Q is the expected excess surprise from using Q as a model instead of P when the actual distribution is P . While it is a measure of how different two distributions are, and in some sense is thus a "distance", it is not actually a metric, which is the most familiar and formal type of distance. In particular, it is not symmetric in the two distributions, and does not satisfy the triangle inequality. Instead, in terms of information geometry, it is a type of divergence, a generalization of squared distance, and for certain classes of distributions, it satisfies a generalized Pythagorean theorem.$

<span class="mw-page-title-main">Dirichlet distribution</span> Probability distribution

In probability and statistics, the Dirichlet distribution (after Peter Gustav Lejeune Dirichlet), often denoted $, is a family of continuous multivariate probability distributions parameterized by a vector of positive reals. It is a multivariate generalization of the beta distribution, hence its alternative name of multivariate beta distribution (MBD) . Dirichlet distributions are commonly used as prior distributions in Bayesian statistics, and in fact, the Dirichlet distribution is the conjugate prior of the categorical distribution and multinomial distribution.$

In information theory, redundancy measures the fractional difference between the entropy $H(X)$ of an ensemble $X$ , and its maximum possible value $. Informally, it is the amount of wasted "space" used to transmit certain data. Data compression is a way to reduce or eliminate unwanted redundancy, while forward error correction is a way of adding desired redundancy for purposes of error detection and correction when communicating over a noisy channel of limited capacity.$

In mathematics, the Bernoulli scheme or Bernoulli shift is a generalization of the Bernoulli process to more than two possible outcomes. Bernoulli schemes appear naturally in symbolic dynamics, and are thus important in the study of dynamical systems. Many important dynamical systems exhibit a repellor that is the product of the Cantor set and a smooth manifold, and the dynamics on the Cantor set are isomorphic to that of the Bernoulli shift. This is essentially the Markov partition. The term shift is in reference to the shift operator, which may be used to study Bernoulli schemes. The Ornstein isomorphism theorem shows that Bernoulli shifts are isomorphic when their entropy is equal.

<span class="mw-page-title-main">Scoring rule</span> Measure for evaluating probabilistic forecasts

In decision theory, a scoring rule provides are evaluation metrics for probabilistic predictions or forecasts. While "regular" loss functions assign a goodness-of-fit score to a predicted value and an observed value, scoring rules assign such a score to a predicted probability distribution and an observed value. On the other hand, a scoring function provides a summary measure for the evaluation of point predictions, i.e. one predicts a property or functional $, like the expectation or the median.$

This article discusses how information theory is related to measure theory.

In probability theory and in particular in information theory, total correlation is one of several generalizations of the mutual information. It is also known as the multivariate constraint or multiinformation. It quantifies the redundancy or dependency among a set of n random variables.

The Kirkwood superposition approximation was introduced in 1935 by John G. Kirkwood as a means of representing a discrete probability distribution. The Kirkwood approximation for a discrete probability density function $is given by$

The entropic vector or entropic function is a concept arising in information theory. It represents the possible values of Shannon's information entropy that subsets of one set of random variables may take. Understanding which vectors are entropic is a way to represent all possible inequalities between entropies of various subsets. For example, for any two random variables $, their joint entropy is at most the sum of the entropies of and of :$

<span class="mw-page-title-main">Conditional mutual information</span> Information theory

In probability theory, particularly information theory, the conditional mutual information is, in its most basic form, the expected value of the mutual information of two random variables given the value of a third.

In information theory, dual total correlation, information rate, excess entropy, or binding information is one of several known non-negative generalizations of mutual information. While total correlation is bounded by the sum entropies of the n elements, the dual total correlation is bounded by the joint-entropy of the n elements. Although well behaved, dual total correlation has received much less attention than the total correlation. A measure known as "TSE-complexity" defines a continuum between the total correlation and dual total correlation.

Partial Information Decomposition is an extension of information theory, that aims to generalize the pairwise relations described by information theory to the interaction of multiple variables.

References

↑ Ting, Hu Kuo (January 1962). "On the Amount of Information". Theory of Probability & Its Applications. 7 (4): 439–447. doi:10.1137/1107041. ISSN 0040-585X.
↑ Wolf, David (May 1, 1996). The Generalization of Mutual Information as the Information between a Set of Variables: The Information Correlation Function Hierarchy and the Information Structure of Multi-Agent Systems (Technical report). NASA Ames Research Center.
1 2 Bell, Anthony (2003). The co-information lattice. 4th Int. Symp. Independent Component Analysis and Blind Source Separation.
1 2 Yeung, R.W. (May 1991). "A new outlook on Shannon's information measures". IEEE Transactions on Information Theory. 37 (3): 466–474. doi:10.1109/18.79902. ISSN 0018-9448.
↑ Baudot, Pierre; Bennequin, Daniel (2015-05-13). "The Homological Nature of Entropy". Entropy. 17 (5): 3253–3318. Bibcode:2015Entrp..17.3253B. doi: 10.3390/e17053253 . ISSN 1099-4300.
↑ McGill, William J. (June 1954). "Multivariate information transmission". Psychometrika. 19 (2): 97–116. doi:10.1007/bf02289159. ISSN 0033-3123. S2CID 126431489.
↑ Krippendorff, Klaus (August 2009). "Information of interactions in complex systems". International Journal of General Systems. 38 (6): 669–680. doi:10.1080/03081070902993160. ISSN 0308-1079. S2CID 13923485.
↑ Killian, Benjamin J.; Yundenfreund Kravitz, Joslyn; Gilson, Michael K. (2007-07-14). "Extraction of configurational entropy from molecular simulations via an expansion approximation". The Journal of Chemical Physics. 127 (2): 024107. Bibcode:2007JChPh.127b4107K. doi:10.1063/1.2746329. ISSN 0021-9606. PMC 2707031 . PMID 17640119.
↑ LeVine, Michael V.; Perez-Aguilar, Jose Manuel; Weinstein, Harel (2014-06-18). "N-body Information Theory (NbIT) Analysis of Rigid-Body Dynamics in Intracellular Loop 2 of the 5-HT2A Receptor". arXiv: 1406.4730 [q-bio.BM].
↑ "InfoTopo: Topological Information Data Analysis. Deep statistical unsupervised and supervised learning - File Exchange - Github". github.com/pierrebaudot/infotopopy/. Retrieved 26 September 2020.

Baudot, P.; Bennequin, D. (2015). "The homological nature of entropy" (PDF). Entropy. 17 (5): 1–66. Bibcode:2015Entrp..17.3253B. doi: 10.3390/e17053253 .
Bell, A J (2003), The co-information lattice
Fano, R M (1961), Transmission of Information: A Statistical Theory of Communications, MIT Press, Cambridge, MA.
Garner W R (1962). Uncertainty and Structure as Psychological Concepts, JohnWiley & Sons, New York.
Han, T S (1978). "Nonnegative entropy measures of multivariate symmetric correlations". Information and Control. 36 (2): 133–156. doi: 10.1016/s0019-9958(78)90275-9 .
Han, T S (1980). "Multiple mutual information and multiple interactions in frequency data". Information and Control. 46: 26–45. doi: 10.1016/s0019-9958(80)90478-7 .
Hu Kuo Tin (1962), On the Amount of Information. Theory Probab. Appl.,7(4), 439-44. PDF
Jakulin A & Bratko I (2003a). Analyzing Attribute Dependencies, in N Lavra\quad{c}, D Gamberger, L Todorovski & H Blockeel, eds, Proceedings of the 7th European Conference on Principles and Practice of Knowledge Discovery in Databases, Springer, Cavtat-Dubrovnik, Croatia, pp. 229–240.
Jakulin A & Bratko I (2003b). Quantifying and visualizing attribute interactions .
Margolin, A; Wang, K; Califano, A; Nemenman, I (2010). "Multivariate dependence and genetic networks inference". IET Syst Biol. 4 (6): 428–440. arXiv: 1001.1681 . doi:10.1049/iet-syb.2010.0009. PMID 21073241. S2CID 14280921.
McGill, W J (1954). "Multivariate information transmission". Psychometrika. 19 (2): 97–116. doi:10.1007/bf02289159. S2CID 126431489.
Moore JH, Gilbert JC, Tsai CT, Chiang FT, Holden T, Barney N, White BC (2006). A flexible computational framework for detecting, characterizing, and interpreting statistical patterns of epistasis in genetic studies of human disease susceptibility, Journal of Theoretical Biology241, 252-261.
Nemenman I (2004). Information theory, multivariate dependence, and genetic network inference .
Pearl, J (1988), Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference, Morgan Kaufmann, San Mateo, CA.
Tsujishita, T (1995), On triple mutual information, Advances in applied mathematics16, 269-274.
Chanda, P; Zhang, A; Brazeau, D; Sucheston, L; Freudenheim, JL; Ambrosone, C; Ramanathan, M (2007). "Information-theoretic metrics for visualizing gene-environment interactions". American Journal of Human Genetics. 81 (5): 939–63. doi:10.1086/521878. PMC 2265645 . PMID 17924337.
Chanda, P; Sucheston, L; Zhang, A; Brazeau, D; Freudenheim, JL; Ambrosone, C; Ramanathan, M (2008). "AMBIENCE: a novel approach and efficient algorithm for identifying informative genetic and environmental associations with complex phenotypes". Genetics. 180 (2): 1191–210. doi:10.1534/genetics.108.088542. PMC 2567367 . PMID 18780753.
Killian, B J; Kravitz, J Y; Gilson, M K (2007). "Extraction of configurational entropy from molecular simulations via an expansion approximation". J. Chem. Phys. 127 (2): 024107. Bibcode:2007JChPh.127b4107K. doi:10.1063/1.2746329. PMC 2707031 . PMID 17640119.
LeVine MV, Weinstein H (2014), NbIT - A New Information Theory-Based Analysis of Allosteric Mechanisms Reveals Residues that Underlie Function in the Leucine Transporter LeuT. PLoS Computational Biology.
Pandey, Biswajit; Sarkar, Suman (2017). "How much a galaxy knows about its large-scale environment?: An information theoretic perspective". Monthly Notices of the Royal Astronomical Society Letters. 467 (1): L6. arXiv: 1611.00283 . Bibcode:2017MNRAS.467L...6P. doi:10.1093/mnrasl/slw250. S2CID 119095496.
https://www3.nd.edu/~jnl/ee80653/Fall2005/tutorials/sunil.pdf
Yeung R W (1992). A new outlook on Shannon's information measures. in IEEE Transactions on Information Theory.

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.

[1] Ting, Hu Kuo (January 1962). "On the Amount of Information". Theory of Probability & Its Applications. 7 (4): 439–447. doi:10.1137/1107041. ISSN 0040-585X.

[2] Wolf, David (May 1, 1996). The Generalization of Mutual Information as the Information between a Set of Variables: The Information Correlation Function Hierarchy and the Information Structure of Multi-Agent Systems (Technical report). NASA Ames Research Center.

[:0-3] 1 2 Bell, Anthony (2003). The co-information lattice. 4th Int. Symp. Independent Component Analysis and Blind Source Separation.

[:1-4] 1 2 Yeung, R.W. (May 1991). "A new outlook on Shannon's information measures". IEEE Transactions on Information Theory. 37 (3): 466–474. doi:10.1109/18.79902. ISSN 0018-9448.

[5] Baudot, Pierre; Bennequin, Daniel (2015-05-13). "The Homological Nature of Entropy". Entropy. 17 (5): 3253–3318. Bibcode:2015Entrp..17.3253B. doi: 10.3390/e17053253 . ISSN 1099-4300.

[6] McGill, William J. (June 1954). "Multivariate information transmission". Psychometrika. 19 (2): 97–116. doi:10.1007/bf02289159. ISSN 0033-3123. S2CID 126431489.

[7] Krippendorff, Klaus (August 2009). "Information of interactions in complex systems". International Journal of General Systems. 38 (6): 669–680. doi:10.1080/03081070902993160. ISSN 0308-1079. S2CID 13923485.

[8] Killian, Benjamin J.; Yundenfreund Kravitz, Joslyn; Gilson, Michael K. (2007-07-14). "Extraction of configurational entropy from molecular simulations via an expansion approximation". The Journal of Chemical Physics. 127 (2): 024107. Bibcode:2007JChPh.127b4107K. doi:10.1063/1.2746329. ISSN 0021-9606. PMC 2707031 . PMID 17640119.

[9] LeVine, Michael V.; Perez-Aguilar, Jose Manuel; Weinstein, Harel (2014-06-18). "N-body Information Theory (NbIT) Analysis of Rigid-Body Dynamics in Intracellular Loop 2 of the 5-HT2A Receptor". arXiv: 1406.4730 [q-bio.BM].

[10] "InfoTopo: Topological Information Data Analysis. Deep statistical unsupervised and supervised learning - File Exchange - Github". github.com/pierrebaudot/infotopopy/. Retrieved 26 September 2020.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]