State-dependent information

In information theory, state-dependent information is the generic name given to the family of state-dependent measures that in expectation converge to the mutual information.

State-dependent informations often appear in neuroscience applications.

Let ${\displaystyle X}$ and ${\displaystyle Y}$ be random variables and ${\displaystyle y}$ be a state within ${\displaystyle Y}$. The state-dependent information between a random variable ${\displaystyle X}$ and a state ${\displaystyle Y=y}$ is written as ${\displaystyle I(X;Y=y)}$. There are currently three known varieties of state-dependent information. They are usually denoted as ${\displaystyle I_{ss}}$, ${\displaystyle I_{si}}$, and ${\displaystyle I_{ssi}}$.

Specific-Surprise

"Specific-surprise", is defined by a Kullback–Leibler divergence,

${\displaystyle \mathrm {I_{ss}} (X;Y=y)\equiv \mathrm {D_{KL}} [P_{X}\parallel P_{X|y}]}$ .

As a special case of the chain-rule for Kullback-Liebler divergerences, specific-surprise follows the chain-rule for variables. Using ${\displaystyle Z}$ as a random variable, this is specifically,

${\displaystyle \mathrm {I_{ss}} (XZ;Y=y)=\mathrm {I_{ss}} (X;Y=y)+\mathrm {I_{ss}} (Z;Y=y|X)}$.

Intuitively, specific-surprise is thought of as “how much did my beliefs about ${\displaystyle X}$ change upon learning that ${\displaystyle Y=y}$”? Which is zero when there’s no change. It is nonnegative. Specific-surprise has also been called “Bayesian Surprise”.

Specific-Information

"Specific-information", is defined by a difference of entropies,

${\displaystyle \mathrm {I_{si}} (X;Y=y)\equiv \mathrm {H} (X)-\mathrm {H} (X|Y=y)}$ .

Specific-information follows the chain-rule for states. Using a state ${\displaystyle z\in Z}$ as a state of random variable ${\displaystyle Z}$, this is specifically,

${\displaystyle \mathrm {I_{si}} (X;YZ=yz)=\mathrm {I_{si}} (X;Y=y)+\mathrm {I_{si}} (X;Z=z|Y=y)}$ .

Specific-information is interpreted as "how did the uncertainty about ${\displaystyle X}$ change upon learning ${\displaystyle Y=y}$?" This can be in the positive or negative. When ${\displaystyle X}$ follows a uniform distribution, the ${\displaystyle \mathrm {I_{ss}} }$ and ${\displaystyle \mathrm {I_{si}} }$ are equivalent.

State-Specific-Information

The state-specific information, ${\displaystyle \mathrm {I_{ssi}} }$, is a synonym for the Pointwise mutual information.

References

• Timme, Nicholas; Lapish, Christopher (2018). "A Tutorial for Information Theory in Neuroscience". eNeuro. 5 (3). doi:10.1523/ENEURO.0052-18.2018.
• Deweese, Michael; Meister, Markus (1999). "How to measure the information gained from one symbol". Network: Computation in Neural Systems. 10 (4): 325–40. CiteSeerX   10.1.1.553.8013 . doi:10.1088/0954-898X/10/4/303. PMID   10695762.
• Butts, Daniel (2003). "How much information is associated with a particular stimulus?". Network: Computation in Neural Systems. 14 (2): 177–87. doi:10.1088/0954-898X/14/2/301. PMID   12790180.
• Itti, Laurent; Baldi, Pierre (2009). "Bayesian surprise attracts human attention". Vision Research. 49 (10): 1295–1306. doi:10.1016/j.visres.2008.09.007.