Maximally informative dimensions

Maximally informative dimensions is a dimensionality reduction technique used in the statistical analyses of neural responses. Specifically, it is a way of projecting a stimulus onto a low-dimensional subspace so that as much information as possible about the stimulus is preserved in the neural response. It is motivated by the fact that natural stimuli are typically confined by their statistics to a lower-dimensional space than that spanned by white noise ^[1] but correctly identifying this subspace using traditional techniques is complicated by the correlations that exist within natural images. Within this subspace, stimulus-response functions may be either linear or nonlinear. The idea was originally developed by Tatyana Sharpee, Nicole C. Rust, and William Bialek in 2003. ^[2]

Mathematical formulation

Neural stimulus-response functions are typically given as the probability of a neuron generating an action potential, or spike, in response to a stimulus ${\displaystyle \mathbf {s} }$. The goal of maximally informative dimensions is to find a small relevant subspace of the much larger stimulus space that accurately captures the salient features of ${\displaystyle \mathbf {s} }$. Let ${\displaystyle D}$ denote the dimensionality of the entire stimulus space and ${\displaystyle K}$ denote the dimensionality of the relevant subspace, such that ${\displaystyle K\ll D}$. We let ${\displaystyle \{\mathbf {v} ^{K}\}}$ denote the basis of the relevant subspace, and ${\displaystyle \mathbf {s} ^{K}}$ the projection of ${\displaystyle \mathbf {s} }$ onto ${\displaystyle \{\mathbf {v} ^{K}\}}$. Using Bayes' theorem we can write out the probability of a spike given a stimulus:

${\displaystyle P(spike|\mathbf {s} ^{K})=P(spike)f(\mathbf {s} ^{K})}$

where

${\displaystyle f(\mathbf {s} ^{K})={\frac {P(\mathbf {s} ^{K}|spike)}{P(\mathbf {s} ^{K})}}}$

is some nonlinear function of the projected stimulus.

In order to choose the optimal ${\displaystyle \{\mathbf {v} ^{K}\}}$, we compare the prior stimulus distribution ${\displaystyle P(\mathbf {s} )}$ with the spike-triggered stimulus distribution ${\displaystyle P(\mathbf {s} |spike)}$ using the Shannon information. The average information (averaged across all presented stimuli) per spike is given by

${\displaystyle I_{spike}=\sum _{\mathbf {s} }P(\mathbf {s} |spike)log_{2}[P(\mathbf {s} |spike)/P(\mathbf {s} )]}$. ^[3]

Now consider a ${\displaystyle K=1}$ dimensional subspace defined by a single direction ${\displaystyle \mathbf {v} }$. The average information conveyed by a single spike about the projection ${\displaystyle x=\mathbf {s} \cdot \mathbf {v} }$ is

${\displaystyle I(\mathbf {v} )=\int dxP_{\mathbf {v} }(x|spike)log2[P_{\mathbf {v} }(x|spike)/P_{\mathbf {v} }(x)]}$,

where the probability distributions are approximated by a measured data set via ${\displaystyle P_{\mathbf {v} }(x|spike)=\langle \delta (x-\mathbf {s} \cdot \mathbf {v} )|spike\rangle _{\mathbf {s} }}$ and ${\displaystyle P_{\mathbf {v} }(x)=\langle \delta (x-\mathbf {s} \cdot \mathbf {v} )\rangle _{\mathbf {s} }}$, i.e., each presented stimulus is represented by a scaled Dirac delta function and the probability distributions are created by averaging over all spike-eliciting stimuli, in the former case, or the entire presented stimulus set, in the latter case. For a given dataset, the average information is a function only of the direction ${\displaystyle \mathbf {v} }$. Under this formulation, the relevant subspace of dimension ${\displaystyle K=1}$ would be defined by the direction ${\displaystyle \mathbf {v} }$ that maximizes the average information ${\displaystyle I(\mathbf {v} )}$.

This procedure can readily be extended to a relevant subspace of dimension ${\displaystyle K>1}$ by defining

${\displaystyle P_{\mathbf {v} ^{K}}(\mathbf {x} |spike)=\langle \prod _{i=1}^{K}\delta (x_{i}-\mathbf {s} \cdot \mathbf {v} _{i})|spike\rangle _{\mathbf {s} }}$

and

${\displaystyle P_{\mathbf {v} ^{K}}(\mathbf {x} )=\langle \prod _{i=1}^{K}\delta (x_{i}-\mathbf {s} \cdot \mathbf {v} _{i})\rangle _{\mathbf {s} }}$

and maximizing ${\displaystyle I({\mathbf {v} ^{K}})}$.

Importance

Maximally informative dimensions does not make any assumptions about the Gaussianity of the stimulus set, which is important, because naturalistic stimuli tend to have non-Gaussian statistics. In this way the technique is more robust than other dimensionality reduction techniques such as spike-triggered covariance analyses.

References

1. D.J. Field. "Relations between the statistics of natural images and the response properties of cortical cells." J. Opt. Soc. am. A 4:2479-2394, 1987.
2. Sharpee, Tatyana, Nicole C. Rust, and William Bialek. Maximally informative dimensions: analyzing neural responses to natural signals . Advances in Neural Information Processing Systems (2003): 277-284.
3. N. Brenner, S. P. Strong, R. Koberle, W. Bialek, and R. R. de Ruyter van Steveninck. "Synergy in a neural code. Neural Comp., 12:1531-1552, 2000.