Modes of variation

In statistics, modes of variation ^[1] are a continuously indexed set of vectors or functions that are centered at a mean and are used to depict the variation in a population or sample. Typically, variation patterns in the data can be decomposed in descending order of eigenvalues with the directions represented by the corresponding eigenvectors or eigenfunctions. Modes of variation provide a visualization of this decomposition and an efficient description of variation around the mean. Both in principal component analysis (PCA) and in functional principal component analysis (FPCA), modes of variation play an important role in visualizing and describing the variation in the data contributed by each eigencomponent. ^[2] In real-world applications, the eigencomponents and associated modes of variation aid to interpret complex data, especially in exploratory data analysis (EDA).

Formulation

Modes of variation are a natural extension of PCA and FPCA.

Modes of variation in PCA

If a random vector ${\displaystyle \mathbf {X} =(X_{1},X_{2},\cdots ,X_{p})^{T}}$ has the mean vector ${\displaystyle {\boldsymbol {\mu }}_{p}}$, and the covariance matrix ${\displaystyle \mathbf {\Sigma } _{p\times p}}$ with eigenvalues ${\displaystyle \lambda _{1}\geq \lambda _{2}\geq \cdots \geq \lambda _{p}\geq 0}$ and corresponding orthonormal eigenvectors ${\displaystyle \mathbf {e} _{1},\mathbf {e} _{2},\cdots ,\mathbf {e} _{p}}$, by eigendecomposition of a real symmetric matrix, the covariance matrix ${\displaystyle \mathbf {\Sigma } }$ can be decomposed as

${\displaystyle \mathbf {\Sigma } =\mathbf {Q} \mathbf {\Lambda } \mathbf {Q} ^{T},}$

where ${\displaystyle \mathbf {Q} }$ is an orthogonal matrix whose columns are the eigenvectors of ${\displaystyle \mathbf {\Sigma } }$, and ${\displaystyle \mathbf {\Lambda } }$ is a diagonal matrix whose entries are the eigenvalues of ${\displaystyle \mathbf {\Sigma } }$. By the Karhunen–Loève expansion for random vectors, one can express the centered random vector in the eigenbasis

${\displaystyle \mathbf {X} -{\boldsymbol {\mu }}=\sum _{k=1}^{p}\xi _{k}\mathbf {e} _{k},}$

where ${\displaystyle \xi _{k}=\mathbf {e} _{k}^{T}(\mathbf {X} -{\boldsymbol {\mu }})}$ is the principal component ^[3] associated with the ${\displaystyle k}$-th eigenvector ${\displaystyle \mathbf {e} _{k}}$, with the properties

${\displaystyle \operatorname {E} (\xi _{k})=0,\operatorname {Var} (\xi _{k})=\lambda _{k},}$ and ${\displaystyle \operatorname {E} (\xi _{k}\xi _{l})=0\ {\text{for}}\ l\neq k.}$

Then the ${\displaystyle k}$-th mode of variation of ${\displaystyle \mathbf {X} }$ is the set of vectors, indexed by ${\displaystyle \alpha }$,

${\displaystyle \mathbf {m} _{k,\alpha }={\boldsymbol {\mu }}\pm \alpha {\sqrt {\lambda _{k}}}\mathbf {e} _{k},\alpha \in [-A,A],}$

where ${\displaystyle A}$ is typically selected as ${\displaystyle 2\ {\text{or}}\ 3}$.

Modes of variation in FPCA

For a square-integrable random function ${\displaystyle X(t),t\in {\mathcal {T}}\subset R^{p}}$, where typically ${\displaystyle p=1}$ and ${\displaystyle {\mathcal {T}}}$ is an interval, denote the mean function by ${\displaystyle \mu (t)=\operatorname {E} (X(t))}$, and the covariance function by

${\displaystyle G(s,t)=\operatorname {Cov} (X(s),X(t))=\sum _{k=1}^{\infty }\lambda _{k}\varphi _{k}(s)\varphi _{k}(t),}$

where ${\displaystyle \lambda _{1}\geq \lambda _{2}\geq \cdots \geq 0}$ are the eigenvalues and ${\displaystyle \{\varphi _{1},\varphi _{2},\cdots \}}$ are the orthonormal eigenfunctions of the linear Hilbert–Schmidt operator

${\displaystyle G:L^{2}({\mathcal {T}})\rightarrow L^{2}({\mathcal {T}}),\,G(f)=\int _{\mathcal {T}}G(s,t)f(s)ds.}$

By the Karhunen–Loève theorem, one can express the centered function in the eigenbasis,

${\displaystyle X(t)-\mu (t)=\sum _{k=1}^{\infty }\xi _{k}\varphi _{k}(t),}$

where

${\displaystyle \xi _{k}=\int _{\mathcal {T}}(X(t)-\mu (t))\varphi _{k}(t)dt}$

is the ${\displaystyle k}$-th principal component with the properties

${\displaystyle \operatorname {E} (\xi _{k})=0,\operatorname {Var} (\xi _{k})=\lambda _{k},}$ and ${\displaystyle \operatorname {E} (\xi _{k}\xi _{l})=0{\text{ for }}l\neq k.}$

Then the ${\displaystyle k}$-th mode of variation of ${\displaystyle X(t)}$ is the set of functions, indexed by ${\displaystyle \alpha }$,

${\displaystyle m_{k,\alpha }(t)=\mu (t)\pm \alpha {\sqrt {\lambda _{k}}}\varphi _{k}(t),\ t\in {\mathcal {T}},\ \alpha \in [-A,A]}$

that are viewed simultaneously over the range of ${\displaystyle \alpha }$, usually for ${\displaystyle A=2\ {\text{or}}\ 3}$. ^[2]

Estimation

The formulation above is derived from properties of the population. Estimation is needed in real-world applications. The key idea is to estimate mean and covariance.

Modes of variation in PCA

Suppose the data ${\displaystyle \mathbf {x} _{1},\mathbf {x} _{2},\cdots ,\mathbf {x} _{n}}$ represent ${\displaystyle n}$ independent drawings from some ${\displaystyle p}$-dimensional population ${\displaystyle \mathbf {X} }$ with mean vector ${\displaystyle {\boldsymbol {\mu }}}$ and covariance matrix ${\displaystyle \mathbf {\Sigma } }$. These data yield the sample mean vector ${\displaystyle {\overline {\mathbf {x} }}}$, and the sample covariance matrix ${\displaystyle \mathbf {S} }$ with eigenvalue-eigenvector pairs ${\displaystyle ({\hat {\lambda }}_{1},{\hat {\mathbf {e} }}_{1}),({\hat {\lambda }}_{2},{\hat {\mathbf {e} }}_{2}),\cdots ,({\hat {\lambda }}_{p},{\hat {\mathbf {e} }}_{p})}$. Then the ${\displaystyle k}$-th mode of variation of ${\displaystyle \mathbf {X} }$ can be estimated by

${\displaystyle {\hat {\mathbf {m} }}_{k,\alpha }={\overline {\mathbf {x} }}\pm \alpha {\sqrt {{\hat {\lambda }}_{k}}}{\hat {\mathbf {e} }}_{k},\alpha \in [-A,A].}$

Modes of variation in FPCA

Consider ${\displaystyle n}$ realizations ${\displaystyle X_{1}(t),X_{2}(t),\cdots ,X_{n}(t)}$ of a square-integrable random function ${\displaystyle X(t),t\in {\mathcal {T}}}$ with the mean function ${\displaystyle \mu (t)=\operatorname {E} (X(t))}$ and the covariance function ${\displaystyle G(s,t)=\operatorname {Cov} (X(s),X(t))}$. Functional principal component analysis provides methods for the estimation of ${\displaystyle \mu (t)}$ and ${\displaystyle G(s,t)}$ in detail, often involving point wise estimate and interpolation. Substituting estimates for the unknown quantities, the ${\displaystyle k}$-th mode of variation of ${\displaystyle X(t)}$ can be estimated by

${\displaystyle {\hat {m}}_{k,\alpha }(t)={\hat {\mu }}(t)\pm \alpha {\sqrt {{\hat {\lambda }}_{k}}}{\hat {\varphi }}_{k}(t),t\in {\mathcal {T}},\alpha \in [-A,A].}$

Applications

The first and second modes of variation of female mortality data from 41 countries in 2003

The first and second modes of variation of male mortality data from 41 countries in 2003

Modes of variation are useful to visualize and describe the variation patterns in the data sorted by the eigenvalues. In real-world applications, modes of variation associated with eigencomponents allow to interpret complex data, such as the evolution of function traits ^[5] and other infinite-dimensional data. ^[6] To illustrate how modes of variation work in practice, two examples are shown in the graphs to the right, which display the first two modes of variation. The solid curve represents the sample mean function. The dashed, dot-dashed, and dotted curves correspond to modes of variation with ${\displaystyle \alpha =\pm 1,\pm 2,}$ and ${\displaystyle \pm 3}$, respectively.

The first graph displays the first two modes of variation of female mortality data from 41 countries in 2003. ^[4] The object of interest is log hazard function between ages 0 and 100 years. The first mode of variation suggests that the variation of female mortality is smaller for ages around 0 or 100, and larger for ages around 25. An appropriate and intuitive interpretation is that mortality around 25 is driven by accidental death, while around 0 or 100, mortality is related to congenital disease or natural death.

Compared to female mortality data, modes of variation of male mortality data shows higher mortality after around age 20, possibly related to the fact that life expectancy for women is higher than that for men.

References

1. Castro, P. E.; Lawton, W. H.; Sylvestre, E. A. (November 1986). "Principal Modes of Variation for Processes with Continuous Sample Curves". Technometrics. 28 (4): 329. doi:10.2307/1268982. ISSN   0040-1706. JSTOR   1268982.
2. Wang, Jane-Ling; Chiou, Jeng-Min; Müller, Hans-Georg (June 2016). "Functional Data Analysis". Annual Review of Statistics and Its Application . 3 (1): 257–295. doi: 10.1146/annurev-statistics-041715-033624 . ISSN   2326-8298.
3. Kleffe, Jürgen (January 1973). "Principal components of random variables with values in a seperable hilbert space". Mathematische Operationsforschung und Statistik. 4 (5): 391–406. doi:10.1080/02331887308801137. ISSN   0047-6277.
4. "Human Mortality Database". www.mortality.org. Retrieved 2020-03-12.
5. Kirkpatrick, Mark; Heckman, Nancy (August 1989). "A quantitative genetic model for growth, shape, reaction norms, and other infinite-dimensional characters". Journal of Mathematical Biology. 27 (4): 429–450. doi:10.1007/bf00290638. ISSN   0303-6812. PMID   2769086. S2CID   46336613.
6. Jones, M. C.; Rice, John A. (May 1992). "Displaying the Important Features of Large Collections of Similar Curves". The American Statistician. 46 (2): 140–145. doi:10.1080/00031305.1992.10475870. ISSN   0003-1305.