Multilinear principal component analysis

Last updated

Multilinear principal component analysis (MPCA) is a multilinear extension of principal component analysis (PCA) that is used to analyze M-way arrays, also informally referred to as "data tensors". M-way arrays may be modeled by linear tensor models, such as CANDECOMP/Parafac, or by multilinear tensor models, such as multilinear principal component analysis (MPCA) or multilinear independent component analysis (MICA). The origin of MPCA can be traced back to the tensor rank decomposition introduced by Frank Lauren Hitchcock in 1927; [1] to the Tucker decomposition; [2] and to Peter Kroonenberg's "3-mode PCA" work. [3] In 2000, De Lathauwer et al. restated Tucker and Kroonenberg's work in clear and concise numerical computational terms in their SIAM paper entitled "Multilinear Singular Value Decomposition", [4] (HOSVD) and in their paper "On the Best Rank-1 and Rank-(R1, R2, ..., RN ) Approximation of Higher-order Tensors". [5]

Contents

Circa 2001, Vasilescu and Terzopoulos reframed the data analysis, recognition and synthesis problems as multilinear tensor problems. Tensor factor analysis is the compositional consequence of several causal factors of data formation, and are well suited for multi-modal data tensor analysis. The power of the tensor framework was showcased by analyzing human motion joint angles, facial images or textures in terms of their causal factors of data formation in the following works: Human Motion Signatures [6] (CVPR 2001, ICPR 2002), face recognition – TensorFaces, [7] [8] (ECCV 2002, CVPR 2003, etc.) and computer graphics – TensorTextures [9] (Siggraph 2004).

Historically, MPCA has been referred to as "M-mode PCA", a terminology which was coined by Peter Kroonenberg in 1980. [3] In 2005, Vasilescu and Terzopoulos introduced the Multilinear PCA [10] terminology as a way to better differentiate between linear and multilinear tensor decomposition, as well as, to better differentiate between the work [6] [7] [8] [9] that computed 2nd order statistics associated with each data tensor mode(axis), and subsequent work on Multilinear Independent Component Analysis [10] that computed higher order statistics associated with each tensor mode/axis.

Multilinear PCA may be applied to compute the causal factors of data formation, or as signal processing tool on data tensors whose individual observation have either been vectorized, [6] [7] [8] [9] or whose observations are treated as a collection of column/row observations, "data matrix" and concatenated into a data tensor. The main disadvantage of this approach is that rather than computing all possible combinations

MPCA computes a set of orthonormal matrices associated with each mode of the data tensor which are analogous to the orthonormal row and column space of a matrix computed by the matrix SVD. This transformation aims to capture as high a variance as possible, accounting for as much of the variability in the data associated with each data tensor mode(axis).

The algorithm

The MPCA solution follows the alternating least square (ALS) approach. It is iterative in nature. As in PCA, MPCA works on centered data. Centering is a little more complicated for tensors, and it is problem dependent.

Feature selection

MPCA features: Supervised MPCA is employed in causal factor analysis that facilitates object recognition [11] while a semi-supervised MPCA feature selection is employed in visualization tasks. [12]

Extensions

Various extension of MPCA:

Related Research Articles

<span class="mw-page-title-main">Tensor</span> Algebraic object with geometric applications

In mathematics, a tensor is an algebraic object that describes a multilinear relationship between sets of algebraic objects related to a vector space. Tensors may map between different objects such as vectors, scalars, and even other tensors. There are many types of tensors, including scalars and vectors, dual vectors, multilinear maps between vector spaces, and even some operations such as the dot product. Tensors are defined independent of any basis, although they are often referred to by their components in a basis related to a particular coordinate system; those components form an array, which can be thought of as a high-dimensional matrix.

<span class="mw-page-title-main">Principal component analysis</span> Method of data analysis

Principal component analysis (PCA) is a linear dimensionality reduction technique with applications in exploratory data analysis, visualization and data preprocessing.

<span class="mw-page-title-main">Eigenface</span> Set of eigenvectors used in the computer vision problem of human face recognition

An eigenface is the name given to a set of eigenvectors when used in the computer vision problem of human face recognition. The approach of using eigenfaces for recognition was developed by Sirovich and Kirby and used by Matthew Turk and Alex Pentland in face classification. The eigenvectors are derived from the covariance matrix of the probability distribution over the high-dimensional vector space of face images. The eigenfaces themselves form a basis set of all images used to construct the covariance matrix. This produces dimension reduction by allowing the smaller set of basis images to represent the original training images. Classification can be achieved by comparing how faces are represented by the basis set.

Dimensionality reduction, or dimension reduction, is the transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties of the original data, ideally close to its intrinsic dimension. Working in high-dimensional spaces can be undesirable for many reasons; raw data are often sparse as a consequence of the curse of dimensionality, and analyzing the data is usually computationally intractable. Dimensionality reduction is common in fields that deal with large numbers of observations and/or large numbers of variables, such as signal processing, speech recognition, neuroinformatics, and bioinformatics.

Non-negative matrix factorization, also non-negative matrix approximation is a group of algorithms in multivariate analysis and linear algebra where a matrix V is factorized into (usually) two matrices W and H, with the property that all three matrices have no negative elements. This non-negativity makes the resulting matrices easier to inspect. Also, in applications such as processing of audio spectrograms or muscular activity, non-negativity is inherent to the data being considered. Since the problem is not exactly solvable in general, it is commonly approximated numerically.

In mathematics, the structure tensor, also referred to as the second-moment matrix, is a matrix derived from the gradient of a function. It describes the distribution of the gradient in a specified neighborhood around a point and makes the information invariant to the observing coordinates. The structure tensor is often used in image processing and computer vision.

In multilinear algebra, a tensor decomposition is any scheme for expressing a "data tensor" as a sequence of elementary operations acting on other, often simpler tensors. Many tensor decompositions generalize some matrix decompositions.

In multilinear algebra, the higher-order singular value decomposition (HOSVD) of a tensor is a specific orthogonal Tucker decomposition. It may be regarded as one type of generalization of the matrix singular value decomposition. It has applications in computer vision, computer graphics, machine learning, scientific computing, and signal processing. Some aspects can be traced as far back as F. L. Hitchcock in 1928, but it was L. R. Tucker who developed for third-order tensors the general Tucker decomposition in the 1960s, further advocated by L. De Lathauwer et al. in their Multilinear SVD work that employs the power method, or advocated by Vasilescu and Terzopoulos that developed M-mode SVD a parallel algorithm that employs the matrix SVD.

Tensor software is a class of mathematical software designed for manipulation and calculation with tensors.

<span class="mw-page-title-main">Multilinear subspace learning</span> Approach to dimensionality reduction

Multilinear subspace learning is an approach for disentangling the causal factor of data formation and performing dimensionality reduction. The Dimensionality reduction can be performed on a data tensor that contains a collection of observations have been vectorized, or observations that are treated as matrices and concatenated into a data tensor. Here are some examples of data tensors whose observations are vectorized or whose observations are matrices concatenated into data tensor images (2D/3D), video sequences (3D/4D), and hyperspectral cubes (3D/4D).

Multiway data analysis is a method of analyzing large data sets by representing a collection of observations as a multiway array, . The proper choice of data organization into (C+1)-way array, and analysis techniques can reveal patterns in the underlying data undetected by other methods.

Robust Principal Component Analysis (RPCA) is a modification of the widely used statistical procedure of principal component analysis (PCA) which works well with respect to grossly corrupted observations. A number of different approaches exist for Robust PCA, including an idealized version of Robust PCA, which aims to recover a low-rank matrix L0 from highly corrupted measurements M = L0 +S0. This decomposition in low-rank and sparse matrices can be achieved by techniques such as Principal Component Pursuit method (PCP), Stable PCP, Quantized PCP, Block based PCP, and Local PCP. Then, optimization methods are used such as the Augmented Lagrange Multiplier Method (ALM), Alternating Direction Method (ADM), Fast Alternating Minimization (FAM), Iteratively Reweighted Least Squares (IRLS ) or alternating projections (AP).

<span class="mw-page-title-main">Mode-k flattening</span> Mathematical operation

In multilinear algebra, mode-m flattening, also known as matrixizing, matricizing, or unfolding, is an operation that reshapes a multi-way array into a matrix denoted by .

The following outline is provided as an overview of, and topical guide to, machine learning:

<span class="mw-page-title-main">René Vidal</span> Chilean computer scientist (born 1974)

René Vidal is a Chilean electrical engineer and computer scientist who is known for his research in machine learning, computer vision, medical image computing, robotics, and control theory. He is the Herschel L. Seder Professor of the Johns Hopkins Department of Biomedical Engineering, and the founding director of the Mathematical Institute for Data Science (MINDS).

<span class="mw-page-title-main">Andrzej Cichocki</span> Polish computer scientist

Andrzej Cichocki is a Polish computer scientist, electrical engineer and a professor at the Systems Research Institute of Polish Academy of Science, Warsaw, and Nicolaus Copernicus University (UMK) in Toruń, Poland, and a visiting professor in several universities and research institutes, especially Riken AIP, Japan. Andrzej Cichocki is among world’s top 1% most-cited researchers in the Web of Science (Clarivate) citation index and named on the annual Highly Cited Researchers 2021--2023 lists. He is most noted for his learning algorithms for  Signal separation (BSS), Independent Component Analysis (ICA), Non-negative matrix factorization (NMF), tensor decomposition,  Deep (Multilayer) Matrix Factorizations for ICA, NMF, PCA, neural networks for optimization and signal processing, Tensor network for Machine Learning and Big Data, and brain–computer interfaces. He is the author of several monographs/books and more than 600 scientific peer-reviewed articles.

<span class="mw-page-title-main">L1-norm principal component analysis</span>

L1-norm principal component analysis (L1-PCA) is a general method for multivariate data analysis. L1-PCA is often preferred over standard L2-norm principal component analysis (PCA) when the analyzed data may contain outliers, as it is believed to be robust.

Jiebo Luo is a Chinese-American computer scientist, the Albert Arendt Hopeman Professor of Engineering and Professor of Computer Science at the University of Rochester. He is interested in artificial intelligence, data science and computer vision.

<span class="mw-page-title-main">Michael J. Black</span> American-born computer scientist

Michael J. Black is an American-born computer scientist working in Tübingen, Germany. He is a founding director at the Max Planck Institute for Intelligent Systems where he leads the Perceiving Systems Department in research focused on computer vision, machine learning, and computer graphics. He is also an Honorary Professor at the University of Tübingen.

In machine learning, the term tensor informally refers to two different concepts for organizing and representing data. Data may be organized in a multidimensional array (M-way array), informally referred to as a "data tensor"; however, in the strict mathematical sense, a tensor is a multilinear mapping over a set of domain vector spaces to a range vector space. Observations, such as images, movies, volumes, sounds, and relationships among words and concepts, stored in an M-way array ("data tensor"), may be analyzed either by artificial neural networks or tensor methods.

References

  1. ↑ F. L. Hitchcock (1927). "The expression of a tensor or a polyadic as a sum of products". Journal of Mathematics and Physics . 6 (1–4): 164–189. doi:10.1002/sapm192761164.
  2. ↑ Tucker, Ledyard R (September 1966). "Some mathematical notes on three-mode factor analysis". Psychometrika . 31 (3): 279–311. doi:10.1007/BF02289464. PMID   5221127.
  3. 1 2 P. M. Kroonenberg and J. de Leeuw, Principal component analysis of three-mode data by means of alternating least squares algorithms, Psychometrika, 45 (1980), pp. 69–97.
  4. ↑ Lathauwer, L.D.; Moor, B.D.; Vandewalle, J. (2000). "A multilinear singular value decomposition". SIAM Journal on Matrix Analysis and Applications. 21 (4): 1253–1278. doi:10.1137/s0895479896305696.
  5. ↑ Lathauwer, L. D.; Moor, B. D.; Vandewalle, J. (2000). "On the best rank-1 and rank-(R1, R2, ..., RN ) approximation of higher-order tensors". SIAM Journal on Matrix Analysis and Applications. 21 (4): 1324–1342. doi:10.1137/s0895479898346995.
  6. 1 2 3 M.A.O. Vasilescu (2002) "Human Motion Signatures: Analysis, Synthesis, Recognition," Proceedings of International Conference on Pattern Recognition (ICPR 2002), Vol. 3, Quebec City, Canada, Aug, 2002, 456–460.
  7. 1 2 3 M.A.O. Vasilescu, D. Terzopoulos (2002) "Multilinear Analysis of Image Ensembles: TensorFaces," Proc. 7th European Conference on Computer Vision (ECCV'02), Copenhagen, Denmark, May, 2002, in Computer Vision – ECCV 2002, Lecture Notes in Computer Science, Vol. 2350, A. Heyden et al. (Eds.), Springer-Verlag, Berlin, 2002, 447–460.
  8. 1 2 3 M.A.O. Vasilescu, D. Terzopoulos (2003) "Multilinear Subspace Analysis for Image Ensembles, M. A. O. Vasilescu, D. Terzopoulos, Proc. Computer Vision and Pattern Recognition Conf. (CVPR '03), Vol.2, Madison, WI, June, 2003, 93–99.
  9. 1 2 3 M.A.O. Vasilescu, D. Terzopoulos (2004) "TensorTextures: Multilinear Image-Based Rendering", M. A. O. Vasilescu and D. Terzopoulos, Proc. ACM SIGGRAPH 2004 Conference Los Angeles, CA, August, 2004, in Computer Graphics Proceedings, Annual Conference Series, 2004, 336–342.
  10. 1 2 M. A. O. Vasilescu, D. Terzopoulos (2005) "Multilinear Independent Component Analysis", "Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, June 2005, vol.1, 547–553."
  11. ↑ M. A. O. Vasilescu, D. Terzopoulos (2003) "Multilinear Subspace Analysis of Image Ensembles", "Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’03), Madison, WI, June, 2003"
  12. ↑ H. Lu, H.-L. Eng, M. Thida, and K.N. Plataniotis, "Visualization and Clustering of Crowd Video Content in MPCA Subspace," in Proceedings of the 19th ACM Conference on Information and Knowledge Management (CIKM 2010), Toronto, ON, Canada, October, 2010.
  13. ↑ K. Inoue, K. Hara, K. Urahama, "Robust multilinear principal component analysis", Proc. IEEE Conference on Computer Vision, 2009, pp. 591–597.
  14. ↑ Khan, Suleiman A.; Leppäaho, Eemeli; Kaski, Samuel (2016-06-10). "Bayesian multi-tensor factorization". Machine Learning. 105 (2): 233–253. arXiv: 1412.4679 . doi:10.1007/s10994-016-5563-y. ISSN   0885-6125.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.