Cholesky decomposition

Last updated

In linear algebra, the Cholesky decomposition or Cholesky factorization (pronounced /ʃo-LESS-key/) is a decomposition of a Hermitian, positive-definite matrix into the product of a lower triangular matrix and its conjugate transpose, which is useful for efficient numerical solutions, e.g., Monte Carlo simulations. It was discovered by André-Louis Cholesky for real matrices. When it is applicable, the Cholesky decomposition is roughly twice as efficient as the LU decomposition for solving systems of linear equations. [1]

Linear algebra branch of mathematics

Linear algebra is the branch of mathematics concerning linear equations such as

In the mathematical discipline of linear algebra, a matrix decomposition or matrix factorization is a factorization of a matrix into a product of matrices. There are many different matrix decompositions; each finds use among a particular class of problems.

In mathematics, a Hermitian matrix is a complex square matrix that is equal to its own conjugate transpose—that is, the element in the i-th row and j-th column is equal to the complex conjugate of the element in the j-th row and i-th column, for all indices i and j:

Contents

Statement

The Cholesky decomposition of a Hermitian positive-definite matrix A is a decomposition of the form

where L is a lower triangular matrix with real and positive diagonal entries, and L* denotes the conjugate transpose of L. Every Hermitian positive-definite matrix (and thus also every real-valued symmetric positive-definite matrix) has a unique Cholesky decomposition. [2]

In mathematics, the conjugate transpose or Hermitian transpose of an m-by-n matrix with complex entries is the n-by-m matrix obtained from by taking the transpose and then taking the complex conjugate of each entry.

If the matrix A is Hermitian and positive semi-definite, then it still has a decomposition of the form A = LL* if the diagonal entries of L are allowed to be zero. [3]

When A has only real entries, L has only real entries as well, and the factorization may be written A = LLT. [4]

The Cholesky decomposition is unique when A is positive definite; there is only one lower triangular matrix L with strictly positive diagonal entries such that A = LL*. However, the decomposition need not be unique when A is positive semidefinite.

The converse holds trivially: if A can be written as LL* for some invertible L, lower triangular or otherwise, then A is Hermitian and positive definite.

LDL decomposition

A closely related variant of the classical Cholesky decomposition is the LDL decomposition,

where L is a lower unit triangular (unitriangular) matrix, and D is a diagonal matrix.

This decomposition is related to the classical Cholesky decomposition of the form LL* as follows:

Or, given the classical Cholesky decomposition , the form can be found by using the property that the diagonal of L must be 1 and that both the Cholesky and the form are lower triangles, [5] if S is a diagonal matrix that contains the main diagonal of , then

The LDL variant, if efficiently implemented, requires the same space and computational complexity to construct and use but avoids extracting square roots. [6] Some indefinite matrices for which no Cholesky decomposition exists have an LDL decomposition with negative entries in D. For these reasons, the LDL decomposition may be preferred. For real matrices, the factorization has the form A = LDLT and is often referred to as LDLT decomposition (or LDLT decomposition, or LDL'). It is closely related to the eigendecomposition of real symmetric matrices, A = QΛQT.

Example

Here is the Cholesky decomposition of a symmetric real matrix:

And here is its LDLT decomposition:

Applications

The Cholesky decomposition is mainly used for the numerical solution of linear equations . If A is symmetric and positive definite, then we can solve by first computing the Cholesky decomposition , then solving for y by forward substitution, and finally solving for x by back substitution.

An alternative way to eliminate taking square roots in the decomposition is to compute the Cholesky decomposition , then solving for y, and finally solving .

For linear systems that can be put into symmetric form, the Cholesky decomposition (or its LDL variant) is the method of choice, for superior efficiency and numerical stability. Compared to the LU decomposition, it is roughly twice as efficient. [1]

Linear least squares

Systems of the form Ax = b with A symmetric and positive definite arise quite often in applications. For instance, the normal equations in linear least squares problems are of this form. It may also happen that matrix A comes from an energy functional, which must be positive from physical considerations; this happens frequently in the numerical solution of partial differential equations.

Non-linear optimization

Non-linear multi-variate functions may be minimized over their parameters using variants of Newton's method called quasi-Newton methods. At iteration k, the search steps in a direction defined by solving = for , where is the step direction, is the gradient, and is an approximation to the Hessian matrix formed by repeating rank-1 updates at each iteration. Two well-known update formulas are called Davidon–Fletcher–Powell (DFP) and Broyden–Fletcher–Goldfarb–Shanno (BFGS). Loss of the positive-definite condition through round-off error is avoided if rather than updating an approximation to the inverse of the Hessian, one updates the Cholesky decomposition of an approximation of the Hessian matrix itself.[ citation needed ]

Monte Carlo simulation

The Cholesky decomposition is commonly used in the Monte Carlo method for simulating systems with multiple correlated variables. The covariance matrix is decomposed to give the lower-triangular L. Applying this to a vector of uncorrelated samples u produces a sample vector Lu with the covariance properties of the system being modeled. [7]

The following simplified example shows the economy one gets from the Cholesky decomposition: suppose the goal is to generate two correlated normal variables and with given correlation coefficient . To accomplish that, it is necessary to first generate two uncorrelated Gaussian random variables and , which can be done using a Box–Muller transform. Given the required correlation coefficient , the correlated normal variables can be obtained via the transformations and .

Kalman filters

Unscented Kalman filters commonly use the Cholesky decomposition to choose a set of so-called sigma points. The Kalman filter tracks the average state of a system as a vector x of length N and covariance as an N × N matrix P. The matrix P is always positive semi-definite and can be decomposed into LLT. The columns of L can be added and subtracted from the mean x to form a set of 2N vectors called sigma points. These sigma points completely capture the mean and covariance of the system state.

Matrix inversion

The explicit inverse of a Hermitian matrix can be computed by Cholesky decomposition, in a manner similar to solving linear systems, using operations ( multiplications). [6] The entire inversion can even be efficiently performed in-place.

A non-Hermitian matrix B can also be inverted using the following identity, where BB* will always be Hermitian:

Computation

There are various methods for calculating the Cholesky decomposition. The computational complexity of commonly used algorithms is O(n3) in general.[ citation needed ] The algorithms described below all involve about n3/3 FLOPs (n3/6 multiplications and the same number of additions), where n is the size of the matrix A. Hence, they have half the cost of the LU decomposition, which uses 2n3/3 FLOPs (see Trefethen and Bau 1997).

Which of the algorithms below is faster depends on the details of the implementation. Generally, the first algorithm will be slightly slower because it accesses the data in a less regular manner.

The Cholesky algorithm

The Cholesky algorithm, used to calculate the decomposition matrix L, is a modified version of Gaussian elimination.

The recursive algorithm starts with i := 1 and

A(1) := A.

At step i, the matrix A(i) has the following form:

where Ii−1 denotes the identity matrix of dimension i − 1.

If we now define the matrix Li by

then we can write A(i) as

where

Note that bib*i is an outer product, therefore this algorithm is called the outer-product version in (Golub & Van Loan).

We repeat this for i from 1 to n. After n steps, we get A(n+1) = I. Hence, the lower triangular matrix L we are looking for is calculated as

The Cholesky–Banachiewicz and Cholesky–Crout algorithms

Access pattern (white) and writing pattern (yellow) for the in-place Cholesky--Banachiewicz algorithm on a 5x5 matrix Chol.gif
Access pattern (white) and writing pattern (yellow) for the in-place Cholesky—Banachiewicz algorithm on a 5×5 matrix

If we write out the equation

we obtain the following:

and therefore the following formulas for the entries of L:

The expression under the square root is always positive if A is real and positive-definite.

For complex Hermitian matrix, the following formula applies:

So we can compute the (i, j) entry if we know the entries to the left and above. The computation is usually arranged in either of the following orders:

Either pattern of access allows the entire computation to be performed in-place if desired.

Stability of the computation

Suppose that we want to solve a well-conditioned system of linear equations. If the LU decomposition is used, then the algorithm is unstable unless we use some sort of pivoting strategy. In the latter case, the error depends on the so-called growth factor of the matrix, which is usually (but not always) small.

Now, suppose that the Cholesky decomposition is applicable. As mentioned above, the algorithm will be twice as fast. Furthermore, no pivoting is necessary, and the error will always be small. Specifically, if we want to solve Ax = b, and y denotes the computed solution, then y solves the perturbed system (A + E)y = b, where

Here ||·||2 is the matrix 2-norm, cn is a small constant depending on n, and ε denotes the unit round-off.

One concern with the Cholesky decomposition to be aware of is the use of square roots. If the matrix being factorized is positive definite as required, the numbers under the square roots are always positive in exact arithmetic. Unfortunately, the numbers can become negative because of round-off errors, in which case the algorithm cannot continue. However, this can only happen if the matrix is very ill-conditioned. One way to address this is to add a diagonal correction matrix to the matrix being decomposed in an attempt to promote the positive-definiteness. [8] While this might lessen the accuracy of the decomposition, it can be very favorable for other reasons; for example, when performing Newton's method in optimization, adding a diagonal matrix can improve stability when far from the optimum.

LDL decomposition

An alternative form, eliminating the need to take square roots, is the symmetric indefinite factorization [9]

If A is real, the following recursive relations apply for the entries of D and L:

For complex Hermitian matrix A, the following formula applies:

Again, the pattern of access allows the entire computation to be performed in-place if desired.

Block variant

When used on indefinite matrices, the LDL* factorization is known to be unstable without careful pivoting; [10] specifically, the elements of the factorization can grow arbitrarily. A possible improvement is to perform the factorization on block sub-matrices, commonly 2 × 2: [11]

where every element in the matrices above is a square submatrix. From this, these analogous recursive relations follow:

This involves matrix products and explicit inversion, thus limiting the practical block size.

Updating the decomposition

A task that often arises in practice is that one needs to update a Cholesky decomposition. In more details, one has already computed the Cholesky decomposition of some matrix , then one changes the matrix in some way into another matrix, say , and one wants to compute the Cholesky decomposition of the updated matrix: . The question is now whether one can use the Cholesky decomposition of that was computed before to compute the Cholesky decomposition of .

Rank-one update

The specific case, where the updated matrix is related to the matrix by , is known as a rank-one update.

Here is a little function [12] written in Matlab syntax that realizes a rank-one update:

function[L] =cholupdate(L, x)n=length(x);fork=1:nr=sqrt(L(k,k)^2+x(k)^2);c=r/L(k,k);s=x(k)/L(k,k);L(k,k)=r;ifk<nL((k+1):n,k)=(L((k+1):n,k)+s*x((k+1):n))/c;x((k+1):n)=c*x((k+1):n)-s*L((k+1):n,k);endendend

Rank-one downdate

A rank-one downdate is similar to a rank-one update, except that the addition is replaced by subtraction: . This only works if the new matrix is still positive definite.

The code for the rank-one update shown above can easily be adapted to do a rank-one downdate: one merely needs to replace the two additions in the assignment to r and L((k+1):n, k) by subtractions.

Adding and Removing Rows and Columns

If we have a symmetric and positive definite matrix represented in block form as

And its upper Cholesky factor

Then, for a new matrix which is the same as but with the insertion of new rows and columns

we are interested in finding the Cholesky factorisation of , which we call , without directly computing the entire decomposition.

Writing for the solution of , which can be found easily for triangular matrices, and for the Cholesky decomposition of , the following relations can be found;

These formulas may be used to determine the Cholesky factor after the insertion of rows or columns in any position, if we set the row and column dimensions appropriately (including to zero). The inverse problem, when we have

with known Cholesky decomposition

And we wish to determine the Cholesky factor

of the matrix with rows and columns removed

yields the following rules

Notice that the equations above that involve finding the Cholesky decomposition of a new matrix are all of the form , which allows them to be efficiently calculated using the update and downdate procedures detailed in the previous section. [13]

Proof for positive semi-definite matrices

The above algorithms show that every positive definite matrix has a Cholesky decomposition. This result can be extended to the positive semi-definite case by a limiting argument. The argument is not fully constructive, i.e., it gives no explicit numerical algorithms for computing Cholesky factors.

If is an positive semi-definite matrix, then the sequence consists of positive definite matrices. (This is an immediate consequence of, for example, the spectral mapping theorem for the polynomial functional calculus.) Also,

in operator norm. From the positive definite case, each has Cholesky decomposition . By property of the operator norm,

So is a bounded set in the Banach space of operators, therefore relatively compact (because the underlying vector space is finite-dimensional). Consequently, it has a convergent subsequence, also denoted by , with limit . It can be easily checked that this has the desired properties, i.e. , and is lower triangular with non-negative diagonal entries: for all and ,

Therefore, . Because the underlying vector space is finite-dimensional, all topologies on the space of operators are equivalent. So tends to in norm means tends to entrywise. This in turn implies that, since each is lower triangular with non-negative diagonal entries, is also.

Generalization

The Cholesky factorization can be generalized [ citation needed ] to (not necessarily finite) matrices with operator entries. Let be a sequence of Hilbert spaces. Consider the operator matrix

acting on the direct sum

where each

is a bounded operator. If A is positive (semidefinite) in the sense that for all finite k and for any

we have , then there exists a lower triangular operator matrix L such that A = LL*. One can also take the diagonal entries of L to be positive.

Implementations in programming libraries

See also

Notes

  1. 1 2 Press, William H.; Saul A. Teukolsky; William T. Vetterling; Brian P. Flannery (1992). Numerical Recipes in C: The Art of Scientific Computing (second ed.). Cambridge University England EPress. p. 994. ISBN   0-521-43108-5.
  2. Golub & Van Loan (1996 , p. 143), Horn & Johnson (1985 , p. 407), Trefethen & Bau (1997 , p. 174).
  3. Golub & Van Loan (1996 , p. 147).
  4. Horn & Johnson (1985 , p. 407).
  5. variance – LDLT decomposition from Cholesky decomposition – Cross Validated. Stats.stackexchange.com (2016-04-21). Retrieved on 2016-11-02.
  6. 1 2 Krishnamoorthy, Aravindh; Menon, Deepak (2011). "Matrix Inversion Using Cholesky Decomposition". 1111: 4144. arXiv: 1111.4144 . Bibcode:2011arXiv1111.4144K.
  7. Matlab randn documentation. mathworks.com.
  8. Fang, Haw-ren; O'Leary, Dianne P. (8 August 2006). "Modified Cholesky Algorithms: A Catalog with New Approaches" (PDF).
  9. Watkins, D. (1991). Fundamentals of Matrix Computations. New York: Wiley. p. 84. ISBN   0-471-61414-9.
  10. Nocedal, Jorge (2000). Numerical Optimization. Springer.
  11. Fang, Haw-ren (24 August 2007). "Analysis of Block LDLT Factorizations for Symmetric Indefinite Matrices".
  12. Based on: Stewart, G. W. (1998). Basic decompositions. Philadelphia: Soc. for Industrial and Applied Mathematics. ISBN   0-89871-414-1.
  13. Osborne, M. (2010), Appendix B.

Related Research Articles

In linear algebra, the determinant is a scalar value that can be computed from the elements of a square matrix and encodes certain properties of the linear transformation described by the matrix. The determinant of a matrix A is denoted det(A), det A, or |A|. Geometrically, it can be viewed as the volume scaling factor of the linear transformation described by the matrix. This is also the signed volume of the n-dimensional parallelepiped spanned by the column or row vectors of the matrix. The determinant is positive or negative according to whether the linear mapping preserves or reverses the orientation of n-space.

In linear algebra, a symmetric real matrix is said to be positive definite if the scalar is strictly positive for every non-zero column vector of real numbers. Here denotes the transpose of .. When interpreting as the output of an operator, , that is acting on an input, , the property of positive definiteness implies that the output always has a positive inner product with the input, as often observed in physical processes.

Multivariate normal distribution

In probability theory and statistics, the multivariate normal distribution, multivariate Gaussian distribution, or joint normal distribution is a generalization of the one-dimensional (univariate) normal distribution to higher dimensions. One definition is that a random vector is said to be k-variate normally distributed if every linear combination of its k components has a univariate normal distribution. Its importance derives mainly from the multivariate central limit theorem. The multivariate normal distribution is often used to describe, at least approximately, any set of (possibly) correlated real-valued random variables each of which clusters around a mean value.

Gram–Schmidt process algorithm

In mathematics, particularly linear algebra and numerical analysis, the Gram–Schmidt process is a method for orthonormalising a set of vectors in an inner product space, most commonly the Euclidean space Rn equipped with the standard inner product. The Gram–Schmidt process takes a finite, linearly independent set S = {v1, ..., vk} for kn and generates an orthogonal set S′ = {u1, ..., uk} that spans the same k-dimensional subspace of Rn as S.

In mathematics, matrix multiplication or matrix product is a binary operation that produces a matrix from two matrices with entries in a field, or, more generally, in a ring or even a semiring. The matrix product is designed for representing the composition of linear maps that are represented by matrices. Matrix multiplication is thus a basic tool of linear algebra, and as such has numerous applications in many areas of mathematics, as well as in applied mathematics, statistics, physics, economics, and engineering. In more detail, if A is an n × m matrix and B is an m × p matrix, their matrix product AB is an n × p matrix, in which the m entries across a row of A are multiplied with the m entries down a column of B and summed to produce an entry of AB. When two linear maps are represented by matrices, then the matrix product represents the composition of the two maps.

Singular value decomposition matrix decomposition

In linear algebra, the singular-value decomposition (SVD) is a factorization of a real or complex matrix. It is the generalization of the eigendecomposition of a positive semidefinite normal matrix to any matrix via an extension of the polar decomposition. It has many useful applications in signal processing and statistics.

In linear algebra, an n-by-n square matrix A is called invertible if there exists an n-by-n square matrix B such that

In linear algebra, a QR decomposition of a matrix is a decomposition of a matrix A into a product A = QR of an orthogonal matrix Q and an upper triangular matrix R. QR decomposition is often used to solve the linear least squares problem and is the basis for a particular eigenvalue algorithm, the QR algorithm.

In mathematics, and in particular linear algebra, a pseudoinverseA+ of a matrix A is a generalization of the inverse matrix. The most widely known type of matrix pseudoinverse is the Moore–Penrose inverse, which was independently described by E. H. Moore in 1920, Arne Bjerhammar in 1951, and Roger Penrose in 1955. Earlier, Erik Ivar Fredholm had introduced the concept of a pseudoinverse of integral operators in 1903. When referring to a matrix, the term pseudoinverse, without further specification, is often used to indicate the Moore–Penrose inverse. The term generalized inverse is sometimes used as a synonym for pseudoinverse.

In linear algebra, a generalized eigenvector of an n × n matrix is a vector which satisfies certain criteria which are more relaxed than those for an (ordinary) eigenvector.

In mathematics, the square root of a matrix extends the notion of square root from numbers to matrices. Matrix B is said to be a square root of A if the matrix product BB is equal to A.

In numerical analysis and linear algebra, lower–upper (LU) decomposition or factorization factors a matrix as the product of a lower triangular matrix and an upper triangular matrix. The product sometimes includes a permutation matrix as well. LU decomposition can be viewed as the matrix form of Gaussian elimination. Computers usually solve square systems of linear equations using LU decomposition, and it is also a key step when inverting a matrix or computing the determinant of a matrix. LU decomposition was introduced by mathematician Tadeusz Banachiewicz in 1938.

In mathematics, specifically multilinear algebra, a dyadic or dyadic tensor is a second order tensor, written in a notation that fits in with vector algebra.

In theoretical physics, the Udwadia–Kalaba equation is a method for deriving the equations of motion of a constrained mechanical system. The equation was first described by Firdaus E. Udwadia and Robert E. Kalaba in 1992. The approach is based on Gauss's principle of least constraint. The Udwadia–Kalaba equation applies to a wide class of constraints, both holonomic constraints and nonholonomic ones, as long as they are linear with respect to the accelerations. The equation generalizes to constraint forces that do not obey D'Alembert's principle.

In probability theory, the family of complex normal distributions characterizes complex random variables whose real and imaginary parts are jointly normal. The complex normal family has three parameters: location parameter μ, covariance matrix , and the relation matrix . The standard complex normal is the univariate distribution with , , and .

In the mathematical discipline of numerical linear algebra, a matrix splitting is an expression which represents a given matrix as a sum or difference of matrices. Many iterative methods depend upon the direct solution of matrix equations involving matrices more general than tridiagonal matrices. These matrix equations can often be solved directly and efficiently when written as a matrix splitting. The technique was devised by Richard S. Varga in 1960.

In numerical linear algebra, a convergent matrix is a matrix that converges to the zero matrix under matrix exponentiation.

In pure and applied mathematics, particularly quantum mechanics and computer graphics and their applications, a spherical basis is the basis used to express spherical tensors. The spherical basis closely relates to the description of angular momentum in quantum mechanics and spherical harmonic functions.

Low-rank matrix approximations are essential tools in the application of kernel methods to large-scale learning problems.

In mathematics, the Frobenius inner product is a binary operation that takes two matrices and returns a number. It is often denoted . The operation is a component-wise inner product of two matrices as though they are vectors. The two matrices must have the same dimension—same number of rows and columns—but are not restricted to be square matrices.

References

History of science

Information

Computer code

Use of the matrix in simulation

Online calculators