Semi-orthogonal matrix

In linear algebra, a semi-orthogonal matrix is a non-square matrix with real entries where: if the number of columns exceeds the number of rows, then the rows are orthonormal vectors; but if the number of rows exceeds the number of columns, then the columns are orthonormal vectors.

Properties

Let ${\displaystyle A}$ be an ${\displaystyle m\times n}$ semi-orthogonal matrix.

• Either ${\displaystyle A^{\operatorname {T} }A=I{\text{ or }}AA^{\operatorname {T} }=I.\,}$ ^{[1] [2] [3]}
• A semi-orthogonal matrix is an isometry. This means that it preserves the norm either in row space, or column space.
• A semi-orthogonal matrix always has full rank.
• A square matrix is semi-orthogonal if and only if it is an orthogonal matrix.
• A real matrix is semi-orthogonal if and only if its non-zero singular values are all equal to 1.
• A semi-orthogonal matrix A is semi-unitary (either A^†A = I or AA^† = I) and either left-invertible or right-invertible (left-invertible if it has more rows than columns, otherwise right invertible).

Examples

Tall matrix (sub-isometry)

Consider the ${\displaystyle 3\times 2}$ matrix whose columns are orthonormal: ${\displaystyle A={\begin{pmatrix}1&0\\0&1\\0&0\end{pmatrix}}}$ Here, its columns are orthonormal. Therefore, it is semi-orthogonal, which is confirmed by: ${\displaystyle A^{T}A={\begin{pmatrix}1&0&0\\0&1&0\end{pmatrix}}{\begin{pmatrix}1&0\\0&1\\0&0\end{pmatrix}}={\begin{pmatrix}1&0\\0&1\end{pmatrix}}=I_{2}}$

Short matrix

Consider the ${\displaystyle 2\times 3}$ matrix whose rows are orthonormal: ${\displaystyle B={\begin{pmatrix}1&0&0\\0&1&0\end{pmatrix}}}$ Here, its rows are orthonormal. Therefore, it is semi-orthogonal, which is confirmed by: ${\displaystyle BB^{T}={\begin{pmatrix}1&0&0\\0&1&0\end{pmatrix}}{\begin{pmatrix}1&0\\0&1\\0&0\end{pmatrix}}={\begin{pmatrix}1&0\\0&1\end{pmatrix}}=I_{2}}$

Non-example

The following ${\displaystyle 3\times 2}$ matrix has orthogonal, but not orthonormal, columns and is therefore not semi-orthogonal: ${\displaystyle C={\begin{pmatrix}2&0\\0&1\\0&0\end{pmatrix}}}$ The calculation confirms this: ${\displaystyle C^{T}C={\begin{pmatrix}2&0&0\\0&1&0\end{pmatrix}}{\begin{pmatrix}2&0\\0&1\\0&0\end{pmatrix}}={\begin{pmatrix}4&0\\0&1\end{pmatrix}}\neq I_{2}}$

Proofs

Preservation of Norm

If a matrix ${\displaystyle A}$ is tall or square (${\displaystyle m\geq n}$), its semi-orthogonality implies ${\displaystyle A^{T}A=I_{n}}$. For any vector ${\displaystyle x\in \mathbb {R} ^{n}}$, ${\displaystyle A}$ preserves its norm: ${\displaystyle \|Ax\|_{2}^{2}=(Ax)^{T}(Ax)=x^{T}A^{T}Ax=x^{T}I_{n}x=\|x\|_{2}^{2}}$ If a matrix ${\displaystyle A}$ is short (${\displaystyle m<n}$), it preserves the norm of vectors in its row space.

Justification for Full Rank

If ${\displaystyle A^{T}A=I_{n}}$, then the columns of ${\displaystyle A}$ are linearly independent, so the rank of ${\displaystyle A}$ must be ${\displaystyle n}$. If ${\displaystyle AA^{T}=I_{m}}$, then the rows of ${\displaystyle A}$ are linearly independent, so the rank of ${\displaystyle A}$ must be ${\displaystyle m}$. In both cases, the matrix has full rank.

Singular Value Property

The statement is that a real matrix ${\displaystyle A}$ is semi-orthogonal if and only if all of its non-zero singular values are 1.

This follows directly from the SVD, ${\displaystyle A=U\Sigma V^{T}}$.

(${\displaystyle \implies }$) Assume ${\displaystyle A}$ is semi-orthogonal. Then either ${\displaystyle A^{T}A=I}$ or ${\displaystyle AA^{T}=I}$. The non-zero singular values of ${\displaystyle A}$ are the square roots of the non-zero eigenvalues of both ${\displaystyle A^{T}A}$ and ${\displaystyle AA^{T}}$. Since one of these "Gramian" matrices is an identity matrix, its eigenvalues are all 1. Thus, the non-zero singular values of ${\displaystyle A}$ must be 1.

(${\displaystyle \Leftarrow }$) Assume all non-zero singular values of ${\displaystyle A}$ are 1. This forces the block of ${\displaystyle \Sigma }$ containing the non-zero values to be an identity matrix. This structure ensures that either ${\displaystyle \Sigma ^{T}\Sigma =I_{n}}$ (if ${\displaystyle A}$ has full column rank) or ${\displaystyle \Sigma \Sigma ^{T}=I_{m}}$ (if ${\displaystyle A}$ has full row rank). Substituting this into the expressions for ${\displaystyle A^{T}A=V(\Sigma ^{T}\Sigma )V^{T}}$ or ${\displaystyle AA^{T}=U(\Sigma \Sigma ^{T})U^{T}}$ respectively shows that one of them must simplify to an identity matrix, satisfying the definition of a semi-orthogonal matrix.

References

1. Abadir, K.M., Magnus, J.R. (2005). Matrix Algebra. Cambridge University Press.
2. Zhang, Xian-Da. (2017). Matrix analysis and applications. Cambridge University Press.
3. Povey, Daniel, et al. (2018). "Semi-Orthogonal Low-Rank Matrix Factorization for Deep Neural Networks." Interspeech.