Multivariate analysis of variance

Last updated
The image above depicts a visual comparison between multivariate analysis of variance (MANOVA) and univariate analysis of variance (ANOVA). In MANOVA, researchers are examining the group differences of a singular independent variable across multiple outcome variables, whereas in an ANOVA, researchers are examining the group differences of sometimes multiple independent variables on a singular outcome variable. In the provided example, the levels of the IV might include high school, college, and graduate school. The results of a MANOVA can tell us whether an individual who completed graduate school showed higher life AND job satisfaction than an individual who completed only high school or college. Results of an ANOVA can only tell us this information for life satisfaction. Analyzing group differences across multiple outcome variables often provides more accurate information as a pure relationship between only X and only Y rarely exists in nature. Univariate vs. Multivariate.jpg
The image above depicts a visual comparison between multivariate analysis of variance (MANOVA) and univariate analysis of variance (ANOVA). In MANOVA, researchers are examining the group differences of a singular independent variable across multiple outcome variables, whereas in an ANOVA, researchers are examining the group differences of sometimes multiple independent variables on a singular outcome variable. In the provided example, the levels of the IV might include high school, college, and graduate school. The results of a MANOVA can tell us whether an individual who completed graduate school showed higher life AND job satisfaction than an individual who completed only high school or college. Results of an ANOVA can only tell us this information for life satisfaction. Analyzing group differences across multiple outcome variables often provides more accurate information as a pure relationship between only X and only Y rarely exists in nature.

In statistics, multivariate analysis of variance (MANOVA) is a procedure for comparing multivariate sample means. As a multivariate procedure, it is used when there are two or more dependent variables, [1] and is often followed by significance tests involving individual dependent variables separately. [2]

Contents

Without relation to the image, the dependent variables may be k life satisfactions scores measured at sequential time points and p job satisfaction scores measured at sequential time points. In this case there are k+p dependent variables whose linear combination follows a multivariate normal distribution, multivariate variance-covariance matrix homogeneity, and linear relationship, no multicollinearity, and each without outliers.

Model

Assume -dimensional observations, where the ’th observation is assigned to the group and is distributed around the group center with Multivariate Gaussian noise:

where is the covariance matrix. Then we formulate our null hypothesis as

Relationship with ANOVA

MANOVA is a generalized form of univariate analysis of variance (ANOVA), [1] although, unlike univariate ANOVA, it uses the covariance between outcome variables in testing the statistical significance of the mean differences.

Where sums of squares appear in univariate analysis of variance, in multivariate analysis of variance certain positive-definite matrices appear. The diagonal entries are the same kinds of sums of squares that appear in univariate ANOVA. The off-diagonal entries are corresponding sums of products. Under normality assumptions about error distributions, the counterpart of the sum of squares due to error has a Wishart distribution.


Hypothesis Testing

First, define the following matrices:

Then the matrix is a generalization of the sum of squares explained by the group, and is a generalization of the residual sum of squares. [3] [4] Note that alternatively one could also speak about covariances when the abovementioned matrices are scaled by 1/(n-1) since the subsequent test statistics do not change by multiplying and by the same non-zero constant.

The most common [5] [6] statistics are summaries based on the roots (or eigenvalues) of the matrix

Discussion continues over the merits of each, [1] although the greatest root leads only to a bound on significance which is not generally of practical interest. A further complication is that, except for the Roy's greatest root, the distribution of these statistics under the null hypothesis is not straightforward and can only be approximated except in a few low-dimensional cases. [8] An algorithm for the distribution of the Roy's largest root under the null hypothesis was derived in [9] while the distribution under the alternative is studied in. [10]

The best-known approximation for Wilks' lambda was derived by C. R. Rao.

In the case of two groups, all the statistics are equivalent and the test reduces to Hotelling's T-square.

Introducing Covariates (MANCOVA)

One can also test if there is a group effect after adjusting for covariates. For this, follow the procedure above but substitute with the predictions of the general linear model, containing the group and the covariates, and substitute with the predictions of the general linear model containing only the covariates (and an intercept). Then are the additional sum of squares explained by adding the grouping information and is the residual sum of squares of the model containing the grouping and the covariates. [4]

Note that in case of unbalanced data, the order of adding the covariates matter.

Correlation of dependent variables

This is a graphical depiction of the required relationship amongst outcome variables in a multivariate analysis of variance. Part of the analysis involves creating a composite variable, which the group differences of the independent variable are analyzed against. The composite variables, as there can be multiple, are different combinations of the outcome variables. The analysis then determines which combination shows the greatest group differences for the independent variable. A descriptive discriminant analysis is then used as a post hoc test to determine what the makeup of that composite variable is that creates the greatest group differences. Outcome Variables.jpg
This is a graphical depiction of the required relationship amongst outcome variables in a multivariate analysis of variance. Part of the analysis involves creating a composite variable, which the group differences of the independent variable are analyzed against. The composite variables, as there can be multiple, are different combinations of the outcome variables. The analysis then determines which combination shows the greatest group differences for the independent variable. A descriptive discriminant analysis is then used as a post hoc test to determine what the makeup of that composite variable is that creates the greatest group differences.
This is a simple visual representation of the effect of two highly correlated dependent variables within a MANOVA. If two (or more) dependent variables are highly correlated, the chances of a Type I error occurring is reduced, but the trade-off is that the power of the MANOVA test is also reduced. MANOVAs and Highly Correlated Dependent Variables.png
This is a simple visual representation of the effect of two highly correlated dependent variables within a MANOVA. If two (or more) dependent variables are highly correlated, the chances of a Type I error occurring is reduced, but the trade-off is that the power of the MANOVA test is also reduced.

MANOVA's power is affected by the correlations of the dependent variables and by the effect sizes associated with those variables. For example, when there are two groups and two dependent variables, MANOVA's power is lowest when the correlation equals the ratio of the smaller to the larger standardized effect size. [11]

See also

Related Research Articles

<span class="mw-page-title-main">Normal distribution</span> Probability distribution

In statistics, a normal distribution or Gaussian distribution is a type of continuous probability distribution for a real-valued random variable. The general form of its probability density function is

<span class="mw-page-title-main">Variance</span> Statistical measure of how far values spread from their average

In probability theory and statistics, variance is the expected squared deviation from the mean of a random variable. The standard deviation is obtained as the square root of the variance. Variance is a measure of dispersion, meaning it is a measure of how far a set of numbers is spread out from their average value. It is the second central moment of a distribution, and the covariance of the random variable with itself, and it is often represented by , , , , or .

<span class="mw-page-title-main">Negative binomial distribution</span> Probability distribution

In probability theory and statistics, the negative binomial distribution is a discrete probability distribution that models the number of failures in a sequence of independent and identically distributed Bernoulli trials before a specified (non-random) number of successes occurs. For example, we can define rolling a 6 on a die as a success, and rolling any other number as a failure, and ask how many failure rolls will occur before we see the third success. In such a case, the probability distribution of the number of failures that appear will be a negative binomial distribution.

<span class="mw-page-title-main">Multivariate normal distribution</span> Generalization of the one-dimensional normal distribution to higher dimensions

In probability theory and statistics, the multivariate normal distribution, multivariate Gaussian distribution, or joint normal distribution is a generalization of the one-dimensional (univariate) normal distribution to higher dimensions. One definition is that a random vector is said to be k-variate normally distributed if every linear combination of its k components has a univariate normal distribution. Its importance derives mainly from the multivariate central limit theorem. The multivariate normal distribution is often used to describe, at least approximately, any set of (possibly) correlated real-valued random variables each of which clusters around a mean value.

In probability theory, Chebyshev's inequality provides an upper bound on the probability of deviation of a random variable from its mean. More specifically, the probability that a random variable deviates from its mean by more than is at most , where is any positive constant.

In probability theory and statistics, a Gaussian process is a stochastic process, such that every finite collection of those random variables has a multivariate normal distribution. The distribution of a Gaussian process is the joint distribution of all those random variables, and as such, it is a distribution over functions with a continuous domain, e.g. time or space.

The general linear model or general multivariate regression model is a compact way of simultaneously writing several multiple linear regression models. In that sense it is not a separate statistical linear model. The various multiple linear regression models may be compactly written as

In probability theory, a compound Poisson distribution is the probability distribution of the sum of a number of independent identically-distributed random variables, where the number of terms to be added is itself a Poisson-distributed variable. The result can be either a continuous or a discrete distribution.

<span class="mw-page-title-main">Noncentral chi-squared distribution</span> Noncentral generalization of the chi-squared distribution

In probability theory and statistics, the noncentral chi-squared distribution is a noncentral generalization of the chi-squared distribution. It often arises in the power analysis of statistical tests in which the null distribution is a chi-squared distribution; important examples of such tests are the likelihood-ratio tests.

<span class="mw-page-title-main">Characteristic function (probability theory)</span> Fourier transform of the probability density function

In probability theory and statistics, the characteristic function of any real-valued random variable completely defines its probability distribution. If a random variable admits a probability density function, then the characteristic function is the Fourier transform of the probability density function. Thus it provides an alternative route to analytical results compared with working directly with probability density functions or cumulative distribution functions. There are particularly simple results for the characteristic functions of distributions defined by the weighted sums of random variables.

In statistics, a generalized additive model (GAM) is a generalized linear model in which the linear response variable depends linearly on unknown smooth functions of some predictor variables, and interest focuses on inference about these smooth functions.

In statistics, Wilks' lambda distribution, is a probability distribution used in multivariate hypothesis testing, especially with regard to the likelihood-ratio test and multivariate analysis of variance (MANOVA).

Mauchly's sphericity test or Mauchly's W is a statistical test used to validate a repeated measures analysis of variance (ANOVA). It was developed in 1940 by John Mauchly.

In probability and statistics, the class of exponential dispersion models (EDM), also called exponential dispersion family (EDF), is a set of probability distributions that represents a generalisation of the natural exponential family. Exponential dispersion models play an important role in statistical theory, in particular in generalized linear models because they have a special structure which enables deductions to be made about appropriate statistical inference.

Multivariate analysis of covariance (MANCOVA) is an extension of analysis of covariance (ANCOVA) methods to cover cases where there is more than one dependent variable and where the control of concomitant continuous independent variables – covariates – is required. The most prominent benefit of the MANCOVA design over the simple MANOVA is the 'factoring out' of noise or error that has been introduced by the covariant. A commonly used multivariate version of the ANOVA F-statistic is Wilks' Lambda (Λ), which represents the ratio between the error variance and the effect variance.

In statistics, principal component regression (PCR) is a regression analysis technique that is based on principal component analysis (PCA). More specifically, PCR is used for estimating the unknown regression coefficients in a standard linear regression model.

In statistics and machine learning, lasso is a regression analysis method that performs both variable selection and regularization in order to enhance the prediction accuracy and interpretability of the resulting statistical model. It was originally introduced in geophysics, and later by Robert Tibshirani, who coined the term.

<span class="mw-page-title-main">Poisson distribution</span> Discrete probability distribution

In probability theory and statistics, the Poisson distribution is a discrete probability distribution that expresses the probability of a given number of events occurring in a fixed interval of time or space if these events occur with a known constant mean rate and independently of the time since the last event. It is named after French mathematician Siméon Denis Poisson. The Poisson distribution can also be used for the number of events in other specified interval types such as distance, area, or volume. It plays an important role for discrete-stable distributions.

In probability theory and statistics, the generalized multivariate log-gamma (G-MVLG) distribution is a multivariate distribution introduced by Demirhan and Hamurkaroglu in 2011. The G-MVLG is a flexible distribution. Skewness and kurtosis are well controlled by the parameters of the distribution. This enables one to control dispersion of the distribution. Because of this property, the distribution is effectively used as a joint prior distribution in Bayesian analysis, especially when the likelihood is not from the location-scale family of distributions such as normal distribution.

In statistics, the complex Wishart distribution is a complex version of the Wishart distribution. It is the distribution of times the sample Hermitian covariance matrix of zero-mean independent Gaussian random variables. It has support for Hermitian positive definite matrices.

References

  1. 1 2 3 Warne, R. T. (2014). "A primer on multivariate analysis of variance (MANOVA) for behavioral scientists". Practical Assessment, Research & Evaluation. 19 (17): 1–10.
  2. Stevens, J. P. (2002). Applied multivariate statistics for the social sciences. Mahwah, NJ: Lawrence Erblaum.
  3. Anderson, T. W. (1994). An Introduction to Multivariate Statistical Analysis. Wiley.
  4. 1 2 Krzanowski, W. J. (1988). Principles of Multivariate Analysis. A User's Perspective. Oxford University Press.
  5. Garson, G. David. "Multivariate GLM, MANOVA, and MANCOVA" . Retrieved 2011-03-22.
  6. UCLA: Academic Technology Services, Statistical Consulting Group. "Stata Annotated Output – MANOVA" . Retrieved 2011-03-22.
  7. "MANOVA Basic Concepts – Real Statistics Using Excel". www.real-statistics.com. Retrieved 5 April 2018.
  8. Camo http://www.camo.com/multivariate_analysis.html
  9. Chiani, M. (2016), "Distribution of the largest root of a matrix for Roy's test in multivariate analysis of variance", Journal of Multivariate Analysis , 143: 467–471, arXiv: 1401.3987v3 , doi:10.1016/j.jmva.2015.10.007, S2CID   37620291
  10. I.M. Johnstone, B. Nadler "Roy's largest root test under rank-one alternatives" arXiv preprint arXiv:1310.6581 (2013)
  11. Frane, Andrew (2015). "Power and Type I Error Control for Univariate Comparisons in Multivariate Two-Group Designs". Multivariate Behavioral Research. 50 (2): 233–247. doi:10.1080/00273171.2014.968836. PMID   26609880. S2CID   1532673.