Alternating conditional expectations

In statistics, Alternating Conditional Expectations (ACE) is a nonparametric algorithm used in regression analysis to find the optimal transformations for both the outcome (response) variable and the input (predictor) variables. ^[1]

For example, in a model that tries to predict house prices based on size and location, ACE helps by figuring out if, for instance, transforming the size (maybe taking the square root or logarithm) or the location (perhaps grouping locations into categories) would make the relationship easier to model and lead to better predictions. The algorithm iteratively adjusts these transformations until it finds the ones that maximize the predictive power of the regression model.

Introduction

In statistics, a nonlinear transformation of variables is commonly used in practice in regression problems. ACE is one of the methods to find those transformations that produce the best fitting additive model. Knowledge of such transformations aids in the interpretation and understanding of the relationship between the response and predictors.

ACE transforms the response variable ${\displaystyle Y}$ and its predictor variables, ${\displaystyle X_{i}}$ to minimize the fraction of variance not explained. The transformation is nonlinear and is iteratively obtained from data.

Mathematical description

Let ${\displaystyle Y,X_{1},\dots ,X_{p}}$ be random variables. We use ${\displaystyle X_{1},\dots ,X_{p}}$ to predict ${\displaystyle Y}$. Suppose ${\displaystyle \theta (Y),\varphi _{1}(X_{1}),\dots ,\varphi _{p}(X_{p})}$ are zero-mean functions and with these transformation functions, the fraction of variance of ${\displaystyle \theta (Y)}$ not explained is

${\displaystyle e^{2}(\theta ,\varphi _{1},\dots ,\varphi _{p})={\frac {\mathbb {E} \left[\theta (Y)-\sum _{i=1}^{p}\varphi _{i}(X_{i})\right]^{2}}{\mathbb {E} [\theta ^{2}(Y)]}}}$

Generally, the optimal transformations that minimize the unexplained part are difficult to compute directly. As an alternative, ACE is an iterative method to calculate the optimal transformations. The procedure of ACE has the following steps:

1. Hold ${\displaystyle \varphi _{1}(X_{1}),\dots ,\varphi _{p}(X_{p})}$ fixed, minimizing ${\displaystyle e^{2}}$gives ${\displaystyle \theta _{1}(Y)=\mathbb {E} \left[\sum _{i=1}^{p}\varphi _{i}(X_{i}){\Bigg |}Y\right]}$
2. Normalize ${\displaystyle \theta _{1}(Y)}$ to unit variance.
3. For each ${\displaystyle k}$, fix other ${\displaystyle \varphi _{i}(X_{i})}$ and ${\displaystyle \theta (Y)}$, minimizing ${\displaystyle e^{2}}$ and the solution is:: ${\displaystyle {\tilde {\varphi }}_{k}=\mathbb {E} \left[\theta (Y)-\sum _{i\neq k}\varphi _{i}(X_{i}){\Bigg |}X_{k}\right]}$
4. Iterate the above three steps until ${\displaystyle e^{2}}$ is within error tolerance.

Bivariate case

The optimal transformation ${\displaystyle \theta ^{*}(Y),\varphi ^{*}(X)}$ for ${\displaystyle p=1}$ satisfies

${\displaystyle \rho ^{*}(X,Y)=\rho ^{*}(\theta ^{*},\varphi ^{*})=\max _{\theta ,\varphi }\rho (\theta (Y),\varphi (X))}$

where ${\displaystyle \rho }$ is Pearson correlation coefficient. ${\displaystyle \rho ^{*}(X,Y)}$ is known as the maximal correlation between ${\displaystyle X}$ and ${\displaystyle Y}$. It can be used as a general measure of dependence.

In the bivariate case, the ACE algorithm can also be regarded as a method for estimating the maximal correlation between two variables.

Software implementation

The algorithm and software were developed as part of Project Orion. ^[2] The R language has a package acepack ^[3] which implements the ACE algorithm. The following example demonstrates its usage:

library(acepack)TWOPI<-8*atan(1)x<-runif(200,0,TWOPI)y<-exp(sin(x)+rnorm(200)/2)a<-ace(x,y)par(mfrow=c(3,1))plot(a$y,a$ty)# view the response transformationplot(a$x,a$tx)# view the carrier transformationplot(a$tx,a$ty)# examine the linearity of the fitted model

Discussion

The ACE algorithm provides a fully automated method for estimating optimal transformations in multiple regression. It also provides a method for estimating the maximal correlation between random variables. Since the process of iteration usually terminates in a limited number of runs, the time complexity of the algorithm is ${\displaystyle O(np)}$ where ${\displaystyle n}$ is the number of samples. The algorithm is reasonably computer efficient.

A strong advantage of the ACE procedure is the ability to incorporate variables of quite different types in terms of the set of values they can assume. The transformation functions ${\displaystyle \theta (y),\varphi _{i}(x_{i})}$ assume values on the real line. Their arguments can, however, assume values on any set. For example, ordered real and unordered categorical variables can be incorporated in the same regression equation. Variables of mixed type are admissible.

As a tool for data analysis, the ACE procedure provides graphical output to indicate a need for transformations as well as to guide in their choice. If a particular plot suggests a familiar functional form for a transformation, then the data can be pre-transformed using this functional form and the ACE algorithm can be rerun.

Wang suggests that the Box-Cox transform, a parametric approach, is a special case of ACE. ^[4]

Limitations

As with any regression procedure, a high degree of association between predictor variables can sometimes cause the individual transformation estimates to be highly variable, even though the complete model is reasonably stable. When this is suspected, running the algorithm on randomly selected subsets of the data, or on bootstrap samples can assist in assessing the variability.

ACE has shown some sensitivity to the order of the predictor variables and extreme outliers. ^[5] Long tailed distributions can lead to the above mentioned instability.

In real world applications one can never be sure that all relevant variables are observed and ACE will always recommend a transform. Thus the recommended transforms can be symptoms of this problem rather than what ACE is trying to solve. ^[6]

References

1. Breiman, L. and Friedman, J. H. Estimating optimal transformations for multiple regression and correlation. J. Am. Stat. Assoc., 80(391):580–598, September 1985b. This article incorporates text from this source, which is in the public domain .
2. Breiman, L., Friedman, J., 1982. Estimating Optimal Transformations for Multiple Regression and Correlation. Technical Report 9. University of California, Berkeley, Dept of Statistics.
3. "DOI:10.32614/CRAN.package.acepack".
4. Wang, D., Murphy, M. 2005. Identifying Nonlinear Relationships in Regression using the ACE Algorithm. Journal of Applied Statistics. 32(3) 243-258.
5. De Veaux, R. 1990. Finding Transformations for Regression Using the ACE Algorithm. Sociological Methods and Research 18(2-3) 327-359.
6. Pregibon, D., Vardi, Y. 1985. Estimating Optimal Transformations for Multiple Regression and Correlation: Comment. Journal of the American Statistical Association. 80(391) 598-601

• This draft contains quotations from Estimating Optimal Transformations For Multiple Regression And Correlation By Leo Breiman And Jerome Freidman. Technical Report No. 9 July 1982, which is in the public domain.