Projection pursuit regression

Last updated April 17, 2024

In statistics, projection pursuit regression (PPR) is a statistical model developed by Jerome H. Friedman and Werner Stuetzle that extends additive models. This model adapts the additive models in that it first projects the data matrix of explanatory variables in the optimal direction before applying smoothing functions to these explanatory variables.

Model overview

The model consists of linear combinations of ridge functions: non-linear transformations of linear combinations of the explanatory variables. The basic model takes the form

y_{i}=\beta _{0}+\sum _{j=1}^{r}f_{j}(\beta _{j}^{\mathrm {T} }x_{i})+\varepsilon _{i},

where x_i is a 1 × p row of the design matrix containing the explanatory variables for example i, y_i is a 1 × 1 prediction, {β_j} is a collection of r vectors (each a unit vector of length p) which contain the unknown parameters, {f_j} is a collection of r initially unknown smooth functions that map from $\mathbb {R} \rightarrow \mathbb {R}$ , and r is a hyperparameter. Good values for r can be determined through cross-validation or a forward stage-wise strategy which stops when the model fit cannot be significantly improved. As r approaches infinity and with an appropriate set of functions {f_j}, the PPR model is a universal estimator, as it can approximate any continuous function in $\mathbb {R} ^{p}$ .

Model estimation

For a given set of data $\{(y_{i},x_{i})\}_{i=1}^{n}$ , the goal is to minimize the error function

S=\sum _{i=1}^{n}\left[y_{i}-\sum _{j=1}^{r}f_{j}(\beta _{j}^{\mathrm {T} }x_{i})\right]^{2}

over the functions $f_{j}$ and vectors $\beta _{j}$ . No method exists for solving over all variables at once, but it can be solved via alternating optimization. First, consider each $(f_{j},\beta _{j})$ pair individually: Let all other parameters be fixed, and find a "residual", the variance of the output not accounted for by those other parameters, given by

r_{i}=y_{i}-\sum _{l\neq j}f_{l}(\beta _{l}^{\mathrm {T} }x_{i})

The task of minimizing the error function now reduces to solving

\min _{f_{j},\beta _{j}}S'=\min _{f_{j},\beta _{j}}\sum _{i=1}^{n}\left[r_{i}-f_{j}(\beta _{j}^{\mathrm {T} }x_{i})\right]^{2}

for each j in turn. Typically new $(f_{j},\beta _{j})$ pairs are added to the model in a forward stage-wise fashion.

Aside: Previously fitted pairs can be readjusted after new fit-pairs are determined by an algorithm known as backfitting, which entails reconsidering a previous pair, recalculating the residual given how other pairs have changed, refitting to account for that new information, and then cycling through all fit-pairs this way until parameters converge. This process typically results in a model that performs better with fewer fit-pairs, though it takes longer to train, and it is usually possible to achieve the same performance by skipping backfitting and simply adding more fits to the model (increasing r).

Solving the simplified error function to determine an $(f_{j},\beta _{j})$ pair can be done with alternating optimization, where first a random $\beta _{j}$ is used to project $X$ in to 1D space, and then the optimal $f_{j}$ is found to describe the relationship between that projection and the residuals via your favorite scatter plot regression method. Then if $f_{j}$ is held constant, assuming $f_{j}$ is once differentiable, the optimal updated weights $\beta _{j}$ can be found via the Gauss–Newton method —a quasi-Newton method in which the part of the Hessian involving the second derivative is discarded. To derive this, first Taylor expand $f_{j}(\beta _{j}^{T}x_{i})\approx f_{j}(\beta _{j,old}^{T}x_{i})+{\dot {f_{j}}}(\beta _{j,old}^{T}x_{i})(\beta _{j}^{T}x_{i}-\beta _{j,old}^{T}x_{i})$ , then plug the expansion back in to the simplified error function $S'$ and do some algebraic manipulation to put it in the form

\min _{\beta _{j}}S'\approx \min _{\beta _{j}}\sum _{i=1}^{n}\underbrace {{\dot {f_{j}}}(\beta _{j,old}^{T}x_{i})^{2}} _{w}{\Bigg [}{\bigg (}\underbrace {\beta _{j,old}^{T}x_{i}+{\frac {r_{i}-f_{j}(\beta _{j,old}^{T}x_{i})}{{\dot {f_{j}}}(\beta _{j,old}^{T}x_{i})}}} _{\hat {b}}{\bigg )}-\beta _{j}^{T}x_{i}{\Bigg ]}^{2}

This is a weighted least squares problem. If we solve for all weights $w$ and put them in a diagonal matrix $W$ , stack all the new targets ${\hat {b}}$ in to a vector, and use the full data matrix $X$ instead of a single example $x_{i}$ , then the optimal $\beta _{j}$ is given by the closed-form

{\underset {\beta _{j}}{\operatorname {arg\,min} }}{\Big \|}{\vec {\hat {b}}}-X\beta _{j}{\Big \|}_{W}^{2}=(X^{\mathrm {T} }WX)^{-1}X^{\mathrm {T} }W{\vec {\hat {b}}}

Use this updated $\beta _{j}$ to find a new projection of $X$ and refit $f_{j}$ to the new scatter plot. Then use that new $f_{j}$ to update $\beta _{j}$ by resolving the above, and continue this alternating process until $(f_{j},\beta _{j})$ converges.

It has been shown that the convergence rate, the bias and the variance are affected by the estimation of $\beta _{j}$ and $f_{j}$ .

Discussion

The PPR model takes the form of a basic additive model but with the additional $\beta _{j}$ component, so each $f_{j}$ fits a scatter plot of $\beta _{j}^{T}X^{T}$ vs the residual (unexplained variance) during training rather than using the raw inputs themselves. This constrains the problem of finding each $f_{j}$ to low dimension, making it solvable with common least squares or spline fitting methods and sidestepping the curse of dimensionality during training. Because $f_{j}$ is taken of a projection of $X$ , the result looks like a "ridge" orthogonal to the projection dimension, so $\{f_{j}\}$ are often called "ridge functions". The directions $\beta _{j}$ are chosen to optimize the fit of their corresponding ridge functions.

Note that because PPR attempts to fit projections of the data, it can be difficult to interpret the fitted model as a whole, because each input variable has been accounted for in a complex and multifaceted way. This can make the model more useful for prediction than for understanding the data, though visualizing individual ridge functions and considering which projections the model is discovering can yield some insight.

Advantages of PPR estimation

It uses univariate regression functions instead of their multivariate form, thus effectively dealing with the curse of dimensionality
Univariate regression allows for simple and efficient estimation
Relative to generalized additive models, PPR can estimate a much richer class of functions
Unlike local averaging methods (such as k-nearest neighbors), PPR can ignore variables with low explanatory power.

Disadvantages of PPR estimation

PPR requires examining an M-dimensional parameter space in order to estimate $\beta _{j}$ .
One must select the smoothing parameter for $f_{j}$ .
The model is often difficult to interpret

Extensions of PPR

Alternate smoothers, such as the radial function, harmonic function and additive function, have been suggested and their performances vary depending on the data sets used.
Alternate optimization criteria have been used as well, such as standard absolute deviations and mean absolute deviations.
Ordinary least squares can be used to simplify calculations as often the data does not have strong non-linearities.
Sliced Inverse Regression (SIR) has been used to choose the direction vectors for PPR.
Generalized PPR combines regular PPR with iteratively reweighted least squares (IRLS) and a link function to estimate binary data.

PPR vs neural networks (NN)

Both projection pursuit regression and fully connected neural networks with a single hidden layer project the input vector onto a one-dimensional hyperplane and then apply a nonlinear transformation of the input variables that are then added in a linear fashion. Thus both follow the same steps to overcome the curse of dimensionality. The main difference is that the functions $f_{j}$ being fitted in PPR can be different for each combination of input variables and are estimated one at a time and then updated with the weights, whereas in NN these are all specified upfront and estimated simultaneously.

Thus, in PPR estimation the transformations of variables in PPR are data driven whereas in a single-layer neural network these transformations are fixed.

Related Research Articles

<span class="mw-page-title-main">Supervised learning</span> A paradigm in machine learning

Supervised learning (SL) is a paradigm in machine learning where input objects and a desired output value train a model. The training data is processed, building a function that maps new data on expected output values. An optimal scenario will allow for the algorithm to correctly determine output values for unseen instances. This requires the learning algorithm to generalize from the training data to unseen situations in a "reasonable" way. This statistical quality of an algorithm is measured through the so-called generalization error.

The method of least squares is a parameter estimation method in regression analysis based on minimizing the sum of the squares of the residuals made in the results of each individual equation.

In statistics, the logistic model is a statistical model that models the log-odds of an event as a linear combination of one or more independent variables. In regression analysis, logistic regression is estimating the parameters of a logistic model. Formally, in binary logistic regression there is a single binary dependent variable, coded by an indicator variable, where the two values are labeled "0" and "1", while the independent variables can each be a binary variable or a continuous variable. The corresponding probability of the value labeled "1" can vary between 0 and 1, hence the labeling; the function that converts log-odds to probability is the logistic function, hence the name. The unit of measurement for the log-odds scale is called a logit, from logistic unit, hence the alternative names. See § Background and § Definition for formal mathematics, and § Example for a worked example.

In statistics, nonlinear regression is a form of regression analysis in which observational data are modeled by a function which is a nonlinear combination of the model parameters and depends on one or more independent variables. The data are fitted by a method of successive approximations (iterations).

Partial least squares regression is a statistical method that bears some relation to principal components regression; instead of finding hyperplanes of maximum variance between the response and independent variables, it finds a linear regression model by projecting the predicted variables and the observable variables to a new space. Because both the X and Y data are projected to new spaces, the PLS family of methods are known as bilinear factor models. Partial least squares discriminant analysis (PLS-DA) is a variant used when the Y is categorical.

In statistics, the coefficient of determination, denoted R² or r² and pronounced "R squared", is the proportion of the variation in the dependent variable that is predictable from the independent variable(s).

In statistics, econometrics, epidemiology and related disciplines, the method of instrumental variables (IV) is used to estimate causal relationships when controlled experiments are not feasible or when a treatment is not successfully delivered to every unit in a randomized experiment. Intuitively, IVs are used when an explanatory variable of interest is correlated with the error term (endogenous), in which case ordinary least squares and ANOVA give biased results. A valid instrument induces changes in the explanatory variable but has no independent effect on the dependent variable and is not correlated with the error term, allowing a researcher to uncover the causal effect of the explanatory variable on the dependent variable.

In statistics, ordinary least squares (OLS) is a type of linear least squares method for choosing the unknown parameters in a linear regression model by the principle of least squares: minimizing the sum of the squares of the differences between the observed dependent variable in the input dataset and the output of the (linear) function of the independent variable.

In statistics, a generalized additive model (GAM) is a generalized linear model in which the linear response variable depends linearly on unknown smooth functions of some predictor variables, and interest focuses on inference about these smooth functions.

In statistics, the variance inflation factor (VIF) is the ratio (quotient) of the variance of a parameter estimate when fitting a full model that includes other parameters to the variance of the parameter estimate if the model is fit with only the parameter on its own. The VIF provides an index that measures how much the variance of an estimated regression coefficient is increased because of collinearity.

Non-linear least squares is the form of least squares analysis used to fit a set of m observations with a model that is non-linear in n unknown parameters (m ≥ n). It is used in some forms of nonlinear regression. The basis of the method is to approximate the model by a linear one and to refine the parameters by successive iterations. There are many similarities to linear least squares, but also some significant differences. In economic theory, the non-linear least squares method is applied in (i) the probit regression, (ii) threshold regression, (iii) smooth regression, (iv) logistic link regression, (v) Box–Cox transformed regressors ( $).$

In statistics, principal component regression (PCR) is a regression analysis technique that is based on principal component analysis (PCA). More specifically, PCR is used for estimating the unknown regression coefficients in a standard linear regression model.

In statistics, an additive model (AM) is a nonparametric regression method. It was suggested by Jerome H. Friedman and Werner Stuetzle (1981) and is an essential part of the ACE algorithm. The AM uses a one-dimensional smoother to build a restricted class of nonparametric regression models. Because of this, it is less affected by the curse of dimensionality than e.g. a p-dimensional smoother. Furthermore, the AM is more flexible than a standard linear model, while being more interpretable than a general regression surface at the cost of approximation errors. Problems with AM, like many other machine-learning methods, include model selection, overfitting, and multicollinearity.

In statistics and machine learning, lasso is a regression analysis method that performs both variable selection and regularization in order to enhance the prediction accuracy and interpretability of the resulting statistical model. The lasso method assumes that the coefficients of the linear model are sparse, meaning that few of them are non-zero. It was originally introduced in geophysics, and later by Robert Tibshirani, who coined the term.

Linear least squares (LLS) is the least squares approximation of linear functions to data. It is a set of formulations for solving statistical problems involved in linear regression, including variants for ordinary (unweighted), weighted, and generalized (correlated) residuals. Numerical methods for linear least squares include inverting the matrix of the normal equations and orthogonal decomposition methods.

The Generalized Additive Model for Location, Scale and Shape (GAMLSS) is an approach to statistical modelling and learning. GAMLSS is a modern distribution-based approach to (semiparametric) regression. A parametric distribution is assumed for the response (target) variable but the parameters of this distribution can vary according to explanatory variables using linear, nonlinear or smooth functions. In machine learning parlance, GAMLSS is a form of supervised machine learning.

In statistics and in machine learning, a linear predictor function is a linear function of a set of coefficients and explanatory variables, whose value is used to predict the outcome of a dependent variable. This sort of function usually comes in linear regression, where the coefficients are called regression coefficients. However, they also occur in various types of linear classifiers, as well as in various other models, such as principal component analysis and factor analysis. In many of these models, the coefficients are referred to as "weights".

The generalized functional linear model (GFLM) is an extension of the generalized linear model (GLM) that allows one to regress univariate responses of various types on functional predictors, which are mostly random trajectories generated by a square-integrable stochastic processes. Similarly to GLM, a link function relates the expected value of the response variable to a linear predictor, which in case of GFLM is obtained by forming the scalar product of the random predictor function $with a smooth parameter function . Functional Linear Regression, Functional Poisson Regression and Functional Binomial Regression, with the important Functional Logistic Regression included, are special cases of GFLM. Applications of GFLM include classification and discrimination of stochastic processes and functional data.$

In statistics, linear regression is a statistical model which estimates the linear relationship between a scalar response and one or more explanatory variables. The case of one explanatory variable is called simple linear regression; for more than one, the process is called multiple linear regression. This term is distinct from multivariate linear regression, where multiple correlated dependent variables are predicted, rather than a single scalar variable. If the explanatory variables are measured with error then errors-in-variables models are required, also known as measurement error models.

In statistics, the class of vector generalized linear models (VGLMs) was proposed to enlarge the scope of models catered for by generalized linear models (GLMs). In particular, VGLMs allow for response variables outside the classical exponential family and for more than one parameter. Each parameter can be transformed by a link function. The VGLM framework is also large enough to naturally accommodate multiple responses; these are several independent responses each coming from a particular statistical distribution with possibly different parameter values.

References

Friedman, J.H. and Stuetzle, W. (1981) Projection Pursuit Regression. Journal of the American Statistical Association, 76, 817–823.
Hand, D., Mannila, H. and Smyth, P, (2001) Principles of Data Mining. MIT Press. ISBN 0-262-08290-X
Hall, P. (1988) Estimating the direction in which a data set is the most interesting, Probab. Theory Related Fields, 80, 51–77.
Hastie, T. J., Tibshirani, R. J. and Friedman, J.H. (2009). The Elements of Statistical Learning: Data Mining, Inference and Prediction. Springer. ISBN 978-0-387-84857-0
Klinke, S. and Grassmann, J. (2000) 'Projection Pursuit Regression' in Smoothing and Regression: Approaches, Computation and Application. Ed. Schimek, M.G.. Wiley Interscience.
Lingjarde, O. C. and Liestol, K. (1998) Generalized Projection Pursuit Regression. SIAM Journal of Scientific Computing, 20, 844–857.

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.