WikiMili The Free Encyclopedia

This article needs additional citations for verification .(August 2017) (Learn how and when to remove this template message) |

**Cross-validation**, sometimes called **rotation estimation**,^{ [1] }^{ [2] }^{ [3] } or **out-of-sample testing** is any of various similar model validation techniques for assessing how the results of a statistical analysis will generalize to an independent data set. It is mainly used in settings where the goal is prediction, and one wants to estimate how accurately a predictive model will perform in practice. In a prediction problem, a model is usually given a dataset of *known data* on which training is run (*training dataset*), and a dataset of *unknown data* (or *first seen* data) against which the model is tested (called the validation dataset or *testing set*).^{ [4] }^{ [5] } The goal of cross-validation is to test the model's ability to predict new data that was not used in estimating it, in order to flag problems like overfitting or selection bias ^{ [6] } and to give an insight on how the model will generalize to an independent dataset (i.e., an unknown dataset, for instance from a real problem).

**Statistics** is a branch of mathematics dealing with data collection, organization, analysis, interpretation and presentation. In applying statistics to, for example, a scientific, industrial, or social problem, it is conventional to begin with a statistical population or a statistical model process to be studied. Populations can be diverse topics such as "all people living in a country" or "every atom composing a crystal". Statistics deals with all aspects of data, including the planning of data collection in terms of the design of surveys and experiments. See glossary of probability and statistics.

**Predictive modelling** uses statistics to predict outcomes. Most often the event one wants to predict is in the future, but predictive modelling can be applied to any type of unknown event, regardless of when it occurred. For example, predictive models are often used to detect crimes and identify suspects, after the crime has taken place.

In statistics, **overfitting** is "the production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit additional data or predict future observations reliably". An **overfitted model** is a statistical model that contains more parameters than can be justified by the data. The essence of overfitting is to have unknowingly extracted some of the residual variation as if that variation represented underlying model structure.

- Purpose of cross-validation
- Common types of cross-validation
- Exhaustive cross-validation
- Non-exhaustive cross-validation
- Nested cross-validation
- k*l-fold cross-validation
- k-fold cross-validation with validation and test set
- Measures of fit
- Statistical properties
- Computational issues
- Limitations and misuse
- Cross validation for time-series models
- Applications
- See also
- Notes and references

One round of cross-validation involves partitioning a sample of data into complementary subsets, performing the analysis on one subset (called the *training set*), and validating the analysis on the other subset (called the *validation set* or *testing set*). To reduce variability, in most methods multiple rounds of cross-validation are performed using different partitions, and the validation results are combined (e.g. averaged) over the rounds to give an estimate of the model's predictive performance.

In mathematics, a **partition of a set** is a grouping of the set's elements into non-empty subsets, in such a way that every element is included in one and only one of the subsets.

**Data** is a set of values of subjects with respect to qualitative or quantitative variables.

In set theory, the **complement** of a set *A* refers to elements not in *A*.

In summary, cross-validation combines (averages) measures of fitness in prediction to derive a more accurate estimate of model prediction performance.^{ [7] }

Suppose we have a model with one or more unknown parameters, and a data set to which the model can be fit (the training data set). The fitting process optimizes the model parameters to make the model fit the training data as well as possible. If we then take an independent sample of validation data from the same population as the training data, it will generally turn out that the model does not fit the validation data as well as it fits the training data. The size of this difference is likely to be large especially when the size of the training data set is small, or when the number of parameters in the model is large. Cross-validation is a way to estimate the size of this effect.

A **statistical model** is a mathematical model that embodies a set of statistical assumptions concerning the generation of sample data. A statistical model represents, often in considerably idealized form, the data-generating process.

In probability theory, two events are **independent**, **statistically independent**, or **stochastically independent** if the occurrence of one does not affect the probability of occurrence of the other. Similarly, two random variables are independent if the realization of one does not affect the probability distribution of the other.

In statistics, a **population** is a set of similar items or events which is of interest for some question or experiment. A statistical population can be a group of existing objects or a hypothetical and potentially infinite group of objects conceived as a generalization from experience. A common aim of statistical analysis is to produce information about some chosen population.

In linear regression we have real *response values**y*_{1}, ..., *y _{n}*, and

In mathematics, a **real number** is a value of a continuous quantity that can represent a distance along a line. The adjective *real* in this context was introduced in the 17th century by René Descartes, who distinguished between real and imaginary roots of polynomials. The real numbers include all the rational numbers, such as the integer −5 and the fraction 4/3, and all the irrational numbers, such as √2. Included within the irrationals are the transcendental numbers, such as π (3.14159265...). In addition to measuring distance, real numbers can be used to measure quantities such as time, mass, energy, velocity, and many more.

In mathematics, physics, and engineering, a **Euclidean vector** is a geometric object that has magnitude and direction. Vectors can be added to other vectors according to vector algebra. A Euclidean vector is frequently represented by a line segment with a definite direction, or graphically as an arrow, connecting an *initial point**A* with a *terminal point**B*, and denoted by

The method of **least squares** is a standard approach in regression analysis to approximate the solution of overdetermined systems, i.e., sets of equations in which there are more equations than unknowns. "Least squares" means that the overall solution minimizes the sum of the squares of the residuals made in the results of every single equation.

If the model is correctly specified, it can be shown under mild assumptions that the expected value of the MSE for the training set is (*n* − *p* − 1)/(*n* + *p* + 1) < 1 times the expected value of the MSE for the validation set^{ [8] } (the expected value is taken over the distribution of training sets). Thus if we fit the model and compute the MSE on the training set, we will get an optimistically biased assessment of how well the model will fit an independent data set. This biased estimate is called the *in-sample* estimate of the fit, whereas the cross-validation estimate is an *out-of-sample* estimate.

In probability theory, the **expected value** of a random variable, intuitively, is the long-run average value of repetitions of the **same experiment** it represents. For example, the expected value in rolling a six-sided die is 3.5, because the average of all the numbers that come up is 3.5 as the number of rolls approaches infinity. In other words, the law of large numbers states that the arithmetic mean of the values almost surely converges to the expected value as the number of repetitions approaches infinity. The expected value is also known as the **expectation**, **mathematical expectation**, **EV**, **average**, **mean value**, **mean**, or **first moment**.

**Statistical bias** is a feature of a statistical technique or of its results whereby the expected value of the results differs from the true underlying quantitative parameter being estimated.

Since in linear regression it is possible to directly compute the factor (*n* − *p* − 1)/(*n* + *p* + 1) by which the training MSE underestimates the validation MSE under the assumption that the model specification is valid, cross-validation can be used for checking whether the model has been overfitted, in which case the MSE in the validation set will substantially exceed its anticipated value. (Cross-validation in the context of linear regression is also useful in that it can be used to select an optimally regularized cost function). In most other regression procedures (e.g. logistic regression), there is no simple formula to compute the expected out-of-sample fit. Cross-validation is, thus, a generally applicable way to predict the performance of a model on unavailable data using numerical computation in place of theoretical analysis.

Two types of cross-validation can be distinguished, exhaustive and non-exhaustive cross-validation.

Exhaustive cross-validation methods are cross-validation methods which learn and test on all possible ways to divide the original sample into a training and a validation set.

Leave-*p*-out cross-validation (**LpO CV**) involves using *p* observations as the validation set and the remaining observations as the training set. This is repeated on all ways to cut the original sample on a validation set of *p* observations and a training set.

LpO cross-validation requires training and validating the model times, where *n* is the number of observations in the original sample, and where is the binomial coefficient. For *p* > 1 and for even moderately large *n*, LpO CV can become computationally infeasible. For example, with *n* = 100 and *p* = 30 = 30 percent of 100 (as suggested above),

Leave-*one*-out cross-validation (**LOOCV**) is a particular case of leave-*p*-out cross-validation with *p* = 1.

The process looks similar to jackknife; however, with cross-validation one computes a statistic on the left-out sample(s), while with jackknifing one computes a statistic from the kept samples only.

LOO cross-validation requires less computation time than LpO cross-validation because there are only passes rather than . However, passes may still require quite a large computation time, in which case other approaches such as k-fold cross validation may be more appropriate.

**Pseudo-Code-Algorithm:**

**Input:**

x, {vector of length N with x-values of data points}

y, {vector of length N with y-values of data points}

**Output:**

err, {estimate for the prediction error}

**Steps:**

err ← 0

for i ← 1, . . . , N do

// define the cross-validation subsets

x_in ← (x[1], . . . , x[i − 1], x[i + 1], . . . , x[N])

y_in ← (y[1], . . . , y[i − 1], y[i + 1], . . . , y[N]

x_out ← x[i]

y_out ← interpolate(x_in, y_in, x_out, y_out)

err ← err + (y[i] − y_out)^2

end for

err ← err/N

Non-exhaustive cross validation methods do not compute all ways of splitting the original sample. Those methods are approximations of leave-*p*-out cross-validation.

In *k*-fold cross-validation, the original sample is randomly partitioned into *k* equal sized subsamples. Of the *k* subsamples, a single subsample is retained as the validation data for testing the model, and the remaining *k* − 1 subsamples are used as training data. The cross-validation process is then repeated *k* times, with each of the *k* subsamples used exactly once as the validation data. The *k* results can then be averaged to produce a single estimation. The advantage of this method over repeated random sub-sampling (see below) is that all observations are used for both training and validation, and each observation is used for validation exactly once. 10-fold cross-validation is commonly used,^{ [9] } but in general *k* remains an unfixed parameter.

For example, setting *k* = *2* results in 2-fold cross-validation. In 2-fold cross-validation, we randomly shuffle the dataset into two sets *d*_{0} and *d*_{1}, so that both sets are equal size (this is usually implemented by shuffling the data array and then splitting it in two). We then train on *d*_{0} and validate on *d*_{1}, followed by training on *d*_{1} and validating on *d*_{0}.

When *k* = *n* (the number of observations), the *k*-fold cross-validation is exactly the leave-one-out cross-validation.

In *stratified**k*-fold cross-validation, the folds are selected so that the mean response value is approximately equal in all the folds. In the case of binary classification, this means that each fold contains roughly the same proportions of the two types of class labels.

In the holdout method, we randomly assign data points to two sets *d*_{0} and *d*_{1}, usually called the training set and the test set, respectively. The size of each of the sets is arbitrary although typically the test set is smaller than the training set. We then train (build a model) on *d*_{0} and test (evaluate its performance) on *d*_{1}.

In typical cross-validation, results of multiple runs of model-testing are averaged together; in contrast, the holdout method, in isolation, involves a single run. It should be used with caution because without such averaging of multiple runs, one may achieve highly misleading results. One's indicator of predictive accuracy (F*), as noted below, will tend to be unstable since it will not be smoothed out by multiple iterations. Similarly, indicators of the specific role played by various predictor variables (e.g., values of regression coefficients) will tend to be unstable.

While the holdout method can be framed as "the simplest kind of cross-validation",^{ [10] } many sources instead classify holdout as a type of simple validation, rather than a simple or degenerate form of cross-validation.^{ [2] }^{ [11] }

This method, also known as Monte Carlo cross-validation,^{ [12] } randomly splits the dataset into training and validation data. For each such split, the model is fit to the training data, and predictive accuracy is assessed using the validation data. The results are then averaged over the splits. The advantage of this method (over *k*-fold cross validation) is that the proportion of the training/validation split is not dependent on the number of iterations (folds). The disadvantage of this method is that some observations may never be selected in the validation subsample, whereas others may be selected more than once. In other words, validation subsets may overlap. This method also exhibits Monte Carlo variation, meaning that the results will vary if the analysis is repeated with different random splits.

As the number of random splits approaches infinity, the result of repeated random sub-sampling validation tends towards that of leave-p-out cross-validation.

In a stratified variant of this approach, the random samples are generated in such a way that the mean response value (i.e. the dependent variable in the regression) is equal in the training and testing sets. This is particularly useful if the responses are dichotomous with an unbalanced representation of the two response values in the data.

When cross-validation is used simultaneously for selection of the best set of hyperparameters and for error estimation (and assessment of generalization capacity), a nested cross-validation is required. Many variants exist. At least two variants can be distinguished:

This is a truly nested variant (for instance used by ^{ [13] }), which contains an outer loop of *k* folds and an inner loop of *l* folds. The total data set is split in *k* sets. One by one, a set is selected as (outer) test set and the *k-1* other sets are combined into the corresponding outer training set. This is repeated for each of the k sets. Each outer training set is further sub-divided into *l* sets. One by one, a set is selected as inner test (validation) set and the *l-1* other sets are combined into the corresponding inner training set. This is repeated for each of the l sets. The inner training sets are used to fit model parameters, while the inner test set is used as a validation set to provides an unbiased evaluation of the model fit. Typically, this is repeated for many different hyperparameters (or even different model types) and the validation set is used to determine the best hyperparameter set (and model type) for this inner training set. After this, a new model is fit on the entire outer training set, using the best set of hyperparameters from the inner cross-validation. The performance of this model is then evaluated using the outer test set.

A slightly less granular approach is to use a single k-fold cross validation with both a validation and test set. The total data set is split in *k* sets. One by one, a set is selected as test set. Then, one by one, one of the remaining sets is used as a validation sets and the other *k-2* sets are used as training sets until all possible combinations have been evaluated. Similar to the k*l-fold cross validation, the training set is used for model fitting and the validation set is used for model evaluation for each of the hyperparameter sets. Finally, for the selected parameter set, the test set is used to evaluate the model with the best parameter set. Here, two variants are possible: either evaluating the model that was trained on the training set or evaluating a new model that was fit on the combination of the train and the validation set.

The goal of cross-validation is to estimate the expected level of fit of a model to a data set that is independent of the data that were used to train the model. It can be used to estimate any quantitative measure of fit that is appropriate for the data and model. For example, for binary classification problems, each case in the validation set is either predicted correctly or incorrectly. In this situation the misclassification error rate can be used to summarize the fit, although other measures like positive predictive value could also be used. When the value being predicted is continuously distributed, the mean squared error, root mean squared error or median absolute deviation could be used to summarize the errors.

Suppose we choose a measure of fit *F*, and use cross-validation to produce an estimate *F ^{*}* of the expected fit

The cross-validation estimator *F ^{*}* is very nearly unbiased for

The variance of *F ^{*}* can be large.

Most forms of cross-validation are straightforward to implement as long as an implementation of the prediction method being studied is available. In particular, the prediction method can be a "black box" – there is no need to have access to the internals of its implementation. If the prediction method is expensive to train, cross-validation can be very slow since the training must be carried out repeatedly. In some cases such as least squares and kernel regression, cross-validation can be sped up significantly by pre-computing certain values that are needed repeatedly in the training, or by using fast "updating rules" such as the Sherman–Morrison formula. However one must be careful to preserve the "total blinding" of the validation set from the training procedure, otherwise bias may result. An extreme example of accelerating cross-validation occurs in linear regression, where the results of cross-validation have a closed-form expression known as the *prediction residual error sum of squares* (PRESS).

Cross-validation only yields meaningful results if the validation set and training set are drawn from the same population and only if human biases are controlled.

In many applications of predictive modeling, the structure of the system being studied evolves over time (i.e. it is "non-stationary"). Both of these can introduce systematic differences between the training and validation sets. For example, if a model for predicting stock values is trained on data for a certain five-year period, it is unrealistic to treat the subsequent five-year period as a draw from the same population. As another example, suppose a model is developed to predict an individual's risk for being diagnosed with a particular disease within the next year. If the model is trained using data from a study involving only a specific population group (e.g. young people or males), but is then applied to the general population, the cross-validation results from the training set could differ greatly from the actual predictive performance.

In many applications, models also may be incorrectly specified and vary as a function of modeler biases and/or arbitrary choices. When this occurs, there may be an illusion that the system changes in external samples, whereas the reason is that the model has missed a critical predictor and/or included a confounded predictor. New evidence is that cross-validation by itself is not very predictive of external validity, whereas a form of experimental validation known as swap sampling that does control for human bias can be much more predictive of external validity.^{ [17] } As defined by this large MAQC-II study across 30,000 models, swap sampling incorporates cross-validation in the sense that predictions are tested across independent training and validation samples. Yet, models are also developed across these independent samples and by modelers who are blinded to one another. When there is a mismatch in these models developed across these swapped training and validation samples as happens quite frequently, MAQC-II shows that this will be much more predictive of poor external predictive validity than traditional cross-validation.

The reason for the success of the swapped sampling is a built-in control for human biases in model building. In addition to placing too much faith in predictions that may vary across modelers and lead to poor external validity due to these confounding modeler effects, these are some other ways that cross-validation can be misused:

- By performing an initial analysis to identify the most informative features using the entire data set – if feature selection or model tuning is required by the modeling procedure, this must be repeated on every training set. Otherwise, predictions will certainly be upwardly biased.
^{ [18] }If cross-validation is used to decide which features to use, an*inner cross-validation*to carry out the feature selection on every training set must be performed.^{ [19] } - By allowing some of the training data to also be included in the test set – this can happen due to "twinning" in the data set, whereby some exactly identical or nearly identical samples are present in the data set. Note that to some extent twinning always takes place even in perfectly independent training and validation samples. This is because some of the training sample observations will have nearly identical values of predictors as validation sample observations. And some of these will correlate with a target at better than chance levels in the same direction in both training and validation when they are actually driven by confounded predictors with poor external validity. If such a cross-validated model is selected from a
*k*-fold set, human confirmation bias will be at work and determine that such a model has been validated. This is why traditional cross-validation needs to be supplemented with controls for human bias and confounded model specification like swap sampling and prospective studies.

Since the order of the data is important, cross-validation might be problematic for time-series models. A more appropriate approach might be to use forward chaining.

However, if performance is describe by a single summary statistic, it is possible that the approach described by ^{ [20] } as a stationary bootstrap will work. The statistic of the bootstrap needs to accept an interval of the time series and return the summary statistic on it. The call to the stationary bootstrap needs to specify an appropriate mean interval length.

Cross-validation can be used to compare the performances of different predictive modeling procedures. For example, suppose we are interested in optical character recognition, and we are considering using either support vector machines (SVM) or *k*-nearest neighbors (KNN) to predict the true character from an image of a handwritten character. Using cross-validation, we could objectively compare these two methods in terms of their respective fractions of misclassified characters. If we simply compared the methods based on their in-sample error rates, the KNN method would likely appear to perform better, since it is more flexible and hence more prone to overfitting ^{[ citation needed ]} compared to the SVM method.

Cross-validation can also be used in *variable selection*.^{ [21] } Suppose we are using the expression levels of 20 proteins to predict whether a cancer patient will respond to a drug. A practical goal would be to determine which subset of the 20 features should be used to produce the best predictive model. For most modeling procedures, if we compare feature subsets using the in-sample error rates, the best performance will occur when all 20 features are used. However under cross-validation, the model with the best fit will generally include only a subset of the features that are deemed truly informative.

A recent development in medical statistics is its use in meta-analysis. It forms the basis of the validation statistic, Vn which is used to test the statistical validity of meta-analysis summary estimates.^{ [22] } It has also been used in a more conventional sense in meta-analysis to estimate the likely prediction error of meta-analysis results.^{ [23] }

Wikimedia Commons has media related to . Cross-validation (statistics) |

- ↑ Geisser, Seymour (1993).
*Predictive Inference*. New York, NY: Chapman and Hall. ISBN 978-0-412-03471-8. - 1 2 Kohavi, Ron (1995). "A study of cross-validation and bootstrap for accuracy estimation and model selection".
*Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence*. San Mateo, CA: Morgan Kaufmann.**2**(12): 1137–1143. CiteSeerX 10.1.1.48.529 . - ↑ Devijver, Pierre A.; Kittler, Josef (1982).
*Pattern Recognition: A Statistical Approach*. London, GB: Prentice-Hall. - ↑ "What is the difference between test set and validation set?" . Retrieved 10 October 2018.
- ↑ "Newbie question: Confused about train, validation and test data!". Archived from the original on 2015-03-14. Retrieved 2013-11-14.CS1 maint: BOT: original-url status unknown (link)
- ↑ Cawley, Gavin C.; Talbot, Nicola L. C. (2010). "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation" (PDF).
**11**. Journal of Machine Learning Research: 2079–2107. - ↑ Grossman, Robert; Seni, Giovanni; Elder, John; Agarwal, Nitin; Liu, Huan (2010). "Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions".
*Synthesis Lectures on Data Mining and Knowledge Discovery*. Morgan & Claypool.**2**: 1–126. doi:10.2200/S00240ED1V01Y200912DMK002. - ↑ Trippa, Lorenzo; Waldron, Levi; Huttenhower, Curtis; Parmigiani, Giovanni (March 2015). "Bayesian nonparametric cross-study validation of prediction methods".
*The Annals of Applied Statistics*.**9**(1): 402–428. arXiv: 1506.00474 . doi:10.1214/14-AOAS798. ISSN 1932-6157. - ↑ McLachlan, Geoffrey J.; Do, Kim-Anh; Ambroise, Christophe (2004).
*Analyzing microarray gene expression data*. Wiley. - ↑ "Cross Validation" . Retrieved 11 November 2012.
- ↑ Arlot, Sylvain; Celisse, Alain (2010). "A survey of cross-validation procedures for model selection".
*Statistics Surveys*.**4**: 40–79. doi:10.1214/09-SS054.In brief, CV consists in averaging several hold-out estimators of the risk corresponding to different data splits.

- ↑ Dubitzky, Werner; Granzow, Martin; Berrar, Daniel (2007).
*Fundamentals of data mining in genomics and proteomics*. Springer Science & Business Media. p. 178. - ↑ "Nested versus non-nested cross-validation" . Retrieved 19 February 2019.
- ↑ "Thoughts on prediction and cross-validation." Ronald Christensen, Department of Mathematics and Statistics University of New Mexico, May 21, 2015. Retrieved from http://www.math.unm.edu/~fletcher/Prediction.pdf on May 31, 2017.
- 1 2 Efron, Bradley; Tibshirani, Robert (1997). "Improvements on cross-validation: The .632 + Bootstrap Method".
*Journal of the American Statistical Association*.**92**(438): 548–560. doi:10.2307/2965703. JSTOR 2965703. MR 1467848. - ↑ Stone, Mervyn (1977). "Asymptotics for and against cross-validation".
*Biometrika*.**64**(1): 29–35. doi:10.1093/biomet/64.1.29. JSTOR 2335766. MR 0474601. - ↑ Consortium, MAQC (2010). "The Microarray Quality Control (MAQC)-II study of common practices for the development and validation of microarray-based predictive models".
*Nature Biotechnology*. London: Nature Publishing Group.**28**(8): 827–838. doi:10.1038/nbt.1665. PMC 3315840 . PMID 20676074. - ↑ Bermingham, Mairead L.; Pong-Wong, Ricardo; Spiliopoulou, Athina; Hayward, Caroline; Rudan, Igor; Campbell, Harry; Wright, Alan F.; Wilson, James F.; Agakov, Felix; Navarro, Pau; Haley, Chris S. (2015). "Application of high-dimensional feature selection: evaluation for genomic prediction in man".
*Sci. Rep.***5**: 10312. Bibcode:2015NatSR...510312B. doi:10.1038/srep10312. - ↑ Varma, Sudhir; Simon, Richard (2006). "Bias in error estimation when using cross-validation for model selection".
*BMC Bioinformatics*.**7**: 91. doi:10.1186/1471-2105-7-91. PMC 1397873 . PMID 16504092. - ↑ https://amstat.tandfonline.com/doi/abs/10.1080/01621459.1994.10476870#.XE-ENlVKilw Politis and Romano
- ↑ Picard, Richard; Cook, Dennis (1984). "Cross-Validation of Regression Models".
*Journal of the American Statistical Association*.**79**(387): 575–583. doi:10.2307/2288403. JSTOR 2288403. - ↑ Willis BH, Riley RD (2017). "Measuring the statistical validity of summary meta-analysis and meta-regression results for use in clinical practice".
*Statistics in Medicine*.**36**(21): 3283–3301. doi:10.1002/sim.7372. PMC 5575530 . PMID 28620945. - ↑ Riley RD, Ahmed I, Debray TP, Willis BH, Noordzij P, Higgins JP, Deeks JJ (2015). "Summarising and validating test accuracy results across multiple studies for use in clinical practice".
*Statistics in Medicine*.**34**(13): 2081–2103. doi:10.1002/sim.6471. PMC 4973708 . PMID 25800943.CS1 maint: Multiple names: authors list (link)

This page is based on this Wikipedia article

Text is available under the CC BY-SA 4.0 license; additional terms may apply.

Images, videos and audio are available under their respective licenses.

Text is available under the CC BY-SA 4.0 license; additional terms may apply.

Images, videos and audio are available under their respective licenses.

**Supervised learning** is the machine learning task of learning a function that maps an input to an output based on example input-output pairs. It infers a function from *labeled training data* consisting of a set of *training examples*. In supervised learning, each example is a *pair* consisting of an input object and a desired output value. A supervised learning algorithm analyzes the training data and produces an inferred function, which can be used for mapping new examples. An optimal scenario will allow for the algorithm to correctly determine the class labels for unseen instances. This requires the learning algorithm to generalize from the training data to unseen situations in a "reasonable" way.

In statistics, the **mean squared error** (**MSE**) or **mean squared deviation** (**MSD**) of an estimator measures the average of the squares of the errors—that is, the average squared difference between the estimated values and what is estimated. MSE is a risk function, corresponding to the expected value of the squared error loss. The fact that MSE is almost always strictly positive is because of randomness or because the estimator does not account for information that could produce a more accurate estimate.

In statistical modeling, **regression analysis** is a set of statistical processes for estimating the relationships among variables. It includes many techniques for modeling and analyzing several variables, when the focus is on the relationship between a dependent variable and one or more independent variables. More specifically, regression analysis helps one understand how the typical value of the dependent variable changes when any one of the independent variables is varied, while the other independent variables are held fixed.

**Bootstrap aggregating**, also called **bagging**, is a machine learning ensemble meta-algorithm designed to improve the stability and accuracy of machine learning algorithms used in statistical classification and regression. It also reduces variance and helps to avoid overfitting. Although it is usually applied to decision tree methods, it can be used with any type of method. Bagging is a special case of the model averaging approach.

**Random forests** or **random decision forests** are an ensemble learning method for classification, regression and other tasks that operates by constructing a multitude of decision trees at training time and outputting the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees. Random decision forests correct for decision trees' habit of overfitting to their training set.

In statistics, the **coefficient of determination**, denoted *R*^{2} or *r*^{2} and pronounced "R squared", is the proportion of the variance in the dependent variable that is predictable from the independent variable(s).

In statistics, **ordinary least squares** (**OLS**) is a type of linear least squares method for estimating the unknown parameters in a linear regression model. OLS chooses the parameters of a linear function of a set of explanatory variables by the principle of least squares: minimizing the sum of the squares of the differences between the observed dependent variable in the given dataset and those predicted by the linear function.

**Regression dilution**, also known as **regression attenuation**, is the biasing of the regression slope towards zero, caused by errors in the independent variable.

In statistics, **resampling** is any of a variety of methods for doing one of the following:

- Estimating the precision of sample statistics by using subsets of available data (
**jackknifing**) or drawing randomly with replacement from a set of data points (**bootstrapping**) - Exchanging labels on data points when performing significance tests
- Validating models by using random subsets

**Nonparametric regression** is a category of regression analysis in which the predictor does not take a predetermined form but is constructed according to information derived from the data. Nonparametric regression requires larger sample sizes than regression based on parametric models because the data must supply the model structure as well as the model estimates.

In statistics, **stepwise regression** is a method of fitting regression models in which the choice of predictive variables is carried out by an automatic procedure. In each step, a variable is considered for addition to or subtraction from the set of explanatory variables based on some prespecified criterion. Usually, this takes the form of a sequence of *F*-tests or *t*-tests, but other techniques are possible, such as adjusted *R*^{2}, Akaike information criterion, Bayesian information criterion, Mallows's *C _{p}*, PRESS, or false discovery rate.

In statistics, **bootstrapping** is any test or metric that relies on random sampling with replacement. Bootstrapping allows assigning measures of accuracy to sample estimates. This technique allows estimation of the sampling distribution of almost any statistic using random sampling methods. Generally, it falls in the broader class of resampling methods.

In statistics, **Mallows’s C_{p}**, named for Colin Lingwood Mallows, is used to assess the fit of a regression model that has been estimated using ordinary least squares. It is applied in the context of model selection, where a number of predictor variables are available for predicting some outcome, and the goal is to find the best model involving a subset of these predictors. A small value of C

The **root-mean-square deviation (RMSD)** or **root-mean-square error (RMSE)** is a frequently used measure of the differences between values predicted by a model or an estimator and the values observed. The RMSD represents the square root of the second sample moment of the differences between predicted values and observed values or the quadratic mean of these differences. These deviations are called *residuals* when the calculations are performed over the data sample that was used for estimation and are called *errors* when computed out-of-sample. The RMSD serves to aggregate the magnitudes of the errors in predictions for various times into a single measure of predictive power. RMSD is a measure of accuracy, to compare forecasting errors of different models for a particular dataset and not between datasets, as it is scale-dependent.

In statistics, **regression validation** is the process of deciding whether the numerical results quantifying hypothesized relationships between variables, obtained from regression analysis, are acceptable as descriptions of the data. The validation process can involve analyzing the goodness of fit of the regression, analyzing whether the regression residuals are random, and checking whether the model's predictive performance deteriorates substantially when applied to data that were not used in model estimation.

In statistics, **principal component regression** (**PCR**) is a regression analysis technique that is based on principal component analysis (PCA). Typically, it considers regressing the outcome on a set of covariates based on a standard linear regression model, but uses PCA for estimating the unknown regression coefficients in the model.

In statistics, **multivariate adaptive regression splines** (**MARS**) is a form of regression analysis introduced by Jerome H. Friedman in 1991. It is a non-parametric regression technique
and can be seen as an extension of linear models that automatically models nonlinearities and interactions between variables.

In statistics and machine learning, the **bias–variance tradeoff** is the property of a set of predictive models whereby models with a lower bias in parameter estimation have a higher variance of the parameter estimates across samples, and vice versa. The **bias–variance dilemma** or **problem** is the conflict in trying to simultaneously minimize these two sources of error that prevent supervised learning algorithms from generalizing beyond their training set:

In statistics, **linear regression** is a linear approach to modelling the relationship between a scalar response and one or more explanatory variables. The case of one explanatory variable is called simple linear regression. For more than one explanatory variable, the process is called **multiple linear regression**. This term is distinct from multivariate linear regression, where multiple correlated dependent variables are predicted, rather than a single scalar variable.