Ordinal regression

Last updated February 13, 2024

In statistics, ordinal regression, also called ordinal classification, is a type of regression analysis used for predicting an ordinal variable, i.e. a variable whose value exists on an arbitrary scale where only the relative ordering between different values is significant. It can be considered an intermediate problem between regression and classification.^[1]^[2] Examples of ordinal regression are ordered logit and ordered probit. Ordinal regression turns up often in the social sciences, for example in the modeling of human levels of preference (on a scale from, say, 1–5 for "very poor" through "excellent"), as well as in information retrieval. In machine learning, ordinal regression may also be called ranking learning.^[3]^{[lower-alpha 1]}

Linear models for ordinal regression

Ordinal regression can be performed using a generalized linear model (GLM) that fits both a coefficient vector and a set of thresholds to a dataset. Suppose one has a set of observations, represented by length- $p$ vectors $x 1$ through $x n$ , with associated responses $y 1$ through $y n$ , where each $y i$ is an ordinal variable on a scale $1, ..., K$ . For simplicity, and without loss of generality, we assume $y$ is a non-decreasing vector, that is, $\leq$ . To this data, one fits a length- $p$ coefficient vector $w$ and a set of thresholds $θ 1, ..., θ K -1$ with the property that $θ 1 < θ 2 < ... < θ K -1$ . This set of thresholds divides the real number line into $K$ disjoint segments, corresponding to the $K$ response levels.

The model can now be formulated as

\Pr(y\leq i\mid \mathbf {x} )=\sigma (\theta _{i}-\mathbf {w} \cdot \mathbf {x} )

or, the cumulative probability of the response $y$ being at most $i$ is given by a function $σ$ (the inverse link function) applied to a linear function of $x$ . Several choices exist for $σ$ ; the logistic function

\sigma (\theta _{i}-\mathbf {w} \cdot \mathbf {x} )={\frac {1}{1+e^{-(\theta _{i}-\mathbf {w} \cdot \mathbf {x} )}}}

gives the ordered logit model, while using the probit function gives the ordered probit model. A third option is to use an exponential function

\sigma (\theta _{i}-\mathbf {w} \cdot \mathbf {x} )=1-\exp(-\exp(\theta _{i}-\mathbf {w} \cdot \mathbf {x} ))

which gives the proportional hazards model.^[4]

Latent variable model

The probit version of the above model can be justified by assuming the existence of a real-valued latent variable (unobserved quantity) $y*$ , determined by^[5]

y^{*}=\mathbf {w} \cdot \mathbf {x} +\varepsilon

where $ε$ is normally distributed with zero mean and unit variance, conditioned on $x$ . The response variable $y$ results from an "incomplete measurement" of $y*$ , where one only determines the interval into which $y*$ falls:

y={\begin{cases}1&{\text{if}}~~y^{*}\leq \theta _{1},\\2&{\text{if}}~~\theta _{1}<y^{*}\leq \theta _{2},\\3&{\text{if}}~~\theta _{2}<y^{*}\leq \theta _{3}\\\vdots \\K&{\text{if}}~~\theta _{K-1}<y^{*}.\end{cases}}

Defining $θ 0 = -\infty$ and $θ K = \infty$ , the above can be summarized as $y = k$ if and only if $θ k -1 < y * \leq θ k$ .

From these assumptions, one can derive the conditional distribution of $y$ as^[5]

{\begin{aligned}P(y=k\mid \mathbf {x} )&=P(\theta _{k-1}<y^{*}\leq \theta _{k}\mid \mathbf {x} )\\&=P(\theta _{k-1}<\mathbf {w} \cdot \mathbf {x} +\varepsilon \leq \theta _{k})\\&=\Phi (\theta _{k}-\mathbf {w} \cdot \mathbf {x} )-\Phi (\theta _{k-1}-\mathbf {w} \cdot \mathbf {x} )\end{aligned}}

where $Φ$ is the cumulative distribution function of the standard normal distribution, and takes on the role of the inverse link function $σ$ . The log-likelihood of the model for a single training example $x i$ , $y i$ can now be stated as^[5]

\log {\mathcal {L}}(\mathbf {w} ,\mathbf {\theta } \mid \mathbf {x} _{i},y_{i})=\sum _{k=1}^{K}[y_{i}=k]\log[\Phi (\theta _{k}-\mathbf {w} \cdot \mathbf {x} _{i})-\Phi (\theta _{k-1}-\mathbf {w} \cdot \mathbf {x} _{i})]

(using the Iverson bracket $[y i = k]$ .) The log-likelihood of the ordered logit model is analogous, using the logistic function instead of $Φ$ .^[6]

Alternative models

In machine learning, alternatives to the latent-variable models of ordinal regression have been proposed. An early result was PRank, a variant of the perceptron algorithm that found multiple parallel hyperplanes separating the various ranks; its output is a weight vector $w$ and a sorted vector of $K -1$ thresholds $θ$ , as in the ordered logit/probit models. The prediction rule for this model is to output the smallest rank $k$ such that $wx < θ k$ .^[7]

Other methods rely on the principle of large-margin learning that also underlies support vector machines.^[8]^[9]

Another approach is given by Rennie and Srebro, who, realizing that "even just evaluating the likelihood of a predictor is not straight-forward" in the ordered logit and ordered probit models, propose fitting ordinal regression models by adapting common loss functions from classification (such as the hinge loss and log loss) to the ordinal case.^[10]

Software

ORCA (Ordinal Regression and Classification Algorithms) is an Octave/MATLAB framework including a wide set of ordinal regression methods.^[11]

R packages that provide ordinal regression methods include MASS^[12] and Ordinal.^[13]

Notes

↑ Not to be confused with learning to rank.

Related Research Articles

<span class="mw-page-title-main">Navier–Stokes equations</span> Equations describing the motion of viscous fluid substances

The Navier–Stokes equations are partial differential equations which describe the motion of viscous fluid substances. They were named after French engineer and physicist Claude-Louis Navier and the Irish physicist and mathematician George Gabriel Stokes. They were developed over several decades of progressively building the theories, from 1822 (Navier) to 1842–1850 (Stokes).

In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statistical model, the observed data is most probable. The point in the parameter space that maximizes the likelihood function is called the maximum likelihood estimate. The logic of maximum likelihood is both intuitive and flexible, and as such the method has become a dominant means of statistical inference.

In statistics, the logistic model is a statistical model that models the log-odds of an event as a linear combination of one or more independent variables. In regression analysis, logistic regression is estimating the parameters of a logistic model. Formally, in binary logistic regression there is a single binary dependent variable, coded by an indicator variable, where the two values are labeled "0" and "1", while the independent variables can each be a binary variable or a continuous variable. The corresponding probability of the value labeled "1" can vary between 0 and 1, hence the labeling; the function that converts log-odds to probability is the logistic function, hence the name. The unit of measurement for the log-odds scale is called a logit, from logistic unit, hence the alternative names. See § Background and § Definition for formal mathematics, and § Example for a worked example.

In probability and statistics, an exponential family is a parametric set of probability distributions of a certain form, specified below. This special form is chosen for mathematical convenience, including the enabling of the user to calculate expectations, covariances using differentiation based on some useful algebraic properties, as well as for generality, as exponential families are in a sense very natural sets of distributions to consider. The term exponential class is sometimes used in place of "exponential family", or the older term Koopman–Darmois family. Sometimes loosely referred to as "the" exponential family, this class of distributions is distinct because they all possess a variety of desirable properties, most importantly the existence of a sufficient statistic.

In statistics, Gibbs sampling or a Gibbs sampler is a Markov chain Monte Carlo (MCMC) algorithm for sampling from a specified multivariate probability distribution when direct sampling from the joint distribution is difficult, but sampling from the conditional distribution is more practical. This sequence can be used to approximate the joint distribution ; to approximate the marginal distribution of one of the variables, or some subset of the variables ; or to compute an integral. Typically, some of the variables correspond to observations whose values are known, and hence do not need to be sampled.

In statistics, a generalized linear model (GLM) is a flexible generalization of ordinary linear regression. The GLM generalizes linear regression by allowing the linear model to be related to the response variable via a link function and by allowing the magnitude of the variance of each measurement to be a function of its predicted value.

Random forests or random decision forests is an ensemble learning method for classification, regression and other tasks that operates by constructing a multitude of decision trees at training time. For classification tasks, the output of the random forest is the class selected by most trees. For regression tasks, the mean or average prediction of the individual trees is returned. Random decision forests correct for decision trees' habit of overfitting to their training set.

In probability and statistics, a circular distribution or polar distribution is a probability distribution of a random variable whose values are angles, usually taken to be in the range [0, 2π). A circular distribution is often a continuous probability distribution, and hence has a probability density, but such distributions can also be discrete, in which case they are called circular lattice distributions. Circular distributions can be used even when the variables concerned are not explicitly angles: the main consideration is that there is not usually any real distinction between events occurring at the opposite ends of the range, and the division of the range could notionally be made at any point.

In statistics, a linear probability model (LPM) is a special case of a binary regression model. Here the dependent variable for each observation takes values which are either 0 or 1. The probability of observing a 0 or 1 in any one case is treated as depending on one or more explanatory variables. For the "linear probability model", this relationship is a particularly simple one, and allows the model to be fitted by linear regression.

In statistics, a probit model is a type of regression where the dependent variable can take only two values, for example married or not married. The word is a portmanteau, coming from probability + unit. The purpose of the model is to estimate the probability that an observation with particular characteristics will fall into a specific one of the categories; moreover, classifying observations based on their predicted probabilities is a type of binary classification model.

In probability theory and statistics, the characteristic function of any real-valued random variable completely defines its probability distribution. If a random variable admits a probability density function, then the characteristic function is the Fourier transform of the probability density function. Thus it provides an alternative route to analytical results compared with working directly with probability density functions or cumulative distribution functions. There are particularly simple results for the characteristic functions of distributions defined by the weighted sums of random variables.

In statistics, the ordered logit model is an ordinal regression model—that is, a regression model for ordinal dependent variables—first considered by Peter McCullagh. For example, if one question on a survey is to be answered by a choice among "poor", "fair", "good", "very good" and "excellent", and the purpose of the analysis is to see how well that response can be predicted by the responses to other questions, some of which may be quantitative, then ordered logistic regression may be used. It can be thought of as an extension of the logistic regression model that applies to dichotomous dependent variables, allowing for more than two (ordered) response categories.

In statistics, ordered probit is a generalization of the widely used probit analysis to the case of more than two outcomes of an ordinal dependent variable. Similarly, the widely used logit method also has a counterpart ordered logit. Ordered probit, like ordered logit, is a particular method of ordinal regression.

In statistics, multinomial logistic regression is a classification method that generalizes logistic regression to multiclass problems, i.e. with more than two possible discrete outcomes. That is, it is a model that is used to predict the probabilities of the different possible outcomes of a categorically distributed dependent variable, given a set of independent variables.

In statistics, binomial regression is a regression analysis technique in which the response has a binomial distribution: it is the number of successes in a series of independent Bernoulli trials, where each trial has probability of success . In binomial regression, the probability of a success is related to explanatory variables: the corresponding concept in ordinary regression is to relate the mean value of the unobserved response to explanatory variables.

For certain applications in linear algebra, it is useful to know properties of the probability distribution of the largest eigenvalue of a finite sum of random matrices. Suppose $is a finite sequence of random matrices. Analogous to the well-known Chernoff bound for sums of scalars, a bound on the following is sought for a given parameter t :$

In statistics, the variance function is a smooth function that depicts the variance of a random quantity as a function of its mean. The variance function is a measure of heteroscedasticity and plays a large role in many settings of statistical modelling. It is a main ingredient in the generalized linear model framework and a tool used in non-parametric regression, semiparametric regression and functional data analysis. In parametric modeling, variance functions take on a parametric form and explicitly describe the relationship between the variance and the mean of a random quantity. In a non-parametric setting, the variance function is assumed to be a smooth function.

The generalized functional linear model (GFLM) is an extension of the generalized linear model (GLM) that allows one to regress univariate responses of various types on functional predictors, which are mostly random trajectories generated by a square-integrable stochastic processes. Similarly to GLM, a link function relates the expected value of the response variable to a linear predictor, which in case of GFLM is obtained by forming the scalar product of the random predictor function $with a smooth parameter function . Functional Linear Regression, Functional Poisson Regression and Functional Binomial Regression, with the important Functional Logistic Regression included, are special cases of GFLM. Applications of GFLM include classification and discrimination of stochastic processes and functional data.$

In statistics, the class of vector generalized linear models (VGLMs) was proposed to enlarge the scope of models catered for by generalized linear models (GLMs). In particular, VGLMs allow for response variables outside the classical exponential family and for more than one parameter. Each parameter can be transformed by a link function. The VGLM framework is also large enough to naturally accommodate multiple responses; these are several independent responses each coming from a particular statistical distribution with possibly different parameter values.

Ordinal data is a categorical, statistical data type where the variables have natural, ordered categories and the distances between the categories are not known. These data exist on an ordinal scale, one of four levels of measurement described by S. S. Stevens in 1946. The ordinal scale is distinguished from the nominal scale by having a ranking. It also differs from the interval scale and ratio scale by not having category widths that represent equal increments of the underlying attribute.

References

↑ Winship, Christopher; Mare, Robert D. (1984). "Regression Models with Ordinal Variables" (PDF). American Sociological Review. 49 (4): 512–525. doi:10.2307/2095465. JSTOR 2095465.
↑ Gutiérrez, P. A.; Pérez-Ortiz, M.; Sánchez-Monedero, J.; Fernández-Navarro, F.; Hervás-Martínez, C. (January 2016). "Ordinal Regression Methods: Survey and Experimental Study". IEEE Transactions on Knowledge and Data Engineering. 28 (1): 127–146. doi:10.1109/TKDE.2015.2457911. hdl: 10396/14494 . ISSN 1041-4347.
↑ Shashua, Amnon; Levin, Anat (2002). Ranking with large margin principle: Two approaches. NIPS.
↑ McCullagh, Peter (1980). "Regression models for ordinal data". Journal of the Royal Statistical Society. Series B (Methodological). 42 (2): 109–142.
1 2 3 Wooldridge, Jeffrey M. (2010). Econometric Analysis of Cross Section and Panel Data. MIT Press. pp. 655–657. ISBN 9780262232586.
↑ Agresti, Alan (23 October 2010). "Modeling Ordinal Categorical Data" (PDF). Retrieved 23 July 2015.
↑ Crammer, Koby; Singer, Yoram (2001). Pranking with ranking. NIPS.
↑ Chu, Wei; Keerthi, S. Sathiya (2007). "Support vector ordinal regression". Neural Computation. 19 (3): 792–815. CiteSeerX 10.1.1.297.3637 . doi:10.1162/neco.2007.19.3.792. PMID 17298234.
↑ Herbrich, Ralf; Graepel, Thore; Obermayer, Klaus (2000). "Large Margin Rank Boundaries for Ordinal Regression". Advances in Large Margin Classifiers. MIT Press. pp. 115–132.
↑ Rennie, Jason D. M.; Srebro, Nathan (2005). Loss Functions for Preference Levels: Regression with Discrete Ordered Labels (PDF). Proc. IJCAI Multidisciplinary Workshop on Advances in Preference Handling.
↑ orca: Ordinal Regression and Classification Algorithms, AYRNA, 2017-11-21, retrieved 2017-11-21
↑ "Modern Applied Statistics with S, 4th ed". www.stats.ox.ac.uk. Retrieved 2020-07-15.
↑ Christensen, Rune Haubo B. (2020-06-05), runehaubo/ordinal , retrieved 2020-07-15