Deming regression

Last updated October 29, 2024

In statistics, Deming regression, named after W. Edwards Deming, is an errors-in-variables model that tries to find the line of best fit for a two-dimensional data set. It differs from the simple linear regression in that it accounts for errors in observations on both the x- and the y- axis. It is a special case of total least squares, which allows for any number of predictors and a more complicated error structure.

Deming regression is equivalent to the maximum likelihood estimation of an errors-in-variables model in which the errors for the two variables are assumed to be independent and normally distributed, and the ratio of their variances, denoted δ, is known.^[1] In practice, this ratio might be estimated from related data-sources; however the regression procedure takes no account for possible errors in estimating this ratio.

The Deming regression is only slightly more difficult to compute than the simple linear regression. Most statistical software packages used in clinical chemistry offer Deming regression.

The model was originally introduced by Adcock (1878) who considered the case δ = 1, and then more generally by Kummell (1879) with arbitrary δ. However their ideas remained largely unnoticed for more than 50 years, until they were revived by Koopmans (1936) and later propagated even more by Deming (1943). The latter book became so popular in clinical chemistry and related fields that the method was even dubbed Deming regression in those fields.^[2]

Specification

Assume that the available data (y_i, x_i) are measured observations of the "true" values (y_i*, x_i*), which lie on the regression line:

{\begin{aligned}y_{i}&=y_{i}^{*}+\varepsilon _{i},\\x_{i}&=x_{i}^{*}+\eta _{i},\end{aligned}}

where errors ε and η are independent and the ratio of their variances is assumed to be known:

\delta ={\frac {\sigma _{\varepsilon }^{2}}{\sigma _{\eta }^{2}}}.

In practice, the variances of the $x$ and $y$ parameters are often unknown, which complicates the estimate of $\delta$ . Note that when the measurement method for $x$ and $y$ is the same, these variances are likely to be equal, so $\delta =1$ for this case.

We seek to find the line of "best fit"

y^{*}=\beta _{0}+\beta _{1}x^{*},

such that the weighted sum of squared residuals of the model is minimized:^[3]

SSR=\sum _{i=1}^{n}{\bigg (}{\frac {\varepsilon _{i}^{2}}{\sigma _{\varepsilon }^{2}}}+{\frac {\eta _{i}^{2}}{\sigma _{\eta }^{2}}}{\bigg )}={\frac {1}{\sigma _{\epsilon }^{2}}}\sum _{i=1}^{n}{\Big (}(y_{i}-\beta _{0}-\beta _{1}x_{i}^{*})^{2}+\delta (x_{i}-x_{i}^{*})^{2}{\Big )}\ \to \ \min _{\beta _{0},\beta _{1},x_{1}^{*},\ldots ,x_{n}^{*}}SSR

See Jensen (2007) for a full derivation.

Solution

The solution can be expressed in terms of the second-degree sample moments. That is, we first calculate the following quantities (all sums go from i = 1 to n):

{\begin{aligned}{\overline {x}}&={\tfrac {1}{n}}\sum x_{i}&{\overline {y}}&={\tfrac {1}{n}}\sum y_{i},\\s_{xx}&={\tfrac {1}{n}}\sum (x_{i}-{\overline {x}})^{2}&&={\overline {x^{2}}}-{\overline {x}}^{2},\\s_{xy}&={\tfrac {1}{n}}\sum (x_{i}-{\overline {x}})(y_{i}-{\overline {y}})&&={\overline {xy}}-{\overline {x}}\,{\overline {y}},\\s_{yy}&={\tfrac {1}{n}}\sum (y_{i}-{\overline {y}})^{2}&&={\overline {y^{2}}}-{\overline {y}}^{2}.\end{aligned}}\,

Finally, the least-squares estimates of model's parameters will be^[4]

{\begin{aligned}&{\hat {\beta }}_{1}={\frac {s_{yy}-\delta s_{xx}+{\sqrt {(s_{yy}-\delta s_{xx})^{2}+4\delta s_{xy}^{2}}}}{2s_{xy}}},\\&{\hat {\beta }}_{0}={\overline {y}}-{\hat {\beta }}_{1}{\overline {x}},\\&{\hat {x}}_{i}^{*}=x_{i}+{\frac {{\hat {\beta }}_{1}}{{\hat {\beta }}_{1}^{2}+\delta }}(y_{i}-{\hat {\beta }}_{0}-{\hat {\beta }}_{1}x_{i}).\end{aligned}}

Orthogonal regression

For the case of equal error variances, i.e., when $\delta =1$ , Deming regression becomes orthogonal regression: it minimizes the sum of squared perpendicular distances from the data points to the regression line. In this case, denote each observation as a point $z_{j}=x_{j}+iy_{j}$ in the complex plane (i.e., the point $(x_{j},y_{j})$ where $i$ is the imaginary unit). Denote as $S=\sum {(z_{j}-{\overline {z}})^{2}}$ the sum of the squared differences of the data points from the centroid ${\overline {z}}={\tfrac {1}{n}}\sum z_{j}$ (also denoted in complex coordinates), which is the point whose horizontal and vertical locations are the averages of those of the data points. Then:^[5]

If $S=0$ , then every line through the centroid is a line of best orthogonal fit.
If $S\neq 0$ , the orthogonal regression line goes through the centroid and is parallel to the vector from the origin to ${\sqrt {S}}$ .

A trigonometric representation of the orthogonal regression line was given by Coolidge in 1913.^[6]

Application

In the case of three non-collinear points in the plane, the triangle with these points as its vertices has a unique Steiner inellipse that is tangent to the triangle's sides at their midpoints. The major axis of this ellipse falls on the orthogonal regression line for the three vertices.^[7] The quantification of a biological cell's intrinsic cellular noise can be quantified upon applying Deming regression to the observed behavior of a two reporter synthetic biological circuit.^[8]

When humans are asked to draw a linear regression on a scatterplot by guessing, their answers are closer to orthogonal regression than to ordinary least squares regression.^[9]

York regression

The York regression extends Deming regression by allowing correlated errors in x and y.^[10]

Related Research Articles

In mathematical physics and mathematics, the Pauli matrices are a set of three $2 \times 2$ complex matrices that are traceless, Hermitian, involutory and unitary. Usually indicated by the Greek letter sigma, they are occasionally denoted by tau when used in connection with isospin symmetries.

The method of least squares is a parameter estimation method in regression analysis based on minimizing the sum of the squares of the residuals made in the results of each individual equation.

In mechanics and geometry, the 3D rotation group, often denoted SO(3), is the group of all rotations about the origin of three-dimensional Euclidean space $under the operation of composition.$

In continuum mechanics, the infinitesimal strain theory is a mathematical approach to the description of the deformation of a solid body in which the displacements of the material particles are assumed to be much smaller than any relevant dimension of the body; so that its geometry and the constitutive properties of the material at each point of space can be assumed to be unchanged by the deformation.

In statistics, the logistic model is a statistical model that models the log-odds of an event as a linear combination of one or more independent variables. In regression analysis, logistic regression estimates the parameters of a logistic model. In binary logistic regression there is a single binary dependent variable, coded by an indicator variable, where the two values are labeled "0" and "1", while the independent variables can each be a binary variable or a continuous variable. The corresponding probability of the value labeled "1" can vary between 0 and 1, hence the labeling; the function that converts log-odds to probability is the logistic function, hence the name. The unit of measurement for the log-odds scale is called a logit, from logistic unit, hence the alternative names. See § Background and § Definition for formal mathematics, and § Example for a worked example.

In applied statistics, total least squares is a type of errors-in-variables regression, a least squares data modeling technique in which observational errors on both dependent and independent variables are taken into account. It is a generalization of Deming regression and also of orthogonal regression, and can be applied to both linear and non-linear models.

In statistics, the theory of minimum norm quadratic unbiased estimation (MINQUE) was developed by C. R. Rao. MINQUE is a theory alongside other estimation methods in estimation theory, such as the method of moments or maximum likelihood estimation. Similar to the theory of best linear unbiased estimation, MINQUE is specifically concerned with linear regression models. The method was originally conceived to estimate heteroscedastic error variance in multiple linear regression. MINQUE estimators also provide an alternative to maximum likelihood estimators or restricted maximum likelihood estimators for variance components in mixed effects models. MINQUE estimators are quadratic forms of the response variable and are used to estimate a linear function of the variances.

In statistics, ordinary least squares (OLS) is a type of linear least squares method for choosing the unknown parameters in a linear regression model by the principle of least squares: minimizing the sum of the squares of the differences between the observed dependent variable in the input dataset and the output of the (linear) function of the independent variable. Some sources consider OLS to be linear regression.

In statistics, simple linear regression (SLR) is a linear regression model with a single explanatory variable. That is, it concerns two-dimensional sample points with one independent variable and one dependent variable and finds a linear function that, as accurately as possible, predicts the dependent variable values as a function of the independent variable. The adjective simple refers to the fact that the outcome variable is related to a single predictor.

In mathematics, in particular in algebraic geometry and differential geometry, Dolbeault cohomology (named after Pierre Dolbeault) is an analog of de Rham cohomology for complex manifolds. Let M be a complex manifold. Then the Dolbeault cohomology groups $depend on a pair of integers p and q and are realized as a subquotient of the space of complex differential forms of degree (p, q).$

Difference in differences is a statistical technique used in econometrics and quantitative research in the social sciences that attempts to mimic an experimental research design using observational study data, by studying the differential effect of a treatment on a 'treatment group' versus a 'control group' in a natural experiment. It calculates the effect of a treatment on an outcome by comparing the average change over time in the outcome variable for the treatment group to the average change over time for the control group. Although it is intended to mitigate the effects of extraneous factors and selection bias, depending on how the treatment group is chosen, this method may still be subject to certain biases.

The partition of sums of squares is a concept that permeates much of inferential statistics and descriptive statistics. More properly, it is the partitioning of sums of squared deviations or errors. Mathematically, the sum of squared deviations is an unscaled, or unadjusted measure of dispersion. When scaled for the number of degrees of freedom, it estimates the variance, or spread of the observations about their mean value. Partitioning of the sum of squared deviations into various components allows the overall variability in a dataset to be ascribed to different types or sources of variability, with the relative importance of each being quantified by the size of each component of the overall sum of squares.

In statistics, a fixed effects model is a statistical model in which the model parameters are fixed or non-random quantities. This is in contrast to random effects models and mixed models in which all or some of the model parameters are random variables. In many applications including econometrics and biostatistics a fixed effects model refers to a regression model in which the group means are fixed (non-random) as opposed to a random effects model in which the group means are a random sample from a population. Generally, data can be grouped according to several observed factors. The group means could be modeled as fixed or random effects for each grouping. In a fixed effects model each group mean is a group-specific fixed quantity.

The Newman–Penrose (NP) formalism is a set of notation developed by Ezra T. Newman and Roger Penrose for general relativity (GR). Their notation is an effort to treat general relativity in terms of spinor notation, which introduces complex forms of the usual variables used in GR. The NP formalism is itself a special case of the tetrad formalism, where the tensors of the theory are projected onto a complete vector basis at each point in spacetime. Usually this vector basis is chosen to reflect some symmetry of the spacetime, leading to simplified expressions for physical observables. In the case of the NP formalism, the vector basis chosen is a null tetrad: a set of four null vectors—two real, and a complex-conjugate pair. The two real members often asymptotically point radially inward and radially outward, and the formalism is well adapted to treatment of the propagation of radiation in curved spacetime. The Weyl scalars, derived from the Weyl tensor, are often used. In particular, it can be shown that one of these scalars— $in the appropriate frame—encodes the outgoing gravitational radiation of an asymptotically flat system.$

In mathematics, the Schur orthogonality relations, which were proven by Issai Schur through Schur's lemma, express a central fact about representations of finite groups. They admit a generalization to the case of compact groups in general, and in particular compact Lie groups, such as the rotation group SO(3).

Non-linear least squares is the form of least squares analysis used to fit a set of m observations with a model that is non-linear in n unknown parameters (m ≥ n). It is used in some forms of nonlinear regression. The basis of the method is to approximate the model by a linear one and to refine the parameters by successive iterations. There are many similarities to linear least squares, but also some significant differences. In economic theory, the non-linear least squares method is applied in (i) the probit regression, (ii) threshold regression, (iii) smooth regression, (iv) logistic link regression, (v) Box–Cox transformed regressors ( $).$

In statistics, principal component regression (PCR) is a regression analysis technique that is based on principal component analysis (PCA). PCR is a form of reduced rank regression.More specifically, PCR is used for estimating the unknown regression coefficients in a standard linear regression model.

The purpose of this page is to provide supplementary materials for the ordinary least squares article, reducing the load of the main article with mathematics and improving its accessibility, while at the same time retaining the completeness of exposition.

In statistics, errors-in-variables models or measurement error models are regression models that account for measurement errors in the independent variables. In contrast, standard regression models assume that those regressors have been measured exactly, or observed without error; as such, those models account only for errors in the dependent variables, or responses.

The hyperbolastic functions, also known as hyperbolastic growth models, are mathematical functions that are used in medical statistical modeling. These models were originally developed to capture the growth dynamics of multicellular tumor spheres, and were introduced in 2005 by Mohammad Tabatabai, David Williams, and Zoran Bursac. The precision of hyperbolastic functions in modeling real world problems is somewhat due to their flexibility in their point of inflection. These functions can be used in a wide variety of modeling problems such as tumor growth, stem cell proliferation, pharma kinetics, cancer growth, sigmoid activation function in neural networks, and epidemiological disease progression or regression.

References

Notes

↑ Linnet 1993.
↑ Cornbleet & Gochman 1979.
↑ Fuller 1987, Ch. 1.3.3.
↑ Glaister 2001.
↑ Minda & Phelps 2008, Theorem 2.3.
↑ Coolidge 1913.
↑ Minda & Phelps 2008, Corollary 2.4.
↑ Quarton 2020.
↑ Ciccione, Lorenzo; Dehaene, Stanislas (August 2021). "Can humans perform mental regression on a graph? Accuracy and bias in the perception of scatterplots". Cognitive Psychology. 128: 101406. doi:10.1016/j.cogpsych.2021.101406.
↑ York, D., Evensen, N. M., Martınez, M. L., and Delgado, J. D. B.: Unified equations for the slope, intercept, and standard errors of the best straight line, Am. J. Phys., 72, 367–375, https://doi.org/10.1119/1.1632486, 2004.

Bibliography

Adcock, R. J. (1878). "A problem in least squares". The Analyst. 5 (2): 53–54. doi: 10.2307/2635758 . JSTOR 2635758 .
Coolidge, J. L. (1913). "Two geometrical applications of the mathematics of least squares". The American Mathematical Monthly. 20 (6): 187–190. doi:10.2307/2973072. JSTOR 2973072.
Cornbleet, P.J.; Gochman, N. (1979). "Incorrect Least–Squares Regression Coefficients". Clinical Chemistry. 25 (3): 432–438. doi: 10.1093/clinchem/25.3.432 . PMID 262186.
Deming, W. E. (1943). Statistical adjustment of data. Wiley, NY (Dover Publications edition, 1985). ISBN 0-486-64685-8.
Fuller, Wayne A. (1987). Measurement error models. John Wiley & Sons, Inc. ISBN 0-471-86187-1.
Glaister, P. (2001). "Least squares revisited". The Mathematical Gazette . 85: 104–107. doi:10.2307/3620485. JSTOR 3620485. S2CID 125949467.
Jensen, Anders Christian (2007). "Deming regression, MethComp package" (PDF). Gentofte, Denmark: Steno Diabetes Center.
Koopmans, T. C. (1936). Linear regression analysis of economic time series. DeErven F. Bohn, Haarlem, Netherlands.
Kummell, C. H. (1879). "Reduction of observation equations which contain more than one observed quantity". The Analyst. 6 (4): 97–105. doi: 10.2307/2635646 . JSTOR 2635646 .
Linnet, K. (1993). "Evaluation of regression procedures for method comparison studies". Clinical Chemistry. 39 (3): 424–432. doi: 10.1093/clinchem/39.3.424 . PMID 8448852.
Minda, D.; Phelps, S. (2008). "Triangles, ellipses, and cubic polynomials". American Mathematical Monthly . 115 (8): 679–689. doi:10.1080/00029890.2008.11920581. MR 2456092. S2CID 15049234.
Quarton, T. G. (2020). "Uncoupling gene expression noise along the central dogma using genome engineered human cell lines". Nucleic Acids Research. 48 (16): 9406–9413. doi: 10.1093/nar/gkaa668 . PMC 7498316 . PMID 32810265.

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.

[FOOTNOTELinnet1993-1] Linnet 1993.

[FOOTNOTECornbleetGochman1979-2] Cornbleet & Gochman 1979.

[FOOTNOTEFuller1987Ch._1.3.3-3] Fuller 1987, Ch. 1.3.3.

[FOOTNOTEGlaister2001-4] Glaister 2001.

[FOOTNOTEMindaPhelps2008Theorem_2.3-5] Minda & Phelps 2008, Theorem 2.3.

[FOOTNOTECoolidge1913-6] Coolidge 1913.

[FOOTNOTEMindaPhelps2008Corollary_2.4-7] Minda & Phelps 2008, Corollary 2.4.

[FOOTNOTEQuarton2020-8] Quarton 2020.

[9] Ciccione, Lorenzo; Dehaene, Stanislas (August 2021). "Can humans perform mental regression on a graph? Accuracy and bias in the perception of scatterplots". Cognitive Psychology. 128: 101406. doi:10.1016/j.cogpsych.2021.101406.

[10] York, D., Evensen, N. M., Martınez, M. L., and Delgado, J. D. B.: Unified equations for the slope, intercept, and standard errors of the best straight line, Am. J. Phys., 72, 367–375, https://doi.org/10.1119/1.1632486, 2004.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]