Polynomial least squares

In mathematical statistics, polynomial least squares comprises a broad range of statistical methods for estimating an underlying polynomial that describes observations. These methods include polynomial regression, curve fitting, linear regression, least squares, ordinary least squares, simple linear regression, linear least squares, approximation theory and method of moments. Polynomial least squares has applications in radar trackers, estimation theory, signal processing, statistics, and econometrics.

Two common applications of polynomial least squares methods are generating a low-degree polynomial that approximates a complicated function and estimating an assumed underlying polynomial from corrupted (also known as "noisy") observations. The former is commonly used in statistics and econometrics to fit a scatter plot with a first degree polynomial (that is, a linear expression). ^{[1] [2] [3]} The latter is commonly used in target tracking in the form of Kalman filtering, which is effectively a recursive implementation of polynomial least squares. ^{[4] [5] [6] [7]} Estimating an assumed underlying deterministic polynomial can be used in econometrics as well. ^[8] In effect, both applications produce average curves as generalizations of the common average of a set of numbers, which is equivalent to zero degree polynomial least squares. ^{[1] [2] [9]}

In the above applications, the term "approximate" is used when no statistical measurement or observation errors are assumed, as when fitting a scatter plot. The term "estimate", derived from statistical estimation theory, is used when assuming that measurements or observations of a polynomial are corrupted.

Polynomial least squares estimate of a deterministic first degree polynomial corrupted with observation errors

Assume the deterministic first degree polynomial equation ${\displaystyle y}$ with unknown coefficients ${\displaystyle \alpha }$ and ${\displaystyle \beta }$ is written as

${\displaystyle y=\alpha +\beta t.}$

This is corrupted with an additive stochastic process ${\displaystyle \varepsilon }$ described as an error (noise in tracking), resulting in

${\displaystyle z=y+\varepsilon =\alpha +\beta t+\varepsilon .}$

Given observations ${\displaystyle z_{n}}$ from a sample, where the subscript ${\displaystyle n}$ is the observation index, the problem is to apply polynomial least squares to estimate ${\displaystyle y(t)}$, and to determine its variance along with its expected value.

Definitions and assumptions

(1) The term linearity in mathematics may be considered to take two forms that are sometimes confusing: a linear system or transformation (sometimes called an operator) ^[9] and a linear equation. The term "function" is often used to describe both a system and an equation, which may lead to confusion. A linear system is defined by

${\displaystyle f(ax+by)=af(x)+bf(y)}$

where ${\displaystyle a}$ and ${\displaystyle b}$ are constants, and where ${\displaystyle x}$ and ${\displaystyle y}$ are variables. In a linear system${\displaystyle E[f(x)]=f(E[x])}$, where ${\displaystyle E}$ is the linear expectation operator. A linear equation is a straight line as is the first degree polynomial described above.

(2) The error ${\displaystyle \varepsilon }$ is modeled as a zero mean stochastic process, sample points of which are random variables that are uncorrelated and assumed to have identical probability distributions (specifically same mean and variance), but not necessarily Gaussian, treated as inputs to polynomial least squares. Stochastic processes and random variables are described only by probability distributions. ^{[1] [9] [2]}

(3) Polynomial least squares is modeled as a linear signal processing system which processes statistical inputs deterministically, the output being the linearly processed empirically determined statistical estimate, variance, and expected value. ^{[6] [7] [8]}

(4) Polynomial least squares processing produces deterministic moments (analogous to mechanical moments), which may be considered as moments of sample statistics, but not of statistical moments. ^[8]

Polynomial least squares and the orthogonality principle

Approximating a function ${\displaystyle z(t)}$ with a polynomial

${\displaystyle {\hat {z}}(t)=\sum _{j=1}^{J}a_{j}t^{j-1}}$

where hat (^) denotes the estimate and (J − 1) is the polynomial degree, can be performed by applying the orthogonality principle. The sum of squared residuals can be written as

${\displaystyle \sum _{n=1}^{N}(z_{n}-{\hat {z}}_{n})^{2}.}$

According to the orthogonality principle, ^{[4] [5] [6] [7] [8] [9] [10] [11]} this is at its minimum when the residual vector (${\displaystyle z-{\hat {z}}}$) is orthogonal to the estimate ${\displaystyle {\hat {z}}}$, that is

${\displaystyle \sum _{n=1}^{N}(z_{n}-{\hat {z}}_{n}){\hat {z}}_{n}=0.}$

This can be described as the orthogonal projection of the data values {${\displaystyle z_{n}}$} onto a solution in the form of the polynomial ${\displaystyle {\hat {z}}(t)}$. ^{[4] [6] [7]} For N > J, orthogonal projection yields the standard overdetermined system of equations (often called normal equations) used to compute the coefficients in the polynomial approximation. ^{[1] [10] [11]} The minimum sum of squared residuals is then

${\displaystyle SSR_{\min }=\sum _{n=1}^{N}(z_{n}-{\hat {z}}_{n})z_{n}}$

The advantage of using orthogonal projection is that ${\displaystyle SSR_{\min }}$ can be determined for use in the polynomial least squares processed statistical variance of the estimate. ^{[8] [9] [11]}

The empirically determined polynomial least squares output of a first degree polynomial corrupted with observation errors

To fully determine the output of polynomial least squares, a weighting function describing the processing must first be structured and then the statistical moments can be computed.

The weighting function describing the linear polynomial least squares "system"

The weighting function ${\displaystyle w_{n}(\tau )}$ can be formulated from polynomial least squares to estimate the unknown ${\displaystyle y(t)}$ as follows: ^[8]

${\displaystyle {\hat {y}}(\tau )={\frac {1}{N}}\sum _{n=1}^{N}z_{n}w_{n}(\tau )={\frac {1}{N}}\sum _{n=1}^{N}(\alpha +\beta t_{n}+\varepsilon _{n})w_{n}(\tau )}$

where N is the number of samples, ${\displaystyle z_{n}}$ are random variables as samples of the stochastic ${\displaystyle z}$ (noisy signal), and the first degree polynomial data weights are

${\displaystyle w_{n}(\tau )\equiv {\frac {[{\bar {t^{2}}}-{\bar {t}}t_{n}+(t_{n}-{\bar {t}})\tau ]}{({\bar {t^{2}}}-{\bar {t}}^{2})}}}$

which represent the linear polynomial least squares "system" and describe its processing. ^[8] The Greek letter ${\displaystyle \tau }$ is the independent variable ${\displaystyle t}$ when estimating the dependent variable ${\displaystyle y(t)}$ after data fitting has been performed. (The letter ${\displaystyle \tau }$ is used to avoid confusion with ${\displaystyle t}$ before and sampling during polynomial least squares processing.) The overbar ( ¯ ) defines the deterministic centroid of ${\displaystyle u_{n}}$ as processed by polynomial least squares ^[8] – i.e., it defines the deterministic first order moment, which may be considered a sample average, but does not here approximate a first order statistical moment:

${\displaystyle {\bar {u}}{\overset {\underset {\mathrm {def} }{}}{=}}{\frac {1}{N}}\sum _{n=1}^{N}u_{n}}$

Empirically determined statistical moments

Applying ${\displaystyle w_{n}(\tau )}$ yields

${\displaystyle {\hat {y}}(\tau )={\hat {\alpha }}+{\hat {\beta }}\tau }$

where

${\displaystyle {\hat {\alpha }}={\frac {({\bar {z}}{\bar {t^{2}}}-{\bar {zt}}{\bar {t}})}{({\bar {t^{2}}}-{\bar {t}}^{2})}}=\alpha +{\frac {({\bar {\varepsilon }}{\bar {t^{2}}}-{\bar {{\varepsilon }t}}{\bar {t}})}{({\bar {t^{2}}}-{\bar {t}}^{2})}}}$

and

${\displaystyle {\hat {\beta }}={\frac {({\bar {zt}}-{\bar {z}}{\bar {t}})}{({\bar {t^{2}}}-{\bar {t}}^{2})}}=\beta +{\frac {({\bar {\varepsilon t}}-{\bar {\varepsilon }}{\bar {t}})}{({\bar {t^{2}}}-{\bar {t}}^{2})}}}$

As linear functions of the random variables ${\displaystyle \varepsilon _{n}}$, both coefficient estimates ${\displaystyle {\hat {\alpha }}}$ and ${\displaystyle {\hat {\beta }}}$ are random variables. ^[8] In the absence of the errors ${\displaystyle \varepsilon _{n}}$, ${\displaystyle {\hat {\alpha }}=\alpha }$ and ${\displaystyle {\hat {\beta }}=\beta }$, as they should to meet that boundary condition.

Because the statistical expectation operator E[•] is a linear function and the sampled stochastic process errors ${\displaystyle \varepsilon _{n}}$ are zero mean, the expected value of the estimate ${\displaystyle {\hat {y}}}$ is the first order statistical moment as follows: ^{[1] [2] [3] [8]}

${\displaystyle E[{\hat {y}}(\tau )]=\alpha +\beta \tau +{\frac {1}{N}}\sum _{n=1}^{N}E[\varepsilon _{n}]w_{n}(\tau )=\alpha +\beta \tau =\alpha +\beta t}$

The statistical variance in ${\displaystyle {\hat {y}}}$ is given by the second order statistical central moment as follows: ^{[1] [2] [3] [8]}

${\displaystyle \sigma _{\hat {y}}^{2}=E[\left({\hat {y}}-E[{\hat {y}}]\right)^{2}]={\frac {1}{N}}{\frac {1}{N}}\sum _{n=1}^{N}\sum _{i=1}^{N}w_{n}(\tau )E[\varepsilon _{n}\varepsilon _{i}]w_{i}(\tau )}$

${\displaystyle =\sigma _{\varepsilon }^{2}{\frac {1}{N}}{\frac {1}{N}}\sum _{n=1}^{N}\sum _{i=1}^{N}w_{n}^{2}(\tau )}$

because

${\displaystyle \sum _{i=1}^{N}E[\varepsilon _{n}\varepsilon _{i}]w_{i}(\tau )=\sigma _{\varepsilon }^{2}w_{n}(\tau )}$

where ${\displaystyle \sigma _{\varepsilon }^{2}}$ is the statistical variance of random variables ${\displaystyle \varepsilon _{n}}$; i.e., ${\displaystyle E[\varepsilon _{n}\varepsilon _{i}]=\sigma _{\varepsilon }^{2}}$ for i = n and (because ${\displaystyle \varepsilon _{n}}$ are uncorrelated) ${\displaystyle \sigma _{\varepsilon }^{2}=0}$ for ${\displaystyle i\neq n}$ ^[8]

Carrying out the multiplications and summations in ${\displaystyle \sigma _{\hat {y}}^{2}}$ yields ^[8]

${\displaystyle \sigma _{\hat {y}}^{2}=\sigma _{\varepsilon }^{2}{\frac {({\bar {t^{2}}}-2{\bar {t}}\tau +\tau ^{2})}{N({\bar {t^{2}}}-{\bar {t}}^{2})}}.}$

Measuring or approximating the statistical variance of the random errors

In a hardware system, such as a tracking radar, the measurement noise variance ${\displaystyle \sigma _{\varepsilon }^{2}}$ can be determined from measurements when there is no target return – i.e., by just taking measurements of the noise alone.

However, if polynomial least squares is used when the variance ${\displaystyle \sigma _{\varepsilon }^{2}}$ is not measurable (such as in econometrics or statistics), it can be estimated with observations in ${\displaystyle e_{\min }}$ from orthogonal projection as follows:

${\displaystyle \sigma _{\varepsilon }^{2}\approx {\hat {\sigma _{\varepsilon }^{2}}}=({\bar {z^{2}}}-{\hat {\alpha }}{\bar {z}}-{\hat {\beta }}{\bar {zt}})}$ ^[8]

As a result, to the first order approximation from the estimates ${\displaystyle {\hat {\alpha }}}$ and ${\displaystyle {\hat {\beta }}}$ as functions of sampled ${\displaystyle z}$ and ${\displaystyle t}$

${\displaystyle \sigma _{\hat {y}}^{2}\approx {\bigg [}{\frac {({\bar {z^{2}}}-{\bar {z}}^{2})}{({\bar {t^{2}}}-{\bar {t}}^{2})}}-{\Biggl (}{\frac {({\bar {zt}}-{\bar {z}}{\bar {t}})}{({\bar {t^{2}}}-{\bar {t}})}}{\Biggl )}^{2}{\bigg ]}{\frac {({\bar {t^{2}}}-2{\bar {t}}\tau +\tau ^{2})}{N}}}$

which goes to zero in the absence of the errors ${\displaystyle \varepsilon _{n}}$, as it should to meet that boundary condition. ^[8]

As a result, the samples ${\displaystyle z_{n}}$ (noisy signal) are considered to be the input to the linear polynomial least squares "system" which transforms the samples into the empirically determined statistical estimate ${\displaystyle {\hat {y}}(\tau )}$, the expected value ${\displaystyle E[{\hat {y}}]}$, and the variance ${\displaystyle \sigma _{\hat {y}}^{2}}$. ^[8]

Properties of polynomial least squares modeled as a linear "system"

(1) The empirical statistical variance ${\displaystyle \sigma _{\hat {y}}^{2}}$ is a function of ${\displaystyle \sigma _{\varepsilon }^{2}}$, N and ${\displaystyle \tau }$. Setting the derivative of ${\displaystyle \sigma _{\hat {y}}^{2}}$ with respect to ${\displaystyle \tau }$ equal to zero shows the minimum to occur at ${\displaystyle \tau ={\bar {t}}}$; i.e., at the centroid (sample average) of the samples ${\displaystyle t_{n}}$. The minimum statistical variance thus becomes ${\displaystyle {\frac {\sigma _{\varepsilon }^{2}}{N}}}$. This is equivalent to the statistical variance from polynomial least squares of a zero degree polynomial – i.e., of the centroid (sample average) of ${\displaystyle \alpha }$. ^{[1] [2] [8] [9]}

(2) The empirical statistical variance ${\displaystyle \sigma _{\hat {y}}^{2}}$ is a function of the quadratic ${\displaystyle \tau ^{2}}$ . Moreover, the further ${\displaystyle \tau }$ deviates from ${\displaystyle {\bar {t}}}$ (even within the data window), the larger is the variance ${\displaystyle \sigma _{\hat {y}}^{2}}$ due to the random variable errors ${\displaystyle \varepsilon _{n}}$ . The independent variable ${\displaystyle \tau }$ can take any value on the ${\displaystyle t}$ axis. It is not limited to the data window. It can extend beyond the data window – and likely will at times depending on the application. If it is within the data window, estimation is described as interpolation. If it is outside the data window, estimation is described as extrapolation. It is both intuitive and well known that the further is extrapolation, the larger is the error. ^[8]

(3) The empirical statistical variance ${\displaystyle \sigma _{\hat {y}}^{2}}$ due to the random variable errors ${\displaystyle \varepsilon _{n}}$ is inversely proportional to N. As N increases, the statistical variance decreases. This is well known and what filtering out the errors ${\displaystyle \varepsilon _{n}}$ is all about. ^{[1] [2] [8] [12]} The underlying purpose of polynomial least squares is to filter out the errors to improve estimation accuracy by reducing the empirical statistical estimation variance. In reality, only two data points are required to estimate ${\displaystyle \alpha }$ and ${\displaystyle \beta }$; albeit the more data points with zero mean statistical errors included, the smaller is the empirical statistical estimation variance as established by N samples.

(4) There is an additional issue to be considered when the noise variance is not measurable: Independent of the polynomial least squares estimation, any new observations would be described by the variance ${\displaystyle \sigma _{\varepsilon }^{2}\approx {\hat {\sigma _{\varepsilon }^{2}}}=({\bar {z^{2}}}-{\hat {\alpha }}{\bar {z}}-{\hat {\beta }}{\bar {zt}})}$. ^{[8] [9]}

Thus, the polynomial least squares statistical estimation variance ${\displaystyle \sigma _{\hat {y}}^{2}}$ and the statistical variance of any new sample in ${\displaystyle \sigma _{\varepsilon }^{2}}$ would both contribute to the uncertainty of any future observation. Both variances are clearly determined by polynomial least squares in advance.

(5) This concept also applies to higher degree polynomials. However, the weighting function ${\displaystyle w_{n}(\tau )}$ is obviously more complicated. In addition, the estimation variances increase exponentially as polynomial degrees increase linearly (i.e., in unit steps). However, there are ways of dealing with this as described in. ^{[6] [7]}

The synergy of integrating polynomial least squares with statistical estimation theory

Modeling polynomial least squares as a linear signal processing "system" creates the synergy of integrating polynomial least squares with statistical estimation theory to deterministically process samples of an assumed polynomial corrupted with a statistically described stochastic error ε. In the absence of the error ε, statistical estimation theory is irrelevant and polynomial least squares reverts to the conventional approximation of complicated functions and scatter plots.

See also

• Multi-fractional order estimator

References

1. Gujarati, Damodar N.; Porter, Dawn C. (2008). Basic Econometrics (PDF) (5 ed.). McGraw-Hill Education. ISBN   978-0073375779.
2. Hansen, Bruce E. (January 16, 2015). Econometrics (PDF).
3. Copeland, Thomas E.; Weston, John Fred; Shastri, Kuldeep (January 10, 2004). Financial Theory and Corporate Policy (4 ed.). Prentice Hall. ISBN   978-0321127211.
4. Kálmán, Rudolf E. (March 1, 1960). "A New Approach to Linear Filtering and Prediction Problems". Journal of Basic Engineering. 82: 35. doi:10.1115/1.3662552.
5. Sorenson, H. W., Least-squares estimation: Gauss to Kalman, IEEE Spectrum, July, 1970.
6. Bell, J. W., Simple Disambiguation Of Orthogonal Projection In Kalman’s Filter Derivation, Proceedings of the International Conference on Radar Systems, Glasgow, UK. October, 2012.
7. Bell, J. W., A Simple Kalman Filter Alternative: The Multi-Fractional Order Estimator, IET-RSN, Vol. 7, Issue 8, October 2013.
8. "Ordinary Least Squares Revolutionized: Establishing the Vital Missing Empirically Determined Statistical Prediction Variance by Jeff Bell". SSRN. doi:10.2139/ssrn.2573840 . Retrieved 2019-02-27.
9. Papoulis, A., Probability, RVs, and Stochastic Processes, McGraw-Hill, New York, 1965
10. Wylie, C. R., Jr., Advanced Engineering Mathematics, McGraw-Hill, New York, 1960.
11. Schied, F., Numerical Analysis, Schaum's Outline Series, McGraw-Hill, New York, 1968.
12. Ordinary least squares