# Kernel (statistics)

Last updated

The term kernel is used in statistical analysis to refer to a window function. The term "kernel" has several distinct meanings in different branches of statistics.

## Bayesian statistics

In statistics, especially in Bayesian statistics, the kernel of a probability density function (pdf) or probability mass function (pmf) is the form of the pdf or pmf in which any factors that are not functions of any of the variables in the domain are omitted.[ citation needed ] Note that such factors may well be functions of the parameters of the pdf or pmf. These factors form part of the normalization factor of the probability distribution, and are unnecessary in many situations. For example, in pseudo-random number sampling, most sampling algorithms ignore the normalization factor. In addition, in Bayesian analysis of conjugate prior distributions, the normalization factors are generally ignored during the calculations, and only the kernel considered. At the end, the form of the kernel is examined, and if it matches a known distribution, the normalization factor can be reinstated. Otherwise, it may be unnecessary (for example, if the distribution only needs to be sampled from).

For many distributions, the kernel can be written in closed form, but not the normalization constant.

An example is the normal distribution. Its probability density function is

${\displaystyle p(x|\mu ,\sigma ^{2})={\frac {1}{\sqrt {2\pi \sigma ^{2}}}}e^{-{\frac {(x-\mu )^{2}}{2\sigma ^{2}}}}}$

and the associated kernel is

${\displaystyle p(x|\mu ,\sigma ^{2})\propto e^{-{\frac {(x-\mu )^{2}}{2\sigma ^{2}}}}}$

Note that the factor in front of the exponential has been omitted, even though it contains the parameter ${\displaystyle \sigma ^{2}}$ , because it is not a function of the domain variable ${\displaystyle x}$ .

## Pattern analysis

The kernel of a reproducing kernel Hilbert space is used in the suite of techniques known as kernel methods to perform tasks such as statistical classification, regression analysis, and cluster analysis on data in an implicit space. This usage is particularly common in machine learning.

## Nonparametric statistics

In nonparametric statistics, a kernel is a weighting function used in non-parametric estimation techniques. Kernels are used in kernel density estimation to estimate random variables' density functions, or in kernel regression to estimate the conditional expectation of a random variable. Kernels are also used in time-series, in the use of the periodogram to estimate the spectral density where they are known as window functions. An additional use is in the estimation of a time-varying intensity for a point process where window functions (kernels) are convolved with time-series data.

Commonly, kernel widths must also be specified when running a non-parametric estimation.

### Definition

A kernel is a non-negative real-valued integrable function K. For most applications, it is desirable to define the function to satisfy two additional requirements:

${\displaystyle \int _{-\infty }^{+\infty }K(u)\,du=1\,;}$
• Symmetry:
${\displaystyle K(-u)=K(u){\mbox{ for all values of }}u\,.}$

The first requirement ensures that the method of kernel density estimation results in a probability density function. The second requirement ensures that the average of the corresponding distribution is equal to that of the sample used.

If K is a kernel, then so is the function K* defined by K*(u) = λKu), where λ > 0. This can be used to select a scale that is appropriate for the data.

### Kernel functions in common use

Several types of kernel functions are commonly used: uniform, triangle, Epanechnikov, [1] quartic (biweight), tricube, [2] triweight, Gaussian, quadratic [3] and cosine.

In the table below, if ${\displaystyle K}$ is given with a bounded support, then ${\displaystyle K(u)=0}$ for values of u lying outside the support.

Kernel Functions, K(u)${\displaystyle \textstyle \int u^{2}K(u)du}$${\displaystyle \textstyle \int K(u)^{2}du}$Efficiency [4] relative to the Epanechnikov kernel
Uniform ("rectangular window")${\displaystyle K(u)={\frac {1}{2}}}$

Support: ${\displaystyle |u|\leq 1}$

${\displaystyle {\frac {1}{3}}}$ ${\displaystyle {\frac {1}{2}}}$92.9%
Triangular${\displaystyle K(u)=(1-|u|)}$

Support: ${\displaystyle |u|\leq 1}$

${\displaystyle {\frac {1}{6}}}$ ${\displaystyle {\frac {2}{3}}}$98.6%
Epanechnikov

(parabolic)

${\displaystyle K(u)={\frac {3}{4}}(1-u^{2})}$

Support: ${\displaystyle |u|\leq 1}$

${\displaystyle {\frac {1}{5}}}$ ${\displaystyle {\frac {3}{5}}}$100%
Quartic
(biweight)
${\displaystyle K(u)={\frac {15}{16}}(1-u^{2})^{2}}$

Support: ${\displaystyle |u|\leq 1}$

${\displaystyle {\frac {1}{7}}}$ ${\displaystyle {\frac {5}{7}}}$99.4%
Triweight${\displaystyle K(u)={\frac {35}{32}}(1-u^{2})^{3}}$

Support: ${\displaystyle |u|\leq 1}$

${\displaystyle {\frac {1}{9}}}$ ${\displaystyle {\frac {350}{429}}}$98.7%
Tricube${\displaystyle K(u)={\frac {70}{81}}(1-{\left|u\right|}^{3})^{3}}$

Support: ${\displaystyle |u|\leq 1}$

${\displaystyle {\frac {35}{243}}}$ ${\displaystyle {\frac {175}{247}}}$99.8%
Gaussian ${\displaystyle K(u)={\frac {1}{\sqrt {2\pi }}}e^{-{\frac {1}{2}}u^{2}}}$  ${\displaystyle 1\,}$ ${\displaystyle {\frac {1}{2{\sqrt {\pi }}}}}$95.1%
Cosine${\displaystyle K(u)={\frac {\pi }{4}}\cos \left({\frac {\pi }{2}}u\right)}$

Support: ${\displaystyle |u|\leq 1}$

${\displaystyle 1-{\frac {8}{\pi ^{2}}}}$ ${\displaystyle {\frac {\pi ^{2}}{16}}}$99.9%
Logistic ${\displaystyle K(u)={\frac {1}{e^{u}+2+e^{-u}}}}$  ${\displaystyle {\frac {\pi ^{2}}{3}}}$ ${\displaystyle {\frac {1}{6}}}$88.7%
Sigmoid function ${\displaystyle K(u)={\frac {2}{\pi }}{\frac {1}{e^{u}+e^{-u}}}}$  ${\displaystyle {\frac {\pi ^{2}}{4}}}$ ${\displaystyle {\frac {2}{\pi ^{2}}}}$84.3%
Silverman kernel [5] ${\displaystyle K(u)={\frac {1}{2}}e^{-{\frac {|u|}{\sqrt {2}}}}\cdot \sin \left({\frac {|u|}{\sqrt {2}}}+{\frac {\pi }{4}}\right)}$  ${\displaystyle 0}$ ${\displaystyle {\frac {3{\sqrt {2}}}{16}}}$not applicable

## Related Research Articles

In probability theory, a normaldistribution is a type of continuous probability distribution for a real-valued random variable. The general form of its probability density function is

In probability theory, the central limit theorem (CLT) establishes that, in many situations, when independent random variables are summed up, their properly normalized sum tends toward a normal distribution even if the original variables themselves are not normally distributed. The theorem is a key concept in probability theory because it implies that probabilistic and statistical methods that work for normal distributions can be applicable to many problems involving other types of distributions. This theorem has seen many changes during the formal development of probability theory. Previous versions of the theorem date back to 1811, but in its modern general form, this fundamental result in probability theory was precisely stated as late as 1920, thereby serving as a bridge between classical and modern probability theory.

In probability theory and statistics, the multivariate normal distribution, multivariate Gaussian distribution, or joint normal distribution is a generalization of the one-dimensional (univariate) normal distribution to higher dimensions. One definition is that a random vector is said to be k-variate normally distributed if every linear combination of its k components has a univariate normal distribution. Its importance derives mainly from the multivariate central limit theorem. The multivariate normal distribution is often used to describe, at least approximately, any set of (possibly) correlated real-valued random variables each of which clusters around a mean value.

In probability and statistics, Student's t-distribution is any member of a family of continuous probability distributions that arise when estimating the mean of a normally distributed population in situations where the sample size is small and the population's standard deviation is unknown. It was developed by English statistician William Sealy Gosset under the pseudonym "Student".

In statistics, an effect size is a number measuring the strength of the relationship between two variables in a population, or a sample-based estimate of that quantity. It can refer to the value of a statistic calculated from a sample of data, the value of a parameter for a hypothetical population, or to the equation that operationalizes how statistics or parameters lead to the effect size value. Examples of effect sizes include the correlation between two variables, the regression coefficient in a regression, the mean difference, or the risk of a particular event happening. Effect sizes complement statistical hypothesis testing, and play an important role in power analyses, sample size planning, and in meta-analyses. The cluster of data-analysis methods concerning effect sizes is referred to as estimation statistics.

In statistical inference, specifically predictive inference, a prediction interval is an estimate of an interval in which a future observation will fall, with a certain probability, given what has already been observed. Prediction intervals are often used in regression analysis.

Linear discriminant analysis (LDA), normal discriminant analysis (NDA), or discriminant function analysis is a generalization of Fisher's linear discriminant, a method used in statistics and other fields, to find a linear combination of features that characterizes or separates two or more classes of objects or events. The resulting combination may be used as a linear classifier, or, more commonly, for dimensionality reduction before later classification.

In statistics, kernel density estimation (KDE) is a non-parametric way to estimate the probability density function of a random variable. Kernel density estimation is a fundamental data smoothing problem where inferences about the population are made, based on a finite data sample. In some fields such as signal processing and econometrics it is also termed the Parzen–Rosenblatt window method, after Emanuel Parzen and Murray Rosenblatt, who are usually credited with independently creating it in its current form. One of the famous applications of kernel density estimation is in estimating the class-conditional marginal densities of data when using a naive Bayes classifier, which can improve its prediction accuracy.

The following is a glossary of terms used in the mathematical sciences statistics and probability.

In statistics, a parametric model or parametric family or finite-dimensional model is a particular class of statistical models. Specifically, a parametric model is a family of probability distributions that has a finite number of parameters.

In probability theory, the inverse Gaussian distribution is a two-parameter family of continuous probability distributions with support on (0,∞).

Bootstrapping is any test or metric that uses random sampling with replacement, and falls under the broader class of resampling methods. Bootstrapping assigns measures of accuracy to sample estimates. This technique allows estimation of the sampling distribution of almost any statistic using random sampling methods.

In probability theory, Dirichlet processes are a family of stochastic processes whose realizations are probability distributions. In other words, a Dirichlet process is a probability distribution whose range is itself a set of probability distributions. It is often used in Bayesian inference to describe the prior knowledge about the distribution of random variables—how likely it is that the random variables are distributed according to one or another particular distribution.

In probability and statistics, the truncated normal distribution is the probability distribution derived from that of a normally distributed random variable by bounding the random variable from either below or above. The truncated normal distribution has wide applications in statistics and econometrics. For example, it is used to model the probabilities of the binary outcomes in the probit model and to model censored data in the tobit model.

Exact statistics, such as that described in exact test, is a branch of statistics that was developed to provide more accurate results pertaining to statistical testing and interval estimation by eliminating procedures based on asymptotic and approximate statistical methods. The main characteristic of exact methods is that statistical tests and confidence intervals are based on exact probability statements that are valid for any sample size.

In statistics, identifiability is a property which a model must satisfy for precise inference to be possible. A model is identifiable if it is theoretically possible to learn the true values of this model's underlying parameters after obtaining an infinite number of observations from it. Mathematically, this is equivalent to saying that different values of the parameters must generate different probability distributions of the observable variables. Usually the model is identifiable only under certain technical restrictions, in which case the set of these requirements is called the identification conditions.

In probability theory, the Mills ratio of a continuous random variable is the function

In probability theory, a log-Cauchy distribution is a probability distribution of a random variable whose logarithm is distributed in accordance with a Cauchy distribution. If X is a random variable with a Cauchy distribution, then Y = exp(X) has a log-Cauchy distribution; likewise, if Y has a log-Cauchy distribution, then X = log(Y) has a Cauchy distribution.

In machine learning, the kernel embedding of distributions comprises a class of nonparametric methods in which a probability distribution is represented as an element of a reproducing kernel Hilbert space (RKHS). A generalization of the individual data-point feature mapping done in classical kernel methods, the embedding of distributions into infinite-dimensional feature spaces can preserve all of the statistical features of arbitrary distributions, while allowing one to compare and manipulate distributions using Hilbert space operations such as inner products, distances, projections, linear transformations, and spectral analysis. This learning framework is very general and can be applied to distributions over any space on which a sensible kernel function may be defined. For example, various kernels have been proposed for learning from data which are: vectors in , discrete classes/categories, strings, graphs/networks, images, time series, manifolds, dynamical systems, and other structured objects. The theory behind kernel embeddings of distributions has been primarily developed by Alex Smola, Le Song , Arthur Gretton, and Bernhard Schölkopf. A review of recent works on kernel embedding of distributions can be found in.

In statistics, the variance function is a smooth function which depicts the variance of a random quantity as a function of its mean. The variance function is a measure of heteroscedasticity and plays a large role in many settings of statistical modelling. It is a main ingredient in the generalized linear model framework and a tool used in non-parametric regression, semiparametric regression and functional data analysis. In parametric modeling, variance functions take on a parametric form and explicitly describe the relationship between the variance and the mean of a random quantity. In a non-parametric setting, the variance function is assumed to be a smooth function.

## References

1. Named for Epanechnikov, V. A. (1969). "Non-Parametric Estimation of a Multivariate Probability Density". Theory Probab. Appl. 14 (1): 153–158. doi:10.1137/1114019.
2. Altman, N. S. (1992). "An introduction to kernel and nearest neighbor nonparametric regression". The American Statistician. 46 (3): 175–185. doi:10.1080/00031305.1992.10475879. hdl:.
3. Cleveland, W. S.; Devlin, S. J. (1988). "Locally weighted regression: An approach to regression analysis by local fitting". Journal of the American Statistical Association. 83 (403): 596–610. doi:10.1080/01621459.1988.10478639.
4. Efficiency is defined as ${\displaystyle {\sqrt {\int u^{2}K(u)\,du}}\int K(u)^{2}\,du}$.
5. Silverman, B. W. (1986). Density Estimation for Statistics and Data Analysis. Chapman and Hall, London.
• Li, Qi; Racine, Jeffrey S. (2007). Nonparametric Econometrics: Theory and Practice. Princeton University Press. ISBN   978-0-691-12161-1.
• Comaniciu, D; Meer, P (2002). "Mean shift: A robust approach toward feature space analysis". IEEE Transactions on Pattern Analysis and Machine Intelligence. 24 (5): 603–619. CiteSeerX  . doi:10.1109/34.1000236.