Kernel embedding of distributions

Last updated

In machine learning, the kernel embedding of distributions (also called the kernel mean or mean map) comprises a class of nonparametric methods in which a probability distribution is represented as an element of a reproducing kernel Hilbert space (RKHS). [1] A generalization of the individual data-point feature mapping done in classical kernel methods, the embedding of distributions into infinite-dimensional feature spaces can preserve all of the statistical features of arbitrary distributions, while allowing one to compare and manipulate distributions using Hilbert space operations such as inner products, distances, projections, linear transformations, and spectral analysis. [2] This learning framework is very general and can be applied to distributions over any space on which a sensible kernel function (measuring similarity between elements of ) may be defined. For example, various kernels have been proposed for learning from data which are: vectors in , discrete classes/categories, strings, graphs/networks, images, time series, manifolds, dynamical systems, and other structured objects. [3] [4] The theory behind kernel embeddings of distributions has been primarily developed by Alex Smola, Le Song , Arthur Gretton, and Bernhard Schölkopf. A review of recent works on kernel embedding of distributions can be found in. [5]

Contents

The analysis of distributions is fundamental in machine learning and statistics, and many algorithms in these fields rely on information theoretic approaches such as entropy, mutual information, or Kullback–Leibler divergence. However, to estimate these quantities, one must first either perform density estimation, or employ sophisticated space-partitioning/bias-correction strategies which are typically infeasible for high-dimensional data. [6] Commonly, methods for modeling complex distributions rely on parametric assumptions that may be unfounded or computationally challenging (e.g. Gaussian mixture models), while nonparametric methods like kernel density estimation (Note: the smoothing kernels in this context have a different interpretation than the kernels discussed here) or characteristic function representation (via the Fourier transform of the distribution) break down in high-dimensional settings. [2]

Methods based on the kernel embedding of distributions sidestep these problems and also possess the following advantages: [6]

  1. Data may be modeled without restrictive assumptions about the form of the distributions and relationships between variables
  2. Intermediate density estimation is not needed
  3. Practitioners may specify the properties of a distribution most relevant for their problem (incorporating prior knowledge via choice of the kernel)
  4. If a characteristic kernel is used, then the embedding can uniquely preserve all information about a distribution, while thanks to the kernel trick, computations on the potentially infinite-dimensional RKHS can be implemented in practice as simple Gram matrix operations
  5. Dimensionality-independent rates of convergence for the empirical kernel mean (estimated using samples from the distribution) to the kernel embedding of the true underlying distribution can be proven.
  6. Learning algorithms based on this framework exhibit good generalization ability and finite sample convergence, while often being simpler and more effective than information theoretic methods

Thus, learning via the kernel embedding of distributions offers a principled drop-in replacement for information theoretic approaches and is a framework which not only subsumes many popular methods in machine learning and statistics as special cases, but also can lead to entirely new learning algorithms.

Definitions

Let denote a random variable with domain and distribution . Given a symmetric, positive-definite kernel the Moore–Aronszajn theorem asserts the existence of a unique RKHS on (a Hilbert space of functions equipped with an inner product and a norm ) for which is a reproducing kernel, i.e., in which the element satisfies the reproducing property

One may alternatively consider an implicit feature mapping from to (which is therefore also called the feature space), so that can be viewed as a measure of similarity between points While the similarity measure is linear in the feature space, it may be highly nonlinear in the original space depending on the choice of kernel.

Kernel embedding

The kernel embedding of the distribution in (also called the kernel mean or mean map) is given by: [1]

If allows a square integrable density , then , where is the Hilbert–Schmidt integral operator. A kernel is characteristic if the mean embedding is injective. [7] Each distribution can thus be uniquely represented in the RKHS and all statistical features of distributions are preserved by the kernel embedding if a characteristic kernel is used.

Empirical kernel embedding

Given training examples drawn independently and identically distributed (i.i.d.) from the kernel embedding of can be empirically estimated as

Joint distribution embedding

If denotes another random variable (for simplicity, assume the co-domain of is also with the same kernel which satisfies ), then the joint distribution can be mapped into a tensor product feature space via [2]

By the equivalence between a tensor and a linear map, this joint embedding may be interpreted as an uncentered cross-covariance operator from which the cross-covariance of functions can be computed as [8]

Given pairs of training examples drawn i.i.d. from , we can also empirically estimate the joint distribution kernel embedding via

Conditional distribution embedding

Given a conditional distribution one can define the corresponding RKHS embedding as [2]

Note that the embedding of thus defines a family of points in the RKHS indexed by the values taken by conditioning variable . By fixing to a particular value, we obtain a single element in , and thus it is natural to define the operator

which given the feature mapping of outputs the conditional embedding of given Assuming that for all it can be shown that [8]

This assumption is always true for finite domains with characteristic kernels, but may not necessarily hold for continuous domains. [2] Nevertheless, even in cases where the assumption fails, may still be used to approximate the conditional kernel embedding and in practice, the inversion operator is replaced with a regularized version of itself (where denotes the identity matrix).

Given training examples the empirical kernel conditional embedding operator may be estimated as [2]

where are implicitly formed feature matrices, is the Gram matrix for samples of , and is a regularization parameter needed to avoid overfitting.

Thus, the empirical estimate of the kernel conditional embedding is given by a weighted sum of samples of in the feature space:

where and

Properties

Convergence of empirical kernel mean to the true distribution embedding

where denotes the unit ball in and is the Gram matrix with

Universal kernels

For to be universal it suffices that the continuous part of in its unique Lebesgue decomposition is non-zero. Furthermore, if
then is the spectral density of frequencies in and is the Fourier transform of . If the support of is all of , then is a characteristic kernel as well. [11] [12] [13]
on compact subsets of is universal.

Parameter selection for conditional distribution kernel embeddings

where is the Hilbert–Schmidt norm.

Rules of probability as operations in the RKHS

This section illustrates how basic probabilistic rules may be reformulated as (multi)linear algebraic operations in the kernel embedding framework and is primarily based on the work of Song et al. [2] [8] The following notation is adopted:

In practice, all embeddings are empirically estimated from data and it assumed that a set of samples may be used to estimate the kernel embedding of the prior distribution .

Kernel sum rule

In probability theory, the marginal distribution of can be computed by integrating out from the joint density (including the prior distribution on )

The analog of this rule in the kernel embedding framework states that the RKHS embedding of , can be computed via

where is the kernel embedding of In practical implementations, the kernel sum rule takes the following form

where

is the empirical kernel embedding of the prior distribution, , and are Gram matrices with entries respectively.

Kernel chain rule

In probability theory, a joint distribution can be factorized into a product between conditional and marginal distributions

The analog of this rule in the kernel embedding framework states that the joint embedding of can be factorized as a composition of conditional embedding operator with the auto-covariance operator associated with

where

In practical implementations, the kernel chain rule takes the following form

Kernel Bayes' rule

In probability theory, a posterior distribution can be expressed in terms of a prior distribution and a likelihood function as

where

The analog of this rule in the kernel embedding framework expresses the kernel embedding of the conditional distribution in terms of conditional embedding operators which are modified by the prior distribution

where from the chain rule:

In practical implementations, the kernel Bayes' rule takes the following form

where

Two regularization parameters are used in this framework: for the estimation of and for the estimation of the final conditional embedding operator

The latter regularization is done on square of because may not be positive definite.

Applications

Measuring distance between distributions

The maximum mean discrepancy (MMD) is a distance-measure between distributions and which is defined as the distance between their embeddings in the RKHS [6]

While most distance-measures between distributions such as the widely used Kullback–Leibler divergence either require density estimation (either parametrically or nonparametrically) or space partitioning/bias correction strategies, [6] the MMD is easily estimated as an empirical mean which is concentrated around the true value of the MMD. The characterization of this distance as the maximum mean discrepancy refers to the fact that computing the MMD is equivalent to finding the RKHS function that maximizes the difference in expectations between the two probability distributions

a form of integral probability metric.

Kernel two-sample test

Given n training examples from and m samples from , one can formulate a test statistic based on the empirical estimate of the MMD

to obtain a two-sample test [15] of the null hypothesis that both samples stem from the same distribution (i.e. ) against the broad alternative .

Density estimation via kernel embeddings

Although learning algorithms in the kernel embedding framework circumvent the need for intermediate density estimation, one may nonetheless use the empirical embedding to perform density estimation based on n samples drawn from an underlying distribution . This can be done by solving the following optimization problem [6] [16]

subject to

where the maximization is done over the entire space of distributions on Here, is the kernel embedding of the proposed density and is an entropy-like quantity (e.g. Entropy, KL divergence, Bregman divergence). The distribution which solves this optimization may be interpreted as a compromise between fitting the empirical kernel means of the samples well, while still allocating a substantial portion of the probability mass to all regions of the probability space (much of which may not be represented in the training examples). In practice, a good approximate solution of the difficult optimization may be found by restricting the space of candidate densities to a mixture of M candidate distributions with regularized mixing proportions. Connections between the ideas underlying Gaussian processes and conditional random fields may be drawn with the estimation of conditional probability distributions in this fashion, if one views the feature mappings associated with the kernel as sufficient statistics in generalized (possibly infinite-dimensional) exponential families. [6]

Measuring dependence of random variables

A measure of the statistical dependence between random variables and (from any domains on which sensible kernels can be defined) can be formulated based on the Hilbert–Schmidt Independence Criterion [17]

and can be used as a principled replacement for mutual information, Pearson correlation or any other dependence measure used in learning algorithms. Most notably, HSIC can detect arbitrary dependencies (when a characteristic kernel is used in the embeddings, HSIC is zero if and only if the variables are independent), and can be used to measure dependence between different types of data (e.g. images and text captions). Given n i.i.d. samples of each random variable, a simple parameter-free unbiased estimator of HSIC which exhibits concentration about the true value can be computed in time, [6] where the Gram matrices of the two datasets are approximated using with . The desirable properties of HSIC have led to the formulation of numerous algorithms which utilize this dependence measure for a variety of common machine learning tasks such as: feature selection (BAHSIC [18] ), clustering (CLUHSIC [19] ), and dimensionality reduction (MUHSIC [20] ).

HSIC can be extended to measure the dependence of multiple random variables. The question of when HSIC captures independence in this case has recently been studied: [21] for more than two variables

Kernel belief propagation

Belief propagation is a fundamental algorithm for inference in graphical models in which nodes repeatedly pass and receive messages corresponding to the evaluation of conditional expectations. In the kernel embedding framework, the messages may be represented as RKHS functions and the conditional distribution embeddings can be applied to efficiently compute message updates. Given n samples of random variables represented by nodes in a Markov random field, the incoming message to node t from node u can be expressed as

if it assumed to lie in the RKHS. The kernel belief propagation update message from t to node s is then given by [2]

where denotes the element-wise vector product, is the set of nodes connected to t excluding node s, , are the Gram matrices of the samples from variables , respectively, and is the feature matrix for the samples from .

Thus, if the incoming messages to node t are linear combinations of feature mapped samples from , then the outgoing message from this node is also a linear combination of feature mapped samples from . This RKHS function representation of message-passing updates therefore produces an efficient belief propagation algorithm in which the potentials are nonparametric functions inferred from the data so that arbitrary statistical relationships may be modeled. [2]

Nonparametric filtering in hidden Markov models

In the hidden Markov model (HMM), two key quantities of interest are the transition probabilities between hidden states and the emission probabilities for observations. Using the kernel conditional distribution embedding framework, these quantities may be expressed in terms of samples from the HMM. A serious limitation of the embedding methods in this domain is the need for training samples containing hidden states, as otherwise inference with arbitrary distributions in the HMM is not possible.

One common use of HMMs is filtering in which the goal is to estimate posterior distribution over the hidden state at time step t given a history of previous observations from the system. In filtering, a belief state is recursively maintained via a prediction step (where updates are computed by marginalizing out the previous hidden state) followed by a conditioning step (where updates are computed by applying Bayes' rule to condition on a new observation). [2] The RKHS embedding of the belief state at time t+1 can be recursively expressed as

by computing the embeddings of the prediction step via the kernel sum rule and the embedding of the conditioning step via kernel Bayes' rule. Assuming a training sample is given, one can in practice estimate

and filtering with kernel embeddings is thus implemented recursively using the following updates for the weights [2]

where denote the Gram matrices of and respectively, is a transfer Gram matrix defined as and

Support measure machines

The support measure machine (SMM) is a generalization of the support vector machine (SVM) in which the training examples are probability distributions paired with labels . [22] SMMs solve the standard SVM dual optimization problem using the following expected kernel

which is computable in closed form for many common specific distributions (such as the Gaussian distribution) combined with popular embedding kernels (e.g. the Gaussian kernel or polynomial kernel), or can be accurately empirically estimated from i.i.d. samples via

Under certain choices of the embedding kernel , the SMM applied to training examples is equivalent to a SVM trained on samples , and thus the SMM can be viewed as a flexible SVM in which a different data-dependent kernel (specified by the assumed form of the distribution ) may be placed on each training point. [22]

Domain adaptation under covariate, target, and conditional shift

The goal of domain adaptation is the formulation of learning algorithms which generalize well when the training and test data have different distributions. Given training examples and a test set where the are unknown, three types of differences are commonly assumed between the distribution of the training examples and the test distribution : [23] [24]

  1. Covariate shift in which the marginal distribution of the covariates changes across domains:
  2. Target shift in which the marginal distribution of the outputs changes across domains:
  3. Conditional shift in which remains the same across domains, but the conditional distributions differ: . In general, the presence of conditional shift leads to an ill-posed problem, and the additional assumption that changes only under location-scale (LS) transformations on is commonly imposed to make the problem tractable.

By utilizing the kernel embedding of marginal and conditional distributions, practical approaches to deal with the presence of these types of differences between training and test domains can be formulated. Covariate shift may be accounted for by reweighting examples via estimates of the ratio obtained directly from the kernel embeddings of the marginal distributions of in each domain without any need for explicit estimation of the distributions. [24] Target shift, which cannot be similarly dealt with since no samples from are available in the test domain, is accounted for by weighting training examples using the vector which solves the following optimization problem (where in practice, empirical approximations must be used) [23]

subject to

To deal with location scale conditional shift, one can perform a LS transformation of the training points to obtain new transformed training data (where denotes the element-wise vector product). To ensure similar distributions between the new transformed training samples and the test data, are estimated by minimizing the following empirical kernel embedding distance [23]

In general, the kernel embedding methods for dealing with LS conditional shift and target shift may be combined to find a reweighted transformation of the training data which mimics the test distribution, and these methods may perform well even in the presence of conditional shifts other than location-scale changes. [23]

Domain generalization via invariant feature representation

Given N sets of training examples sampled i.i.d. from distributions , the goal of domain generalization is to formulate learning algorithms which perform well on test examples sampled from a previously unseen domain where no data from the test domain is available at training time. If conditional distributions are assumed to be relatively similar across all domains, then a learner capable of domain generalization must estimate a functional relationship between the variables which is robust to changes in the marginals . Based on kernel embeddings of these distributions, Domain Invariant Component Analysis (DICA) is a method which determines the transformation of the training data that minimizes the difference between marginal distributions while preserving a common conditional distribution shared between all training domains. [25] DICA thus extracts invariants, features that transfer across domains, and may be viewed as a generalization of many popular dimension-reduction methods such as kernel principal component analysis, transfer component analysis, and covariance operator inverse regression. [25]

Defining a probability distribution on the RKHS with

DICA measures dissimilarity between domains via distributional variance which is computed as

where

so is a Gram matrix over the distributions from which the training data are sampled. Finding an orthogonal transform onto a low-dimensional subspace B (in the feature space) which minimizes the distributional variance, DICA simultaneously ensures that B aligns with the bases of a central subspaceC for which becomes independent of given across all domains. In the absence of target values , an unsupervised version of DICA may be formulated which finds a low-dimensional subspace that minimizes distributional variance while simultaneously maximizing the variance of (in the feature space) across all domains (rather than preserving a central subspace). [25]

Distribution regression

In distribution regression, the goal is to regress from probability distributions to reals (or vectors). Many important machine learning and statistical tasks fit into this framework, including multi-instance learning, and point estimation problems without analytical solution (such as hyperparameter or entropy estimation). In practice only samples from sampled distributions are observable, and the estimates have to rely on similarities computed between sets of points. Distribution regression has been successfully applied for example in supervised entropy learning, and aerosol prediction using multispectral satellite images. [26]

Given training data, where the bag contains samples from a probability distribution and the output label is , one can tackle the distribution regression task by taking the embeddings of the distributions, and learning the regressor from the embeddings to the outputs. In other words, one can consider the following kernel ridge regression problem

where

with a kernel on the domain of -s , is a kernel on the embedded distributions, and is the RKHS determined by . Examples for include the linear kernel , the Gaussian kernel , the exponential kernel , the Cauchy kernel , the generalized t-student kernel , or the inverse multiquadrics kernel .

The prediction on a new distribution takes the simple, analytical form

where , , , . Under mild regularity conditions this estimator can be shown to be consistent and it can achieve the one-stage sampled (as if one had access to the true -s) minimax optimal rate. [26] In the objective function -s are real numbers; the results can also be extended to the case when -s are -dimensional vectors, or more generally elements of a separable Hilbert space using operator-valued kernels.

Example

In this simple example, which is taken from Song et al., [2] are assumed to be discrete random variables which take values in the set and the kernel is chosen to be the Kronecker delta function, so . The feature map corresponding to this kernel is the standard basis vector . The kernel embeddings of such a distributions are thus vectors of marginal probabilities while the embeddings of joint distributions in this setting are matrices specifying joint probability tables, and the explicit form of these embeddings is

The conditional distribution embedding operator,

is in this setting a conditional probability table

and

Thus, the embeddings of the conditional distribution under a fixed value of may be computed as

In this discrete-valued setting with the Kronecker delta kernel, the kernel sum rule becomes

The kernel chain rule in this case is given by

Related Research Articles

<span class="mw-page-title-main">Normal distribution</span> Probability distribution

In statistics, a normal distribution or Gaussian distribution is a type of continuous probability distribution for a real-valued random variable. The general form of its probability density function is

<span class="mw-page-title-main">Dirac delta function</span> Generalized function whose value is zero everywhere except at zero

In mathematical analysis, the Dirac delta function, also known as the unit impulse, is a generalized function on the real numbers, whose value is zero everywhere except at zero, and whose integral over the entire real line is equal to one. Since there is no function having this property, to model the delta "function" rigorously involves the use of limits or, as is common in mathematics, measure theory and the theory of distributions.

In probability theory, the central limit theorem (CLT) states that, under appropriate conditions, the distribution of a normalized version of the sample mean converges to a standard normal distribution. This holds even if the original variables themselves are not normally distributed. There are several versions of the CLT, each applying in the context of different conditions.

<span class="mw-page-title-main">Navier–Stokes equations</span> Equations describing the motion of viscous fluid substances

The Navier–Stokes equations are partial differential equations which describe the motion of viscous fluid substances. They were named after French engineer and physicist Claude-Louis Navier and the Irish physicist and mathematician George Gabriel Stokes. They were developed over several decades of progressively building the theories, from 1822 (Navier) to 1842–1850 (Stokes).

<span class="mw-page-title-main">Multivariate normal distribution</span> Generalization of the one-dimensional normal distribution to higher dimensions

In probability theory and statistics, the multivariate normal distribution, multivariate Gaussian distribution, or joint normal distribution is a generalization of the one-dimensional (univariate) normal distribution to higher dimensions. One definition is that a random vector is said to be k-variate normally distributed if every linear combination of its k components has a univariate normal distribution. Its importance derives mainly from the multivariate central limit theorem. The multivariate normal distribution is often used to describe, at least approximately, any set of (possibly) correlated real-valued random variables each of which clusters around a mean value.

<span class="mw-page-title-main">Fourier transform</span> Mathematical transform that expresses a function of time as a function of frequency

In physics, engineering and mathematics, the Fourier transform (FT) is an integral transform that converts a function into a form that describes the frequencies present in the original function. The output of the transform is a complex-valued function of frequency. The term Fourier transform refers to both this complex-valued function and the mathematical operation. When a distinction needs to be made the Fourier transform is sometimes called the frequency domain representation of the original function. The Fourier transform is analogous to decomposing the sound of a musical chord into the intensities of its constituent pitches.

<span class="mw-page-title-main">Log-normal distribution</span> Probability distribution

In probability theory, a log-normal (or lognormal) distribution is a continuous probability distribution of a random variable whose logarithm is normally distributed. Thus, if the random variable X is log-normally distributed, then Y = ln(X) has a normal distribution. Equivalently, if Y has a normal distribution, then the exponential function of Y, X = exp(Y), has a log-normal distribution. A random variable which is log-normally distributed takes only positive real values. It is a convenient and useful model for measurements in exact and engineering sciences, as well as medicine, economics and other topics (e.g., energies, concentrations, lengths, prices of financial instruments, and other metrics).

In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statistical model, the observed data is most probable. The point in the parameter space that maximizes the likelihood function is called the maximum likelihood estimate. The logic of maximum likelihood is both intuitive and flexible, and as such the method has become a dominant means of statistical inference.

<span class="mw-page-title-main">Noether's theorem</span> Statement relating differentiable symmetries to conserved quantities

Noether's theorem states that every continuous symmetry of the action of a physical system with conservative forces has a corresponding conservation law. This is the first of two theorems proven by mathematician Emmy Noether in 1915 and published in 1918. The action of a physical system is the integral over time of a Lagrangian function, from which the system's behavior can be determined by the principle of least action. This theorem only applies to continuous and smooth symmetries of physical space.

<span class="mw-page-title-main">Radon transform</span> Integral transform

In mathematics, the Radon transform is the integral transform which takes a function f defined on the plane to a function Rf defined on the (two-dimensional) space of lines in the plane, whose value at a particular line is equal to the line integral of the function over that line. The transform was introduced in 1917 by Johann Radon, who also provided a formula for the inverse transform. Radon further included formulas for the transform in three dimensions, in which the integral is taken over planes. It was later generalized to higher-dimensional Euclidean spaces and more broadly in the context of integral geometry. The complex analogue of the Radon transform is known as the Penrose transform. The Radon transform is widely applicable to tomography, the creation of an image from the projection data associated with cross-sectional scans of an object.

<span class="mw-page-title-main">Characteristic function (probability theory)</span> Fourier transform of the probability density function

In probability theory and statistics, the characteristic function of any real-valued random variable completely defines its probability distribution. If a random variable admits a probability density function, then the characteristic function is the Fourier transform of the probability density function. Thus it provides an alternative route to analytical results compared with working directly with probability density functions or cumulative distribution functions. There are particularly simple results for the characteristic functions of distributions defined by the weighted sums of random variables.

In machine learning, kernel machines are a class of algorithms for pattern analysis, whose best known member is the support-vector machine (SVM). These methods involve using linear classifiers to solve nonlinear problems. The general task of pattern analysis is to find and study general types of relations in datasets. For many algorithms that solve these tasks, the data in raw representation have to be explicitly transformed into feature vector representations via a user-specified feature map: in contrast, kernel methods require only a user-specified kernel, i.e., a similarity function over all pairs of data points computed using inner products. The feature map in kernel machines is infinite dimensional but only requires a finite dimensional matrix from user-input according to the Representer theorem. Kernel machines are slow to compute for datasets larger than a couple of thousand examples without parallel processing.

In directional statistics, the von Mises–Fisher distribution, is a probability distribution on the -sphere in . If the distribution reduces to the von Mises distribution on the circle.

In probability theory, the family of complex normal distributions, denoted or , characterizes complex random variables whose real and imaginary parts are jointly normal. The complex normal family has three parameters: location parameter μ, covariance matrix , and the relation matrix . The standard complex normal is the univariate distribution with , , and .

Static force fields are fields, such as a simple electric, magnetic or gravitational fields, that exist without excitations. The most common approximation method that physicists use for scattering calculations can be interpreted as static forces arising from the interactions between two bodies mediated by virtual particles, particles that exist for only a short time determined by the uncertainty principle. The virtual particles, also known as force carriers, are bosons, with different bosons associated with each force.

<span class="mw-page-title-main">Logit-normal distribution</span>

In probability theory, a logit-normal distribution is a probability distribution of a random variable whose logit has a normal distribution. If Y is a random variable with a normal distribution, and t is the standard logistic function, then X = t(Y) has a logit-normal distribution; likewise, if X is logit-normally distributed, then Y = logit(X)= log (X/(1-X)) is normally distributed. It is also known as the logistic normal distribution, which often refers to a multinomial logit version (e.g.).

Lagrangian field theory is a formalism in classical field theory. It is the field-theoretic analogue of Lagrangian mechanics. Lagrangian mechanics is used to analyze the motion of a system of discrete particles each with a finite number of degrees of freedom. Lagrangian field theory applies to continua and fields, which have an infinite number of degrees of freedom.

In the mathematical theory of probability, multivariate Laplace distributions are extensions of the Laplace distribution and the asymmetric Laplace distribution to multiple variables. The marginal distributions of symmetric multivariate Laplace distribution variables are Laplace distributions. The marginal distributions of asymmetric multivariate Laplace distribution variables are asymmetric Laplace distributions.

In representation theory of mathematics, the Waldspurger formula relates the special values of two L-functions of two related admissible irreducible representations. Let k be the base field, f be an automorphic form over k, π be the representation associated via the Jacquet–Langlands correspondence with f. Goro Shimura (1976) proved this formula, when and f is a cusp form; Günter Harder made the same discovery at the same time in an unpublished paper. Marie-France Vignéras (1980) proved this formula, when and f is a newform. Jean-Loup Waldspurger, for whom the formula is named, reproved and generalized the result of Vignéras in 1985 via a totally different method which was widely used thereafter by mathematicians to prove similar formulas.

In set theory and logic, Buchholz's ID hierarchy is a hierarchy of subsystems of first-order arithmetic. The systems/theories are referred to as "the formal theories of ν-times iterated inductive definitions". IDν extends PA by ν iterated least fixed points of monotone operators.

References

  1. 1 2 A. Smola, A. Gretton, L. Song, B. Schölkopf. (2007). A Hilbert Space Embedding for Distributions Archived 2013-12-15 at the Wayback Machine . Algorithmic Learning Theory: 18th International Conference. Springer: 13–31.
  2. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 L. Song, K. Fukumizu, F. Dinuzzo, A. Gretton (2013). Kernel Embeddings of Conditional Distributions: A unified kernel framework for nonparametric inference in graphical models. IEEE Signal Processing Magazine30: 98–111.
  3. J. Shawe-Taylor, N. Christianini. (2004). Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge, UK.
  4. T. Hofmann, B. Schölkopf, A. Smola. (2008). Kernel Methods in Machine Learning. The Annals of Statistics36(3):1171–1220.
  5. Muandet, Krikamol; Fukumizu, Kenji; Sriperumbudur, Bharath; Schölkopf, Bernhard (2017-06-28). "Kernel Mean Embedding of Distributions: A Review and Beyond". Foundations and Trends in Machine Learning. 10 (1–2): 1–141. arXiv: 1605.09522 . doi:10.1561/2200000060. ISSN   1935-8237.
  6. 1 2 3 4 5 6 7 8 9 L. Song. (2008) Learning via Hilbert Space Embedding of Distributions. PhD Thesis, University of Sydney.
  7. K. Fukumizu, A. Gretton, X. Sun, and B. Schölkopf (2008). Kernel measures of conditional independence. Advances in Neural Information Processing Systems20, MIT Press, Cambridge, MA.
  8. 1 2 3 L. Song, J. Huang, A. J. Smola, K. Fukumizu. (2009).Hilbert space embeddings of conditional distributions. Proc. Int. Conf. Machine Learning. Montreal, Canada: 961–968.
    • Steinwart, Ingo; Christmann, Andreas (2008). Support Vector Machines. New York: Springer. ISBN   978-0-387-77241-7.
  9. Sriperumbudur, B. K.; Fukumizu, K.; Lanckriet, G.R.G. (2011). "Universality, Characteristic Kernels and RKHS Embedding of Measures". Journal of Machine Learning Research. 12 (70).
  10. Liang, Percy (2016), CS229T/STAT231: Statistical Learning Theory (PDF), Stanford lecture notes
  11. Sriperumbudur, B. K.; Fukumizu, K.; Lanckriet, G.R.G. (2010). On the relation between universality, characteristic kernels and RKHS embedding of measures. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. Italy.
  12. Micchelli, C.A.; Xu, Y.; Zhang, H. (2006). "Universal Kernels". Journal of Machine Learning Research. 7 (95): 2651–2667.
  13. S. Grunewalder, G. Lever, L. Baldassarre, S. Patterson, A. Gretton, M. Pontil. (2012). Conditional mean embeddings as regressors. Proc. Int. Conf. Machine Learning: 1823–1830.
  14. A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, A. Smola. (2012). A kernel two-sample test. Journal of Machine Learning Research, 13: 723–773.
  15. M. Dudík, S. J. Phillips, R. E. Schapire. (2007). Maximum Entropy Distribution Estimation with Generalized Regularization and an Application to Species Distribution Modeling. Journal of Machine Learning Research, 8: 1217–1260.
  16. A. Gretton, O. Bousquet, A. Smola, B. Schölkopf. (2005). Measuring statistical dependence with Hilbert–Schmidt norms. Proc. Intl. Conf. on Algorithmic Learning Theory: 63–78.
  17. L. Song, A. Smola, A. Gretton, K. Borgwardt, J. Bedo. (2007). Supervised feature selection via dependence estimation. Proc. Intl. Conf. Machine Learning, Omnipress: 823–830.
  18. L. Song, A. Smola, A. Gretton, K. Borgwardt. (2007). A dependence maximization view of clustering. Proc. Intl. Conf. Machine Learning. Omnipress: 815–822.
  19. L. Song, A. Smola, K. Borgwardt, A. Gretton. (2007). Colored maximum variance unfolding. Neural Information Processing Systems.
  20. Zoltán Szabó, Bharath K. Sriperumbudur. Characteristic and Universal Tensor Product Kernels. Journal of Machine Learning Research, 19:1–29, 2018.
  21. 1 2 K. Muandet, K. Fukumizu, F. Dinuzzo, B. Schölkopf. (2012). Learning from Distributions via Support Measure Machines. Advances in Neural Information Processing Systems: 10–18.
  22. 1 2 3 4 K. Zhang, B. Schölkopf, K. Muandet, Z. Wang. (2013). Domain adaptation under target and conditional shift. Journal of Machine Learning Research, 28(3): 819–827.
  23. 1 2 A. Gretton, A. Smola, J. Huang, M. Schmittfull, K. Borgwardt, B. Schölkopf. (2008). Covariate shift and local learning by distribution matching. In J. Quinonero-Candela, M. Sugiyama, A. Schwaighofer, N. Lawrence (eds.). Dataset shift in machine learning, MIT Press, Cambridge, MA: 131–160.
  24. 1 2 3 K. Muandet, D. Balduzzi, B. Schölkopf. (2013).Domain Generalization Via Invariant Feature Representation. 30th International Conference on Machine Learning.
  25. 1 2 Z. Szabó, B. Sriperumbudur, B. Póczos, A. Gretton. Learning Theory for Distribution Regression. Journal of Machine Learning Research, 17(152):1–40, 2016.