Bhattacharyya distance

In statistics, the Bhattacharyya distance is a quantity which represents a notion of similarity between two probability distributions. ^[1] It is closely related to the Bhattacharyya coefficient, which is a measure of the amount of overlap between two statistical samples or populations.

It is not a metric, despite being named a "distance", since it does not obey the triangle inequality.

History

Both the Bhattacharyya distance and the Bhattacharyya coefficient are named after Anil Kumar Bhattacharyya, a statistician who worked in the 1930s at the Indian Statistical Institute. ^[2] He has developed this through a series of papers. ^{[3] [4] [5]} He developed the method to measure the distance between two non-normal distributions and illustrated this with the classical multinomial populations, ^[3] this work despite being submitted for publication in 1941, appeared almost five years later in Sankhya. ^{[3] [2]} Consequently, Professor Bhattacharyya started working toward developing a distance metric for probability distributions that are absolutely continuous with respect to the Lebesgue measure and published his progress in 1942, at Proceedings of the Indian Science Congress ^[4] and the final work has appeared in 1943 in the Bulletin of the Calcutta Mathematical Society. ^[5]

Definition

For probability distributions${\displaystyle P}$ and ${\displaystyle Q}$ on the same domain ${\displaystyle {\mathcal {X}}}$, the Bhattacharyya distance is defined as

${\displaystyle D_{B}(P,Q)=-\ln \left(BC(P,Q)\right)}$

where

${\displaystyle BC(P,Q)=\sum _{x\in {\mathcal {X}}}{\sqrt {P(x)Q(x)}}}$

is the Bhattacharyya coefficient for discrete probability distributions.

For continuous probability distributions, with ${\displaystyle P(dx)=p(x)dx}$ and ${\displaystyle Q(dx)=q(x)dx}$ where ${\displaystyle p(x)}$ and ${\displaystyle q(x)}$ are the probability density functions, the Bhattacharyya coefficient is defined as

${\displaystyle BC(P,Q)=\int _{\mathcal {X}}{\sqrt {p(x)q(x)}}\,dx}$.

More generally, given two probability measures ${\displaystyle P,Q}$ on a measurable space ${\displaystyle ({\mathcal {X}},{\mathcal {B}})}$, let ${\displaystyle \lambda }$ be a (sigma finite) measure such that ${\displaystyle P}$ and ${\displaystyle Q}$ are absolutely continuous with respect to ${\displaystyle \lambda }$ i.e. such that ${\displaystyle P(dx)=p(x)\lambda (dx)}$, and ${\displaystyle Q(dx)=q(x)\lambda (dx)}$ for probability density functions ${\displaystyle p,q}$ with respect to ${\displaystyle \lambda }$ defined ${\displaystyle \lambda }$-almost everywhere. Such a measure, even such a probability measure, always exists, e.g. ${\displaystyle \lambda ={\tfrac {1}{2}}(P+Q)}$. Then define the Bhattacharyya measure on ${\displaystyle ({\mathcal {X}},{\mathcal {B}})}$ by

${\displaystyle bc(dx|P,Q)={\sqrt {p(x)q(x)}}\,\lambda (dx)={\sqrt {{\frac {P(dx)}{\lambda (dx)}}(x){\frac {Q(dx)}{\lambda (dx)}}(x)}}\lambda (dx).}$

It does not depend on the measure ${\displaystyle \lambda }$, for if we choose a measure ${\displaystyle \mu }$ such that ${\displaystyle \lambda }$ and an other measure choice ${\displaystyle \lambda '}$ are absolutely continuous i.e. ${\displaystyle \lambda =l(x)\mu }$ and ${\displaystyle \lambda '=l'(x)\mu }$, then

${\displaystyle P(dx)=p(x)\lambda (dx)=p'(x)\lambda '(dx)=p(x)l(x)\mu (dx)=p'(x)l'(x)\mu (dx)}$,

and similarly for ${\displaystyle Q}$. We then have

${\displaystyle bc(dx|P,Q)={\sqrt {p(x)q(x)}}\,\lambda (dx)={\sqrt {p(x)q(x)}}\,l(x)\mu (x)={\sqrt {p(x)l(x)q(x)\,l(x)}}\mu (dx)={\sqrt {p'(x)l'(x)q'(x)l'(x)}}\,\mu (dx)={\sqrt {p'(x)q'(x)}}\,\lambda '(dx)}$.

We finally define the Bhattacharyya coefficient

${\displaystyle BC(P,Q)=\int _{\mathcal {X}}bc(dx|P,Q)=\int _{\mathcal {X}}{\sqrt {p(x)q(x)}}\,\lambda (dx)}$.

By the above, the quantity ${\displaystyle BC(P,Q)}$ does not depend on ${\displaystyle \lambda }$, and by the Cauchy inequality ${\displaystyle 0\leq BC(P,Q)\leq 1}$. In particular if ${\displaystyle P(dx)=p(x)Q(dx)}$ is absolutely continuous wrt to ${\displaystyle Q}$ with Radon Nikodym derivative ${\displaystyle p(x)={\frac {P(dx)}{Q(dx)}}(x)}$, then

$${\displaystyle BC(P,Q)=\int _{\mathcal {X}}{\sqrt {p(x)}}Q(dx)=\int _{\mathcal {X}}{\sqrt {\frac {P(dx)}{Q(dx)}}}Q(dx)=E_{Q}\left[{\sqrt {\frac {P(dx)}{Q(dx)}}}\right]}$$

Gaussian case

Let ${\displaystyle p\sim {\mathcal {N}}(\mu _{p},\sigma _{p}^{2})}$, ${\displaystyle q\sim {\mathcal {N}}(\mu _{q},\sigma _{q}^{2})}$, where ${\displaystyle {\mathcal {N}}(\mu ,\sigma ^{2})}$ is the normal distribution with mean ${\displaystyle \mu }$ and variance ${\displaystyle \sigma ^{2}}$; then

${\displaystyle D_{B}(p,q)={\frac {1}{4}}{\frac {(\mu _{p}-\mu _{q})^{2}}{\sigma _{p}^{2}+\sigma _{q}^{2}}}+{\frac {1}{2}}\ln \left({\frac {\sigma _{p}^{2}+\sigma _{q}^{2}}{2\sigma _{p}\sigma _{q}}}\right)}$.

And in general, given two multivariate normal distributions ${\displaystyle p_{i}={\mathcal {N}}({\boldsymbol {\mu }}_{i},\,{\boldsymbol {\Sigma }}_{i})}$,

${\displaystyle D_{B}(p_{1},p_{2})={1 \over 8}({\boldsymbol {\mu }}_{1}-{\boldsymbol {\mu }}_{2})^{T}{\boldsymbol {\Sigma }}^{-1}({\boldsymbol {\mu }}_{1}-{\boldsymbol {\mu }}_{2})+{1 \over 2}\ln \,\left({\det {\boldsymbol {\Sigma }} \over {\sqrt {\det {\boldsymbol {\Sigma }}_{1}\,\det {\boldsymbol {\Sigma }}_{2}}}}\right)}$,

where ${\displaystyle {\boldsymbol {\Sigma }}={{\boldsymbol {\Sigma }}_{1}+{\boldsymbol {\Sigma }}_{2} \over 2}.}$ ^[6] Note that the first term is a squared Mahalanobis distance.

Properties

${\displaystyle 0\leq BC\leq 1}$ and ${\displaystyle 0\leq D_{B}\leq \infty }$.

${\displaystyle D_{B}}$ does not obey the triangle inequality, though the Hellinger distance ${\displaystyle {\sqrt {1-BC(p,q)}}}$ does.

Bounds on Bayes Error

The Bhattacharyya distance can be used to upper and lower bound the Bayes error rate:

$${\displaystyle {\frac {1}{2}}-{\frac {1}{2}}{\sqrt {1-4\rho ^{2}}}\leq L^{*}\leq \rho }$$

where ${\displaystyle \rho =\mathbb {E} {\sqrt {\eta (X)(1-\eta (X))}}}$ and ${\displaystyle \eta (X)=\mathbb {P} (Y=1|X)}$ is the posterior probability. ^[7]

Applications

The Bhattacharyya coefficient quantifies the "closeness" of two random statistical samples.

Given two sequences from distributions ${\displaystyle P,Q}$, bin them into ${\displaystyle n}$ buckets, and let the frequency of samples from ${\displaystyle P}$ in bucket ${\displaystyle i}$ be ${\displaystyle p_{i}}$, and similarly for ${\displaystyle q_{i}}$, then the sample Bhattacharyya coefficient is

${\displaystyle BC(\mathbf {p} ,\mathbf {q} )=\sum _{i=1}^{n}{\sqrt {p_{i}q_{i}}},}$

which is an estimator of ${\displaystyle BC(P,Q)}$. The quality of estimation depends on the choice of buckets; too few buckets would overestimate ${\displaystyle BC(P,Q)}$, while too many would underestimate.

A common task in classification is estimating the separability of classes. Up to a multiplicative factor, the squared Mahalanobis distance is a special case of the Bhattacharyya distance when the two classes are normally distributed with the same variances. When two classes have similar means but significantly different variances, the Mahalanobis distance would be close to zero, while the Bhattacharyya distance would not be.

The Bhattacharyya coefficient is used in the construction of polar codes. ^[8]

The Bhattacharyya distance is used in feature extraction and selection, ^[9] image processing, ^[10] speaker recognition, ^[11] phone clustering, ^[12] and in genetics. ^[13]

See also

• Bhattacharyya angle
• Kullback–Leibler divergence
• Hellinger distance
• Mahalanobis distance
• Chernoff bound
• Rényi entropy
• F-divergence
• Fidelity of quantum states

References

1. Dodge, Yadolah (2003). The Oxford Dictionary of Statistical Terms. Oxford University Press. ISBN   978-0-19-920613-1.
2. Sen, Pranab Kumar (1996). "Anil Kumar Bhattacharyya (1915-1996): A Reverent Remembrance". Calcutta Statistical Association Bulletin. 46 (3–4): 151–158. doi:10.1177/0008068319960301. S2CID   164326977.
3. Bhattacharyya, A. (1946). "On a Measure of Divergence between Two Multinomial Populations". Sankhyā. 7 (4): 401–406. JSTOR   25047882.
4. Bhattacharyya, A (1942). "On discrimination and divergence". Proceedings of the Indian Science Congress. Asiatic Society of Bengal.
5. Bhattacharyya, A. (March 1943). "On a measure of divergence between two statistical populations defined by their probability distributions". Bulletin of the Calcutta Mathematical Society . 35: 99–109. MR   0010358.
6. Kashyap, Ravi (2019). "The Perfect Marriage and Much More: Combining Dimension Reduction, Distance Measures and Covariance". Physica A: Statistical Mechanics and its Applications. 536: 120938. arXiv: 1603.09060 . doi:10.1016/j.physa.2019.04.174.
7. Devroye, L., Gyorfi, L. & Lugosi, G. A Probabilistic Theory of Pattern Recognition. Discrete Appl Math 73, 192–194 (1997).
8. Arıkan, Erdal (July 2009). "Channel polarization: A method for constructing capacity-achieving codes for symmetric binary-input memoryless channels". IEEE Transactions on Information Theory. 55 (7): 3051–3073. arXiv: 0807.3917 . doi:10.1109/TIT.2009.2021379. S2CID   889822.
9. Euisun Choi, Chulhee Lee, "Feature extraction based on the Bhattacharyya distance", Pattern Recognition, Volume 36, Issue 8, August 2003, Pages 1703–1709
10. François Goudail, Philippe Réfrégier, Guillaume Delyon, "Bhattacharyya distance as a contrast parameter for statistical processing of noisy optical images", JOSA A, Vol. 21, Issue 7, pp. 1231−1240 (2004)
11. Chang Huai You, "An SVM Kernel With GMM-Supervector Based on the Bhattacharyya Distance for Speaker Recognition", Signal Processing Letters, IEEE, Vol 16, Is 1, pp. 49-52
12. Mak, B., "Phone clustering using the Bhattacharyya distance", Spoken Language, 1996. ICSLP 96. Proceedings., Fourth International Conference on, Vol 4, pp. 2005–2008 vol.4, 3−6 Oct 1996
13. Chattopadhyay, Aparna; Chattopadhyay, Asis Kumar; B-Rao, Chandrika (2004-06-01). "Bhattacharyya's distance measure as a precursor of genetic distance measures". Journal of Biosciences. 29 (2): 135–138. doi:10.1007/BF02703410. ISSN   0973-7138.

External links

• "Bhattacharyya distance", Encyclopedia of Mathematics , EMS Press, 2001 [1994]
• Statistical Intuition of Bhattacharyya's distance
• Some of the properties of Bhattacharyya Distance
• Nielsen, F.; Boltz, S. (2010). "The Burbea–Rao and Bhattacharyya centroids". IEEE Transactions on Information Theory. 57 (8): 5455–5466. ^[1]
• Kailath, T. (1967). "The Divergence and Bhattacharyya Distance Measures in Signal Selection". IEEE Transactions on Communication Technology. 15 (1): 52–60. ^[2]
• Djouadi, A.; Snorrason, O.; Garber, F. (1990). "The quality of Training-Sample estimates of the Bhattacharyya coefficient". IEEE Transactions on Pattern Analysis and Machine Intelligence. 12 (1): 92–97. ^[3]

1. Nielsen, Frank; Boltz, Sylvain (2011). "The Burbea-Rao and Bhattacharyya Centroids". IEEE Transactions on Information Theory. 57 (8): 5455–5466. arXiv: 1004.5049 . doi:10.1109/TIT.2011.2159046. ISSN   0018-9448. S2CID   14238708.
2. Kailath, T. (1967). "The Divergence and Bhattacharyya Distance Measures in Signal Selection". IEEE Transactions on Communications. 15 (1): 52–60. doi:10.1109/TCOM.1967.1089532. ISSN   0096-2244.
3. Djouadi, A.; Snorrason, O.; Garber, F.D. (1990). "The quality of training sample estimates of the Bhattacharyya coefficient". IEEE Transactions on Pattern Analysis and Machine Intelligence. 12 (1): 92–97. doi:10.1109/34.41388.