Berk-Jones test

The Berk-Jones test (or BJ test) refers to a class of non-parametric goodness-of-fit statistical tests used to determine whether a set of observed data follow a specific probability distribution. Introduced by Robert H. Berk and Douglas H. Jones in 1979, the test is specifically designed to be more powerful than the Kolmogorov-Smirnov test in the tails of the distribution. ^[1]

Background

In statistical hypothesis testing, a goodness-of-fit test compares an empirical distribution function (EDF) to a theoretical cumulative distribution function (CDF). While the Kolmogorov-Smirnov test is arguably the most well-known method, it is often criticized for its lack of sensitivity to deviations occurring at the extremes (tails) of the distribution.

The Berk-Jones test addresses this shortcoming by adopting a "pointwise" maximum likelihood ratio approach. It still belongs to the family of supremum-type statistics but also incorporates information-theoretic properties, specifically the Kullback-Leibler divergence.

Mathematical formulation

Let ${\displaystyle X_{1},X_{2},\dots ,X_{n}}$ be independent and identically distributed (i.i.d.) random variables with an empirical distribution function ${\displaystyle \mathbb {F} _{n}(x)}$. We wish to test the null hypothesis ${\displaystyle H_{0}:F=F_{0}}$.

The Berk-Jones statistic ${\displaystyle R_{n}}$ is defined as:

${\displaystyle R_{n}=\sup _{x}K(\mathbb {F} _{n}(x),F_{0}(x))}$

where ${\displaystyle K(p,q)}$ is the binary Kullback-Leibler divergence (or relative entropy), defined as:

${\displaystyle K(p,q)=p\log \left({\frac {p}{q}}\right)+\left(1-p\right)\log \left({\frac {1-p}{1-q}}\right)}$

The statistic can also be expressed in terms of the order statistics ${\displaystyle X_{(1)}<X_{(2)}<\cdots <X_{(n)}}$. For a fixed ${\displaystyle x}$, ${\displaystyle n\,\mathbb {F} _{n}(x)}$ follows a Binomial distribution ${\displaystyle {\text{Bin}}(n,F_{0}(x))}$. The BJ statistic effectively finds the ${\displaystyle x}$ that maximizes the likelihood ratio for testing the success probability of this binomial. ^[2]

Relationship with Tukey's Higher Criticism statistic

The Berk-Jones test is closely related to the Higher Criticism statistic (HC), originally suggested by John Tukey in 1976 and further developed in 2004 by David Donoho and Jianshun Jin. Both tests are used for sparse signal detection.

While the BJ test is based on the likelihood ratio, the HC statistic is based on a standardized distance (essentially a Z score approach):

${\displaystyle {\text{HC}}_{n}=\sup _{x}{\frac {{\sqrt {n}}\left(\mathbb {F} _{n}(x)-F_{0}(x)\right)}{\sqrt {F_{0}(x)\left(1-F_{0}(x)\right)}}}}$

Asymptotically (as ${\displaystyle n}$ approaches infinity), the BJ test and HC are equivalent in their ability to detect sparse mixtures. Specifically, they both belong to the Rényi family of statistics. However, the BJ test is over viewed as the "information-theoretic version" of HC. When applying a Taylor expansion to the Kullback-Leibler divergence ${\displaystyle K(p,q)}$ around ${\displaystyle p=q}$, the first non-vanishing term in the expansion is proportional to the square of the HC statistic: ^[3]

${\displaystyle K(p,q)\approxeq {\frac {(p-q)^{2}}{2q(1-q)}}}$

This shows that the Higher Criticism statistic is essentially a local quadratic approximation of the Berk-Jones statistic.

In high-dimensional settings where the signal is "rare and weak," both tests have the same detection boundary (the point at which it becomes mathematically possible to distinguish signal from noise). However, the BJ test is often preferred in finite samples of the ${\displaystyle [0,1]}$ interval more naturally than the ${\displaystyle Z}$-score denominator in HC.

Comparison of power

The effectiveness of the Berk-Jones test compared with other methods depends heavily on the nature of the deviation from the null hypothesis.

Test Method	Sensitivity Focus	Best Use Case
Kolmogorov–Smirnov	Median / Center	General deviations in the bulk of the data.
Anderson–Darling	Weighted Tails	Moderate tail sensitivity; common in engineering.
Berk–Jones	Extreme Tails	Sparse signals; likelihood-ratio based optimality.
Higher Criticism	Extreme Tails	Sparse signals; ${\displaystyle Z}$-score based optimality.

Applications

The Berk-Jones test is widely used in:

• Genomics: for identifying differentially expressed genes where only a small fraction of genes are expected to show a signal.
• Signal detection: for detecting weak, sparse signals against a background of noise.
• Meta-analysis: for combining p-values from multiple independent studies to detect a global effect.

See also

• Anderson–Darling test
• Empirical distribution function
• Kolmogorov–Smirnov test
• Kullback–Leibler divergence

References

1. Berk, R. H.; Jones, D. H. (1979). "Relatively Optimal Combinations of Test Statistics" (PDF). Scandinavian Journal of Statistics. 6 (1): 33–36.
2. Wellner, J. A.; Jager, L. (2007). "Goodness-of-fit tests via Phi-divergences". The Annals of Statistics. 35 (5): 2018–2053.
3. Donoho, D.; Jin, J. (2004). "Higher Criticism for detecting sparse heterogeneous mixtures". The Annals of Statistics. 32 (3): 962–994.