Quantile

Last updated
Probability density of a normal distribution, with quantiles shown. The area below the red curve is the same in the intervals (-[?],Q1), (Q1,Q2), (Q2,Q3), and (Q3,+[?]). Iqr with quantile.png
Probability density of a normal distribution, with quantiles shown. The area below the red curve is the same in the intervals (−∞,Q1), (Q1,Q2), (Q2,Q3), and (Q3,+∞).

In statistics and probability, quantiles are cut points dividing the range of a probability distribution into continuous intervals with equal probabilities, or dividing the observations in a sample in the same way. There is one fewer quantile than the number of groups created. Common quantiles have special names, such as quartiles (four groups), deciles (ten groups), and percentiles (100 groups). The groups created are termed halves, thirds, quarters, etc., though sometimes the terms for the quantile are used for the groups created, rather than for the cut points.

Contents

q-quantiles are values that partition a finite set of values into q subsets of (nearly) equal sizes. There are q − 1 partitions of the q-quantiles, one for each integer k satisfying 0 < k < q. In some cases the value of a quantile may not be uniquely determined, as can be the case for the median (2-quantile) of a uniform probability distribution on a set of even size. Quantiles can also be applied to continuous distributions, providing a way to generalize rank statistics to continuous variables (see percentile rank). When the cumulative distribution function of a random variable is known, the q-quantiles are the application of the quantile function (the inverse function of the cumulative distribution function) to the values {1/q, 2/q, …, (q − 1)/q}.

Quantiles of a population

As in the computation of, for example, standard deviation, the estimation of a quantile depends upon whether one is operating with a statistical population or with a sample drawn from it. For a population, of discrete values or for a continuous population density, the k-th q-quantile is the data value where the cumulative distribution function crosses k/q. That is, x is a k-th q-quantile for a variable X if

Pr[X < x] ≤ k/q or, equivalently, Pr[Xx] ≥ 1 − k/q

and

Pr[Xx] ≥ k/q.

For a finite population of N equally probable values indexed 1, …, N from lowest to highest, the k-th q-quantile of this population can equivalently be computed via the value of Ip = Nk/q. If Ip is not an integer, then round up to the next integer to get the appropriate index; the corresponding data value is the k-th q-quantile. On the other hand, if Ip is an integer then any number from the data value at that index to the data value of the next index can be taken as the quantile, and it is conventional (though arbitrary) to take the average of those two values (see Estimating quantiles from a sample).

If, instead of using integers k and q, the "p-quantile" is based on a real number p with 0 < p < 1 then p replaces k/q in the above formulas. This broader terminology is used when quantiles are used to parameterize continuous probability distributions. Moreover, some software programs (including Microsoft Excel) regard the minimum and maximum as the 0th and 100th percentile, respectively. However, this broader terminology is an extension beyond traditional statistics definitions.

Examples

The following two examples use the Nearest Rank definition of quantile with rounding. For an explanation of this definition, see percentiles.

Even-sized population

Consider an ordered population of 10 data values [3, 6, 7, 8, 8, 10, 13, 15, 16, 20]. What are the 4-quantiles (the "quartiles") of this dataset?

QuartileCalculationResult
Zeroth quartileAlthough not universally accepted, one can also speak of the zeroth quartile. This is the minimum value of the set, so the zeroth quartile in this example would be 3.3
First quartileThe rank of the first quartile is 10×(1/4) = 2.5, which rounds up to 3, meaning that 3 is the rank in the population (from least to greatest values) at which approximately 1/4 of the values are less than the value of the first quartile. The third value in the population is 7.7
Second quartileThe rank of the second quartile (same as the median) is 10×(2/4) = 5, which is an integer, while the number of values (10) is an even number, so the average of both the fifth and sixth values is takenthat is (8+10)/2 = 9, though any value from 8 through to 10 could be taken to be the median.9
Third quartileThe rank of the third quartile is 10×(3/4) = 7.5, which rounds up to 8. The eighth value in the population is 15.15
Fourth quartileAlthough not universally accepted, one can also speak of the fourth quartile. This is the maximum value of the set, so the fourth quartile in this example would be 20. Under the Nearest Rank definition of quantile, the rank of the fourth quartile is the rank of the biggest number, so the rank of the fourth quartile would be 10.20

So the first, second and third 4-quantiles (the "quartiles") of the dataset [3, 6, 7, 8, 8, 10, 13, 15, 16, 20] are [7, 9, 15]. If also required, the zeroth quartile is 3 and the fourth quartile is 20.

Odd-sized population

Consider an ordered population of 11 data values [3, 6, 7, 8, 8, 9, 10, 13, 15, 16, 20]. What are the 4-quantiles (the "quartiles") of this dataset?

QuartileCalculationResult
Zeroth quartileAlthough not universally accepted, one can also speak of the zeroth quartile. This is the minimum value of the set, so the zeroth quartile in this example would be 3.3
First quartileThe first quartile is determined by 11×(1/4) = 2.75, which rounds up to 3, meaning that 3 is the rank in the population (from least to greatest values) at which approximately 1/4 of the values are less than the value of the first quartile. The third value in the population is 7.7
Second quartileThe second quartile value (same as the median) is determined by 11×(2/4) = 5.5, which rounds up to 6. Therefore, 6 is the rank in the population (from least to greatest values) at which approximately 2/4 of the values are less than the value of the second quartile (or median). The sixth value in the population is 9.9
Third quartileThe third quartile value for the original example above is determined by 11×(3/4) = 8.25, which rounds up to 9. The ninth value in the population is 15.15
Fourth quartileAlthough not universally accepted, one can also speak of the fourth quartile. This is the maximum value of the set, so the fourth quartile in this example would be 20. Under the Nearest Rank definition of quantile, the rank of the fourth quartile is the rank of the biggest number, so the rank of the fourth quartile would be 11.20

So the first, second and third 4-quantiles (the "quartiles") of the dataset [3, 6, 7, 8, 8, 9, 10, 13, 15, 16, 20] are [7, 9, 15]. If also required, the zeroth quartile is 3 and the fourth quartile is 20.

Relationship to the mean

For any population probability distribution on finitely many values, and generally for any probability distribution with a mean and variance, it is the case that

where Q(p) is the value of the p-quantile for 0 < p < 1 (or equivalently is the k-th q-quantile for p = k/q), where μ is the distribution's arithmetic mean, and where σ is the distribution's standard deviation. [1] In particular, the median (p = k/q = 1/2) is never more than one standard deviation from the mean.

The above formula can be used to bound the value μ + in terms of quantiles. When z ≥ 0, the value that is z standard deviations above the mean has a lower bound

For example, the value that is z = 1 standard deviation above the mean is always greater than or equal to Q(p = 0.5), the median, and the value that is z = 2 standard deviations above the mean is always greater than or equal to Q(p = 0.8), the fourth quintile.

When z ≤ 0, there is instead an upper bound

For example, the value μ + for z = −3 will never exceed Q(p = 0.1), the first decile.

Estimating quantiles from a sample

One problem which frequently arises is estimating a quantile of a (very large or infinite) population based on a finite sample of size N.

Modern statistical packages rely on a number of techniques to estimate the quantiles.

Hyndman and Fan compiled a taxonomy of nine algorithms [2] used by various software packages. All methods compute Qp, the estimate for the p-quantile (the k-th q-quantile, where p = k/q) from a sample of size N by computing a real valued index h. When h is an integer, the h-th smallest of the N values, xh, is the quantile estimate. Otherwise a rounding or interpolation scheme is used to compute the quantile estimate from h, xh, and xh. (For notation, see floor and ceiling functions).

The first three are piecewise constant, changing abruptly at each data point, while the last six use linear interpolation between data points, and differ only in how the index h used to choose the point along the piecewise linear interpolation curve, is chosen.

Mathematica, [3] Matlab, [4] R [5] and GNU Octave [6] programming languages support all nine sample quantile methods. SAS includes five sample quantile methods, SciPy [7] and Maple [8] both include eight, EViews [9] and Julia [10] include the six piecewise linear functions, Stata [11] includes two, Python [12] includes two, and Microsoft Excel includes two. Mathematica, SciPy and Julia support arbitrary parameters for methods which allow for other, non-standard, methods.

The estimate types and interpolation schemes used include:

TypehQpNotes
R‑1, SAS‑3, Maple‑1NpxhInverse of empirical distribution function.
R‑2, SAS‑5, Maple‑2, StataNp + 1/2(xh – 1/2⌉ + xh + 1/2⌋) / 2The same as R-1, but with averaging at discontinuities.
R‑3, SAS‑2Np − 1/2xhThe observation numbered closest to Np. Here, h indicates rounding to the nearest integer, choosing the even integer in the case of a tie.
R‑4, SAS‑1, SciPy‑(0,1), Julia‑(0,1), Maple‑3Npxh + (h − ⌊h⌋) (xhxh)Linear interpolation of the inverse of the empirical distribution function.
R‑5, SciPy‑(1/2,1/2), Julia‑(1/2,1/2), Maple‑4Np + 1/2Piecewise linear function where the knots are the values midway through the steps of the empirical distribution function.
R‑6, Excel, Python, SAS‑4, SciPy‑(0,0), Julia-(0,0), Maple‑5, Stata‑altdef(N + 1)pLinear interpolation of the expectations for the order statistics for the uniform distribution on [0,1]. That is, it is the linear interpolation between points (ph, xh), where ph = h/(N+1) is the probability that the last of (N+1) randomly drawn values will not exceed the h-th smallest of the first N randomly drawn values.
R‑7, Excel, Python, SciPy‑(1,1), Julia-(1,1), Maple‑6, NumPy(N − 1)p + 1Linear interpolation of the modes for the order statistics for the uniform distribution on [0,1].
R‑8, SciPy‑(1/3,1/3), Julia‑(1/3,1/3), Maple‑7(N + 1/3)p + 1/3Linear interpolation of the approximate medians for order statistics.
R‑9, SciPy‑(3/8,3/8), Julia‑(3/8,3/8), Maple‑8(N + 1/4)p + 3/8The resulting quantile estimates are approximately unbiased for the expected order statistics if x is normally distributed.

Notes:

Of the techniques, Hyndman and Fan recommend R-8, but most statistical software packages have chosen R-6 or R-7 as the default. [13]

The standard error of a quantile estimate can in general be estimated via the bootstrap. The Maritz–Jarrett method can also be used. [14]

The asymptotic distribution of the sample median

The sample median is the most examined one amongst quantiles, being an alternative to estimate a location parameter, when the expected value of the distribution does not exist, and hence the sample mean is not a meaningful estimator of a population characteristic. Moreover, the sample median is a more robust estimator than the sample mean.

One peculiarity of the sample median is its asymptotic distribution: when the sample comes from a continuous distribution, then the sample median has the anticipated Normal asymptotic distribution,

This extends to the other quantiles,

where f(xp) is the value of the distribution density at the p-th population quantile (). [15]

But when the distribution is discrete, then the distribution of the sample median and the other quantiles fails to be Normal (see examples in https://stats.stackexchange.com/a/86638/28746).

A solution to this problem is to use an alternative definition of sample quantiles through the concept of the "mid-distribution" function, which is defined as

The definition of sample quantiles through the concept of mid-distribution function can be seen as a generalization that can cover as special cases the continuous distributions. For discrete distributions the sample median as defined through this concept has an asymptotically Normal distribution, see Ma, Y., Genton, M. G., & Parzen, E. (2011). Asymptotic properties of sample quantiles of discrete distributions. Annals of the Institute of Statistical Mathematics, 63(2), 227-243.

Approximate quantiles from a stream

Computing approximate quantiles from data arriving from a stream can be done efficiently using compressed data structures. The most popular methods are t-digest [16] and KLL. [17] These methods read a stream of values in a continuous fashion and can, at any time, be queried about the approximate value of a specified quantile.

Both algorithms are based on a similar idea: compressing the stream of values by summarizing identical or similar values with a weight. If the stream is made of a repetition of 100 times v1 and 100 times v2, there is no reason to keep a sorted list of 200 elements, it is enough to keep two elements and two counts to be able to recover the quantiles. With more values, these algorithms maintain a trade-off between the number of unique values stored and the precision of the resulting quantiles. Some values may be discarded from the stream and contribute to the weight of a nearby value without changing the quantile results too much. The t-digest maintains a data structure of bounded size using an approach motivated by k-means clustering to group similar values. The KLL algorithm uses a more sophisticated "compactor" method that leads to better control of the error bounds at the cost of requiring an unbounded size if errors must be bounded relative to p.

Both methods belong to the family of data sketches that are subsets of Streaming Algorithms with useful properties: t-digest or KLL sketches can be combined. Computing the sketch for a very large vector of values can be split into trivially parallel processes where sketches are computed for partitions of the vector in parallel and merged later.

Discussion

Standardized test results are commonly reported as a student scoring "in the 80th percentile", for example. This uses an alternative meaning of the word percentile as the interval between (in this case) the 80th and the 81st scalar percentile. [18] This separate meaning of percentile is also used in peer-reviewed scientific research articles. [19] The meaning used can be derived from its context.

If a distribution is symmetric, then the median is the mean (so long as the latter exists). But, in general, the median and the mean can differ. For instance, with a random variable that has an exponential distribution, any particular sample of this random variable will have roughly a 63% chance of being less than the mean. This is because the exponential distribution has a long tail for positive values but is zero for negative numbers.

Quantiles are useful measures because they are less susceptible than means to long-tailed distributions and outliers. Empirically, if the data being analyzed are not actually distributed according to an assumed distribution, or if there are other potential sources for outliers that are far removed from the mean, then quantiles may be more useful descriptive statistics than means and other moment-related statistics.

Closely related is the subject of least absolute deviations, a method of regression that is more robust to outliers than is least squares, in which the sum of the absolute value of the observed errors is used in place of the squared error. The connection is that the mean is the single estimate of a distribution that minimizes expected squared error while the median minimizes expected absolute error. Least absolute deviations shares the ability to be relatively insensitive to large deviations in outlying observations, although even better methods of robust regression are available.

The quantiles of a random variable are preserved under increasing transformations, in the sense that, for example, if m is the median of a random variable X, then 2m is the median of 2X, unless an arbitrary choice has been made from a range of values to specify a particular quantile. (See quantile estimation, above, for examples of such interpolation.) Quantiles can also be used in cases where only ordinal data are available.

Other quantifications

Values that divide sorted data into equal subsets other than four have different names.

See also

Related Research Articles

In statistics, a central tendency is a central or typical value for a probability distribution.

<span class="mw-page-title-main">Interquartile range</span> Measure of statistical dispersion

In descriptive statistics, the interquartile range (IQR) is a measure of statistical dispersion, which is the spread of the data. The IQR may also be called the midspread, middle 50%, fourth spread, or H‑spread. It is defined as the difference between the 75th and 25th percentiles of the data. To calculate the IQR, the data set is divided into quartiles, or four rank-ordered even parts via linear interpolation. These quartiles are denoted by Q1 (also called the lower quartile), Q2 (the median), and Q3 (also called the upper quartile). The lower quartile corresponds with the 25th percentile and the upper quartile corresponds with the 75th percentile, so IQR = Q3 − Q1.

<span class="mw-page-title-main">Median</span> Middle quantile of a data set or probability distribution

In statistics and probability theory, the median is the value separating the higher half from the lower half of a data sample, a population, or a probability distribution. For a data set, it may be thought of as "the middle" value. The basic feature of the median in describing data compared to the mean is that it is not skewed by a small proportion of extremely large or small values, and therefore provides a better representation of the center. Median income, for example, may be a better way to describe the center of the income distribution because increases in the largest incomes alone have no effect on the median. For this reason, the median is of central importance in robust statistics.

<span class="mw-page-title-main">Normal distribution</span> Probability distribution

In statistics, a normal distribution or Gaussian distribution is a type of continuous probability distribution for a real-valued random variable. The general form of its probability density function is

In statistics, quartiles are a type of quantiles which divide the number of data points into four parts, or quarters, of more-or-less equal size. The data must be ordered from smallest to largest to compute quartiles; as such, quartiles are a form of order statistic. The three quartiles, resulting in four data divisions, are as follows:

<span class="mw-page-title-main">Standard deviation</span> In statistics, a measure of variation

In statistics, the standard deviation is a measure of the amount of variation of a random variable expected about its mean. A low standard deviation indicates that the values tend to be close to the mean of the set, while a high standard deviation indicates that the values are spread out over a wider range.

<span class="mw-page-title-main">Skewness</span> Measure of the asymmetry of random variables

In probability theory and statistics, skewness is a measure of the asymmetry of the probability distribution of a real-valued random variable about its mean. The skewness value can be positive, zero, negative, or undefined.

In statistics, a k-thpercentile, also known as percentile score or centile, is a score below which a given percentage k of scores in its frequency distribution falls or a score at or below which a given percentage falls. Percentiles are expressed in the same unit of measurement as the input scores, not in percent; for example, if the scores refer to human weight, the corresponding percentiles will be expressed in kilograms or pounds. In the limit of an infinite sample size, the percentile approximates the percentile function, the inverse of the cumulative distribution function.

In statistical inference, specifically predictive inference, a prediction interval is an estimate of an interval in which a future observation will fall, with a certain probability, given what has already been observed. Prediction intervals are often used in regression analysis.

A tolerance interval (TI) is a statistical interval within which, with some confidence level, a specified sampled proportion of a population falls. "More specifically, a 100×p%/100×(1−α) tolerance interval provides limits within which at least a certain proportion (p) of the population falls with a given level of confidence (1−α)." "A (p, 1−α) tolerance interval (TI) based on a sample is constructed so that it would include at least a proportion p of the sampled population with confidence 1−α; such a TI is usually referred to as p-content − (1−α) coverage TI." "A (p, 1−α) upper tolerance limit (TL) is simply a 1−α upper confidence limit for the 100 p percentile of the population."

In probability theory and statistics, the coefficient of variation (CV), also known as Normalized Root-Mean-Square Deviation (NRMSD), Percent RMS, and relative standard deviation (RSD), is a standardized measure of dispersion of a probability distribution or frequency distribution. It is defined as the ratio of the standard deviation to the mean , and often expressed as a percentage ("%RSD"). The CV or RSD is widely used in analytical chemistry to express the precision and repeatability of an assay. It is also commonly used in fields such as engineering or physics when doing quality assurance studies and ANOVA gauge R&R, by economists and investors in economic models, and in psychology/neuroscience.

In statistics, the mode is the value that appears most often in a set of data values. If X is a discrete random variable, the mode is the value x at which the probability mass function takes its maximum value. In other words, it is the value that is most likely to be sampled.

This glossary of statistics and probability is a list of definitions of terms and concepts used in the mathematical sciences of statistics and probability, their sub-disciplines, and related fields. For additional related terms, see Glossary of mathematics and Glossary of experimental design.

<span class="mw-page-title-main">Q–Q plot</span> Plot of the empirical distribution of p-values against the theoretical one

In statistics, a Q–Q plot (quantile–quantile plot) is a probability plot, a graphical method for comparing two probability distributions by plotting their quantiles against each other. A point (x, y) on the plot corresponds to one of the quantiles of the second distribution (y-coordinate) plotted against the same quantile of the first distribution (x-coordinate). This defines a parametric curve where the parameter is the index of the quantile interval.

Bootstrapping is any test or metric that uses random sampling with replacement, and falls under the broader class of resampling methods. Bootstrapping assigns measures of accuracy to sample estimates. This technique allows estimation of the sampling distribution of almost any statistic using random sampling methods.

In statistics, the median absolute deviation (MAD) is a robust measure of the variability of a univariate sample of quantitative data. It can also refer to the population parameter that is estimated by the MAD calculated from a sample.

<span class="mw-page-title-main">Quantile function</span> Statistical function that defines the quantiles of a probability distribution

In probability and statistics, the quantile function outputs the value of a random variable such that its probability is less than or equal to an input probability value. Intuitively, the quantile function associates with a range at and below a probability input the likelihood that a random variable is realized in that range for some probability distribution. It is also called the percentile function, percent-point function, inverse cumulative distribution function or inverse distribution function.

In probability theory and statistics, the index of dispersion, dispersion index,coefficient of dispersion,relative variance, or variance-to-mean ratio (VMR), like the coefficient of variation, is a normalized measure of the dispersion of a probability distribution: it is a measure used to quantify whether a set of observed occurrences are clustered or dispersed compared to a standard statistical model.

In statistics, robust measures of scale are methods that quantify the statistical dispersion in a sample of numerical data while resisting outliers. The most common such robust statistics are the interquartile range (IQR) and the median absolute deviation (MAD). These are contrasted with conventional or non-robust measures of scale, such as sample standard deviation, which are greatly influenced by outliers.

In statistics and probability theory, the nonparametric skew is a statistic occasionally used with random variables that take real values. It is a measure of the skewness of a random variable's distribution—that is, the distribution's tendency to "lean" to one side or the other of the mean. Its calculation does not require any knowledge of the form of the underlying distribution—hence the name nonparametric. It has some desirable properties: it is zero for any symmetric distribution; it is unaffected by a scale shift; and it reveals either left- or right-skewness equally well. In some statistical samples it has been shown to be less powerful than the usual measures of skewness in detecting departures of the population from normality.

References

  1. Bagui, S.; Bhaumik, D. (2004). "Glimpses of inequalities in probability and statistics" (PDF). International Journal of Statistical Sciences. 3: 9–15. ISSN   1683-5603.
  2. Hyndman, Rob J.; Fan, Yanan (November 1996). "Sample Quantiles in Statistical Packages". American Statistician. American Statistical Association. 50 (4): 361–365. doi:10.2307/2684934. JSTOR   2684934.
  3. Mathematica Documentation See 'Details' section
  4. "Quantile calculation". uk.mathworks.com.
  5. Frohne, Ivan; Hyndman, Rob J. (2009). Sample Quantiles. R Project. ISBN   978-3-900051-07-5.
  6. "Function Reference: quantile - Octave-Forge - SourceForge" . Retrieved 6 September 2013.
  7. "scipy.stats.mstats.mquantiles — SciPy v1.4.1 Reference Guide". docs.scipy.org.
  8. "Statistics - Maple Programming Help". www.maplesoft.com.
  9. "EViews 9 Help". Archived from the original on April 16, 2016. Retrieved April 4, 2016.
  10. "Statistics - Julia Documentation" . Retrieved June 17, 2023.
  11. Stata documentation for the pctile and xtile commands See 'Methods and formulas' section.
  12. "statistics — Mathematical statistics functions — Python 3.8.3rc1 documentation". docs.python.org.
  13. Hyndman, Rob J. (28 March 2016). "Sample quantiles 20 years later". Hyndsignt blog. Retrieved 2020-11-30.
  14. Wilcox, Rand R. (2010). Introduction to Robust Estimation and Hypothesis Testing. Academic Press. ISBN   978-0-12-751542-7.
  15. Stuart, Alan; Ord, Keith (1994). Kendall's Advanced Theory of Statistics. London: Arnold. ISBN   0340614307.
  16. Dunning, Ted; Ertl, Otmar (February 2019). "Computing Extremely Accurate Quantiles Using t-Digests". arXiv: 1902.04023 [stat.CO].
  17. Zohar Karnin; Kevin Lang; Edo Liberty (2016). "Optimal Quantile Approximation in Streams". arXiv: 1603.05346 [cs.DS].
  18. "percentile". Oxford Reference. Retrieved 2020-08-17.
  19. Kruger, J.; Dunning, D. (December 1999). "Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments". Journal of Personality and Social Psychology. 77 (6): 1121–1134. doi:10.1037/0022-3514.77.6.1121. ISSN   0022-3514. PMID   10626367. S2CID   2109278.
  20. Walker, Helen Mary; Lev, Joseph (1969). Elementary Statistical Methods. Holt, Rinehart and Winston. ISBN   978-0-03-081130-2.
  21. Stephen B. Vardeman (1992). "What about the Other Intervals?". The American Statistician. 46 (3): 193–197. doi:10.2307/2685212. JSTOR   2685212.

Further reading