Percentile

Last updated

In statistics, a k-thpercentile (percentile score or centile), denoted , [1] is a score below which a given percentage k of scores in its frequency distribution falls (exclusive definition) or a score at or below which a given percentage falls (inclusive definition). For example, the 50th percentile (the median) is the score below which (exclusive) or at or below which (inclusive) 50% of the scores in the distribution may be found. Percentiles are expressed in the same unit of measurement as the input scores; for example, if the scores refer to human weight, the corresponding percentiles will be expressed in kilograms or pounds.

Contents

The percentile score and the percentile rank are related terms. The percentile rank of a score is the percentage of scores in its distribution that are less than it, an exclusive definition, and one that can be expressed with a single, simple formula. Percentile scores and percentile ranks are often used in the reporting of test scores from norm-referenced tests, but, as just noted, they are not the same. For percentile rank, a score is given and a percentage is computed. Percentile ranks are exclusive. If the percentile rank for a specified score is 90%, then 90% of the scores were lower. In contrast, for percentiles a percentage is given and a corresponding score is determined, which can be either exclusive or inclusive. The score for a specified percentage (e.g., 90th) indicates a score below which (exclusive definition) or at or below which (inclusive definition) other scores in the distribution fall.

The 25th percentile is also known as the first quartile (Q1), the 50th percentile as the median or second quartile (Q2), and the 75th percentile as the third quartile (Q3).

Applications

When ISPs bill "burstable" internet bandwidth, the 95th or 98th percentile usually cuts off the top 5% or 2% of bandwidth peaks in each month, and then bills at the nearest rate. In this way, infrequent peaks are ignored, and the customer is charged in a fairer way. The reason this statistic is so useful in measuring data throughput is that it gives a very accurate picture of the cost of the bandwidth. The 95th percentile says that 95% of the time, the usage is below this amount: so, the remaining 5% of the time, the usage is above that amount.

Physicians will often use infant and children's weight and height to assess their growth in comparison to national averages and percentiles which are found in growth charts.

The 85th percentile speed of traffic on a road is often used as a guideline in setting speed limits and assessing whether such a limit is too high or low. [2] [3]

In finance, value at risk is a standard measure to assess (in a model-dependent way) the quantity under which the value of the portfolio is not expected to sink within a given period of time and given a confidence value.

The normal distribution and percentiles

Representation of the three-sigma rule. The dark blue zone represents observations within one standard deviation (s) to either side of the mean (m), which accounts for about 68.3% of the population. Two standard deviations from the mean (dark and medium blue) account for about 95.4%, and three standard deviations (dark, medium, and light blue) for about 99.7%. Standard deviation diagram.svg
Representation of the three-sigma rule. The dark blue zone represents observations within one standard deviation (σ) to either side of the mean (μ), which accounts for about 68.3% of the population. Two standard deviations from the mean (dark and medium blue) account for about 95.4%, and three standard deviations (dark, medium, and light blue) for about 99.7%.

The methods given in the definitions section (below) are approximations for use in small-sample statistics. In general terms, for very large populations following a normal distribution, percentiles may often be represented by reference to a normal curve plot. The normal distribution is plotted along an axis scaled to standard deviations, or sigma () units. Mathematically, the normal distribution extends to negative infinity on the left and positive infinity on the right. Note, however, that only a very small proportion of individuals in a population will fall outside the −3σ to +3σ range. For example, with human heights very few people are above the +3σ height level.

Percentiles represent the area under the normal curve, increasing from left to right. Each standard deviation represents a fixed percentile. Thus, rounding to two decimal places, −3σ is the 0.13th percentile, −2σ the 2.28th percentile, −1σ the 15.87th percentile, 0σ the 50th percentile (both the mean and median of the distribution), +1σ the 84.13th percentile, +2σ the 97.72nd percentile, and +3σ the 99.87th percentile. This is related to the 68–95–99.7 rule or the three-sigma rule. Note that in theory the 0th percentile falls at negative infinity and the 100th percentile at positive infinity, although in many practical applications, such as test results, natural lower and/or upper limits are enforced.

Definitions

There is no standard definition of percentile, [4] [5] [6] however all definitions yield similar results when the number of observations is very large and the probability distribution is continuous. [7] In the limit, as the sample size approaches infinity, the 100pth percentile (0<p<1) approximates the inverse of the cumulative distribution function (CDF) thus formed, evaluated at p, as p approximates the CDF. This can be seen as a consequence of the Glivenko–Cantelli theorem. Some methods for calculating the percentiles are given below.

Calculation methods

Interpolated and nearest-rank, exclusive and inclusive, percentiles for 10-score distribution. Frequency histogram and exclusive and inclusive percentiles 2.png
Interpolated and nearest-rank, exclusive and inclusive, percentiles for 10-score distribution.

There are many formulas or algorithms [8] for a percentile score. Hyndman and Fan [4] identified nine and most statistical and spreadsheet software use one of the methods they describe. [9] Algorithms either return the value of a score that exists in the set of scores (nearest-rank methods) or interpolate between existing scores and are either exclusive or inclusive.

Nearest-rank methods (exclusive/inclusive)
PC: percentile specified0.100.250.500.750.90
N: Number of scores1010101010
OR: ordinal rank = PC × N12.557.59
Rank: >OR / ≥OR2/13/36/58/810/9
Score at rank (exc/inc)2/13/34/35/57/5

The figure shows a 10-score distribution, illustrates the percentile scores that result from these different algorithms, and serves as an introduction to the examples given subsequently. The simplest are nearest-rank methods that return a score from the distribution, although compared to interpolation methods, results can be a bit crude. The Nearest-Rank Methods table shows the computational steps for exclusive and inclusive methods.

Interpolated methods (exclusive/inclusive)
PC: percentile specified0.100.250.500.750.90
N: number of scores1010101010
OR: PC×(N+1) / PC×(N−1)+11.1/1.92.75/3.255.5/5.58.25/7.759.9/9.1
LoRank: OR truncated1/12/35/58/79/9
HIRank: OR rounded up2/23/46/69/810/10
LoScore: score at LoRank1/12/33/35/45/5
HiScore: score at HiRank2/23/34/45/57/7
Difference: HiScore − LoScore1/11/01/10/12/2
Mod: fractional part of OR0.1/0.90.75/0.250.5/0.50.25/0.750.9/0.1
Interpolated score (exc/inc)
= LoScore + Mod × Difference
1.1/1.92.75/33.5/3.55/4.756.8/5.2

Interpolation methods, as the name implies, can return a score that is between scores in the distribution. Algorithms used by statistical programs typically use interpolation methods, for example, the percentile.exl and percentile.inc function in Microsoft Excel. The Interpolated Methods table shows the computational steps.

The nearest-rank method

The percentile values for the ordered list {15, 20, 35, 40, 50} Percentile.png
The percentile values for the ordered list {15, 20, 35, 40, 50}

One definition of percentile, often given in texts, is that the P-th percentile of a list of N ordered values (sorted from least to greatest) is the smallest value in the list such that no more than P percent of the data is strictly less than the value and at least P percent of the data is less than or equal to that value. This is obtained by first calculating the ordinal rank and then taking the value from the ordered list that corresponds to that rank. The ordinal rank n is calculated using this formula

Note the following:

Worked examples of the nearest-rank method

Example 1

Consider the ordered list {15, 20, 35, 40, 50}, which contains 5 data values. What are the 5th, 30th, 40th, 50th and 100th percentiles of this list using the nearest-rank method?

Percentile
P
Number in list
N
Ordinal rank
n
Number from the ordered list
that has that rank
Percentile
value
Notes
5th5the first number in the ordered list, which is 151515 is the smallest element of the list; 0% of the data is strictly less than 15, and 20% of the data is less than or equal to 15.
30th5the 2nd number in the ordered list, which is 202020 is an element of the ordered list.
40th5the 2nd number in the ordered list, which is 2020In this example, it is the same as the 30th percentile.
50th5the 3rd number in the ordered list, which is 353535 is an element of the ordered list.
100th5the last number in the ordered list, which is 5050The 100th percentile is defined to be the largest value in the list, which is 50.

So the 5th, 30th, 40th, 50th and 100th percentiles of the ordered list {15, 20, 35, 40, 50} using the nearest-rank method are {15, 20, 20, 35, 50}.

Example 2

Consider an ordered population of 10 data values {3, 6, 7, 8, 8, 10, 13, 15, 16, 20}. What are the 25th, 50th, 75th and 100th percentiles of this list using the nearest-rank method?

Percentile
P
Number in list
N
Ordinal rank
n
Number from the ordered list
that has that rank
Percentile
value
Notes
25th10the 3rd number in the ordered list, which is 777 is an element of the list.
50th10the 5th number in the ordered list, which is 888 is an element of the list.
75th10the 8th number in the ordered list, which is 151515 is an element of the list.
100th10Last20, which is the last number in the ordered list20The 100th percentile is defined to be the largest value in the list, which is 20.

So the 25th, 50th, 75th and 100th percentiles of the ordered list {3, 6, 7, 8, 8, 10, 13, 15, 16, 20} using the nearest-rank method are {7, 8, 15, 20}.

Example 3

Consider an ordered population of 11 data values {3, 6, 7, 8, 8, 9, 10, 13, 15, 16, 20}. What are the 25th, 50th, 75th and 100th percentiles of this list using the nearest-rank method?

Percentile
P
Number in list
N
Ordinal rank
n
Number from the ordered list
that has that rank
Percentile
value
Notes
25th11the 3rd number in the ordered list, which is 777 is an element of the list.
50th11the 6th number in the ordered list, which is 999 is an element of the list.
75th11the 9th number in the ordered list, which is 151515 is an element of the list.
100th11Last20, which is the last number in the ordered list20The 100th percentile is defined to be the largest value in the list, which is 20.

So the 25th, 50th, 75th and 100th percentiles of the ordered list {3, 6, 7, 8, 8, 9, 10, 13, 15, 16, 20} using the nearest-rank method are {7, 9, 15, 20}.

The linear interpolation between closest ranks method

An alternative to rounding used in many applications is to use linear interpolation between adjacent ranks.

Commonalities between the variants of this method

All of the following variants have the following in common. Given the order statistics

we seek a linear interpolation function that passes through the points . This is simply accomplished by

where uses the floor function to represent the integral part of positive x, whereas uses the mod function to represent its fractional part (the remainder after division by 1). (Note that, though at the endpoint , is undefined, it does not need to be because it is multiplied by .) As we can see, x is the continuous version of the subscript i, linearly interpolating v between adjacent nodes.

There are two ways in which the variant approaches differ. The first is in the linear relationship between the rankx, the percent rank, and a constant that is a function of the sample size N:

There is the additional requirement that the midpoint of the range , corresponding to the median, occur at :

and our revised function now has just one degree of freedom, looking like this:

The second way in which the variants differ is in the definition of the function near the margins of the range of p: should produce, or be forced to produce, a result in the range , which may mean the absence of a one-to-one correspondence in the wider region. One author has suggested a choice of where ξ is the shape of the Generalized extreme value distribution which is the extreme value limit of the sampled distribution.

First variant, C = 1/2

The result of using each of the three variants on the ordered list {15, 20, 35, 40, 50} Percentile interpolation.png
The result of using each of the three variants on the ordered list {15, 20, 35, 40, 50}

(Sources: Matlab "prctile" function, [10] [11] )

where

Furthermore, let

The inverse relationship is restricted to a narrower region:

Worked example of the first variant

Consider the ordered list {15, 20, 35, 40, 50}, which contains five data values. What are the 5th, 30th, 40th and 95th percentiles of this list using the Linear Interpolation Between Closest Ranks method? First, we calculate the percent rank for each list value.

List value
Position of that value
in the ordered list
i
Number of values
N
Calculation of
percent rank
Percent rank,
151510
202530
353550
404570
505590

Then we take those percent ranks and calculate the percentile values as follows:

Percent rank
P
Number of values
N
Is ?Is ?Is there a
percent rank
equal to P?
What do we use for percentile value?Percentile value

Notes
55YesNoNoWe see that , which is less than the first percent rank , so use the first list value , which is 151515 is a member of the ordered list
305NoNoYesWe see that is the same as the second percent rank , so use the second list value , which is 202020 is a member of the ordered list
405NoNoNoWe see that is between percent rank and , so we take
.

Given those values we can then calculate v as follows:

27.527.5 is not a member of the ordered list
955NoYesNoWe see that , which is greater than the last percent rank , so use the last list value, which is 505050 is a member of the ordered list

So the 5th, 30th, 40th and 95th percentiles of the ordered list {15, 20, 35, 40, 50} using the Linear Interpolation Between Closest Ranks method are {15, 20, 27.5, 50}

Second variant, C = 1

(Source: Some software packages, including NumPy [12] and Microsoft Excel [6] (up to and including version 2013 by means of the PERCENTILE.INC function). Noted as an alternative by NIST [9] )

Note that the relationship is one-to-one for , the only one of the three variants with this property; hence the "INC" suffix, for inclusive, on the Excel function.

Worked examples of the second variant

Example 1

Consider the ordered list {15, 20, 35, 40, 50}, which contains five data values. What is the 40th percentile of this list using this variant method?

First we calculate the rank of the 40th percentile:

So, x=2.6, which gives us and . So, the value of the 40th percentile is

Example 2

Consider the ordered list {1,2,3,4} which contains four data values. What is the 75th percentile of this list using the Microsoft Excel method?

First we calculate the rank of the 75th percentile as follows:

So, x=3.25, which gives us an integral part of 3 and a fractional part of 0.25. So, the value of the 75th percentile is

Third variant, C = 0

(The primary variant recommended by NIST. [9] Adopted by Microsoft Excel since 2010 by means of PERCENTIL.EXC function. However, as the "EXC" suffix indicates, the Excel version excludes both endpoints of the range of p, i.e., , whereas the "INC" version, the second variant, does not; in fact, any number smaller than is also excluded and would cause an error.)

The inverse is restricted to a narrower region:

Worked example of the third variant

Consider the ordered list {15, 20, 35, 40, 50}, which contains five data values. What is the 40th percentile of this list using the NIST method?

First we calculate the rank of the 40th percentile as follows:

So x=2.4, which gives us and . So the value of the 40th percentile is calculated as:

So the value of the 40th percentile of the ordered list {15, 20, 35, 40, 50} using this variant method is 26.

The weighted percentile method

In addition to the percentile function, there is also a weighted percentile, where the percentage in the total weight is counted instead of the total number. There is no standard function for a weighted percentile. One method extends the above approach in a natural way.

Suppose we have positive weights associated, respectively, with our N sorted sample values. Let

the sum of the weights. Then the formulas above are generalized by taking

when ,

or

for general ,

and

The 50% weighted percentile is known as the weighted median.

See also

Related Research Articles

Binomial distribution Probability distribution

In probability theory and statistics, the binomial distribution with parameters n and p is the discrete probability distribution of the number of successes in a sequence of n independent experiments, each asking a yes–no question, and each with its own Boolean-valued outcome: success or failure. A single success/failure experiment is also called a Bernoulli trial or Bernoulli experiment, and a sequence of outcomes is called a Bernoulli process; for a single trial, i.e., n = 1, the binomial distribution is a Bernoulli distribution. The binomial distribution is the basis for the popular binomial test of statistical significance.

Binary search algorithm Search algorithm finding the position of a target value within a sorted array

In computer science, binary search, also known as half-interval search, logarithmic search, or binary chop, is a search algorithm that finds the position of a target value within a sorted array. Binary search compares the target value to the middle element of the array. If they are not equal, the half in which the target cannot lie is eliminated and the search continues on the remaining half, again taking the middle element to compare to the target value, and repeating this until the target value is found. If the search ends with the remaining half being empty, the target is not in the array.

Normal distribution Probability distribution

In probability theory, a normaldistribution is a type of continuous probability distribution for a real-valued random variable. The general form of its probability density function is

Quantile Statistical method of dividing data into equal-sized intervals for analysis

In statistics and probability, quantiles are cut points dividing the range of a probability distribution into continuous intervals with equal probabilities, or dividing the observations in a sample in the same way. There is one fewer quantile than the number of groups created. Common quantiles have special names, such as quartiles, deciles, and percentiles. The groups created are termed halves, thirds, quarters, etc., though sometimes the terms for the quantile are used for the groups created, rather than for the cut points.

Triangle wave

A triangular wave or triangle wave is a non-sinusoidal waveform named for its triangular shape. It is a periodic, piecewise linear, continuous real function.

Floor and ceiling functions Mathematical functions rounding a number to the two closest integers

In mathematics and computer science, the floor function is the function that takes as input a real number x, and gives as output the greatest integer less than or equal to x, denoted floor(x) or x. Similarly, the ceiling function maps x to the least integer greater than or equal to x, denoted ceil(x) or x.

Students <i>t</i>-distribution Probability distribution

In probability and statistics, Student's t-distribution is any member of a family of continuous probability distributions that arise when estimating the mean of a normally-distributed population in situations where the sample size is small and the population's standard deviation is unknown. It was developed by English statistician William Sealy Gosset under the pseudonym "Student".

Golomb coding is a lossless data compression method using a family of data compression codes invented by Solomon W. Golomb in the 1960s. Alphabets following a geometric distribution will have a Golomb code as an optimal prefix code, making Golomb coding highly suitable for situations in which the occurrence of small values in the input stream is significantly more likely than large values.

In computer science, the Akra–Bazzi method, or Akra–Bazzi theorem, is used to analyze the asymptotic behavior of the mathematical recurrences that appear in the analysis of divide and conquer algorithms where the sub-problems have substantially different sizes. It is a generalization of the master theorem for divide-and-conquer recurrences, which assumes that the sub-problems have equal size. It is named after mathematicians Mohamad Akra and Louay Bazzi.

Spearmans rank correlation coefficient Nonparametric measure of rank correlation

In statistics, Spearman's rank correlation coefficient or Spearman's ρ, named after Charles Spearman and often denoted by the Greek letter (rho) or as , is a nonparametric measure of rank correlation. It assesses how well the relationship between two variables can be described using a monotonic function.

In number theory, a formula for primes is a formula generating the prime numbers, exactly and without exception. No such formula which is efficiently computable is known. A number of constraints are known, showing what such a "formula" can and cannot be.

In statistical inference, specifically predictive inference, a prediction interval is an estimate of an interval in which a future observation will fall, with a certain probability, given what has already been observed. Prediction intervals are often used in regression analysis.

In number theory, the integer square root (isqrt) of a positive integer n is the positive integer m which is the greatest integer less than or equal to the square root of n,

In mathematics, a pairing function is a process to uniquely encode two natural numbers into a single natural number.

In computing, the modulo operation returns the remainder or signed remainder of a division, after one number is divided by another.

Bakers map

In dynamical systems theory, the baker's map is a chaotic map from the unit square into itself. It is named after a kneading operation that bakers apply to dough: the dough is cut in half, and the two halves are stacked on one another, and compressed.

Conway–Maxwell–Poisson distribution

In probability theory and statistics, the Conway–Maxwell–Poisson distribution is a discrete probability distribution named after Richard W. Conway, William L. Maxwell, and Siméon Denis Poisson that generalizes the Poisson distribution by adding a parameter to model overdispersion and underdispersion. It is a member of the exponential family, has the Poisson distribution and geometric distribution as special cases and the Bernoulli distribution as a limiting case.

Dedekind number Combinatorial sequence of numbers

In mathematics, the Dedekind numbers are a rapidly growing sequence of integers named after Richard Dedekind, who defined them in 1897. The Dedekind number M(n) counts the number of monotone boolean functions of n variables. Equivalently, it counts the number of antichains of subsets of an n-element set, the number of elements in a free distributive lattice with n generators, or the number of abstract simplicial complexes with n elements.

In mathematics and computer science, the binary Goppa code is an error-correcting code that belongs to the class of general Goppa codes originally described by Valerii Denisovich Goppa, but the binary structure gives it several mathematical advantages over non-binary variants, also providing a better fit for common usage in computers and telecommunication. Binary Goppa codes have interesting properties suitable for cryptography in McEliece-like cryptosystems and similar setups.

In stochastic analysis, a rough path is a generalization of the notion of smooth path allowing to construct a robust solution theory for controlled differential equations driven by classically irregular signals, for example a Wiener process. The theory was developed in the 1990s by Terry Lyons. Several accounts of the theory are available.

References

  1. Percentile Symbol - does it exist or not?, Mathematics Stack Exchange
  2. Johnson, Robert; Kuby, Patricia (2007), "Applied Example 2.15, The 85th Percentile Speed Limit: Going With 85% of the Flow", Elementary Statistics (10th ed.), Cengage Learning, p. 102, ISBN   9781111802493 .
  3. "Rational Speed Limits and the 85th Percentile Speed" (PDF). lsp.org. Louisiana State Police. Archived from the original (PDF) on 23 September 2018. Retrieved 28 October 2018.
  4. 1 2 Hyndman RH, Fan Y (1996). "Sample quantiles in statistical packages". The American Statistician. 50 (4): 361–365. doi:10.2307/2684934. JSTOR   2684934.
  5. Lane, David. "Percentiles" . Retrieved 2007-09-15.
  6. 1 2 Pottel, Hans. "Statistical flaws in Excel" (PDF). Archived from the original (PDF) on 2013-06-04. Retrieved 2013-03-25.
  7. Schoonjans F, De Bacquer D, Schmid P (2011). "Estimation of population percentiles". Epidemiology. 22 (5): 750–751. doi:10.1097/EDE.0b013e318225c1de. PMC   3171208 . PMID   21811118.
  8. Wessa, P (2021). "Percentiles in Free Statistics Software". Office for Research Development and Education. Retrieved 13 November 2021.
  9. 1 2 3 "Engineering Statistics Handbook: Percentile". NIST . Retrieved 2009-02-18.
  10. "Matlab Statistics Toolbox – Percentiles" . Retrieved 2006-09-15., This is equivalent to Method 5 discussed here
  11. Langford, E. (2006). "Quartiles in Elementary Statistics". Journal of Statistics Education. 14 (3). doi: 10.1080/10691898.2006.11910589 .
  12. "NumPy 1.12 documentation". SciPy . Retrieved 2017-03-19.