False positives and false negatives

Last updated

A false positive is an error in binary classification in which a test result incorrectly indicates the presence of a condition (such as a disease when the disease is not present), while a false negative is the opposite error, where the test result incorrectly indicates the absence of a condition when it is actually present. These are the two kinds of errors in a binary test, in contrast to the two kinds of correct result (a true positive and a true negative). They are also known in medicine as a false positive (or false negative) diagnosis, and in statistical classification as a false positive (or false negative) error. [1]

Contents

In statistical hypothesis testing, the analogous concepts are known as type I and type II errors, where a positive result corresponds to rejecting the null hypothesis, and a negative result corresponds to not rejecting the null hypothesis. The terms are often used interchangeably, but there are differences in detail and interpretation due to the differences between medical testing and statistical hypothesis testing.

False positive error

A false positive error, or false positive, is a result that indicates a given condition exists when it does not. For example, a pregnancy test which indicates a woman is pregnant when she is not, or the conviction of an innocent person. [2]

A false positive error is a type I error where the test is checking a single condition, and wrongly gives an affirmative (positive) decision. However it is important to distinguish between the type 1 error rate and the probability of a positive result being false. The latter is known as the false positive risk (see Ambiguity in the definition of false positive rate, below). [3]

False negative error

A false negative error, or false negative, is a test result which wrongly indicates that a condition does not hold. For example, when a pregnancy test indicates a woman is not pregnant, but she is, or when a person guilty of a crime is acquitted, these are false negatives. The condition "the woman is pregnant", or "the person is guilty" holds, but the test (the pregnancy test or the trial in a court of law) fails to realize this condition, and wrongly decides that the person is not pregnant or not guilty. [4]

A false negative error is a type II error occurring in a test where a single condition is checked for, and the result of the test is erroneous, that the condition is absent. [5]

False positive and false negative rates

The false positive rate (FPR) is the proportion of all negatives that still yield positive test outcomes, i.e., the conditional probability of a positive test result given an event that was not present. [6] The false positive rate depends on the significance level. The specificity of the test is equal to 1 minus the false positive rate. [7]

In statistical hypothesis testing, this fraction is given the Greek letter α, and 1 − α is defined as the specificity of the test. Increasing the specificity of the test lowers the probability of type I errors, but may raise the probability of type II errors (false negatives that reject the alternative hypothesis when it is true). [lower-alpha 1]

Complementarily, the false negative rate (FNR) is the proportion of positives which yield negative test outcomes with the test, i.e., the conditional probability of a negative test result given that the condition being looked for is present. [8]

In statistical hypothesis testing, this fraction is given the letter β. The "power" (or the "sensitivity") of the test is equal to 1 − β.

Ambiguity in the definition of false positive rate

The term false discovery rate (FDR) was used by Colquhoun (2014) [9] to mean the probability that a "significant" result was a false positive. Later Colquhoun (2017) [3] used the term false positive risk (FPR) for the same quantity, to avoid confusion with the term FDR as used by people who work on multiple comparisons. Corrections for multiple comparisons aim only to correct the type I error rate, so the result is a (corrected) p-value. Thus they are susceptible to the same misinterpretation as any other p-value. The false positive risk is always higher, often much higher, than the p-value. [9] [3]

Confusion of these two ideas, the error of the transposed conditional, has caused much mischief. [10] Because of the ambiguity of notation in this field, it is essential to look at the definition in every paper. The hazards of reliance on p-values was emphasized in Colquhoun (2017) [3] by pointing out that even an observation of p = 0.001 was not necessarily strong evidence against the null hypothesis. Despite the fact that the likelihood ratio in favor of the alternative hypothesis over the null is close to 100, if the hypothesis was implausible, with a prior probability of a real effect being 0.1, even the observation of p = 0.001 would have a false positive rate of 8 percent. It wouldn't even reach the 5 percent level. As a consequence, it has been recommended [3] [11] that every p-value should be accompanied by the prior probability of there being a real effect that it would be necessary to assume in order to achieve a false positive risk of 5%. For example, if we observe p = 0.05 in a single experiment, we would have to be 87% certain that there was a real effect before the experiment was done to achieve a false positive risk of 5%.

Receiver operating characteristic

The article "Receiver operating characteristic" discusses parameters in statistical signal processing based on ratios of errors of various types.

See also

Notes

  1. ↑ When developing detection algorithms or tests, a balance must be chosen between risks of false negatives and false positives. Usually there is a threshold of how close a match to a given sample must be achieved before the algorithm reports a match. The higher this threshold, the more false negatives and the fewer false positives.

Related Research Articles

<span class="mw-page-title-main">Statistical hypothesis test</span> Method of statistical inference

A statistical hypothesis test is a method of statistical inference used to decide whether the data sufficiently support a particular hypothesis. A statistical hypothesis test typically involves a calculation of a test statistic. Then a decision is made, either by comparing the test statistic to a critical value or equivalently by evaluating a p-value computed from the test statistic. Roughly 100 specialized statistical tests have been defined.

Binary classification is the task of classifying the elements of a set into one of two groups. Typical binary classification problems include:

In statistics, the power of a binary hypothesis test is the probability that the test correctly rejects the null hypothesis when a specific alternative hypothesis is true. It is commonly denoted by , and represents the chances of a true positive detection conditional on the actual existence of an effect to detect. Statistical power ranges from 0 to 1, and as the power of a test increases, the probability of making a type II error by wrongly failing to reject the null hypothesis decreases.

In null-hypothesis significance testing, the -value is the probability of obtaining test results at least as extreme as the result actually observed, under the assumption that the null hypothesis is correct. A very small p-value means that such an extreme observed outcome would be very unlikely under the null hypothesis. Even though reporting p-values of statistical tests is common practice in academic publications of many quantitative fields, misinterpretation and misuse of p-values is widespread and has been a major topic in mathematics and metascience. In 2016, the American Statistical Association (ASA) made a formal statement that "p-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone" and that "a p-value, or statistical significance, does not measure the size of an effect or the importance of a result" or "evidence regarding a model or hypothesis". That said, a 2019 task force by ASA has issued a statement on statistical significance and replicability, concluding with: "p-values and significance tests, when properly applied and interpreted, increase the rigor of the conclusions drawn from data".

In statistics, the term "error" arises in two ways. Firstly, it arises in the context of decision making, where the probability of error may be considered as being the probability of making a wrong decision and which would have a different value for each type of error. Secondly, it arises in the context of statistical modelling where the model's predicted value may be in error regarding the observed outcome and where the term probability of error may refer to the probabilities of various amounts of error occurring.

<span class="mw-page-title-main">Receiver operating characteristic</span> Diagnostic plot of binary classifier ability

A receiver operating characteristic curve, or ROC curve, is a graphical plot that illustrates the performance of a binary classifier model at varying threshold values.

<span class="mw-page-title-main">Data dredging</span> Misuse of data analysis

Data dredging is the misuse of data analysis to find patterns in data that can be presented as statistically significant, thus dramatically increasing and understating the risk of false positives. This is done by performing many statistical tests on the data and only reporting those that come back with significant results.

In statistics, the false discovery rate (FDR) is a method of conceptualizing the rate of type I errors in null hypothesis testing when conducting multiple comparisons. FDR-controlling procedures are designed to control the FDR, which is the expected proportion of "discoveries" that are false. Equivalently, the FDR is the expected ratio of the number of false positive classifications to the total number of positive classifications. The total number of rejections of the null include both the number of false positives (FP) and true positives (TP). Simply put, FDR = FP /. FDR-controlling procedures provide less stringent control of Type I errors compared to family-wise error rate (FWER) controlling procedures, which control the probability of at least one Type I error. Thus, FDR-controlling procedures have greater power, at the cost of increased numbers of Type I errors.

In statistics, family-wise error rate (FWER) is the probability of making one or more false discoveries, or type I errors when performing multiple hypotheses tests.

<span class="mw-page-title-main">Sensitivity and specificity</span> Statistical measure of a binary classification

In medicine and statistics, sensitivity and specificity mathematically describe the accuracy of a test that reports the presence or absence of a medical condition. If individuals who have the condition are considered "positive" and those who do not are considered "negative", then sensitivity is a measure of how well a test can identify true positives and specificity is a measure of how well a test can identify true negatives:

In statistical hypothesis testing, a type I error, or a false positive, is the rejection of the null hypothesis when it is actually true. For example, an innocent person may be convicted. A type II error, or a false negative, is the failure to reject a null hypothesis that is actually false. For example: a guilty person may be not convicted.

<span class="mw-page-title-main">Multiple comparisons problem</span> Statistical interpretation with many tests

In statistics, the multiple comparisons, multiplicity or multiple testing problem occurs when one considers a set of statistical inferences simultaneously or estimates a subset of parameters selected based on the observed values.

Medical statistics deals with applications of statistics to medicine and the health sciences, including epidemiology, public health, forensic medicine, and clinical research. Medical statistics has been a recognized branch of statistics in the United Kingdom for more than 40 years, but the term has not come into general use in North America, where the wider term 'biostatistics' is more commonly used. However, "biostatistics" more commonly connotes all applications of statistics to biology. Medical statistics is a subdiscipline of statistics.

It is the science of summarizing, collecting, presenting and interpreting data in medical practice, and using them to estimate the magnitude of associations and test hypotheses. It has a central role in medical investigations. It not only provides a way of organizing information on a wider and more formal basis than relying on the exchange of anecdotes and personal experience, but also takes into account the intrinsic variation inherent in most biological processes.

<span class="mw-page-title-main">Precision and recall</span> Pattern-recognition performance metrics

In pattern recognition, information retrieval, object detection and classification, precision and recall are performance metrics that apply to data retrieved from a collection, corpus or sample space.

<span class="mw-page-title-main">David Colquhoun</span> British pharmacologist (born 1936)

David Colquhoun is a British pharmacologist at University College London (UCL). He has contributed to the general theory of receptor and synaptic mechanisms, and in particular the theory and practice of single ion channel function. He held the A.J. Clark chair of Pharmacology at UCL from 1985 to 2004, and was the Hon. Director of the Wellcome Laboratory for Molecular Pharmacology. He was elected a Fellow of the Royal Society (FRS) in 1985 and an honorary fellow of UCL in 2004. Colquhoun runs the website DC's Improbable Science, which is critical of pseudoscience, particularly alternative medicine, and managerialism.

In statistics, when performing multiple comparisons, a false positive ratio is the probability of falsely rejecting the null hypothesis for a particular test. The false positive rate is calculated as the ratio between the number of negative events wrongly categorized as positive and the total number of actual negative events.

In statistics, a false coverage rate (FCR) is the average rate of false coverage, i.e. not covering the true parameters, among the selected intervals.

<span class="mw-page-title-main">Evaluation of binary classifiers</span> Statistics and machine learning concept

Evaluation of a binary classifier assigns a numerical value, or values, to a classifier that represent its accuracy. An example is error rate, which measures how frequently the classifier makes a mistake.

Misuse of p-values is common in scientific research and scientific education. p-values are often used or interpreted incorrectly; the American Statistical Association states that p-values can indicate how incompatible the data are with a specified statistical model. From a Neyman–Pearson hypothesis testing approach to statistical inferences, the data obtained by comparing the p-value to a significance level will yield one of two results: either the null hypothesis is rejected, or the null hypothesis cannot be rejected at that significance level. From a Fisherian statistical testing approach to statistical inferences, a low p-value means either that the null hypothesis is true and a highly improbable event has occurred or that the null hypothesis is false.

<i>q</i>-value (statistics) Statistical hypothesis testing measure

In statistical hypothesis testing, specifically multiple hypothesis testing, the q-value in the Storey procedure provides a means to control the positive false discovery rate (pFDR). Just as the p-value gives the expected false positive rate obtained by rejecting the null hypothesis for any result with an equal or smaller p-value, the q-value gives the expected pFDR obtained by rejecting the null hypothesis for any result with an equal or smaller q-value.

References

  1. ↑ False Positives and False Negatives
  2. ↑ Robinson A, Keller LR, del Campo C. Building insights on true positives vs. false positives: Bayes’ rule. Decision Sciences Journal of Innovative Education. 2022;20(4):224-234. doi:10.1111/dsji.12265
  3. 1 2 3 4 5 Colquhoun, David (2017). "The reproducibility of research and the misinterpretation of p-values". Royal Society Open Science. 4 (12): 171085. doi:10.1098/rsos.171085. PMC   5750014 . PMID   29308247.
  4. ↑ Alnabulsi, Hussein; Islam, Rafiqui; Mamun, Qasi (2018). "A novel algorithm to protect code injection attacks". In Abawajy, Jemal H.; Choo, Kim-Kwang Raymond; Islam, Rafiqul (eds.). International Conference on Applications and Techniques in Cyber Security and Intelligence: Applications and Techniques in Cyber Security and Intelligence. Cham, Switzerland: Springer International Publishing. p. 288. ISBN   978-3-31967-071-3.
  5. ↑ Banerjee, A; Chitnis, UB; Jadhav, SL; Bhawalkar, JS; Chaudhury, S (2009). "Hypothesis testing, type I and type II errors". Ind Psychiatry J. 18 (2): 127–31. doi: 10.4103/0972-6748.62274 . PMC   2996198 . PMID   21180491.
  6. ↑ Bose, Prosenjit; Guo, Hua; Kranakis, Evangelos; Maheshwari, Anil; Morin, Pat; Morrison, Jason; Smid, Michiel; Tang, Yihui (2008). "On the false-positive rate of Bloom filters". Information Processing Letters. 108 (4): 210. doi:10.1016/j.ipl.2008.05.018.
  7. ↑ Cronin, Paul; Kelly, Aine Marie (2011). "Influence of population prevalences on numbers of false positives: an overlooked entity". Academic Radiology. 18 (9): 1088. doi:10.1016/j.acra.2011.04.011.
  8. ↑ Cronin & Kelly, 2011, p.1087
  9. 1 2 Colquhoun, David (2014). "An investigation of the false discovery rate and the misinterpretation of p-values". Royal Society Open Science. 1 (3): 140216. arXiv: 1407.5296 . Bibcode:2014RSOS....140216C. doi:10.1098/rsos.140216. PMC   4448847 . PMID   26064558.
  10. ↑ Colquhoun, David. "The problem with p-values". Aeon. Aeon Magazine. Retrieved 11 December 2016.
  11. ↑ Colquhoun, David (2018). "The false positive risk: A proposal concerning what to do about p values". The American Statistician. 73: 192–201. arXiv: 1802.04888 . doi:10.1080/00031305.2018.1529622. S2CID   85530643.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.