Brier score

Last updated December 05, 2024

The Brier score is a strictly proper scoring rule that measures the accuracy of probabilistic predictions. For unidimensional predictions, it is strictly equivalent to the mean squared error as applied to predicted probabilities.

The Brier score is applicable to tasks in which predictions must assign probabilities to a set of mutually exclusive discrete outcomes or classes. The set of possible outcomes can be either binary or categorical in nature, and the probabilities assigned to this set of outcomes must sum to one (where each individual probability is in the range of 0 to 1). It was proposed by Glenn W. Brier in 1950.^[1]

The Brier score can be thought of as a cost function. More precisely, across all items $i\in {1...N}$ in a set of N predictions, the Brier score measures the mean squared difference between:

The predicted probability assigned to the possible outcomes for item i
The actual outcome $o_{i}$

Therefore, the lower the Brier score is for a set of predictions, the better the predictions are calibrated. Note that the Brier score, in its most common formulation, takes on a value between zero and one, since this is the square of the largest possible difference between a predicted probability (which must be between zero and one) and the actual outcome (which can take on values of only 0 or 1). In the original (1950) formulation of the Brier score, the range is double, from zero to two.

The Brier score is appropriate for binary and categorical outcomes that can be structured as true or false, but it is inappropriate for ordinal variables which can take on three or more values.

Definition

The most common formulation of the Brier score is

BS={\frac {1}{N}}\sum \limits _{t=1}^{N}(f_{t}-o_{t})^{2}\,\!

in which $f_{t}$ is the probability that was forecast, $o_{t}$ the actual outcome of the event at instance $t$ ( $0$ if it does not happen and $1$ if it does happen) and $N$ is the number of forecasting instances. In effect, it is the mean squared error of the forecast. This formulation is mostly used for binary events (for example "rain" or "no rain"). The above equation is a proper scoring rule only for binary events; if a multi-category forecast is to be evaluated, then the original definition given by Brier below should be used.

Example

Suppose that one is forecasting the probability $P$ that it will rain on a given day. Then the Brier score is calculated as follows:

If the forecast is 100% ( $P$ = 1) and it rains, then the Brier Score is 0, the best score achievable.
If the forecast is 100% and it does not rain, then the Brier Score is 1, the worst score achievable.
If the forecast is 70% ( $P$ = 0.70) and it rains, then the Brier Score is (0.70−1)² = 0.09.

In contrast, if the forecast is 70% ( $P$ = 0.70) and it does not rain, then the Brier Score is (0.70−0)² = 0.49.
Similarly, if the forecast is 30% ( $P$ = 0.30) and it rains, then the Brier Score is (0.30−1)² = 0.49.
If the forecast is 50% ( $P$ = 0.50), then the Brier score is (0.50−1)² = (0.50−0)² = 0.25, regardless of whether it rains.

Original definition by Brier

Although the above formulation is the most widely used, the original definition by Brier^[1] is applicable to multi-category forecasts as well as it remains a proper scoring rule, while the binary form (as used in the examples above) is only proper for binary events. For binary forecasts, the original formulation of Brier's "probability score" has twice the value of the score currently known as the Brier score.

BS={\frac {1}{N}}\sum \limits _{t=1}^{N}\sum \limits _{i=1}^{R}(f_{ti}-o_{ti})^{2}\,\!

In which $R$ is the number of possible classes in which the event can fall, and $N$ the overall number of instances of all classes. $f_{ti}$ is the predicted probability for class $i.o_{ti}$ is $1$ if it is $i$ -th class in instance $t$ ; $0$ , otherwise. For the case Rain / No rain, $R=2$ , while for the forecast Cold / Normal / Warm, $R=3$ .

Decompositions

There are several decompositions of the Brier score which provide a deeper insight on the behavior of a binary classifier.

3-component decomposition

The Brier score can be decomposed into 3 additive components: Uncertainty, Reliability, and Resolution. (Murphy 1973)^[2]

BS=REL-RES+UNC

Each of these components can be decomposed further according to the number of possible classes in which the event can fall. Abusing the equality sign:

BS={\frac {1}{N}}\sum \limits _{k=1}^{K}{n_{k}(\mathbf {f_{k}} -\mathbf {\bar {o}} _{\mathbf {k} })}^{2}-{\frac {1}{N}}\sum \limits _{k=1}^{K}{n_{k}(\mathbf {{\bar {o}}_{k}} -{\bar {\mathbf {o} }})}^{2}+\mathbf {\bar {o}} \left({1-\mathbf {\bar {o}} }\right)

With $\textstyle N$ being the total number of forecasts issued, $\textstyle K$ the number of unique forecasts issued, $\mathbf {\bar {o}} ={\sum _{t=1}^{N}}\mathbf {o_{t}} /N$ the observed climatological base rate for the event to occur, $n_{k}$ the number of forecasts with the same probability category and $\mathbf {\overline {o}} _{\mathbf {k} }$ the observed frequency, given forecasts of probability $\mathbf {f_{k}}$ . The bold notation in the above formula indicates vectors, which is another way of denoting the original definition of the score and decomposing it according to the number of possible classes in which the event can fall. For example, a 70% chance of rain and an occurrence of no rain are denoted as $\mathbf {f} =(0.3,0.7)$ and $\mathbf {o} =(1,0)$ respectively. Operations like the square and multiplication on these vectors are understood to be component wise. The Brier Score is then the sum of the resulting vector on the right hand side.

Reliability

The reliability term measures how close the forecast probabilities are to the true probabilities, given that forecast. Reliability is defined in the contrary direction compared to English language. If the reliability is 0, the forecast is perfectly reliable. For example, if we group all forecast instances where 80% chance of rain was forecast, we get a perfect reliability only if it rained 4 out of 5 times after such a forecast was issued.

Resolution

The resolution term measures how much the conditional probabilities given by the different forecasts differ from the climatic average. The higher this term is, the better. In the worst case, when the climatic probability is always forecast, the resolution is zero. In the best case, when the conditional probabilities are zero and one, the resolution is equal to the uncertainty.

Uncertainty

The uncertainty term measures the inherent uncertainty in the outcomes of the event. For binary events, it is at a maximum when each outcome occurs 50% of the time, and is minimal (zero) if an outcome always occurs or never occurs.

Two-component decomposition

An alternative (and related) decomposition generates two terms instead of three.

BS=CAL+REF

BS={\frac {1}{N}}\sum \limits _{k=1}^{K}{n_{k}(\mathbf {f_{k}} -\mathbf {\bar {o}} _{\mathbf {k} })}^{2}+{\frac {1}{N}}\sum \limits _{k=1}^{K}{n_{k}(\mathbf {{\bar {o}}_{k}} (1-\mathbf {{\bar {o}}_{k}} }))

The first term is known as calibration (and can be used as a measure of calibration, see statistical calibration), and is equal to reliability. The second term is known as refinement, and it is an aggregation of resolution and uncertainty, and is related to the area under the ROC Curve.

The Brier Score, and the CAL + REF decomposition, can be represented graphically through the so-called Brier Curves,^[3] where the expected loss is shown for each operating condition. This makes the Brier Score a measure of aggregated performance under a uniform distribution of class asymmetries.^[4]

Brier skill score (BSS)

A skill score for a given underlying score is an offset and (negatively-) scaled variant of the underlying score such that a skill score value of zero means that the score for the predictions is merely as good as that of a set of baseline or reference or default predictions, while a skill score value of one (100%) represents the best possible score. A skill score value less than zero means that the performance is even worse than that of the baseline or reference predictions. When the underlying score is the Brier score (BS), the Brier skill score (BSS) is calculated as

BSS=1-{\frac {BS}{BS_{ref}}}

where $BS_{ref}$ is the Brier score of reference or baseline predictions which we seek to improve on. While the reference predictions could in principle be given by any pre-existing model, by default one can use the naïve model that predicts the overall proportion or frequency of a given class in the data set being scored, as the constant predicted probability of that class occurring in each instance in the data set. This baseline model would represent a "no skill" model that one seeks to improve on. Skill scores originate in the meteorological prediction literature, where the naïve default reference predictions are called the "in-sample climatology" predictions, where climatology means a long-term or overall average of weather predictions, and in-sample means as calculated from the present data set being scored.^[5]^[6] In this default case, for binary (two-class) classification, the reference Brier score is given by (using the notation of the first equation of this article, at the top of the Definition section):

BS_{ref}={\frac {1}{N}}\sum \limits _{t=1}^{N}({\bar {o}}-o_{t})^{2}\,

where ${\bar {o}}$ is simply the average actual outcome, i.e. the overall proportion of true class 1 in the data set:

{\bar {o}}={\frac {1}{N}}\sum \limits _{t=1}^{N}o_{t}.

With a Brier score, lower is better (it is a loss function) with 0 being the best possible score. But with a Brier skill score, higher is better with 1 (100%) being the best possible score.

The Brier skill score can be more interpretable than the Brier score because the BSS is simply the percentage improvement in the BS compared to the reference model, and a negative BSS means you are doing even worse than the reference model, which may not be obvious from looking at the Brier score itself. However, a BSS near 100% should not typically be expected because this would require that every probability prediction was nearly 0 or 1 (and was correct of course).

Even if the Brier score is a strictly proper scoring rule, the BSS is not strictly proper: indeed, skill scores are generally non-proper even if the underlying scoring rule is proper.^[7] Still, Murphy (1973)^[8] proved that the BSS is asymptotically proper with a large number of samples.

You might notice that classification's (probability estimation's) BSS is to its BS, as regression's coefficient of determination ( $R^{2}$ ) is to its mean squared error (MSE).

Shortcomings

The Brier score becomes inadequate for very rare (or very frequent) events, because it does not sufficiently discriminate between small changes in forecast that are significant for rare events.^[9] Wilks (2010) has found that "[Q]uite large sample sizes, i.e. n > 1000, are required for higher-skill forecasts of relatively rare events, whereas only quite modest sample sizes are needed for low-skill forecasts of common events."^[10]

Related Research Articles

In probability theory and statistics, a normal distribution or Gaussian distribution is a type of continuous probability distribution for a real-valued random variable. The general form of its probability density function is $The parameter is the mean or expectation of the distribution, while the parameter is the variance. The standard deviation of the distribution is (sigma). A random variable with a Gaussian distribution is said to be normally distributed, and is called a normal deviate .$

<span class="mw-page-title-main">Central limit theorem</span> Fundamental theorem in probability theory and statistics

In probability theory, the central limit theorem (CLT) states that, under appropriate conditions, the distribution of a normalized version of the sample mean converges to a standard normal distribution. This holds even if the original variables themselves are not normally distributed. There are several versions of the CLT, each applying in the context of different conditions.

A likelihood function measures how well a statistical model explains observed data by calculating the probability of seeing that data under different parameter values of the model. It is constructed from the joint probability distribution of the random variable that (presumably) generated the observations. When evaluated on the actual data points, it becomes a function solely of the model parameters.

In statistics, the logistic model is a statistical model that models the log-odds of an event as a linear combination of one or more independent variables. In regression analysis, logistic regression estimates the parameters of a logistic model. In binary logistic regression there is a single binary dependent variable, coded by an indicator variable, where the two values are labeled "0" and "1", while the independent variables can each be a binary variable or a continuous variable. The corresponding probability of the value labeled "1" can vary between 0 and 1, hence the labeling; the function that converts log-odds to probability is the logistic function, hence the name. The unit of measurement for the log-odds scale is called a logit, from logistic unit, hence the alternative names. See § Background and § Definition for formal mathematics, and § Example for a worked example.

In probability theory, the multinomial distribution is a generalization of the binomial distribution. For example, it models the probability of counts for each side of a k-sided dice rolled n times. For n independent trials each of which leads to a success for exactly one of k categories, with each category having a given fixed success probability, the multinomial distribution gives the probability of any particular combination of numbers of successes for the various categories.

Cohen's kappa coefficient is a statistic that is used to measure inter-rater reliability for qualitative (categorical) items. It is generally thought to be a more robust measure than simple percent agreement calculation, as κ takes into account the possibility of the agreement occurring by chance. There is controversy surrounding Cohen's kappa due to the difficulty in interpreting indices of agreement. Some researchers have suggested that it is conceptually simpler to evaluate disagreement between items.

The Jaccard index is a statistic used for gauging the similarity and diversity of sample sets. It is defined in general taking the ratio of two sizes, the intersection size divided by the union size, also called intersection over union (IoU).

<span class="mw-page-title-main">Scoring rule</span> Measure for evaluating probabilistic forecasts

In decision theory, a scoring rule provides evaluation metrics for probabilistic predictions or forecasts. While "regular" loss functions assign a goodness-of-fit score to a predicted value and an observed value, scoring rules assign such a score to a predicted probability distribution and an observed value. On the other hand, a scoring function provides a summary measure for the evaluation of point predictions, i.e. one predicts a property or functional $, like the expectation or the median.$

In statistics, multinomial logistic regression is a classification method that generalizes logistic regression to multiclass problems, i.e. with more than two possible discrete outcomes. That is, it is a model that is used to predict the probabilities of the different possible outcomes of a categorically distributed dependent variable, given a set of independent variables.

The softmax function, also known as softargmax or normalized exponential function, converts a vector of $K$ real numbers into a probability distribution of $K$ possible outcomes. It is a generalization of the logistic function to multiple dimensions, and is used in multinomial logistic regression. The softmax function is often used as the last activation function of a neural network to normalize the output of a network to a probability distribution over predicted output classes.

The ensemble Kalman filter (EnKF) is a recursive filter suitable for problems with a large number of variables, such as discretizations of partial differential equations in geophysical models. The EnKF originated as a version of the Kalman filter for large problems, and it is now an important data assimilation component of ensemble forecasting. EnKF is related to the particle filter but the EnKF makes the assumption that all probability distributions involved are Gaussian; when it is applicable, it is much more efficient than the particle filter.

In probability theory and statistics, a categorical distribution is a discrete probability distribution that describes the possible results of a random variable that can take on one of K possible categories, with the probability of each category separately specified. There is no innate underlying ordering of these outcomes, but numerical labels are often attached for convenience in describing the distribution,. The K-dimensional categorical distribution is the most general distribution over a K-way event; any other discrete distribution over a size-K sample space is a special case. The parameters specifying the probabilities of each possible outcome are constrained only by the fact that each must be in the range 0 to 1, and all must sum to 1.

<span class="mw-page-title-main">Precision and recall</span> Pattern-recognition performance metrics

In pattern recognition, information retrieval, object detection and classification, precision and recall are performance metrics that apply to data retrieved from a collection, corpus or sample space.

In statistics, the phi coefficient is a measure of association for two binary variables.

In statistics and combinatorial mathematics, group testing is any procedure that breaks up the task of identifying certain objects into tests on groups of items, rather than on individual ones. First studied by Robert Dorfman in 1943, group testing is a relatively new field of applied mathematics that can be applied to a wide range of practical applications and is an active area of research today.

Statistical Football prediction is a method used in sports betting, to predict the outcome of football matches by means of statistical tools. The goal of statistical match prediction is to outperform the predictions of bookmakers, who use them to set odds on the outcome of football matches.

The system size expansion, also known as van Kampen's expansion or the Ω-expansion, is a technique pioneered by Nico van Kampen used in the analysis of stochastic processes. Specifically, it allows one to find an approximation to the solution of a master equation with nonlinear transition rates. The leading order term of the expansion is given by the linear noise approximation, in which the master equation is approximated by a Fokker–Planck equation with linear coefficients determined by the transition rates and stoichiometry of the system.

In information theory, Shannon–Fano–Elias coding is a precursor to arithmetic coding, in which probabilities are used to determine codewords. It is named for Claude Shannon, Robert Fano, and Peter Elias.

Mean directional accuracy (MDA), also known as mean direction accuracy, is a measure of prediction accuracy of a forecasting method in statistics. It compares the forecast direction to the actual realized direction. It is defined by the following formula:

The hyperbolastic functions, also known as hyperbolastic growth models, are mathematical functions that are used in medical statistical modeling. These models were originally developed to capture the growth dynamics of multicellular tumor spheres, and were introduced in 2005 by Mohammad Tabatabai, David Williams, and Zoran Bursac. The precision of hyperbolastic functions in modeling real world problems is somewhat due to their flexibility in their point of inflection. These functions can be used in a wide variety of modeling problems such as tumor growth, stem cell proliferation, pharma kinetics, cancer growth, sigmoid activation function in neural networks, and epidemiological disease progression or regression.

References

1 2 Brier (1950). "Verification of Forecasts Expressed in Terms of Probability" (PDF). Monthly Weather Review. 78 (1): 1–3. Bibcode:1950MWRv...78....1B. doi:10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2. S2CID 122906757. Archived from the original (PDF) on 2017-10-23.
↑ Murphy, A. H. (1973). "A new vector partition of the probability score". Journal of Applied Meteorology. 12 (4): 595–600. Bibcode:1973JApMe..12..595M. doi: 10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2 .
↑ Hernandez-Orallo, J.; Flach, P.A.; Ferri, C. (2011). "Brier curves: a new cost-based visualisation of classifier performance" (PDF). Proceedings of the 28th International Conference on Machine Learning (ICML-11). pp. 585–592.
↑ Hernandez-Orallo, J.; Flach, P.A.; Ferri, C. (2012). "A unified view of performance metrics: translating threshold choice into expected classification loss" (PDF). Journal of Machine Learning Research. 13: 2813–2869.
↑ A bias-corrected decomposition of the Brier score. (Notes and Correspondence.) C. A. T. Ferro and T. E. Fricker in Quarterly Journal of the Royal Meteorological Society, Volume 138, Issue 668, October 2012 Part A, Pages 1954-1960
↑ "Numerical Weather Prediction: The MOGREPS short-range ensemble prediction system: Verification report: Trial Performance of MOGREPS: January 2006 - March 2007. Forecasting Research Technical Report No. 503." Neill Bowler, Marie Dando, Sarah Beare & Ken Mylne
↑ Gneiting, Tilmann; Raftery, Adrian E. (2007). "Strictly Proper Scoring Rules, Prediction, and Estimation" (PDF). Journal of the American Statistical Association. 102 (447): 359–378. doi:10.1198/016214506000001437. S2CID 1878582.
↑ Murphy, A. H. (1973). "Hedging and skill scores for probability forecasts". Journal of Applied Meteorology. 12: 215–223.
↑ Riccardo Benedetti (2010-01-01). "Scoring Rules for Forecast Verification". Monthly Weather Review. 138 (1): 203–211. Bibcode:2010MWRv..138..203B. doi: 10.1175/2009MWR2945.1 .
↑ Wilks, D. S. (2010). "Sampling distributions of the Brier score and Brier skill score under serial dependence". Quarterly Journal of the Royal Meteorological Society. 136 (1): 2109–2118. Bibcode:2010QJRMS.136.2109W. doi:10.1002/qj.709. S2CID 121504347.