Forecast verification

Last updated

Forecast verification is a subfield of the climate, atmospheric and ocean sciences dealing with validating, verifying and determining the predictive power of prognostic model forecasts. Because of the complexity of these models, forecast verification goes a good deal beyond simple measures of statistical association or mean error calculations.

Contents

Defining the problem

To determine the value of a forecast, we need to measure it against some baseline, or minimally accurate forecast. There are many types of forecast that, while producing impressive-looking skill scores, are nonetheless naive. A "persistence" forecast can still rival even those of the most sophisticated models. An example is: "What is the weather going to be like today? Same as it was yesterday." This could be considered analogous to a "control" experiment. Another example would be a climatological forecast: "What is the weather going to be like today? The same as it was, on average, for all the previous days this time of year for the past 75 years".

The second example suggests a good method of normalizing a forecast before applying any skill measure. Most weather situations will cycle, since the Earth is forced by a highly regular energy source. A numerical weather model must accurately model both the seasonal cycle and (if finely resolved enough) the diurnal cycle. This output, however, adds no information content, since the same cycles are easily predicted from climatological data. Climatological cycles may be removed from both the model output and the "truth" data. Thus, the skill score, applied afterward, is more meaningful.

One way of thinking about it is, "how much does the forecast reduce our uncertainty?"

Christensen et al. (1981) [1] used entropy minimax entropy minimax pattern discovery based on information theory to advance the science of long range weather prediction. Previous computer models of weather were based on persistence alone and reliable to only 5–7 days into the future. Long range forecasting was essentially random. Christensen et al. demonstrated the ability to predict the probability that precipitation will be below or above average with modest but statistically significant skill one, two and even three years into the future. Notably, this pioneering work discovered the influence of El Nino El Nino/Southern Oscillation (ENSO) on U.S. weather forecasting.

Tang et al. (2005) [2] used the conditional entropy to characterize the uncertainty of ensemble predictions of the El Nino/Southern Oscillation (ENSO):

where p is the ensemble distribution and q is the climatological distribution.

Further information

The World Meteorological Organization maintains a webpage on forecast verification. [3]

For more in-depth information on how to verify forecasts see the book by Jolliffe and Stephenson [4] or the book chapter by Daniel Wilks. [5]

Related Research Articles

<span class="mw-page-title-main">El Niño</span> Warm phase of a cyclic climatic phenomenon in the Pacific Ocean

El Niño is the warm phase of the El Niño–Southern Oscillation (ENSO) and is associated with a band of warm ocean water that develops in the central and east-central equatorial Pacific, including the area off the Pacific coast of South America. The ENSO is the cycle of warm and cold sea surface temperature (SST) of the tropical central and eastern Pacific Ocean.

<span class="mw-page-title-main">La Niña</span> Coupled ocean-atmosphere phenomenon that is the counterpart of El Niño

La Niña is an oceanic and atmospheric phenomenon that is the colder counterpart of El Niño, as part of the broader El Niño–Southern Oscillation (ENSO) climate pattern. The name La Niña originates from Spanish for "the girl", by analogy to El Niño, meaning "the boy". In the past, it was also called an anti-El Niño and El Viejo, meaning "the old man."

<span class="mw-page-title-main">Climatology</span> Scientific study of climate, defined as weather conditions averaged over a period of time

Climatology or climate science is the scientific study of Earth's climate, typically defined as weather conditions averaged over a period of at least 30 years. Climate concerns the atmospheric condition during an extended to indefinite period of time; weather is the condition of the atmosphere during a relative brief period of time. The main topics of research are the study of climate variability, mechanisms of climate changes and modern climate change. This topic of study is regarded as part of the atmospheric sciences and a subdivision of physical geography, which is one of the Earth sciences. Climatology includes some aspects of oceanography and biogeochemistry.

<span class="mw-page-title-main">El Niño–Southern Oscillation</span> Physical oceanography

El Niño–Southern Oscillation (ENSO) is an irregular periodic variation in winds and sea surface temperatures over the tropical eastern Pacific Ocean, affecting the climate of much of the tropics and subtropics. The warming phase of the sea temperature is known as El Niño and the cooling phase as La Niña. The Southern Oscillation is the accompanying atmospheric component, coupled with the sea temperature change: El Niño is accompanied by high air surface pressure in the tropical western Pacific and La Niña with low air surface pressure there. The two periods last several months each and typically occur every few years with varying intensity per period.

The Tropical Ocean Global Atmosphere program (TOGA) was a ten-year study (1985–1994) of the World Climate Research Programme (WCRP) aimed specifically at the prediction of climate phenomena on time scales of months to years.

<span class="mw-page-title-main">Pacific decadal oscillation</span> Recurring pattern of climate variability

The Pacific decadal oscillation (PDO) is a robust, recurring pattern of ocean-atmosphere climate variability centered over the mid-latitude Pacific basin. The PDO is detected as warm or cool surface waters in the Pacific Ocean, north of 20°N. Over the past century, the amplitude of this climate pattern has varied irregularly at interannual-to-interdecadal time scales. There is evidence of reversals in the prevailing polarity of the oscillation occurring around 1925, 1947, and 1977; the last two reversals corresponded with dramatic shifts in salmon production regimes in the North Pacific Ocean. This climate pattern also affects coastal sea and continental surface air temperatures from Alaska to California.

<span class="mw-page-title-main">Ensemble forecasting</span> Multiple simulation method for weather forecasting

Ensemble forecasting is a method used in or within numerical weather prediction. Instead of making a single forecast of the most likely weather, a set of forecasts is produced. This set of forecasts aims to give an indication of the range of possible future states of the atmosphere. Ensemble forecasting is a form of Monte Carlo analysis. The multiple simulations are conducted to account for the two usual sources of uncertainty in forecast models: (1) the errors introduced by the use of imperfect initial conditions, amplified by the chaotic nature of the evolution equations of the atmosphere, which is often referred to as sensitive dependence on initial conditions; and (2) errors introduced because of imperfections in the model formulation, such as the approximate mathematical methods to solve the equations. Ideally, the verified future atmospheric state should fall within the predicted ensemble spread, and the amount of spread should be related to the uncertainty (error) of the forecast. In general, this approach can be used to make probabilistic forecasts of any dynamical system, and not just for weather prediction.

The Brier Score is a strictly proper score function or strictly proper scoring rule that measures the accuracy of probabilistic predictions. For unidimensional predictions, it is strictly equivalent to the mean squared error as applied to predicted probabilities.

<span class="mw-page-title-main">Tropical cyclone forecast model</span> Computer program that uses meteorological data to forecast tropical cyclones

A tropical cyclone forecast model is a computer program that uses meteorological data to forecast aspects of the future state of tropical cyclones. There are three types of models: statistical, dynamical, or combined statistical-dynamic. Dynamical models utilize powerful supercomputers with sophisticated mathematical modeling software and meteorological data to calculate future weather conditions. Statistical models forecast the evolution of a tropical cyclone in a simpler manner, by extrapolating from historical datasets, and thus can be run quickly on platforms such as personal computers. Statistical-dynamical models use aspects of both types of forecasting. Four primary types of forecasts exist for tropical cyclones: track, intensity, storm surge, and rainfall. Dynamical models were not developed until the 1970s and the 1980s, with earlier efforts focused on the storm surge problem.

Upper-atmospheric models are simulations of the Earth's atmosphere between 20 and 100 km that comprises the stratosphere, mesosphere, and the lower thermosphere. Whereas most climate models simulate a region of the Earth's atmosphere from the surface to the stratopause, there also exist numerical models which simulate the wind, temperature and composition of the Earth's tenuous upper atmosphere, from the mesosphere to the exosphere, including the ionosphere. This region is affected strongly by the 11 year Solar cycle through variations in solar UV/EUV/Xray radiation and solar wind leading to high latitude particle precipitation and aurora. It has been proposed that these phenomena may have an effect on the lower atmosphere, and should therefore be included in simulations of climate change. For this reason there has been a drive in recent years to create whole atmosphere models to investigate whether or not this is the case.

<span class="mw-page-title-main">Tropical cyclogenesis</span> Development and strengthening of a tropical cyclone in the atmosphere

Tropical cyclogenesis is the development and strengthening of a tropical cyclone in the atmosphere. The mechanisms through which tropical cyclogenesis occurs are distinctly different from those through which temperate cyclogenesis occurs. Tropical cyclogenesis involves the development of a warm-core cyclone, due to significant convection in a favorable atmospheric environment.

In weather forecasting, model output statistics (MOS) is a multiple linear regression technique in which predictands, often near-surface quantities, are related statistically to one or more predictors. The predictors are typically forecasts from a numerical weather prediction (NWP) model, climatic data, and, if applicable, recent surface observations. Thus, output from NWP models can be transformed by the MOS technique into sensible weather parameters that are familiar to a layperson.

<span class="mw-page-title-main">Tropical cyclone forecasting</span> Science of forecasting how a tropical cyclone moves and its effects

Tropical cyclone forecasting is the science of forecasting where a tropical cyclone's center, and its effects, are expected to be at some point in the future. There are several elements to tropical cyclone forecasting: track forecasting, intensity forecasting, rainfall forecasting, storm surge, tornado, and seasonal forecasting. While skill is increasing in regard to track forecasting, intensity forecasting skill remains unchanged over the past several years. Seasonal forecasting began in the 1980s in the Atlantic basin and has spread into other basins in the years since.

Teleconnection in atmospheric science refers to climate anomalies being related to each other at large distances. The most emblematic teleconnection is that linking sea-level pressure at Tahiti and Darwin, Australia, which defines the Southern Oscillation. Another well-known teleconnection links the sea-level pressure over Iceland with the one over the Azores, traditionally defining the North Atlantic Oscillation (NAO).

<span class="mw-page-title-main">Global Energy and Water Exchanges</span>

The Global Energy and Water Exchanges Project is an international research project and a core project of the World Climate Research Programme (WCRP).

In the fields of forecasting and prediction, forecasting skill or prediction skill is any measure of the accuracy and/or degree of association of prediction to an observation or estimate of the actual value of what is being predicted ; it may be quantified as a skill score.

<span class="mw-page-title-main">Tuvalu Meteorological Service</span>

The Tuvalu Meteorological Service (TMS) is the principal meteorological observatory of Tuvalu and is responsible for providing weather services to the islands of Tuvalu. A meteorological office was established on Funafuti at the time the islands of Tuvalu were administered as parts of the Gilbert and Ellice Islands colony of the United Kingdom. The meteorological office is now an agency of the government of Tuvalu.

In the natural sciences, especially in atmospheric and Earth sciences involving applied statistics, an anomaly is a persisting deviation in a physical quantity from its expected value, e.g., the systematic difference between a measurement and a trend or a model prediction. Similarly, a standardized anomaly equals an anomaly divided by a standard deviation. A group of anomalies can be analyzed spatially, as a map, or temporally, as a time series. It should not be confused for an isolated outlier. There are examples in atmospheric sciences and in geophysics.

<span class="mw-page-title-main">Pacific Meridional Mode</span> Climate mode in the North Pacific

Pacific Meridional Mode (PMM) is a climate mode in the North Pacific. In its positive state, it is characterized by the coupling of weaker trade winds in the northeast Pacific Ocean between Hawaii and Baja California with decreased evaporation over the ocean, thus increasing sea surface temperatures (SST); and the reverse during its negative state. This coupling develops during the winter months and spreads southwestward towards the equator and the central and western Pacific during spring, until it reaches the Intertropical Convergence Zone (ITCZ), which tends to shift north in response to a positive PMM.

Lisa Marie Goddard was an American climate scientist who was director at the International Research Institute for Climate and Society (IRI). She joined the institute in 1995 and served as IRI's director from 2012 to 2020. Goddard was also an adjunct associate professor at Columbia University.

References

  1. Ronald A. Christensen and Richard F. Eilbert and Orley H. Lindgren and Laurel L. Rans (1981). "Successful Hydrologic Forecasting for California Using an Information Theoretic Model". Journal of Applied Meteorology. 20 (6): 706–712. doi: 10.1175/1520-0450(1981)020<0706:SHFFCU>2.0.CO;2 .
  2. Youmin Tang and Richard Kleeman and Andrew M. Moore (2005). "Reliability of ENSO Dynamical Predictions". Journal of the Atmospheric Sciences. 62 (6): 1770–1791. Bibcode:2005JAtS...62.1770T. doi:10.1175/JAS3445.1.
  3. WMO Joint Working Group on Forecast Verification Research. "Forecast Verification: Issues, Methods and FAQ" . Retrieved July 30, 2013.
  4. Ian T. Jolliffe and David B. Stephenson (2011). Forecast Verification: A Practitioner's Guide in Atmospheric Science. Wiley.
  5. Wilks, Daniel (2011). "Chapter 8: Forecast Verification". Statistical Methods in the Atmospheric Sciences (3rd ed.). Elsevier. ISBN   9780123850225.