Backtesting

Last updated

Backtesting is a term used in modeling to refer to testing a predictive model on historical data. Backtesting is a type of retrodiction, and a special type of cross-validation applied to previous time period(s).

Contents

Financial analysis

In the economic and financial field, backtesting seeks to estimate the performance of a strategy or model if it had been employed during a past period. This requires simulating past conditions with sufficient detail, making one limitation of backtesting the need for detailed historical data. A second limitation is the inability to model strategies that would affect historic prices. Finally, backtesting, like other modeling, is limited by potential overfitting. That is, it is often possible to find a strategy that would have worked well in the past, but will not work well in the future. [1] Despite these limitations, backtesting provides information not available when models and strategies are tested on synthetic data.

Backtesting has historically only been performed by large institutions and professional money managers due to the expense of obtaining and using detailed datasets. However, backtesting is increasingly used on a wider basis, and independent web-based backtesting platforms have emerged. Although the technique is widely used, it is prone to weaknesses. [2] Basel financial regulations require large financial institutions to backtest certain risk models.

For a Value at Risk 1-day at 99% backtested 250 days in a row, the test is considered green (0-95%), orange (95-99.99%) or red (99.99-100%) depending on the following table: [3]

backtesting exceptions 1Dx250 Backtesting exceptions 1Dx250.png
backtesting exceptions 1Dx250
1-day VaR at 99% backtested 250 days
Zone Number exceptions Probability Cumul
Green 0 8.11% 8.11%
1 20.47% 28.58%
2 25.74% 54.32%
3 21.49% 75.81%
4 13.41% 89.22%
Orange 5 6.66% 95.88%
6 2.75% 98.63%
7 0.97% 99.60%
8 0.30% 99.89%
9 0.08% 99.97%
Red 10 0.02% 99.99%
11 0.00% 100.00%
... ... ...

For a Value at Risk 10-day at 99% backtested 250 days in a row, the test is considered green (0-95%), orange (95-99.99%) or red (99.99-100%) depending on the following table:

backtesting exceptions 10Dx250 Backtesting exceptions 10Dx250.png
backtesting exceptions 10Dx250
10-day VaR at 99% backtested 250 days
Zone Number exceptions Probability Cumul
Green 0 36.02% 36.02%
1 15.99% 52.01%
2 11.58% 63.59%
3 8.90% 72.49%
4 6.96% 79.44%
5 5.33% 84.78%
6 4.07% 88.85%
7 3.05% 79.44%
8 2.28% 94.17%
Orange 9 1.74% 95.91%
... ... ...
24 0.01% 99.99%
Red 25 0.00% 99.99%
... ... ...

Hindcast

Temporal representation of hindcasting. Hindcasting.jpeg
Temporal representation of hindcasting.

In oceanography [5] and meteorology, [6] backtesting is also known as hindcasting: a hindcast is a way of testing a mathematical model; researchers enter known or closely estimated inputs for past events into the model to see how well the output matches the known results.

Hindcasting usually refers to a numerical-model integration of a historical period where no observations have been assimilated. This distinguishes a hindcast run from a reanalysis. Oceanographic observations of salinity and temperature as well as observations of surface-wave parameters such as the significant wave height are much scarcer than meteorological observations, making hindcasting more common in oceanography than in meteorology. Also, since surface waves represent a forced system where the wind is the only generating force, wave hindcasting is often considered adequate for generating a reasonable representation of the wave climate with little need for a full reanalysis. Hydrologists use hindcasting for model stream flows. [7]

An example of hindcasting would be entering climate forcings (events that force change) into a climate model. If the hindcast showed reasonably-accurate climate response, the model would be considered successful.

The ECMWF re-analysis is an example of a combined atmospheric reanalysis coupled with a wave-model integration where no wave parameters were assimilated, making the wave part a hindcast run.

See also

Related Research Articles

<span class="mw-page-title-main">European Centre for Medium-Range Weather Forecasts</span> European intergovernmental weather computation organisation based in the UK

The European Centre for Medium-Range Weather Forecasts (ECMWF) is an independent intergovernmental organisation supported by most of the nations of Europe. It is based at three sites: Shinfield Park, Reading, United Kingdom; Bologna, Italy; and Bonn, Germany. It operates one of the largest supercomputer complexes in Europe and the world's largest archive of numerical weather prediction data.

<span class="mw-page-title-main">Value at risk</span> Estimated potential loss for an investment under a given set of conditions

Value at risk (VaR) is a measure of the risk of loss of investment/Capital. It estimates how much a set of investments might lose, given normal market conditions, in a set time period such as a day. VaR is typically used by firms and regulators in the financial industry to gauge the amount of assets needed to cover possible losses.

<span class="mw-page-title-main">General circulation model</span> Type of climate model

A general circulation model (GCM) is a type of climate model. It employs a mathematical model of the general circulation of a planetary atmosphere or ocean. It uses the Navier–Stokes equations on a rotating sphere with thermodynamic terms for various energy sources. These equations are the basis for computer programs used to simulate the Earth's atmosphere or oceans. Atmospheric and oceanic GCMs are key components along with sea ice and land-surface components.

Forecasting is the process of making predictions based on past and present data. Later these can be compared (resolved) against what happens. For example, a company might estimate their revenue in the next year, then compare it against the actual results creating a variance actual analysis. Prediction is a similar but more general term. Forecasting might refer to specific formal statistical methods employing time series, cross-sectional or longitudinal data, or alternatively to less formal judgmental methods or the process of prediction and resolution itself. Usage can vary between areas of application: for example, in hydrology the terms "forecast" and "forecasting" are sometimes reserved for estimates of values at certain specific future times, while the term "prediction" is used for more general estimates, such as the number of times floods will occur over a long period.

<span class="mw-page-title-main">Jule Gregory Charney</span> US meteorologist

Jule Gregory Charney was an American meteorologist who played an important role in developing numerical weather prediction and increasing understanding of the general circulation of the atmosphere by devising a series of increasingly sophisticated mathematical models of the atmosphere. His work was the driving force behind many national and international weather initiatives and programs.

<span class="mw-page-title-main">Numerical weather prediction</span> Weather prediction using mathematical models of the atmosphere and oceans

Numerical weather prediction (NWP) uses mathematical models of the atmosphere and oceans to predict the weather based on current weather conditions. Though first attempted in the 1920s, it was not until the advent of computer simulation in the 1950s that numerical weather predictions produced realistic results. A number of global and regional forecast models are run in different countries worldwide, using current weather observations relayed from radiosondes, weather satellites and other observing systems as inputs.

Retrodiction is the act of making a prediction about the past. It is also known as postdiction.

The ECMWF reanalysis project is a meteorological reanalysis project carried out by the European Centre for Medium-Range Weather Forecasts (ECMWF). The first reanalysis product, ERA-15, generated reanalyses for approximately 15 years, from December 1978 to February 1994. The second product, ERA-40 begins in 1957 and covers 45 years to 2002. As a precursor to a revised extended reanalysis product to replace ERA-40, ECMWF released ERA-Interim, which covers the period from 1979 to 2019. A new reanalysis product ERA5 has recently been released by ECMWF as part of Copernicus Climate Change Services. This product has higher spatial resolution and covers the period from 1979 to present. Extension up to 1940 became available in 2023.

<span class="mw-page-title-main">Ensemble forecasting</span> Multiple simulation method for weather forecasting

Ensemble forecasting is a method used in or within numerical weather prediction. Instead of making a single forecast of the most likely weather, a set of forecasts is produced. This set of forecasts aims to give an indication of the range of possible future states of the atmosphere. Ensemble forecasting is a form of Monte Carlo analysis. The multiple simulations are conducted to account for the two usual sources of uncertainty in forecast models: (1) the errors introduced by the use of imperfect initial conditions, amplified by the chaotic nature of the evolution equations of the atmosphere, which is often referred to as sensitive dependence on initial conditions; and (2) errors introduced because of imperfections in the model formulation, such as the approximate mathematical methods to solve the equations. Ideally, the verified future atmospheric state should fall within the predicted ensemble spread, and the amount of spread should be related to the uncertainty (error) of the forecast. In general, this approach can be used to make probabilistic forecasts of any dynamical system, and not just for weather prediction.

Data assimilation is a mathematical discipline that seeks to optimally combine theory with observations. There may be a number of different goals sought – for example, to determine the optimal state estimate of a system, to determine initial conditions for a numerical forecast model, to interpolate sparse observation data using knowledge of the system being observed, to set numerical parameters based on training a model from observed data. Depending on the goal, different solution methods may be used. Data assimilation is distinguished from other forms of machine learning, image analysis, and statistical methods in that it utilizes a dynamical model of the system being analyzed.

<span class="mw-page-title-main">Atlantic hurricane reanalysis project</span> Project to add new information about past North Atlantic hurricanes

The Atlantic hurricane reanalysis project of the National Oceanic and Atmospheric Administration seeks to correct and add new information about past North Atlantic hurricanes. It was started around 2000 to update HURDAT, the official hurricane database for the Atlantic Basin, which has become outdated since its creation due to various systematic errors introduced into the database over time. This effort has involved reanalyses of ship observations from the International Comprehensive Ocean-Atmosphere Data Set (ICOADS) as well as reanalyses done by other researchers over the years. It has been ongoing as of 2024.

<span class="mw-page-title-main">Norwegian Meteorological Institute</span> National meteorological service of Norway

The Norwegian Meteorological Institute, also known internationally as MET Norway, is Norway's national meteorological institute. It provides weather forecasts for civilian and military uses and conducts research in meteorology, oceanography and climatology. It is headquartered in Oslo and has offices and stations in other cities and places. It has around 500 full-time staff and was founded in 1866.

The NCEP/NCAR Reanalysis is an atmospheric reanalysis produced by the National Centers for Environmental Prediction (NCEP) and the National Center for Atmospheric Research (NCAR). It is a continually updated globally gridded data set that represents the state of the Earth's atmosphere, incorporating observations and numerical weather prediction (NWP) model output from 1948 to present.

<span class="mw-page-title-main">Jagadish Shukla</span> Indian meteorologist

Jagadish Shukla is an Indian meteorologist and Distinguished University Professor at George Mason University in the United States.

<span class="mw-page-title-main">Wind wave model</span> Numerical modelling of the sea state

In fluid dynamics, wind wave modeling describes the effort to depict the sea state and predict the evolution of the energy of wind waves using numerical techniques. These simulations consider atmospheric wind forcing, nonlinear wave interactions, and frictional dissipation, and they output statistics describing wave heights, periods, and propagation directions for regional seas or global oceans. Such wave hindcasts and wave forecasts are extremely important for commercial interests on the high seas. For example, the shipping industry requires guidance for operational planning and tactical seakeeping purposes.

In finance, model risk is the risk of loss resulting from using insufficiently accurate models to make decisions, originally and frequently in the context of valuing financial securities.

An atmospheric reanalysis is a meteorological and climate data assimilation project which aims to assimilate historical atmospheric observational data spanning an extended period, using a single consistent assimilation scheme throughout.

Walk forward optimization is a method used in finance to determine the optimal parameters for a trading strategy and to determine the robustness of the strategy. Walk Forward Analysis was created by Robert E. Pardo in 1992 and expanded in the second edition. Walk Forward Analysis is now widely considered the "gold standard" in trading strategy validation. The trading strategy is optimized with in-sample data for a time window in a data series. The remaining data is reserved for out of sample testing. A small portion of the reserved data following the in-sample data is tested and the results are recorded. The in-sample time window is shifted forward by the period covered by the out of sample test, and the process repeated. Lastly, all of the recorded results are used to assess the trading strategy.

Masao Kanamitsu was a Japanese and American atmospheric scientist working in the field of data assimilation. His research greatly influenced global and regional climate change studies including development of breakthrough reanalysis and downscaling datasets and weather forecasting studies. He was the co-author of one of the most cited geophysics paper in his time.

The Simple Ocean Data Assimilation (SODA) analysis is an oceanic reanalysis data set consisting of gridded state variables for the global ocean, as well as several derived fields. SODA was developed in the 1990s as a collaborative project between the Department of Atmospheric and Oceanic Science at the University of Maryland and the Department of Oceanography at Texas A&M University with the goal of providing an improved estimate of ocean state from those based solely on observations or numerical simulations. Since its first release there have been several updates, the most recent of which extends from 1958 to 2008, as well as a “beta release” of a long-term reanalysis for 1871–2008.

References

  1. Bailey, Borwein, Lopez de Prado, Zhu (2014). "Pseudo-mathematics and financial charlatanism. Notices of the American Mathematical Society, Volume 61, Number 5, pp. 458-471" (PDF).{{cite web}}: CS1 maint: multiple names: authors list (link) CS1 maint: numeric names: authors list (link)
  2. FinancialTrading (2013-04-27). "Issues related to back testing".
  3. "Supervisory framework for the use of "backtesting" in conjunction with the internal models approach to market risk capital requirements" (PDF). Basle Committee on Banking Supervision. January 1996. p. 14.
  4. Taken from p.145 of Yeates, L.B., Thought Experimentation: A Cognitive Approach, Graduate Diploma in Arts (By Research) dissertation, University of New South Wales, 2004.
  5. "Hindcast approach". OceanWeather Inc. Retrieved 22 January 2013.
  6. Huijnen, V.; J. Flemming; J. W. Kaiser; A. Inness; J. Leitão; A. Heil; H. J. Eskes; M. G. Schultz; A. Benedetti; J. Hadji-Lazaro; G. Dufour; M. Eremenko (2012). "Hindcast experiments of tropospheric composition during the summer 2010 fires over western Russia". Atmos. Chem. Phys. 12 (9): 4341–4364. Bibcode:2012ACP....12.4341H. doi: 10.5194/acp-12-4341-2012 . Retrieved 22 January 2013.
  7. "Guidance on Conducting Streamflow Hindcasting in CHPS" (PDF). NOAA. Retrieved 22 January 2013.