Natural experiment

Last updated

A natural experiment is an empirical study in which individuals (or clusters of individuals) are exposed to the experimental and control conditions that are determined by nature or by other factors outside the control of the investigators. The process governing the exposures arguably resembles random assignment. Thus, natural experiments are observational studies and are not controlled in the traditional sense of a randomized experiment (an intervention study). Natural experiments are most useful when there has been a clearly defined exposure involving a well defined subpopulation (and the absence of exposure in a similar subpopulation) such that changes in outcomes may be plausibly attributed to the exposure. [1] [2] In this sense, the difference between a natural experiment and a non-experimental observational study is that the former includes a comparison of conditions that pave the way for causal inference, but the latter does not.


Natural experiments are employed as study designs when controlled experimentation is extremely difficult to implement or unethical, such as in several research areas addressed by epidemiology (like evaluating the health impact of varying degrees of exposure to ionizing radiation in people living near Hiroshima at the time of the atomic blast [3] ) and economics (like estimating the economic return on amount of schooling in US adults [4] ). [1] [2]


John Snow's map showing the clustering of cholera cases in Soho during the London epidemic of 1854 Snow-cholera-map-1.jpg
John Snow's map showing the clustering of cholera cases in Soho during the London epidemic of 1854

One of the best-known early natural experiments was the 1854 Broad Street cholera outbreak in London, England. On 31 August 1854, a major outbreak of cholera struck Soho. Over the next three days, 127 people near Broad Street died. By the end of the outbreak 616 people died. The physician John Snow identified the source of the outbreak as the nearest public water pump, using a map of deaths and illness that revealed a cluster of cases around the pump.

In this example, Snow discovered a strong association between the use of the water from the pump, and deaths and illnesses due to cholera. Snow found that the Southwark and Vauxhall Waterworks Company, which supplied water to districts with high attack rates, obtained the water from the Thames downstream from where raw sewage was discharged into the river. By contrast, districts that were supplied water by the Lambeth Waterworks Company, which obtained water upstream from the points of sewage discharge, had low attack rates. Given the near-haphazard patchwork development of the water supply in mid-nineteenth century London, Snow viewed the developments as "an experiment...on the grandest scale." [5] Of course, the exposure to the polluted water was not under the control of any scientist. Therefore, this exposure has been recognized as being a natural experiment. [6] [7] [8]

Recent examples

Family size

An aim of a study Angrist and Evans (1998) [9] was to estimate the effect of family size on the labor market outcomes of the mother. For at least two reasons, the correlations between family size and various outcomes (e.g., earnings) do not inform us about how family size causally affects labor market outcomes. First, both labor market outcomes and family size may be affected by unobserved "third" variables (e.g., personal preferences). Second, labor market outcomes themselves may affect family size (called "reverse causality"). For example, a woman may defer having a child if she gets a raise at work. The authors observed that two-child families with either two boys or two girls are substantially more likely to have a third child than two-child families with one boy and one girl. The sex of the first two children, then, constitutes a kind of natural experiment: it is as if an experimenter had randomly assigned some families to have two children and others to have three. The authors were then able to credibly estimate the causal effect of having a third child on labor market outcomes. Angrist and Evans found that childbearing had a greater impact on poor and less educated women than on highly educated women although the earnings impact of having a third child tended to disappear by that child's 13th birthday. They also found that having a third child had little impact on husbands' earnings. [9]

Game shows

Within economics, game shows are a frequently studied form of natural experiment. While game shows might seem to be artificial contexts, they can be considered natural experiments due to the fact that the context arises without interference of the scientist. Game shows have been used to study a wide range of different types of economic behavior, such as decision making under risk [10] and cooperative behavior. [11]

Smoking ban

In Helena, Montana a smoking ban was in effect in all public spaces, including bars and restaurants, during the six-month period from June 2002 to December 2002. Helena is geographically isolated and served by only one hospital. The investigators observed that the rate of heart attacks dropped by 40% while the smoking ban was in effect. Opponents of the law prevailed in getting the enforcement of the law suspended after six months, after which the rate of heart attacks went back up. [12] This study was an example of a natural experiment, called a case-crossover experiment, where the exposure is removed for a time and then returned. The study also noted its own weaknesses which potentially suggest that the inability to control variables in natural experiments can impede investigators from drawing firm conclusions.' [12]

Nuclear weapons testing

Nuclear weapons testing released large quantities of radioactive isotopes into the atmosphere, some of which could be incorporated into biological tissues. The release stopped after the Partial Nuclear Test Ban Treaty in 1963, which prohibited atmospheric nuclear tests. This resembled a large-scale pulse-chase experiment, but could not have been performed as a regular experiment in humans due to scientific ethics. Several types of observations were made possible (in people born before 1963), such as determination of the rate of replacement for cells in different human tissues.

Vietnam War draft

An important question in economics research is what determines earnings. Angrist (1990) evaluated the effects of military service on lifetime earnings. [13] Using statistical methods developed in econometrics, [14] Angrist capitalized on the approximate random assignment of the Vietnam War draft lottery, and used it as an instrumental variable associated with eligibility (or non-eligibility) for military service. Because many factors might predict whether someone serves in the military, the draft lottery frames a natural experiment whereby those drafted into the military can be compared against those not drafted because the two groups should not differ substantially prior to military service. Angrist found that the earnings of veterans were, on average, about 15 percent less than the earnings of non-veterans.

Industrial melanism

With the Industrial Revolution in the nineteenth century, many species of moth, including the well-studied peppered moth, responded to the atmospheric pollution of sulphur dioxide and soot around cities with industrial melanism, a dramatic increase in the frequency of dark forms over the formerly abundant pale, speckled forms. In the twentieth century, as regulation improved and pollution fell, providing the conditions for a large-scale natural experiment, the trend towards industrial melanism was reversed, and melanic forms quickly became scarce. The effect led the evolutionary biologists L. M. Cook and J. R. G. Turner to conclude that "natural selection is the only credible explanation for the overall decline". [15]

See also

Related Research Articles

Econometrics is the application of statistical methods to economic data in order to give empirical content to economic relationships. More precisely, it is "the quantitative analysis of actual economic phenomena based on the concurrent development of theory and observation, related by appropriate methods of inference". An introductory economics textbook describes econometrics as allowing economists "to sift through mountains of data to extract simple relationships". Jan Tinbergen is one of the two founding fathers of econometrics. The other, Ragnar Frisch, also coined the term in the sense in which it is used today.

Epidemiology aspect of health and disease science

Epidemiology is the study and analysis of the distribution, patterns and determinants of health and disease conditions in defined populations.

A cohort study is a particular form of longitudinal study that samples a cohort, performing a cross-section at intervals through time. It is a type of panel study where the individuals in the panel share a common characteristic.

Case–control study Type of observational study comparing two existing groups differing in outcome

A case–control study is a type of observational study in which two existing groups differing in outcome are identified and compared on the basis of some supposed causal attribute. Case–control studies are often used to identify factors that may contribute to a medical condition by comparing subjects who have that condition/disease with patients who do not have the condition/disease but are otherwise similar. They require fewer resources but provide less evidence for causal inference than a randomized controlled trial. A case–control study produces only an odds ratio, which is an inferior measure of strength of association compared to relative risk.

John Snow English epidemiologist and physician (1813–1858)

John Snow was an English physician and a leader in the development of anaesthesia and medical hygiene. He is considered one of the founders of modern epidemiology, in part because of his work in tracing the source of a cholera outbreak in Soho, London, in 1854, which he curtailed by removing the handle of a water pump. Snow's findings inspired the adoption of anaesthesia as well as fundamental changes in the water and waste systems of London, which led to similar changes in other cities, and a significant improvement in general public health around the world.

Field experiment

Field experiments are experiments carried out outside of laboratory settings.

Health geography

Health geography is the application of geographical information, perspectives, and methods to the study of health, disease, and health care. Medical geography, a sub-discipline of or sister field of health geography, focuses on understanding spatial patterns of health and disease as related to the natural and social environment. Conventionally, there are two primary areas of research within medical geography: the first deals with the spatial distribution and determinants of morbidity and mortality, while the second deals with health planning, help-seeking behavior, and the provision of health services.

The Rubin causal model (RCM), also known as the Neyman–Rubin causal model, is an approach to the statistical analysis of cause and effect based on the framework of potential outcomes, named after Donald Rubin. The name "Rubin causal model" was first coined by Paul W. Holland. The potential outcomes framework was first proposed by Jerzy Neyman in his 1923 Master's thesis, though he discussed it only in the context of completely randomized experiments. Rubin extended it into a general framework for thinking about causation in both observational and experimental studies.

Mendelian randomization Statistical method in genetic epidemiology

In epidemiology, Mendelian randomization is a method using measured variation in genes to interrogate the causal effect of an exposure on an outcome. Under key assumptions, the design reduces both reverse causation and confounding, which often substantially impede or mislead the interpretation of results from epidemiological studies.

Environmental epidemiology

Environmental epidemiology is a branch of epidemiology concerned with determining how environmental exposures impact human health. This field seeks to understand how various external risk factors may predispose to or protect against disease, illness, injury, developmental abnormalities, or death. These factors may be naturally occurring or may be introduced into environments where people live, work, and play.

Joshua Angrist Israeli American economist

Joshua David Angrist is an Israeli-American economist and Ford Professor of Economics at the Massachusetts Institute of Technology. Angrist, together with Guido Imbens, was awarded the Nobel Memorial Prize in Economics in 2021 "for their methodological contributions to the analysis of causal relationships".

In statistics, econometrics, political science, epidemiology, and related disciplines, a regression discontinuity design (RDD) is a quasi-experimental pretest-posttest design that aims to determine the causal effects of interventions by assigning a cutoff or threshold above or below which an intervention is assigned. By comparing observations lying closely on either side of the threshold, it is possible to estimate the average treatment effect in environments in which randomisation is unfeasible. However, it remains impossible to make true causal inference with this method alone, as it does not automatically reject causal effects by any potential confounding variable. First applied by Donald Thistlethwaite and Donald Campbell to the evaluation of scholarship programs, the RDD has become increasingly popular in recent years. Recent study comparisons of randomised controlled trials (RCTs) and RDDs have empirically demonstrated the internal validity of the design.

1846–1860 cholera pandemic The third major outbreak of cholera, 1846–1860 worldwide pandemic

The third cholera pandemic (1846–1860) was the third major outbreak of cholera originating in India in the nineteenth century that reached far beyond its borders, which researchers at UCLA believe may have started as early as 1837 and lasted until 1863. In Russia, more than one million people died of cholera. In 1853–54, the epidemic in London claimed over 10,000 lives, and there were 23,000 deaths for all of Great Britain. This pandemic was considered to have the highest fatalities of the 19th-century epidemics.

1854 Broad Street cholera outbreak Severe outbreak of cholera that occurred in London in 1854

The Broad Street cholera outbreak was a severe outbreak of cholera that occurred in 1854 near Broad Street in the Soho district of the City of Westminster, London, England, and occurred during the 1846–1860 cholera pandemic happening worldwide. This outbreak, which killed 616 people, is best known for the physician John Snow's study of its causes and his hypothesis that germ-contaminated water was the source of cholera, rather than particles in the air. This discovery came to influence public health and the construction of improved sanitation facilities beginning in the mid-19th century. Later, the term "focus of infection" started to be used to describe sites, such as the Broad Street pump, in which conditions are good for transmission of an infection. Snow's endeavour to find the cause of the transmission of cholera caused him to unknowingly create a double-blind experiment.

Matching is a statistical technique which is used to evaluate the effect of a treatment by comparing the treated and the non-treated units in an observational study or quasi-experiment. The goal of matching is to reduce bias for the estimated treatment effect in an observational-data study, by finding, for every treated unit, one non-treated unit(s) with similar observable characteristics against who the covariates are balanced out. By matching treated units to similar non-treated units, matching enables a comparison of outcomes among treated and non-treated units to estimate the effect of the treatment reducing bias due to confounding. Propensity score matching, an early matching technique, was developed as part of the Rubin causal model, but has been shown to increase model dependence, bias, inefficiency, and power and is no longer recommended compared to other matching methods.

Causal inference is the process of determining the independent, actual effect of a particular phenomenon that is a component of a larger system. The main difference between causal inference and inference of association is that causal inference analyzes the response of an effect variable when a cause of the effect variable is changed. The science of why things occur is called etiology. Causal inference is said to provide the evidence of causality theorized by causal reasoning.

Causation in economics has a long history with Adam Smith explicitly acknowledging its importance via his (1776) An Inquiry into the Nature and Causes of the Wealth of Nations and David Hume and John Stuart Mill (1848) both offering important contributions with more philosophical discussions. Hoover (2006) suggests that a useful way of classifying approaches to causation in economics might be to distinguish between approaches that emphasize structure and those that emphasise process and to add to this a distinction between approaches that adopt a priori reasoning and those that seek to infer causation from the evidence provided by data. He represented by this little table which useful identifies key works in each of the four categories.

Guido Wilhelmus Imbens is a Dutch-American economist. In 2021 Imbens was awarded half of the Nobel Memorial Prize in Economic Sciences jointly with Joshua Angrist "for their methodological contributions to the analysis of causal relationships", with David Card awarded the other half. He has been Professor of Economics at the Stanford Graduate School of Business at Stanford University since 2012.

The experimentalist approach to econometrics is a way of doing econometrics that, according to Angrist and Krueger (1999): … puts front and center the problem of identifying causal effects from specific events or situations. These events or situations are thought of as natural experiments that generate exogenous variations in variables that would otherwise be endogenous in the behavioral relationship of interest. An example from the economic study of education can be used to illustrate the approach. Here we might be interested in the effect of effect of an additional year of education on earnings. Those working with an experimentalist approach to econometrics would argue that such a question is problematic to answer because, and this is using their terminology, education is not randomly assigned. That is those with different education levels would tend to also have different levels of other variables. And these other variable, many of which would be unobserved, also affect earnings. This renders the causal effect of extra years of schooling difficult to identify. The experimentalist approach looks for an instrumental variable that is correlated with X but uncorrelated with the unobservables.

Victor Chaim Lavy is an Israeli economist and professor at the University of Warwick and the Hebrew University of Jerusalem. His research interests include labour economics, the economics of education, and development economics. Lavy belongs to the most prominent education economists in the world.


  1. 1 2 DiNardo, J. (2008). "Natural experiments and quasi-natural experiments". In Durlauf, Steven N.; Blume, Lawrence E (eds.). The New Palgrave Dictionary of Economics (Second ed.). Palgrave Macmillan. pp. 856–864. doi:10.1057/9780230226203.1162. ISBN   978-0-333-78676-5.
  2. 1 2 Dunning, Thad (2012). Natural Experiments in the Social Sciences: A Design-Based Approach. Cambridge University Press.
  3. Friedman, G. D. (1980). Primer of Epidemiology (2nd ed.). New York: McGraw-Hill. ISBN   978-0-07-022434-6.
  4. Rosenzweig, M. R.; Wolpin, K. I. (2000). "Natural 'Natural Experiments' in Economics". Journal of Economic Literature . 38 (4): 827–874. doi:10.1257/jel.38.4.827.
  5. Snow, J. (1855). On the Mode of Communication of Cholera (2nd ed.). London: Churchill. Excerpted in MacMahon, B. & Pugh, T.F. (1970). Epidemiology. Boston: Little Brown.
  6. The 1854 cholera outbreak is the example of a natural experiment discussed often by David A. Freedman, e.g. in Statistical Models: Theory and Practice (Cambridge University Press) , chapter 1.3 (pp. 6–9).
  7. MacMahon, B., & Pugh, T.F. (1970). Epidemiology: Principles and Methods. Boston: Little, Brown.
  8. Snow's studies of the pattern of the disease were convincing enough to persuade the local council to disable the well pump by removing its handle. After the handle of the well-pump was replaced, the incidence of new cases dropped. In stopping the use of water from the well-pump, the authorities conducted, in effect, a second study, an uncontrolled intervention study, a study with an intervention group but no control group.
  9. 1 2 Angrist, J.; Evans, W. (1998). "Children and Their Parents' Labor Supply: Evidence from Exogenous Variation in Family Size". American Economic Review . 88 (3): 450–477. JSTOR   116844.
  10. Post; Van den Assem; Baltussen; Thaler (2008). "Deal or No Deal? Decision Making under Risk in a Large-Payoff Game Show" (PDF). American Economic Review . 98 (1): 38–71. doi:10.1257/aer.98.1.38. JSTOR   29729963. S2CID   12816022. SSRN   636508 .
  11. van den Assem; van Dolder; Thaler (2012). "Split or Steal? Cooperative Behavior When the Stakes Are Large". Management Science. 58 (1): 2. doi:10.1287/mnsc.1110.1413. hdl: 1765/31292 . S2CID   1371739. SSRN   1592456 .
  12. 1 2 Sargent, R. P.; Shepard, R. M.; Glantz, S. A. (2004). "Reduced incidence of admissions for myocardial infarction associated with public smoking ban: before and after study". British Medical Journal . 328 (7446): 977–980. doi:10.1136/bmj.38055.715683.55. PMC   404491 . PMID   15066887.
  13. Angrist, Joshua D. (1990). "Lifetime Earnings and the Vietnam Draft Lottery: Evidence from Social Security Administrative Records". American Economic Review . 80 (3): 313–336. JSTOR   2006669.
  14. Lambert, B. (2013). Natural experiments in econometrics. Youtube video
  15. Cook, L. M.; Turner, J. R. G. (2008). "Decline of melanism in two British moths: spatial, temporal and inter-specific variation". Heredity. 101 (6): 483–489. doi: 10.1038/hdy.2008.105 . PMID   18941471.