Goodhart's law

Last updated

Goodhart's law is an adage named after economist Charles Goodhart, which has been phrased by Marilyn Strathern as "When a measure becomes a target, it ceases to be a good measure." [1] One way in which this can occur is individuals trying to anticipate the effect of a policy and then taking actions that alter its outcome. [2]

Contents

Formulation and precedence

Goodhart first advanced the idea in a 1975 article, which later became used popularly to criticize the United Kingdom government of Margaret Thatcher for trying to conduct monetary policy on the basis of targets for broad and narrow money.[ clarification needed ] His original formulation was: [3]

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

However, there have been numerous concepts related to this idea, at least one of which pre-dates Goodhart's statement in 1975. [4] Notably, Campbell's law likely has precedence, as Jeff Rodamar has argued, since various formulations date to 1969. [5] Other academics had similar insights during this time period. Jerome R. Ravetz's 1971 book Scientific Knowledge and Its Social Problems [6] also predates Goodhart, though it does not formulate the same law. He discusses how systems in general can be gamed, focuses on cases where the goals of a task are complex, sophisticated, or subtle. In such cases, the persons possessing the skills to execute the tasks properly are instead able to achieve their own goals to the detriment of the assigned tasks. When the goals are instantiated as metrics, this could be seen as equivalent to Goodhart and Campbell's claim.

Shortly after Goodhart's publication, others suggested closely related ideas, including the Lucas critique (1976). As applied in economics, the law is also implicit in the economic idea of rational expectations, a theory in economics that states that those who are aware of a system of rewards and punishments will optimize their actions within that system to achieve their desired results. For example, if an employee is rewarded by the number of cars she sells each month, she will try to sell more cars, whether or not she makes a profit on any.

While it originated in the context of market responses, the law has profound implications for the selection of high-level targets in organizations. [7] Jón Danı́elsson quotes the law as "Any statistical relationship will break down when used for policy purposes" and suggests a corollary to the law for use in financial risk modelling: "A risk model breaks down when used for regulatory purposes." [8] Mario Biagioli has related the concept to consequences of using citation impact measures to estimate the importance of scientific publications: [9]

All metrics of scientific evaluation are bound to be abused. Goodhart's law (named after the British economist who may have been the first to announce it) states that when a feature of the economy is picked as an indicator of the economy, then it inexorably ceases to function as that indicator because people start to game it.

The law is illustrated in the 2018 book The Tyranny of Metrics by Jerry Z. Muller. [10]

See also

Related Research Articles

Alcubierre drive Hypothetical mode of transportation by warping space

The Alcubierre drive, Alcubierre warp drive, or Alcubierre metric is a speculative idea based on a solution of Einstein's field equations in general relativity as proposed by theoretical physicist Miguel Alcubierre, by which a spacecraft could achieve apparent faster-than-light travel if a configurable energy-density field lower than that of vacuum could be created.

arXiv Online digital archive for preprints of scientific papers

arXiv is an open-access repository of electronic preprints approved for posting after moderation, but not full peer review. It consists of scientific papers in the fields of mathematics, physics, astronomy, electrical engineering, computer science, quantitative biology, statistics, mathematical finance and economics, which can be accessed online. In many fields of mathematics and physics, almost all scientific papers are self-archived on the arXiv repository. Begun on August 14, 1991, arXiv.org passed the half-million-article milestone on October 3, 2008, and had hit a million by the end of 2014. By October 2016 the submission rate had grown to more than 10,000 per month.

Post-Keynesian economics is a school of economic thought with its origins in The General Theory of John Maynard Keynes, with subsequent development influenced to a large degree by Michał Kalecki, Joan Robinson, Nicholas Kaldor, Sidney Weintraub, Paul Davidson, Piero Sraffa and Jan Kregel. Historian Robert Skidelsky argues that the post-Keynesian school has remained closest to the spirit of Keynes' original work. It is a heterodox approach to economics.

This aims to be a complete article list of economics topics:

Brane cosmology Several theories in particle physics and cosmology related to superstring theory and M-theory

Brane cosmology refers to several theories in particle physics and cosmology related to string theory, superstring theory and M-theory.

The Lucas critique, named for Robert Lucas's work on macroeconomic policymaking, argues that it is naive to try to predict the effects of a change in economic policy entirely on the basis of relationships observed in historical data, especially highly aggregated historical data. More formally, it states that the decision rules of Keynesian models—such as the consumption function—cannot be considered as structural in the sense of being invariant with respect to changes in government policy variables. The Lucas critique is significant in the history of economic thought as a representative of the paradigm shift that occurred in macroeconomic theory in the 1970s towards attempts at establishing micro-foundations.

The impact factor (IF) or journal impact factor (JIF) of an academic journal is a scientometric index that reflects the yearly average number of citations that articles published in the last two years in a given journal received. It is frequently used as a proxy for the relative importance of a journal within its field; journals with higher impact factors are often deemed to be more important than those with lower ones.

Nassim Nicholas Taleb French-Lebanese-American essayist, scholar, statistician, former trader and risk analyst

Nassim Nicholas Taleb is a Lebanese-American essayist, scholar, mathematical statistician, and former option trader and risk analyst, whose work concerns problems of randomness, probability, and uncertainty. His 2007 book The Black Swan has been described by The Sunday Times as one of the twelve most influential books since World War II.

Scientometrics is the field of study which concerns itself with measuring and analysing scientific literature. Scientometrics is a sub-field of bibliometrics. Major research issues include the measurement of the impact of research papers and academic journals, the understanding of scientific citations, and the use of such measurements in policy and management contexts. In practice there is a significant overlap between scientometrics and other scientific fields such as information systems, information science, science of science policy, sociology of science, and metascience.

Post-normal science use of science on urgent issues involving uncertainty in facts and moral values

Post-normal science (PNS) represents a novel approach for the use of science on issues where "facts [are] uncertain, values in dispute, stakes high and decisions urgent". PNS was developed in the 1990s by Silvio Funtowicz and Jerome R. Ravetz. It can be considered as a reaction to the styles of analysis based on risk and cost-benefit analysis prevailing at that time, and as an embodiment of concepts of a new "critical science" developed in previous works by the same authors. In a more recent work PNS is described as "the stage where we are today, where all the comfortable assumptions about science, its production and its use, are in question".

Citation impact quantifies the citation usage of scholarly works. It is a result of citation analysis or bibliometrics. Among the measures that have emerged from citation analysis are the citation counts for an individual article, an author, and an academic journal.

The h-index is an author-level metric that attempts to measure both the productivity and citation impact of the publications of a scientist or scholar. The h-index correlates with obvious success indicators such as winning the Nobel Prize, being accepted for research fellowships and holding positions at top universities. The index is based on the set of the scientist's most cited papers and the number of citations that they have received in other publications. The index can also be applied to the productivity and impact of a scholarly journal as well as a group of scientists, such as a department or university or country. The index was suggested in 2005 by Jorge E. Hirsch, a physicist at UC San Diego, as a tool for determining theoretical physicists' relative quality and is sometimes called the Hirsch index or Hirsch number.

Charles Goodhart British economist

Charles Albert Eric Goodhart, is a British economist. He was a member of the Bank of England's Monetary Policy Committee from June 1997 to May 2000 and a professor at the London School of Economics. He is the developer of Goodhart's law, an economic law named after him. He is the son of Arthur Lehman Goodhart, and the brother of William Goodhart and Sir Philip Goodhart.

Campbell's law is an adage developed by Donald T. Campbell, a psychologist and social scientist who often wrote about research methodology, which states:

"The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."

Reference class forecasting or comparison class forecasting is a method of predicting the future by looking at similar past situations and their outcomes. The theories behind reference class forecasting were developed by Daniel Kahneman and Amos Tversky. The theoretical work helped Kahneman win the Nobel Prize in Economics.

Jerome Ravetz American philosopher

Jerome (Jerry) Ravetz is a philosopher of science. He is best known for his books analysing scientific knowledge from a social and ethical perspective, focussing on issues of quality. He is the co-author of the NUSAP notational system and of Post-normal science. He is currently an Associate Fellow at the Institute for Science, Innovation and Society, University of Oxford.

Gaming the system can be defined as using the rules and procedures meant to protect a system to, instead, manipulate the system for a desired outcome.

The McNamara fallacy, named for Robert McNamara, the US Secretary of Defense from 1961 to 1968, involves making a decision based solely on quantitative observations and ignoring all others. The reason given is often that these other observations cannot be proven.

The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can't be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can't be measured easily really isn't important. This is blindness. The fourth step is to say that what can't be easily measured really doesn't exist. This is suicide.

Sensitivity auditing is an extension of sensitivity analysis for use in policy-relevant modelling studies. Its use is recommended - e.g. in the European Commission Impact assessment guidelines and by the European Science Academies - when a sensitivity analysis (SA) of a model-based study is meant to demonstrate the robustness of the evidence provided by the model, but in a context where the inference feeds into a policy or decision-making process.

<i>Science on the Verge</i>

Science on the verge is a book written in 2016 by group of eight scholars working in the tradition of Post-normal science. The book analyzes the main features and possible causes of the present science's crisis.

References

  1. Strathern, Marilyn. "Improving Ratings: Audit in the British University System". European Review. 5 (3): 305–321. doi:10.1002/(SICI)1234-981X(199707)5:3%3C305::AID-EURO184%3E3.0.CO;2-4.
  2. Manheim, David; Garrabrant, Scott (2018). "Categorizing Variants of Goodhart's Law". arXiv: 1803.04585 [cs.AI].
  3. Goodhart, Charles (1981). "Problems of Monetary Management: The U.K. Experience". In Courakis, Anthony S. (ed.). Inflation, Depression, and Economic Policy in the West. pp. 111–146. ISBN   978-0-389-20144-1.
  4. "Overpowered Metrics Eat Underspecified Goals". Ribbonfarm. Accessed 26 January 2017.
  5. Rodamar, Jeffrey (November 2018). "There ought to be a law! Campbell versus Goodhart". Significance. doi: 10.1111/j.1740-9713.2018.01205.x .
  6. Ravetz, Jerome R. (1971). Scientific Knowledge and Its Social Problems. Transaction Publishers. pp. 295–296. ISBN   978-1-4128-3378-3.
  7. Goodhart, C. A. E. (1975). "Problems of Monetary Management: The U.K. Experience". Papers in Monetary Economics. I. Reserve Bank of Australia.
  8. Daníelsson, Jón (July 2002). "The Emperor Has No Clothes: Limits to Risk Modelling". Journal of Banking & Finance. 26 (7): 1273–1296. CiteSeerX   10.1.1.27.3392 . doi:10.1016/S0378-4266(02)00263-7.
  9. Biagioli, Mario (12 July 2016). "Watch out for cheats in citation game" (PDF). Nature. 535 (7611): 201. Bibcode:2016Natur.535..201B. doi:10.1038/535201a. PMID   27411599.
  10. Muller, Jerry Z. (2018). The Tyranny of Metrics. Princeton University Press. ISBN   978-0-691-19126-3.

Further reading