HARKing

Last updated

HARKing (hypothesizing after the results are known) is an acronym coined by social psychologist Norbert Kerr [1] that refers to the questionable research practice of "presenting a post hoc hypothesis in the introduction of a research report as if it were an a priori hypothesis". [1] [2] Hence, a key characteristic of HARKing is that post hoc hypothesizing is falsely portrayed as a priori hypothesizing. [3] HARKing may occur when a researcher tests an a priori hypothesis but then omits that hypothesis from their research report after they find out the results of their test; inappropriate forms of post hoc analysis or post hoc theorizing then may lead to a post hoc hypothesis.

Contents

Types

Several types of HARKing have been distinguished, including:

THARKing
Transparently hypothesizing after the results are known, rather than the secretive, undisclosed, HARKing that was first proposed by Kerr. In this case, researchers openly declare that they developed their hypotheses after they observed their research results. [4]
CHARKing (or Pure HARKing)
CHARKing [5] or "pure HARKing" [1] refers to the practice of constructing new hypotheses after the results are known and presenting them as a priori hypotheses. [1] [5] CHARKing is often regarded as the prototypical form of HARKing.
RHARKing
RHARKing refers to retrieving old hypotheses from the existing literature after the results are known and presenting them as a priori hypotheses [5] Note that RHARKed hypotheses can be considered to be a priori hypotheses in the sense that they were developed and published prior to knowledge of the current research results. [5] [6]
SHARKing
Suppressing a priori hypotheses after the results of tests of those hypotheses are known. [1] [5]
Active and passive HARKing
Active HARKing occurs when researchers HARK prior to submitting their research report for publication. Passive HARKing occurs when researchers HARK in response to requests by editors and reviewers during the peer review process. [5] : 317 

Prevalence among researchers

Citations to the original Kerr (1998) article on HARKing Kerr (1998).png
Citations to the original Kerr (1998) article on HARKing

Concerns about HARKing appear to be increasing in the scientific community, as shown by the increasing number of citations to Kerr's seminal article. [7] A 2017 review of six surveys found that an average of 43% of researchers reported HARKing "at least once". [5] This figure may be an underestimate if researchers are concerned about reporting questionable research practices, do not perceive themselves to be responsible for HARKing that is proposed by editors and reviewers (i.e., passive HARKing), or do not recognize their HARKing due to hindsight or confirmation biases.

Researchers' motivation

HARKing appears to be motivated by a desire to publish research in a publication environment that values a priori hypotheses over post hoc hypotheses and contains a publication bias against null results. In order to improve their chances of publishing their results, researchers may secretly suppress any a priori hypotheses that fail to yield significant results, construct or retrieve post hoc hypotheses that account for any unexpected significant results, and then present these new post hoc hypotheses in their research reports as if they are a priori hypotheses. [1] [8] [9] [5] [10]

Prediction and accommodation

HARKing is associated with the debate regarding prediction and accommodation. [11] In the case of prediction, hypotheses are deduced from a priori theory and evidence. In the case of accommodation, hypotheses are induced from the current research results. [7] One view is that HARKing represents a form of accommodation in which researchers induce ad hoc hypotheses from their current results. [1] [3] Another view is that HARKing represents a form of prediction in which researchers deduce hypotheses from a priori theory and evidence after they know their current results. [7]

Potential costs to science

Potential costs of HARKing include: [1] : 211 

  1. Translating Type I errors into hard-to-eradicate theory
  2. Propounding theories that cannot (pending replication) pass Popper's disconfirmability test
  3. Disguising post hoc explanations as a priori explanations
  4. Not communicating valuable information about what did not work
  5. Taking unjustified statistical licence
  6. Presenting an inaccurate model of science to students
  7. Encouraging questionable practices in other grey areas
  8. Making us less receptive to serendipitous findings
  9. Encouraging adoption of narrow, context-bound new theory
  10. Encouraging retention of too-broad, disconfirmable old theory
  11. Inhibiting identification of plausible alternative hypotheses
  12. Implicitly violating basic ethical principles

In 2022, Rubin provided a critical analysis of Kerr's 12 costs of HARKing. He concluded that these costs "are either misconceived, misattributed to HARKing, lacking evidence, or that they do not take into account pre- and post-publication peer review and public availability to research materials and data." [7]

HARKing and the replication crisis

Some of the costs of HARKing are thought to have led to the replication crisis in science. [4] Hence, Bishop described HARKing as one of "the four horsemen of the reproducibility apocalypse," with publication bias, low statistical power, and p-hacking [12] being the other three. [13] An alternative view is that it is premature to conclude that HARKing has contributed to the replication crisis. [7] [5] [14]

The preregistration of research hypotheses prior to data collection has been proposed as a method of identifying and deterring HARKing. However, the use of preregistration to prevent HARKing is controversial. [3]

Ethical concerns

Kerr pointed out that "HARKing can entail concealment. The question then becomes whether what is concealed in HARKing can be a useful part of the 'truth' ...or is instead basically uninformative (and may, therefore, be safely ignored at an author's discretion)". [1] : 209  Three different positions about the ethics of HARKing depend on whether HARKing conceals "a useful part of the 'truth'".

The first position is that all HARKing is unethical under all circumstances because it violates a fundamental principle of communicating scientific research honestly and completely. [1] : 209  According to this position, HARKing always conceals a useful part of the truth.

A second position, mostly supported by perpetrators and profiteers, is that HARKing falls into a "gray zone" of ethical practice. [1] [15] According to this position, some forms of HARKing are more or less ethical under some circumstances. [16] [5] [17] [7] Hence, only some forms of HARKing conceal a useful part of the truth under some conditions. Consistent with this view, a 2018 survey of 119 USA researchers found that HARKing ("reporting an unexpected result as having been hypothesized from the start") was associated with "ambiguously unethical" research practices more than with "unambiguously unethical" research practices. [18]

A third position is that HARKing is acceptable provided that hypotheses are explicitly deduced from a priori theory and evidence, as explained in a theoretical rationale, and readers have access to the relevant research data and materials. [7] According to this position, HARKing does not prevent readers from making an adequately informed evaluation of the theoretical quality and plausibility of the HARKed hypotheses and the methodological rigor with which the hypotheses have been tested. [7] [17] In this case, HARKing does not conceal a useful part of the truth. Furthermore, researchers may claim that a priori theory and evidence predict their results even if the prediction is deduced after they know their results. [7] [19]

See also

Related Research Articles

The scientific method is an empirical method for acquiring knowledge that has characterized the development of science since at least the 17th century. The scientific method involves careful observation coupled with rigorous scepticism, because cognitive assumptions can distort the interpretation of the observation. Scientific inquiry includes creating a hypothesis through inductive reasoning, testing it through experiments and statistical analysis, and adjusting or discarding the hypothesis based on the results.

Social psychology is the scientific study of how thoughts, feelings, and behaviors are influenced by the actual, imagined, or implied presence of others. Social psychologists typically explain human behavior as a result of the relationship between mental states and social situations, studying the social conditions under which thoughts, feelings, and behaviors occur, and how these variables influence social interactions.

<span class="mw-page-title-main">Statistical hypothesis test</span> Method of statistical inference

A statistical hypothesis test is a method of statistical inference used to decide whether the data sufficiently support a particular hypothesis. A statistical hypothesis test typically involves a calculation of a test statistic. Then a decision is made, either by comparing the test statistic to a critical value or equivalently by evaluating a p-value computed from the test statistic. Roughly 100 specialized statistical tests have been defined.

The phrase "correlation does not imply causation" refers to the inability to legitimately deduce a cause-and-effect relationship between two events or variables solely on the basis of an observed association or correlation between them. The idea that "correlation implies causation" is an example of a questionable-cause logical fallacy, in which two events occurring together are taken to have established a cause-and-effect relationship. This fallacy is also known by the Latin phrase cum hoc ergo propter hoc. This differs from the fallacy known as post hoc ergo propter hoc, in which an event following another is seen as a necessary consequence of the former event, and from conflation, the errant merging of two events, ideas, databases, etc., into one.

Confirmation bias is the tendency to search for, interpret, favor, and recall information in a way that confirms or supports one's prior beliefs or values. People display this bias when they select information that supports their views, ignoring contrary information, or when they interpret ambiguous evidence as supporting their existing attitudes. The effect is strongest for desired outcomes, for emotionally charged issues, and for deeply entrenched beliefs. Confirmation bias is insuperable for most people, but they can manage it, for example, by education and training in critical thinking skills.

<span class="mw-page-title-main">Experiment</span> Scientific procedure performed to validate a hypothesis

An experiment is a procedure carried out to support or refute a hypothesis, or determine the efficacy or likelihood of something previously untried. Experiments provide insight into cause-and-effect by demonstrating what outcome occurs when a particular factor is manipulated. Experiments vary greatly in goal and scale but always rely on repeatable procedure and logical analysis of the results. There also exist natural experimental studies.

The Texas sharpshooter fallacy is an informal fallacy which is committed when differences in data are ignored, but similarities are overemphasized. From this reasoning, a false conclusion is inferred. This fallacy is the philosophical or rhetorical application of the multiple comparisons problem and apophenia. It is related to the clustering illusion, which is the tendency in human cognition to interpret patterns where none actually exist.

In published academic research, publication bias occurs when the outcome of an experiment or research study biases the decision to publish or otherwise distribute it. Publishing only results that show a significant finding disturbs the balance of findings in favor of positive results. The study of publication bias is an important topic in metascience.

<span class="mw-page-title-main">Data dredging</span> Misuse of data analysis

Data dredging is the misuse of data analysis to find patterns in data that can be presented as statistically significant, thus dramatically increasing and understating the risk of false positives. This is done by performing many statistical tests on the data and only reporting those that come back with significant results.

Social comparison theory, initially proposed by social psychologist Leon Festinger in 1954, centers on the belief that individuals drive to gain accurate self-evaluations. The theory explains how individuals evaluate their opinions and abilities by comparing themselves to others to reduce uncertainty in these domains and learn how to define the self. Comparing oneself to others socially is a form of measurement and self-assessment to identify where an individual stands according to their own set of standards and emotions about themselves.

In science and philosophy, a just-so story is an untestable narrative explanation for a cultural practice, a biological trait, or behavior of humans or other animals. The pejorative nature of the expression is an implicit criticism that reminds the listener of the fictional and unprovable nature of such an explanation. Such tales are common in folklore genres like mythology. A less pejorative term is a pourquoi story, which has been used to describe usually more mythological or otherwise traditional examples of this genre, aimed at children.

<span class="mw-page-title-main">Paul E. Meehl</span> American psychologist (1920–2003)

Paul Everett Meehl was an American clinical psychologist. He was the Hathaway and Regents' Professor of Psychology at the University of Minnesota, and past president of the American Psychological Association. A Review of General Psychology survey, published in 2002, ranked Meehl as the 74th most cited psychologist of the 20th century, in a tie with Eleanor J. Gibson. Throughout his nearly 60-year career, Meehl made seminal contributions to psychology, including empirical studies and theoretical accounts of construct validity, schizophrenia etiology, psychological assessment, behavioral prediction, metascience, and philosophy of science.

<span class="mw-page-title-main">Duhem–Quine thesis</span> Principle in the philosophy of science

In philosophy of science, the Duhem–Quine thesis, also called the Duhem–Quine problem, posits that it is impossible to experimentally test a scientific hypothesis in isolation, because an empirical test of the hypothesis requires one or more background assumptions : the thesis says that unambiguous scientific falsifications are impossible. It is named after French theoretical physicist Pierre Duhem and American logician Willard Van Orman Quine, who wrote about similar concepts.

<span class="mw-page-title-main">Research design</span> Overall strategy utilized to carry out research

Research design refers to the overall strategy utilized to answer research questions. A research design typically outlines the theories and models underlying a project; the research question(s) of a project; a strategy for gathering data and information; and a strategy for producing answers from the data. A strong research design yields valid answers to research questions while weak designs yield unreliable, imprecise or irrelevant answers.

Evolutionary psychology seeks to identify and understand human psychological traits that have evolved in much the same way as biological traits, through adaptation to environmental cues. Furthermore, it tends toward viewing the vast majority of psychological traits, certainly the most important ones, as the result of past adaptions, which has generated significant controversy and criticism from competing fields. These criticisms include disputes about the testability of evolutionary hypotheses, cognitive assumptions such as massive modularity, vagueness stemming from assumptions about the environment that leads to evolutionary adaptation, the importance of non-genetic and non-adaptive explanations, as well as political and ethical issues in the field itself.

<span class="mw-page-title-main">Hypothesis</span> Proposed explanation for an observation, phenomenon, or scientific problem

A hypothesis is a proposed explanation for a phenomenon. For a hypothesis to be a scientific hypothesis, the scientific method requires that one can test it. Scientists generally base scientific hypotheses on previous observations that cannot satisfactorily be explained with the available scientific theories. Even though the words "hypothesis" and "theory" are often used interchangeably, a scientific hypothesis is not the same as a scientific theory. A working hypothesis is a provisionally accepted hypothesis proposed for further research in a process beginning with an educated guess or thought.

<span class="mw-page-title-main">Replication crisis</span> Observed inability to reproduce scientific studies

The replication crisis is an ongoing methodological crisis in which the results of many scientific studies are difficult or impossible to reproduce. Because the reproducibility of empirical results is an essential part of the scientific method, such failures undermine the credibility of theories building on them and potentially call into question substantial parts of scientific knowledge.

Preregistration is the practice of registering the hypotheses, methods, and/or analyses of a scientific study before it is conducted. Clinical trial registration is similar, although it may not require the registration of a study's analysis protocol. Finally, registered reports include the peer review and in principle acceptance of a study protocol prior to data collection.

Researcher degrees of freedom is a concept referring to the inherent flexibility involved in the process of designing and conducting a scientific experiment, and in analyzing its results. The term reflects the fact that researchers can choose between multiple ways of collecting and analyzing data, and these decisions can be made either arbitrarily or because they, unlike other possible choices, produce a positive and statistically significant result. The researcher degrees of freedom has positives such as affording the ability to look at nature from different angles, allowing new discoveries and hypotheses to be generated. However, researcher degrees of freedom can lead to data dredging and other questionable research practices where the different interpretations and analyses are taken for granted Their widespread use represents an inherent methodological limitation in scientific research, and contributes to an inflated rate of false-positive findings. They can also lead to overestimated effect sizes.

Crowdsourced science refers to collaborative contributions of a large group of people to the different steps of the research process in science. In psychology, the nature and scope of the collaborations can vary in their application and in the benefits it offers.

References

  1. 1 2 3 4 5 6 7 8 9 10 11 Kerr, N. L. (1998). "HARKing: Hypothesizing after the results are known". Personality and Social Psychology Review. 2 (3): 196–217. doi: 10.1207/s15327957pspr0203_4 . PMID   15647155.
  2. ↑ John, L. K.; Loewenstein, G.; Prelec, D. (2012). "Measuring the prevalence of questionable research practices with incentives for truth telling". Psychological Science. 23 (5): 524–532. doi:10.1177/0956797611430953. PMID   22508865. S2CID   8400625.
  3. 1 2 3 Lishner, D. A. (2021). "HARKing: Conceptualizations, harms, and two fundamental remedies". Journal of Theoretical and Philosophical Psychology. 41 (4): 248–263. doi:10.1037/teo0000182. S2CID   234886782.
  4. 1 2 Hollenbeck, J. R.; Wright, P. M. (2017). "Harking, sharking, and tharking: Making the case for post hoc analysis of scientific data". Journal of Management. 43: 5–18. doi: 10.1177/0149206316679487 .
  5. 1 2 3 4 5 6 7 8 9 10 Rubin, M. (2017). "When does HARKing hurt? Identifying when different types of undisclosed post hoc hypothesizing harm scientific progress". Review of General Psychology. 21 (4): 308–320. doi:10.1037/gpr0000128. S2CID   149228437.
  6. ↑ Vancouver, J. B. (2020). "Navigating the review process through the holier than thou". Industrial and Organizational Psychology. 13: 72–75. doi:10.1017/iop.2020.8. S2CID   219044743.
  7. 1 2 3 4 5 6 7 8 9 Rubin, M. (2022). "The costs of HARKing" (PDF). British Journal for the Philosophy of Science. 73 (2): 535–560. doi:10.1093/bjps/axz050.
  8. ↑ Mazzola, J. J.; Deuling, J. K. (2013). "Forgetting what we learned as graduate students: HARKing and selective outcome reporting in I–O journal articles". Industrial and Organizational Psychology: Perspectives on Science and Practice. 6 (3): 279–284. doi:10.1111/iops.12049. S2CID   142493071.
  9. ↑ O'Boyle, E. H. Jr.; Banks, G. C.; Gonzalez-Mulé, E. (2017). "The chrysalis effect: How ugly initial results metamorphosize into beautiful articles". Journal of Management. 43: 367–399. doi:10.1177/0149206314527133. S2CID   145237761.
  10. ↑ Cairo, A. H.; Green, J. D.; Forsyth, D. R.; Behler, A. C.; Raldiris, T. L. (2020). "Gray (literature) matters: Evidence of selective hypothesis reporting in social psychological research". Personality and Social Psychology Bulletin. 46 (9): 1344–1362. doi:10.1177/0146167220903896. PMID   32093574. S2CID   211475516.
  11. ↑ Barnes, Eric Christian (2022), "Prediction versus Accommodation", in Zalta, Edward N.; Nodelman, Uri (eds.), The Stanford Encyclopedia of Philosophy (Winter 2022 ed.), Metaphysics Research Lab, Stanford University, retrieved 2022-10-28
  12. ↑ Simmons, Joseph P.; Nelson, Leif D.; Simonsohn, Uri (17 October 2011). "False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant". Psychological Science. 22 (11): 1359–1366. doi: 10.1177/0956797611417632 . PMID   22006061.
  13. ↑ Bishop, D. (2019). "Rein in the four horsemen of irreproducibility". Nature. 568 (7753): 435. Bibcode:2019Natur.568..435B. doi: 10.1038/d41586-019-01307-2 . PMID   31019328.
  14. ↑ Mohseni, A. (2020). "HARKing: From misdiagnosis to misprescription" (PDF). Retrieved 12 November 2020.{{cite journal}}: Cite journal requires |journal= (help)
  15. ↑ Butler, N.; Delaney, H.; Spoelstra, S. (2017). "The gray zone: Questionable research practices in the business school". Academy of Management Learning & Education. 16: 94–109. doi:10.5465/amle.2015.0201.
  16. ↑ Leung, K. (2011). "Presenting post hoc hypotheses as a priori: Ethical and theoretical issues". Management and Organization Review. 7: 471–479. doi:10.1017/CBO9781139171434.009.
  17. 1 2 Vancouver, J. N. (2018). "In defense of HARKing". Industrial and Organizational Psychology. 11: 73–80. doi:10.1017/iop.2017.89. S2CID   149220988.
  18. ↑ Sacco, D. F.; Bruton, S. V.; Brown, M. (2018). "In defense of the questionable: Defining the basis of research scientists' engagement in questionable research practices". Responsible Conduct of Research and Research Integrity. 13 (1): 101–110. doi: 10.1177/1556264617743834 . PMID   29179623.
  19. ↑ Worrall, J. (2014). "Prediction and accommodation revisited". Studies in History and Philosophy of Science. 45: 54–61. Bibcode:2014SHPSA..45...54W. doi:10.1016/j.shpsa.2013.10.001. PMID   24984450.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.