Researcher degrees of freedom

Last updated

Researcher degrees of freedom is a concept referring to the inherent flexibility involved in the process of designing and conducting a scientific experiment, and in analyzing its results. The term reflects the fact that researchers can choose between multiple ways of collecting and analyzing data, and these decisions can be made either arbitrarily or because they, unlike other possible choices, produce a positive and statistically significant result. [1] The researcher degrees of freedom has positives such as affording the ability to look at nature from different angles, allowing new discoveries and hypotheses to be generated. [2] [3] [4] However, researcher degrees of freedom can lead to data dredging and other questionable research practices where the different interpretations and analyses are taken for granted [5] [6] Their widespread use represents an inherent methodological limitation in scientific research, and contributes to an inflated rate of false-positive findings. [1] They can also lead to overestimated effect sizes. [7]

Contents

Though the concept of researcher degrees of freedom has mainly been discussed in the context of psychology, it can affect any scientific discipline. [1] [8] Like publication bias, the existence of researcher degrees of freedom has the potential to lead to an inflated degree of funnel plot asymmetry. [9] It is also a potential explanation for p-hacking, as researchers have so many degrees of freedom to draw on, especially in the social and behavioral sciences. Multiverse analysis is a method that helps bring these degrees of freedom to light. Studies with smaller sample sizes are more susceptible to the biasing influence of researcher degrees of freedom. [10]

Examples

Steegen et al. (2016) showed how, starting from a single raw data set, applying different reasonable data processing decisions can give rise to a multitude of processed data sets (called the data multiverse), often leading to different statistical results. [11] Wicherts et al. (2016) provided a list of 34 degrees of freedom (DFs) researchers have when conducting psychological research. The DFs listed span every stage of the research process, from formulating a hypothesis to the reporting of results. They include conducting exploratory, hypothesis-free research, which the authors note "...pervades many of the researcher DFs that we describe below in the later phases of the study." Other DFs listed in this paper include the creation of multiple manipulated independent variables and the measurement of additional variables that may be selected for analysis later on. [7]

See also

Related Research Articles

Robopsychology is the study of the personalities and behavior of intelligent machines. The term was coined by Isaac Asimov in the short stories collected in I, Robot, which featured robopsychologist Dr. Susan Calvin, and whose plots largely revolved around the protagonist solving problems connected with intelligent robot behaviour. The term has been also used in some academic studies from the field of psychology and human–computer interactions, and it refers to the study of the psychological consequences of living in societies where the application of robotics is becoming increasingly common.

Insight is the understanding of a specific cause and effect within a particular context. The term insight can have several related meanings:

<span class="mw-page-title-main">Self-determination theory</span> Macro theory of human motivation and personality

Self-determination theory (SDT) is a macro theory of human motivation and personality that concerns people's innate growth tendencies and innate psychological needs. It pertains to the motivation behind people's choices in the absence of external influences and distractions. SDT focuses on the degree to which human behavior is self-motivated and self-determined.

Ego depletion is the controversial idea that self-control or willpower draws upon a limited pool of mental resources that can be used up. When the energy for mental activity is low, self-control is typically impaired, which would be considered a state of ego depletion. In particular, experiencing a state of ego depletion impairs the ability to control oneself later on. A depleting task requiring self-control can have a hindering effect on a subsequent self-control task, even if the tasks are seemingly unrelated. Self-control plays a valuable role in the functioning of the self on both individualistic and interpersonal levels. Ego depletion is therefore a critical topic in experimental psychology, specifically social psychology, because it is a mechanism that contributes to the understanding of the processes of human self-control. There have both been studies to support and to question the validity of ego-depletion as a theory.

Type D personality, a concept used in the field of medical psychology, is defined as the joint tendency towards negative affectivity and social inhibition. The letter D stands for "distressed".

The name-letter effect is the tendency of people to prefer the letters in their name over other letters in the alphabet. Whether subjects are asked to rank all letters of the alphabet, rate each of the letters, choose the letter they prefer out of a set of two, or pick a small set of letters they most prefer, on average people consistently like the letters in their own name the most. Crucially, subjects are not aware that they are choosing letters from their name.

The interplay of exercise and music has long been discussed, crossing the disciplines of biomechanics, neurology, physiology, and sport psychology. Research and experimentation on the relation between music and exercise dates back to the early 1900s, when investigator Leonard Ayres found that cyclists pedaled faster in the presence of a band and music, as opposed to when it was silent. Since then, hundreds of studies have been conducted on both the physiological and psychological relationship between music and physical activity, with a number of clear cut relationships and trends emerging. Exercise and music involves the use of music before, during, and/or after performing a physical activity. Listening to music while exercising is done to improve aspects of exercise, such as strength output, exercise duration, and motivation. The use of music during exercise can provide physiological benefits as well as psychological benefits.

The self-expansion model proposes that individuals seek to expand their sense of self by acquiring resources, broadening their perspectives, and increase competency to ultimately optimize their ability to thrive in their environment. It was developed in 1986 by Arthur Aron and Elaine Aron to provide a framework for the underlying experience and behavior in close relationships. The model has two distinct but related core principles: the motivational principle and the inclusion-of-other-in-self principle. The motivational principle refers to an individual's inherent desire to improve their self-efficacy and adapt, survive, and reproduce in their environment. The inclusion-of-other-in-self principle posits that close relationships serve as the primary way to expand our sense of self as we incorporate the identities, perspectives, resources, and experiences of others as our own through these relationships.

<span class="mw-page-title-main">Replication crisis</span> Observed inability to reproduce scientific studies

The replication crisis is an ongoing methodological crisis in which the results of many scientific studies are difficult or impossible to reproduce. Because the reproducibility of empirical results is an essential part of the scientific method, such failures undermine the credibility of theories building on them and potentially call into question substantial parts of scientific knowledge.

Relief is a positive emotion experienced when something unpleasant, painful or distressing has not happened or has come to an end.

<span class="mw-page-title-main">Facial width to height ratio</span>

Facial width to height ratio (fWHR) is a measure of the width of a person’s face compared to its height. Research has shown that higher FWHR is associated with various physical and behavioral traits, such as adolescent testosterone, aggression, attractiveness to women, cause of death by violence, CEO success as measured by organizational and financial performance, and success in sports. While most studies have found some significance, other studies found little correlation. The metric has also been used in primate studies with similar findings.

The Meta-Research Center at Tilburg University is a metascience research center within the School of Social and Behavioral Sciences at the Dutch Tilburg University. They were profiled in a September 2018 article in Science.

Jelte Michiel "J.M." Wicherts is a Dutch psychologist and professor in the Tilburg School of Social and Behavioral Sciences at Tilburg University. His research interests include biases in decision making, as well as scientific misconduct and reproducibility. He has also researched group differences in IQ scores and the Flynn effect.

<span class="mw-page-title-main">Preregistration (science)</span>

Preregistration is the practice of registering the hypotheses, methods, and/or analyses of a scientific study before it is conducted. Clinical trial registration is similar, although it may not require the registration of a study's analysis protocol. Finally, registered reports include the peer review and in principle acceptance of a study protocol prior to data collection.

HARKing is an acronym coined by social psychologist Norbert Kerr that refers to the questionable research practice of “presenting a post hoc hypothesis in the introduction of a research report as if it were an a priori hypothesis”. Hence, a key characteristic of HARKing is that post hoc hypothesizing is falsely portrayed as a priori hypothesizing. HARKing may occur when a researcher tests an a priori hypothesis but then omits that hypothesis from their research report after they find out the results of their test; inappropriate forms of post hoc analysis and/or post hoc theorizing then may lead to a post hoc hypothesis.

Crowdsourced science refers to collaborative contributions of a large group of people to the different steps of the research process in science. In psychology, the nature and scope of the collaborations can vary in their application and in the benefits it offers.

Sharlene D. Newman is an American cognitive neuroscientist, executive director of the Alabama Life Research Institute at the University of Alabama (UA), Professor in the Department of Psychology at UA, and an adjunct professor in the Department of Psychological and Brain Sciences at Indiana University.

The reverse correlation technique is a data driven study method used primarily in psychological and neurophysiological research. This method earned its name from its origins in neurophysiology, where cross-correlations between white noise stimuli and sparsely occurring neuronal spikes could be computed quicker when only computing it for segments preceding the spikes. The term has since been adopted in psychological experiments that usually do not analyze the temporal dimension, but also present noise to human participants. In contrast to the original meaning, the term is here thought to reflect that the standard psychological practice of presenting stimuli of defined categories to the participants is "reversed": Instead, the participant's mental representations of categories are estimated from interactions of the presented noise and the behavioral responses. It is used to create composite pictures of individual and/or group mental representations of various items that depict characteristics of said items. This technique is helpful when evaluating the mental representations of those with and without mental illnesses.

Do-gooder derogation is a phenomenon where a person's morally motivated behavior leads to them being perceived negatively by others. The term "do-gooder" refers to a person who deviates from the majority in terms of behavior, because of their morality.

Multiverse analysis is a scientific method that specifies and then runs a set of plausible alternative models or statistical tests for a single hypothesis. It is a method to address the issue that the "scientific process confronts researchers with a multiplicity of seemingly minor, yet nontrivial, decision points, each of which may introduce variability in research outcomes". A problem also known as Researcher degrees of freedom. It is a method arising in response to the credibility and replication crisis taking place in science, because it can diagnose p-hacking. It is also a form of meta-analysis allowing researchers to provide evidence on how different model specifications impact results for the same hypothesis, and thus can point scientists toward where they might need better theory or causal models.

References

  1. 1 2 3 Simmons, Joseph P.; Nelson, Leif D.; Simonsohn, Uri (November 2011). "False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant". Psychological Science. 22 (11): 1359–1366. doi:10.1177/0956797611417632. ISSN   0956-7976. PMID   22006061. S2CID   13802986.
  2. ↑ Tukey, J.W. (1977). Exploratory data analysis Volume 2. Pearson.
  3. ↑ Box, George E. P. (December 1976). "Science and Statistics". Journal of the American Statistical Association. 71 (356): 791–799. doi:10.2307/2286841. JSTOR   2286841.
  4. ↑ Groot, Adriaan D. de; Groot, Adrianus Dingeman de (1978). Thought and choice in chess. Psychological studies (2 ed.). The Hague: Mouton. ISBN   978-90-279-7914-8.
  5. ↑ Wicherts, Jelte M.; Veldkamp, Coosje L. S.; Augusteijn, Hilde E. M.; Bakker, Marjan; van Aert, Robbie C. M.; van Assen, Marcel A. L. M. (2016). "Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking". Frontiers in Psychology. 7: 1832. doi: 10.3389/fpsyg.2016.01832 . ISSN   1664-1078. PMC   5122713 . PMID   27933012.
  6. ↑ Simmons, Joseph P.; Nelson, Leif D.; Simonsohn, Uri (2011-10-17). "False-Positive Psychology". Psychological Science. 22 (11): 1359–1366. doi:10.1177/0956797611417632. ISSN   0956-7976. PMID   22006061. S2CID   13802986.
  7. 1 2 Wicherts, Jelte M.; Veldkamp, Coosje L. S.; Augusteijn, Hilde E. M.; Bakker, Marjan; van Aert, Robbie C. M.; van Assen, Marcel A. L. M. (2016-11-25). "Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking". Frontiers in Psychology. 7: 1832. doi: 10.3389/fpsyg.2016.01832 . ISSN   1664-1078. PMC   5122713 . PMID   27933012.
  8. ↑ Coretta, Stefano; Casillas, Joseph V.; Roessig, Simon; Franke, Michael; Ahn, Byron; Al-Hoorie, Ali H.; Al-Tamimi, Jalal; Alotaibi, Najd E.; AlShakhori, Mohammed K.; Altmiller, Ruth M.; Arantes, Pablo; Athanasopoulou, Angeliki; Baese-Berk, Melissa M.; Bailey, George; Sangma, Cheman Baira A (July 2023). "Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses". Advances in Methods and Practices in Psychological Science. 6 (3). doi: 10.1177/25152459231162567 . ISSN   2515-2459.
  9. ↑ Carter, Evan C.; McCullough, Michael E. (2014-07-30). "Publication bias and the limited strength model of self-control: has the evidence for ego depletion been overestimated?". Frontiers in Psychology. 5: 823. doi: 10.3389/fpsyg.2014.00823 . ISSN   1664-1078. PMC   4115664 . PMID   25126083.
  10. ↑ Schweizer, Geoffrey; Furley, Philip (March 2016). "Reproducible research in sport and exercise psychology: The role of sample sizes". Psychology of Sport and Exercise. 23: 114–122. doi:10.1016/j.psychsport.2015.11.005.
  11. ↑ Steegen, Sara; Tuerlinckx, Francis; Gelman, Andrew; Vanpaemel, Wolf (September 2016). "Increasing Transparency Through a Multiverse Analysis". Perspectives on Psychological Science. 11 (5): 702–712. doi: 10.1177/1745691616658637 . ISSN   1745-6916. PMID   27694465.
Fisher iris versicolor sepalwidth.svg

This statistics-related article is a stub. You can help Wikipedia by expanding it.

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.