Outcome switching

Last updated

Outcome switching is the practice of changing the primary or secondary outcomes of a clinical trial after its initiation. An outcome is the goal of the clinical trial, such as survival after five years for cancer treatment. Outcome switching can lead to bias and undermine the reliability of the trial, for instance when outcomes are switched after researchers already have access to trial data. That way, researchers can cherry pick an outcome which is statistically significant.

Contents

Problem

Outcome switching can undermine the reliability of the trial, for instance when outcomes are switched when researchers already have access to the trial data. It can lead to bias in terms of benefits and harms. For example, when the findings using the original protocol were statistically insignificant, a study may cherry pick a new outcome measure that is statistically significant. [1] If there are a large number of outcomes to choose from, it is likely at least one will show significant findings, assuming no correction is made for multiple testing. [2] Incomplete or misleading reporting of outcomes is an example of unethical research practice. [1]

Possible solutions

Primary outcomes for clinical trials need to be defined upfront to prevent a biased selection of outcomes. Some medical journals require adherence to the CONSORT standards, which require authors to document any deviations from the initial study protocol and explain why changes were made. For instance, JAMA and the British Medical Journal require inclusion of the CONSORT checklist. [1] When pre-specified outcomes are defined but unclear, researchers still have wriggle room to choose the most favourable option. A clear pre-defined outcome is therefore important to avoid outcome switching. [3]

Study authors may engage in outcome switching due to high pressures to publish. It is more difficult to publish negative results, especially in high-impact journals. Registered reports—a type of journal article where editors pledge to publish clinical trials irrespective of their results—may help combat this pressure to produce positive results. Initial peer review takes place based on the methodology and the reasoning behind the study. [4]

Prevalence

Despite problems with outcome switching, the practice is common. Changes in primary outcome metrics are present in nearly one in three studies. [5] Outcome switching also occurs frequently in follow-up studies. [6] In an analysis of oncology trials, outcome switching was more common in studies with a male first author, and in studies funded by non-profits. [7] One study analysed outcome switching in five top medical journals, writing letters for each misreported trial outcome. Journal editors and clinical trial authors typically responded dismissively when concerns were raised, misrepresenting ethical standards and including ad hominem attacks. [8]

Examples

A drug against major depressive disorder, paroxetine, was marketed for children and teenagers for years, based on a flawed trial that involved outcome switching. [9] The trial's protocol had described two primary and six secondary outcomes by which it would measure efficacy. [10] [11] The data showed that, according to those eight outcomes, paroxetine was no more effective than placebo. According to Melanie Newman, writing for the BMJ , "[t]he drug only produced a positive result when four new secondary outcome measures, which were introduced following the initial data analysis, were used instead. Fifteen other new secondary outcome measures failed to throw up positive results." [lower-alpha 1] [11]

See also

Related Research Articles

<span class="mw-page-title-main">Antidepressant</span> Class of medication used to treat depression and other conditions

Antidepressants are a class of medications used to treat major depressive disorder, anxiety disorders, chronic pain, and addiction.

<span class="mw-page-title-main">Meta-analysis</span> Statistical method that summarizes data from multiple sources

Meta-analysis is the statistical combination of the results of multiple studies addressing a similar research question. An important part of this method involves computing an effect size across all of the studies; this involves extracting effect sizes and variance measures from various studies. Meta-analyses are integral in supporting research grant proposals, shaping treatment guidelines, and influencing health policies. They are also pivotal in summarizing existing research to guide future studies, thereby cementing their role as a fundamental methodology in metascience. Meta-analyses are often, but not always, important components of a systematic review procedure. For instance, a meta-analysis may be conducted on several clinical trials of a medical treatment, in an effort to obtain a better understanding of how well the treatment works.

<span class="mw-page-title-main">Paroxetine</span> SSRI antidepressant medication

Paroxetine, sold under the brand names Paxil and Seroxat among others, is an antidepressant of the selective serotonin reuptake inhibitor (SSRI) class. It is used to treat major depressive disorder, obsessive-compulsive disorder, panic disorder, social anxiety disorder, post-traumatic stress disorder, generalized anxiety disorder, and premenstrual dysphoric disorder. It has also been used in the treatment of premature ejaculation and hot flashes due to menopause. It is taken orally.

<span class="mw-page-title-main">Randomized controlled trial</span> Form of scientific experiment

A randomized controlled trial is a form of scientific experiment used to control factors not under direct experimental control. Examples of RCTs are clinical trials that compare the effects of drugs, surgical techniques, medical devices, diagnostic procedures, diets or other medical treatments.

In a blind or blinded experiment, information which may influence the participants of the experiment is withheld until after the experiment is complete. Good blinding can reduce or eliminate experimental biases that arise from a participants' expectations, observer's effect on the participants, observer bias, confirmation bias, and other sources. A blind can be imposed on any participant of an experiment, including subjects, researchers, technicians, data analysts, and evaluators. In some cases, while blinding would be useful, it is impossible or unethical. For example, it is not possible to blind a patient to their treatment in a physical therapy intervention. A good clinical protocol ensures that blinding is as effective as possible within ethical and practical constraints.

In published academic research, publication bias occurs when the outcome of an experiment or research study biases the decision to publish or otherwise distribute it. Publishing only results that show a significant finding disturbs the balance of findings in favor of positive results. The study of publication bias is an important topic in metascience.

A pilot experiment, pilot study, pilot test or pilot project is a small-scale preliminary study conducted to evaluate feasibility, duration, cost, adverse events, and improve upon the study design prior to performance of a full-scale research project.

In medicine an intention-to-treat (ITT) analysis of the results of a randomized controlled trial is based on the initial treatment assignment and not on the treatment eventually received. ITT analysis is intended to avoid various misleading artifacts that can arise in intervention research such as non-random attrition of participants from the study or crossover. ITT is also simpler than other forms of study design and analysis, because it does not require observation of compliance status for units assigned to different treatments or incorporation of compliance into the analysis. Although ITT analysis is widely employed in published clinical trials, it can be incorrectly described and there are some issues with its application. Furthermore, there is no consensus on how to carry out an ITT analysis in the presence of missing outcome data.

In a randomized experiment, allocation concealment hides the sorting of trial participants into treatment groups so that this knowledge cannot be exploited. Adequate allocation concealment serves to prevent study participants from influencing treatment allocations for subjects. Studies with poor allocation concealment are prone to selection bias.

<span class="mw-page-title-main">Randomized experiment</span> Experiment using randomness in some aspect, usually to aid in removal of bias

In science, randomized experiments are the experiments that allow the greatest reliability and validity of statistical estimates of treatment effects. Randomization-based inference is especially important in experimental design and in survey sampling.

In epidemiology, reporting bias is defined as "selective revealing or suppression of information" by subjects. In artificial intelligence research, the term reporting bias is used to refer to people's tendency to under-report all the information available.

Clinical trials are medical research studies conducted on human subjects. The human subjects are assigned to one or more interventions, and the investigators evaluate the effects of those interventions. The progress and results of clinical trials are analyzed statistically.

<span class="mw-page-title-main">Selective serotonin reuptake inhibitor</span> Class of antidepressant medication

Selective serotonin reuptake inhibitors (SSRIs) are a class of drugs that are typically used as antidepressants in the treatment of major depressive disorder, anxiety disorders, and other psychological conditions.

<i>Journal of the American Academy of Child and Adolescent Psychiatry</i> Peer-reviewed scientific journal

The Journal of the American Academy of Child and Adolescent Psychiatry is a peer-reviewed medical journal covering pediatric psychiatry. It is published by Elsevier and is the official journal of the American Academy of Child and Adolescent Psychiatry. The editor-in-chief is Douglas Novins.

The relationship between antidepressant use and suicide risk is a subject of medical research and has faced varying levels of debate. This problem was thought to be serious enough to warrant intervention by the U.S. Food and Drug Administration to label greater likelihood of suicide as a risk of using antidepressants. Some studies have shown that the use of certain antidepressants correlate with an increased risk of suicide in some patients relative to other antidepressants. However, these conclusions have faced considerable scrutiny and disagreement: A multinational European study indicated that antidepressants decrease risk of suicide at the population level, and other reviews of antidepressant use claim that there is not enough data to indicate antidepressant use increases risk of suicide.

<span class="mw-page-title-main">Study 329</span> Scientific article

Study 329 was a clinical trial which was conducted in North America from 1994 to 1998 to study the efficacy of paroxetine, an SSRI anti-depressant, in treating 12- to 18-year-olds diagnosed with major depressive disorder. Led by Martin Keller, then professor of psychiatry at Brown University, and funded by the British pharmaceutical company SmithKline Beecham—known since 2000 as GlaxoSmithKline (GSK)—the study compared paroxetine with imipramine, a tricyclic antidepressant, and placebo. SmithKline Beecham had released paroxetine in 1991, marketing it as Paxil in North America and Seroxat in the UK. The drug attracted sales of $11.7 billion in the United States alone from 1997 to 2006, including $2.12 billion in 2002, the year before it lost its patent.

The Centre for Evidence-Based Medicine (CEBM), based in the Nuffield Department of Primary Care Health Sciences at the University of Oxford, is an academic-led centre dedicated to the practice, teaching, and dissemination of high quality evidence-based medicine to improve healthcare in everyday clinical practice. CEBM was founded by David Sackett in 1995. It was subsequently directed by Brian Haynes and Paul Glasziou. Since 2010 it has been led by Professor Carl Heneghan, a clinical epidemiologist and general practitioner.

In medicine, a stepped-wedge trial is a type of randomised controlled trial (RCT). An RCT is a scientific experiment that is designed to reduce bias when testing a new medical treatment, a social intervention, or another testable hypothesis.

<span class="mw-page-title-main">Clinical Trials Registry – India</span> Indias clinical trial registry

Clinical Trials Registry – India (CTRI) is the government of India's official clinical trial registry. The National Institute of Medical Statistics of the Indian Council of Medical Research established the CTRI on 20 July 2007. Since 2009 the Central Drugs Standard Control Organisation has mandated that anyone conducting clinical trials in India must preregister before enrolling any research participants.

References

  1. ↑ Melanie Newman, BMJ, 2010: "Study 329's results showed that paroxetine was no more effective than the placebo according to measurements of eight outcomes specified by Martin Keller, professor of psychiatry at Brown University, when he first drew up the trial.
    "Two of these were primary outcomes: the change in total Hamilton Rating Scale (HAM-D) score, and the proportion of 'responders' at the end of the eight-week acute treatment phase (those with a ≥50% reduction in HAM-D or a HAM-D score ≤8). The drug also showed no significant effect for the initial six secondary outcome measures.
    "The drug only produced a positive result when four new secondary outcome measures, which were introduced following the initial data analysis, were used instead. Fifteen other new secondary outcome measures failed to throw up positive results.". [12]
  1. 1 2 3 Altman DG, Moher D, Schulz KF (February 2017). "Harms of outcome switching in reports of randomised trials: CONSORT perspective". BMJ (Clinical Research Ed.). 356: j396. doi:10.1136/bmj.j396. PMID   28196813. S2CID   206916748.
  2. ↑ Wicherts JM (2017). "The Weak Spots in Contemporary Science (and How to Fix Them)". Animals. 7 (12): 90. doi: 10.3390/ani7120090 . ISSN   2076-2615. PMC   5742784 . PMID   29186879.
  3. ↑ Kahan BC, Jairath V (2018-05-02). "Outcome pre-specification requires sufficient detail to guard against outcome switching in clinical trials: a case study". Trials. 19 (1): 265. doi: 10.1186/s13063-018-2654-z . ISSN   1745-6215. PMC   5932799 . PMID   29720248.
  4. ↑ Chambers C (2017-08-24). "Clinical trials revolution could change the future of medical research". The Guardian. ISSN   0261-3077 . Retrieved 2024-02-19.
  5. ↑ Jones CW, Keil LG, Holland WC, Caughey MC, Platts-Mills TF (November 2015). "Comparison of registered and published outcomes in randomized controlled trials: a systematic review". BMC Medicine. 13: 282. doi: 10.1186/s12916-015-0520-3 . PMC   4650202 . PMID   26581191.
  6. ↑ Kampman JM, Sperna Weiland NH, Hollmann MW, Repping S, Hermanides J (September 2021). "High incidence of outcome switching observed in follow-up publications of randomized controlled trials: Meta-research study". Journal of Clinical Epidemiology. 137: 236–240. doi: 10.1016/j.jclinepi.2021.05.003 . PMID   34004339.
  7. ↑ Falk Delgado A, Falk Delgado A (August 2017). "Outcome switching in randomized controlled oncology trials reporting on surrogate endpoints: a cross-sectional analysis". Scientific Reports. 7 (1): 9206. Bibcode:2017NatSR...7.9206F. doi:10.1038/s41598-017-09553-y. PMC   5569019 . PMID   28835682.
  8. ↑ Goldacre B, Drysdale H, Marston C, Mahtani KR, Dale A, Milosevic I, Slade E, Hartley P, Heneghan C (February 2019). "COMPare: Qualitative analysis of researchers' responses to critical correspondence on a cohort of 58 misreported trials". Trials. 20 (1): 124. doi: 10.1186/s13063-019-3172-3 . PMC   6374909 . PMID   30760328.
  9. ↑ Belluz J (2015-12-29). "How researchers dupe the public with a sneaky practice called "outcome switching"". Vox. Retrieved 2024-03-18.
  10. ↑ "Study drug: BRL29060/Paroxetine (Paxil)", SmithKline Beecham, 20 August 1993, amended 24 March 1994.
  11. 1 2 Jureidini, Jon N.; McHenry, Leemon B.; Mansfield, Peter R. (2008). "Clinical trials and drug promotion: Selective reporting of study 329", International Journal of Risk & Safety in Medicine, 20(1–2) (pp. 73–81), p. 74. doi:10.3233/JRS-2008-0426
  12. ↑ Newman, Melanie (11 December 2010). "The Rules of Retraction", BMJ, 341(7785), pp. 1246–1248. doi:10.1136/bmj.c6985 PMID   21138994 JSTOR   20800711
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.