Observational study

Last updated
Anthropological survey paper from 1961 by Juhan Aul (et) from University of Tartu who measured about 50 000 people Blankett, Antropoloogiline vaatlusleht, vorm nr. 4, 1961.jpg
Anthropological survey paper from 1961 by Juhan Aul (et) from University of Tartu who measured about 50 000 people

In fields such as epidemiology, social sciences, psychology and statistics, an observational study draws inferences from a sample to a population where the independent variable is not under the control of the researcher because of ethical concerns or logistical constraints. One common observational study is about the possible effect of a treatment on subjects, where the assignment of subjects into a treated group versus a control group is outside the control of the investigator. [1] [2] This is in contrast with experiments, such as randomized controlled trials, where each subject is randomly assigned to a treated group or a control group. Observational studies, for lacking an assignment mechanism, naturally present difficulties for inferential analysis.

Contents

Motivation

The independent variable may be beyond the control of the investigator for a variety of reasons:

Types

Degree of usefulness and reliability

"Although observational studies cannot be used to make definitive statements of fact about the "safety, efficacy, or effectiveness" of a practice, they can: [4]

  1. provide information on 'real world' use and practice;
  2. detect signals about the benefits and risks of...[the] use [of practices] in the general population;
  3. help formulate hypotheses to be tested in subsequent experiments;
  4. provide part of the community-level data needed to design more informative pragmatic clinical trials; and
  5. inform clinical practice." [4]

Bias and compensating methods

In all of those cases, if a randomized experiment cannot be carried out, the alternative line of investigation suffers from the problem that the decision of which subjects receive the treatment is not entirely random and thus is a potential source of bias. A major challenge in conducting observational studies is to draw inferences that are acceptably free from influences by overt biases, as well as to assess the influence of potential hidden biases. The following are a non-exhaustive set of problems especially common in observational studies.

Matching techniques bias

In lieu of experimental control, multivariate statistical techniques allow the approximation of experimental control with statistical control by using matching methods. Matching methods account for the influences of observed factors that might influence a cause-and-effect relationship. In healthcare and the social sciences, investigators may use matching to compare units that nonrandomly received the treatment and control. One common approach is to use propensity score matching in order to reduce confounding, [5] although this has recently come under criticism for exacerbating the very problems it seeks to solve. [6]

Multiple comparison bias

Multiple comparison bias can occur when several hypotheses are tested at the same time. As the number of recorded factors increases, the likelihood increases that at least one of the recorded factors will be highly correlated with the data output simply by chance. [7]

Omitted variable bias

An observer of an uncontrolled experiment (or process) records potential factors and the data output: the goal is to determine the effects of the factors. Sometimes the recorded factors may not be directly causing the differences in the output. There may be more important factors which were not recorded but are, in fact, causal. Also, recorded or unrecorded factors may be correlated which may yield incorrect conclusions. [8]

Selection bias

Another difficulty with observational studies is that researchers may themselves be biased in their observational skills. This would allow for researchers to (either consciously or unconsciously) seek out the information they're looking for while conducting their research. For example, researchers may exaggerate the effect of one variable, or downplay the effect of another: researchers may even select in subjects that fit their conclusions. This selection bias can happen at any stage of the research process. This introduces bias into the data where certain variables are systematically incorrectly measured. [9]

Quality

A 2014 (updated in 2024) Cochrane review concluded that observational studies produce results similar to those conducted as randomized controlled trials. [10] The review reported little evidence for significant effect differences between observational studies and randomized controlled trials, regardless of design. [10] Differences need to be evaluated by looking at population, comparator, heterogeneity, and outcomes. [10]

See also

Related Research Articles

<span class="mw-page-title-main">Experiment</span> Scientific procedure performed to validate a hypothesis

An experiment is a procedure carried out to support or refute a hypothesis, or determine the efficacy or likelihood of something previously untried. Experiments provide insight into cause-and-effect by demonstrating what outcome occurs when a particular factor is manipulated. Experiments vary greatly in goal and scale but always rely on repeatable procedure and logical analysis of the results. There also exist natural experimental studies.

<span class="mw-page-title-main">Randomized controlled trial</span> Form of scientific experiment

A randomized controlled trial is a form of scientific experiment used to control factors not under direct experimental control. Examples of RCTs are clinical trials that compare the effects of drugs, surgical techniques, medical devices, diagnostic procedures or other medical treatments.

<span class="mw-page-title-main">Clinical trial</span> Phase of clinical research in medicine

Clinical trials are prospective biomedical or behavioral research studies on human participants designed to answer specific questions about biomedical or behavioral interventions, including new treatments and known interventions that warrant further study and comparison. Clinical trials generate data on dosage, safety and efficacy. They are conducted only after they have received health authority/ethics committee approval in the country where approval of the therapy is sought. These authorities are responsible for vetting the risk/benefit ratio of the trial—their approval does not mean the therapy is 'safe' or effective, only that the trial may be conducted.

In a blind or blinded experiment, information which may influence the participants of the experiment is withheld until after the experiment is complete. Good blinding can reduce or eliminate experimental biases that arise from a participants' expectations, observer's effect on the participants, observer bias, confirmation bias, and other sources. A blind can be imposed on any participant of an experiment, including subjects, researchers, technicians, data analysts, and evaluators. In some cases, while blinding would be useful, it is impossible or unethical. For example, it is not possible to blind a patient to their treatment in a physical therapy intervention. A good clinical protocol ensures that blinding is as effective as possible within ethical and practical constraints.

Selection bias is the bias introduced by the selection of individuals, groups, or data for analysis in such a way that proper randomization is not achieved, thereby failing to ensure that the sample obtained is representative of the population intended to be analyzed. It is sometimes referred to as the selection effect. The phrase "selection bias" most often refers to the distortion of a statistical analysis, resulting from the method of collecting samples. If the selection bias is not taken into account, then some conclusions of the study may be false.

A cohort study is a particular form of longitudinal study that samples a cohort, performing a cross-section at intervals through time. It is a type of panel study where the individuals in the panel share a common characteristic.

<span class="mw-page-title-main">Field experiment</span> Experiment conducted outside the laboratory

Field experiments are experiments carried out outside of laboratory settings.

<span class="mw-page-title-main">Scientific control</span> Methods employed to reduce error in science tests

A scientific control is an experiment or observation designed to minimize the effects of variables other than the independent variable. This increases the reliability of the results, often through a comparison between control measurements and the other measurements. Scientific controls are a part of the scientific method.

In the design of experiments, hypotheses are applied to experimental units in a treatment group. In comparative experiments, members of a control group receive a standard treatment, a placebo, or no treatment at all. There may be more than one treatment group, more than one control group, or both.

Clinical study design is the formulation of trials and experiments, as well as observational studies in medical, clinical and other types of research involving human beings. The goal of a clinical study is to assess the safety, efficacy, and / or the mechanism of action of an investigational medicinal product (IMP) or procedure, or new drug or device that is in development, but potentially not yet approved by a health authority. It can also be to investigate a drug, device or procedure that has already been approved but is still in need of further investigation, typically with respect to long-term effects or cost-effectiveness.

<span class="mw-page-title-main">Confounding</span> Variable or factor in causal inference

In causal inference, a confounder is a variable that influences both the dependent variable and independent variable, causing a spurious association. Confounding is a causal concept, and as such, cannot be described in terms of correlations or associations. The existence of confounders is an important quantitative explanation why correlation does not imply causation. Some notations are explicitly designed to identify the existence, possible existence, or non-existence of confounders in causal relationships between elements of a system.

<span class="mw-page-title-main">Randomized experiment</span> Experiment using randomness in some aspect, usually to aid in removal of bias

In science, randomized experiments are the experiments that allow the greatest reliability and validity of statistical estimates of treatment effects. Randomization-based inference is especially important in experimental design and in survey sampling.

<span class="mw-page-title-main">Quasi-experiment</span> Empirical interventional study

A quasi-experiment is an empirical interventional study used to estimate the causal impact of an intervention on target population without random assignment. Quasi-experimental research shares similarities with the traditional experimental design or randomized controlled trial, but it specifically lacks the element of random assignment to treatment or control. Instead, quasi-experimental designs typically allow the researcher to control the assignment to the treatment condition, but using some criterion other than random assignment.

The average treatment effect (ATE) is a measure used to compare treatments in randomized experiments, evaluation of policy interventions, and medical trials. The ATE measures the difference in mean (average) outcomes between units assigned to the treatment and units assigned to the control. In a randomized trial, the average treatment effect can be estimated from a sample using a comparison in mean outcomes for treated and untreated units. However, the ATE is generally understood as a causal parameter that a researcher desires to know, defined without reference to the study design or estimation procedure. Both observational studies and experimental study designs with random assignment may enable one to estimate an ATE in a variety of ways.

In the statistical analysis of observational data, propensity score matching (PSM) is a statistical matching technique that attempts to estimate the effect of a treatment, policy, or other intervention by accounting for the covariates that predict receiving the treatment. PSM attempts to reduce the bias due to confounding variables that could be found in an estimate of the treatment effect obtained from simply comparing outcomes among units that received the treatment versus those that did not. Paul R. Rosenbaum and Donald Rubin introduced the technique in 1983.

Clinical trials are medical research studies conducted on human subjects. The human subjects are assigned to one or more interventions, and the investigators evaluate the effects of those interventions. The progress and results of clinical trials are analyzed statistically.

A glossary of terms used in clinical research.

<span class="mw-page-title-main">Placebo-controlled study</span>

Placebo-controlled studies are a way of testing a medical therapy in which, in addition to a group of subjects that receives the treatment to be evaluated, a separate control group receives a sham "placebo" treatment which is specifically designed to have no real effect. Placebos are most commonly used in blinded trials, where subjects do not know whether they are receiving real or placebo treatment. Often, there is also a further "natural history" group that does not receive any treatment at all.

<span class="mw-page-title-main">Between-group design experiment</span>

In the design of experiments, a between-group design is an experiment that has two or more groups of subjects each being tested by a different testing factor simultaneously. This design is usually used in place of, or in some cases in conjunction with, the within-subject design, which applies the same variations of conditions to each subject to observe the reactions. The simplest between-group design occurs with two groups; one is generally regarded as the treatment group, which receives the ‘special’ treatment, and the control group, which receives no variable treatment and is used as a reference. The between-group design is widely used in psychological, economic, and sociological experiments, as well as in several other fields in the natural or social sciences.

Experimental benchmarking allows researchers to learn about the accuracy of non-experimental research designs. Specifically, one can compare observational results to experimental findings to calibrate bias. Under ordinary conditions, carrying out an experiment gives the researchers an unbiased estimate of their parameter of interest. This estimate can then be compared to the findings of observational research. Note that benchmarking is an attempt to calibrate non-statistical uncertainty. When combined with meta-analysis this method can be used to understand the scope of bias associated with a specific area of research.

References

  1. "Observational study". Archived from the original on 2016-04-27. Retrieved 2008-06-25.
  2. Porta M, ed. (2008). A Dictionary of Epidemiology (5th ed.). New York: Oxford University Press. ISBN   9780195314496.
  3. Kennedy-Martin T, Curtis S, Faries D, Robinson S, Johnston J (November 2015). "A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results". Trials. 16 (1): 495. doi: 10.1186/s13063-015-1023-4 . PMC   4632358 . PMID   26530985.
  4. 1 2 "Although observational studies cannot provide definitive evidence of safety, efficacy, or effectiveness, they can: 1) provide information on "real world" use and practice; 2) detect signals about the benefits and risks of complementary therapies use in the general population; 3) help formulate hypotheses to be tested in subsequent experiments; 4) provide part of the community-level data needed to design more informative pragmatic clinical trials; and 5) inform clinical practice." "Observational Studies and Secondary Data Analyses To Assess Outcomes in Complementary and Integrative Health Care." Archived 2019-09-29 at the Wayback Machine Richard Nahin, Ph.D., M.P.H., Senior Advisor for Scientific Coordination and Outreach, National Center for Complementary and Integrative Health, June 25, 2012
  5. Rosenbaum, Paul R. 2009. Design of Observational Studies. New York: Springer.
  6. King, Gary; Nielsen, Richard (2019-05-07). "Why Propensity Scores Should Not Be Used for Matching". Political Analysis. 27 (4): 435–454. doi:10.1017/pan.2019.11. hdl: 1721.1/128459 . ISSN   1047-1987. S2CID   53585283. | link to the full article (from the author's homepage
  7. Benjamini, Yoav (2010). "Simultaneous and selective inference: Current successes and future challenges". Biometrical Journal. 52 (6): 708–721. doi:10.1002/bimj.200900299. PMID   21154895. S2CID   8806192.
  8. "Introductory Econometrics Chapter 18: Omitted Variable Bias". www3.wabash.edu. Retrieved 2022-07-16.
  9. Hammer, Gaël P; du Prel, Jean-Baptist; Blettner, Maria (2009-10-01). "Avoiding Bias in Observational Studies". Deutsches Ärzteblatt International. 106 (41): 664–668. doi:10.3238/arztebl.2009.0664. ISSN   1866-0452. PMC   2780010 . PMID   19946431.
  10. 1 2 3 Toews, Ingrid; Anglemyer, Andrew; Nyirenda, John Lz; Alsaid, Dima; Balduzzi, Sara; Grummich, Kathrin; Schwingshackl, Lukas; Bero, Lisa (2024-01-04). "Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials: a meta-epidemiological study". The Cochrane Database of Systematic Reviews. 1 (1): MR000034. doi:10.1002/14651858.MR000034.pub3. ISSN   1469-493X. PMC  10765475. PMID   38174786.

Further reading