Megastudy

A megastudy or mega-study is a research study in which a large number of different treatments or interventions are tested at the same time, on the same sample or similar samples, using a common outcome measure, and using the same experimental protocol. ^{[1] [2] [3]}

Megastudy examples

• Exercise encouragement ^[1]
• Vaccination nudges ^[4]
• Strengthening of democratic attitudes ^[5]
• Interventions against climate change ^[3]

Many-lab studies

The megastudy technique can be combined with the many-labs approach, so that teams of researchers from across the planet conduct the same experiment locally. ^{[3] [6]}

Megastudy criticisms

• Statistical power: While the overall megastudy sample size may be large, the sample size per intervention may be relatively small, leading to underpowered designs with wide confidence intervals. As a result, while interventions may be comparable, their relative ranking by outcome measure may be noisy. Increased sample size can help address this issue. ^[7]

• Lack of theory: The megastudy technique may be considered a form of "fishing expedition" for what interventions have strongest effect on the outcome measure, without much theory building. ^[7]

References

1. Milkman, K.L., Gromet, D., Ho, H. et al. Megastudies improve the impact of applied behavioural science. Nature 600, 478–483 (2021). https://doi.org/10.1038/s41586-021-04128-4
2. Angela L Duckworth, Katherine L Milkman, A guide to megastudies, PNAS Nexus, Volume 1, Issue 5, November 2022, https://doi.org/10.1093/pnasnexus/pgac214
3. Doell, K.C. Megastudies to test the efficacy of behavioural interventions. Nat Rev Psychol 2, 263 (2023). https://doi.org/10.1038/s44159-023-00174-z
4. Milkman, Katherine L., et al. "A 680,000-person megastudy of nudges to encourage vaccination in pharmacies." Proceedings of the National Academy of Sciences 119.6 (2022): e2115126119. https://doi.org/10.1073/pnas.2115126119
5. Voelkel, Jan G., et al. "Megastudy identifying successful interventions to strengthen Americans’ democratic attitudes." Northwestern University: Evanston, IL, USA (2022). https://www.ipr.northwestern.edu/documents/working-papers/2022/wp-22-38.pdf
6. Pavlov, Yuri G.; Adamian, Nika; Appelhoff, Stefan; Arvaneh, Mahnaz; Benwell, Christopher S. Y.; Beste, Christian; Bland, Amy R.; Bradford, Daniel E.; Bublatzky, Florian; Busch, Niko A.; Clayson, Peter E.; Cruse, Damian; Czeszumski, Artur; Dreber, Anna; Dumas, Guillaume (2021-11-01). "#EEGManyLabs: Investigating the replicability of influential EEG experiments". Cortex. 144: 213–229. doi:10.1016/j.cortex.2021.03.013. hdl: 1885/295623 . ISSN   0010-9452.
7. Collins J (2022-05-27). "Megastudy scepticism". jasoncollins.blog. Retrieved 2023-12-13.