Web-based experiments

Last updated

A web-based experiment or Internet-based experiment is an experiment that is conducted over the Internet. In such experiments, the Internet is either "a medium through which to target larger and more diverse samples with reduced administrative and financial costs" or "a field of social science research in its own right." [1] Psychology and Internet studies are probably the disciplines that have used these experiments most widely, although a range of other disciplines including political science and economics also use web-based experiments. Within psychology most web-based experiments are conducted in the areas of cognitive psychology and social psychology. [2] [3] This form of experimental setup has become increasingly popular because researchers can cheaply collect large amounts of data from a wider range of locations and people. A web-based experiment is a type of online research method. Web based experiments have become significantly more widespread since the COVID-19 pandemic, as researchers have been unable to conduct lab-based experiments. [4]

Contents

Introduction

Experiments are an integral part of research, however, their integration with the Internet has been gradual. There are three main categories of experiments:

The adaption of each type of experiment online faces some hurdles.

Benefits

Web-based experiments are significantly less expensive, potentially allowing the researcher to:

These benefits have the potential to translate into greater external validity and generalizability for the study. For instance, in web-based experiments there is less reliance on data gathered from populations of Western undergraduate students who are often used as the default research subjects in social science disciplines. [9] Because participants remain in their homes or offices while participating in the experiment, scholars have also argued that such experiments have greater ecological validity. [5]

Criticisms and limitations

Web-based experiments may have weaker experimental controls compared to laboratory-based experiments, and may face greater difficulties coming up with procedures that ensure reliability and internal validity. [10] Online natural and field experiments may also face challenges generalizing findings beyond the online context in which they were conducted. Some potential difficulties faced by web-based experiments include:

In the face of these criticisms, some researchers have argued that brick-and-mortar experiments are just as affected by these problems, if not more so. [15] [16] [17]

Studies have been conducted to test the internal validity of web-based experiments, comparing across experimental conditions (online and offline) and successfully replicating findings. For example, Schoeffler et al. (2013) compared laboratory- and web-based results (62 and 1,168 subjects) of an auditory experiment and found no significant differences. [18] A paired experiment in behavioral economics split into online and traditional lab environments produced substantively similar results. [1] Uncompensated and unsupervised subjects on LabintheWild have been shown to replicate previous in-lab study results with comparable data quality. [19]

Methodologies

Experimental protocols have been suggested to prevent or control difficulties associated with web-based experimentation. Methods like sequential subject matching, background timing and mouse use tracking, and instantaneous compensation through PayPal have the potential to address many of the concerns about the internal validity of web-based experiments. [1] These methods control for differences in response times, address issues of selective attrition, concentration, and distraction, minimize subject concern about compensation, improve subject confidence that they have a real human partner in the experiment, and ensure that subjects have an appropriate understanding of the instructions and the decision problems in the experiment. [1]

Scholars have also formulated techniques to decrease or account for drop-outs, including the high-hurdle technique (motivationally adverse information is clustered at the beginning of the study), the seriousness check (requesting participant's probability estimate that they'll complete the study), and the warm-up phase (placing consent forms or other pre-study materials first to winnow the samples before the study begins). [5]

Examples

Use in psychology

A wide range of psychology experiments are conducted on the web. The Web Experiment List provides a way to recruit participants and archives past experiments (over 700 and growing). [20] A good resource for designing a web experiment is the free Wextor tool, which "dynamically creates the customized Web pages needed for the experimental procedure" and is remarkably easy to use. [21] Web experiments have been used to validate results from laboratory research and field research and to conduct new experiments that are only feasible if done online. [5] Further, the materials created for web experiments can be used in a traditional laboratory setting if later desired.

Interdisciplinary research using web experiments is rising. For example, a number of psychology and law researchers have used the web to collect data. Lora Levett and Margaret Bull Kovera examined whether opposing expert witnesses are effective in educating jurors about unreliable expert evidence. [22] Rather than sensitizing jurors to flaws in the other expert's testimony, the researchers found that jurors became more skeptical of all expert testimony. In her experiment, this led to more guilty verdicts. Levett and Kovera's research used a written transcript of a trial, which participants then read before making their decision. This type of stimulus has been criticized by some researchers as lacking ecological validity—that is, it does not closely approximate a real-life trial. Many recommend the use of video where possible. Researchers at New York University are currently conducting a psychology and law study that uses video of a criminal trial. [23]

Researchers at University of Salford are currently conducting a number of studies online to explore sound perception. [24] Sound experiments over the web are particularly difficult due to lack of control over sound reproduction equipment.

Salganik, Dodds, and Watts conducted an experiment to measure social influence, specifically in the popularity rating of songs. Their use of the Internet allowed them to collect over 14,000 participants and examine the relationship between individual and collective behavior. [7]

Use in economics

As more experiments have been conducted in economics, questions about appropriate methodology and study organization has been raised. Jerome Hergueux and Nicolas Jacquemet developed an "online laboratory" to compare social preferences and risk aversion online and in person. They administered a risk aversion assessment, Public good game, a Trust game, a Dictator game, and an Ultimatum game to groups both online and in a lab as a way of assessing the internal validity of web-based experimentation in economics. [1]

Use in political science

An online field experiment conducted on 61 million Facebook users tested whether receiving information about voting, polling places, and the voting behavior of one's friends led individuals to seek out political information, influenced political self-expression, and changed real-world voting behavior. [8]

Use in internet studies

Web-based experiments have particular salience in studies of how online communities operate. Internet studies, including studies of online communities and social networks, have used natural and field experiments to understand the effects of informal rewards in peer production on Wikipedia, [25] [26] as well as the impact of early recognition and support on future successes on Kickstarter, Change.org, Epinions, and Wikipedia. [27] Another experiment looked at the effect on edit rates of introducing a program of intelligent task assignment on Wikipedia. [28]

Related Research Articles

Psychology is the study of mind and behavior in humans and non-humans. Psychology includes the study of conscious and unconscious phenomena, including feelings and thoughts. It is an academic discipline of immense scope, crossing the boundaries between the natural and social sciences. Psychologists seek an understanding of the emergent properties of brains, linking the discipline to neuroscience. As social scientists, psychologists aim to understand the behavior of individuals and groups.

Social psychology is the scientific study of how thoughts, feelings, and behaviors are influenced by the real or imagined presence of other people or by social norms. Social psychologists typically explain human behavior as a result of the relationship between mental states and social situations, studying the social conditions under which thoughts, feelings, and behaviors occur, and how these variables influence social interactions.

Introspection is the examination of one's own conscious thoughts and feelings. In psychology, the process of introspection relies on the observation of one's mental state, while in a spiritual context it may refer to the examination of one's soul. Introspection is closely related to human self-reflection and self-discovery and is contrasted with external observation.

Obedience, in human behavior, is a form of "social influence in which a person yields to explicit instructions or orders from an authority figure". Obedience is generally distinguished from compliance, which some authors define as behavior influenced by peers while others use it as a more general term for positive responses to another individual's request, and from conformity, which is behavior intended to match that of the majority. Depending on context, obedience can be seen as moral, immoral, or amoral. For example, in psychological research, individuals are usually confronted with immoral demands designed to elicit an internal conflict. If individuals still choose to submit to the demand, they are acting obediently.

Experimental psychology refers to work done by those who apply experimental methods to psychological study and the underlying processes. Experimental psychologists employ human participants and animal subjects to study a great many topics, including sensation, perception, memory, cognition, learning, motivation, emotion; developmental processes, social psychology, and the neural substrates of all of these.

Experimental economics is the application of experimental methods to study economic questions. Data collected in experiments are used to estimate effect size, test the validity of economic theories, and illuminate market mechanisms. Economic experiments usually use cash to motivate subjects, in order to mimic real-world incentives. Experiments are used to help understand how and why markets and other exchange systems function as they do. Experimental economics have also expanded to understand institutions and the law.

The Web Experimental Psychology Lab is a website for participating in Web-based experiments, a method used in experimental psychology. The Web Experimental Psychology Lab was founded in 1994-1995, by Ulf-Dietrich Reips at the University of Tübingen, then moved to the University of Zürich and on to the Universidad de Deusto, and is now at the University of Konstanz. For the first time, participants were able to take part in studies via a web browser in a virtual psychology laboratory.

Prof. Dr. Ulf-Dietrich Reips is a full professor in the Faculty of Sciences at the University of Konstanz, where he holds the Chair for Psychological Methods, Assessment, and iScience. Between 2009 and 2013 he was a full-time tenured IKERBASQUE research professor at University of Deusto in Bilbao, Spain, and remains affiliated with Ikerbasque. Until 2009 he was an assistant professor and lecturer ('Oberassistent') at the Psychology Department of the University of Zurich, Switzerland. He received his PhD in 1997 and his habilitation in 2004 from the University of Tübingen, Germany. In 1992, he received an M.A. in Psychology from Sonoma State University, California. Reips spent most of his undergraduate and graduate years at the University of Tübingen, where he had attended the Leibniz Kolleg. He majored in both Psychology and General Rhetoric and had a minor in Political Science. In 2012, Ulf-Dietrich Reips received a FIRST award from University of Colorado Boulder and is since affiliated on an honorable basis with its Department of Psychology and Neuroscience. Based on his publications' impact and his affiliation with IKERBASQUE, the Consejo Superior de Investigaciones Cientificas, Spain, ranked him 7th of "Top Scientists working at Spanish Private Universities" in 2014. In Fall 2015, Reips was offered to direct the Leibniz Institute for Psychology Information in Trier, in association with a full professorship for Psychology at University of Trier.

<span class="mw-page-title-main">Response bias</span> Type of bias

Response bias is a general term for a wide range of tendencies for participants to respond inaccurately or falsely to questions. These biases are prevalent in research involving participant self-report, such as structured interviews or surveys. Response biases can have a large impact on the validity of questionnaires or surveys.

<span class="mw-page-title-main">Field experiment</span>

Field experiments are experiments carried out outside of laboratory settings.

Internal validity is the extent to which a piece of evidence supports a claim about cause and effect, within the context of a particular study. It is one of the most important properties of scientific studies and is an important concept in reasoning about evidence more generally. Internal validity is determined by how well a study can rule out alternative explanations for its findings. It contrasts with external validity, the extent to which results can justify conclusions about other contexts. Both internal and external validity can be described using qualitative or quantitative forms of causal notation.

External validity is the validity of applying the conclusions of a scientific study outside the context of that study. In other words, it is the extent to which the results of a study can be generalized to and across other situations, people, stimuli, and times. In contrast, internal validity is the validity of conclusions drawn within the context of a particular study. Because general conclusions are almost always a goal in research, external validity is an important property of any study. Mathematical analysis of external validity concerns a determination of whether generalization across heterogeneous populations is feasible, and devising statistical and computational methods that produce valid generalizations.

In social research, particularly in psychology, the term demand characteristic refers to an experimental artifact where participants form an interpretation of the experiment's purpose and subconsciously change their behavior to fit that interpretation. Typically, demand characteristics are considered an extraneous variable, exerting an effect on behavior other than that intended by the experimenter. Pioneering research was conducted on demand characteristics by Martin Orne.

<span class="mw-page-title-main">Quasi-experiment</span> Empirical interventional study

A quasi-experiment is an empirical interventional study used to estimate the causal impact of an intervention on target population without random assignment. Quasi-experimental research shares similarities with the traditional experimental design or randomized controlled trial, but it specifically lacks the element of random assignment to treatment or control. Instead, quasi-experimental designs typically allow the researcher to control the assignment to the treatment condition, but using some criterion other than random assignment.

Online research methods (ORMs) are ways in which researchers can collect data via the internet. They are also referred to as Internet research, Internet science or iScience, or Web-based methods. Many of these online research methods are related to existing research methodologies but re-invent and re-imagine them in the light of new technologies and conditions associated with the internet. The field is relatively new and evolving. With the growth of social media, a new level of complexity and opportunity has been created. The inclusion of social media research can provide unique insights into consumer and societal segments and gaining an "emotional" measure of a population on issues of interest.

Psychological research refers to research that psychologists conduct for systematic study and for analysis of the experiences and behaviors of individuals or groups. Their research can have educational, occupational and clinical applications.

Experimental political science is the use of experiments, which may be natural or controlled, to implement the scientific method in political science.

Adam N. Joinson is a British author, academic and public speaker within the area of cyberpsychology. He is Professor of Information Systems at University of Bath, following posts at the University of West of England and the Open University. and has conducted ground breaking research into the psychology of Internet usage.

The seriousness check is a technique that can be used in online research to improve data quality. Nowadays, many scientific studies with human participants are conducted online and are accessible to a large diversity of participants. Nonetheless, many people just want to look at the different pages of the questionnaire, instead of giving carefully chosen answers to the questions. The seriousness check addresses this problem: In this approach the respondents are asked about the seriousness of their participation or for a probability estimate that they will complete the entire study or experiment. Thus, by using the seriousness check irrelevant data entries can be easily identified and be excluded from the data analysis.

The multiple site entry technique is a strategy that can be used in online research to target different samples via different recruitment sites and compare their data. It is a method used in behavioral and social research to assess the presence and impact of self-selection effects. Self-selection effects can be considered a major challenge in social science research. With the invention of online research in the 1990s the multiple site entry technique became possible, because the recruitment of participants via different links (URLs) is very easy to implement. It can be assumed that there is no self-selection bias if the data sets coming from different recruitment sites do not differ systematically.

References

  1. 1 2 3 4 5 6 Hergueux, J.; Jacquemet, N. (2014). "Social preferences in the online laboratory: a randomized experiment" (PDF). Experimental Economics. 18 (2): 251–283. doi:10.1007/s10683-014-9400-5. S2CID   14347177.
  2. ↑ Reips, U.-D. (2007). The methodology of Internet-based experiments. In A. Joinson, K. McKenna, T. Postmes, & U.-D. Reips (Eds.), The Oxford Handbook of Internet Psychology (pp. 373-390). Oxford: Oxford University Press.
  3. ↑ Reips, U.-D. & Krantz, J. H. (2010). Conducting true experiments on the Web. In S. Gosling & J. Johnson, Advanced Internet Methods in the Behavioral Sciences (pp. 193-216). Washington, DC: American Psychological Association.
  4. ↑ Lourenco, Stella F.; Tasimi, Arber (2020-08-01). "No Participant Left Behind: Conducting Science During COVID-19". Trends in Cognitive Sciences. 24 (8): 583–584. doi: 10.1016/j.tics.2020.05.003 . ISSN   1364-6613. PMC   7211671 . PMID   32451239.
  5. 1 2 3 4 5 6 7 Reips, Ulf-Dietrich (2002). "Standards for Internet-Based Experimenting" (PDF). Experimental Psychology. 49 (4): 243–56. doi:10.1026/1618-3169.49.4.243. PMID   12455331. S2CID   2404134.
  6. ↑ Kramer, A. D. I.; Guillory, J. E.; Hancock, J. T. (2014). "Experimental evidence of massive-scale emotional contagion through social networks". Proceedings of the National Academy of Sciences. 111 (24): 8788–8790. Bibcode:2014PNAS..111.8788K. doi: 10.1073/pnas.1320040111 . PMC   4066473 . PMID   24889601.
  7. 1 2 Salganik, M. J.; Dodds, P. S.; Watts, D. J. (2006). "Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market". Science. 311 (5762): 854–856. Bibcode:2006Sci...311..854S. doi:10.1126/science.1121066. PMID   16469928. S2CID   7310490.
  8. 1 2 Bond, R. M.; Fariss, C. J.; Jones, J. J.; Kramer, A. D. I.; Marlow, C.; Settle, J. E.; Fowler, J. H. (2012). "A 61-million-person experiment in social influence and political mobilization". Nature. 489 (7415): 295–298. Bibcode:2012Natur.489..295B. doi:10.1038/nature11421. PMC   3834737 . PMID   22972300.
  9. ↑ Henrich, J.; Heine, S. J.; Norenzayan, A. (2010). "The weirdest people in the world?" (PDF). Behavioral and Brain Sciences. 33 (2–3): 61–83. doi:10.1017/s0140525x0999152x. hdl: 11858/00-001M-0000-0013-26A1-6 . PMID   20550733. S2CID   220918842.
  10. ↑ Hoffman, M., & Morgan, J. (2011). Who’s Naughty? Who’s Nice? Social Preferences in Online Industries. UC Berkeley Working Paper.
  11. ↑ Anderhub, V.; Müller, R.; Schmidt, C. (2001). "Design and evaluation of an economic experiment via the Internet". Journal of Economic Behavior & Organization. 46 (2): 227–247. doi: 10.1016/s0167-2681(01)00195-0 .
  12. ↑ Kahneman, D. (2003). "Maps of bounded rationality: Psychology for behavioral economics". The American Economic Review. 93 (5): 1449–1475. doi:10.1257/000282803322655392. S2CID   15131441.
  13. ↑ Shavit, T.; Sonsino, D.; Benzion, U. (2001). "A comparative study of lotteries-evaluation in class and on the Web". Journal of Economic Psychology. 22 (4): 483–491. doi:10.1016/S0167-4870(01)00048-4.
  14. ↑ Eckel, C. C.; Wilson, R. K. (2006). "Internet cautions: Experimental games with Internet partners". Experimental Economics. 9: 53–66. doi:10.1007/s10683-006-4307-4. S2CID   30675518.
  15. ↑ Reips, U.-D. (1996, October). Experimenting on the World WideWeb. Paper presented at the 1996 Society for Computers in Psychology conference, Chicago.
  16. ↑ "Virtual labs: Is there wisdom in the crowd?". New Scientist. 16 March 2007.
  17. ↑ "A Web of research". www.apa.org. Archived from the original on 2000-05-10.
  18. ↑ Schoeffler et al. (2013). "An Experiment About Estimating the Number of Instruments in Polyphonic Music: A Comparison Between Internet and Laboratory Results"
  19. ↑ Reinecke, Katharina; Gajos, Krzysztof Z. (2015-01-01). "LabintheWild". Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. CSCW '15. New York, NY, USA: ACM. pp. 1364–1378. doi:10.1145/2675133.2675246. ISBN   9781450329224. S2CID   2739130.
  20. ↑ "Home". wexlist.net.
  21. ↑ "WEXTOR Webserver".
  22. ↑ Levett; Kovera (2008). "The Effectiveness of Opposing Expert Witnesses for Educating Jurors about Unreliable Expert Evidence". Law and Human Behavior. 32 (4): 363–74. doi:10.1007/s10979-007-9113-9. PMID   17940854. S2CID   809273.
  23. ↑ Virginia vs. McNamara
  24. ↑ "Internet sound experiments | psychoacoustic tests University of Salford".
  25. ↑ Restivo, Michael; Rijt, Arnout van de (2012-03-29). "Experimental Study of Informal Rewards in Peer Production". PLOS ONE. 7 (3): e34358. Bibcode:2012PLoSO...734358R. doi: 10.1371/journal.pone.0034358 . ISSN   1932-6203. PMC   3315525 . PMID   22479610.
  26. ↑ Zhu, Haiyi; Zhang, Amy; He, Jiping; Kraut, Robert E.; Kittur, Aniket (2013-01-01). "Effects of peer feedback on contribution". Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI '13. New York, NY, USA: ACM. pp. 2253–2262. doi:10.1145/2470654.2481311. ISBN   9781450318990. S2CID   17167144.
  27. ↑ Rijt, Arnout van de; Kang, Soong Moon; Restivo, Michael; Patil, Akshay (2014-05-13). "Field experiments of success-breeds-success dynamics". Proceedings of the National Academy of Sciences. 111 (19): 6934–6939. Bibcode:2014PNAS..111.6934V. doi: 10.1073/pnas.1316836111 . ISSN   0027-8424. PMC   4024896 . PMID   24778230.
  28. ↑ Cosley, Dan; Frankowski, Dan; Terveen, Loren; Riedl, John (2007-01-01). "SuggestBot". Proceedings of the 12th international conference on Intelligent user interfaces. IUI '07. New York, NY, USA: ACM. pp. 32–41. doi:10.1145/1216295.1216309. ISBN   978-1595934819. S2CID   906034.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.