Online Experiments
By Janet Salmons, Ph.D. Manager, Sage Research Methods Community
Dr. Salmons is the author of Doing Qualitative Research Online, and Gather Your Data Online. Use the code COMMUNIT24 for 25% off through December 31, 2024 if you purchase research books from Sage.
Use the code COMMUNIT24 for 25% off through December 31, 2024 .
Designing Online Experiments for the Social Sciences is a new book from Dr. Guiseppe Veltri.
Engage participants in online research experiments
Surveys and interviews aren’t the only ways to conduct research online! Today’s researchers are using the Internet is by conducting experimental studies. [Learn more about experimental design in this Sage Research Methods Community post, Quantitative Research with Experimental Designs. ]
This collection of open access articles offers multidisciplinary examples, guidance, and perspectives on this type of research. Also see the interview with Dr. Guiseppe Veltri and see his new book: Designing Online Experiments for the Social Sciences.
Demirović, M., Rogers, J., & Robbins, B. G. (2023). Gender and Gender-Role Attitudes in Wage Negotiations: Evidence from an Online Experiment. Social Psychology Quarterly, 0(0). https://doi.org/10.1177/01902725231195889 (Open access link.)
Abstract. Gender differences in wage negotiations have been offered as a popular explanation for why the gender gap in pay persists in the United States. In this study, we use data from an artificial wage negotiation experiment (N = 307) to examine the relationship between gender and wage negotiations and to test whether gender-role attitudes moderate this relationship. We find that gender-role attitudes moderate how gender influences the decision to negotiate, but not the outcomes of negotiations, and that forced negotiations do not lead to additional gains for women regardless of their gender-role attitudes. We conclude with a discussion of implications and directions for future research.
Jiang R, Pullenayegum E, Shaw JW, et al. Comparison of Preferences and Data Quality between Discrete Choice Experiments Conducted in Online and Face-to-Face Respondents. Medical Decision Making. 2023;43(6):667-679. doi:10.1177/0272989X231171912
Abstract. Discrete choice experiments (DCE) are increasingly being conducted using online panels. However, the comparability of such DCE-based preferences to traditional modes of data collection (e.g., in-person) is not well established. In this study, supervised, face-to-face DCE was compared with its unsupervised, online facsimile on face validity, respondent behavior, and modeled preferences. Data from face-to-face and online EQ-5D-5L health state valuation studies were compared, in which each used the same experimental design and quota sampling procedure. Respondents completed 7 binary DCE tasks comparing 2 EQ-5D-5L health states presented side by side (health states A and B). Data face validity was assessed by comparing preference patterns as a function of the severity difference between 2 health states within a task. The prevalence of potentially suspicious choice patterns (i.e., all As, all Bs, and alternating As/Bs) was compared between studies. Preference data were modeled using multinomial logit regression and compared based on dimensional contribution to overall scale and importance ranking of dimension-levels.
Kim, J. (2023). Why do Experiments Fail? Six Practical Suggestions for Successful Online Experiments. Journal of Travel Research, 0(0). https://doi.org/10.1177/00472875231209983
Abstract. Experiments are frequently used in the travel and hospitality literature to provide stronger evidence of causal relationships between various constructs. Recently, despite the use of online platforms for experimental studies, such studies have often failed to find significant results, as expected. To increase the probability of obtaining significant results for experiments using online panels, this paper suggests six practical recommendations across three categories: (i) handling less homogenous online participants; (ii) understanding and managing different motivations and abilities in online participants; and (iii) using effective and transparent experimental designs and procedures. This paper provides the results of three empirical investigations to support these recommendations.
Lee, L., Guzzo, R. F., Madera, J. M., & Guchait, P. (2021). Examining Applicant Online Recruitment: The Use of Fictitious Websites in Experimental Studies. Cornell Hospitality Quarterly, 62(1), 76-88. https://doi.org/10.1177/1938965520965223
Abstract. Advancements in technology enable hospitality organizations to rely on digital recruitment efforts such as websites to attract applicants. Reflecting this industry trend, a small, but growing body of literature from the hospitality industry examines how applicants react to online recruiting using fictitious websites of hypothetical companies in experiments. The purpose of this article is to validate the use of fictitious websites as an experimental data collection method. Two quasi-experiments were guided by theories and model of applicant perceptions of fit and organizational attraction. Fit was manipulated by matching the career preference of active job seekers (e.g., a job seeker in the hotel sector) with a fictitious website (e.g., a hotel’s careers page) or not (control group). The results from the two quasi-experiments showed person–organization fit (Study 1) and person–job fit (Study 2) led to more organizational attraction under conditions of matches (e.g., a job seeker in the hotel sector evaluating a hotel’s careers page) than in the control groups. The findings of the two studies not only support the use of fictitious websites as a viable data collection method but also open a new line of research for hospitality research and human resources. Future hospitality scholars can use this technique to manipulate organization’s human resource practices (e.g., recruitment, selection, training, performance evaluation, compensation, and benefits) and examine attitudes of individuals (e.g., applicants, employees, and managers). The current data collection method also allows for researchers to not only manipulate information but also maximize the realism of the experimental stimuli.
Gong, X., & Huskey, R. (2023). Moving Behavioral Experimentation Online: A Tutorial and Some Recommendations for Drift Diffusion Modeling. American Behavioral Scientist, 0(0). https://doi.org/10.1177/00027642231207073
Abstract. Behavioral science demands skillful experimentation and high-quality data that are typically gathered in person. However, the COVID-19 pandemic forced many behavioral research laboratories to close. Thankfully, new tools for conducting online experiments allow researchers to elicit psychological responses and gather behavioral data with unprecedented precision. It is now possible to quickly conduct large-scale high-quality behavioral experiments online, even for studies designed to generate data necessary for complex computational models. However, these techniques require new skills that might be unfamiliar to behavioral researchers who are more familiar with laboratory-based experimentation. We present a detailed tutorial introducing an end-to-end build of an online experimental pipeline and corresponding data analysis. We provide an example study investigating people’s media preferences using drift-diffusion modeling (DDM), paying particular attention to potential issues that come with online behavioral experimentation. This tutorial includes sample data and code for conducting and analyzing DDM data gathered in an online experiment, thereby mitigating the extent to which researchers must reinvent the wheel.
Rong, K., Zhou, D., Shi, X., & Huang, W. (2022). Social Information Disclosure of Friends in Common in an E‐commerce Platform Ecosystem: An Online Experiment. Production and Operations Management, 31(3), 984-1005. https://doi.org/10.1111/poms.13591
Abstract. Real social information is believed to cause a trust–privacy paradox in e‐commerce platforms, with the degree of information disclosure affecting platform performance. In this study, we split the mixed influence into the direct trust mechanism and the indirect privacy mechanism, studying each in the context of partial and full social information disclosure, respectively. Using unique data from an 80‐day large‐scale online experiment conducted in an e‐commerce platform in China, we show that a partial social information disclosure could reduce the conversion rate, while a full social information disclosure could promote both the conversion and sales rates of the e‐commerce platform through a trust mechanism. Moreover, social information disclosure would reduce the sales rate and promote the conversion rate through a privacy mechanism. However, even though the degree of privacy could enhance the privacy mechanism, this mechanism is still much weaker than a trust mechanism. Overall, social information could play a positive role in e‐commerce platforms if it is fully disclosed. This study encourages cooperation between social media platforms and e‐commerce platforms by introducing a real social relationship into the online trading market.
Zhao, Y. (2024). TikTok and Researcher Positionality: Considering the Methodological and Ethical Implications of an Experimental Digital Ethnography. International Journal of Qualitative Methods, 23. https://doi.org/10.1177/16094069231221374
Abstract. In this article I examine the opportunities and challenges arising from an experimental digital ethnography I conducted as a digital content creator in response to social restrictions during COVID-19. To explore the perceptions and performances of masculinity among young Uzbek men in Uzbekistan, I created 50 TikTok videos between 2021 and 2022. These videos received more than 300,000 likes in total, not only significantly broadening the reach of my research recruitment but also serving as a substantial source of ethnographic data during the pandemic. Throughout the creation of these digital videos, I assumed a dual role as an agent in the research and an object of observation. This dual role underscores the agency of both researchers and the researched in navigating the digital platform, which allows for the challenging of conventional research gazes and relationships. This digital approach also unveils the complex spatial dynamics that underlie interactions in both online and offline realms, shedding light on how digital platforms can both enhance and constrain research efforts. Moreover, this article delves into the ethical implications of this experimental digital ethnography, which revolve around potential physical and mental risks to researchers, challenges related to the re-definition of research participation, and issues pertaining to obtaining informed consent. The findings provide insights and make contributions to problematising the conceptualisation of digital spaces, online communities/publics and digital ethnography. I conclude by offering insights for researchers who face restrictions in field access or are interested in studying youth culture on social media platforms, particularly in the role of a content creator, an area that has been relatively underexplored in previous research.
Sage Research Methods Community posts about online research
The wealth of material available online is irresistible to social researchers who are trying to understand contemporary experiences, perspectives, and events. The ethical collection and -use of such material is anything but straightforward. Find open-access articles that explore different approaches.