More Ways to Conduct Research Online: Open Access Examples
In June we are focusing on a range of qualitative and quantitative methods for collecting data online. Find the whole series through this link.
When you think of "online research," what comes to mind? Expand the possibilities by looking at this eclectic, multidisciplinary, global collection of open access studies that use a wide range of methods, from diaries to surveys, focus groups to content analysis!
Anson, I. G. (2018). Taking the time? Explaining effortful participation among low-cost online survey participants. Research & Politics. https://doi.org/10.1177/2053168018785483
Abstract. Recent research has shown that Amazon MTurk workers exhibit substantially more effort and attention than respondents in student samples when participating in survey experiments. In this paper, I examine when and why low-cost online survey participants provide effortful responses to survey experiments in political science. I compare novice and veteran MTurk workers to participants in Qualtrics’s qBus, a comparable online omnibus program. The results show that MTurk platform participation is associated with substantially greater effort across a variety of indicators of effort relative to demographically-matched peers. This effect endures even when compensating for the amount of survey experience accumulated by respondents, suggesting that MTurk workers may be especially motivated due to an understudied self-selection mechanism. Together, the findings suggest that novice and veteran MTurk workers alike are preferable to comparable convenience sample participants when performing complex tasks.
>> More on MethodSpace about MTurk
Burles, M. C., & Bally, J. M. G. (2018). Ethical, Practical, and Methodological Considerations for Unobtrusive Qualitative Research About Personal Narratives Shared on the Internet. International Journal of Qualitative Methods. https://doi.org/10.1177/1609406918788203
Abstract. As Internet research grows in popularity, attention to the ethics of studying online content is crucial to ensuring ethical diligence and appropriateness. Over recent years, ethical guidelines and recommendations have emerged to advise researchers and institutional review boards on best practices. However, these guidelines are sometimes irrelevant, overly rigid, or lack recognition of the contingent nature of ethical decision-making in qualitative research. Furthermore, varied ethical stances and practices are evident in existing literature. This article explores key ethical issues for qualitative research involving online content, with a focus on the unobtrusive study of personal narratives shared via the Internet. Principles of informed consent and confidentiality are examined in depth alongside practical and methodological considerations for unobtrusive qualitative research. This critical exploration contributes to ongoing discussion of ethical conduct of Internet research and promotes ethically aware yet flexible approaches to online qualitative research and creative methodological efforts to overcoming ethical challenges.
Coppock, A., & McClellan, O. A. (2019). Validating the demographic, political, psychological, and experimental results obtained from a new source of online survey respondents. Research & Politics. https://doi.org/10.1177/2053168018822174
Abstract. Researchers have increasingly turned to online convenience samples as sources of survey responses that are easy and inexpensive to collect. As reliance on these sources has grown, so too have concerns about the use of convenience samples in general and Amazon’s Mechanical Turk in particular. We distinguish between “external validity” and theoretical relevance, with the latter being the more important justification for any data collection strategy. We explore an alternative source of online convenience samples, the Lucid Fulcrum Exchange, and assess its suitability for online survey experimental research. Our point of departure is the 2012 study by Berinsky, Huber, and Lenz that compares Amazon’s Mechanical Turk to US national probability samples in terms of respondent characteristics and treatment effect estimates. We replicate these same analyses using a large sample of survey responses on the Lucid platform. Our results indicate that demographic and experimental findings on Lucid track well with US national benchmarks, with the exception of experimental treatments that aim to dispel the “death panel” rumor regarding the Affordable Care Act. We conclude that subjects recruited from the Lucid platform constitute a sample that is suitable for evaluating many social scientific theories, and can serve as a drop-in replacement for many scholars currently conducting research on Mechanical Turk or other similar platforms.
Decorte, T., Malm, A., Sznitman, S. R., Hakkarainen, P., Barratt, M. J., Potter, G. R., Werse, B., Kamphausen, G., Lenton, S., & Asmussen Frank, V. (2019). The challenges and benefits of analyzing feedback comments in surveys: Lessons from a cross-national online survey of small-scale cannabis growers. Methodological Innovations. https://doi.org/10.1177/2059799119825606
Abstract. It is common practice in survey questionnaires to include a general open and non-directive feedback question at the end, but the analysis of this type of data is rarely discussed in the methodological literature. While these open-ended comments can be useful, most researchers fail to report on this issue. The aim of this article is to illustrate and reflect upon the benefits and challenges of analyzing responses to open-ended feedback questions. The article describes the experiences of coding and analyzing data generated through a feedback question at the end of an international online survey with small-scale cannabis cultivators carried out by the Global Cannabis Cultivation Research Consortium. After describing the design and dataset of the web survey, the analytical approach and coding frame are presented. The analytical strategies chosen in this study illustrate the diversity and complexity of feedback comments which pose methodological challenges to researchers wishing to use them for data analyses. In this article, three types of feedback comments (political/policy comments, general comments of positive and negative appreciation, and methodological comments) are used to illustrate the difficulties and advantages of analyzing this type of data. The advantages of analyzing feedback comments are well known, but they seem to be rarely exploited. General feedback questions at the end of surveys are typically non-directive. If researchers want to use these data for research and analyses, they need a clear strategy. They ought to give enough thought to why they are including this type of question, and develop an analytical strategy at the design stage of the study.
de Groot, R., Kaal, H. L., & Stol, W. P. (2019). Studying Problematic Online Behavior of Adolescents With Mild Intellectual Disabilities and Borderline Intellectual Functioning: Methodological and Ethical Considerations for Data Collection. International Journal of Qualitative Methods. https://doi.org/10.1177/1609406919857978
Abstract. The aim of this study is to find a research method that results in capturing lived experiences of problematic online behavior of adolescents with mild intellectual disabilities and borderline intellectual functioning within four risk domains: commercial interests, aggression, sexuality, and values/ideology. Three research methods were examined and field-tested in small sample pilot studies: an online questionnaire (N = 16), two focus group interviews (N = 6 and N = 14), and a combination of participatory observations and visual elicitation (N = 2). Both the questionnaire and the focus group studies were not able to generate sufficient knowledge to capture lived experiences. Key issues that arose were the respondents’ comprehension of the research questions, their tendency to give socially desirable answers, the influence of group dynamics, and a lack of rapport between researcher and respondent. Results generated from the third pilot study were more promising. Participatory observations in the form of deep hanging out combined with conversational interviewing and elements of visual elicitation mended these issues and helped to create an authentic research environment, build real relationships, and level the playing field between researcher and respondents. Additionally, it invited the respondents to voice their opinions and feelings about their online experiences. Finally, the study inspires to use different communicational means with the adolescents to increase the understanding of their virtual world. Some important ethical and methodological limitations to these findings are discussed.
Gordon, A. R., Calzo, J. P., Eiduson, R., Sharp, K., Silverstein, S., Lopez, E., Thomson, K., & Reisner, S. L. (2021). Asynchronous Online Focus Groups for Health Research: Case Study and Lessons Learned. International Journal of Qualitative Methods. https://doi.org/10.1177/1609406921990489
Abstract. Increasingly, social life—and accordingly, social research—is conducted in online environments. Asynchronous online focus groups (AOFGs) have emerged as an important tool to conduct remote research with geographically diverse populations. However, there remain few systematic accounts of AOFG methods to guide researchers’ decision-making in designing and implementing studies. This paper seeks to address this gap by describing a recent study on body image and health among transgender and gender diverse (TGD) young adults. In this study, eight AOFGs were conducted in August-October 2019 with 66 TGD young adults residing in 25 U.S. states. Each AOFG lasted four consecutive days with two prompts posted by moderators per day. Overall, participant satisfaction with AOFGs was high: 98% reported their experience was excellent, very good, or good and 95% would be somewhat or very likely to sign up for another AOFG. This example is used to illustrate key methodological decision-points, acceptability of the method to participants, and lessons learned. The goal of this paper is to encourage other researchers, particularly health researchers, to consider using AOFGs and to engage with the method’s strengths and limitations in order to develop new opportunities for online technologies to enrich the field of qualitative health research.
Kumar, P. (2018). Rerouting the Narrative: Mapping the Online Identity Politics of the Tamil and Palestinian Diaspora. Social Media + Society. https://doi.org/10.1177/2056305118764429
Abstract. Drawing on the e-Diasporas Atlas project (www.e-diasporas.fr) and original empirical research, this study examines the complex role of the World Wide Web in supporting and enabling new types of diaspora identity politics. It compares the online identity politics of two conflict-generated diasporas: Tamils and Palestinians. Both of these stateless diaspora communities maintain a strong web presence and have mobilized around various secessionist attempts, grievance narratives, issue-agendas, and calls for the right to self-determination that have garnered significant attention from the international community and mainstream media in recent times. Analytical concepts from transnational advocacy networks (TANs) and social movement literature are used to draw attention to the dynamic identity-based processes and framing mechanisms that connect diasporic demands and political claims across online and offline environments. The data combine Tamil and Palestinian e-Diasporas hyperlink network maps with web-based content analysis and key respondent interviews. The study argues that online diasporic exchanges transcend host–homeland territorial boundaries and invite comparatively expressive forms of identity-based political engagements that are simultaneously both deeply local and digitally global. In particular, the analysis demonstrates that human rights–based language offers a unique streamlining bridge between various locales, countries of settlement, and the international system more broadly.
Lathen, L., & Laestadius, L. (2021). Reflections on Online Focus Group Research With Low Socio-Economic Status African American Adults During COVID-19. International Journal of Qualitative Methods. https://doi.org/10.1177/16094069211021713
Abstract. The COVID-19 pandemic has sped the adoption of online data collection approaches among qualitative researchers. While videoconferencing software has been a tremendous resource for replicating key aspects of the face-to-face focus group environment, online approaches to data collection also face unique challenges. Prior work has offered insights on the value of face-to-face versus online focus groups and strategies for improving the online focus group experience for participants and moderators. However, little has been published on the unique needs of participants from low socio-economic status (SES) populations. In light of the digital divide and the ways in which COVID-19 has exacerbated existing inequalities, researchers must seriously reflect on the ways in which SES and online methods intersect. To address this gap, we present reflections on two online focus groups conducted with low SES African American adults during COVID-19 to offer recommendations on areas of concern and potential solutions for researchers to consider. Three areas stand out as particularly important for reflection: (1) participant privacy, (2) online connectivity, (3) support and time allocations. Greater attention to the impact of SES can help ensure improved opportunities for full and equitable participation, allowing the voices of those who have been marginalized to be heard more clearly.
Lev-On, A., & Lowenstein-Barkai, H. (2019). Viewing diaries in an age of new media: An exploratory analysis of mobile phone app diaries versus paper diaries. Methodological Innovations. https://doi.org/10.1177/2059799119844442
This exploratory study inquires into the validity and reliability of dedicated mobile phone diary applications. We developed Watchy, a dedicated mobile viewing diary application, and compared users’ compliance and usage patterns with those of users of the paper viewing diaries. Participants received paper diaries or installed mobile diary apps, with or without daily reminders, to document their viewings over a 4-day period. Documentation was more extensive in the smartphone app with reminder group compared to the paper diary group. Reminders increased documentation rates. Extent of documentation decreased as the experiment progressed for mobile app users. Findings suggest that mobile viewing diaries are an important tool for viewing studies, yet their use requires careful planning.
Liu, M. (2020). Soliciting email addresses to re-contact online survey respondents: Results from web experiments. Methodological Innovations. https://doi.org/10.1177/2059799120937237
Abstract. There are many occasions where contact information needs to be collected from survey participants in order to achieve future contacts and conducting follow-up surveys. This article reports findings from two experiments into collecting respondent emails and sending the second survey invites. In the email collection experiment, when only one follow-up survey was mentioned, more respondents provided their emails, compare to when the emphasis was on the research purpose of the follow-up survey. However, the follow-up survey participation rates are similar among respondents who provided their emails regardless of the wording of the request. The invitation email subject line experiment shows that a generic requesting for opinion reduces the follow-up survey participation compared to the elements emphasizing survey sponsor and specialty opinions.
Richard, B., Sivo, S. A., Ford, R. C., Murphy, J., Boote, D. N., Witta, E., & Orlowski, M. (2021). A Guide to Conducting Online Focus Groups via Reddit. International Journal of Qualitative Methods. https://doi.org/10.1177/16094069211012217
Now more than ever there exists a need to conduct data collection online in a safe environment while ensuring that methodological rigor is not sacrificed. Widely available online platforms allow for text-based focus groups to be conducted quickly, easily, and efficiently, but protocols must be maintained to ensure they do not descend into casual observation of naturally occurring conversations. Various online platform options and their merits are discussed. Reddit is provided as a case study to illustrate the steps through which researchers can conduct an asynchronous online focus group. Key opportunities such as a similar quality of results, a lower cost, easier recruitment, and the ability to accommodate more sensitive topics are discussed, as well as challenges including a stigma against online focus groups, when they are most appropriate, and the potential for deviant behavior.
Tartory, R. (2020). Critical Discourse Analysis of Online Publications Ideology: A Case of Middle Eastern Online Publications. SAGE Open. https://doi.org/10.1177/2158244020941471
Abstract. This study assesses the reflection of Middle Eastern media networks on the states and the news concerning the oil crisis and Organization of the Petroleum Exporting Countries (OPEC) conditions. A critical discourse analysis approach is adopted to analyze 22 articles from Al Jazeera and Al Arabiya regarding the Qatar withdrawal from OPEC. The ideological choices vary as Al Jazeera is focused on the Qatar economy, while Al Arabiya on Saudi Arabia. These online publications have presented a positive self-presentation of their funded country while the negative representation of the other. Al Jazeera has pointed that the exit of Qatar from the OPEC is to focus on its other non–oil production sector, while Al Arabiya has pointed that this exit is due to the primary role of Saudi Arabia, with which Qatar has an on-going conflict. Qatar is symbolic to broaden regional division, which may later diffuse to other OPEC members and will leave no mark on the decision-making process of the alliance.
Young, A., Espinoza, F., Dodds, C., Rogers, K., & Giacoppo, R. (2021). Adapting an Online Survey Platform to Permit Translanguaging. Field Methods. https://doi.org/10.1177/1525822X21993966
Abstract. This article concerns online data capture using survey methods when the target population(s) comprise not just of several different language-using groups, but additionally populations who may be multilingual and whose total language repertoires are commonly employed in meaning-making practices—commonly referred to as translanguaging. It addresses whether current online data capture survey methods adequately respond to such population characteristics and demonstrates a worked example of how we adapted one electronic data capture software platform (REDCap) to present participants with not just multilingual but translanguaging engagement routes that also encompassed multimodal linguistic access in auditory, orthographic, and visual media. The study population comprised deaf young people. We share the technical (coding) adaptations made and discuss the relevance of our work for other linguistic populations.
Wang, D., & Liu, S. (2021). Doing Ethnography on Social Media: A Methodological Reflection on the Study of Online Groups in China. Qualitative Inquiry. https://doi.org/10.1177/10778004211014610
Abstract. This article draws on the two authors’ extensive fieldwork experiences in studying Chinese feminists and lawyers on social media to offer some thoughts on how to conduct qualitative research in the digitalized world. We argue that qualitative methods such as participation observation, in-depth interview, and textual analysis can provide thick descriptions and deep, localized knowledge of social processes that go far beyond the sketches of Big Data. Social science data collection and analysis on social media need not only Big Data’s bird’s-eye view, but also the day-to-day ethnographic immersion—“living on the sites” and interacting with research subjects over a long period of time. The rise of social media has not changed the basic principles of doing ethnography, such as the importance of immersion and reflexivity. Nevertheless, ethnography of online groups presents new challenges and opportunities in terms of accessing field sites, analyzing ethnographic data, and research ethics.