Annotiva - Developing a Prototype
Researchers:
Nako Abdullah, School of Education, University of Bristol
Basit Ansari, Bristol Composites Institute, University of Bristol
Dr Vahid Goodarzi Ardakani, Visiting Post-doctorate Researcher, University of Bristol
Qualitative coding, especially with extensive datasets, poses challenges for novice coders with diverse backgrounds (Evers, 2018). While existing qualitative data analysis systems (QDAS) aim to streamline coding processes, there is a gap in leveraging AI for collaborative coding refinement. Notably, little is known about how automated system supports transparency and refines inconsistency in coded suggestions to improve rigour in qualitative coding. Recent studies (Chen et al., 2018; Rietz & Maedche, 2021) have shown that commercial QDAS such as such as NVivo and MAQDAS lacks integration of interactive machine learning (ML) to support coding refinement that ultimately feeds up to a more transparent coding process. In response, our team introduces Annotiva, an AI-based coding assistant prototype that aims to semi-automate qualitative coding refinement by predicting ambiguity within and across coder interpretations.
Annotiva: the prototype
Annotiva is an interactive AI-supported coding system that covers not only prime coding procedures such as: (i) coding and annotating strings of lexical, phrases or sentences, but also complements a seamless collaborative coding experience by integrating the following features: (ii) definition of coding cookbook/coding rules, and (iii) ambiguity index that predict (dis)agreement between coded interpretations. Drawing from the QDAS literature highlighting challenges in ML interactivity and coding accuracy (see Chen et al, 2018; Jeannie, 2018), our prototype is designed based on the four main requirements, as depicted in Figure 1. While Annotiva’s main functionality is still in progress, the interface snapshots below (Figure 2 and 3) capture our current work.
Figure 1: Main functionality and features of Annotiva
Figure 2 illustrates the developing interface of Annotiva during the coding process and Figure 3 highlights how our AIx feature may indicate potential disagreements between coders’ interpretations.
Figure 2: The developing Annotiva user interface (UI)
Figure 3: Potential Annotiva AIx functionality to feed forward coding reflection and refinement.
Some challenges:
During Annotiva's development, we grappled with several significant challenges, including:
Accessing open-source datasets and language learning models (LLMs):
Devising algorithms and methodologies to identify patterns, themes, and relationships within unstructured data of multiple linguistic and contextual backgrounds was a daunting process. Specifically, we encountered challenges in refining algorithms that could accurately categorise and code qualitative responses, especially when faced with ambiguous or contradictory information. Pre-training dataset to discern linguistic ambiguities and interpret similar meanings from different terminologies is not a straightforward task. The Annotiva system must recognise terms such as ‘specimen’ and ‘sample’ provided by different coders whose background knowledge can be linguistically and contextually diverse. Given the limited expertise and access to datasets my team and perhaps other educational researchers currently have, training the prototype to undergo extensive training across various datasets can be challenging.
User interface (UI) design for intuitive use:
Designing a UI that is effective yet intuitive and user-friendly was another challenge. As Annotiva is designed for emerging researchers or novice coders to support collaborative coding, we need to ensure that potential users can navigate the tool with ease while maintaining robust features for data analysis. On this, we strive to strike a balance between functionality and simplicity, requiring careful consideration of overall UI design principles, such as layout, navigation, and information architecture.
What’s next?
Moving forward, we aim to pilot Annotiva's usability to assess how coder disagreements correlate with perceived data ambiguity. Particularly, we are interested in exploring how Annotiva, with its developing AIx functionality, addresses coding inconsistencies resulting from semantic ambiguity and raters' subjectivity across diverse contexts. It would be an intriguing next-step to explore how the granularity of coders’ interpretations can be enhanced and to an extent, partially automated through Annotiva without compromising the human touch to data interpretation, which is key in any qualitative research. Moreover, we are also keen to engage in more collaboration and knowledge sharing with other qualitative researchers and SAGE communities to improve and scale up Annotiva for overcome barriers to access and accelerate innovation in qualitative data analysis. We also welcome feedback from all researchers from all backgrounds on this exciting journey. Once the prototype is ready, you can be among the first to try Annotiva by visiting our page link here: https://annotiva.wixsite.com/annotiva. For any collaboration ideas and plans, feel free to reach out to us at annotiva.uk@gmail.com and we are keen to be engaged!
Links:
References:
Chen, N. C., Drouhard, M., Kocielnik, R., Suh, J., & Aragon, C. R. (2018). Using machine learning to support qualitative coding in social science: Shifting the focus to ambiguity. ACM Transactions on Interactive Intelligent Systems (TiiS), 8(2), 1-20.
Evers, J. C. (2018). Current Issues in Qualitative Data Analysis Sotware (QDAS): A User and Developer Perspective. The Qualitative Report, 23(13), 61-73. Retrieved from htp://nsuworks.nova.edu/tqr/vol23/iss13/5
Rietz, T., & Maedche, A. (2021, May). Cody: An AI-based system to semi-automate coding for qualitative research. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-14). https://doi.org/10.1145/3411764.3445591