Culture and Evaluation

SAGE Methodspace hosted a webinar about culture and research design in March 2022, and cultural issues have been the subject of numerous posts. In April 2022 the focus is on evaluation. Since evaluation often entails reviewing community needs and priorities, or measuring success of projects or programs, culture is important to consider. Will the evaluation give an accurate picture and include all relevant perspectives? Will the researcher be able to access key members or stakeholders? This list of open-access articles offers some suggestions and examples to guide you.

---

Aguila, E., Weidmer, B. A., Rivera Illingworth, A., & Martinez, H. (2016). Culturally Competent Informed-Consent Process to Evaluate a Social Policy for Older Persons With Low Literacy: The Mexican Case. SAGE Open. https://doi.org/10.1177/2158244016665886

Abstract. The informed-consent process seeks to provide complete information to participants about a research project and to protect personal information they may disclose. In this article, we present an informed-consent process that we piloted and improved to obtain consent from older adults in Yucatan, Mexico. Respondents had limited fluency in Spanish, spoke the local Mayan language, and had some physical limitations due to their age. We describe how we adapted the informed-consent process to comply with U.S. and Mexican regulations, while simplifying the forms and providing them in Spanish and Mayan. We present the challenges and lessons learned when dealing with low-literacy older populations, some with diminished autonomy, in a bilingual context and a binational approach to the legal framework.

Clarke, G. S., Douglas, E. B., House, M. J., Hudgins, K. E. G., Campos, S., & Vaughn, E. E. (2021). Empowering Indigenous Communities Through a Participatory, Culturally Responsive Evaluation of a Federal Program for Older Americans. American Journal of Evaluation. https://doi.org/10.1177/10982140211030557

Abstract. This article describes our experience of conducting a 5-year, culturally responsive evaluation of a federal program with Indigenous communities. It describes how we adapted tenets from “participatory evaluation models” to ensure cultural relevance and empowerment. We provide recommendations for evaluators engaged in similar efforts. The evaluation included stakeholder engagement through a Steering Committee and an Evaluation Working Group in designing and implementing the evaluation. That engagement facilitated attention to Indigenous cultural values in developing a program logic model and medicine wheel and in gathering local perspectives through storytelling to facilitate understanding of community traditions. Our ongoing assessment of program grantees’ needs shaped our approach to evaluation capacity building and development of a diverse array of experiential learning opportunities and user-friendly tools and resources. We present practical strategies from lessons learned during the evaluation design and implementation phases of our project that might be useful for other evaluators.

Hanberger, A. (2010). Multicultural Awareness in Evaluation: Dilemmas and Challenges. Evaluation, 16(2), 177–191. https://doi.org/10.1177/1356389010361561

Abstract. The purpose of this article is to discuss what is meant by multicultural competence in evaluation and how policies and programmes aiming at multicultural awareness and ‘validity’ can be evaluated. The article discusses three main ways of understanding multiculturalism and how multicultural competence in evaluation can be defined. It also develops evaluation criteria that can be used for assessing the multicultural implications of policies and programmes. The article suggests that a multiculturally competent evaluator should be well informed about minority and majority norms and also familiar with different models of multiculturalism. The multiculturally aware evaluator employs an appreciative approach to traditional cultures that is consistent with human rights and international law. A multiculturally relevant evaluation should stimulate a discussion that facilitates inter-cultural understanding and multicultural awareness. Developing multicultural awareness in evaluation can be seen as a way of developing democratic evaluation.

Hayward, A., Sjoblom, E., Sinclair, S., & Cidro, J. (2021). A New Era of Indigenous Research: Community-based Indigenous Research Ethics Protocols in Canada. Journal of Empirical Research on Human Research Ethics, 16(4), 403–417. https://doi.org/10.1177/15562646211023705

Abstract. Indigenous communities across Canada have established principles to guide ethical research within their respective communities. Thorough cataloging and description of these would inform university research ethics boards, researchers, and scholars and facilitate meaningful research that respects Indigenous-defined ethical values. A scoping study was conducted of all relevant peer-reviewed literature and public-facing Indigenous research ethical guidelines from First Nations, Metis, and Inuit communities and organizations in Canada. A total of 20 different Indigenous research ethics boards, frameworks, and protocols were identified. Analysis resulted in three key themes: (1) balancing individual and collective rights; (2) upholding culturally-grounded ethical principles; and (3) ensuring community-driven/self-determined research. Findings demonstrate how employment of Indigenous ethical principles in research positively contributes to research outcomes.

Pertiwi, Y. G., Geers, A. L., & Lee, Y.-T. (2020). Rethinking intergroup contact across cultures: Predicting outgroup evaluations using different types of contact, group status, and perceived sociopolitical contexts. Journal of Pacific Rim Psychology. https://doi.org/10.1017/prp.2020.9

Abstract. Two studies were conducted to examine the relationship between three different types of intergroup contact (i.e., direct contact, extended contact, and online contact) and outgroup evaluation, and the moderating effect of group status and sociopolitical contexts across two cultural contexts. A total of 75 European Americans (majority) and 44 Chinese Americans (minority) participated in Study 1; whereas 61 Javanese (majority) and 72 Chinese Indonesians (minority) participated in Study 2. In both studies, participants completed an outgroup feeling thermometer as well as a set of questionnaires measuring intergroup contact, perceived outgroup political power, perceived outgroup economic power, perceived government support, and perceived quality of the current intergroup relations. Results from the two studies revealed that although contact was beneficial in both cultural contexts, there were notable moderators of the links between contact and outgroup evaluations. Specifically, the value of direct contact was greater for the minority group members in the United States, extended contact only mattered in a specific condition when the perceived government support was taken into account in the United States, and online contact was beneficial across group status in the Indonesian context. Overall, the findings provide evidence of the need to take into account the role of specific sociopolitical relations between the two groups in intergroup relations research.

Riediger, N. D., Cyr, M., & Mignone, J. (2020). An Evaluation of an Experiential Learning Program in Global and Indigenous Health: The University of Manitoba’s Queen Elizabeth II Diamond Jubilee Scholarship Program. INQUIRY: The Journal of Health Care Organization, Provision, and Financing. https://doi.org/10.1177/0046958020951002

Abstract. We conducted a mixed-methods outcome evaluation to examine student experiences and learning in the University of Manitoba’s Queen Elizabeth II Diamond Jubilee Scholarship Program in Global and Indigenous Health. Our scholarship program is a bi-directional, 3-month international experiential learning program, including both undergraduate and graduate students, with associated online course focused on community engagement. Students completed a semi-structured narrative report at the conclusion of their funding related to their experience and learning. The Likert questions were analyzed descriptively and student responses to the open-ended questions were utilized for thematic analysis. Also included in this paper is a summary of our lessons learned through program administration. A total of 38 students completed the program between 2016 and 2018, with 95% reporting that they either met or exceeded their goals in the program. Three overarching and inter-related themes emerged in our thematic-analysis of students’ narrative reports, including success through relationships and new perspectives, challenges of the unfamiliar, and personal growth through strong emotions. Many students reported personal growth as their greatest success and linked this with new perspectives and awareness of how different contexts shaped their understanding of health issues. Overcoming challenges in their placements contributed to students’ confidence in their ability to problem-solve. Overall, students reported value in their experiential learning, which further supports the growing trend to incorporate both experiential learning and formal education in community engagement in public health pedagogy. However, international experiential learning requires considerable financial and human resource commitments to ensure its success.

van den Berg, R., Bours, D., Brousselle, A., Čekan, J., Chaplowe, S., Chelimsky, E., Davies, I., Felcis, W., Leiter, T., Menezes, D., Picciotto, R., Rogers, P., Rowe, A., & Uitto, J. (2022). What should evaluation learn from COP 26? Views of evaluation practitioners. Evaluation, 28(1), 7–35. https://doi.org/10.1177/13563890221074173

Abstract. Leading evaluation practitioners were asked about lessons from the recent 26th Conference of the Parties (COP26) for evaluation practice. Contributors emphasize the importance of evaluating equity between rich and poor countries and other forms of climate injustice. The role of the evaluation is questioned: what can evaluation be expected to do on its own and what requires collaboration across disciplines, professions and civil society – and across generations? Contributors discuss the implications of the post-Glasgow climate ‘pact’ for the continued relevance of evaluation. Should evaluators advocate for the marginalized and become activists on behalf of sustainability and climate justice – as well as advocates of evidence? Accountability-driven and evidence-based evaluation is needed to assess the effectiveness of investments in adaptation and mitigation. Causal pathways in different settings and ‘theories of no-change’ are needed to understand gaps between stakeholder promises and delivery. Evaluators should measure unintended consequences and what is often left unmeasured, and be sensitive to failure and unanticipated effects of funded actions. Evaluation timescales and units of analysis beyond particular programmes are needed to evaluate the complexities of climate change, sustainability and to take account of natural systems. The implications for evaluation commissioning and funding are discussed as well as the role of evaluation in programme-design and implementation.

Waddell-Henowitch, C., Gobeil, J., Tacan, F., Ford, M., Herron, R. V., Allan, J. A., Kruth, M. L., & Spence, S. (2022). A Collaborative Multi-Method Approach to Evaluating Indigenous Land-Based Learning With Men. International Journal of Qualitative Methods. https://doi.org/10.1177/16094069221082359

Abstract. In Canada, a vast majority of urban Indigenous people face distinct challenges accessing and connecting to Indigenous cultural practices. Research has found that colonial policies and practices continue to disrupt and fracture traditional methods of passing down cultural teachings, including dispossession from traditional lands in which cultural practices are rooted. This disruption continues to affect the availability of educational programming by and with Indigenous people and in Indigenous languages. This research involves a multi-method approach to observe and engage with a land-based traditional drum-making program for Indigenous men in an urban center in Southwestern Manitoba. By participating, watching, and listening to the men within the workshops through unstructured observation, Sharing Circles, individual interviews, and photovoice, we aim to understand the impacts of land-based learning on Indigenous men’s well-being. The study is designed in accordance with University and Tri-Agency ethical guidelines, integrating ownership, control, access and possession (OCAP), as well as the principles of respect, relevance, reciprocity, and responsibility within all phases of the research. The research is co-created by the university researchers, community collaborators, and other relevant stakeholders.

---

More Methodspace posts about culture and research