Bringing Rigorous Methods to Real-World Problems

This post is the second of three by guest contributors Matthew Aplin-Houtz and Sarah Willey. See the first post, Asking the Right Question, for more real-world research advice.

There is often an assumption that research can be truly rigorous, or it can be “real world” research, and that the two are somehow mutually exclusive or at polar opposite ends of a spectrum. However, the world is not that black and white, and it is possible to have our cake and eat it, too.

In 2021, we took an interest based on one of our team’s professional experiences in the nonprofit sector in examining the content of nonprofit mission statements. Sarah was in the middle of strategic planning and rebranding with one of her clients, including considering edits to the organization’s mission statement. At the same time, as students in a Doctor of Business Administration program, we were learning new qualitative and quantitative analysis methods. We decided to ask some questions about mission statements and try out some of the new methodologies we were learning to bring academic rigor to our own real-world business problems.

 The first of two studies we conducted on a sample of 200 randomly selected mission statements from US-based nonprofit organizations serving the homeless population employed a sentiment analysis and moderation analysis. The second study involved manually coding the mission statement text based on theory-informed a priori code set and then using exploratory factor analysis (EFA) as a mathematical method for theme development, using the percent of coverage of each code within the mission statement set as variables.

This seemingly simple use of EFA likely would not be exciting to a traditional quantitative researcher; EFA tends only to be a starting point of research efforts and is commonplace in the literature. However, the implications for a qualitative researcher are quite profound. Using EFA in the format of exploratory theme analysis (ETA) gives a new way to find latent themes and validate them mathematically. To our knowledge, this method is novel for the qualitative realm and provides an innovative way to evaluate the interactions and relationships of qualitative coding because the use of ETA can be a gateway to establishing relationships that may not be available to the naked eye. The combination of the percentage of coding coverage alongside categorical variables such as demographics is not something readily available to a qualitative researcher. Many qualitative researchers strive to utilize tools from the quantitative realm to provide increased academic rigor via statistical validation. An example of how qualitative researchers use pseudo-quantitative methods can be found in inter-rater reliability, wherein researchers attempt to find a minimum consistency of coding efforts between coders as a metric of scientific reliability.

In our ETA study of mission statements, we were able to identify a need for inductive coding solely based on a simple a priori code set and the mathematical value assigned through sentiment analysis. In traditional qualitative content analysis, a researcher would need to spend a significant amount of time juxtaposing each mission statement against the other, and a sample of 200 mission statements would be unwieldy. These efforts take significant amounts of time. Exploratory theme analysis allows for the evaluation of much larger data sets in far less time.

The results of this study are directly applicable to practice in the real world. Armed with the information that just these initial studies present, nonprofit executives, boards of directors, consultants, and others involved in crafting organizational mission statements can make different choices that may very well impact their bottom line – and their ability to transform lives through their missions. This presents a new challenge that we will pick up in a future post – how do we make sure our research results reach an interested practitioner audience?

---

Matthew Aplin-Houtz: While acting as COO for Body Balance Physical Medicine and as a business advisor for the Indiana Small Business Development Center, he researches in organizational behavior. Current DBA student at UMSL. MBA - Indiana State University, Master of Music degrees from Butler University and University of Maine.

Sarah Willey: A nonprofit executive, fundraising consultant, and thought leader, Sarah is passionate about bridging worlds of practice and theory, with a particular focus on issues related to management and fundraising in the nonprofit sector. She is a DBA student at UMSL and holds a Master’s in Nonprofit Management from Washington University.

---