1-3 Common Methods of Education Research

Overview

Many research methodologies are closely linked to specific fields, but in the health professions, we borrow methodologies from various disciplines (See Chapter 1-1). For example, carefully controlled experiments, in which all known variables are measured or standardized, are required in studies of chemical reactions. Researchers must be able to clearly report the sequence of methods used and the precise measurements of all elements included in each study in order to allow for replication of their study design and possibly also their results. In these contexts, the ability to follow closely the same steps and reproduce the same results is synonymous with scientific rigour. In health professions education research, the main focus is on the experience and study of phenomena related to humans. Humans (and the contexts in which they interact) are complex and cannot be controlled as well as chemicals. Furthermore, most contexts present significant constraints on the kind of research that can be conducted. While a chemical reaction can be replicated several times using identical elements, each human participant in a study brings a plethora of experiences that can create unpredictable outcomes. Therefore, research methods must be used more flexibly, so that the concept of scientific rigour is not as closely tied to standardized procedures, precisely measured elements or replication. Indeed, it is more likely that replication is not exactly possible across diverse contexts. Additionally, study results may need to be interpreted flexibly; inductively or deductively (Chapter 1-2). This chapter first describes several well-defined methods and then introduces the reader to some central principles to guide the selection of a research methodology. It then briefly presents worked examples of how to translate interesting phenomena into a researchable question that can be matched to a feasible research method.

Vignette

Having successfully published their first qualitative research paper, an ambitious constructivist grounded theory study, Rayna was now interested in testing the prevalence of the phenomenon, and wanted to know if these experiences were universal. They realize that this new work may bring them back to a quantitative research wheelhouse. They begin to construct a team with the goal of designing an experiment. However, at their first meeting, there is strong disagreement amongst the team that an experiment is the best choice.

Deeper Dive on this Concept

All phenomena can be studied using methods that are classified as qualitative, quantitative or both (mixed). Qualitative methods rely on data that are less structured: words, pictures. These can include drawings, live observation, interview-based conversations and focus group interactions. Quantitative methods rely on data that are more structured: numbers. These can include test scores, survey responses to scaled responses, counts of events, and counts of people. As implied, mixed methods integrate both kinds of data in one study. We will first explore the elements of specific designs. You may find that your prior experience with research will anchor you to certain types of research questions and study designs; if your thesis included regression modelling, all papers seem to be able to be done via regression modeling from then on. However, in HPER, as we have demonstrated earlier, limiting yourself to one approach often limits your goals and your ability to have a broader impact (Chapter 1-1). The best way to break that cycle is to expand your worldview. In Chapter 2-1, we describe how you might DO a literature review for METHODS in your domain. Use this to understand what other types of science are being used to better understand your topic.

Expanding your own world view does not mean you have to be an expert in every design; methodological playfulness is not for everyone. The best way to learn new methods is to partner and collaborate. Consider the principles covered in Chapter 4-2 on working collaboratively to build a research team.

Common Methodologies in Health Professions Education Research

Collaboration generally begins with some shared understanding. To begin building your knowledge of different research methodologies, we describe seven common research methods used in HPE, across qualitative, quantitative and mixed methodologies

Qualitative

Qualitative research is increasingly common in medical education, and is used to describe a phenomenon or to understand how or why the phenomenon works the way it does. In what follows, we offer an introduction to three common qualitative methodologies. And while there is overlap between most qualitative methodologies, each has unique goals and procedures. Differentiating between them will help you make choices about which methodology might be the best fit for your research goals and context

Constructivist Grounded Theory (1) is a methodology commonly used in HPE research. As its name suggests, it is best used to generate new theory on a social process or phenomenon. To do this, it relies on hallmark methods such as concurrent analysis (data are generated and analyzed at the same time), theoretical sampling (researchers use concurrent analyses to direct their sampling toward the participants best able to inform the development of their theory), and constant comparison (researchers iteratively return to the data to compare data to each other, and their developing analyses to the data).

Phenomenology is the study of experience. Rather than seeking to theorize experience across participants (as in Constructivist Grounded Theory), they seek to deeply understand the nature of a phenomenon for those who experience it first-hand. A hallmark of phenomenology include deep engagement with individual participants, often through lengthy and recurring interviews.

Ethnography is one of the oldest qualitative research methodologies. Ethnographers study culture, usually within defined social groups (in interprofessional teams, for example, or within a class of first-year medical students). They spend significant time in the field and conduct observations to study the group’s social processes and practices; they also often interview participants or review documents to try to make meaning out of what they observe, from different perspectives.

Quantitative

While the culture of research in the health professions is changing, there remain strong beliefs regarding the rigour of quantitative methodologies. One reason is that the Randomized Control Trial (RCT) is held as the gold standard for clinical research, and so remains a dominant paradigm in the mind of most clinicians. While it is possible to design an experiment similar to an RCT to evaluate education theory, often it is not possible to maintain the strict constraints in education studies. For example, it would be impossible to control multiple participant characteristics, or ensure that participants are not exposed to other training or experiences which could influence study results. Therefore, most experiments in education are really practical trials.

Practical trials are essentially RCTs in which one or more elements are not possible. For example, in a longitudinal study of different approaches to education, it may not be possible to randomly assign students to different groups. In this case, the researcher may have to rely on naturally occurring cohorts to act as ‘groups’. Health professions education can benefit from practical trials that evaluate whether proposed theories stand up to the complexity of semi-controlled contexts. Convenience sampling and pseudo-random assignment are acceptable techniques.

Surveys are structured questionnaires which measure self-reported attitudes, experiences or behaviours. These responses are expected to define how much of a given construct a respondent has. Health professions education can benefit from surveys that detect common patterns in attitudes and experiences. However, this methodology is easy to overuse or use poorly. Consider reviewing guides on survey design (2-4).

Case audits are studies that examine how often clinical practice guidelines are followed.  These studies can be useful to identify areas in health professions education that require more attention. Alternatively, audits can measure the effectiveness of a recent education intervention on changing practice or improving patient safety. For example, an audit approach combined with an in situ training can measure the impact of quality improvement initiatives.

A Note about Mixed Methods

Mixed methods designs are increasingly popular in health professions education and, while they are not the right answer to every research question, they can offer the benefits of both qualitative and quantitative study designs.

Good mixed methods designs are thoughtful about how qualitative and quantitative data will be combined to best answer the research questions. There are many different ways of thinking about mixed methods design, but a common way of thinking about mixed methods involves distinguishing between convergent/parallel designs (in which qualitative and quantitative data are generated separately but simultaneously, then integrated in the final analysis) and sequential designs (in which one type of data is generated first and used to inform the other). Sequential designs can be exploratory (where qualitative data are first generated to explore the phenomenon that quantitative data can then confirm) or explanatory (where quantitative data are generated first, then used to inform qualitative exploration of the phenomenon).

Making Choices

The choice of which method to use depends on the available resources (which includes methodological expertise), the aspect of the phenomenon that is being studied, and the perceived impact of the kind of data that will be produced. The first step in designing any study is to clearly identify your goal. Do you want to inform? Do you want to better understand? Do you want to change behaviour? A different version of these goals is described by Cook et al. (4). In this chapter we explore the goals of informing, understanding and changing behaviour because the central focus is not an education intervention; rather we consider the broader concept of exploring phenomena in health professions education. The second step is always to read around the topic in depth; re-evaluating your goal if necessary. The third step is to translate your goal into a research question, which will likely point you to the best research methodology. Let’s consider these potential research goals, using the phenomenon of ‘burnout’ to explore how each goal might translate into a research question.

To Inform

At one point, the concept of burnout among health professionals was not well understood. In this case your goal may be to describe burnout to increase awareness and motivation to study it closely. This goal seems best aligned with an observational study, or a survey. In an observational study, you would act much like an anthropologist, or ethnographer, observing all evidence of the phenomenon in practice. You could rely on a rigorous literature review to refine your approach and the contexts you choose to observe. An observational study of burnout could be quite ambitious as it may require following individuals closely, observing people at their lowest points and multiple observer perspectives at multiple time points. On the other hand, a survey allows various individuals to report their experiences in an anonymized and structured manner. A carefully crafted survey, supported by a rigorous literature review, can tap into shared experiences, common definitions and frequencies of events related to burnout. Your own preference for one of these designs and access to the necessary resources will ultimately drive your decision. If you refined your goal into a specific research question, such as “How often is burnout experienced among emergency department nurses?”, the alignment to a survey becomes clearer.

To Understand

Perhaps you have read about a phenomenon and you want to explain it further. If we stay with the burnout example, you would be joining many health professions education researchers who have explored this phenomenon in a variety of ways. Your literature review will help you identify what aspect of burnout you want to understand better, which could lead you to a very specific question: “How does family support impact the experience of and recovery from burnout?”, for example. This particular question is not well aligned to a quantitative methodology. Primarily because this is not a phenomenon that can ethically be studied using an experiment, and a survey would not facilitate the rich data required to explain the role of family support. In a situation like this, a study that relies on data gathered during interviews with health professionals who experienced burnout would be more valuable. You might use phenomenology to develop a rich appreciation of the experience or constructivist grounded theory to develop a theory of the process of recovering from burnout.  Subsequently, once you feel you have a strong understanding of the phenomenon, and have developed a theory of how it manifests, you may choose to design a survey to support your theory.

To Change Behaviour

Building on the prior goals of exploring and understanding, you may now be interested in changing how health professionals respond to signs of burnout. Your previous research or literature review may have provided you with insights regarding how best to prevent burnout or reduce its impact. In this case you might be interested in gathering evidence in favour of these strategies, and exploring options for funding to introduce these strategies at your local institution so that you can study their impact. Some researchers benefit from, for lack of a better word, luck. For example, if a new program to help with burnout is introduced at your institution, you suddenly have the ideal context (informally known as a ‘natural experiment’) for gathering evidence of outcomes. Much like an experimental design, you could work with the program directors to design two intervention groups and measure the outcomes quantitatively, through a survey of burnout, or qualitatively, through a series of interviews. Your findings may then support investment of additional resources in the strategies that had the most positive impact, or the one that was most feasible and cost effective.

By framing research through these 3 potential goals, we hoped to highlight how fluid choices in research methodology can be. In contrast to the notion that there is one ‘gold standard’ methodology, the goal of this chapter is to emphasize that the choice of research methodology is dependent on numerous factors. Indeed, the same group of researchers may use a multitude of methodologies to investigate the same phenomenon, creating a robust and impactful program of research. However, your specific and immediate goal will determine which approach you should choose. Consider reading more on this topic (6, 7). Table 1.3.1 details some worked examples.

Vignette Conclusion

After some discussion, Rayna and the rest of the research team agreed to proceed with a few study designs. The group felt it was best to first conduct an observational/qualitative study before proceeding to identify quantitative methodologies for exploring how different factors affect the phenomenon.

References

About the authors

name: Renate Kahlke

institution: McMaster University

Renate Kahlke is an Assistant Professor within the Division of Education & Innovation, Department of Medicine. She is a scientist within the McMaster Education Research, Innovation and Theory (MERIT) Program.

name: Sandra Monteiro

institution: McMaster University

Sandra Monteiro is an Associate Professor within the Department of Medicine, Division of Education and Innovation, Faculty of Health Sciences, McMaster University. She holds a joint appointment within the Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University.