2-3 Setting and Defending your Methodological Choices

Overview

In this Chapter, you will explore strategies for aligning and articulating your methodological choices for your audience. Whether this is for your readers or to survive peer review, it is important to articulate your perspective on WHY you have chosen your methodology and  methods and HOW they are linked to your research question.

Vignette

Xiang sighed with exasperation. Days away from his thesis defense, he had just received more questions from his external examiner around his methodological choices within his master’s thesis. The examiner had written notes that suggested that he needed to “substantiate” the rationale for his study design choices more. Based on his reading, he had never seen many of the sections the examiner requested within other people’s papers, and yet for his thesis he was expected to have a whole chapter around methodology. He shot an email off to his supervisor, Dr. Ubuntu, who was always there for him. Dr. U always did a great job at clarifying things via a quick email or more in depth pep talk…  and always gave great insights about how things fit together.

After quickly booking the meeting in his calendar, Xiang clicked on the link to read with great interest.

Deeper Dive on this Concept

Setting and NOT forgetting

Whether you’re a graduate student (like Xiang in our vignette) or an experienced researcher (like Camille), writing your methods  can be very important for your success when making presentations or preparing a publication. Whether you are submitting a study protocol to the research ethics board for review, a grant to a competition, or a paper for peer review, the readers of your research work will want to understand why and how you decided upon your research methods and how they align to your research questions and your outcomes. They may also expect you to state your epistemic roots (e.g. state your philosophy of science and how it informs your view on the question/methods).

For those new to research, it is important that there is a logical flow between your philosophy of science & epistemology and also between the research question, methodology, data generation and analysis methods, and the outcomes of your project. We will call this the “throughline” of your paper. When a project’s throughline is broken, that is a red flag for editors and reviewers – and is often what these individuals might consider a fatal flaw of the paper. As it is impossible to comment on every possible nuance involved in research alignment, we will focus on common mistakes in methodological alignment and communication.

Here are some common pitfalls that can occur with project throughlines:

1. Conflict or misalignment between your epistemology and the research question: There is an inextricable link between your epistemology as a scientist with the research question. One common problem can be that there is a misalignment between your view on the construction of knowledge and your research question. For instance, if you wanted to know how many people engaged in a certain faculty development activity, you probably shouldn’t use a constructivist approach to base your research design – this question likely is better framed within a post-positivist lens.
2. Contradiction between your research question and your selected theories: It can be very confusing when trying to articulate your theory, theoretical frameworks, and conceptual frameworks, let alone link them to your research question. (You may wish to review these concepts by listening again to the MERIT podcast episode on this topic (Theory, Theoretical Frameworks, and Conceptual Frameworks). In the podcast episode,  we explore how theory intersects with health professions education research and scholarship) However, it is very important to link how your specific research question is situated within all of these above frameworks.
3. Misalignment between your research question and  methods: Similar to the above situation, when deriving your research question it is important that you ensure that your methods are properly aligned to address and/or answer your research question. If you’re interested in deeply understanding the nature of someone’s experience with a certain clinical learning environment, it may not make sense to use a survey full of Likert scales and “select from the below list”-type questions. It may make more sense for this particular research question to be more qualitative in nature, and perhaps to adopt a phenomenological approach.
4. Lack of connection between your research question, methods, and intended outcomes: It is important for your outcomes to be lined up with all the components of your methods (e.g. data generation & analysis methods) as a misalignment can result in the production of data that can’t answer your research question, or leading to analyses that seem disconnected from the literature gap you sought to address.. Sometimes this misalignment comes when a research team gets lost in the data analysis phase. In many studies, the volume of data and plethora of possible interpretations can make it difficult to retain the original intention of the research, and results can fail to reflect the original research questions asked by the team. To prevent this, many quantitative and qualitative scholars will outline the outcomes they plan to deliver in their study protocols and grant applications. These statements can serve later as touchstones for your team. In qualitative research, in particular, study designs often evolve as the phenomenon is explored – as these changes occur, it’s critical that researchers maintain focus on their research questions to avoid straying too far from their original intent.

Thinking through and managing these potential pitfalls can ensure rigorous research design and head off many issues when it comes to peer review.

Defending or justifying your choices

When a graduate student presents their project at the end of training, this session is usually called a “defense”. Traditionally, this is a rite of passage that marks entry into the academy. Historically it was a public event where the other members of the academy would be invited to openly ask you questions and you would have to “defend” yourself against this public questioning. In today’s world, the “defense” is not nearly as open nor so adversarial- even if others within the community are invited to attend, it’s mainly to watch and rarely to ask difficult questions of the candidate. Instead, the thesis committee members and the external examiner usually aim to guide graduate students in displaying their thinking and justifying their research design and methodological choices. Likewise, when manuscripts are submitted to a journal for peer review, the conversation should be collegial; the editor and reviewers’ intent should be to offer advice on how to improve the manuscript and prepare it for publication, whether or not it is accepted at that specific journal.

There is an old adage that says that the best defense is a good offense, but, in less adversarial terms, the best way to make it through peer review or a defense  is to ensure that your choices are rigorous and defensible. And so, it can be quite useful for new researchers in the health professions education domain to imagine that they will need to “defend” the choices they make when designing and writing up their studies. Being articulate in prophylactically “defending” your choices can serve three main purposes:

1. If you are writing a study protocol: Working with your research team to articualte and justify your methodological choices (e.g. analysis plan, data collection choices, etc.) can help to align your research and your team, getting everyone  on the same page. This will help your team focus on the big picture and maintain the project throughline even when drowning in data or making sense out of analyses.
2. If you are intending to submit a grant application: Clear purpose and methodological alignment is one of the key factors that grant reviewers look for, and reviewers often heavily scrutinize the methods section. The merits of your science are often driven by the project logic, and so being thoughtful about why and how you selected your methods will help a grant reviewer better buy into your intended plan.
3. If you are writing your paper: Being transparent about your methodological choices will strengthen your methods sections and demonstrate the rigour of your work for editors and reviewers, since you will not be leaving your readers to guess at your intentions. Instead, plainly listing your intentions may provide a refreshing amount of clarity that will make your paper stand out.

For more on this topic, please consider listening to the listed podcast within the HPER curriculum:

Defending your Methods

In this podcast you will explore the ways in which you may need to message and explain your methodological choices. In this episode, Drs. Sandra Monteiro & Teresa Chan discuss the ways in which you might explain and/or defend your methodological choices.

Theory, Theoretical Frameworks, and Conceptual Frameworks

In this podcast you will explore how theory intersects with health professions education research and scholarship. Drs. Teresa Chan and Lara Varpio take you through these concepts together.

Vignette Conclusion

Xiang spends a lot of time reviewing the HPER resources and combing through academic articles to find ways to articulate his methods. He comes across O’Brien et al.’s Standards for Reporting Qualitative Research (1), which he finds extremely helpful in determining the level and type of detail he should include to instill confidence in the rigour of his work and satisfy his external reviewer. He realizes that many of the manuscripts he’s come across likely do not provide sufficient detail on their methods. But he’s still struggling with this question of alignment – how would he know if his project doesn’t have a clear “throughline”? One of his colleagues recommends Varpio et al, (2017) and he realizes that some of the language he has used to describe his methods (such as strategies to eliminate bias and member checking to confirm the ‘truth’ of each participant’s story) were more in keeping with post-positivist paradigms that he learned about in HPER, and didn’t really jive with his constructivist paradigm and choice of constructivist grounded theory as a methodology. He begins to revise his methodology chapter with an eye to ensuring that his decisions reflect the constructivist paradigm his study is rooted in.

References

1. O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. 2014;89(9):1245-1251. doi: 10.1097/ACM.0000000000000388
2. Varpio L, Paradis E, Uijtdehaage S, Young M. The Distinctions between Theory, Theoretical Framework, and Conceptual Framework. Acad Med. 2020;95(7):989-994. doi: 10.1097/ACM.0000000000003075
3. Varpio L, Ajjawi R, Monrouxe L, O’Brien B, Rees C. Shedding the cobra effect: problematizing thematic emergence, triangulation, saturation and member checking. Med Educ. 2017;51(1):40-50. doi: 10.1111/medu.13124

About the author

name: Teresa Chan

institution: McMaster University

Teresa Chan is an Associate Professor in the Department of Medicine within the Division of Emergency Medicine and cross-appointed to the Division of Education & Innovation. She is also the Associate Dean, Continuing Professional Development Office (CPD) and a Clinician Scientist, McMaster Education Research, Innovation & Theory (MERIT) Program, Faculty of Health Sciences, McMaster University.