Unit 3: Compiling Multiple Sources of Evidence
Analyze HyFlex research with a clear purpose in mind
In Unit 2, you were introduced to a variety of individual strategies that can be used to evaluate your HyFlex teaching at the lesson and course levels.
These techniques should not be viewed as distinct, siloed activities. Rather, evaluation, regardless of modality, should consist of obtaining specific information from multiple sources, and then interpreting the information holistically. Combined findings from the various sources of evidence tell a story about your teaching and its effectiveness.
To illustrate how multiple sources of data can be pulled together to provide a comprehensive picture of your course, let’s take a look at some existing research for HyFlex.
Example
Expanding Learning Opportunities for Graduate Students with HyFlex Course Design
Abdelmalak and Parra (2016) redesigned a course so that it included face-to-face components as well as synchronous online and an asynchronous option that involved watching recorded classes. The study involved only 6 graduate students at one institution in the United States and data collection methods were:
- Interviews: 60-minute interviews were conducted with each student at the end of the course. The interview questions focused on the following: What benefits did you get from having both face-to-face and online approaches in the same course with giving you the opportunity to choose your mode of participation? What were the disadvantages or challenges?
- Observations: “The face-to-face and the online synchronous meetings were observed for 5 hours per month for one semester (approximately 3.5 months). The observation protocol included: the implementation of HyFlex and the physical manifestations of participants’ responses to the learning experience.” (p. 23)
- Recordings of Class Meetings: Each class was recorded and transcribed. The recordings helped the researchers better understand HyFlex implementation and student responses to the learning experience.
- Online Course Artifacts: The syllabus and online content were reviewed. Student coursework was analyzed and used to contextualize students’ responses about their learning experiences.
In the end, researchers concluded the following:
…participants perceived HyFlex to be a good way to accommodate student needs and their life circumstances, increase student access to course content and instruction, differentiate instruction to meet adult students’ different learning styles and strategies, and give students a sense of control over their learning. (p. 19)
Though these data collection approaches were used for research, specifically for exploring student perspectives about the HyFlex design, you can use similar strategies when evaluating the effectiveness of your teaching.
Let’s explore another research study:
Example
Understanding the Needs of Adult Graduate Students: An Exploratory Case Study of a HyFlex Learning Environment
Kostinen (2018) conducted a qualitative study to understand the experiences of students in a Hyflex environment. The setting was three 15-week graduate or upper-year computer science courses at one institution in the United States.
They collected data using the following strategies:
- Student Survey: The survey was sent to all students in each of the classes and focused on student preferences for the instructional modalities and preferences in taking a HyFlex course
- They used a five-point Likert scale ranging from “Strongly Agree” to “Strongly Disagree” with a couple open-ended questions. Sample questions were:
- I am satisfied with the way my HyFlex class was delivered/presented
- I found it valuable to have a recording of the entire class
- I found it valuable to be able to stream into class when I could not physically attend
- I found it valuable to have short video lectures in addition to the recording of the class
- I feel that having the option to choose how I attended each class helped me get a better grade(s)
- If I had the opportunity, I would like to take another HyFlex class
- Taking a HyFlex course allowed me to better learn the content by giving me flexibility
- HyFlex courses fit into my schedule easier than a full face-to-face course
- What influenced your decision to attend the Hyflex class the way you chose to (in person, online, or on video conferencing)?
- Did you find that your performance in the course was affected by your choice of attendance? Why or why not?
- They used a five-point Likert scale ranging from “Strongly Agree” to “Strongly Disagree” with a couple open-ended questions. Sample questions were:
- Student Interviews: Eight interviews were conducted.
- Examples of questions were:
- How did the flexibility help or hinder the ability to learn about the subject course material?
- If you missed a class either face-to-face or through video conferencing, what steps did you take to learn the material independently? Did you find you were able to understand the course lectures and course material in a sufficient manner?
- What factors did you consider in making your decision whether to attend class each week?
- How did you get to know your classmates?
- Over the course of the semester, did you feel a sense of community with your classmates?
- What would you say were the overall strengths of a HyFlex learning environment for graduate classes? What were the weaknesses?
- Examples of questions were:
- Instructor Survey: The instructor for each of the courses was asked about the course design of the Hyflex model.
- Observations: The researchers observed both classes in face-to-face and synchronous formats. These observations focused on: “how students interact with each other, how the instructor teaches through the technology and ways in which communication with synchronous helps or hinders the class” (p. 43).
- Other Data: Syllabi of the courses, documentation provided by the institution to instructors of HyFlex courses, and data on the activity of students through the Learning Management System.
Overall Findings:
The flexibility of the HyFlex format was the main benefit reported by participants, and they also noted that they preferred HyFlex in comparison to other formats like fully online or hybrid.
The difficulties with HyFlex focused on communication challenges, particularly with the students who were not face-to-face. Face-to-face was also the preferred modality for the participants.
Both examples provide data-collecting strategies in assessing the quality of HyFlex teaching and design. While you may never conduct an official study of your own, analyzing the existing research on HyFlex can give you some ideas about gathering constructive feedback to improve your Hyflex course.
Record your findings
In the same way that research is strengthened by collecting information from multiple sources of data, so too is the evaluation of your HyFlex teaching. Simply relying on one strategy or source of data to determine your effectiveness will not provide a complete picture.
The goal is to be intentional about the information you collect and from whom.
There are a variety of different ways to record the evaluation strategies you want to use and the sources from which that information is derived.
Generally, the process for planning these strategies involves identifying what you want to know. What specific indicators do you want to explore? Identify the focus and then consider the most appropriate way of collecting the information.
Example
In their Guide for Providing Evidence of Teaching (CC BY-NC 4.0), Kenny et al. (2018) focus on various facets of teaching: teaching and supporting learning, supervision and mentorship, professional learning and development, and educational leadership.
View page 2 of the guide for a completed template. Though the focus is more broad and applicable to who you are as an instructor, the same template can be applied at the course level.
Activity
This activity involves considering how you are going to think about obtaining feedback while delivering your course.
Step 1. Download the worksheet Lesson or Course-Level Evaluation Plan [.docx].
Step 2. Complete the worksheet by planning when you are going to collect different kinds of information throughout your course, what you are going to focus on, and the source of evidence. We provided some additional prompts for you to consider as you develop your plan.
If you are new to the process of information-collection planning, the task may seem daunting. Our recommendation is to start small. Currently, you most likely use student course evaluations at the end of the term; what is one additional piece of information you can collect in your next HyFlex course?
Gradually, you will become more comfortable with this process and will continue to refine your teaching evaluation practices at the course level.
Using multiple data collection techniques at the instructor level
Although the focus of this module has been about evaluating the effectiveness of your HyFlex teaching at the course level, let’s now broaden this discussion to the instructor level. Consider how your engagement with HyFlex teaching complements who you are as an educator.
A teaching portfolio/dossier is one source of information that can help you assess your effectiveness at the instructor level. Because of the popularity of teaching portfolios/dossiers, we are going to focus on these for the remainder of this unit.
Overview of Teaching Dossiers
The Canadian Association of University Teachers (2018) suggests that the measurement of effective teaching is best captured through the use of a teaching dossier and the multiple sources of evidence contained within it (e.g., statement of teaching philosophy, examples of lesson plans, information from colleagues, students, and others).
You may, or may not have a teaching dossier as part of your reflective practice. Teaching dossiers (typically Canadian) or teaching portfolios (more commonly used in the U.S.) are accepted as ways in which effective teaching can be measured.
Your teaching dossier is a living document that should be continuously updated.
At times, a teaching dossier is used to evaluate an individual’s work for tenure and promotion; it is also a way in which individuals reflect on their teaching. It is recommended that you would construct your teaching dossiers for a particular purpose.
Some people have expressed discomfort with this idea, that a dossier is different depending on the purpose. The criteria for a teaching dossier for an award may be different than that for tenure and promotion, and you might see other elements as more representative of your teaching.
At its most basic level though, a teaching dossier is your compilation of educational artifacts that reflect your teaching and learning endeavours.
Activity
If you are not familiar with teaching dossiers, we encourage you to work through the self-paced activity below, Module 1 Section 1: What is a Teaching Dossier (CC BY-NC-SA 4.0). The modules were co-created by Ryerson (X) University, University of Toronto, University of Waterloo, University of Windsor and Western University.
This module provides a general overview of what a teaching dossier is and how it is used. Though we have not adapted this for the HyFlex environment, it gives you some background on dossiers in general.
Now is a good time to pull out your teaching dossier and take a close look at what is included. Consider how your teaching in the Hyflex environment could be captured in this portfolio.
For example, what are the principles of your teaching philosophy and practice and how do these align (or not) to what you know about Hyflex teaching?
Contents of Teaching Dossiers
Teaching dossiers can include a range of items. Typically, there is the main body to the dossier as well as appendices, which consist of artifacts.
The basic structure of a teaching dossier should include the statement of teaching philosophy, evidence to support the claims that you made in that statement, and then descriptions and/or analysis of the evidence that you detailed. This description is done using a narrative format.
A small sample of potential items that could go into a teacher dossier includes the statement of teaching philosophy, list of courses taught, instructional strategies, formative evaluation strategies, lesson plans, or peer observation.
For Hyflex teaching specifically, you might include these items:
- Statement of teaching philosophy focused on HyFlex
- Courses you have taught using HyFlex (and other modalities)
- Design for a HyFlex course
- Lesson plan for HyFlex lessons
- Formative and summative feedback directly related to the HyFlex nature of the course
- Teaching observation from a peer or member of a teaching and learning centre
Activity
Part 1
If you are new to teaching dossiers, please continue the self-paced activity below, Module 1 Section 2: Characteristics and Components of a Teaching Dossier
(CC BY-NC-SA 4.0). Please note that these have not been adapted for HyFlex.
Part 2
Now let’s consider adapting your teaching dossier for HyFlex teaching-related activities.
Step 1 – Collect and organize materials based on modality. For example, if you are new to teaching HyFlex but have taught fully online asynchronous courses, compile a folder of materials that speak to your teaching-related activities in this capacity. As you gain more experience with HyFlex, you could create a folder of items you have collected that are linked to teaching using this approach.
Step 2 – Determine your purpose. When creating a teaching dossier for a specific purpose, like an award, you can look across your folders for different modalities and pull together the relevant documents.
If it is an award or job promotion that highlights teaching with technology or experience in HyFlex specifically, you will already have your folder of information to draw on and synthesize.
Teaching Philosophies
Statements about your teaching philosophies describe your beliefs and values, where these beliefs and values come from, why you have them, and how you use them in your teaching practice. These statements can often be challenging to write; it can be difficult to communicate exactly what is important to you, why, and how you apply it.
Similar to teaching dossier in general, these teaching philosophies will also evolve as we continue to develop our teaching practices. For example, if you have never taught HyFlex before, but are teaching a HyFlex course this term, you may wish to integrate aspects of that new experience into your teaching philosophy.
It is important to engage in reflective practice and evolve as an educator.
Kenny (2015) proposes that a teaching philosophy statement can include this structure: beliefs, strategies, impact, and goals.
Kenny (2015) describes each of these components in the following ways:
Kenny (2015) also provides guiding questions for each of these sections to help you write a teaching philosophy statement.
While the main questions below have been copied verbatim from Kenny’s blog post, we added questions with a HyFlex focus.
Activity
In a blank text document, brainstorm answers to the questions above.
If you have never taught HyFlex, focus on the Beliefs and Goals sections. You could also provide hypothetical answers for the Strategies and Impact sections. What strategies do you plan on using in HyFlex? How will you collect evidence of your impact in the HyFlex classroom?
Treat this as a living document. When you gain experience and learn more about HyFlex, come back to it to capture additional thoughts.
This brainstormed list of answers can become the first document in your teaching dossier of HyFlex-related teaching activities.
Unit Summary
In this unit, you learned about how researchers utilize multiple data collection strategies to obtain a fuller picture of what takes place in a HyFlex classroom. You then considered how this applied to the HyFlex context and practiced planning your HyFlex course evaluation strategies.
Lastly, you explored how you can integrate your knowledge, beliefs, and experience related to HyFlex into your teaching dossier, and began establishing a HyFlex-informed teaching philosophy statement.
