Evaluating the effectiveness of HyFlex can be complex

In Unit 1 of this module, we referred to a broad definition of research that included asking questions, collecting data, and analyzing data. Data can broadly refer to various pieces of information or sources of evidence that are collected to increase knowledge about a topic or to answer the questions (Plano Clark & Creswell, 2015).

The goal of this unit is to introduce various sources of information to measure the effectiveness of your HyFlex teaching. You will be given a few sources of evidence you could use, and considerations for implementing these strategies for your HyFlex lesson or course.  That said, common strategies used to measure the effectiveness of face-to-face teaching are somewhat challenging when applied to online or HyFlex contexts.

In their book, Tobin et al. (2015, p. 15) outline some considerations when establishing and implementing evaluation methods for online environments:

• What is the online equivalent of visiting a colleague’s classroom for ninety minutes?
• In an online class, which elements count as “design” and which count as “teaching”?
• How much time should peer and administrative evaluators spend observing online teaching?
• Should observation of online teaching take place during or after the course?
• To what purposes will student ratings, self-evaluation, peer evaluation, and administrative observation be used?

Similar (and additional!) challenges are present within the HyFlex environment. For example, let’s explore the question “What is the online equivalent to visiting a colleague’s classroom for ninety minutes?” by considering additional questions per the following modalities:

• Online Asynchronous
  • Communication does not take place during a set time of day. Would an observation involve observing a Learning Management System for a week and looking at both the content of the course and other engagement tools like discussion boards, etc.?
  • How would indicators of effective teaching compare to those of other modalities?
• Online Synchronous
  • Does an observation include one period of class time?
  • Are the indicators of effective teaching in this synchronous environment the same as those for face-to-face?

There are certainly many complexities when it comes to identifying sources of information or implementing strategies for use in HyFlex. Additionally, there is no prescriptive way of selecting the strategies you are going to use. As Berk (2005) points out:

“A virtual smorgasbord of data sources awaits you.” (p. 48)

Selecting evaluations strategies for HyFlex

Evaluation strategies should not be chosen at random; they should be intentionally selected based on a variety of factors such as the information they provide, source of data, and/or program or institutional guidelines.

Some instructors will use certain strategies that are different from others – that’s okay!

Below, you will see a “smorgasbord” of strategies that have emerged from the academic literature. As you read through these strategies, think about the following:

1. Overall, which strategies apply more to the lesson or course level, in comparison to the instructor level holistically?
2. Within each resource, which strategies or criteria are more applicable to the lesson or course level than the instructor level?
3. For items that can be used for specific lessons or courses, how can they be applied to the HyFlex lesson or course specifically?

For an accessible-friendly version of the information above, download Sample of Strategies for Evaluation Teaching Effectiveness [.docx]

So, what did you determine after reviewing these literature-based strategies?

Though there is no right or wrong answer, one perspective is viewing Resource 1 (12 Strategies to Measure Teaching Effectiveness) and Resource 4 (Aspirational Model Teaching Criteria for Psychology) as more applicable to the instructor and their effectiveness as a whole, rather than the effectiveness of teaching in HyFlex within a lesson or course.

Resource 1 includes items such as employer ratings and teaching awards, and Resource 4 has items such as records of continuing education and local dissemination of knowledge about teaching. While you can certainly engage in all of these practices with a HyFlex viewpoint (e.g., disseminating knowledge about your experiences with HyFlex), they speak to who you are as an instructor and your effectiveness holistically.

Resource 2 (Administrative Courage to Evaluate the Complexities of Teaching) and Resource 3 (Statement on Student Evaluations of Teaching) include information that can speak to your effectiveness as an instructor, but more specifically, assess the effectiveness of your HyFlex teaching.

All items under Resource 2 (peer evaluation, student evaluation and input, and personal reflection) can be used at different points in a course and within the HyFlex context as well. Similar to Resource 2, all items in Resource 3 (peer observations, reviews of teaching materials, and self-reflection) can be used within a lesson or course, and also, within the HyFlex modality.

Even though there is a myriad of possibilities for what you could use to measure and demonstrate the effectiveness of your teaching, it is important to remember the purpose of your evaluation and how you are using the information.

Using a holistic approach to evaluation

There has been an increasing need for more holistic approaches when measuring the effectiveness of teaching, rather than relying solely on student evaluations. Dalhousie University is an example of one institution that has implemented a holistic evaluation of teaching policy, which was approved in 2021.

The aim of the policy is to ensure that evaluation of teaching takes multiple sources of teaching evidence into account (Dalhousie University Centre for Learning and Teaching, n.d.). The three sources of evidence of teaching effectiveness (student, peer, and self) and the relationship between them, are illustrated in the following diagram:

This policy and the diagram are referring to the instructor level, and how you can measure and describe your effectiveness as an instructor for a variety of purposes, including decision-making such as tenure, reappointment, promotion, and hiring. However, we believe that this framework can also be used to think about evaluation practices at the lesson and course levels, and within the HyFlex context.

Dalhousie’s framework is helpful in the way that it categorizes sources of evidence as students, peers, and self, which we will do for the rest of this unit.

In the next sections, we are going to provide examples of different sources of evidence (categorized as students, peers, and self) and we will also discuss how they can be used to measure the effectiveness of your HyFlex teaching within lessons and course levels.

Using students as a source for evidence

Instructors are typically most familiar with using students as sources teaching effectiveness through the use of course evaluations. Even though reflecting on course evaluations can be helpful, there are many other ways to collect feedback from students about their learning experiences.

An important consideration when collecting feedback from students through formative assessment techniques is the timing.

Only collecting end-of-term feedback is a missed opportunity to make immediate changes.

In their guide called “Making Sense of Student Feedback” (CC BY-NC 4.0), Berenson and Jeffs (2021) define these different types of feedback:

“Summative feedback is typically an institutional or department requirement at the end of a course, often used to assess and improve teaching practices for future course offerings” (p. 5)

“Formative feedback can be instructor initiated during a course with the intent to improve teaching, this could include a mid-term check-in, one minute paper, stop-start-continue, or other technique” (p. 5)

Collecting summative, end-of-term feedback

There is an ongoing debate about the value of end-of-term teaching evaluations due to biased concerns. Some instructors have solicited end-of-term feedback using other external surveying tools because they perceive that the feedback is sometimes different and more helpful than the institutional student evaluation questionnaire.

Either way, end-of-term feedback allows you to reflect on and make informed decisions about future iterations of your course. If you are teaching a HyFlex course for the first time, this feedback could be about course design, and appropriate suggestions can be implemented in the next delivery.

Even though there are documented issues with institutional student evaluation surveys, the feedback may be valuable, and you may wish to consult your centre for teaching and learning to see what support they provide regarding these surveys.

Collecting formative feedback throughout the course

Informal and formal feedback collected from students at different points in the course can help you reflect on your teaching and design approaches and make appropriate changes right away.

Another strategy to obtain student feedback is the Start-Stop-Continue method. Conestoga College’s Teaching and Learning website uses this image to describe the strategy:

The structure and intent of the feedback is also important. If students complete the reflection for themselves, they are considering their role in the class and how they can regulate their learning. If completing it for the class in general, sometimes students can make helpful suggestions for participants in different modalities.

One small change can make a big difference to the learning experience.

For example, a HyFlex instructor may learn early in the semester that the online synchronous students (or the asynchronous online students viewing a class recording) cannot identify who is speaking in the class when there is a discussion. Moving forward, the in-person students can identify their names before speaking to fix this clarity issue.

Here are some other considerations when generating student feedback:

Using peers as a source for evidence

Peers can be an invaluable source of feedback in teaching and design practices. We often discuss the role of peers as classroom observers, however, the Holistic Evaluation Framework from earlier describes how your peers can play an important role in your evaluation plan:

• Peer observations of teaching
• Peer review of teaching materials
• External peer review of teaching materials by an appropriate (disciplinary?) expert
• Other formal and informal peer feedback

In this section, we focus on the use of teaching observations as sources of evidence, which can involve the review of teaching materials, external reviews, and other formal and informal feedback.

Teaching Observation

Teaching observations are one source of evidence that can be used to evaluate the effectiveness of your HyFlex teaching. An important consideration about observations is identifying why you are collecting the information.

As we discussed earlier, evaluation of your teaching can serve different purposes. While some types of evidence can be primarily used at the lesson and/or course level, other evidence may be used for broader implications such as decisions related to awards, tenure, promotion etc.

For teaching observations specifically, they may be used in a formative way at any point during a course to obtain feedback on how to improve your teaching in your course. Conversely, summative observations may be used for administrative purposes.

While observations can improve teaching, they also identify existing effective practices (McMahan et al., 2007).

Fullerton (2003) suggests the following:

…teaching observations are beneficial for both the observer and the observed, and that they are an opportunity to: “validate what is working well, to exchange practice ideas, to solve problems and to explore and align practice with a developing understanding of theory, as well as to learn from the practice of others” (p. 227). 

Keep in mind that an observation captures a snapshot of the classroom at one point in time. Having your course observed multiple times can improve the reliability of these types of assessments, and also provide you with more meaningful feedback as you reflect and refine your teaching practices.

Who will be completing the teaching observation?

There has been some discussion of terminology regarding teaching observation, with some noting a difference between peer observation and third-party observation. We view peer observation as one form of observation under the umbrella of third-party observation. In addition to peer observation, third-party observation might include observation by educational developers or administrators.

This is important to note because observations can be completed by a variety of individuals – though we often use peer observation most frequently. For a further discussion of how to conceptualize third-party observation, see McMahon et al. (2007) and Gosling (2002).

For our purposes of evaluation – reflecting on and improving teaching practice – having a colleague from your program might be appropriate, an Educational Developer from your institution’s Teaching and Learning Center, or someone external to the institution.

What is the process for having a teaching observation completed?

In face-to-face classrooms, a teaching observation typically involves having an observer come to your classroom or review a video of your teaching while completing some sort of template about what they observed. When utilizing different types of delivery modes, as with HyFlex, teaching observations can become more complex.

One way of thinking about the process is through this four-stage framework:

How will the peer record their observations?

Regardless of who is doing the observation, it is important that they are given guidelines about what they should be observing. Without this guidance, various observers may be using different kinds of criteria to evaluate your teaching. For example, they may be more focused on the subject matter and content rather than your teaching strategies.

While a variety of templates for teaching observations can also be found online, at the time of writing, there are none specific to HyFlex.

In the interim, try creating a template centred around Beatty’s Four Fundamental Values for HyFlex: Learner Choice, Equivalency, Reusability, Accessibility, as discussed in Module 1, Unit 1, and Module 2, Unit 2. Each value could be given a 1-5 rating scale with additional space for comments.

Using self as a source for evidence

YOU are the final source for measuring the effectiveness of your HyFlex teaching. When discussing the Holistic Evaluation Framework mentioned earlier, self-reflection is most applicable to the lesson and course level.

Wilfrid Laurier’s Guide for Effective Remote Teaching, Learning, and Assessment (CC BY 4.0) suggests that self-reflection can build a strong awareness of your practice and its relative strengths and development opportunities.

Teaching is a constantly changing field; adopting a reflective approach allows you to connect your teaching practices to the changing landscape while addressing your own values, priorities and teaching philosophy.

Determine the focus of your self-reflection

Reflecting on your teaching can involve a lot of different things such as rapport with students, classroom management, clarity of instructions etc. A teaching journal could be used to capture your thoughts after a lesson, a module, and/or a course; what you record is up to you. However, it is important that there is a written record of these reflections so that you can return to them later. You can also record short audio clips on the fly.

In their discussion of Reflective Memos, the University of Nebraska-Lincoln offers great reflective questions you can use:

• From my perspective, how did the course go this semester? What aspects went really well? What could have gone better?
• What was the general atmosphere in the class? Did my students seem excited to be there? Did I feel enthusiasm for each class period? If not, what were the factors that prevented it from being a more enjoyable experience?
• Did my students meet my expectations with respect to attendance, attention, and participation in class? Can I do anything to assist or encourage my future students to do better in this regard?
• Did my students achieve the course learning goals? If so, how do I know? If not, what should I change to improve student outcomes? Could I do anything differently to communicate the learning goals to my students more effectively?

While these are general questions, try addressing them with specific HyFlex teaching needs and goals in mind. For instance, in Module 3, Unit 2, we recommended selecting the right tools and technology for HyFlex based on the following:

• Functionality
• Accessibility
• Technical
• Mobile design
• Privacy and data
• Social presence
• Teaching presence
• Cognitive presence

These topics can provide a clear purpose when reflecting on any technical challenges you have encountered. Let’s say you tried a new tool for the first time, and you noticed some accessibility issues. It may be useful to jot down those issues to determine whether there is a solution or if the tool should be replaced altogether.

Similarly, you might think about the Equivalency value of HyFlex and reflect on whether the learning activities for each learning modality led to an equivalent learning outcome. Maybe you thought they would when you designed the lesson, but it didn’t work out the way you thought it would. That’s okay! Take note of why it didn’t work and make changes.

Observations of your teaching and course design are important, especially if you are new to HyFlex. By adopting a growth mindset and welcoming constructive feedback from your students, your peers, and yourself, you can make necessary changes to create an engaging and effective HyFlex course.

Unit Summary

In this unit, you began by brainstorming current and future sources of evidence of teaching effectiveness. You considered why you use certain strategies in your practice so that moving forward, you can be intentional about selecting sources of evidence that are meaningful. You were also given a “smorgasbord” of strategies that could be used to determine the effectiveness of your HyFlex teaching at the lesson and/or course levels.

You also considered how the Holistic Evaluation of Teaching Framework from Dalhousie University could help you use evaluation sources such as students, peers, and self. Though the framework is broader in scope (beyond lesson and course level evaluation practices) we considered how to evaluate the effectiveness of your teaching on a smaller scale, within a lesson and/or course. You were also given some examples and strategies on how to use these sources in your HyFlex course.

In the next unit, you will plan how to pull different sources of evidence together to help determine your areas of strength and improvements for HyFlex teaching and design.