Unit 2: Possible Strategies for Evaluating Effectiveness

Photo by Eugen Str on Unsplash

Evaluating the effectiveness of HyFlex can be complex

In Unit 1 of this module, we referred to a broad definition of research that included asking questions, collecting data, and analyzing data. Data can broadly refer to various pieces of information or sources of evidence that are collected to increase knowledge about a topic or to answer the questions (Plano Clark & Creswell, 2015).

The goal of this unit is to introduce various sources of information to measure the effectiveness of your HyFlex teaching. You will be given a few sources of evidence you could use, and considerations for implementing these strategies for your HyFlex lesson or course.  That said, common strategies used to measure the effectiveness of face-to-face teaching are somewhat challenging when applied to online or HyFlex contexts.

In their book, Tobin et al. (2015, p. 15) outline some considerations when establishing and implementing evaluation methods for online environments:

  • What is the online equivalent of visiting a colleague’s classroom for ninety minutes?
  • In an online class, which elements count as “design” and which count as “teaching”?
  • How much time should peer and administrative evaluators spend observing online teaching?
  • Should observation of online teaching take place during or after the course?
  • To what purposes will student ratings, self-evaluation, peer evaluation, and administrative observation be used?

Similar (and additional!) challenges are present within the HyFlex environment. For example, let’s explore the question “What is the online equivalent to visiting a colleague’s classroom for ninety minutes?” by considering additional questions per the following modalities:

  • Online Asynchronous
    • Communication does not take place during a set time of day. Would an observation involve observing a Learning Management System for a week and looking at both the content of the course and other engagement tools like discussion boards, etc.?
    • How would indicators of effective teaching compare to those of other modalities?
  • Online Synchronous
    • Does an observation include one period of class time?
    • Are the indicators of effective teaching in this synchronous environment the same as those for face-to-face?

There are certainly many complexities when it comes to identifying sources of information or implementing strategies for use in HyFlex. Additionally, there is no prescriptive way of selecting the strategies you are going to use. As Berk (2005) points out:

“A virtual smorgasbord of data sources awaits you.” (p. 48)

Computer with lightbulbs Activity

What’s on your smorgasbord?

Step 1. Take a few minutes to brainstorm the following:

  1. Current Smorgasbord: What sources of information/evidence or strategies do you currently use to evaluate and/or demonstrate the effectiveness of your teaching?
  2. Potential Smorgasbord: What sources of information/evidence or strategies are you aware of that could be used to measure and/or demonstrate the effectiveness of teaching?

Think about why you are using the items you listed for number one (current sources of information). Is it about…

  • Familiarity?
  • Ease of implementation?
  • Comfort?
  • Program or institutional guidelines?
  • Something else?

You might already see areas for improvement in your evaluation practices based on this brainstorming activity. Is there a gap between how many strategies you are currently using and how many you are aware of?

Step 2. Review each of the lists you identified in Step 1 and circle or highlight the sources of information/evidence and strategies that could be used to measure the effectiveness of your HyFlex teaching at the lesson or course level.

Step 3. Review the items that you circled or highlighted in Step 2 and consider the following questions:

  1. Are there sources of information/evidence and strategies that could not be used in the HyFlex context? Why couldn’t they be used?
  2. For the sources of information/evidence and strategies that could be used with HyFlex, how might you adapt or customize them to this modality of teaching that includes face-to-face, online synchronous, and online asynchronous students at the same time?
  3. Which sources of information/evidence and strategies would you use to review one specific HyFlex lesson? Which ones could you use to look at the HyFlex course? Think about how they apply to the lesson or course level of evaluation, in comparison to those that are more applicable to who you are and what you do as an educator holistically.

Selecting evaluations strategies for HyFlex

Evaluation strategies should not be chosen at random; they should be intentionally selected based on a variety of factors such as the information they provide, source of data, and/or program or institutional guidelines.

Some instructors will use certain strategies that are different from others – that’s okay!

Below, you will see a “smorgasbord” of strategies that have emerged from the academic literature. As you read through these strategies, think about the following:

  1. Overall, which strategies apply more to the lesson or course level, in comparison to the instructor level holistically?
  2. Within each resource, which strategies or criteria are more applicable to the lesson or course level than the instructor level?
  3. For items that can be used for specific lessons or courses, how can they be applied to the HyFlex lesson or course specifically?

For an accessible-friendly version of the information above, download Sample of Strategies for Evaluation Teaching Effectiveness [.docx]

So, what did you determine after reviewing these literature-based strategies?

Though there is no right or wrong answer, one perspective is viewing Resource 1 (12 Strategies to Measure Teaching Effectiveness) and Resource 4 (Aspirational Model Teaching Criteria for Psychology) as more applicable to the instructor and their effectiveness as a whole, rather than the effectiveness of teaching in HyFlex within a lesson or course.

Resource 1 includes items such as employer ratings and teaching awards, and Resource 4 has items such as records of continuing education and local dissemination of knowledge about teaching. While you can certainly engage in all of these practices with a HyFlex viewpoint (e.g., disseminating knowledge about your experiences with HyFlex), they speak to who you are as an instructor and your effectiveness holistically.

Resource 2 (Administrative Courage to Evaluate the Complexities of Teaching) and Resource 3 (Statement on Student Evaluations of Teaching) include information that can speak to your effectiveness as an instructor, but more specifically, assess the effectiveness of your HyFlex teaching.

All items under Resource 2 (peer evaluation, student evaluation and input, and personal reflection) can be used at different points in a course and within the HyFlex context as well. Similar to Resource 2, all items in Resource 3 (peer observations, reviews of teaching materials, and self-reflection) can be used within a lesson or course, and also, within the HyFlex modality.

Even though there is a myriad of possibilities for what you could use to measure and demonstrate the effectiveness of your teaching, it is important to remember the purpose of your evaluation and how you are using the information.

Using a holistic approach to evaluation

There has been an increasing need for more holistic approaches when measuring the effectiveness of teaching, rather than relying solely on student evaluations. Dalhousie University is an example of one institution that has implemented a holistic evaluation of teaching policy, which was approved in 2021.

The aim of the policy is to ensure that evaluation of teaching takes multiple sources of teaching evidence into account (Dalhousie University Centre for Learning and Teaching, n.d.). The three sources of evidence of teaching effectiveness (student, peer, and self) and the relationship between them, are illustrated in the following diagram:


A Venn diagram with three overlapping circles. Circle 1 lists: self-reflection, professional development, classroom observations, teaching artifacts, and communities of practice. Circle 2 lists: teaching dossier, teaching philosophy, SLEQ formative, and SLEQ summative. Circle 3 lists: solicited feedback, unsolicited feedback, and scholarship of teaching and learning. Where Circle 1 and 2 overlap, the word “self” is written. Where Circle 1 and 3 overlap, the word “peers” is written. Where Circle 2 and 3 overlap, the word “students” is written. Around the outside of the Venn diagram, the phrases “Documenting teaching effectiveness”, “Assessing teaching effectiveness”, and “Improved processes for tenure reappointment, promotion, and hiring” are written.” Alt text description prepared by the Centre for Learning and Teaching at Dalhousie University.
Figure 2.2. Holistic evaluation of teaching. Image shared with permission from the Centre for Learning and Teaching at Dalhousie University.

This policy and the diagram are referring to the instructor level, and how you can measure and describe your effectiveness as an instructor for a variety of purposes, including decision-making such as tenure, reappointment, promotion, and hiring. However, we believe that this framework can also be used to think about evaluation practices at the lesson and course levels, and within the HyFlex context.

Dalhousie’s framework is helpful in the way that it categorizes sources of evidence as students, peers, and self, which we will do for the rest of this unit.

Computer with lightbulbs Activity

While the diagram above illustrates some examples of evidence sources for each of these categories, you can uncover additional sources by clicking on the buttons below. As you read through the strategies, consider the following:

  1. For each source (students, peers, and self), which strategies can be used at the lesson and course levels?
  2. For the items you identified in number one, consider how they could be done in your HyFlex class.


In the next sections, we are going to provide examples of different sources of evidence (categorized as students, peers, and self) and we will also discuss how they can be used to measure the effectiveness of your HyFlex teaching within lessons and course levels.

Using students as a source for evidence

Instructors are typically most familiar with using students as sources teaching effectiveness through the use of course evaluations. Even though reflecting on course evaluations can be helpful, there are many other ways to collect feedback from students about their learning experiences.

Screen with a pen and charts Example

For a great overview of how one instructor collects and uses student feedback in a course, listen to Episode 9: Using Student Feedback[12m53s] from the University of Alberta’s Teaching Plus podcast episode.

You may also view the transcript here. [.docx]

An important consideration when collecting feedback from students through formative assessment techniques is the timing.

Only collecting end-of-term feedback is a missed opportunity to make immediate changes.

In their guide called “Making Sense of Student Feedback” (CC BY-NC 4.0), Berenson and Jeffs (2021) define these different types of feedback:

Summative feedback is typically an institutional or department requirement at the end of a course, often used to assess and improve teaching practices for future course offerings” (p. 5)

Formative feedback can be instructor initiated during a course with the intent to improve teaching, this could include a mid-term check-in, one minute paper, stop-start-continue, or other technique” (p. 5)

Collecting summative, end-of-term feedback

There is an ongoing debate about the value of end-of-term teaching evaluations due to biased concerns. Some instructors have solicited end-of-term feedback using other external surveying tools because they perceive that the feedback is sometimes different and more helpful than the institutional student evaluation questionnaire.

Either way, end-of-term feedback allows you to reflect on and make informed decisions about future iterations of your course. If you are teaching a HyFlex course for the first time, this feedback could be about course design, and appropriate suggestions can be implemented in the next delivery.

Even though there are documented issues with institutional student evaluation surveys, the feedback may be valuable, and you may wish to consult your centre for teaching and learning to see what support they provide regarding these surveys.

Screen with a pen and charts Example

Check out the website for the Center for Teaching at Vanderbilt University. Content on their website is licensed under CC BY-NC 4.0 and therefore you can use any resources that are available, with attribution.

The website provides the following suggestions for end-of-term student evaluations:

  • Designate time in class for students to complete evaluations
  • Tell students that you value their honest and constructive feedback
  • Let your students know that you are interested in both positive and negative feedback on the course
  • Describe the kind of feedback you find most helpful
  • Remind students that the evaluations are anonymous
  • Let students know that you are the primary audience but that others will potentially read their evaluations
  • Consider including language in your syllabus that addresses student evaluations

Collecting formative feedback throughout the course

Informal and formal feedback collected from students at different points in the course can help you reflect on your teaching and design approaches and make appropriate changes right away.

Screen with a pen and charts Example

The University of Waterloo has a section on their Centre for Teaching Excellence website entitled Tools for Reflecting on Your Teaching (CC BY-NC 4.0).

To gather student feedback, they list and describe various tools, variations of the tool, and caveats related to the tool.

Another strategy to obtain student feedback is the Start-Stop-Continue method. Conestoga College’s Teaching and Learning website uses this image to describe the strategy:


An infographic with icons for start, stop and continue. Start: Offer a suggestion of something this person could start doing to better support your learning. Give an example of how it can help you learn. Stop: Offer a suggestion of something this person could stop doing, as it hinders your learning. Give an example of how it may disadvantage your learning. Continue: Offer positive feedback on something this person could continue doing, as it supports your learning. Give an example of how you have benefitted from it.
Figure 2.3. Image courtesy of Teaching and Learning Conestoga, CC BY-NC-SA 4.0.


A head with a pencil, ruler and highlighter Take Note

For HyFlex, it is important to determine how you are collecting feedback since you have to consider all three modalities – will you use a data collection tool like Google Forms to collect the information?

Use this as an opportunity to teach students about constructive feedback and self-reflection. You could model this through reflective practice: “What is something I could start to support students’ learning, what is something I could stop, what is something I could continue?”

The structure and intent of the feedback is also important. If students complete the reflection for themselves, they are considering their role in the class and how they can regulate their learning. If completing it for the class in general, sometimes students can make helpful suggestions for participants in different modalities.

One small change can make a big difference to the learning experience.

For example, a HyFlex instructor may learn early in the semester that the online synchronous students (or the asynchronous online students viewing a class recording) cannot identify who is speaking in the class when there is a discussion. Moving forward, the in-person students can identify their names before speaking to fix this clarity issue.

Here are some other considerations when generating student feedback:

Computer with lightbulbs Activity

In this activity, consider the tools that you could use to collect formative feedback.

Step 1. Find a list of existing formative feedback strategies. Here are some examples:

Step 2. Select three formative feedback strategies that you would use in your HyFlex course.

Step 3. Consider any modifications to these strategies for the HyFlex environment.

Step 4. Think about when you want to collect feedback throughout your course.

We will return to this when you plan your evaluation practices in Unit 3 – Compiling Multiple Sources of Evidence.

Using peers as a source for evidence

Peers can be an invaluable source of feedback in teaching and design practices. We often discuss the role of peers as classroom observers, however, the Holistic Evaluation Framework from earlier describes how your peers can play an important role in your evaluation plan:

  • Peer observations of teaching
  • Peer review of teaching materials
  • External peer review of teaching materials by an appropriate (disciplinary?) expert
  • Other formal and informal peer feedback

In this section, we focus on the use of teaching observations as sources of evidence, which can involve the review of teaching materials, external reviews, and other formal and informal feedback.

Teaching Observation

Teaching observations are one source of evidence that can be used to evaluate the effectiveness of your HyFlex teaching. An important consideration about observations is identifying why you are collecting the information.

As we discussed earlier, evaluation of your teaching can serve different purposes. While some types of evidence can be primarily used at the lesson and/or course level, other evidence may be used for broader implications such as decisions related to awards, tenure, promotion etc.

For teaching observations specifically, they may be used in a formative way at any point during a course to obtain feedback on how to improve your teaching in your course. Conversely, summative observations may be used for administrative purposes.

While observations can improve teaching, they also identify existing effective practices (McMahan et al., 2007).

Fullerton (2003) suggests the following:

…teaching observations are beneficial for both the observer and the observed, and that they are an opportunity to: “validate what is working well, to exchange practice ideas, to solve problems and to explore and align practice with a developing understanding of theory, as well as to learn from the practice of others” (p. 227). 

Keep in mind that an observation captures a snapshot of the classroom at one point in time. Having your course observed multiple times can improve the reliability of these types of assessments, and also provide you with more meaningful feedback as you reflect and refine your teaching practices.

Who will be completing the teaching observation?

There has been some discussion of terminology regarding teaching observation, with some noting a difference between peer observation and third-party observation. We view peer observation as one form of observation under the umbrella of third-party observation. In addition to peer observation, third-party observation might include observation by educational developers or administrators.

This is important to note because observations can be completed by a variety of individuals – though we often use peer observation most frequently. For a further discussion of how to conceptualize third-party observation, see McMahon et al. (2007) and Gosling (2002).

For our purposes of evaluation – reflecting on and improving teaching practice – having a colleague from your program might be appropriate, an Educational Developer from your institution’s Teaching and Learning Center, or someone external to the institution.

Computer with lightbulbs Activity

Who do you want as an observer in your HyFlex class?

Consider the following questions in relation to the colleagues in the program in which you teach, your institution’s Teaching and Learning Center, and/or external individuals:

  • Should the observer be familiar with the values and principles of HyFlex teaching?
  • Should the observer have experience teaching in the HyFlex modality?

If your response to both questions is yes, and there is limited familiarity with HyFlex at your institution, perhaps you will ask someone in your professional networks with more familiarity and experience to observe your class.

What is the process for having a teaching observation completed?

In face-to-face classrooms, a teaching observation typically involves having an observer come to your classroom or review a video of your teaching while completing some sort of template about what they observed. When utilizing different types of delivery modes, as with HyFlex, teaching observations can become more complex.

One way of thinking about the process is through this four-stage framework:

A diagram of four labeled boxes with arrows pointing from pre-observation to observation to post-observation to reflective summary.
Figure 2.5. Image courtesy of University of Texas at Austin Faculty Innovation Center, CC BY-NC-SA 4.0.

How will the peer record their observations?

Regardless of who is doing the observation, it is important that they are given guidelines about what they should be observing. Without this guidance, various observers may be using different kinds of criteria to evaluate your teaching. For example, they may be more focused on the subject matter and content rather than your teaching strategies.

Screen with a pen and charts Example

Read through Thompson Rivers University’s Centre for Excellence in Learning and Teaching website for their peer-review process. It may give you some ideas about creating your own process for HyFlex.

While a variety of templates for teaching observations can also be found online, at the time of writing, there are none specific to HyFlex.

In the interim, try creating a template centred around Beatty’s Four Fundamental Values for HyFlex: Learner Choice, Equivalency, Reusability, Accessibility, as discussed in Module 1, Unit 1, and Module 2, Unit 2. Each value could be given a 1-5 rating scale with additional space for comments.

Using self as a source for evidence

YOU are the final source for measuring the effectiveness of your HyFlex teaching. When discussing the Holistic Evaluation Framework mentioned earlier, self-reflection is most applicable to the lesson and course level.

Wilfrid Laurier’s Guide for Effective Remote Teaching, Learning, and Assessment (CC BY 4.0) suggests that self-reflection can build a strong awareness of your practice and its relative strengths and development opportunities.

Teaching is a constantly changing field; adopting a reflective approach allows you to connect your teaching practices to the changing landscape while addressing your own values, priorities and teaching philosophy.

Determine the focus of your self-reflection

Reflecting on your teaching can involve a lot of different things such as rapport with students, classroom management, clarity of instructions etc. A teaching journal could be used to capture your thoughts after a lesson, a module, and/or a course; what you record is up to you. However, it is important that there is a written record of these reflections so that you can return to them later. You can also record short audio clips on the fly.

In their discussion of Reflective Memos, the University of Nebraska-Lincoln offers great reflective questions you can use:

  • From my perspective, how did the course go this semester? What aspects went really well? What could have gone better?
  • What was the general atmosphere in the class? Did my students seem excited to be there? Did I feel enthusiasm for each class period? If not, what were the factors that prevented it from being a more enjoyable experience?
  • Did my students meet my expectations with respect to attendance, attention, and participation in class? Can I do anything to assist or encourage my future students to do better in this regard?
  • Did my students achieve the course learning goals? If so, how do I know? If not, what should I change to improve student outcomes? Could I do anything differently to communicate the learning goals to my students more effectively?

While these are general questions, try addressing them with specific HyFlex teaching needs and goals in mind. For instance, in Module 3, Unit 2, we recommended selecting the right tools and technology for HyFlex based on the following:

  • Functionality
  • Accessibility
  • Technical
  • Mobile design
  • Privacy and data
  • Social presence
  • Teaching presence
  • Cognitive presence

These topics can provide a clear purpose when reflecting on any technical challenges you have encountered. Let’s say you tried a new tool for the first time, and you noticed some accessibility issues. It may be useful to jot down those issues to determine whether there is a solution or if the tool should be replaced altogether.

Similarly, you might think about the Equivalency value of HyFlex and reflect on whether the learning activities for each learning modality led to an equivalent learning outcome. Maybe you thought they would when you designed the lesson, but it didn’t work out the way you thought it would. That’s okay! Take note of why it didn’t work and make changes.

Observations of your teaching and course design are important, especially if you are new to HyFlex. By adopting a growth mindset and welcoming constructive feedback from your students, your peers, and yourself, you can make necessary changes to create an engaging and effective HyFlex course.

Unit Summary

In this unit, you began by brainstorming current and future sources of evidence of teaching effectiveness. You considered why you use certain strategies in your practice so that moving forward, you can be intentional about selecting sources of evidence that are meaningful. You were also given a “smorgasbord” of strategies that could be used to determine the effectiveness of your HyFlex teaching at the lesson and/or course levels.

You also considered how the Holistic Evaluation of Teaching Framework from Dalhousie University could help you use evaluation sources such as students, peers, and self. Though the framework is broader in scope (beyond lesson and course level evaluation practices) we considered how to evaluate the effectiveness of your teaching on a smaller scale, within a lesson and/or course. You were also given some examples and strategies on how to use these sources in your HyFlex course.

In the next unit, you will plan how to pull different sources of evidence together to help determine your areas of strength and improvements for HyFlex teaching and design.




Icon for the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

HyFlex Course Design and Teaching Strategies Copyright © 2022 by Angela Barclay; Krista Ceccolini; Kathleen Clarke; Nicole Domonchuk; Sidney Shapiro; Jupsimar Singh; Mel Young; Jenni Hayman; Joseph Beer; and Courtney Arseneau is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted.

Share This Book