Determining the success of HyFlex

At this point, you have learned how to approach and plan for a HyFlex course, but how do you know if your efforts are successful? In this unit, you will explore the background and meaning behind the terms “evaluation” and “effectiveness” in the context of HyFlex teaching and learning.

Caution: there is limited information about the effectiveness of HyFlex.

In their chapter titled “Evaluating the Impact of Hybrid-Flexible Courses and Programs”, Beatty (2019) notes that because the HyFlex approach to teaching and learning is still relatively new, much of the literature has been descriptive case studies focused on one instructor and their classroom at one institution. Examples of studies looking at entire programs, campus-wide implementation, or multi-institutional implementation, are scarce.

Beatty highlights that “Though fewer studies report the impact on student learning and associated metrics of interest (retention, passing grade rate, GPA, time to graduation, etc.), some have been published already, and more are expected within the coming years” (para. 1). This particular chapter in the book focuses on thirteen studies that Beatty says go beyond case study and literature review approaches.

The use of only thirteen studies in this book highlights how limited the research is on the impact of HyFlex. Given the lack of empirical literature on the effectiveness of HyFlex teaching, it is not feasible to provide a step-by-step, evidence-based guide rooted in HyFlex research.

For this reason, throughout this module, you will be presented with a variety of concepts related to teaching effectiveness, tools to measure teaching effectiveness, and frameworks to guide your evaluation practice that are from the broader teaching and learning literature. Each of these topics will be discussed and applied to the HyFlex environment so that you may have a starting point for developing your own evaluation plan.

Let’s begin with you…

Some of you may be considered experienced and have considerable background knowledge about evaluation, thinking about effectiveness, and/or considering and measuring the effectiveness of your teaching in face-to-face and/or online modalities. Others may be in the preliminary stages of determining what all of this means.

When this module was developed in 2022 during the Covid-19 pandemic, very little was known about the effectiveness of HyFlex teaching and how it could be measured. This means that in the immediate years after this book is published, learners who take this course will likely have limited experience measuring the effectiveness of HyFlex teaching, specifically because the modality is still currently emerging and growing in popularity.

Long-term, HyFlex may become a more popular teaching modality; more evidence of its effectiveness and how to measure it will likely surface. Regardless of when you are reading this book and this specific module, it is important that you take a few moments to identify and reflect on your prior knowledge and consider what you want to know about evaluating the effectiveness of HyFlex teaching.

Conceptualizing Evaluation

Evaluation can be broadly described as a process that involves the collection and analysis of various forms of information with the purpose of making improvements and decisions.

Evaluation processes are used in a variety of fields outside of education, and there is even a program evaluation profession that focuses on the evaluation of social programs (e.g., human service programs like health, education, employment, housing, poverty) (Rossi et al., 2019). At times, various stakeholders need to determine the effectiveness of various programs, launch new ones, or improve existing ones so that they may achieve better outcomes (Rossi et al., 2019).

One way to think about evaluation is how it connects to research processes:

“There is a place at which research and evaluation intersect – when research provides information about the need for, improvement of, or effects of programs or policies.” (Mertens, 2009, p. 2)

Generally, researchers pose a question, collect data about the question, and analyze the data to answer the question (Plano Clark & Creswell, 2015) – similar to evaluation.

This is not to say that the terms “evaluation” and “research” should be used synonymously.

Rather, considering the research cycle can be helpful in also understanding the role of evaluation in measuring effectiveness.

Another important consideration to understanding the concept of evaluation is the connection between evaluation and assessment within the teaching context. Hoey et al. (2021) argue that these terms are often used interchangeably and that although there are similarities between them, they are also distinct. Banta and Palomba (2014) define assessment as:

“The process of providing credible evidence of resources, implementation, action and outcomes undertaking the purpose of improving the effectiveness of instruction, programs, and services in higher education.” (p. 2)

While the primary focus of assessment is on collecting data about student learning (or program effectiveness, depending on the goal), evaluation may be distinct in the way that it involves analyzing and interpreting the assessment data, and then making judgments or decisions based on that information (Hoey et al., 2021).

The point here is that when it comes to identifying a definition of evaluation, it can become complex and there are related terms to consider, such as research and assessment. Researchers, instructors, and even institutions may use terms in a variety of ways.

For the purpose of this module, we use the term evaluation to refer to the collection and analysis of different forms of information that are used to measure the effectiveness of teaching. These measurements are then used to make decisions and improvements.

Scope and Purpose of Evaluation in Higher Education

Within the postsecondary context, evaluation can take place at a variety of levels: particular lessons, a course, a program, or an institution.

Evaluation of effectiveness might involve using institutional quality assurance frameworks, accreditation guidelines for professional programs (e.g., Ontario College of Teachers), or provincial quality assurance frameworks (See Ontario Universities Council on Quality Assurance, 2021).

At a course level, Kember and Ginns (2012) suggest the purpose of evaluation should be on:

“Improving the quality of teaching, providing a better learning and teaching environment, and improving the chances that desired learning outcomes will be attained.” (p. 1)

This is done by collecting different types of information about teaching and learning and then interpreting and making improvements based on that information.

The quotation above highlights three areas of focus and each of them raises some additional questions:

These three points frame our subsequent discussion about effectiveness, particularly in relation to attempting to conceptualize effectiveness and then in considering what an effective teaching and learning environment in HyFlex would look like.

Conceptualizing Effectiveness

When talking about effectiveness at the course level, we often focus on how to measure students’ learning.

At first glance, this might seem straightforward – but student learning is difficult to measure – and there has been ongoing and significant debate about learning outcomes and outcomes-based learning in general (See Bagnall 1994; Evans et al., 2018).

With the use of learning outcomes comes more of a focus on the student – a learning-centred focus – rather than a focus on the instructor and teaching objectives. Tam (2014) describes this distinction in the following way:

The traditional way of curriculum design, the teacher-centered approach focuses on the teacher’s input and on assessment in terms of how well the students absorb the materials taught. A departure from this traditional paradigm is the student-centered approach where the emphasis is on what the students are expected to be able to do at the end of the learning experience…Implicit in the student-centered model is the idea that teachers are facilitators of learning, who create and sustain an effective learning environment and experience based on a wide range of best practices in teaching and learning. (p. 161)

What this means is that the instructor needs to consider evidence-based, promising teaching practices and how they can apply them in their own practice. However, identifying what is considered an effective, best practice can vary.

In their book titled “Evaluating Online Teaching: Implementing Best Practices”, Tobin et al. (2015) similarly share examples of frameworks of best practices for the online teaching context. After doing so, however, they note the following:

Even when researchers are purposely looking to create lists of online-only teaching behaviours, such as the frequency of instructor log-ins to the course environment, the list (including our own) inevitably includes general best practices that could apply in various teaching modalities. This is cause for hope as well as reason to continue to examine what is unique about teaching online. (p. 10)

Unit Summary

In this unit, you began by describing what you currently know and want to know about evaluation, teaching effectiveness, and evaluating the effectiveness of HyFlex teaching specifically.

You then reviewed what is meant by the terms evaluation and effectiveness, to provide a foundation from which we can develop a greater understanding of how to evaluate the effectiveness of HyFlex teaching.

Lastly, you reviewed what has been discussed throughout this book that could be utilized as best practices for HyFlex teaching. These practices and the supplementary frameworks focus on online and face-to-face teaching in general and provide you with a sample of possible indicators that can be used in the evaluation of the effectiveness of your HyFlex teaching.

In the next unit, you will reflect on the strategies you are currently using in your teaching (regardless of the modality) to measure the effectiveness of your teaching. You will investigate possible strategies for measuring the effectiveness of teaching and distinguish their applicability to the HyFlex context.