Evaluation

Mary Wilson; Kyle Mackie

Evaluation

Teaching in a college or university setting is busy, demanding work so it can be challenging to carve out time to review a course systematically and deeply; however, as educators committed to continuous professional growth and development, it is well worth investing the effort and time associated with undertaking a course evaluation process. Course evaluation is beneficial to your teaching practice, ensuring the coherence of the course and its role in the broader curriculum, building the quality of the student learning experience, and improving the experience of your industry/community partner(s). A planned approach to course evaluation allows you to frame questions in advance, collect meaningful data throughout the term that you can use to identify aspects that have worked really well and met expectations, as well as components of the course that would benefit from refinement or greater development. And finally, having an evaluation plan in place before the course starts can help you to define what sorts of documentation will be suited to your evaluation purposes and prove useful in the future so that you can start to organize your documentation that way from the outset.

Given the demands placed on our time as educators during the term, ideally, a plan for course evaluation will be developed during the analysis, design and development phases of the ADDIE process. Planning the course review process in advance will give the instructor the benefit of forethought about what aspects of the course can and should be reviewed for the purposes of making ongoing improvements to the course.

Let’s begin by reviewing a few basic principles of course evaluation as they apply to experiential education courses. For any type of course delivery, including experiential learning courses, course evaluation can play a formative role, a summative role, or both. It can be used to improve a course, and it can be used to determine the fate of a course. Course evaluation can inform academic governance decisions about credit weight allocation, the appropriate course pre-requisites and level of study, the costs associated with delivery, appropriate section sizes and staffing models, and can also serve as an effective means of gaining information that can be used to promote the course.

Experiential education courses can largely be reviewed and improved using the same principles of quality assurance and quality improvement that apply to other forms of instruction such as lecture/lab based courses or blended/online courses, but there are, of course, some unique aspects of experiential education courses to take into account when undertaking evaluation and improvement efforts. Throughout this course, we have explored the aspects of experiential education that set it apart from other forms of education – its particular purposes, strengths and requirements – that we can now consider as we discuss methods for evaluating aspects of course design and delivery. While conceiving of your evaluation questions, paradigms, models, tools, data sets and approaches to documenting and sharing your evaluation findings and recommendations, do consider the ways in which experiential education influences roles, responsibilities and risks, changes the nature of interactions and communications, complicates formative and summative assessments, and shapes engagement with course-related resources and materials. Make sure that the approaches that you choose to employ take into account the distinctive features of experiential education.

The six-step evaluation process discussed by Stirling et. al. (see: Stirling, A., Kerr, G., Banwell, J., MacPherson, E., & Heron, A. (2016). A Practical Guide to Work-Integrated Learning. Retrieved September 14, 2017, from http://www.heqco.ca/SiteCollectionDocuments/HEQCO_WIL_Guide_ENG_ACC.pdf) provides a useful guideline for how to proceed with developing and executing a plan for evaluating your experiential education course. Stirling et. al. recommend these steps:

Develop an evaluation question
Choose an evaluation paradigm
Select an evaluation model
Develop evaluation tools
Collect and analyze data
Document and present findings

For the first step, developing evaluation questions, you may find that drawing upon the order of thinking promoted by backward design can prove to be a helpful mindset to adopt. Given that the main purpose of any course is always to cultivate student learning, it makes sense to privilege experiential education course evaluation questions that will help you to get at how well the course functioned to help students achieve intended learning outcomes. When planning evaluation questions, consider first what learning outcomes students are meant to achieve through the experiential education course and what level of competence or sophistication was desired.

While developing your evaluation questions, you may want to consider formulating questions associated with key components of the experiential learning course, such as its learning outcomes, assessments, learning activities and resources such as:

To what extent did students in this course achieve the intended learning outcomes?
How effective were the formative and summative assessments in surfacing evidence of student learning? Do you feel confident that students have reflected productively on their learning, drawn connections between theory and practice, and made progress in consolidating their learning through experience?
How effective were the activities, interactions, course documents, scholarly resources and communications that you contributed throughout the experiential learning course? Were they productive and timely in nature? Were you accurate in anticipating the needs of the students and the industry/community partners? Were the sites of student learning and the nature of the student learning experiences safe and appropriate as well as aligned to the intended purpose of the course? Were the costs associated with running this experiential learning opportunity reasonable and manageable for all involved?

The next step in the evaluation process is to choose among evaluation paradigms. Stirling et. al. outline four main paradigms: postpositive, constructivist, transformative and pragmatic. Your choice of paradigm is likely to be influenced your personal teaching philosophy as it applies to experiential education, your beliefs about how course design, teaching and learning can best be evaluated in experiential education and by the questions you have about your own experiential education course.

In the next and final session of the course, you will compose and submit a polished teaching philosophy statement for experiential education. But as you examine the evaluation paradigms outlined in the assigned chapter for this week, it is useful to begin to think concretely about exactly what philosophy of teaching you have developed in relation to experiential education so that you can reflect more critically upon the four paradigms presented in the chapter on evaluation by Stirling et. al.

In endeavouring to define your teaching philosophy for experiential education, reflect upon your perspectives and beliefs about experiential education. Characterize your goals for student learning through direct experience and capture your deliberate and focused efforts to ensure that your students achieve those goals through the ways in which you design, deliver, and assess an experiential education course. Consider how your approaches to experiential education align with your beliefs, goals and ethical values as an educator. Then turn to the paradigms described in the chapter and consider whether post-positivist, pragmatic, constructivist, or transformative paradigms of evaluation would work best. Once you have determined an evaluation paradigm, it will be easy to select among the evaluation models outlined or to adopt a model that is not outlined in the chapter, but is one that you are familiar with and find to be associated with a chosen paradigm.

Stirling et. al. provide a detailed description of the post-positivist Kirkpatrick evaluation model (Kirkpatrick & Kirkpatrick, 2006), the pragmatic RE-AIM framework (Glasgow, Vogt and Boles, 1999) and the constructivist CIPP model (Stuffelbeam, 2002). Review the models as described in the chapter assigned and critically compare them in terms of their methodology and assumptions, and consider their relative utility for your purposes. Your review should prepare you to select appropriate tools.

The fourth step in the model is the development of evaluation tools. When you consider the questions that you wish to address in your evaluation of your experiential education course, what kind of data will serve best to answer those questions? What are the best sources of that information – documents, material objects, or people? Knowing the answers to these questions will help you to determine what evaluation tools, whether a teaching journal, surveys, focus groups, document reviews or some other tools, will be most effective. You may wish to gather mid-semester feedback from your students about how they feel that they are progressing toward achievement of learning outcomes, or you may wish to consult with trusted peers to gain input on the clarity and effectiveness of a communication or activity that you have designed. Who you ask, and what you ask for, will depend on what questions you want to answer about your course.

The fifth step in the model focuses on making decisions about data collection and analysis for the evaluation of your experiential education course. How much data will you require in order for you to use it meaningfully? Who is best situated to provide or collect the data objectively and ethically – is it you, your students, your colleagues or administrators or the industry/community partner(s), or some combination? Your students, the industry/community partners and the professional staff of your experiential education office will undoubtedly have rich insights to share about how the learning opportunity could be improved, but to be respectful of their expertise and time, it will be valuable for you to think about what questions you wish to ask in advance. You will also need to identify what ethical considerations should you address for data collection, storage and use? What techniques for data analysis will function best and do you have the skills required to conduct the analysis, or should you seek instruction or resources to assist you? Knowing the answers to questions of this nature in advance will help you to plan for a course evaluation model that will be tailored to your purposes, resources, and capabilities.

Finally, the last step in the evaluation process centres on the use of the findings that emerge from the evaluation process. Will your evaluation serve formative or summative purposes or both? Who is your audience for your findings? Will you need to share your evaluation findings with your academic leadership, colleagues, students or industry/community partners, or is the evaluation for your edification as an instructor alone? Will you wish to include your evaluation documentation in your teaching portfolio? Questions of purpose, scope, and audience for the documentation and reporting of your evaluation can help you to make decisions early on about how you will plan for and structure your evaluation process throughout the term and following the conclusion of the course. Effective course evaluation will lead to informed decision-making.