Effective Assessment Design

Backwards design (Wiggins and McTighe, 1998) has been a solid resource to outline the process used when designing assessment of learning and assessment for learning.  Start with the learning outcomes you are trying to achieve with your course.  Learning outcomes are assessable statements that indicate what students should know, value, or be able to do by the end of a course, unit, or program. Learning outcomes must be observable and measurable (Goff et al., 2015; Potter & Kustra, 2012).  From there, you should be able to get a sense of what achieving that learning outcome would look like and how to build an assessment to measure that outcome.  In between, is the journey from the outcome to the assessment which takes the form of classroom activities, lectures and resources that should build up the learning necessary to achieve the outcome that is being assessed.

Formative assessments are used to help build and reinforce concepts to partner with instruction.  In this case, they are used as assessments for learning.  Short quizzes, reflective questions, exercises, flashcards, exit questions, detail the muddiest point, etc.,  are all examples of what instructors can use for formative assessments.  Formative assessments not only help the student – they can also assist the instructor to know what concepts are readily comprehended and what is clearly misunderstood.  There are many resources available on formative assessments that you can seek out to learn more about.

Summative assessments are those that culminate and are to provide a report on the degree of knowledge or proficiency achieved at the end of a module, course, unit, etc. (McTighe, J., & Ferrara, S., 2021).

According to McTighe and Ferrara there are five principles to inform an effective assessment:

• Assessments should serve learning.
• Multiple measures provide more evidence.
• Assessments should align with goals.
• Assessments should measure what matters.
• Assessments should be fair.

McTighe and Ferrara argue to help achieve these principles, students should know in advance what is expected to achieve the goals, models of the work expected should be provided, assessments are realistic, challenging but attainable.  Students should be able to show evidence of their learning and the more artefacts of evidence of their learning over time will help to provide a realistic view of their actual learning of the material.  If there are limited chances to demonstrate learning and students are only provided with high-stakes assessment events, then the end result is limited for both the instructor and the student to be actually meaningful.  A single snapshot does not tell a whole story – only a moment in time.

Alignment to the learning outcome being measured is also a critical component in the assessment design.  A great way to check this out is to ask a trusted colleague to review your proposed assessment(s) and see if they can guess what outcome(s) you were trying to assess (McTighe, J., & Ferrara, S., 2021).

BetterExaminations is a tool that can help achieve both formative and summative assessments.  One way to help gauge learning is to have students articulate how they came up with an answer in addition to actually providing that answer.  A way to do that in BetterExaminations is to include an audio recorded or video recorded response from the student after a specific question and answer set.  According to Weleschuk, Dyjur and Kelly, 2019, interactive and higher-order learning opportunities can increase engagement with assessments.  Further, case studies can present a series of facts, images, videos, audio or details about a scenario that can be used to build a series of questions off of the original case that can be a more engaging and authentic way to traverse course materials in the assessment.

References

Goff, Lori; Potter, Michael K.; Pierre, Eleanor; Carey, Thomas; Gullage, Amy; Kustra, Erika; Lee, Rebecca; Lopes, Valerie; Marshall, Leslie; Martin, Lynn; Raffoul, Jessica; Siddiqui, Abeer; and Van Gaste, Greg, “Learning Outcomes Assessment A Practitioner’s Handbook” (2015). Centre for Teaching and Learning Reports. https://scholar.uwindsor.ca/ctlreports/

McTighe, J., & Ferrara, S. (2021). Assessing Student Learning by Design: Principles and Practices for Teachers and School Leaders. Teachers College Press.

Weleschuk, A., Dyjur, P., & Kelly, P. (2019). Online Assessment in Higher Education. Taylor Institute for Teaching and Learning Guide Series. Calgary, AB: Taylor Institute for Teaching and Learning at the University of Calgary. Retrieved from https://taylorinstitute.ucalgary.ca/resources/guides

Wiggins, Grant, and McTighe, Jay. (1998). Backward Design. In Understanding by Design (pp. 13-34). ASCD.

Thinking through exam design prior to building an online exam

Below are some questions to think through before starting to build an exam in BetterExaminations so you can use your time as efficiently as possible using the tool to its maximum capacity.

Preparing the Student for the Exam and the Exam for the Student

• Are there images/diagrams that are necessary for students to fill in answers on the image e.g. chemistry molecules, graphs, charts, diagrams?  Are there any copyright implications for using this image? 
• Does the student require additional tools e.g. calculator (simple or scientific), protractor/ruler? 
• Will you allow the student additional tools to use during the exam e.g. highlighter, notepad, drawing tool, sticky note? Why?  What would happen if you enabled all of the possible tools or none of the tools?  Would that change the results the students could generate and provide an advantage or disadvantage regarding their use?
• Do you want questions to appear in a specific order or do you want them randomized in their appearance? 
• Do you know the initial values you might need to establish for some questions in order for students to process using the software tool to get the correct results (e.g. bar charts, line graphs)
• What settings/values should be enabled or avoided to prevent unnecessary errors or invalid results (e.g. avoid overlaid duplicate shapes)? 

Beginning the Exam

• Do you have any introductory elements to the exam for the student?  Does it require the student to completing anything e.g. provide an identification photo, video of their voice, text passage?  Do you have to introduce anything relevant to the exam (e.g. instructions, video, audio)? 
• Will there be any media needed for question(s)?  If so, what type of media? (for the instructor to share with the student, or for the student to share with an instructor?).  How will the student generate the media (e.g. image, video, audio) and do they require any additional equipment to do so? How would you ensure that equipment would be in place without disadvantaging the student? 

During the Exam

• Will you provide an option for students to re-enter the exam if they get disconnected?  If not, what possible implications would that have on the student? 
• Students will be able to flag a question and return to it if they need to rethink it before submission.  Does that impact any of your designs? 

Written Responses

• Will there be a maximum word limit on your student’s response?  Why would that be an important design factor?  Are there pedagogical considerations with that choice if a student exceeded the limit or drastically underproduced content?  How would you manage that type of grading?

Case Studies

• Do you have a case study where there may be multiple questions following an example, video, or answer choice (choose your own adventure)?  Are any of these questions or answers dependent upon an earlier choice/answer in the scenario?

Scoring Answers

• Does the scoring of the current question rely on other factors, e.g. the answer generated in a previous question?  What pedagogical implications would occur if a student got their answer partially correct?  Completely incorrect?  

• Do you want to use “ranges” for a correct answer e.g. correct, partially correct, incorrect, or only a correct/incorrect response?  Why?  What would be the pedagogical considerations/implications for that choice? 

• When using autoscoring for math questions, scoring methods are available into two categories –Evaluation Types and Form Checking 

• • Evaluation Types – equivSymbolic (symbolic equivalence), equiLiteral (literal equivalence), equiValue (value equivalence)
  • Form Checking – isSimplified (simplified), isFactorised (factorization), isExpanded (expansion), isTrue (Boolean evaluation), isUnit (unit comparison), stringMatch (string comparison), equivSyntax (syntax comparison).

• Are you familiar with what you want to check for regarding the student’s results against these terms?

After the Exam

• Do you require the ability for another instructor to use questions from your course? 

Once you have reviewed these questions and have answers in mind, you are ready to begin building your test!