Evaluating OER
Why evaluate an OER?
Just as with commercial textbooks, not all OER are the same quality and one single OER may not meet all the learning objectives set for a course. It is therefore important to spend time evaluating OER to see if they meet our needs and those of our students and to adjust as necessary.
What do we mean by “quality”?
OER are not necessarily created using the same editorial process used for conventional educational textbooks produced by publishers. This difference can negatively affect the reputation of OER, leading some people to perceive them as being lower-quality resources. This misconception persists despite many OER undergoing peer review processes before or after their publication. For example, OpenStax, an open textbook initiative from Rice University, closely mirrors the processes used by commercial publishers. Through its Textbook Success Program and other resources, Rebus Community supports OER projects from start to finish, including peer review. OER already published can also be evaluated by subject-matter experts who are often faculty or instructors who may use the materials in their own courses. eCampus Ontario’s Open Library, BCcampus’ Open Textbook Collection, and the Open Textbook Library all rely on qualified experts to evaluate the OER in their collections.
OER creators rarely have a graphic design team or a budget at their disposal so their resources may not have the same visual appeal as commercial textbooks. Because people still tend to judge books by their covers, OER can be disregarded. However, an OER’s cover and overall appearance have nothing to do with the accuracy and quality of its content. In 2013, David Wiley called into question publishers’ way of judging resource quality and proposed that, regardless of the commercial or free nature of the resource, the only important question to ask is: Does the material help students meet the course’s learning objectives?[1] That is why we chose not to include pictures of the OER suggested in this guide and instead emphasize the importance of evaluating these resources.
How to evaluate an OER?
There are many elements to keep in mind while evaluating an OER. BCOER and Affordable Learning Georgia provide the following evaluation rubrics:
- BCOER, Faculty Guide for Evaluating Open Educational Resources [PDF]
- Affordable Learning Georgia, OER Evaluation Criteria [PDF]
Generally, an exhaustive OER evaluation considers the following criteria[2]:
- Scope: The resource appropriately covers material and ideas related to the subject.
- Accuracy: The content is accurate, objective, and free from errors. If it has been translated, the quality of the translation reflects the original content.
- Relevancy: The content is current or written/organized in a manner that facilitates making the necessary updates. The content is suitable for the desired education level.
- Authority: The resource was created or adapted by a subject-matter expert. Note that cocreating resources with students is an increasingly common practice in open education. Their participation in creation should not automatically signal a lower quality resource. In these cases, the context in which the resource was created, such as if it was created within the course and/or under faculty supervision, can determine the resource’s authority.
- Accessibility and ease of access: The resource includes measures that support accessibility such as subtitles for videos or transcripts of podcasts, multi-format availability, and a similar experience across any method used to access the content.
- Modularity: The resource can easily be divided into sections that can be assigned at different times throughout the course without causing confusion for the learner.
- Cultural relevance and inclusion: The content includes examples that represent diversity in gender, race, culture, and experience. The content is appropriate in the local context in which it will be used or can be easily adapted to be applicable.
- Additional resources: The resource comes with supplementary materials (videos, interactive activities, question bank, etc.).
- Licensing: The resource is available through open licence terms that allow it to be reused, modified, and distributed without the usual copyright restrictions.
Evaluation of OER suggested in this guide
Librarians conducted a preliminary evaluation of the OER suggested in this guide to ensure they are relevant to specific courses based on the available descriptions in the University of Ottawa’s course catalogue. Evaluation is indicated by an asterisk (*) after the resource’s title.
The criteria below were considered for this preliminary evaluation:
- Relevancy based on a comparison of the course description with the resource’s table of contents
- Correspondence between the course and resource learning levels
- Functionality of the resource (ease of navigation, modularity)
- If the resource has been evaluated by experts and/or users, do these evaluations indicate significant problems with the resource?
- Other details that highlight the relevance of the resource (geographic or cultural context)
- David Wiley, “On Quality and OER,” Improving Learning, 10 October 2013, https://opencontent.org/blog/archives/2947, CC BY 4.0. ↵
- This list of criteria is a combination of the following sources: BCOER, Faculty Guide for Evaluating Open Educational Resources, April 2015, https://open.bccampus.ca/files/2014/07/Faculty-Guide-22-Apr-15.pdf, CC BY 4.0 and Open Textbook Library, Open Textbooks Review Criteria, n.d., https://open.umn.edu/opentextbooks/reviews/rubric, CC BY 3.0. ↵
