Outputs prioritized over unclear desired outcomes

Institutional leadership and faculty members reported the perception that the focus for the HOQAS review process was placed on the completion of reviews as opposed to the outcomes of the review process, with the ambitious goal of approximately 20% of courses (translating to 300 courses) having been set for the number of courses to be reviewed each year. One of the impacts of this goal, as reported in the findings on scores, feedback, and communication, was that the process ended up being oftentimes an impersonal and negative experience for faculty members; the process focused on the production of reviews and reports, without making obvious to stakeholders what the purpose or outcome of the process was supposed to be. As institutional leadership noted, the increasing number of hybrid and online courses being offered at Algonquin College, and the lack of existing standards with which to design and assess them, provided urgency to find standards against which the courses could be assessed; Quality Matters was chosen because it was well-known for online learning and it was felt that an externally validated instrument would have better institutional buy-in. As some faculty noted, however, it remained unclear what specific outcomes were trying to be achieved and what measures of outcomes or evidence would be used to support whether or not they were being achieved; as such, the review process ended up feeling like an administrative or bureaucratic burden, more so than a tool to improve the quality of their courses.

Student course satisfaction and student course achievement were looked at in relation to the HOQAS standards to see if standards compliance as assessed by the review score was reflected in either of these measures. HOQAS review reports were matched with their corresponding student course feedback survey results using the SCF-HOQAS concordance table that was developed. Only inferential statistics were tested as the pausing of the HOQAS review program made it impossible to construct a robust pre-revision/post-revision experiment. Several statistically significant correlations, as measured by Spearman’s rho, were detected between items on the SCF survey (both the 2015/2016 and 2016/2017 versions) and the HOQAS standards. Both positive and negative correlations were found: in some cases, the HOQAS review score increased with the SCF survey score; and in some cases, the HOQAS review score decreased with higher SCF survey scores (and vice versa). In general, the strength of the correlations found were weak. For the 2016/2017 SCF survey specifically, moderate strength negative correlations were detected: higher HOQAS standards scores were correlated with lower ratings on the 2016/2017 SCF items and vice versa. Course mean grades had weak, statistically-significant correlations with both individual HOQAS standards from the review reports as well as standards section scores. For individual standards, four correlations were negative and two were positive; for the standards section, both correlations were negative.