HOQAS review reports
HOQAS review process tracking spreadsheet versus parsed reports
While the HOQAS review process tracking spreadsheet provided more final report scores (n = 187) than the parsed reports (n = 140, plus 4 audits of revisions), only the parsed reports contained the scoring for each standard. Thus, for any in-depth analysis of the HOQAS review standards, the parsed reports would need to be used. The two sets have similar but not identical characteristics, as compared below.
| HOQAS spreadsheet | HOQAS review reports | |
|---|---|---|
| Count | 187 | 140 |
| Maximum | 99 | 88 |
| Minimum | 21 | 28 |
| Mean | 57.2 | 56.4 |
| SD | 15.9 | 14.2 |
| Median | 55 | 54 |
To ensure the validity of using this subset for analysis, both the means and distributions of the two groups were compared. The difference between the means of the two groups was tested with an independent samples t-test and was not found to be statistically significant, t(325) = .431, p = .667; similarly, no statistically significant difference was found in the distributions of scores of the two groups using an independent samples Mann-Whitney U test, U = 12 803, z = -.339, p = .734. We can thus conclude that the subset could be suitable our analysis.
HOQAS review reports compliance rates
Of the 140 reviews included for analysis, 4 courses had overall review scores greater than or equal to the 85 points required to be considered standards compliant. The mean score was 56.4 (SD = 14.2) and the median score was 54. In terms of meeting the essential standards requirement, however, no courses received full marks in all 21 3-point sub-standards. Per-section compliance rate is tabulated below. No course fully met all essential standards for sections 2, 3, and 5, and only one course met all essential standards for section 8.
Essential standards compliance by HOQAS section
Non-compliant
| Section | % | Count |
|---|---|---|
| 1. Course overview and introduction | 69.3% | 97 |
| 2. Learning outcomes | 100.0% | 140 |
| 3. Assessment and measurement | 100.0% | 140 |
| 4. Instructional materials | 70.0% | 98 |
| 5. Learning interaction and engagement | 100.0% | 140 |
| 6. Course technology | 19.3% | 27 |
| 7. Learner support | 62.1% | 87 |
| 8. Accessibility | 99.3% | 139 |
Compliant
| Section | % | Count |
|---|---|---|
| 1. Course overview and introduction | 30.7% | 43 |
| 2. Learning outcomes | 0% | 0 |
| 3. Assessment and measurement | 0% | 0 |
| 4. Instructional materials | 30.0% | 42 |
| 5. Learning interaction and engagement | 0% | 0 |
| 6. Course technology | 80.7% | 113 |
| 7. Learner support | 37.9% | 53 |
| 8. Accessibility | 0.7% | 1 |
In terms of how many standards sections per course were compliant, based on meeting the essential standards, no course fully met the essential standards of all 8 sections. The highest number of section compliant within a course was 5, which was achieved by 1 course. The mean number of compliant sections was 1.8 (SD = 1.1), with a median of 2 compliant sections.
Distribution of the number of compliant sections per course
| Compliant sections | Course Count | % |
|---|---|---|
| 0 | 9 | 6.4% |
| 1 | 59 | 42.1% |
| 2 | 38 | 27.1% |
| 3 | 20 | 14.3% |
| 4 | 13 | 9.3% |
| 5 | 1 | 0.7% |
In terms of the individual sub-standards, 21 sub-standards had the majority (>50%) of the ratings as being 0, while 21 standards had the majority of ratings being the category maximum. (see Appendix A for the data tables). Additionally, though partial marks for a standard could occur (e.g., a 1 or 2 out of 3, instead of only 0 or 3), they were exceedingly rare, accounting for 0.3% of all ratings (18 out of 140 × 43 = 6 020).
