Linking with the student course feedback survey (2015/2016)
Given an ongoing discussion about the use of required resources, an exploratory analysis was conducted looking at two questions:
- Was the use of required resources, as rated on the student course feedback survey, different between eText/IPM and non-eText/IPM courses?
- Is there a difference between the mean achievements of eText/IPM courses based on high and low rated usage of a resource?
The student course feedback survey from the 2015/2016 reference period offered the opportunity to explore these questions as it included a question on required resources. Specifically, students were asked to rate, on a scale of 1 to 5, with 5 being high, the following statement: “The required course materials (e.g. textbooks, manuals, software, etc.) are used in the course.” Unfortunately, no other student course feedback survey reference period available (e.g. 2014/2015, 2016/2017) appears to contain that question.
Use of required resources by eText/IPM flag
The 2015/2016 student course feedback survey covered 7 341 course sections. Consideration was given to the generally low response rates of the survey, and a response rate cut-off of 50% was set, leaving 511 course sections, for which the mean score of the required resource question was calculated. Joining this with the numeric grade set resulted in 459 course sections remaining.
| Course sections eText = N | Course sections eText = Y | |
|---|---|---|
| Count | 370 | 89 |
| Mean (required resource) | 4.103 | 3.996 |
| SD (required resource) | .478 | .533 |
The presence of outliers in each of the two groups was assessed with boxplots; normality of the distribution was assessed with a Kolmogorov-Smirnov test. Significant outliers were observed in the eText = N group; the eText = Y group exhibited only a handful of outliers; deviation from normality was detected in both groups (p < 0.05). While independent t-tests can be robust to violations of normality, the extensive presence of outliers in at least one of the groups may too strongly affect the power (Type II error rate) of the test. Instead, the non-parametric equivalent, the Mann-Whitney U test of distribution, was used. The test did not detect a statistically significant difference in the distributions of the required resource score, between the two groups of the eText/IPM flag, U = 14 479, z = -1.768, p = 0.077. We, thus, retain the null hypothesis and conclude that there is no difference between the two groups.
Required resource usage impact on achievement
Explore the impact of required resource usage on achievement, the required resource variable was recoded such that a score greater than or equal to 4 was coded as “High”, and a score below 4 was coded as “Low”.
| eText/IPM | Usage (Num. courses) Low | Usage (Num. courses) High |
|---|---|---|
| N | 113 | 257 |
| Y | 35 | 54 |
Each category of eText/IPM was assessed for the impact of usage on the average grade obtained for the section. Boxplots were used to assess for outliers; Shapiro-Wilk tests were used to test for normality where appropriate.
For the non-eText/IPM course sections, the boxplot revealed outliers for the high-usage group; one extreme outlier was removed, and the rest were retained; both groups deviated from normality (p < .05); equality of variances was met (p = .615). An independent samples t-test was run and no statistically significant difference was found between the group means, t(367) = -.776, p = .438, leading to the null hypothesis being retained.
For the eText/IPM course sections, no outliers were detected; no deviation from normality was detected for either group (Low, p = 0.691; High, p = .358); equality of variances was met (p > 0.686). An independent samples t-test was run and no statistically significant difference was found between the group means, t(87) = -1.533, p = .129, leading to the null hypothesis being retained.
