An analysis of pooled numeric grades was conducted to test whether the eText/IPM factor affected student grades. The data were selected based on the condition of the pass/fail grading = ‘N’, to indicate that they were using a numeric grading system. The grades were distributed between reference periods as described in the table below.

Numeric grades by reference period and eText/IPM flag

Mean difference (all periods)

An independent samples t-test was initially conducted to determine if a student’s grade in a given course was affected by that course being included in the eText/IPM initiative. An independent samples t-test tests the null hypothesis that there is no statistically significant difference between two groups (i.e. that the means are equal). The grouping, in this case, was on the eText/IPM flag, with non-eText/IPM (n = 666 424) and eText/IPM (n = 193 310). Outliers were noted in the boxplot of the non-eText/IPM group and were retained as necessary to the analysis (the outliers were all of the “Fail” grades for the group), accepting that they may increase the Type II error rate. No outliers were noted in the boxplot of the eText/IPM group. Data from both groups were non-normal, as assessed by Kolmogorov-Smirnov (both groups, p < 0.001), however the independent-samples t-test is considered to be robust to deviations from normality, especially with large sample sizes. The assumption of homogeneity of variances was violated, as assed by Levene’s test for equality of variances (p < 0.001).

A statistically significant mean difference was detected between the non-eText/IPM (M = 2.86, SD = 1.21) and eText/IPM (M = 2.56, SD = 1.33) groups, with non-eText/IPM higher than eText/IPM, M = 0.29, SE = 0.003, t(291 737.815) = 87.708, p < 0.001. Thus, the null hypothesis is rejected as we found a significant difference between the groups.

Mean difference (by reference period)

The procedure above was repeated for each reference period, retaining any outliers. The results have been tabulated below:

For each reference period, a statistically significant mean difference was detected between the non-eText and eText groups, with the non-eText groups higher than the eText groups. Thus, for each of the reference periods, we would reject the null hypothesis.

Effect size (two-way ANOVA, eText and Academic Year on grades)

A two-way ANOVA was conducted to examine the effects of the eText/IPM flag and academic year on grades. Residual analysis was performed to test the assumptions of the two-way ANOVA: boxplots were used to assess outliers, normality was assessed by a Kolmogorov-Smirnov test, and homogeneity of variances was assessed with Levene’s test. While some outliers were detected in some cells, no extreme outliers (more than 3 box lengths) were detected and all data points were retained; the residuals were determined to deviate from normality (p < .001); and homogeneity of variances was violated (p < .001). Nonetheless, as two-way ANOVA is considered robust to deviations from normality and homogeneity when sample sizes are sufficiently large, the two-way ANOVA was run.

A statistically significant interaction effect was detected between the eText/IPM flag and academic year on grades, F(3, 859726) = 55.817, p < .001, partial η2 < .001. Of the main effects, both the eText flag, F(1, 859726) = 9091.972, p < .001, partial η2 = .010, and academic year, F(3, 859726) = 279.801, p < .001, partial η2 = .001, had statistically significant effects. Thus, while all of the effects were statistically significant, the effect size of academic year, and the interaction term of academic year and the eText/IPM flag, were negligible, while the eText/IPM flag’s effect was small.