Overall student analysis

It could be argued that the pooled numeric grade analysis described above did not appropriately respect the independence of observations, an assumption of t-tests. Thus, an analysis of student GPAs obtained from averages of both non-eText and eText flagged grades was undertaken as part of a repeated measures design. The resulting dataset was based on an aggregation of the numeric grades data over all reference periods, containing 55 202 unique students. This was further reduced by requiring a student to have taken both eText/IPM and non-eText/IPM courses, a requirement for a repeated measures analysis. The new dataset contains 32 371 unique students, and the mean number of non-eText flagged courses (M = 12.08, SD = 8.75) and the mean number of eText flagged courses (M = 5.81, SD = 4.55) were calculated.

A boxplot was used to check for outliers in the distribution of the differences; a Kolmogorov-Smirnov test was used to check for normality of the distribution of the differences. Significant outliers were detected in the differences boxplot; likewise, the distribution appeared to deviate from normality (p < .001). The paired samples test was run and the result was not statistically significant, t(32370) = 1.223, p = .221, leading us to retain the null hypothesis that the mean difference of GPA, based on the eText/IPM and non-eText/IPM flag, within each student, is 0.

Considering that the non-statistical significance could be due to the outliers’ effect on the Type II error rate, further examination of the outliers was conducted. One hypothesis was that the calculated GPAs for students were being affected by instances where few grades in a given group had been earned, for example, where only 1 eText/IPM course had been taken, the eText/IPM GPA for that student would be based solely on that one course. Thus, different cut points, i.e. minimum numbers of courses required for both eText and non-eText courses for the difference to be included, were tested for their effect on the significance of the paired samples t-test, detailed in the table below.

Paired t-tests of student eText and non-eText GPAs by minimum number of courses

Note: Mean difference is non-eText GPA – eText GPA

Cohen’s d, calculated above as d = M/SD, gives us a measure of effect size for statistically significant differences. According to Cohen (1998), values of d < .2 would be considered small-to-negligible. Thus, with the changes made to the minimum number of courses that form part of a GPA, the paired t test found a statistically difference at a cut point of 3 courses, t(20610) = 2.470, p = .014, with the non-eText GPA higher than the eText GPA by a mean difference of .010 grade points, SD = .581, SE = .004, with small-to-negligible strength of effect size, d = .017. The null hypothesis of no mean difference between groups can be rejected.

Analysis of student GPAs by faculty or school

Further segmenting of student GPAs by faculty or school, for students who had taken 3 or more eText/IPM and non-eText/IPM courses each, allowed for an investigation of whether or not the impact of eText/IPM varied by faculty or school. Paired t-tests were run for students in each faculty or school. 5 faculties or schools showed statistically significant differences between eText and non-eText GPAs. Effect size for the Faculty of Technology and Trades was small-to-moderate, favouring eText GPAs, d = -.228; the other effect sizes were small-to-negligible, with an absolute value of Cohen’s d < .2. The results of the analysis are summarized below.

Paired t-tests of student eText and non-eText GPAs by faculty or school

Note: Mean difference is non-eText GPA – eText GPA.