Torgny Roxå: On Students and Educational Reform
December 20, 2016
The following text is an edited transcription of an interview conducted with Torgny Roxå of Lund University on October 5th 2016. As a Distinguished Scholar at the Macpherson Institute, Roxå lends his 27 years of expertise as an academic developer to support teachers in their endeavor to improve student learning. In this interview, Roxå sat down with two McMaster undergraduate students, Raj Jain and Hana Faidi, to share additional insights and observations related to student evaluations of teaching and academic cultures, among other pedagogical themes. The interview has been edited to reduce length and increase clarity.
Raj:
I want to start by asking you about a recent article you published on DisruptED, the MacPherson Institute’s teaching blog dedicated to explorative pedagogical themes and discussions. In that article, titled “Assessing and Rewarding Good Teaching,” you assert that student evaluations of teaching (SETs) are “irresponsible and unworthy of academic institutions.” In your opinion, what could enhance or supplement SETs besides the idea of integrating a process of “critical interpretation” which you also write about?
Torgny:
The question that I basically posed in the blog is that the use of student evaluations can sometimes be unworthy and irresponsible of academic institutions, not that student evaluations of teaching are in and of themselves unworthy and irresponsible. I would say any day that if you don’t ask students about their experiences of teaching and learning, that’s irresponsible and unworthy of academic organizations. But in many institutions and preferably, to my experiences, in North America, student evaluations of teaching are used instrumentally to produce numbers, and these numbers become crucial for academic teachers to be promoted and for the development of salary — all that kind of stuff. That is irresponsible, because there are so many factors that influence student evaluations of teaching.
A lot of these factors teachers cannot control. I can give you a few examples. The most striking example, which is up for debate quite a lot, is that female teachers are treated differently than male teachers in student evaluations of teaching – a point that comes across really, really clearly in research. The explanation given for bias or discrimination against female teachers is that students are a part of society, and if you take a bunch of people and put them in a room, those values will come into the classroom as well. So it’s not strange, but it’s also not fair, to use student evaluations as an objective measure of teaching given this context.
We also know that if you teach courses in a three level building, student evaluations get weaker and produce lower scores on level three than if you teach students on level one. We also know that the faculty of Engineering usually gets lower student evaluations of teaching than those in the Humanities. There is a study in Italy which I think is taking things a bit too far, claiming that student evaluations of teaching vary according to weather. Yeah, they show that. I think there’s something wrong with the basic assumptions, but it’s yet another example of how complex this issue is.
Just asking students the simple question “Was this a good course?” will not produce valid information. It produces some information that can be vital to developing teaching. If you don’t have the opinion of students, you lack very important information. But you also can’t use students as judges of what is good teaching alone, that’s what I mean. When you do that, that’s irresponsible.
Raj:
When you integrate this sort of critical interpretation into student evaluations of teaching, what types of reform could you bring to eliminate the gender bias that you spoke about?
Torgny:
Actually, what I think is important to remember is that you cannot define good teaching. That has never happened because every piece of teaching can be done in good and bad ways. What you can do is look for good learning. You can try to develop an opinion about what good learning is and there are lots of ways to determine effective forms of learning in research and in literature. Basically, very simplistically, you can say that students can move towards passing the exam instrumentally, or they can develop a deeper understanding and personal meaning. That is the difference between what’s called the surface approach to learning, which is an instrumental way of learning, compared to the deep approach, which fosters understanding and personal meaning.
Since you’ve been in the system of higher education as students, you’ve probably been both surface and deep learners. If you’re forced into a course where you don’t like the course content, where you can’t see the point of what you’re learning but you need the course and the grade, it’s really easy to go instrumental and just pass the exam. That’s a surface approach. But it’s not a way of valuing students. On the other hand, if you’re sort of intrigued by the subject and you find that subject meaningful the learning improves. You probably can agree with me on that. That is the deep approach.
What you can do is to look for aspects in teaching that favours a deep approach. Aspects in a course that are likely to produce a greater number of students using a deep approach tend to produce better courses and reflect better teachers. However, you can also easily organize courses that make students gravitate towards surface learning. The easiest thing I say to teachers is that if you are rude and bad and evil to students, they will go for surface learning because they don’t like the whole experience. Who would? Students are people, just like anyone else. If you have a high workload, superficial learning, you have a lot of data, a lot of factual little pieces, and you threaten students, and there is an exam which is a multiple choice questionnaire without any sort of things … Why wouldn’t you go surface? That’s an example of a bad course.
If you have student evaluations, you should ask students to help you differentiate between courses leading to good learning instead of bad learning. You see what I mean?
Raj:
Yeah. Do you think that the temptation for students and teachers to use an instrumental approach, as opposed to the deep approach, is also because they find that it is easier to get better grades? If both teachers and students focus less on understanding the content and more on passing the tests…
Torgny:
It’s rational for students to use the surface approach and instrumental forms of learning because you get higher grades if you do that, generally speaking. But that’s still an example of a bad course. You can’t blame the students because they are in a situation they have to deal with — they need to pass the exams, so they adapt to the learning that goes on.
This has been researched, quite interestingly, in China, because they have another way of picturing the relationship between students and teachers. It’s built into the pedagogic tradition that students should be able to repeat what the teacher is saying and they get high grades for that. You might say this is a surface approach that encourages surface learning, but if that were the case, graduates from China would be poor professionals but they’re not. They’re clearly not.
They looked into this problem and what they found was that these bright students in China talked with teachers as if they were using a surface approach but they were actually using a deep approach in their own study rooms and they just played the exam game, because they knew that if they started to question the professor, they would risk getting low grades. They can’t question the professor so they kept that higher level learning to themselves while demonstrating learning tailored for exams and assessment systems. It did not allow for students to produce and express their own critical thinking or their own personal views or arguments. They just keep that hidden. That’s what researchers found.
Students react rationally to the way we organize teaching. If they use a surface approach, it is because we have organized a course which favours a surface approach. Now I hear myself, and it almost sounds like students are cattle, as if you can wave them here or there. That will not happen. But even so, the way courses are organized and designed will influence the directions students will want to go while learning. I just want to stress this. We have to organize courses in a meaningful way for meaningful outcomes.
Raj:
In much of your work, you write about creating what you call a quality academic culture, one in which “teaching develops slowly but constantly, by the active involvement of academic teachers.” I’m wondering if there are any efforts that students can put forth to contribute to academic cultures and the types of educational changes that you write about?
Torgny:
I love this question! I find it to be a really intriguing and a very good question. I’m not sure I have a really good answer. One bit of phenomena is that students sort of circulate into academic cultures. As you might know, academic culture is really stable stuff. It doesn’t change easily. It’s the reason why universities have been around. Society changes, universities remain. Students circulate in for a while. Even though we always have students in the campus, many thousands of them, there are constantly new people arriving, new individuals coming into an institution. That is interesting when it comes to looking at the possibility for students to organize and carry on influential strategies in academia, because it takes so much time. To do that, you have to have a strategy that generations of students can sign up to, and usually they don’t, because they have their own priorities and they’re only here temporarily. Time is a limiting factor here.
On the other hand, I’ve seen groups of academics listen to students in a very different way. Academic teachers can go from zero to one hundred because a group of students said something about their teaching. Student behaviour has an immense effect on academic teachers. Since teaching is not necessarily the prime conversational subject between academic teachers, they very seldom sit down and have a focused conversation about teaching, where they usually do in research. That means that it’s really hard to become a teacher, to construct an identity as a teacher, by interacting with your colleagues, because it’s so scarce. It’s not stable. Things happen. While with students, when teachers interact with students, there can be an immense sense of “Now I’m a teacher. This is my teacher identity, when I see these faces, work with these students, talk to them.” Of course, if students react in a negative or positive way, it has an immense effect on teachers. But we haven’t explored that very much. Research is scarce on the effect the multitude of faces has on teachers.
Research is scarce on the effect the multitude of faces has on teachers.
We lack research on how these impacts actually work. We know as anecdotes, and you can see yourself, if my relation with students is really crucial in terms of my identity as a teacher, and then I get the student evaluations and there might be 30 of them or 40 of them, and two of them write, “You’re really a lousy teacher“, that stands out. It has a fantastic effect on the teachers. I’ve known many teachers who, for years and years, go back to these instances when students said something that hurt them. But the opposite is also true: teachers go back to when students said something positive and think, “Wow, they like me. They like what I’m doing.” That’s a sign of their importance, but we know too little. We don’t know enough about how that relation operates and how to support teachers to actually use that relation in significant pedagogical ways.
As an institution, we don’t know enough about how to support students in a better way, to actually engage closer, more within the institution. Student partnership movements are definitely one way of doing it, but that is quite recent. It hasn’t been around for very long. That is one strategy, but there might be other strategies where we can do that, because in the end, students benefit a lot from getting closer to the culture of academia and teachers benefit a lot with positive and constructive relationships with students. Teachers get a lot of ideas that are actually comments from students, but we haven’t really figured it out, we haven’t organized it. It’s slippery stuff, this. We should do better here. I think there is a lot of room for improvement in that sense.
Hana:
I think one limitation that I’ve seen is that, in year one, students are dumbed down. We are not exposed to things like we should be. What I mean is that students are not treated like someone who will be a professional in the field. We aren’t actually treated like someone who understands the field until the second semester of the third year of our programs. We are exposed to names and expected to understand theories and where they fit into the global world we live in, but we are less frequently asked to challenge them. Nor are we asked to really get involved with these topics, at least not until much later in our undergraduate careers.
Torgny:
That is very interesting because what you refer to is a socially constructed norm in teaching cultures, because there is no law — it is not in the DNA, it is not science claims — but sociology that explains that sort of assessment about what students should know and when. And all these norms can be broken in academic cultures, but it is hard. To me, it is really important that teachers start acting as a group because, if they did, they would probably have the kind of reforms they talk about. That’s why I stressed in my lecture at McMaster this week the importance of having teachers talking about teaching with other teachers. Because that is when teachers can start to coordinate with each other, it is not impossible. If you do it individually, you become a problem, a deviation in the group of academic teachers. Academics live in social worlds organized into disciplines. There are hierarchies and statuses and they are being assessed all the time, so it is really risky to make changes individually. But when teachers go together, they can actually invent something like problem-based learning which before 1967 had not been around. And these people here in the medical faculty in your university, they did something that no one had done before and now there are many places in the world following that! So of course, teachers can invent something similar if they put their mind to it. This is fantastic!
To put their minds to it, teachers have to feel safe, they have to start talking with each other, and to do that they also need a senior leadership who appreciates and sends signals to the organization: “Hey! They are talking about something and that’s good!” Give teachers something, lift them up, so they feel that they have the organization on their side. When we looked at Lund University for example, senior leadership initially had no idea conversations about teaching were happening. They knew where the bad examples of teaching were but when asked about the good, they did not know. This says something critical: there is something wrong in our view of the governance of the university. Leadership should know both good and bad examples of teaching, and how can you improve unless they know where the good examples are? In order to do that, they need good observations and that is also why we also need good student evaluations and partnerships with students in university institutions to enhance educational practices overall.
