Introduction

This case study is an example of the use of a team evaluation tool, ITP Metrics, used in the self-evaluation of teams for a software engineering design course. The use of this tool is motivated by the Canadian Engineering graduate attribute accreditation requirement to evaluate students individual and teamwork in the course.  One of the graduates attributes program indicators is to demonstrate the ability to establish and monitor team organization structure by assessing individual contributions to a team activity and provide feedback.

The ITP metrics team at the University of Calgary have developed a number of team assessment tools of which the peer feedback is leveraged in this case study. This particular tool supports peer evaluation of team members based on five competencies: commitment to the team; communication; the foundation of knowledge, skills and abilities; encouraging high performance; keeping the team focused.

Once team members complete the online peer evaluation they receive an individualized peer feedback report. The report contains sections where an individual can reflect on competency and list specific, actionable, and measurable development goals. At the end of each report, there is a Team Debrief section where teams have to meet and capture individual and team strengths and development areas. In this case study, I focus on the completion and submission of this team debrief as a way to assess the teams’ effort and ability to assess each of the individuals’ contributions to the team and improvement areas.

This particular case study also leveraged two other teaching and learning environments that had to be manually integrated. These are a GitHub Classroom roster and a group assignment repository (used for the project submission) and the Blackboard learning module system (used for a team debrief submission and grading.) The focus of the case study is on the ITP metrics peer and team assessment tool, but they are all related through the management and grading of the teams.

Resources

• https://www.itpmetrics.com/ 
• https://classroom.github.com/
  

Steps for Implementation

The steps to set up ITP Metrics for particular teams is well documented so the focus here is on the full use case that leverages both GitHub classrooms and Blackboard for the team assessment.

Step 1. Setting up a class roster in a GitHub classroom

As the core repository for student projects is GitHub, the number of groups and membership size were first specified in a GitHub classroom account (note: a GitHub classroom account is required) and a classroom organization has to be created (the details of these steps are not part of this chapter.)

Once a classroom is available a class roster has to be uploaded in order to associate the students’ individual GitHub accounts to the institutions’ student ID [you may need to consider this in terms of GDPR complinace]. For our case, this is done through a CSV file where each row is simply a student ID.

Figure 2: Sample of CSV file (a column of student IDs)

Step 2. Creating a GitHub group assignment

Students submit their project work through a group assignment. The group assignment settings determine the number of groups and members in a group. See image below.

Step 3. Creation of GitHub groups (teams)

Creation of groups (teams) in GitHub is dynamically performed and is fairly convoluted because students have their own personal GitHub account and can create any identifying name for the teams.  Students use the assignment link to access the assignment. Then are requested to sign into their GitHub account and associate themselves to a particular student ID and then either create or join an existing team (first team member to sign in creates the team.)

The final result after all students have done this is a roster that looks something like that below.

The only 2 team identifiers that matter are in bold italics. The student ID (identifier) and the group name (group_name). Without the specification of a roster, it would be impossible to associate the student IDs (typically use in the Blackboard system) to the group name.

Step 4. Creation of ITP Metric teams

The creation of and ITP account and set up of the peer evaluation in ITP Metric is well documented in the following link https://www.itpmetrics.com/resources/info/en/2_Instructions_PeerFeedback_EN.pdf so the focus here is on the creation of the student/team list.

Similar to GitHub a CSV file is required that specifies the last and first names of the participants, an email, and a group name as shown in the example below. The instructor has to manually associate the correct students’ names and email addresses to the GitHub group that was created by the students.

Step 5. Deployment and completion of the peer evaluation

Once the student and team list is uploaded to the peer assessment it can be made available to the students. The ITP system takes care to notify the students via the email in the uploaded file (this is typically an institution email) with sign up ad completion instructions.

Step 6. ITP Monitoring

 Student progress in the completion of the peer evaluation can be monitored online via an interface as shown below. 

Once a team has completed all the peer evaluations the individual reports become available to them. The instructor can view the reports by choosing the Results link next to a particular student’s name through the Dashboard. One can also download a number of statistics but in this particular case study this is not used for grading the students as it is very challenging to associate a grade to a peer feedback value this metric on its own is not sufficient to meet the individual and teamwork engineering graduate attribute (this metric was previously used by the instructor.) For that reason, a group assignment dropbox was created for students to submit a completed team debrief (last page of any individual peer report https://www.itpmetrics.com/questionnaire.view?guid=21b5fdcb53a041d2804b769ab955e3e0 ) that consists of tables that summarize the key strengths and contributions as well as the strengths and development areas of the team. The peer feedback form lists steps on completing the team debrief.

Step 7. Setting a Blackboard Team Debrief Submission

Most teaching institutions leverage an LMS like Blackboard and so the set up of a group assignment is fairly well understood and documented. The challenge is to set up the groups for this particular case given the uniqueness of the team names.  

Blackboard supports the uploading of custom groups into a group set via the specification of the groups in a CSV file with a structure to that shown below. The key columns here are the Group Code (unique name for the group), Title (matches the GitHub group name), Group Set name, and no self enrolment (group members to be uploaded in step 8).

After creating the Blackboard project groups, it is fairly straightforward to upload the group members to the groups by creating another CSV file that relates groups (identified by the Group Code) to student IDs (identified in our BlackBoard system as User Name.)

Step 8. Evaluating the Team Debrief

This final step is simply a matter of creating a group assignment dropbox in Blackboard and associating it to the project groups created in step 8. Note that the group name that appears to the students is, in fact, the same name of that that they used in the creation of the GitHub teams.

The evaluation of the submission is done using a grading rubric as that presented below.

Dealing with Challenges

As one can probably summarize from this case study that there is a significant amount of work required to make certain that the 3 systems used are in proper synchronization. Concerning the use of the ITP metrics tool, it is rather straightforward and simple to use for the scenario presented in this chapter.

If the readers are not using a tool like GitHub then the assignment of groups and their names is simpler as these can be specified by the instructor in Blackboard and then uploaded to the ITP Metrics tool. Using any tools that are not integrated with the institutions LMS will add another level of coordination. In this case the IT Metrics uses student email addresses as identifiers while the LMS uses student IDs.

Encouraging students to complete the peer assessment in time is rather challenging. This was facilitated by the fact that the actual deadline that counted was the completion of the team debrief that was created in the Blackboard LMS which students utilized on a daily basis. ITP Metrics has no date reminder for completion.

Having students complete the assessments in a meaningful manner is challenging. The rubric can help since we do not teach in this particular course team dynamics and management. A more effective way to use the tool would be to perform the exercise near the start of the project and evaluate how well the team has improved. ITP Metrics does not seem to have a tool to help perform this second assessment.  

Author

Dr . Liscano received his Ph.D in Systems Design Engineering from the University of Waterloo in 1998 and MASc in Mechanical Engineering in 1984. Dr. Liscano is now an Associate Professor in the Department of Electrical and Software Engineering, Faculty of Engineering and Applied Science, at Ontario Tech University. Before joining UOIT; he was an assistant professor at the School for Information Technology and Engineering of the University of Ottawa; a senior research engineer at Mitel Networks from 2001-2003; and a research scientist at the National Research Council from 1998 – 2001. Dr. Liscano’s main areas of research are Mobile and Pervasive Computing, mobile sensor networks and Software Engineering.