10
University of Tasmania
- The University of Tasmania, Australia, executed a pilot project involving a learning design team and academic educators to showcase the practical implementation of AI chatbots in assessment design.
- The University’s learning design team organized a workshop with the aim of aiding academic educators in enhancing their assessment redesign efforts.
- The key objectives included integrating skills development into student learning journeys and enhancing the authenticity of assessments, while also addressing concerns related to AI’s impact on academic integrity.
- To streamline the assessment design process, the team leveraged ChatGPT, an AI chatbot.
- The approach aligned with recommendations to prioritize authentic assessment methods and minimize reliance on AI tools.
- The methodology included a structured sequence of prompts. ChatGPT was trained using the curriculum and assessment framework, enabling it to generate relevant and context-sensitive responses.
- This approach aimed to stimulate both divergent and convergent thinking, assisting educators in efficiently selecting and formulating assessment tasks.
- Multiple iterations were conducted to fine-tune the prompts and ensure consistent and contextually appropriate responses.
- Feedback received from academic staff participating in the workshop indicated that the process was both valid and reliable.
In conclusion, the University of Tasmania’s initiative showcases the potential of AI chatbots, like ChatGPT, in expediting assessment design. The process encourages collaboration between learning designers and educators, aligning with the goal of enhancing authentic assessment practices.
ChatGPT being prompted to assume a given role and clarify the user’s intended task.
Prompt:
Act as a Learning Designer. You are tasked with supporting an academic to redesign 3 assessments, for a 3rd university unit. The focus of this redesign is to brainstorm authentic assessment options for the unit. (Note: a unit is a subject or course within a degree).
ChatGPT Response:
Understood. As a Learning Designer, I have been tasked with supporting an academic to redesign three assessments for a third university unit. The focus of this redesign is to brainstorm authentic assessment options for the unit.
Next, provide specific conditions for ChatGPT to consider, ensuring it comprehended the desired assessment type, while also enhancing confidence in meeting the requirements.
ChatGPT provided additional context to complete the task.
Prompt:
In the process of collaborating with the academic, we need to use the following Authentic Assessment Types. Proximal assessment – This type of assessment measures a student’s ability to perform a task similar to what they would do in the real world. However, it is still a simplified or artificial version of the real thing. For example, a culinary student might be asked to prepare a dish under time pressure in a simulated kitchen environment.
Intermediate assessment – This type of assessment requires students to apply their knowledge and skills to real-world situations that are more complex and authentic than proximal tasks. For example, an engineering student might be asked to design a prototype for a product that meets a real-world need.
Authentic assessment – This type of assessment requires students to apply their knowledge and skills to real-world situations that closely mirror the complexity and demands of the tasks they will face in their future careers. For example, a medical student might be asked to diagnose and treat a patient with a complex medical condition.
Simulation assessment – This type of assessment involves using simulations or virtual environments that replicate real-world situations, allowing students to practice their skills in a safe and controlled environment. For example, a pilot trainee might be asked to fly a simulated aircraft under different weather conditions and emergency situations.
State that your confirm understanding.
Next in the prompt process is the presentation of vocational skills aligned with learning goals for a study unit, offering varied assessment choices and serving as a valuable brainstorming tool for academics.
ChatGPT prompts output to include focus on vocational skills alignment, helping to facilitate practical thinking.
ChatGPT Response:
Based on the Unit Description and Intended Learning Outcomes, it is clear that this unit aims to prepare students for future work opportunities and develop their personal and professional skills. In order to be successful in the field of STEM, there are several contemporary skills that are highly valued by industry. Here are five critical skills that are relevant to this unit:
- Collaboration and teamwork: In the context of this unit, students will need to work collaboratively and ethically with colleagues throughout the STEM experience. This requires strong collaboration and teamwork skills, including the ability to communicate effectively, listen actively, and work towards shared goals.
Authentic Assessment Option: Intermediate assessment. Students could be tasked with working in a small group to develop a solution to a complex STEM challenge, demonstrating their ability to collaborate effectively, communicate ideas clearly, and work towards shared goals.
- Critical thinking and problem-solving: STEM professionals need to be able to analyze complex problems and develop effective solutions. This requires strong critical thinking and problem-solving skills, including the ability to identify relevant information, evaluate different options, and make informed decisions.
- Authentic Assessment Option: Authentic assessment. Students could be presented with a real-world STEM problem, such as developing a sustainable energy solution for a community, and be required to apply their critical thinking and problem-solving skills to develop a comprehensive solution.
The next step in the prompt sequence was to identify assessable skills or curriculum components, enabling an efficient assessment style by suggesting options aligned with multiple learning goals from previous prompt outputs. This streamlined the design, avoiding redundancy and ensuring comprehensive coverage of intended learning outcomes and assessment criteria.
ChatGPT prompt used to guide output to align skills with curriculum components.
Prompt:
Using the above, suggest which intended learning outcomes and assessment criteria could be assessed in a common assessment method/task. If there isn’t a common assessment method/ task, suggest a final assessment method, based on the authentic assessment brief.Ensure assessment Criteria are considered.
ChatGPT Response:
Based on the suggested authentic assessment methods, it is possible to identify some intended learning outcomes and assessment criteria that could be assessed through a common assessment method/task.
For example, the simulation assessment option could be used to assess ILO 2-“Collaborate professionally and ethically with colleagues throughout a STEM experience” and its corresponding assessment criteria 2.1, 2.2, and 2.3. The simulation could involve a team- based project where students are required to work together to solve a complex STEM- related problem while demonstrating professional and ethical behavior in their interactions with their team members. The simulation could be designed to mimic a real-world scenario, where students must navigate challenges that are common in professional settings, such as differing opinions and priorities, conflicting schedules, and communication barriers.
Alternatively, a final assessment method that could be used to assess all three intended learning outcomes and their corresponding assessment criteria could be a portfolio. The portfolio could be a collection of artifacts, reflections, and self-assessments that demonstrate the student’s ability to assess the impact of STEM on society, collaborate professionally and ethically with colleagues throughout a STEM experience, and evaluate their readiness for future work opportunities by reflecting on personal and professional development during a specific experience. The portfolio could include a variety of authentic assessments, such as case studies, project reports, peer evaluations, self-reflections, and action plans, that provide evidence of the student’s achievement of each intended learning outcome and their corresponding assessment criteria.
Next (Eager, B et al, 2023) prompted the AI model to identify six implicit skills associated with Portfolio assessments, revealing valuable insights for academic educators. Understanding these skills helped educators determine the necessary teaching content to prepare students for the assessment tasks and aided in developing relevant learning activities with AI-generated support.
ChatGPT prompt used to identify underlying or implicit skills within an assessment.
Prompt:
Propose 6 underlying skills that students will need to develop to perform well in the STEM and Society Portfolio assessment. The proposed list of skills should include 3 that are industry specific and 3 that are academic-specific.
ChatGPT Response:
Here are six underlying skills that students will need to develop to perform well in the STEM and Society Portfolio assessment:
Industry-specific skills:
- Collaboration and teamwork: The ability to work effectively with others, share ideas and responsibilities, and collaborate towards a common goal is essential for success in any STEM-related field.
- Problem-solving: The ability to identify, analyze, and solve complex problems is a critical skill in the STEM industry. Students need to demonstrate the ability to apply logical and analytical thinking to develop innovative solutions to real-world challenges.
- Ethical and professional behavior: Students must demonstrate professional conduct and ethics when working with colleagues, clients, and stakeholders in the STEM industry. This includes showing respect, honesty, and integrity in all interactions.
Academic-specific skills:
- Critical thinking: The ability to think critically, analyze information, and evaluate arguments is crucial for success in academic settings. Students must be able to draw conclusions and make informed decisions based on evidence and logical reasoning.
- Research and analysis: Students must demonstrate the ability to conduct independent research, gather and analyze data, and synthesize information from multiple sources.
- Self-reflection and evaluation: Students must be able to reflect on their own learning and evaluate their own progress towards achieving the learning outcomes of the assessment. This includes identifying areas of strength and weakness, setting goals for improvement, and seeking feedback to enhance their performance.
References
Eager, B., & Brunton, R. (2023). Prompting Higher Education Towards AI-Augmented Teaching and Learning Practice. Journal of University Teaching & Learning Practice, 20(5). https://doi.org/10.53761/1.20.5.02
Last Updated November 20, 2024
