Monitoring, Evaluation, and Reflection on AI Use
Evaluating Impact
In an earlier section of this guide, we introduced exercises to help you identify specific problems or areas of curosity regarding AI. Now that we’ve explored a range of tools and ideas for embedding AI in your educational and research practices, it’s timely to reflect on those initial exercises.
The emerging field of AI in education presents opportunities to develop methods for assessing the impact of AI on teaching and research. Below, you’ll find a collection of research questions / curiosities, with potential methodologies to assess the impact and effectiveness of AI tools.
Quantitative Insights
Grades
Curiosity: How have AI tools influenced the average grades or assessment scores in your courses?
Methodology: Compare average grades, test scores, and assignment scores before and after AI implementation.
Engagement and Participation
Curiosity: In what ways has AI contributed to increased student participation and engagement in classroom activities?
Methodology: Track participation rates in discussions, attendance, and engagement in online platforms or forums.
Critical Thinking and Problem-Solving
Curiosity: Can you identify specific improvements in students’ critical thinking and problem-solving skills due to AI tools?
Methodology: Use pre- and post-implementation surveys, assessments focusing on critical thinking, and problem-solving tasks.
Accessibility, Diversity, Inclusivity
Curiosity: Have you observed any changes in the diversity and inclusivity of classroom interactions since implementing AI?
Methodology: Analyze demographic data of class participation, conduct surveys on student perceptions of inclusivity.
Teaching Practices and Workload Management
Curiosity: What changes, if any, have you noticed in your own teaching practices and workload since adopting AI tools?
Methodology: Self-assessment of teaching methods, time spent on grading and administrative tasks, educator surveys.
Scope and Depth of Research Findings
Curiosity: How has the use of AI tools influenced the scope and depth of your research findings?
Methodology: Evaluate the number of new research questions addressed, depth of analysis in research papers, and peer reviews.
Efficiency in Research Processes
Curiosity: In what ways has AI integration affected the efficiency and speed of your research processes?
Methodology: Time taken for data collection and analysis, number of research processes automated by AI, time-to-publication.
Publication and Dissemination
Curiosity: Have you observed any significant changes in the rate of publication or dissemination of research findings?
Methodology: Track the number of publications before and after AI implementation, citation rates, and reach of research findings.
Qualitative Insights
Student Feedback on AI-Enhanced Learning
Curiosity: What are students’ perceptions of AI tools in their learning experience?
Methodology: Analyze themes from student surveys, focus group discussions, and feedback forms regarding AI’s usability, engagement, and impact on learning.
Faculty Perspectives on AI Integration
Curiosity: How do faculty members perceive the integration of AI into their teaching methods?
Methodology: Conduct interviews or focus groups with faculty to gather insights on changes in teaching practices, challenges, and benefits of using AI.
Impact of AI on Classroom Dynamics
Curiosity: How has the introduction of AI tools affected the dynamics and interactions within the classroom?
Methodology: Observe classroom interactions and gather anecdotal evidence or case studies on changes in student collaboration and participation.
AI’s Role in Inclusivity and Accessibility
Curiosity: What is the perceived impact of AI on inclusivity and accessibility in educational settings?
Methodology: Collect qualitative feedback from diverse student groups on AI’s accessibility and its role in fostering an inclusive learning environment.
Changes in Student Motivation and Attitudes
Curiosity: How has AI implementation affected students’ motivation and attitudes towards learning?
Methodology: Assess student reflections and narratives on their learning journey with AI, focusing on motivation, interest, and attitudes.
Faculty Workload and Time Management
Curiosity: How has AI impacted faculty workload and time management in terms of teaching and administrative responsibilities?
Methodology: Evaluate faculty self-reports and diary studies on time allocation and perceived workload changes.
AI’s Influence on Research Methodologies
Curiosity: How has AI integration transformed traditional research methodologies and approaches?
Methodology: Gather insights through in-depth interviews with researchers about changes in research design, methodology, and data analysis.
Perceptions of AI in Enhancing Research Quality
Curiosity: What are researchers’ perceptions of the impact of AI on the quality and innovation of their research?
Methodology: Analyze researcher testimonies on the contribution of AI to the depth, breadth, and innovation in their research.
Challenges and Ethical Considerations in AI-Driven Research
Curiosity: What are the perceived challenges and ethical considerations when using AI in research?
Methodology: Collect viewpoints from ethical reviews, focus groups, and interviews on the challenges and ethical dilemmas faced when integrating AI into research.