GenAI Output – Critical Evaluation
Learners should approach Generative AI content with the understanding that while these technologies can produce content that seems to be accurate and appropriate, it may not be. This critical analysis is something your learners need to practice. Consider modelling this process and including this type of activity as a learning activity in your course.
Humber College identified the following essential considerations:
Content Accuracy
Although AI models can generate grammatically correct text that may seem logical and informative, they can produce content that is misleading and/or incorrect. Always cross-verify AI generated content with trusted resources to ensure accuracy.
Bias Awareness
Generative AI can (and will) generate biased or skewed content. It is vital to critically evaluate the output for gender, racial, cultural, and ableist biases.
Ethical Considerations
Generative AI is powerful and can create content that seems incredibly human-like. This can lead to misuse, such as generating deepfakes or disseminating false information. As responsible learners, it’s crucial to understand and respect the ethical boundaries surrounding the use of these technologies.
Source Verification
Always verify the source of the information. Generative AI, including models like ChatGPT don’t generate information based on new data after their training cut-off. Furthermore, GAI is widely known to “hallucinate” meaning it will make things up and present the content as the truth.
Start by talking to your students. Using critical media literacy, the following guide offers foundational questions you can prompt your students with regarding Generative AI. The questions focus both on the forward-facing content as well as behind-the-scenes work for the medium. The questions address both representation of the power of construction and of distribution. The questions are intentionally broad – they will best be used to begin the process of analysis.
Questions About the AI Writing Tool
- Who created the AI writing tool?
- Who worked on training the AI writing tool?
- What dataset was used to train the AI writing tool? How does the diversity (or lack thereof) of the dataset influence the output of the AI writing tool?
- What do the privacy policies say?
- What are the limitations of this tool? (e.g., ChatGPT is not connected to the Internet, and therefore, cannot generally (though not always) draw connections to present-day events ; ChatGPT has a limit for how much text you can upload)
- Who is harmed and who benefits from this tool?
- What are the unintended and unexpected benefits and consequences of using this tool?
Questions About the Text Produced by the AI Writing Tool
- What information is presented in the text?
- What information is missing from the text? Why do you think that information is missing? (consider that ChatGPT generates text based on its training dataset)
- What type of language and word choices are used to convey ideas and information in the text?
- How are the language and word choices different from, or similar to, the way humans write? Why do you think that is?
- Who is the target audience(s) for this text? How do you know this?
- How reliable, accurate, and credible is the text? How did you determine this?
- What sources , if any, are cited? How accurate and relevant are those sources?
- What biases are present in the text? Why might this be?
- What might be the original sources used to generate this text? Conduct an Internet search and see if you can find the original source (it’s likely more than one source!) that the AI tool used to generate this text.
Attribution
Considerations for Your Course (Talk to Your Students) by the Centre for Teaching and Learning at Queen’s University is licensed under CC BY SA 4.0
