A Field Guide to Applied AI Literacy
WCUPA Teaching & Learning Center
Entry 4.2
Assessment in an AI World
Assessment in an AI World
Assessment in an AI World
Making the AI conversation part of the assignment
One strategy for dealing with AI-use on assignments is to design assessments that require students to use AI and submit their dialogue. This makes AI use visible and assessable.
The concept
While most conversations about AI and assessment focus on whether students are using AI, with a few modifications, instructors can start asking are students using AI on coursework effectively. Requiring students to use AI and to submit their interactions creates a visible and assessable interaction instead of a hidden one behind closed doors.
If students are provided sufficent structure and guidance, a conversation between student and AI can show how students are thinking through problems, articulating uncertainty, and where they are thinking critically. This potententially provides more insight into a student’s thought process than a final product.
What the conversation reveals — shallow vs. substantive
Shallow exchange
Student
Write me a discussion post about whether social media harms teenagers.
AI
Social media has become a central part of modern teenage life. Research suggests… [400 words]
Student
Make it shorter.
AI
Social media presents both benefits and challenges for teenagers… [200 words]
No position taken. No prior knowledge engaged. The student directed formatting, not thinking.
Substantive exchange
Student
I think the research on social media and teen mental health is more mixed than people say. Can you push back on that?
AI
You’re partly right. Jonathan Haidt’s work is widely cited but contested — critics argue the effect sizes are small. However, longitudinal studies suggest…
Student
You cited a 2021 study by Orben and Przybylski. I can’t find that — are you sure it exists?
AI
You’re right to check — I may have confabulated that citation. Orben has published in this area but you should verify the specific study…
Student brought a prior position, challenged an AI claim, caught a hallucination. The conversation extended their thinking.
What to ask students to submit
1
The conversation transcript
The full, unedited exchange — exported, copied, or screenshotted. Make clear that they should not clean it up or remove dead ends; those are often the most revealing moments.
Tip: Specify the format. Most LLMs allow conversation export. If you’re accepting screenshots, require a full capture, not excerpts.
2
A brief annotation (250–400 words)
The annotation is the real assessment. Ask students to identify 3–4 specific moments in the conversation and describe: what they accepted and why, what they rejected or pushed back on, what they caught the AI getting wrong, and how the exchange changed (or didn’t change) their thinking.
Tip: Require moments with specific timestamps or quotes. “I generally questioned AI’s claims” is not an annotation — “I flagged the citation in turn 4 because I’d read the original study and the summary was misleading” is.
3
The final work
Whatever the actual assignment is — the discussion post, the analysis, the argument. Read it alongside the conversation. A final that looks nothing like the conversation’s starting point is worth examining.
Tip: Ask students explicitly: “How did this conversation change your final response, if at all?” A student who says it didn’t change anything has told you something interesting too.
What a strong annotation looks like
Sample student annotation — three tagged moments
Pushed back
“Turn 2 — AI said social media harm is ‘well-established by research’”
I pushed back here because I’d read a meta-analysis for another class that found effect sizes much smaller than popular coverage suggests. I asked AI to name the specific studies it was drawing on, and several of them turned out to be the same three papers I’d already encountered. I didn’t include the “well-established” framing in my final post.
Caught an error
“Turn 4 — AI cited a 2021 Orben & Przybylski study”
I recognized the authors but not the specific paper. I searched for it and couldn’t find it. When I asked AI to confirm, it acknowledged the citation might be inaccurate. I left it out of my post entirely and found a real study by Orben (2019) that made a similar point.
Accepted
“Turn 6 — AI’s counterargument about benefits of online community for isolated teens”
This was a point I hadn’t considered. I know from my own experience that online communities helped me during a period of social isolation, but I hadn’t thought to include it. I incorporated a version of this into my third paragraph and made it personal.
Evaluating the conversation — what to look for
You are not grading the AI’s performance. You’re assessing the quality of the student’s engagement with it.
| Criterion | Strong | Weak |
|---|---|---|
| Starting point | Student brings a prior position, question, or attempt — not a blank prompt | Student opens with a pure generation request; AI does all the initial thinking |
| Follow-up questions | Student asks why, asks for evidence, digs into specific claims | Student only requests formatting changes or length adjustments |
| Critical response | Student pushes back on at least one AI claim, or expresses skepticism | Student accepts all AI output without apparent evaluation |
| Error detection | Student notices or attempts to verify at least one factual claim | No evidence of fact-checking; citations accepted without scrutiny |
| Annotation specificity | Annotation names specific turns, quotes, and explains the student’s reasoning | Annotation is generic; could apply to any AI conversation on any topic |
AI prompt — design a conversation-as-artifact assignment
Step 1 — adapt your existing assignment
I teach a course on [course topic]. I have an existing assignment that asks students to [describe the assignment].
I want to redesign this so that students are required to: (1) have a substantive AI conversation as part of their process, (2) submit the unedited conversation transcript, (3) annotate 3–4 specific moments in the conversation. Give me revised assignment instructions that include all three requirements, with a clear explanation to students of why the conversation is being submitted and how it will be evaluated.
Step 2 — write annotation instructions for students
Write annotation instructions I can give to students that: (1) require them to identify specific moments, not general impressions, (2) ask them to categorize each moment (accepted, rejected, pushed back, caught an error), (3) explain their reasoning for each choice. Make it clear that quantity of AI use is not being assessed — quality of engagement is.
Caveats
- Different AI tools have different export methods. ChatGPT, Claude, and Gemini all allow conversation export; others may require screenshots. Decide in advance what format you’ll accept and communicate it clearly.
- Some students will use this as an invitation to outsource all thinking to AI and annotate what the AI said about itself. Build in a requirement that the annotation references the student’s prior knowledge or course readings — not just the AI’s own self-assessments.
- This works best for assignments where the thinking process matters — discussion posts, argument development, analysis. For short responses or quizzes, the overhead isn’t worth it.
- A conversation log doesn’t capture everything. Students may have had useful AI interactions outside the submitted exchange. Make clear that you’re not requiring them to document all AI use — just the conversation that most shaped their final work.
Reflection questions
- Which of your current assignments would be most illuminated by seeing how a student used AI to think through it — not just what they finally produced?
- If you imagined your own ideal AI conversation for one of your assignments, what would it look like? What would distinguish a student who engaged well from one who didn’t?
- How would you explain to students that submitting this conversation is about developing their AI literacy — not catching them cheating?
- What would you do if a student submitted a conversation that showed genuinely sophisticated engagement but a weak final product — or the reverse?
