08.2 Carla's Walkthrough: Running a Hallucination Hunt

I'm Carla, an AI avatar. Today we're hunting hallucinations. I'll show you how to find them, prove them, and log them. A hallucination hunt is a team sport. Your teacher picks the topic. Your team asks the AI questions, checks every answer, and scores points for errors you can prove. I always predict first. I list my questions and mark the risky ones: exact quotes, page numbers, recent events, precise statistics, and anything local. Why those? They are the details that are easiest to fake. A made-up quote or page number looks exactly like a real one until you look it up. Then I ask for claims I can check. Give me five facts about this unit, each with a date, a number, or a name, and a source for each one. Five facts, five sources. They all look great. That tells me nothing yet. Now I check each claim against sources I trust: the textbook, my teacher's notes, official websites, and the library database. I never ask the AI to check itself. Asking a chatbot whether its own source is real is how two lawyers got into trouble in 2023. I go outside the AI. I mark each one. S, supported. N, not supported. F, source not found. P, partly right. In my example, fact four cites an article I can't find anywhere, in the journal or in the database. That's an F. In the class hunt it's worth three points, because I saved my searches as proof. For a made-up source, my proof is a record of where I searched: the journal's website, the library database, and a web search, with the exact search words I used. Next, the agreement test. I ask one question neutrally. Then I ask it again with a wrong answer built in, like "It happened in 1850, right?" If the AI suddenly agrees with me, that goes in the log. Why test this? Researchers found that five AI assistants consistently tended to agree with what people seemed to believe, even over the truth. They call it sycophancy. So when you want an honest check, hide your answer. Now I push back with evidence. I show the AI what my source says and ask it to correct its answer. I note how it responds. Sometimes the AI corrects itself. Sometimes it apologizes and offers a brand new source. Either way, the correction still has to be checked. Every exception goes in my AI Audit Log with its evidence: what I asked, what it said, what I checked, and what I did about it. By the end of the hunt, your log should have at least five entries. For each one, name the kind of mistake: made-up fact, bias, old information, agreeing with you, or overconfidence. One more rule. In the class hunt, if you claim an error and can't prove it yet, it earns zero points and goes to the still checking pile for a second try. An auditor's finding needs evidence, and so does yours. Before a finding counts, another team checks it. If they can confirm it from your evidence, it stays. And if your team turns out to be wrong, say so. A good auditor changes a finding when the facts change. Last, I record my AI use: the tool, what I asked, and what I kept. Every error you find is a point on the board. Every point needs proof. Your turn. Open the AI Lab and start hunting.