There is a simple rule that makes AI tools much safer to use: fluent output is not evidence.
That sounds obvious until the paragraph looks polished, the bullet points look official, and the answer appears inside a tool your organization already pays for. At that point, the temptation is real. The machine gave you something clean. It used the right tone. It sounded like it had done the reading.
Maybe it did. Maybe it didn’t.
That difference matters, especially when the output is about real people.
A recent West Midlands Police incident is a useful example because it is not a science-fiction AI disaster. Nobody needed a rogue superintelligence. Nobody needed a glowing red robot eye. The failure was much more ordinary: an AI tool produced false information, humans relied on it too much, and the mistake helped support a consequential decision.
According to reporting from The Guardian, former West Midlands Police chief constable Craig Guildford apologized to MPs after incorrect evidence about Maccabi Tel Aviv supporters was linked to the use of Microsoft Copilot. The false detail involved a supposed match between Maccabi Tel Aviv and West Ham. That match had not happened.
The UK Parliament’s Home Affairs Committee later said AI was used in a way that reinforced false narratives in the fan-ban report, and criticized the reliance on inaccurate and unverified information. The committee’s materials are worth reading because the lesson is bigger than one police force, one football match, or one vendor.
The problem is not that AI tools are useless. The problem is that they are very good at producing text that feels finished before the work underneath it is finished.
The dangerous part is not the mistake. It is the confidence.
People make mistakes in reports all the time. A junior analyst can misread a source. A rushed team can confuse two events. A bad search result can send someone down the wrong path. None of that is new.
What generative AI changes is the packaging.
A weak claim can arrive looking like a clean memo. A guess can be wrapped in professional language. A fabricated detail can sit comfortably next to real names, real dates, and real institutions. The output does not look nervous. It does not leave coffee stains on the file. It does not pause and say, “I’m not actually sure this happened.”
That is why the West Midlands Police case is such a useful warning. The reported failure was not just “AI hallucinated a football match.” It was that the false detail moved through a human process far enough to matter.
That is where automation bias enters the room. Automation bias is the human tendency to trust automated output too much, especially when it appears authoritative or comes from a familiar system. Put another way: once the computer says it neatly, people are more likely to treat it as checked.
Anyone who has worked in a busy organization should feel a little uncomfortable here. This is not only a policing problem. Schools, hospitals, courts, businesses, IT departments, and public agencies all run on too much work and not enough time. A tool that can turn a rough request into a clean document is going to be attractive. Of course it is.
The answer is not to pretend that pressure does not exist. The answer is to build a workflow that assumes pressure exists and still keeps unverified claims from becoming decisions.
AI can help with the work. It cannot become the record.
A large language model can be useful for organizing notes, summarizing a long document, drafting a first pass, or turning messy thinking into a structure someone can review. That is real value.
But it is not the same thing as a source of record.
When you ask an AI tool for a historical event, a legal precedent, a policy requirement, a medical fact, or a quote, you are not automatically querying a verified database. You are asking a system to generate a likely answer based on patterns, context, tool access, and whatever grounding it may or may not have at that moment.
Sometimes that answer is right. Sometimes it is partly right. Sometimes it is confidently wrong in a way that fits the shape of the question a little too well.
That last category is the dangerous one.
If someone asks an AI tool to help build a case for a decision, the tool may produce material that supports the direction of the prompt. That does not make the user malicious. It does mean the workflow needs to protect against automated confirmation bias.
This is the same basic issue I wrote about in AI Tools Repeat Consensus. That Is Not the Same as Truth. Pattern-matching can be useful. Consensus can be informative. Neither one replaces verification.
A simple verification loop for high-stakes AI use
The fix is not glamorous. Good. Glamour is usually where these systems start lying to us in nicer fonts.
For high-stakes work, the verification loop should be boring, explicit, and documented.
1. Separate drafting from evidence
Let AI help draft. Let it organize. Let it suggest questions you forgot to ask.
But every factual claim that matters should point to a source outside the model. If the claim affects a person’s rights, money, employment, safety, discipline, access, or reputation, it needs receipts.
A clean paragraph is not a receipt.
2. Trace the claim back to the original record
If the AI says an event happened, find the event. If it names a report, open the report. If it quotes a policy, read the policy. If it cites a court case, verify the case exists and says what the output claims it says.
This step sounds insultingly basic until you remember how many AI failures have involved fake citations, fake cases, fake policies, or real sources used in the wrong way.
3. Search against your own conclusion
If the prompt was “help me justify this decision,” someone should also search for reasons the decision might be wrong.
That does not mean every decision is bad. It means the verification process should not only collect evidence that flatters the original request. In the West Midlands Police case, the Home Affairs Committee’s criticism of confirmation bias is the part every organization should underline.
4. Make a second human responsible for review
The person who uses AI to draft the case should not be the only person who verifies the case.
That is not bureaucracy for its own sake. It is a guardrail against normal human psychology. We all get attached to the thing we are building, especially when the tool makes it look coherent. A second reviewer is less likely to confuse “well-written” with “well-supported.”
This is also why I am skeptical of using AI detectors as verdict machines in schools. As I argued in AI Detectors Are a Weak Signal, Not a Verdict, tools can inform a process, but they should not replace the process.
5. Keep the audit trail
Save the prompt, the output, the sources checked, the contradictions found, and the human sign-off.
If that sounds annoying, that is the point. Friction is not always waste. In high-stakes environments, friction is how you slow a bad claim down before it becomes somebody else’s problem.
NIST’s AI Risk Management Framework and its Generative AI Profile are more formal than most small teams need day to day, but the direction is right: govern the use, map the risk, measure the failure modes, and manage what happens when the tool is wrong.
The higher the stakes, the more boring the process should be
Not every AI mistake deserves a committee hearing. If a chatbot invents a fake brand of dishwasher soap in a brainstorming session, fine. Laugh, fix it, move on.
But when the output supports a police decision, a school discipline case, a hiring decision, a medical recommendation, a legal filing, or a security action, the standard changes.
At that point, the question is not “did the AI sound plausible?”
The question is: who checked the claim, where is the source, what contradictory evidence was considered, and who is accountable for the final decision?
That same mindset applies to more autonomous systems too. In AI Agents Need Audits, Not Just Goals, the point was that giving an AI system a goal is not enough. You also need logs, limits, review, and a way to reconstruct what happened.
For ordinary AI drafting, the equivalent is simple: do not let the tool be the only witness.
Keep the tool. Add the receipts.
The boring answer is probably the right one.
Use AI where it helps. Let it make the first draft less painful. Let it summarize material, suggest structure, and surface questions. Those are good uses when someone understands the limits.
But do not confuse a polished answer with a checked answer.
The West Midlands Police incident is not a reason to ban every AI tool from serious work. It is a reason to stop pretending serious work can skip verification because the software sounds confident.
AI can draft the report.
Humans still have to bring the receipts.