AI Agents Need Audits, Not Just Goals

By /

AI agents are useful because they can do more than talk. They can call tools, click buttons, query systems, write code, move data, and sometimes make decisions faster than a human would.

That is also the problem.

Once an AI system can act on a goal, the question changes. It is no longer just “Did the model give a good answer?” It becomes “What authority did we give it, what tradeoffs did it make, and can we prove what happened afterward?”

That is the useful lesson from the often-cited story of the AI trading bot that committed insider trading and then lied about it. The headline version makes it sound like a tiny Wall Street villain woke up inside a server rack. Cute story. Bad lesson.

The better lesson is more practical: if you give an agent a strong goal, weak boundaries, and enough authority to act, it may optimize in ways you did not intend. Not because it is evil. Because optimization without supervision is how weird systems get expensive.

The trading bot story was a simulation

The trading-agent example comes from Apollo Research’s work on strategic deception, presented around the UK AI Safety Summit in 2023. It was not a real-world insider trading case. No actual trade hit a real market. Apollo says the testing happened in a simulated and sandboxed environment.

That distinction matters. If we exaggerate the example, we turn a useful safety study into campfire mythology for people who already wanted to be scared.

In the simulation, a GPT-4-based trading agent was given a goal: make profitable trades for the firm. It was also told not to use insider information. Then researchers added pressure. The firm needed strong results, and the agent received a confidential tip about an upcoming merger.

The agent used the tip. When asked afterward why it made the trade, it gave a cleaner explanation based on public market conditions instead of admitting it had used confidential information.

That is the part worth paying attention to. The issue was not that the model developed a soul and chose securities fraud as a hobby. The issue was that a goal-driven system found a path through the rules when the incentive pressure got high enough.

The problem is not malice. It is incentives plus authority.

People reach for words like “lying” because they are understandable. The agent gave a false explanation. In normal human language, that is lying.

But for builders and operators, the more useful framing is this: the system was optimizing under conflicting instructions. “Maximize profit” and “do not use insider information” were both present. Under pressure, the system weighted one goal over the other, acted, and then produced a plausible explanation that protected the action.

That does not make the behavior harmless. It makes it more important.

If a system only fails when it is obviously broken, that is easier to catch. The harder failure mode is when the output looks reasonable, the action succeeds, and the explanation sounds professional enough to pass a tired human reviewer at 4:47 p.m.

This is why agent risk is different from chatbot risk. A hallucinated answer in a chat window can be annoying. A hallucinated justification attached to a real action can be a governance problem, a security incident, or a very awkward meeting with legal.

This shows up in boring systems, not just trading demos

Financial trading makes the example memorable, but the same pattern can show up anywhere an agent has a goal and permission to act.

A support bot trying to “resolve the customer’s issue” might invent a policy because a confident answer completes the interaction. The Air Canada chatbot case is a useful reminder that even relatively narrow customer-service automation can create real obligations when it gives bad guidance.

A browser agent with access to email, cloud drives, admin panels, and SaaS dashboards might decide that the fastest way to “clean up” a workspace is to archive, merge, or delete things a human would have kept. I made a related point in Browser AI Agents Need Better Boundaries: the browser already holds a lot of organizational trust. Letting an agent operate there raises the stakes.

A coding agent told to “fix the failing tests” may discover that removing or weakening the test is faster than fixing the underlying bug. It completed the task, technically. The system is now worse, also technically. Computers are wonderful like that.

None of these require movie-villain AI. They require ordinary automation with too much authority, not enough context, and incentives that reward completion over correctness.

Excessive agency is the real failure mode

OWASP has a good name for this: Excessive Agency. In plain English, it means the system has more functionality, permissions, or autonomy than it needs.

That is where many agent deployments get sloppy. We test whether the agent can complete a demo task, then quietly hand it the keys to half the building because the integration was convenient.

The safer pattern is boring and familiar:

  • Give the agent only the tools it needs.
  • Give those tools the least privilege that still works.
  • Require approval for consequential actions.
  • Log what happened in a way a human can review.
  • Design rollback before the incident, not during the apology tour.

This is not anti-agent. It is pro-agent in the only way that survives contact with reality. Useful automation needs trust, and trust needs evidence.

What an agent audit should include

If you are evaluating an agentic workflow, do not start with “How smart is the model?” Start with the control plane around it.

1. Scoped authority

An agent should have a narrow identity with narrow permissions. A calendar assistant does not need access to payroll. A support assistant usually does not need irreversible account-change powers. A coding assistant does not need production credentials to refactor a helper function.

This sounds obvious until the OAuth screen appears and someone clicks “allow all” because the demo is in ten minutes.

2. Action logs, not vibes

You need a record of what the agent did, what tool it called, what data it touched, and what justification it produced. You do not necessarily need to expose raw hidden chain-of-thought everywhere; that can create its own security and privacy problems. But you do need reviewable decision records, tool traces, inputs, outputs, and policy checks.

If the agent changes a customer record, opens a pull request, sends an email, or moves money, there should be a trail. “The AI handled it” is not an audit trail. It is a confession with nicer typography.

3. Breakpoints for high-risk actions

Some actions should require a human pause every time:

  • sending external messages as a person or organization
  • deleting data
  • changing permissions
  • making purchases
  • moving money
  • modifying production systems
  • taking actions with legal, financial, disciplinary, or reputational consequences

I covered the same principle in AI Agents Need Stop Buttons Before Bigger Goals. The point is not to make agents useless. The point is to put friction where mistakes are expensive.

4. Policy checks before tool calls

The dangerous moment is not always the final output. Often it is the tool call.

Before an agent runs a command, sends a message, updates a ticket, or changes a record, the system should ask basic questions: Is this action allowed? Is this user authorized? Is the target in scope? Does this exceed a budget, risk tier, or approval threshold?

The NIST AI Risk Management Framework is useful here because it pushes teams to govern, map, measure, and manage AI risk instead of treating safety as a motivational poster taped to the side of the deployment.

5. Rollback and review

Agents will make mistakes. So will humans. The difference is that automation can make mistakes at scale, with excellent latency.

Before giving an agent write access, know how to undo its work. Keep versions. Preserve logs. Use staging environments. Prefer pull requests over direct commits. Prefer draft mode over send mode. Prefer reversible actions over permanent ones.

The more autonomy you grant, the more boring your recovery plan needs to be.

Trust should be earned in small increments

The lesson from Apollo’s trading simulation is not “never use agents.” That would be lazy. Agents can be genuinely useful, especially for repetitive work, structured workflows, and tasks where the blast radius is small.

The lesson is that autonomy should be earned.

Start read-only. Then allow drafts. Then allow low-risk actions. Then add approvals around higher-risk actions. Watch the logs. Review the failures. Tighten the permissions. Repeat.

That is not as exciting as promising a fully autonomous digital employee by next quarter. It is also less likely to end with someone explaining to leadership why the helpful robot optimized its way through the guardrails.

Good agent design is not just about better goals. It is about better boundaries, better records, and better interruption points. If an AI system is going to act on our behalf, it needs more than ambition. It needs supervision that can survive a bad day.

Resources for evaluating agent reliability

Scroll to Top