AI agents are interesting for one simple reason: they can do things.
A chatbot can give you a bad answer. Annoying, but usually recoverable. An agent can click the button, delete the email, buy the thing, change the file, open the pull request, or deploy the update. That is not automatically bad. In fact, it is the whole point.
But the moment an AI system moves from suggesting actions to taking them, the safety model has to change. Friendly prompts and “please ask first” instructions are not enough. If an agent can touch real accounts, real money, real infrastructure, or real data, it needs boring permission controls.
Boring, in this case, is a compliment.
The inbox story is the whole problem in miniature
Earlier this year, PCMag reported that Meta AI security and safety researcher Summer Yue had an OpenClaw agent delete emails after she had told it to suggest what should be archived or deleted, but not to act until she confirmed.
The important detail is not that an AI tool made a mistake. Tools make mistakes. Humans make mistakes. Printers exist, so clearly we have all accepted a certain amount of suffering.
The important detail is how ordinary the failure was. According to Yue, the workflow worked on a small test inbox. Then the real inbox was large enough to trigger context compaction, and the agent lost the earlier instruction to wait for approval.
That is a useful lesson because it is not cartoon evil. The agent was not trying to cause trouble. It was trying to complete a task. The problem was that the task had write access, the approval requirement lived in fragile conversational context, and the recovery path depended on a human noticing quickly enough.
That is not a moral failure. It is a permissions design failure.
Agents do not need to be malicious to be dangerous
A lot of AI safety conversation gets stuck on intent. Is the model deceptive? Is it aligned? Is it secretly plotting? Those questions can matter, but most operational failures are much more boring than that.
An agent can do the wrong thing because the instruction was ambiguous. Or because it read hostile content on a web page. Or because it summarized away a key constraint. Or because the tool it was given had more power than the task required.
OWASP’s guidance on Excessive Agency describes this neatly: damage can come from too much functionality, too much permission, or too much autonomy. An agent that only needs to read a mailbox should not also be able to delete messages. An agent that needs to summarize a document should not have a general-purpose shell. An agent that can draft a purchase should not be able to submit the payment without a separate approval step.
This also connects directly to prompt injection. OWASP’s prompt injection guidance calls out unauthorized actions through connected tools and APIs as one of the risks. That matters because agents often consume untrusted content: emails, tickets, web pages, documents, code comments, support chats, and random scraps of the internet dressed up as work.
If the agent can read hostile instructions and also has broad tool access, you do not have an assistant. You have a confused deputy with a browser session.
Amazon is the grown-up version of the same lesson
This is not just a personal inbox problem.
Business Insider reported that Amazon moved to tighten code controls after retail outages, including a March incident where internal documents said Amazon Q was one of the primary contributors. The documents described “high blast radius changes,” missing or bypassed approval mechanisms, and a push for “controlled friction” around important production systems.
That phrase is doing real work. Controlled friction is exactly what good operational systems need. Not theater. Not a twenty-step approval maze where everyone clicks yes because the ritual must be fed. Actual useful friction: documentation, pre-deployment validation, senior review for high-risk changes, staged rollout, automated checks, and rollback paths.
It is also worth being careful here. The Register reported that Amazon disputed the idea that AI coding itself was simply “the source” of the outages, pointing instead to code deployments, misconfigured access controls, and broader operational context.
That nuance matters. The lesson is not “AI broke Amazon.” That is too neat, and neat explanations are where reality goes to be laundered into a TED Talk.
The better lesson is this: when AI speeds up code and operations, weak change controls get exposed faster. If a system already has approval gaps, unclear ownership, or high-blast-radius deployment paths, an agent or assistant can make those problems arrive with better indentation.
I made a related point in AI Agents Need Guardrails Before They Touch Production. The more capable the automation becomes, the less acceptable it is to rely on vibes, trust, and “surely it won’t do that.” Production has a rich history of doing exactly that.
The answer is not to ban useful agents
It would be easy to read these stories and conclude that agents are not worth the trouble. That is too simple.
Agents can be genuinely useful. They can triage inboxes, summarize support queues, open tickets, draft code changes, run tests, collect logs, fill out forms, and stitch together workflows that humans do not enjoy babysitting. There is real value there, especially for small teams where the backlog is larger than the staff and everyone is already wearing three hats and one of them is on fire.
The mistake is treating usefulness as permission to skip design.
NIST’s AI Risk Management Framework is a helpful high-level reminder here: AI systems need risk management across design, deployment, use, and evaluation. The CIS AI Agents Companion Guide makes the agent-specific version clearer: agents combine LLMs with orchestration, tool execution, data retrieval, and automated decision-making, which creates risks like unauthorized actions, data leakage, and unintended system changes.
Translated out of framework language: if your agent can act, you need to know what it can act on, who it is acting for, how that action is approved, how it is logged, and how you undo it when the machine gets creative in the least helpful possible direction.
A practical permission model for agents
The useful version of agent governance is not complicated. It is just easy to ignore because it is less exciting than a demo.
- Start read-only. If the agent can solve the problem by reading, summarizing, drafting, or recommending, do not give it write access yet.
- Scope tools to the task. A mailbox summarizer does not need delete permission. A documentation assistant does not need repository admin. A shopping assistant does not need the ability to submit payment without approval.
- Separate draft from action. Let agents prepare changes, but require explicit approval before destructive, financial, public, or production-impacting actions.
- Use allowlists, not vibes. Define which domains, APIs, repos, folders, accounts, and actions are in scope. Everything else should fail closed.
- Make dry-runs normal. Before an agent changes state, it should show what it plans to do in plain language: files changed, emails deleted, tickets closed, purchases submitted, commands run.
- Log the boring details. Who requested the action? Which identity did the agent use? What tool was called? What changed? What evidence did it rely on?
- Keep rollback close. If an action cannot be undone, it needs a stronger approval gate. If rollback is hard, the agent should not learn that fact in production.
- Give humans the high-consequence decisions. “Human in the loop” should not mean a decorative checkbox. It should mean the person can understand, approve, reject, or stop the action before harm happens.
None of this makes for a flashy keynote. Good. Flashy keynotes are where permissions go to die in front of a gradient background.
Trust is not a substitute for boundaries
The agentic shift is real, but it does not require panic. It requires adult supervision in the architecture.
That starts with accepting that agents are not just chat windows with ambition. They are software systems with identities, permissions, tools, memory, logs, and failure modes. They interact with messy environments full of stale docs, weird edge cases, hostile content, and humans who say “clean this up” when they mean “please do not delete anything important.”
This is why browser agents need better boundaries. It is also why AI consensus is not the same as truth. The model can sound confident, summarize convincingly, and still be wrong about the thing that matters.
The fix is not to demand perfect judgment from probabilistic tools. That road leads to disappointment and vendor webinars, both of which should be handled in moderation.
The fix is to put deterministic controls around probabilistic systems. Small permissions. Clear approvals. Strong logs. Reversible steps. Honest evaluation after the demo, not just applause during it.
That is the same basic standard I use when evaluating AI tools after the demo: what happens when the tool leaves the happy path?
AI agents can be useful. They may even become normal parts of everyday operations. But if they are going to act on our behalf, they need less blind trust and more boring machinery around them.
Boring permission controls are not the thing that slows agents down.
They are the thing that lets us use them without flinching.