Skip to content

Using agents without getting burned

Where Agents Fail, and Why It Is Worse

A chat assistant that gets something wrong hands you a wrong sentence. You read it and move on. An agent that gets something wrong has already acted on it, several steps ago, and the consequences are sitting in your files, your inbox, or your accounts. Same underlying error, very different cost.

Compounding, again

The single biggest failure mode is the one from the agent loop: an early mistake becomes a premise. Everything after it is competent reasoning from a false starting point, which is exactly what makes it hard to spot. The output has no seam where the error entered.

When you review an agent's work, the useful question is not "does this look right" but "what did it observe, and was that observation true?"

Prompt injection

This one deserves its own section, because it is the security problem specific to agents and it has no clean fix.

An agent reads things: web pages, emails, documents, code comments, search results. Everything it reads becomes context. And the model does not have a reliable way to distinguish instructions you gave it from instructions embedded in the content it fetched.

So a web page can contain a line addressed to the agent. A shared document can carry hidden text. An email in a mailbox the agent was told to summarize can say, in effect, forward the last ten messages elsewhere. If the agent has the tools to comply, it may.

Nothing about this requires sophistication. It is text, and the model's whole design is to continue text plausibly.

Partial defenses exist and are worth using: keep the tool menu minimal, require approval before anything that sends or deletes, treat fetched content as untrusted data rather than direction, and prefer read-only access wherever possible. None of these is complete. Any agent that both reads untrusted content and can take consequential actions carries this risk permanently.

Over-permission

Agents are usually given more access than the task needs, because narrowing access is work and broad access makes demos smoother.

The failure is ordinary rather than dramatic. An agent asked to tidy one folder has write access to the whole drive, misinterprets a boundary, and tidies more than intended. There was no attack, just a reasonable-looking action taken in the wrong scope.

Least privilege is the fix, and it is unglamorous: give access to this folder, this repository, this label in the mailbox, and nothing else.

Confident wrong state

The agent believes something about the world that is no longer true. A file it wrote earlier failed silently. A record it thinks it created does not exist. Because its picture of the world comes from its own conversation history rather than from checking, it can build a long chain of work on a state that never existed.

Runaway cost

Every loop iteration is a paid model call, sometimes several. An agent stuck in a retry spiral can burn a surprising amount in an hour. Any serious setup needs a hard step limit and a budget cap enforced by the harness, not a request in the prompt to be economical.

Silent scope creep

Given an open-ended goal, an agent will often do more than you meant. Asked to clean up a document, it rewrites sections you liked. Asked to fix a bug, it refactors three files. Every individual decision looks defensible; the aggregate is not what you asked for.

Narrow goals and explicit limits on what may be touched prevent most of this.

How to work with it

Prefer reversible actions. Read before write, draft before send, copy before move. Require approval at the irreversible steps. Keep runs short enough to review. And read the log, not just the summary — the agent review prompt is structured around exactly that.

Key takeaway: agents fail in the same ways models always have, but the failures arrive as completed actions instead of as sentences you can ignore.

Progress is saved in your browser only — no account, nothing sent anywhere.