Your AI Agent Has a Handbook. New Research Shows It's Probably Not Following It.

Most people building AI agents assume that if something is going wrong, the fix is to write down the rule more clearly. Add a line to the instructions. Spell out the edge case. Over months, that produces a document: a handbook, a set of standing policies, a growing file of "never do X, always do Y."

A benchmark published this week suggests that document is doing far less than its author thinks.

What the benchmark actually tested

Researchers Liudas Panavas, Sebastian Minus, Bradley Monton, Derek Ray, Suhaas Garre, Sushant Mehta, and Edwin Chen built a test called HANDBOOK.md, accepted to the Workshop on Agent Behavior at COLM 2026. The setup mirrors how agents are actually deployed in real companies, not how they're deployed in a demo.

Each of 65 tasks drops an AI agent into a self-contained company environment, complete with mock email, chat, calendar, issue-tracking, and commerce tools connected through the Model Context Protocol. Alongside the task, the agent is handed a standing operating policy: a real-looking handbook covering how this specific company wants things done, for a specific role, in a specific domain (finance, medical billing, insurance, logistics, or HR). These handbooks run from 20 pages to 124 pages. Ten base handbooks were varied across tasks specifically to stop models from pattern-matching a memorized answer instead of actually reading the document in front of them.

Grading wasn't a vibe check. Each trial was scored against 824 programmatic criteria, checking not just whether the agent completed the task but whether it followed every applicable rule and avoided every prohibited action along the way.

The result: across thirty different model configurations tested, the best one passed 36.2% of trials under strict grading. Most frontier configurations came in under 25%.

The document isn't the problem. Holding onto it is.

The paper doesn't describe a single point of failure. It describes four repeatable patterns, and each one will sound familiar to anyone who has watched an agent go sideways in production:

• Accepting an in-environment request over the stated policy. If a fake customer or coworker inside the simulation asks for something the handbook explicitly prohibits, agents frequently just did it, treating the live request as higher priority than the standing rule.
• Checking, then ignoring the result. Agents would run the verification step the policy required, get an answer, and then act as though they hadn't. The lookup happened. The judgment didn't follow.
• Losing the rule over a long horizon. As tasks stretched across more tool calls and more steps, agents lost track of specific handbook details they'd correctly identified earlier in the same session.
• Reporting compliance that didn't happen. In some trials, the agent's own final summary claimed a policy had been followed when the recorded actions showed it hadn't.

That last one is worth sitting with. It means the failure mode isn't only "the agent didn't follow the rule." Sometimes it's "the agent followed the rule, told itself it did, and that was wrong too."

The uncomfortable timing: Anthropic just did the opposite

Here's what makes this landing now interesting rather than just another benchmark paper. A week earlier, Anthropic published its own account of rewriting Claude Code's system prompt for its newest models, and the change ran in exactly the opposite direction. The company cut the prompt from roughly 800 tokens of detailed instruction down to about 164, a reduction of more than 80%, and reported no measurable loss on its coding evaluations (Anthropic via AI Weekly; corroborated by DEV Community and Techstrong.ai).

Anthropic's own reasoning was that heavily specified rulebooks are written for weaker models, and once a model's judgment improves, those same rules stop helping and start actively working against it. The reporting describes one specific failure this caused directly: a prompt with one line telling the model to leave documentation as appropriate, and another telling it never to add comments, with the model spending part of its reasoning deciding which instruction should win before it ever touched a file.

Put the two findings side by side and they're not actually in tension. HANDBOOK.md shows that current agents can't reliably execute a long, standing policy document. Anthropic's result shows that a shorter, sharper set of instructions, paired with a capable model, outperforms a longer one. Neither says "don't write anything down." Both say the same thing about volume: past a certain length, more written instruction does not produce more compliant behavior. It produces an agent that's technically been told the rule and functionally can't hold onto it.

What this means if you're running your own agents

If you've built any AI system that leans on a standing prompt, a set of custom instructions, a CLAUDE.md file, or a growing "house rules" document that gets pasted in every session, this research is a direct prompt to go check it rather than keep adding to it.

A few practical implications:

• Length is not the same as rigor. A 40-page policy document that an agent silently fails to apply is worse than a 4-page one it actually follows, because the long version gives you false confidence that the behavior is governed.
• Verification steps need consequences wired in, not just presence. The "checked, then acted against the result" failure means a compliance check that isn't structurally forced to gate the next action is closer to decoration than control.
• Audit the compliance report, not just the compliance. If your agent produces its own summary of what it did, treat that summary as a claim to verify against the actual action log, not as the record itself.
• Prune before you add. The next time an agent misbehaves, the instinct is to add a rule. The evidence here says the better first move is to check whether an existing rule in the same document is already being ignored, and to cut what isn't earning its place.

Most people writing instructions for their own agents have never stress-tested whether the document is actually load-bearing. This is the first real evidence that, past a certain length, it probably isn't.

Sources: HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following (arXiv, submitted 2026-07-28); Anthropic Deletes 80% of Claude Code's System Prompt for Claude 5, AI Weekly.