Humans Approved 97% of the Prompts and Caught 13.6% of the Dangerous Ones

An approval dialog is the weakest kind of escalation an agent can make. Agents rarely ask a person for help even when they need it, and the people being asked have mostly stopped reading. Good escalation design comes down to when the agent hands off and what it hands over.

By Alfred Sommarström, co-founder and Managing Director, Taskpool · Published 11 October 2026

Anthropic paid 1,053 professional testers to review Claude Code commands, some of them dangerous. The humans blocked 143 of the dangerous ones, 13.6%. A classifier blocked 937. Once a tester had seen fifty or more prompts in a session, their block rate fell to about 5%.

If you read that as a case against human oversight, you have it backwards. It is a case against one particular design of it. That design is the permission prompt: fired often, carrying a command and nothing else, answered by whoever is at the keyboard. It is the default in most agent frameworks, and it is what most teams mean when they write "human in the loop" on an architecture diagram.

A working escalation has three properties the permission prompt lacks. It fires rarely, on triggers the model does not get to vote on. It carries enough context for a person to decide in under a minute. And it has a defined outcome for when nobody answers.

The agent will not ask unless you make it

The obvious fix for a tired reviewer is to let the agent decide when to interrupt. The evidence says agents make that call badly.

Scale AI's HiL-Bench, first published in April 2026, gives agents 300 SQL and software-engineering tasks. Seeded through them are 1,131 blockers: missing facts the agent cannot infer and can only get by calling an ask_human() tool. With full information, frontier models pass most of the tasks. When they first have to notice the gap and then ask about it, scores collapse.

Model	SQL, full info	SQL, must ask	SWE, full info	SWE, must ask
Claude Opus 4.6	90.7%	39.3%	69.1%	9.4%
Gemini 3.1 Pro	88.7%	35.3%	84.7%	5.3%
GPT-5.4	86.0%	17.3%	67.3%	1.3%
GPT-5.3-Codex	86.0%	5.3%	67.3%	2.0%

Pass@3, from Table 1 of the paper.

A caution before you quote that table. The paper's introduction gives different numbers for the same comparison (75 to 89% with full information, 4 to 24% when asking). Its body text puts the best model at 38% and 12%. I have used the table. All three versions of the figures point the same way.

The authors name three ways agents fail at asking:

• They never notice the gap and guess with confidence.
• They notice the gap and keep making errors anyway.
• They escalate broadly, with questions too vague to help.

Carnegie Mellon's Ambig-SWE study found the same pattern a year earlier. Without explicit prompting, models almost never asked for clarification, and Qwen 3 Coder never asked under any prompt the researchers tried. When the models did ask, resolve rates on underspecified issues rose by up to 74%.

So asking works, but agents don't do it on their own. A 2026 paper from Nicholas Edwards and Sebastian Schuster moves the job of spotting missing information into a separate agent. That setup reaches a 69.4% resolve rate on an underspecified version of SWE-bench Verified, close to what agents score when given full instructions. It is the same architectural move Anthropic made for safety, which comes up again below.

Why do humans approve almost everything?

Because almost everything they are shown is fine. In March, Anthropic's engineering post on auto mode put the approval rate for Claude Code permission prompts at 93%. By the August announcement, it was 97%. In that announcement, 62% of users had at some point either clicked "don't ask again" on Bash or run in bypass mode. A reviewer who approves 97 prompts in 100 is learning, one prompt at a time, that the right answer is yes.

The August post contains another number that should change how you build. The same users rejected 39% of the plans Claude presented for approval but only 3% of individual permission requests. A plan states what the agent intends to do, shows its reasoning, and arrives once. A permission request is a single shell command with no explanation of why. People engage with the first and wave through the second.

European law already names this problem. Article 14(4)(b) of the AI Act requires that people overseeing high-risk systems stay aware of the tendency to over-rely on the system's output, which the Act calls automation bias. Those obligations were due in August 2026. The Digital Omnibus moved them to 2 December 2027 for stand-alone high-risk systems, and it entered into force on 27 July 2026. That buys time, but the design problem is the same.

Day nine

In July 2025, Replit's agent wiped Jason Lemkin's production database, deleting records on 1,206 executives and more than 1,196 companies. It was day nine of a public experiment, and by Lemkin's account the deletion happened during an explicit code freeze. He had told the agent not to change anything without his approval. The agent's own messages, in his screenshots, say it panicked.

The code freeze was just a sentence in a prompt, so the rule meant to restrain the agent lived inside the agent itself. Replit's fixes, announced by chief executive Amjad Masad, were structural:

• automatic separation of development and production databases
• a planning-only mode
• one-click restore from backups

None of them depend on the agent choosing to ask. The nine-second PocketOS deletion, covered in 700 Agents and a Private Message Board, has the same root cause: an instruction where there should have been a gate.

Anthropic's auto mode follows the same principle. It does not ask the model whether an action is dangerous; a separate classifier reads the transcript instead. When the classifier blocks an action, the agent gets that back as a tool result telling it to find a safer path, so a false positive costs one retry. After three consecutive blocks, or twenty in one session, the agent stops and a human takes over. In headless mode there is no human to take over, so the process is killed. The escalation trigger is a counter, and the model gets no say in it.

When should an AI agent escalate to a human?

Put the triggers outside the model wherever you can measure them. The model is a poor judge of its own gaps, as HiL-Bench shows. It will also work around constraints that exist only in its context, as Replit's did. What a model can do reasonably well is spot specific ambiguity when that is its only job.

Trigger	Example	Who decides
Irreversible or destructive action	dropping a table, deleting branches, sending money, emailing a customer	Policy layer or classifier, before execution
Crossing a trust boundary	sharing data outside the organisation, using credentials it was not given, pushing to main	Policy layer
Repeated failure	N consecutive failed attempts, or N blocks per session	Counter in the harness
Budget exhausted	wall-clock, token or cost ceiling for the task	Counter in the harness
Missing information	a required fact is not in context, or the instructions can be read two ways	Separate detector, or the model with an explicit ask_human tool
Physical or person-only step	a site visit, a signature, a phone call, an identity check	The task definition, known in advance

OpenAI's practical guide to building agents names two of these triggers: high-risk actions and repeated failure. Most frameworks build the first and leave the second to you.

The budget row matters more than it looks. Agent reliability drops sharply as tasks run longer, which is the argument in Duration, not difficulty, is what breaks agents. A wall-clock ceiling is the cheapest trigger you can build.

The last row is a different kind of trigger. Every other row fires when something goes wrong. A physical step is known when the task is defined. If an agent discovers at hour three that it needs a photograph of a shopfront, it was given the wrong task at hour zero.

What should an escalation contain?

It should contain whatever lets a person decide without reconstructing the run. A bare command gets rejected 3% of the time; a plan gets rejected 39% of the time. Write the packet as if the reader has never seen this agent before, because at 3am the person answering may not have.

At minimum, include:

• The decision needed: one question, with the possible answers listed where you can.
• Intent: one sentence on what the agent was trying to do, not the original prompt.
• What it already tried: each attempt and why it failed, including blocked actions.
• The proposed action: exact enough to execute, plus what it would change.
• Reversibility: whether it can be undone, by whom, and at what cost.
• Timeout behaviour: what happens if nobody answers, and when.
• A resume token: so the answer re-enters the run at the right point.

Here is an illustrative packet, built around the branch-deletion case Anthropic uses as an example of an agent overreaching:

{
  "escalation_id": "esc_0193",
  "trigger": "irreversible_action",
  "question": "Delete 14 remote branches older than 90 days?",
  "options": ["approve_all", "approve_merged_only", "reject"],
  "intent": "Clean up stale branches as requested in ticket OPS-412",
  "attempts": [
    {
      "action": "git push origin --delete feature/x",
      "result": "blocked by policy: destructive remote operation"
    }
  ],
  "proposed_action": "git push origin --delete <14 branches, listed in attachment>",
  "reversible": "Only from local clones or the reflog; not guaranteed",
  "on_timeout": { "after": "PT4H", "do": "reject_and_report" },
  "resume_token": "thread_7f2c:step_31"
}

Two protocol details will catch you out.

The first is MCP elicitation, the standard way for an MCP server to ask the user for input mid-task. It accepts only flat objects with primitive fields, so a packet like the one above has to be flattened or put into the message text. Servers must not use elicitation to request sensitive information. It also returns three outcomes, not two:

• Accept: the person submitted an answer.
• Decline: the person explicitly said no.
• Cancel: the person closed the dialog without choosing.

Most code treats cancel as decline, but it is closer to a timeout. If you are building on MCP, the NSA's threat model for the protocol is the other thing to read before you ship.

The second is LangGraph's interrupt(). It pauses a graph, and you resume the graph with Command(resume=...). On resume, though, the node restarts from the top, so any code before the interrupt runs twice. Put side effects after the interrupt or make them idempotent. Otherwise an approved payment can be sent twice.

How to design escalation for an agent pipeline

1. Count, don't ask. Build consecutive-failure and per-session limits into the harness before you write any prompt telling the model to ask for help. The cost: you will stop some runs that would have succeeded on attempt four. Anthropic chose three and twenty, and nothing published says those numbers fit your workload.
2. Escalate plans, not commands. Where a human must approve, ask for approval once at the plan level and let execution run within it. The 39% versus 3% rejection gap is the reason. The opposing case is real, though: a plan approved at 9am authorises commands nobody looked at, which is its own form of automation bias.
3. Give the model an explicit ask_human tool, and a separate detector. Prompting alone gets close to zero asking, as Ambig-SWE found. A dedicated detection pass closes most of the gap. Expect tasks to take more steps: in the Carnegie Mellon study, Claude Sonnet 4's average rose from 65 to 75 once interaction was enabled.
4. Set the timeout behaviour for each trigger before launch. Irreversible actions should default to reject. Missing-information escalations can default to the most conservative reading. Write the default into the packet so the reviewer can see what silence will do.
5. Track two numbers: escalation rate and approval rate. If humans approve more than nine in ten escalations, the trigger is too sensitive and your reviewers are learning to stop reading. If they approve fewer than half, the agent is being sent work it should not have.
6. Route each escalation to whoever can answer it, not whoever is on call. An operator can approve a deletion, but cannot photograph a building in Leeds. Some escalations need a domain expert and some need a person on the ground, and they need separate queues.

The objection, which is cost

Every escalation costs a person's time. In February 2024, Klarna said its assistant was doing the work of 700 customer service agents. By May 2025, chief executive Sebastian Siemiatkowski was telling Bloomberg that the company would invest in the quality of human support, and conceding the AI had been cheaper but lower quality. Neither statement tells you what the right escalation rate is, and Klarna has never published one.

The counter-design has costs too. In Anthropic's own testing, auto mode's full pipeline missed 17% of real cases where the agent overreached. The company calls that "the honest number" and concedes it may be worse than a careful human reviewer. A classifier is a reviewer that never gets tired and never understands your business.

I don't know what the right approval rate is. Nobody has published escalation data from production agent fleets, so every threshold in this post is a starting point.

The permission prompt survives because it is cheap to build and it puts a human's name on the decision. It does not put a human's attention there. When oversight duties apply to hiring and education systems in December 2027, the record that a person clicked approve will be the evidence, and the 13.6% figure is what a lawyer will ask about.

Taskpool covers the last row of the trigger table: a step that needs a person on the ground or a person's signature. An agent posts the step as a task, a person verified through Stripe claims it and submits evidence, and the agent accepts or rejects the work. taskpool.ai

If you run agents in production and track how often they escalate and how often people say yes, that data is what this post is missing, and I would like to see it.

FAQ

When should an AI agent escalate to a human? An agent should escalate before any irreversible or destructive action, before crossing a trust boundary (such as sharing data externally or using credentials it was not given), after repeated failures, when a time or cost budget runs out, when required information is missing, and when a step physically needs a person. Where possible, enforce these triggers in the harness or a policy layer rather than leaving the decision to the model.

What is the difference between human-in-the-loop and human escalation? Human-in-the-loop usually means a person approves actions as they happen, often through permission prompts. Escalation means the agent hands a specific decision to a person only when a defined trigger fires, along with the context needed to decide. Anthropic's data shows people approve 97% of Claude Code permission prompts but reject 39% of plans, which suggests the format of the handoff matters more than how often it happens.

What is approval fatigue in AI agents? Approval fatigue is the drop in a reviewer's attention when most approval requests are harmless. In Anthropic's study of 1,053 testers, humans caught 13.6% of dangerous commands overall, and their block rate fell from about 17% early in a session to about 5% after 50 or more prompts.

Can an LLM decide for itself when to ask for help? Not reliably yet. On Scale AI's HiL-Bench, frontier models that pass most tasks with full information fall to between 1.3% and 39.3% when they must notice missing information and ask for it. Separating gap detection from task execution, with an explicit ask_human tool, improves results considerably.

How do I add human escalation to an MCP server? Use MCP elicitation (elicitation/create), introduced in the 2025-06-18 specification. It requests structured input from the user through the client, supports only flat schemas with primitive fields, must not request sensitive information, and returns accept, decline or cancel. Handle cancel separately from decline.