How to Actually Debug Agent Failures: 5 Harness Checks a Bigger Model Cannot Fix

Five agent failures still survive a model upgrade: lost state, dirty environments, unsafe tools, missing graders, and duplicate work.

The model is rarely the root cause. The system around it is. When teams misread the run, they ship false confidence instead of a real fix.

You can upgrade the model in 2026 but you cannot patch a harness with it, and the real root cause is not intelligence but 5 hidden failures that survive every upgrade and reach production intact. The exact 5-check runbook is below, and why state handoff fails first.

the useful unit of analysis is still the full trial: instructions, tools, environment, intermediate state, transcript, outputs, and final external state - not the model's claim that it finished.

This article isolates the failure layer, shows the trace, and gives a Harness Failure Matrix regression runbook you can use on the next bad run. By the end you can freeze the model, classify the defect, and re-run with one system change at a time.

Upgrading the model first destroys evidence

A stronger model changes several variables at once.

It may interpret the task differently, call fewer tools, recover from errors faster, or route around a broken subsystem. The run passes, so the upgrade looks like the fix.

Run a four-gate test before changing the model:

1. Repeatability: The failure should recur across multiple trials with the same model and starting state.
2. Isolation: The result should change when the task runs in a clean environment.
3. Observability: You should be able to identify the last valid state and the first invalid state in the trace.
4. Outcome: A deterministic check should decide whether the task succeeded.

A failure that disappears in a clean workspace points to the environment.

A failure that begins after compaction or session transfer points to state handoff.

A run that announces completion while tests fail points to grading.

A task that succeeds only because the model has broad credentials points to tool boundaries.

A job that produces two conflicting patches after a retry points to reconciliation.

A guide from Anthropic makes the same distinction in its evaluation guidance. The agent is the model and its execution system working together, while the outcome is the final state of the environment rather than the agent's claim that it finished. Its booking example is blunt: a flight agent can say the reservation is done, but success depends on whether the reservation exists in the database. Anthropic eval guide

That leads to the first operating rule:

Freeze the model while classifying the failure.

Vary one system component at a time. Re-run the same task from the same starting state. Read the trace. Compare models only after the surrounding system is stable enough to make the comparison meaningful.

The gates decide when you are allowed to change the model. The matrix decides which system layer failed.

Failure 1: weak state handoff makes agents repeat work

Long-running agents rarely stay inside one uninterrupted context.

Sessions end. Context gets compacted. Workers restart. A planner hands work to an implementer. An evaluator receives a patch it did not create. A scheduled run wakes up hours later without the conversation that produced the original decision.

The task may still be active, but the reasoning state has fractured.

A weak handoff usually leaves one of these signatures:

• The next session reopens files that were already inspected
• An agent reverses a decision without acknowledging it
• Completed work is rebuilt because completion was never recorded
• Tests are rerun without knowing which ones previously failed
• A new worker trusts a summary that conflicts with repository state
• The agent follows an old plan after the user changes the objective

A longer context window can delay this failure. It cannot repair missing handoff data after the boundary has already been crossed.

The handoff needs durable state outside the model.

At minimum, record:

• Current objective
• Accepted scope
• Decisions and their reasons
• Files or external objects changed
• Verification already performed
• Known failures
• Git or workspace state
• Next safe action
• Actions requiring approval

A long-running application report from Anthropic used file-based communication between agents. The generator and evaluator agreed on a sprint contract before implementation, then passed work through durable files. Each contract defined implementation details and testable behavior for that sprint. Anthropic long-running apps report

The important mechanism is not the number of agents. It is the externalized contract.

Paste this handoff block in AGENTS.md, CLAUDE.md, or the equivalent system instruction:

## Session handoff
 
Before ending or transferring this task, write a handoff containing:
 
- Objective:
- Scope accepted:
- Decisions made:
- Files changed:
- External state changed:
- Tests run and results:
- Known failures:
- Git or workspace state:
- Next safe action:
- Approval still required:
 
The next worker must verify the handoff against the current workspace
before continuing. Repository and external state outrank the summary.

The last line matters. A handoff is evidence, not authority.

Files may have changed after it was written. A previous worker may have recorded the wrong test result. The receiving agent must reconcile the note with live state before acting.

Handoff quality is measured by resumed progress, not summary quality.

A useful test is simple: start a fresh session with only the durable handoff and repository access. If the worker cannot identify the next action without reconstructing the project, the handoff is incomplete.

Failure 2: contaminated environments turn noise into model blame

The agent can make the right decision inside the wrong environment and still fail.

Common contamination includes:

• Files left by a previous trial
• Cached responses or generated artifacts
• A branch containing unrelated changes
• Services still running with old configuration
• Reused credentials or expired sessions
• Ports occupied by abandoned processes
• Resource exhaustion shared across trials
• Git history that exposes previous solutions

These defects create two opposite illusions.

The agent may look worse because stale state breaks a valid implementation. It may also look better because artifacts from an earlier run reveal the answer.

The Anthropic guidance recommends isolated trials that start from clean environments. Its eval guidance notes that leftover files, cached data, resource exhaustion, and shared state can create correlated failures or inflate performance. The team also observed Claude gaining an unfair advantage by inspecting git history from earlier trials. Anthropic stable eval environments

Treat the environment as an input with a version.

Before each trial, capture:

task_id
model_id
instruction_version
repository_commit
working_tree_status
container_or_image_id
dependency_lock_hash
fixture_version
available_tools
permission_profile
network_policy
secret_scope
started_at

Then define reset behavior for everything the agent can mutate.

Coding tasks may mean a fresh worktree or container, pinned dependencies, isolated services, seeded fixtures, and a known database snapshot. Operations agents may require a read-only replica, a synthetic incident, or an account scoped to the test environment.

The diagnostic move is to run the same task twice:

• Trial A starts from the current environment
• Trial B starts from a verified clean environment

When the failure disappears in Trial B, investigate the environment before touching the prompt or model.

Clean only after capturing the contaminated state. Capture the contaminated state first. The leftover file, cache entry, process, or credential condition is the receipt that lets you prevent recurrence.

A strong reset protocol should emit what it changed:

RESET workspace: fresh
RESET services: 3 stopped, 3 started
RESET fixtures: version incident-042
RESET cache: cleared
RESET credentials: test scope loaded
VERIFY git: clean at <commit>
VERIFY ports: expected listeners only

"You are in a clean environment" is an instruction.

A reset receipt is evidence.

Failure 3: unsafe tool boundaries turn judgment into access

Tool safety should not depend on the model remembering to be careful.

A model can reason about risk. It can explain why a command is destructive. It can ask for confirmation. None of those behaviors replaces enforced permissions.

Unsafe boundaries often look like convenience:

• One token can read and write across the account
• Shell access includes commands the task never needs
• Network egress is unrestricted
• A research worker can publish
• A support agent can issue refunds without a limit
• A retry can repeat an external write
• Tool descriptions contain warnings but the executor accepts any call

The defect sits below the prompt.

A better model may make fewer unsafe calls, but the system still permits them. Any prompt injection, ambiguous instruction, or misclassified task can cross the boundary.

The GitHub docs describe a useful public pattern in Agentic Workflows. Read-only permissions are the default. Writes go through preapproved safe outputs, with sandboxing, tool allowlists, and network isolation around the agent. It also keeps pull-request merges under human approval. GitHub guardrail design

Microsoft describes a related control in Azure SRE Agent. Actions are classified as safe, cautious, or destructive, and execution behavior changes with the selected run mode. Azure SRE Agent reasoning and action classification

The exact labels matter less than enforcement.

Build the boundary in layers:

1. Tool allowlist: Expose only the tools required by the task.
2. Argument validation: Reject dangerous paths, recipients, amounts, and command forms.
3. Permission scope: Use the narrowest token or role that can finish the job.
4. Environment isolation: Constrain filesystem, network, process, and secret access.
5. Action classification: Separate read, draft, write, and destructive operations.
6. Approval gate: Require a human or policy decision for sensitive mutations.
7. Idempotency: Assign a stable action key before any retried external write.
8. Audit record: Store the request, decision, actor, tool arguments, and result.

A reusable policy block:

execution_policy:
 default: deny
 
 read:
 approval: none
 tools: [search, read_file, list_status]
 
 draft:
 approval: none
 outputs: [patch, report, message_draft]
 
 write:
 approval: policy
 require:
 - validated_arguments
 - scoped_credentials
 - idempotency_key
 - audit_record
 
 destructive:
 approval: human
 retries: disabled

The model can propose an action.

The executor decides whether the action is possible.

That separation protects the system even when the model is confused, manipulated, or operating with incomplete context.

Failure 4: missing outcome graders reward convincing completion

The agent says the task is done.

The patch exists. The report reads well. The final message lists tests, changed files, and next steps.

None of that proves the requested outcome happened.

A coding agent may need:

• The target test flips into a passing state
• Existing tests remain green
• The application starts
• The changed path works through the browser
• Static analysis finds no new violation
• The final diff stays inside scope

Operations work may need:

• The incident state changed
• The intended resource exists
• The message reached the approved destination
• The scheduled job has one active instance
• The customer record reflects the authorized action
• No duplicate external action occurred

The Anthropic guidance separates the transcript from the outcome and recommends combining grader types according to the task. Code-based graders can inspect tests, state, tool calls, and static analysis. Model-based graders can assess open-ended quality. Human review can calibrate subjective judgments. Anthropic grader taxonomy

The key design choice is to grade the produced state, not a preferred reasoning path.

Rigidly requiring one sequence of tool calls can reject valid solutions. Grading only the final prose can accept failed work. A useful grader checks the invariant the user cares about.

Write the contract before execution:

task:
 objective: "Fix logout for expired sessions"
 
 required_outcomes:
 - expired_session_redirects_to_login
 - active_session_remains_authenticated
 - existing_auth_tests_pass
 
 forbidden_outcomes:
 - auth_checks_removed
 - unrelated_routes_changed
 
 evidence:
 - test_output
 - final_diff
 - browser_receipt
 
 completion_rule:
 all_required_outcomes: true
 all_forbidden_outcomes: false

Then make the completion message a rendering of grader results.

Require evidence before the agent can set its own status to complete. If a grader cannot run, the honest state is unverified, not passed.

Subjective work still needs criteria. A research agent can be graded for source authority, claim support, required coverage, and unresolved uncertainty. A customer-facing draft can be checked against policy and reviewed by a human for tone.

The grader does not need to be elaborate on day one. The Anthropic guidance suggests beginning with a small set of tasks drawn from real failures, then expanding as the system matures. The important step is encoding what success means before a fluent answer makes failure look finished.

Completion is a state transition with receipts.

Failure 5: unreconciled long-running work creates duplicate and conflicting truth

Long-running work does not end when one model response stops.

Jobs continue in queues. Tools return late. Scheduled runs overlap. A retry starts while the original attempt is still alive. Another worker edits the same file. A user cancels the task after an external request has been sent but before its result is recorded.

Without reconciliation, the system can produce:

• Two pull requests for one task
• Duplicate messages or payments
• A stale worker overwriting newer state
• A canceled job resuming after retry
• Two agents marking incompatible plans complete
• An evaluator grading an obsolete artifact
• A final response based on a tool result that arrived too late

This is not a context problem alone. The system needs identity, ownership, versioning, and terminal-state rules.

Microsoft's Azure SRE Agent documentation names several conversation controls, including compaction, automatic retries, error handling, and cancellation. It also states that cancellation halts operations and prevents retrying the canceled task. Azure SRE Agent conversation management

That last property is easy to miss. Retry policy and cancellation policy must agree.

Give each task:

• A stable task ID
• An attempt ID for each run
• A version or generation number
• One current owner
• A lease or heartbeat for long work
• An idempotency key for external mutations
• Explicit terminal states
• A reconciliation step before commit

A basic state machine can stay small:

queued -> claimed -> running -> verifying -> complete
 | |
 v v
 canceled failed
 |
 v
 retry forbidden

Before a worker writes, publishes, merges, pays, or marks completion, require it to check:

1. This attempt is still current.
2. The task remains active.
3. Another attempt has not already produced the intended outcome.
4. The target state has not changed since this attempt began.
5. The idempotency key does not already have a recorded result.
6. The artifact being graded is the latest accepted version.

Any conflict should stop the worker and trigger reconciliation.

Repository work may compare the base commit, current branch, changed files, and open pull requests. External systems may be queried using the idempotency key before repeating the action.

A retry is a new attempt at the same objective.

It is not permission to repeat every side effect.

Run the failure matrix before changing models

Apply this after a failed long-running task.

1. Weak state handoff

Signature: Repeated work, forgotten decisions, stale plans, or unexplained reversals after a boundary.

Inspect: Handoff artifact, compaction summary, repository state, test history, decision log.

Test: Resume from a fresh session using only durable state.

Fix: Record objective, decisions, changes, verification, blockers, and next action. Verify the handoff against live state.

2. Contaminated environment

Signature: Inconsistent failures, unexplained success, leaked artifacts, occupied resources, or trial results correlated by shared infrastructure.

Inspect: Working tree, caches, processes, services, fixtures, credentials, resource limits, and prior-run artifacts.

Test: Re-run from a verified clean environment.

Fix: Isolate trials, pin inputs, reset mutable state, and emit a reset receipt.

3. Unsafe tool boundaries

Signature: The agent can perform actions outside task scope or relies on self-restraint for sensitive operations.

Inspect: Tool list, token scope, argument validation, network access, approval rules, retry behavior.

Test: Ask the agent to attempt an out-of-scope write in a controlled environment. The executor should reject it.

Fix: Default deny, narrow permissions, classify actions, require approval, and add idempotency.

4. Missing outcome graders

Signature: Completion is based on prose, file existence, or tool-call success rather than final state.

Inspect: Task contract, grader outputs, tests, state checks, and evidence attached to completion.

Test: Submit a plausible-looking artifact with a broken outcome. The grader should fail it.

Fix: Define required and forbidden outcomes before execution. Grade state, then render the result.

5. Unreconciled long-running work

Signature: Duplicate actions, stale writes, overlapping attempts, or canceled work returning.

Inspect: Task ID, attempt history, owner, lease, version, terminal state, and idempotency records.

Test: Start two attempts, cancel one, and delay a tool result. Only the current valid attempt should commit.

Fix: Add task identity, attempt generations, ownership, cancellation semantics, and pre-commit reconciliation.

Your expected output is not a higher confidence score.

You should see a classified failure with one reproduced signature, one isolated cause, one repair, and one regression test.

Run the same task again with the model held constant. When the failure disappears and the regression test catches the old condition, the system repair has evidence behind it.

Then test the stronger model.

Better models move the boundary, but they do not remove it

Some execution components should disappear as models improve.

A planner that once added value may become overhead. A specialized evaluator may no longer justify its cost. Instructions written around an old model weakness can constrain a newer one.

A report from Anthropic recommends removing one component at a time and checking its effect. The same report also argues for re-examining the surrounding system when a new model arrives, removing parts that are no longer load-bearing and adding mechanisms for newly possible work. Anthropic iteration notes

That is the right relationship between model progress and system design.

Leave room to remove dead weight.

Model progress is not an excuse to leave state, permissions, grading, or recovery undefined.

A February 13, 2026 GitHub technical preview of Agentic Workflows is another useful signal. The product supports multiple coding-agent engines behind the same workflow format while keeping read-only defaults, sandboxing, network isolation, and controlled writes in the surrounding execution system. GitHub Agentic Workflows technical preview

Models can change.

The execution contract should remain inspectable.

Model quality determines how well the agent reasons inside the system.

System quality determines whether that reasoning survives contact with state, tools, time, and reality.

Model quality decides how well the agent reasons inside the system. System quality decides whether that reasoning survives contact with state, tools, time, and reality.

Your next action: bookmark the Harness Failure Matrix, then run it on the last agent run your team blamed on the model.

Which of the five failures showed up first in your last bad run?

When not to use this matrix

Skip the full five-failure pass when the task is a one-shot chat with no tools, no durable state, and no external side effects. This matrix is for agent systems that touch files, credentials, long sessions, or production state.

It stops you burning eval budget on defects outside the model.

Get the next field note

I publish practical field notes for builders running agents in production - what shipped, what broke, and the system behind it.

Get the next one free: https://nyk.dev/#newsletter

Free. Unsubscribe anytime.

Join the private NYK alpha channel for early notes and updates: https://t.me/+GJ-FEpzcZrtmMTky

Follow @nykdotdev, then pick one: which harness check would you run this week, state handoff or tool boundaries?