01 · Opening

The model is not the problem

The harness around the model decides what the model can do.

20 minutes. Engineers and leadership.

01 · Opening

It works in the demo. Then it meets our repo.

01

The demo

Clean prompt
One file
The happy path

02

Our reality

A large repo
House conventions
Flaky tests
Real users

What breaks half-done features declares victory early no way to check it lost context

~1:00. The gap is systematic, not a model bug. Anthropic, "Effective harnesses for long-running agents" (Nov 2025): a frontier model (Opus 4.5) running on the Claude Agent SDK in a loop, given only a high-level prompt, "will fall short of building a production-quality web app." Same model, same loop, wrong harness.

Anthropic names four observed failure modes, and they are the chips on this slide: the agent tries to one-shot the work and runs out of context, it declares the job done, it marks features complete without testing, and it has to spend time working out how to run the app. All four were addressed by changing the environment around the model: an initializer run, a feature list, a progress file, git commits, an init.sh. The model did not change. In their setup the system prompt, the tools and the harness stayed the same between the two agent roles; only the starting prompt and the environment around the model changed.

Transition: "If the harness is the variable, then the harness is the thing to engineer."

01 · Opening

The harness is the product.

A harness is the loop, tools, context, memory, evals and guardrails around a model.

01

What a harness is

02

Why agents fail

03

The evidence

04

Building the team

~1:15. Define it in one breath: "A harness is everything a model runs inside: the loop that decides what happens next, the tools it can call, the context it sees, the memory it keeps, the evals that tell us if it works, and the guardrails that stop it."

Preview the arc, one sentence each. First, what a harness is made of. Second, why capable agents still fail without one. Third, what the public evidence says about scaffolding versus model. Fourth, who we need to build and run it.

Evidence to keep in your pocket. SWE-agent (Yang et al., arXiv:2405.15793): "interface design affects the performance of language model agents"; their custom agent-computer interface reached 12.5% pass@1 on SWE-bench. Anthropic, "Building effective agents": while building their SWE-bench agent, "we actually spent more time optimizing our tools than the overall prompt." Anthropic, "Effective context engineering": context is a finite attention budget, and recall degrades as tokens grow ("context rot").

Transition: "Start with the parts."

02 · Anatomy

Six parts, one job.

loop — the turn cycle

Model call, tool run, observation, repeat. It ends when the model stops or a turn limit fires.

~1:30. Ten seconds a part, one example each. Loop: OpenAI's function-calling flow is five steps — send tools, receive a call, run it in your application, send the output back, receive the next response; the Agents SDK repeats that until final output or max_turns. Tools: Anthropic's SWE-bench team spent more time optimizing tools than the overall prompt, and requiring absolute filepaths fixed a recurring failure. Context: Claude Code sends the full conversation with every request, so the window is a harness budget. Memory: Claude Code re-injects CLAUDE.md and auto memory from disk after compaction. Evals: SWE-bench scores a task only when the tests pass, so the number is execution, not opinion. Guardrails: Codex separates the sandbox (technical boundary) from the approval policy (when the agent must stop and ask).

The point: each part is owned, testable, and has failure modes. None of them is "the model". Next, one turn in sequence.

02 · Anatomy

One turn, in order.

01

Assemble context

Task, rules, memory, last result.

02

Model call

One action at a time.

03

Pick tools

Name, arguments, intent.

04

Execute in sandbox

Inside the boundary.

05

Read observation

Output, exit code, error.

06

Compact and repeat

Trim, then next turn.

The last observation becomes the next turn's first input.

~1:15. Anthropic defines an agent as "LLMs using tools based on environmental feedback in a loop". OpenAI's function-calling flow makes the mechanics explicit: send the tool definitions, receive a tool call, execute it in your application, send the result back, receive the next response. The Agents SDK repeats that until final output and raises MaxTurnsExceeded past max_turns. Anthropic ends a model turn with stop_reason: "tool_use"; your code runs the block and returns a tool_result.

This is a harness loop, not a model loop. Claude Code sends the full conversation with every request and auto-compacts near the context limit. SWE-agent's ablation on SWE-bench Lite scored 18.0% when the agent saw the last five observations and 15.0% with full history, so what you feed back matters. Step four runs in the Codex sandbox: the sandbox sets the technical boundary, and the approval policy decides when the agent must stop and ask.

Six stages, one pass. The next slide shows what sits around them.

02 · Anatomy

The harness sits between model and world.

01

The world supplies raw state.

02

The harness builds the turn.

03

The model returns one next step.

~1:30. Three layers. The world is the repo, the shell, the tests, the user and the APIs. The harness is everything in between: it assembles context, routes tools, runs them in a sandbox, reloads memory, scores changes and enforces guardrails. The model is the smallest layer: it returns one next step and keeps no state.

Anthropic's agents post defines agents as "LLMs using tools based on environmental feedback in a loop", and its SWE-bench team spent more time optimizing tools than the overall prompt. Claude Code sends the full conversation with every request; a subagent runs in its own context window and returns only a summary, so what crosses the boundary is a design choice. Codex makes the boundary explicit: the sandbox sets the technical limit, and the approval policy decides when the agent must stop and ask. SWE-agent's ablation shows the layer moves the number: a 100-line file view scored 18.0% on Lite; the full file scored 12.7%.

The model is one component. Next, how each part fails.

03 · Why they fail

Four ways a good model fails.

Symptom

Sharp early. Deep into the run, it contradicts its own decision.

Cause

Every token competes for attention; recall fades as the window fills.

Harness fix

Compact the history, clear stale tool output, load files on demand.

~1:15. Four ways the same good model fails, and each one is a harness part, not a model part. Context rot: Anthropic's context engineering post (29 Sep 2025) states that as the number of tokens in the window increases, the model's ability to recall information from that context decreases, across all models. The tell in a trace is a file re-read that was already read, or a decision reversed late in the run.

Tool sprawl: the same post calls bloated tool sets that overlap in function "one of the most common failure modes we see", because the model faces ambiguous decision points about which tool applies. The tell is a failed call followed immediately by a near-identical call to a sibling tool.

Silent success: Anthropic's Claude Code best practices say it plainly — "Claude stops when the work looks done. Without a check it can run, 'looks done' is the only signal available, and you become the verification loop." The tell is a final message that claims the tests pass, with no test command anywhere in the trace.

No memory: the context post recommends structured note-taking precisely because state held only in the window is lost across dozens of tool calls. The tell is a repeat of last session's dead end, discovered at the same cost.

Transition: that is the theory. Here is one task, run twice through the same model.

03 · Why they fail

Same task. Two harnesses.

One tool. No scope, no check, no exit condition.

// the whole interface: one tool, no rules
{
  "name": "shell",
  "description": "Run any command.",
  "input_schema": { "command": "string" }
}

// loop: run it, append output, repeat until "done"

done No check ran. The word is the only evidence.

Three scoped tools. One verify gate it cannot skip.

// three tools, one verb each
read(path, range)     // read-only
edit(path, old, new)  // exact match or fail
verify(cmd)           // exit code is the truth

// loop: plan -> act -> verify
// verify fails: read the error, fix, verify again

passes, verified Exit 0 recorded in the trace.

The model is identical in both runs. Only the tool schema and the loop changed.

~1:15. Same task, same model, same repository. The before run gets one general shell and a loop that stops when the model says it is done. The after run gets three tools, each with one job, plus an explicit verify call whose exit code the loop can read.

The before trace is the one you have seen: a long chain of shell calls, a plausible patch, and a confident final message. Nothing false was said. Nothing was checked either, so a wrong patch and a right patch leave identical traces and identical cost. That is why the failure survives review — the harness had no opinion about truth.

The after trace costs one more step and produces a pass or fail the loop can act on. This is the claim from Anthropic's Claude Code guidance: give the model a check it can run and "the loop closes on its own"; without one, "you become the verification loop".

Nothing in either block is model code. It is harness code, and it is the part we own.

Transition: if that sounds like an anecdote, the published ablations say the same thing at scale.

03 · Why they fail

Scaffolding moves the number.

SWE-bench Lite, 300 tasks · GPT-4 Turbo in all four runs · Yang et al., SWE-agent, NeurIPS 2024 (Tables 1 and 3)

Same model: 2.7% to 18.0% resolved. Third bar is that agent with its edit linter removed.

~1:30. This is the same model four times. GPT-4 Turbo, SWE-bench Lite, 300 real GitHub issues, scored by the repository's own tests. The only thing that changes across the bars is the interface and the loop around the model.

With the model reading retrieved files and writing a patch in one shot, 2.67% resolve. Give it a plain Linux shell and let it work interactively, 11.00%. Give it purpose-built tools — a file viewer with line numbers, a search that summarises, an editor that refuses edits that do not apply — 18.00%. The paper reports that as a 64% relative increase over the shell-only agent, same model.

The fourth comparison is the one I care about: take the full agent and remove only the linter on edits. 15.0%. The agent still reads, still searches, still edits. One harness check that returns a signal costs three points.

What this does not prove: it is one model from 2024, one benchmark, 300 tasks, and the authors report that per-instance results move between runs even when the average holds. It does not prove a gain of the same size transfers to our repository. Nothing in a public benchmark is our codebase.

Which is the point of the next slide: if the number moves this much on someone else's repo, you cannot tune a harness for yours without measuring yours.

03 · Why they fail

You cannot tune what you do not measure.

20–50

tasks, from your own repo

01 Start from a failure you already had.
02 End in a state a script can check.
03 Two engineers agree on the verdict.

$ harness-eval run --suite ./evals # 42 tasks
  pass@1        0.78 → 0.71  regression
  cost/success  $0.38 → $0.44
  merge         blocked

Example run. The suite runs on every pull request that touches the harness.

~1:15. Public benchmarks measure a model against someone else's repository. SWE-bench Verified is a 500-task subset of SWE-bench, reviewed by humans so each task is solvable (Anthropic, "Raising the bar on SWE-bench Verified", Jan 2025). By early 2026 that benchmark is near saturation above 80% (Anthropic, "Demystifying evals for AI agents", Jan 2026). Neither number tells you whether the harness can fix your backlog.

Anthropic's guidance is specific: 20 to 50 simple tasks drawn from real failures is a great start, because early changes have a large effect and small samples suffice (same post, Step 0). An ambiguous task becomes noise in the metric.

What you get in week one is a number that survives an argument. Before the suite, "the agent feels worse" is an opinion. After it, a regression is a red build.

Signals: pass@1 is Anthropic's definition — the fraction of tasks solved on the first try. Cost per success is the metric Anthropic tracks as "cost per task". False completion is documented directly in the SWE-bench post: the model cannot see the grading tests, so it often "thinks" it has succeeded when the task actually failed. Review minutes is our own metric; no external source for it, treat the number as illustrative.

The gate is the whole point. If a harness change lowers pass@1, it does not merge.

Next: the suite is not free to run.

04 · Cost and risk

Every harness change has a bill.

Retrieved context, per turn Illustrative 6,000 tok Turns per task Illustrative 20 Added input per task 120k tok At $2 / MTok input, Sonnet 5.5 $0.24 Same tokens, cache read at $0.10 / MTok $0.012

Anthropic API pricing, Claude Sonnet 5.5: $2 input / $10 output per MTok. Read 24 Oct 2026.

Same model. Different harness. Different score.

~1:00. Three costs. Only one of them appears on the invoice.

Tokens: context assembly and memory retrieval are re-sent on every turn. The arithmetic on screen is 6,000 tokens per turn times 20 turns — 120,000 input tokens per task. Both inputs are illustrative, not a published measurement. At Claude Sonnet 5.5's $2 per million input tokens that is $0.24 per task. The same tokens served as cache reads at $0.10 per million cost $0.012 (Anthropic API pricing page, read 24 Oct 2026). Caching is a harness feature, not a billing trick.

Latency: every extra model call and tool round trip is serial. One verifier call that prevents five turns of rework pays for itself. A retriever that blocks every turn does not.

Review: a harness that verifies charges minutes at the diff. A harness that does not charges incident time later. Anthropic's evals post describes teams without evals as stuck in a loop: fix one failure, create another, find out in production.

The bottom line is sourced: Anthropic states that SWE-bench Verified performance "can vary significantly" with the scaffolding, on the same model. The line in the review tab — a cheap model with a good harness beats an expensive model with a bad one — I could not source as a measured comparison. It is labelled opinion on the slide, and it is opinion.

Measure cost per successful task, not cost per call. Next: where the harness is allowed to act.

04 · Cost and risk

Draw the trust boundary.

01

Read only

Files, logs, test output.

path allowlist

02

Write in the repo

Edits to tracked files.

sandbox

03

Execute

Build, test, migrate.

approval gate for
irreversible actions

04

Reach the network

APIs, registries, the open web.

egress allowlist

Move right only when the task needs it.

Prompt injection is an input-trust problem, not a model-behaviour problem.

~1:15. Four tiers, in the order you should grant them. Read only comes first, always.

Write inside the repo is tier two. OpenAI's Codex calls this workspace-write and describes it as the default low-friction mode: the agent reads and edits inside a boundary, and routine commands run there without a prompt. When a task needs to go beyond the boundary, the approval flow takes over (OpenAI Codex sandbox documentation).

Execute is where the blast radius jumps. Claude Code's manual mode starts read-only and asks before edits or commands; unmatched commands require approval by default, and an allow rule for part of a command does not skip the prompt (Anthropic, Claude Code security documentation). Put the approval gate on irreversible actions only: migrations, deletes, force-push, deploys. Approving everything trains people to click yes.

Network last, and never as a blanket "internet on". Claude Code's cloud sessions limit network access by default and can allow only specific domains; its local sandbox isolates bash commands at the filesystem and network level.

The callout is the part leadership needs. OWASP LLM01:2025 defines indirect prompt injection as instructions arriving through websites or files, and states that fool-proof prevention is unclear. Its mitigations are privilege control, least privilege, human approval for high-risk actions, and segregating untrusted content. OWASP LLM06:2025 names the root cause of excessive agency: excessive functionality, excessive permissions, excessive autonomy. None of that is fixed by a better prompt.

So untrusted text never carries instructions. A README, an issue body, a web page, a test failure: all data. If any of them can issue orders, the agent is a confused deputy.

Guardrails are the floor. Next: who builds and owns this.

05 · Team and process

Nobody owns the harness, so it rots.

01

Harness engineer

owns the loop and the tool layer

02

Eval owner

owns the task suite and the gate

03

Context owner

owns what enters the window

04

Security reviewer

owns the trust boundary: tools, permissions, data

05

Product owner

owns which workflows get a harness

Two or three people can start this, part time.

~1:00. Ownership is the difference between a demo and infrastructure. Without one, the prompt grows, the tool list grows, and nobody can say whether quality moved.

Anthropic's evals guidance, "Demystifying evals for AI agents," reports that a suite stays useful only with ownership: an eval suite is, in their words, "a living artifact that needs ongoing attention and clear ownership to remain useful." They found dedicated evals teams owning the core infrastructure works, with product teams contributing the tasks.

Platform engineering's "What is platform engineering?" makes the same point: start with a small minimum viable platform, not a department. The smallest viable team here is two people, part time; a third makes it durable. A harness engineer and an eval owner run the loop. The product owner picks the first workflow. Security review is borrowed for the trust boundary, because Anthropic's "Trustworthy agents in practice" says a well-trained model can still be exploited through "a poorly configured harness, an overly permissive tool, or an exposed environment."

Next: how the loop actually runs.

05 · Team and process

Change one thing. Measure again.

01
Pick a failed trace
02
Name the harness cause
03
Change one harness part
04
Rerun the suite
05
Keep or revert

One change per run, or you learn nothing.

~1:00. This is ordinary engineering discipline applied to a probabilistic system. Anthropic's "Demystifying evals for AI agents" defines a transcript, or trace, as "the complete record of a trial, including outputs, tool calls, reasoning, intermediate results, and any other interactions," and says reading transcripts is how you verify the eval measures what matters.

Hamel Husain's "Your AI Product Needs Evals" makes logging traces the prerequisite, then says to remove all friction from looking at them. Pick the failure, name which harness part caused it, change that one part, rerun the same suite.

One change per run: change two and a pass tells you nothing about which one worked. Keep the change only if the suite moves; otherwise revert it and name the next cause. Traces are the spec.

Next: the first thirty days.

05 · Team and process

What the first thirty days look like.

Week 1
Twenty real tasks. Pass or fail, by hand.
Week 2
A full trace of every run.
Week 3
One verification step. One scoped tool set.
Week 4
The suite in CI. A regression gate.
Ongoing
One named owner. A weekly trace review.

~1:00. Month one is measurement, not modernization. Anthropic's "Demystifying evals for AI agents" says to start with "20-50 simple tasks drawn from real failures," and to keep regression evals at "a nearly 100% pass rate."

Week 1 is manual on purpose: you are learning the failure modes, not building tooling. Week 2 makes every run inspectable, because a failure you cannot replay is a rumor. Week 3 changes the harness: one verification step, and a tool set scoped to the task.

Week 4 moves the suite into CI so the gate runs on every commit, which Anthropic lists as a property of automated evals. Then keep one named owner and a weekly trace review.

Skip new models, new frameworks, and fine-tuning in month one. With no baseline you cannot attribute the change. Anthropic notes that teams without evals face weeks of testing when a new model lands, while teams with evals upgrade in days.

Next: how harness work goes wrong.

05 · Team and process

How harness work goes wrong.

The tell

Your score climbs. Production does not.

The harness learned the tasks, not the work.

The countermeasure

Hold out ten tasks. Rotate them each quarter.

Report the held-out number, never the tuned one.

~1:00. Each of these looks like progress for a month, then stops paying.

Overfit. SWE-Bench+ (Aleithan et al., 2024) re-checked the top agent's passing patches: 32.67% had the fix leaked in the issue, 31.08% passed on tests too weak to prove correctness. After removing those tasks the resolve rate fell from 12.47% to 3.97%. Your twenty tasks can do the same thing to you.

Sprawl. Anthropic's Claude Code guidance: if CLAUDE.md is too long, Claude "ignores half of it because important rules get lost in the noise." Prune, then turn hard rules into hooks, which run deterministically instead of advising.

Hiding failure. METR (June 2025) reports frontier models "attempting (often successfully) to get a higher score by modifying the tests or scoring code." Anthropic's long-running agent harness keeps its feature list in JSON, and its prompt says it is "unacceptable to remove or edit tests because this could lead to missing or buggy functionality." The check has to sit outside the agent's write scope.

Shadow harness. GitHub's cloud agent works in a branch and opens a pull request, and its docs say session logs "do not replace your own review and testing." A private stack on one laptop skips exactly that path.

Transition: none of these are model failures. They are process failures, and they are fixable this quarter.

06 · Payoff

What changes for you.

If you lead

01Stop shopping for models first.
02Fund the harness and its evals.
03Ask for cost per successful task.
04Reward the boring fixes.

If you build

01Own the loop, not the prompt.
02Instrument every run.
03Add verification before autonomy.
04Keep the harness in the repo.

06 · Payoff

Pick one agent. Give it a harness.

The model will keep changing. The harness is the part you own.

This week: pick 20 tasks Questions

The model is not the problem

It works in the demo. Then it meets our repo.

The demo

Our reality

The harness is the product.

What a harness is

Why agents fail

The evidence

Building the team

Six parts, one job.

One turn, in order.

Assemble context

Model call

Pick tools

Execute in sandbox

Read observation

Compact and repeat

The harness sits between model and world.

Four ways a good model fails.

Same task. Two harnesses.

Scaffolding moves the number.

You cannot tune what you do not measure.

Every harness change has a bill.

Harness that verifies

Harness that does not

Draw the trust boundary.

Read only

Write in the repo

Execute

Reach the network

Nobody owns the harness, so it rots.

Harness engineer

Eval owner

Context owner

Security reviewer

Product owner

Change one thing. Measure again.

What the first thirty days look like.

How harness work goes wrong.

What changes for you.

If you lead

If you build

Pick one agent. Give it a harness.