Harness-Building for Coding Agents

Executive summary#

Recent writing by Martin Fowler, LangChain, OpenAI, GitHub, Anthropic, and open-source coding-agent projects converges on the same broad intuition: a coding agent is not just a model. It is a model embedded in a system of prompts, tools, files, plans, approvals, execution environments, and feedback loops.¹²³

A useful definition is therefore not the negative definition, “everything except the model,” but a functional one:

The harness of a coding agent is the configurable control substrate that mediates perception of the software environment, curates context for inference, executes and constrains actions, preserves state, and closes the loop with evaluation, policy, and human oversight so that model outputs become reliable software work.

This formulation fits three related but distinct usages:

1. LangChain’s broad usage: agent = model + harness; the harness includes code, configuration, orchestration, tools, memory, and execution logic.
2. Fowler’s bounded usage: the “outer harness” that coding-agent users build around vendor-provided agents for their own codebase and workflow.
3. OpenAI’s Codex usage: the harness as the core loop, prompt/tool wiring, execution logic, and context-window management around a coding model.

The key analytical distinction is between access and selection.⁴⁵⁶ In classical AI, sensors and percepts are not the same as working memory or control state. Likewise, in coding agents:

• Perception/access is how the agent inspects repositories, terminals, issues, browsers, logs, CI systems, and external tools.
• Context management is how the harness decides what to load, compress, retain, offload, or discard before the next model call.

That distinction becomes more important in coding than in many classical toy domains because the “world” varies sharply across cases. A one-file bug fix, a multi-repo migration, a browser-driven frontend repair, a CI failure triage task, and a vague product-code request expose very different observable states and require different salience policies.

Classical AI offers direct analogues.⁴⁷⁸⁹¹⁰ PEAS gives a framework for specifying performance measures, environment, actuators, and sensors. BDI captures beliefs, desires, intentions, and commitment. Blackboard systems anticipate shared workspaces, opportunistic reasoning, and dynamic control. MAPE-K formalizes monitor-analyze-plan-execute loops over shared knowledge. Cognitive architectures such as Soar show how memory, control, and learning can be structured across time.

Modern LLM-agent work re-instantiates these ideas under newer names:^{11121314151617} ReAct, Toolformer, Reflexion, Self-Refine, Tree of Thoughts, LATS, Voyager, SWE-agent, AutoCodeRover, Agentless, CodeAct, OpenHands, Deep Agents, Codex, Claude Code, GitHub Copilot, Aider, and Continue.^{1819202122232425262728}

1. Defining harness precisely#

The broad industry definition is:²

Agent = Model + Harness

In this formulation, the harness is every piece of code, configuration, execution logic, tool wiring, memory, state, and orchestration that is not the model itself.²²⁹

That definition has two advantages:

• It rejects the mistaken view that a coding agent is just a better prompt.
• It foregrounds the fact that agency requires state, tools, feedback, and constraints outside the model weights.

But as a design definition, it is too residual. It defines harness by what it is not, rather than by what it does. It does not distinguish:

• builder-side harness from user-side harness;
• generic infrastructure from agent-specific control logic;
• perception from context management;
• evaluation from orchestration;
• safety policy from task execution.

A more precise definition for coding agents is:

Harness = the layered control substrate that turns model competence into dependable software-engineering behavior by managing access, interpretation, salience, action, memory, orchestration, evaluation, and human oversight.

This definition is positive rather than residual. It defines a harness by function.

Boundary test#

A component belongs to the harness if removing it materially changes how the agent:

• observes the environment,
• chooses what to do,
• acts on the software world,
• remembers state,
• is evaluated,
• is constrained,
• or hands control back to a human.

Examples of harness components include:

• system prompts and repository instructions;
• file and code-search tools;
• shell execution;
• edit tools;
• browser tools;
• retrieval and ranking;
• context compaction;
• memory stores;
• task plans and todo files;
• sandboxing;
• test and lint loops;
• review agents;
• approval policies;
• trace logging;
• human-in-the-loop controls.

The model contributes latent competence. The harness determines the operating regime in which that competence is expressed.

2. Perception, context, and why coding-agent “worlds” differ#

The distinction between perception and context management is central.

2.1 Classical perception#

In classical AI, perception usually means the transformation of sensor input into percepts or beliefs about the environment.

A simplified classical loop is:⁴

environment → sensors → percepts → internal state → decision/action

For a robot, perception might be camera frames, lidar, audio, or force sensors converted into object locations and scene state.

2.2 Coding-agent perception#

For a coding agent, perception is not usually visual or physical. It is symbolic, tool-mediated, and often textual.

Examples include:

• reading files;
• searching code;
• inspecting symbols;
• running tests;
• reading compiler output;
• reading CI logs;
• querying issue trackers;
• inspecting git history;
• using a browser;
• reading API documentation;
• observing runtime traces;
• viewing screenshots;
• receiving human review comments.

A coding agent’s “world” is therefore not just a filesystem. It is a layered software-technical-social environment.

2.3 Context management#

Context management is the process that turns available observations, memories, constraints, and goals into the bounded working state supplied to the model.²³⁶

It includes:

• retrieval;
• ranking;
• chunking;
• summarization;
• compaction;
• offloading to files;
• long-term memory updates;
• subagent context isolation;
• prompt assembly;
• instruction precedence;
• preserving task invariants.

A useful distinction:

Perception changes what the agent can know about the environment. Context management changes what the model keeps active for reasoning.

Search, retrieval, and summarization often do both. That is why the boundary is blurry in coding agents.

2.4 Why coding-agent worlds make the distinction harder#

The “world” changes radically across task types.

The harder the world is to observe directly, the more perception and context management become entangled.

3. Functional taxonomy of coding-agent harnesses#

This taxonomy clarifies Fowler’s “guides” and “sensors” distinction:

• Guides mostly live in interpretation, salience, and regulation.
• Sensors mostly live in perception, evaluation, and human supervision.
• Computational vs. inferential tools cut across many functions rather than forming a separate layer.

Context engineering is therefore a major part of harness engineering, but not the whole of it.

4. Classical AI analogues#

4.1 PEAS and intelligent-agent specification#

Russell and Norvig’s PEAS framework asks designers to specify:⁴

• Performance measure
• Environment
• Actuators
• Sensors

For coding agents, this maps cleanly:

PEAS is a useful antidote to vague agent design. It forces the harness builder to state what success means and what the agent is allowed to observe or change.

4.2 BDI: beliefs, desires, intentions#

BDI architectures model agents in terms of:⁷

• beliefs: what the agent takes to be true;
• desires: goals or states it would like to bring about;
• intentions: commitments to particular courses of action.

Coding tasks are commitment-heavy. An agent must maintain:

• beliefs about how the codebase works;
• goals implied by the user request;
• constraints such as “do not change public API”;
• intentions such as “fix failing parser test by modifying tokenizer logic only.”

Planning artifacts, todo lists, branch notes, and task summaries are not mere UX. They function as externalized intention-management machinery.

4.3 Blackboard systems#

Blackboard architectures coordinate multiple knowledge sources through a shared workspace.⁸ Different modules contribute partial results, and a control mechanism decides what to do next.

The analogy to coding agents is strong:

Subagent systems are often blackboard systems in modern form: specialized agents work with scoped context and coordinate through shared artifacts.

4.4 MAPE-K#

Autonomic computing’s MAPE-K loop consists of:⁹

Monitor → Analyze → Plan → Execute
             ↑          ↓
             Knowledge

Coding agents often follow this same loop:

inspect repo/tests/logs → diagnose → plan edit → apply edit/run command → update state

The shared knowledge substrate is crucial. It includes codebase rules, retrieved files, task memory, test results, traces, and user constraints.

4.5 Cognitive architectures#

Cognitive architectures such as Soar show that intelligent behavior often depends on persistent structures for:¹⁰

• working memory;
• long-term memory;
• production rules;
• goals and subgoals;
• learning from impasses;
• action selection.

Modern LLM agents reintroduce similar concerns through prompt context, retrieval, memory, tool state, plans, and external files. The difference is that much of the “working memory” is serialized into text.

5. Modern LLM-agent research analogues#

Many LLM-agent papers study harness components without using the word “harness.”^{11121314151617}

Two conclusions follow.

First, much of the performance of modern agents comes from the loop around the model: tools, memory, search, validation, context selection, and control.

Second, coding-agent research shows that better harnesses often mean better interfaces to the software world, not merely better prompting.

6. Industry and open-source harnesses#

The current ecosystem can be compared by which harness functions each system treats as first-class.^2324252630

A major trend is that “memory” is becoming inseparable from harness design. If the harness controls what is persisted, compacted, retrieved, or forgotten, then harness ownership also implies memory ownership.

7. Evaluating harness effectiveness#

A harness should be evaluated as a system under a controlled model.^2464748 The basic question is:

What additional capability, reliability, or governance does this harness create for a given model on a defined coding-task family?

The right experimental logic is ablation:

• same model, different harness;
• same harness, one component removed;
• same task suite, varied context policy;
• same model and tools, different action interface;
• same agent loop, different evaluators or approval gates.

7.1 Evaluation dimensions#

7.2 Useful benchmark families#

• SWE-bench / SWE-bench Verified: repository issue resolution.⁴⁷⁴⁹
• Terminal-Bench: long-horizon terminal tasks.⁴⁸⁵⁰
• HumanEvalFix-style tasks: repair-focused code tasks.
• τ-bench-style evaluations: tool-use and rule-following under policy constraints.⁵¹
• Internal CI suites: real codebase behavior.
• Replay-based harness ablations: evaluate alternative context/action/eval policies from the same checkpoint.

7.3 Six-number deployment scorecard#

A practical harness scorecard should include:

1. resolved-task rate;
2. regression-free rate;
3. median wall-clock time;
4. cost per successful task;
5. human minutes per successful task;
6. policy-violation rate.

A single benchmark score is insufficient. Harnesses often trade correctness, autonomy, cost, latency, and governance against one another.

8. Reference architecture#

A coding-agent harness should be treated as a closed-loop engineering system, not as a chat wrapper.¹³⁹²³

  flowchart TD
    U[Human operator] --> T[Task specification and constraints]
    T --> I[Instruction layer<br/>system prompt + repo guidance + policies]
    I --> O[Planner and orchestrator]
    O --> C[Context assembler<br/>retrieval + ranking + compaction + memory]
    C --> M[Model inference]
    M --> A[Action gateway<br/>edit tools + shell + git + browser + MCP]
    A --> S[Sandbox and software world<br/>repo + tests + CI + docs + issue tracker]
    S --> P[Observations<br/>logs + diffs + test results + screenshots]
    P --> E[Evaluators and regulators<br/>tests + linters + rules + review agent]
    E --> C
    E --> H{Human approval needed?}
    H -->|yes| U
    H -->|no| O
    C --> R[State and memory<br/>files + checkpoints + long-term memory]
    R --> C
    O --> R

9. Design patterns#

9.1 Repository knowledge as stable steering layer#

Keep durable instructions in repo-controlled files such as:⁵²²⁵²⁶

• AGENTS.md;
• CLAUDE.md;
• .github/copilot-instructions.md;
• tool-specific skills or rule files.

Treat these as operational policy, not prompt decoration.

Good repository instructions specify:

• architecture constraints;
• build/test commands;
• style and formatting rules;
• public API constraints;
• security constraints;
• common pitfalls;
• review expectations;
• non-goals.

9.2 Context is budgeted, not merely added#

Strong harnesses treat context as a scarce resource.²³⁶³¹

Patterns:

• keep stable instructions separate from volatile evidence;
• retrieve minimally;
• prefer symbol-level slices over whole files;
• offload bulky artifacts to files;
• compact old conversation aggressively;
• isolate subagents so local exploration does not pollute the main thread;
• preserve invariants explicitly.

9.3 Executable feedback beats passive advice#

Prefer machine-checkable feedback:^1463740

• tests;
• lint;
• type checking;
• formatting;
• static analysis;
• browser assertions;
• screenshot comparisons;
• diff review;
• policy checks.

A harness that says “be careful” is weaker than a harness that runs the relevant test and returns the failure.

9.4 Scoped subagents, not one giant context#

Subagents are most useful as context-isolation devices.²³²⁴²⁵

Good subagent design:

• narrow task;
• limited tool scope;
• explicit output contract;
• handoff through files or structured summaries;
• main agent retains global intent and final responsibility.

9.5 Security at the environment boundary#

Do not rely on model self-policing.²³³⁷

Use:

• ephemeral sandboxes;
• filesystem restrictions;
• network restrictions;
• secret isolation;
• approval gates;
• command allow/deny lists;
• review before commit or deploy.

9.6 Observe everything, then improve the harness#

Many agent failures are harness-level failures:³⁴⁶

• wrong context loaded;
• too much irrelevant context loaded;
• relevant file not found;
• dangerous tool exposed;
• poor stop condition;
• missing evaluator;
• ambiguous approval default;
• bad retry loop;
• inadequate task decomposition.

Trace logging, replay, checkpoints, and evaluation dashboards are therefore core harness infrastructure.

10. Prompt and context strategy#

A compact layered context strategy:

The central principle:

The harness should make the correct behavior easier than the incorrect behavior.

11. Open research questions#

11.1 How much of the harness should be open and portable?#

Integrated products often produce better surface coherence because they control the runtime, memory, permissions, and UI. Open harnesses offer inspectability and portability. The trade-off remains unresolved.

11.2 How can harness quality be evaluated independently of model quality?¹⁴⁷⁴⁸#

Benchmarks still confound model ability with harness design. The field needs harness-specific diagnostics:

• context-selection quality;
• retrieval-localization precision;
• evaluator coverage;
• approval-policy calibration;
• tool-action safety;
• interruption and recovery quality;
• trace explainability;
• compaction regret.

11.3 When is less agency better?#

Agentless and AutoCodeRover-style approaches suggest that some software tasks are better handled by staged pipelines than by fully open-ended agents.¹⁹²⁰ The frontier is adaptive autonomy:

• tight pipeline for well-scoped repairs;
• ReAct-style exploration for uncertain debugging;
• tree search for hard reasoning tasks;
• human escalation for ambiguous product or architecture decisions.

11.4 How should memory and instructions be standardized?#

Current ecosystems expose different formats and semantics:^29232526

• AGENTS.md;
• CLAUDE.md;
• Copilot instructions;
• skills;
• hooks;
• MCP servers;
• proprietary memories.

If memory is part of the harness, portability will require better interfaces for working memory, long-term memory, skills, and policy metadata.

11.5 Can harnesses self-improve?#

Future harnesses may use traces to repair themselves:⁴⁶⁵³

• identify recurrent context failures;
• add new tests or sensors;
• refine repository instructions;
• generate task-specific skills;
• tune retrieval policy;
• adjust approval thresholds;
• synthesize new evals from observed failures.

That shifts human work upward: from writing every patch to designing the environment and feedback loops in which agents write patches.

12. Bottom line#

“Harness” is a useful term, but it should not remain a vague synonym for “everything around the model.”

For coding agents, the clearest formulation is:

A harness is the layered control, context, tool, memory, evaluation, and supervision system that turns a model into a situated software-engineering actor.

The closest classical term is not one thing, but a bundle:

• agent architecture;
• task-environment interface;
• sensors and actuators;
• working memory;
• blackboard control;
• BDI commitment;
• MAPE-K loop;
• cognitive architecture.

The most important practical insight is this:

Better coding agents are not produced only by better models. They are produced by better operating environments for models: better perception, better context, better tools, better feedback, better constraints, and better handoff to humans.

Footnotes#