Agent harness engineering: improving AI without fine-tuning

An agent harness can improve the task performance of a fixed language model by changing the context it receives, the tools it can call and the rules governing execution. At StaffOS, we build these controls into the agent runtime and support reviewed changes to business instructions and configuration. This paper sets out the architecture, the published evidence behind it and a method for measuring whether a harness change helps.

The paper's figures use public research results. Its workflow examples are fictional, and its evaluation protocol uses engineer-authored scenarios with synthetic records. Model weights remain fixed throughout that protocol.

Abstract

Business agents operate across conversations, databases and external services. Their failures often begin at those boundaries: missing context, ambiguous tool results, stale state or an unsupported claim that an action succeeded. Harness engineering addresses these failure modes through explicit software contracts. We examine three published experiments covering tool interfaces, prompt optimization and context adaptation, then describe how these ideas apply to StaffOS. We define an evaluation protocol that holds the model fixed, measures verified outcomes and accounts for latency, inference cost and operational load. The benchmark figures below come from the cited studies. The architecture and evaluation specification are our engineering contribution.

1. What an agent harness controls

An agent harness is the software around a language model that determines what the model sees, which actions it can request, how those actions execute and when the work ends. Its scope includes context assembly, retrieval, tool contracts, state management, execution limits, approval rules and evaluation feedback.

Consider a fictional appointment request. The model must identify the requested service, obtain valid slots, interpret the selected time, request a booking and report the result. The surrounding software determines whether the slots are current, whether a retry creates another booking and whether the returned status gives the model enough information to answer correctly.

For analysis, represent an agent as:

Agent outcome = F(model, harness, task, environment)

Harness = {
  instructions,
  context selection,
  tool interfaces,
  execution policy,
  working state and evaluation feedback
}

Changing the harness changes the conditions under which the model works. A useful comparison therefore fixes the model and environment while changing a declared part of the harness. Switching models, adding more attempts and revising tools at the same time makes attribution difficult.

We use three units of change:

• A prompt or playbook revision: a change to the instructions used for a particular decision.
• A tool contract revision: a change to the actions available or the information returned by an action.
• A runtime revision: a change to retrieval, context assembly, execution limits or state handling.

Each unit has a different failure mode and should have its own test cases.

2. Published results with fixed model weights

The studies in this section examine different tasks and metrics. Each comparison holds the deployed model fixed within its experiment. The values are useful as evidence for specific mechanisms; combining them into an average would obscure their experimental conditions.

Tool interfaces: FrogNano

Kim et al. report the following SWE-bench Verified solve rates when replacing R2E-Gym with their Leaf harness. Leaf changes the tool interface, instructions and termination behavior together. 1

Fixed model	R2E-Gym	Leaf	Absolute change
Qwen3.5-4B	8.3%	37.2%	+28.9 percentage points
MiniMax-M2.5	66.5%	66.5%	0.0 percentage points

Figure 1. Published harness comparison from FrogNano, Section 2. Axis starts at zero. The larger model shows no change.

For Qwen3.5-4B, the ratio is 37.2 / 8.3 = 4.48. That is a model-specific result from an interface experiment. FrogNano's subsequent reinforcement-learning results involve weight updates and are outside this comparison.

Prompt optimization: GEPA

GEPA uses execution feedback to propose and evaluate prompt revisions. On HotpotQA, the paper's Qwen3-8B baseline scores 42.33%, and GEPA scores 62.33%, an increase of 20.00 percentage points. The official experiment uses answer exact match. On IFBench, the corresponding improvement is smaller: 36.90% to 38.61%. These results show why prompt optimization needs a task-specific evaluation. 2, 4

Context adaptation: ACE

ACE maintains structured playbook entries through generation, reflection and curation. With DeepSeek-V3.1 on AppWorld, its offline configuration with ground-truth labels improves the reported average from 42.4% to 59.4%, an increase of 17.0 percentage points. That average combines task and scenario completion on the normal and challenge splits. The offline configuration without ground-truth labels reaches 57.2%. 3

Figure 2. Separate experiments from ACE Table 1 and GEPA Table 1. AppWorld uses a four-metric completion average; HotpotQA uses answer exact match. The panels have independent experimental conditions.

The practical question for an engineering team is which failure a change addresses. A clearer tool result can resolve an interface problem. A playbook can supply a missing procedure. A retrieval change can make an existing fact available at the right step. Each claim can be tested directly.

3. How we structure the StaffOS harness

StaffOS combines context retrieval, structured tools, bounded execution and reviewed configuration changes. The boundaries between these components matter because a single request can involve language interpretation, a business policy and a state change. We keep the policy and state checks in application code wherever they can be expressed reliably.

Context: select information for the current decision

Our knowledge retrieval applies business and agent scope, activity and validity rules, similarity thresholds and entry limits. Procedural playbooks use a separate retrieval path. This lets the runtime distinguish a fact, such as a product specification, from a procedure, such as the steps for qualifying a lead.

Conversation history supplies context for handling the active request and has a budget. Long agent loops can compact older context while retaining system instructions and recent tool exchanges. The operational questions are concrete: Was the relevant source selected? Did its content fit? Was a necessary fact removed during compaction? Did the mechanism actually run?

A context setting alone answers none of those questions. Evaluation needs the effective configuration and observations from the executed path.

Tools: make actions and results explicit

Tools carry names, descriptions and structured parameter schemas. The registry applies execution gates and stages applicable configuration changes for approval. Individual tools enforce their own business preconditions.

Our native appointment flow illustrates the contract:

appointment_offer_slots
  -> available slots with exact start times

appointment_book(starts_at: selected_slot.starts_at)
  -> booked appointment
  -> existing booking reused
  -> slot_unavailable

The model receives distinct outcomes for a new booking, a retry that reuses an existing booking and a slot conflict. A conflict gives it a reason to fetch fresh availability. An existing booking gives it a stable result to report.

An appointment tool also has to survive concurrent requests. Retry behavior belongs to the operation itself. Repeating a well-formed request must not silently multiply the customer's bookings.

Execution: bound the loop and inspect completion claims

The runtime limits tool rounds and introduces a wind-down instruction near the cap. It also applies targeted checks to selected claims about completed actions, including escalation and configuration changes. Those checks inspect tool evidence or application state, then replace an unsupported escalation claim or append the appropriate status clarification.

The scope of a completion check should be explicit. A rule that verifies an escalation cannot establish that every factual statement in the reply is correct. Each additional claim family needs its own evidence and failure behavior.

A completion claim needs the same evidence as the business operation it describes.

Configuration: review and version changes

StaffOS supports reviewed changes to five kinds of configuration: persona, prompts, playbooks, knowledge and product information. These controls let operators specify how an agent should handle their business's workflows. Prompt changes use versioned writers, and the evaluation system supports scripted cases and test runs against a model provider.

Each configuration has an owner and a purpose. A wrong product fact belongs in the knowledge source. A required sequence of actions belongs in a playbook. An ambiguous success response belongs in the tool contract. Putting all three into the system prompt would make ownership and testing harder.

4. Building evaluation cases from workflow specifications

Engineers can write evaluation cases from workflow requirements, business rules and tool contracts. Each case defines a decision, the information available at that point and the expected outcome. For a task that changes state, the test also specifies which records should exist after execution.

The case specification requires the following fields:

Field	What the case needs
Decision point	The synthetic message or simulated tool result that starts the test
Available context	Only information accessible before that decision
Initial state	Relevant records, permissions, availability and clock assumptions
Expected behavior	Required action, acceptable response or justified handoff
Forbidden behavior	Duplicate write, unauthorized action or unsupported success claim
Outcome check	A deterministic assertion where possible, otherwise a reviewed rubric
Source identity	Scenario family, specification reference and fixture provenance
Version identity	Model, harness, prompt, tools, fixture and scorer versions

For a synthetic appointment case, set up a selected slot that becomes unavailable before the booking call. The test checks that the agent reports the conflict and offers alternatives. The expected text can vary. The business invariant stays fixed: the reply must not describe a booking that does not exist.

Controlled variants expose neighboring failures. The same case can vary the date wording, simulate a timeout or repeat a request after a successful write. Each variant requires a valid initial state and a checked expected outcome. Keep variants of a scenario in the same evaluation split so a nearly identical case cannot appear in both development and final testing.

Score the response and the resulting state separately. A clearer reply can still describe an action that failed. A successful test booking can still produce a confusing confirmation. The case needs checks for both.

Build fixtures with invented messages, identifiers and business records. Record their provenance and keep production transcripts, account identifiers, credentials and private documents out of the test material. The examples and downloadable assets accompanying this paper contain only public research information and fictional workflow descriptions.

5. Evaluation protocol for a harness release

A harness experiment compares two declared configurations against equivalent starting conditions. We hold the model identity, provider settings, case inputs and budgets constant, then change one mechanism or a clearly identified bundle. The result should show which cases improved, which regressed and what the change cost.

Our evaluation specification has six steps:

1. Freeze the cases and scoring rules. Establish task success and hard business constraints before running the candidate.
2. Group related examples. Keep each synthetic scenario and its variants in one split. Reserve independent cases for final testing.
3. Declare the intervention. Record the prompt, tool, retrieval or runtime change being tested.
4. Reset state for every attempt. Both configurations receive the same appointment availability, records and permissions.
5. Repeat within a fixed budget. Report attempts, confidence intervals and paired outcome differences; account for repeated runs from the same case.
6. Inspect regressions before release. Review failures by workflow, language and action type, including cases the old harness handled correctly.

Preserve the order of events in each simulation. At a decision point, the model receives only the information available at that step. Expected answers, later simulated messages and final transaction state belong to the evaluator's checks.

Metrics that identify useful improvement

Metric	Definition	Engineering use
Verified task success	Attempts satisfying the declared outcome divided by all eligible attempts	Main effectiveness measure
Hard-constraint violations	Attempts with a forbidden action divided by eligible attempts	Release gate
Unsupported completion claims	Checked action claims without supporting state or receipts divided by checked action claims	Measures a specific reliability failure
Tool error rate	Failed tool invocations divided by attempted invocations, separated by error class	Distinguishes interface, policy and infrastructure failures
Human intervention	Attempts requiring operator repair or takeover divided by eligible attempts	Measures operating burden
p50, p95 and p99 latency	End-to-end duration, with queue and provider time recorded separately	Reveals typical and slow-path behavior
Cost per verified success	Total metered cost of all attempts divided by verified successes	Includes the cost of failed attempts
Evidence coverage	Attempts with required observations divided by independently counted attempts	Indicates how much the report can establish

Define eligibility before execution. Timeouts, provider errors and incomplete traces remain in the attempted-case counts. Pending and unknown outcomes remain visible. Missing usage is reported separately from measured spend. A report that omits failed runs from its denominator can make a slower or less reliable harness look better.

Aggregate scores need slices. A booking improvement that damages escalation handling can disappear inside an overall mean. High-volume language groups can conceal regressions in smaller groups. The release report should retain those distinctions.

6. Performance and storage are part of the design

Harness improvements consume resources. Extra context costs input tokens. More tool rounds add latency and opportunities for failure. Detailed evaluation traces create writes and storage costs. Offline evaluation consumes provider budget and worker capacity. These costs belong in the same engineering review as task success.

The performance specification assigns each kind of work an execution boundary and a cost limit:

Work	Execution boundary	Cost control
Authorization, preconditions and required approvals	Before the affected action	Short, indexed state checks
Operational counters, timings and error codes	At defined runtime checkpoints	Bounded fields and recording time
Detailed synthetic evaluation traces	Selected evaluation runs	Sampling and per-run byte budgets
Synthetic case generation and test-result analysis	Background evaluation jobs	Batch, concurrency and spend limits
Candidate comparisons	Isolated evaluation runs	Fixed case, repeat and time budgets

Optional telemetry needs a deadline. Catching a database exception does not prevent a slow connection or lock wait from delaying a customer response. Required authorization and approval records have a different contract: their failure must prevent the dependent action.

The operational counters in this specification measure timing, status and resource use. Synthetic evaluation traces carry the detailed test inputs and outputs. Estimate their storage from evaluation volume and retained evidence size:

Daily evaluation evidence volume
  = recorded evaluation runs per day
  x mean retained bytes per recorded run

Index, replication and backup costs sit on top of that estimate. Sampling by scenario also needs measurement: a small fraction of long scenarios can account for a large fraction of tool calls and retained bytes.

Repeated evaluation context snapshots deserve particular attention. Later rounds often repeat earlier synthetic messages and tool results. Immutable fragments plus ordered manifests can reduce duplication, provided reconstruction and fixture versioning remain reliable.

7. Where harness engineering has limits

A harness can supply a missing fact, expose a clearer action and prevent an invalid write. Some failures persist after those corrections. We classify these separately: ambiguous task requirements, missing business data, unavailable integrations and reasoning failures with sufficient context and working tools. The remedy depends on the category.

The published benchmarks also have boundaries. Coding, question answering and application automation use different success criteria. Our business workflows add tenant policies, human decisions and external systems whose state changes over time. The experiments establish that harness changes can matter substantially; deployment-specific evaluation determines the gain for a particular workflow.

A production release therefore needs both outcome evidence and operating evidence. The task must succeed under the declared rules, and the cost, latency and failure behavior must fit the service it supports.

Conclusion

Harness engineering gives us explicit places to improve an agent: the facts it receives, the procedures it follows, the actions it can request and the checks surrounding those actions. StaffOS uses that structure to support reviewed business configuration and explicit runtime controls.

For each change, the evaluation protocol identifies the failure, reproduces the decision in a controlled case and compares the revised harness with the model held fixed. The release decision includes success, regressions, latency and cost. The case remains in the evaluation set so the failure can be detected again.

Author and citation

Vin Lim is CTO of StaffOS. He works on agent orchestration, business integrations, context management and evaluation.

Suggested citation: Lim, V. (2026). Agent harness engineering: improving AI without fine-tuning. StaffOS technical paper, version 1.1, September 17. https://staffos.xyz/blog/agent-harness-engineering

Download the BibTeX citation · Download benchmark data and source URLs · Figure 1 as SVG · Figure 2 as SVG

Charts reproduce the selected published values. Percentage-point differences use subtraction; the solve-rate multiple uses division. The figures do not pool results across benchmarks. For business context, see our articles on lead qualification and lead response time research.