Skip to content
Kiberon LabsAgentic Engineering Ebook

The Evaluation Harness

Skill Evaluation measures a single packaged skill. Context Evaluation measures a context layer. This chapter covers the level above both: evaluating the whole system. Does the end-to-end pipeline (enrich, plan, implement, review, merge) actually resolve issues correctly, and did a change to any part of it make the system better or worse?

The framing that matters: you are building an eval harness, not just an agent. The MAST study’s conclusion is blunt: “the gap between a good agent system and a bad one is almost never the framework; it is the eval pipeline, the observability setup, and the failure-recovery logic.” Aider’s entire design is downstream of a fixed benchmark. Without a system-level eval, every architectural change is a guess.

Build a private, contamination-resistant eval

Section titled “Build a private, contamination-resistant eval”

Public benchmarks are now benchmark theater. SWE-bench Verified was deprecated over training-data contamination with leaders sitting near 95 percent; a public score is closer to marketing than to a quality signal.

The replacement is a private evaluation set built from your own history: real issues and their merged PRs, drawn from the repositories Ikidna actually works on. Because they are private and post-date model training, they resist contamination. The core metric mirrors Aider’s: did the system’s change apply cleanly and pass the tests without human intervention?

Feed the set continuously, and rank the sources: production failures first (each one becomes a regression case via the capture loop: log it, classify it, reduce it to a minimal reproduction, add it, verify), user-reported problems second, synthetic cases a distant third. An eval set also goes stale, and staleness has signals: a pass rate pinned at 100 percent, production failures the suite never predicted, cases exercising features that no longer exist. Retire and replenish on a schedule, and hold out a slice of cases the team never optimises against, so improvement on the suite still means improvement in the world.

Sources: OpenAI: SWE-bench Verified · MAST (arXiv:2503.13657)

Isolate the variable under test

Section titled “Isolate the variable under test”

The single most important measurement technique, from LongMemEval: an agent’s score collapses when you change retrieval versus when you change reading, and if both move at once you cannot attribute the regression. So:

  • Freeze everything except the part you changed. To test a retrieval or context change, hold the model and prompt fixed. To test a model change, hold context fixed.
  • Oracle vs interactive split. Run each task twice: once with oracle context (only the relevant material handed in) and once interactive (the system must find it amid distractors). The gap between the two, often 30 to 60 points, is precisely the cost of your retrieval and context-management layers, isolated from raw model capability.

Statistical hygiene is part of isolation. Models are probabilistic (see Working With Models), so single runs are anecdotes: repeat runs and report variance, pin exact model versions for the duration of a comparison, and version the rubrics themselves so a graded number means the same thing next quarter.

Source: LongMemEval (arXiv:2410.10813)

Tier the suites, and analyse errors by hand

Section titled “Tier the suites, and analyse errors by hand”

An evaluation harness that only runs nightly is a post-mortem tool. Structure the suites by feedback speed, so the right check runs at the right moment: a smoke tier of a handful of cases that runs in under a minute on every prompt or config change; a core tier of a few dozen cases before anything merges; the full suite on a schedule. This mirrors the verification-speed argument in Verification one level up.

When the numbers move, the highest-value activity is not more automation; it is reading the traces. The discipline, due to Hamel Husain: sample a meaningful batch of failures, annotate each in free form, then group the annotations into categories and count them, iterating until new traces stop producing new categories. The counts tell you where the system actually fails, which is routinely not where intuition says. Two annotation rules make the labels trustworthy: a single accountable domain expert beats a committee, and binary pass-fail judgments beat scales, because “3 out of 5” hides the disagreement that “fail, because X” exposes.

Decompose the pipeline; A/B every change

Section titled “Decompose the pipeline; A/B every change”

Evaluate components, not just the whole. Each stage of the pipeline has its own quality question and its own sub-metric: enrichment completeness, plan quality, implementation pass rate, review catch rate, compaction recall. Ablate each so a system-level regression can be traced to a stage.

Then gate changes: every memory, retrieval, routing, or prompt change runs as a with-vs-without A/B against the current baseline before promotion. The recurring lesson from the memory-systems literature is that most additions help on a demo and regress on distractor-heavy real tasks; the A/B is what catches that.

What to measure

Section titled “What to measure”

Tie system evaluation to the goal KPIs from Minimal Goals, grounded in objective state rather than model opinion:

  • Task success rate: change applied and tests pass without human help.
  • Recovery rate: failures the system recovered from autonomously.
  • Review acceptance rate and regression rate: quality as seen by the next stage.
  • Cost per successful outcome and latency to completion: economics, per Smart Routing. Cost per successful task, not cost per run; a cheap agent that fails often is expensive.
  • Autonomy rate: share of goals completed without human intervention.
  • Abstention: does the system correctly recognise an unworkable or underspecified task and decline or escalate, rather than confidently producing a wrong result? This is the ability most systems fail and the one that prevents confident-but-wrong delivery.

Alongside capability, measure reliability as its own axis: consistency across repeated runs, robustness to paraphrased inputs, calibration (does stated confidence track actual success), and safety violations, which are a blocking count, not an average. Published reliability work finds these improve far more slowly than headline accuracy, which is exactly why the Controls deployment thresholds are stated in these terms.

LLM-as-judge, bounded

Section titled “LLM-as-judge, bounded”

For the dimensions that are not mechanically checkable (plan quality, review usefulness), use a single-call, explicit-rubric judge (see Verification), never as the primary signal for whether code works, and always paired with human spot-checks, because automated evals miss emergent failures. Calibrate the judge against a set of human-labelled cases before trusting it, re-calibrate when the judge’s model changes, and test it for the documented biases: length, style, position, and recency. Start with roughly 20 representative cases; early iterations move fast, so even small sets are informative.

Integration

Section titled “Integration”

System evaluation is the observe step of the Context Development Lifecycle operating at fleet scale, and it consumes the Execution Ledger as its data source; ledger traces are the eval corpus. It runs on a schedule (an Orchestration Trigger heartbeat) and on every significant change, feeding regressions back into the Minimal Bootstrap experimentation loop.

Section titled “Related chapters”