Skip to content
Kiberon LabsAgentic Engineering Ebook

Skill Evaluation

Purpose

Section titled “Purpose”

Skill evaluation is the process of measuring whether a skill package is doing its job: producing better, more consistent agent behavior for the task class it targets. Without evaluation, skill quality is invisible changes can degrade performance silently, model updates can break previously reliable behaviors, and skill coverage gaps stay undetected until they surface as production failures.

This page defines the generic process. Specific tooling options are covered in the software catalog: Promptfoo and Tessl.

When to evaluate

Section titled “When to evaluate”

Evaluation should happen at three points in the skill lifecycle:

  1. Before promotion a new or modified skill is evaluated before it reaches agents in any active environment
  2. On model change when the underlying model version or provider changes, existing skills should be re-evaluated since behavioral contracts can silently shift
  3. On schedule even stable skills should be evaluated periodically to detect drift from context changes, shifting task distributions, or gradual model updates

Evaluation dimensions

Section titled “Evaluation dimensions”

A complete skill evaluation covers four dimensions:

Correctness

Section titled “Correctness”

Does the agent, using this skill, produce outputs that satisfy the task objective?

  • Define expected outcomes for a representative set of task scenarios
  • Measure pass/fail rate across those scenarios
  • Use LLM-as-judge, assertion checks, or semantic similarity depending on the output type
  • Track correct answer inclusion for factual or retrieval-dependent outputs (maps to Context Evaluation correctness metric)

Consistency

Section titled “Consistency”

Does the skill produce stable outputs across runs, models, and prompt variations?

  • Run the same scenario set multiple times to detect non-deterministic failure modes
  • Test the skill across at least two model variants to identify provider-specific weaknesses
  • Vary prompt framing within the skill’s activation scope to check for fragile wording dependencies

Efficiency

Section titled “Efficiency”

Does the skill avoid wasteful token use?

  • Measure output verbosity against task requirements (maps to Context Evaluation verbosity metric)
  • Measure information density: how much actionable guidance is encoded per token of skill content
  • Identify dead weight: skill sections that add tokens but do not improve outcome quality

Coverage

Section titled “Coverage”

Does the skill handle the full scope of the task class it claims to address?

  • Enumerate known sub-cases and edge cases for the task type
  • Test failure modes: what happens at the boundary of the skill’s intended scope?
  • Document uncovered cases as explicit gaps rather than leaving them implicit

Evaluation artifacts

Section titled “Evaluation artifacts”

Each evaluated skill should produce:

  • Scenario suite the set of test inputs and expected outputs used for evaluation, versioned alongside the skill
  • Results report pass/fail summary per dimension, with per-scenario detail for failures
  • Drift log a record of how results change across evaluations over time
  • Gap register known uncovered cases and out-of-scope behaviors

These artifacts travel with the skill. When a skill is distributed, its scenario suite should be distributable too so consumers can re-run evaluation in their own environment.

Evaluation pipeline integration

Section titled “Evaluation pipeline integration”

Skill evaluation should be treated as a first-class step in the skill release pipeline, not a manual one-off:

  1. Skill change is committed
  2. Evaluation suite runs automatically (CI gate)
  3. Results are compared against the previous baseline
  4. Regressions in correctness or consistency block promotion
  5. Efficiency and coverage gaps are reported as non-blocking warnings
  6. Passing results trigger promotion to the distribution layer

This maps to the evaluate step of the Context Development Lifecycle.

Relationship to context evaluation

Section titled “Relationship to context evaluation”

Skill evaluation is a specialisation of Context Evaluation. The same core metrics apply correctness, verbosity, information density but skill evaluation applies them to a bounded, packaged context artifact rather than to a context layer as a whole.

The output of skill evaluation feeds back into the broader context quality loop: skills that consistently underperform are candidates for rework or decomposition; skills that consistently overperform their scope are candidates for generalisation into shared context.

Section titled “Related”