Skip to content
Kiberon LabsAgentic Engineering Ebook

Smart Routing

Description

Section titled “Description”

Smart routing is the repository’s first explicit optimization strategy: match work with the cheapest model and execution path that preserves answer quality, escalating to expensive frontier models only when complexity, risk, or confidence demands it. Routing is not arbitrary it is driven by signals produced upstream and enforced at dispatch.

It acts on the capability of an Agent, since model choice is one of the core factors shaping agent behavior, and it extends beyond model selection to retrieval strategy selection (focused vs exhaustive see Routing Heuristics).

Routing Signals

Section titled “Routing Signals”
  • Complexity classification from Ticket Enrichment the primary input.
  • Runtime feedback from prior similar tasks what model class succeeded on this kind of work before.
  • Domain sensitivity code ownership, compliance boundaries, and information-leak risk that may force self-hosted execution.
  • Budget envelopes for a team, repo, or workflow stage (see Guardrails).
  • Human escalation preference when stepping up to a stronger model is permitted.

Routing Mechanisms

Section titled “Routing Mechanisms”

The patterns that work in production:

  • Classifier + rules a lightweight classifier or explicit heuristics route most traffic; add model-based routing only when heuristic precision is insufficient (mirrors the Adoption Sequence: start simple, escalate by evidence).
  • Cascading try a cheap model first, escalate to a stronger one on low confidence or failed verification. This is the planner/implementer/validator role split applied as a cost ladder (see Model Usage).
  • Effort-scaling rules bound how much compute a task may allocate to itself. Without explicit rules, agents over-allocate on trivial work; the rule (“simple task → one agent and few tool calls; complex → escalate”) belongs in the orchestrator, not the agent’s discretion.
  • Semantic caching cache answers on semantic similarity, not exact match. A cache hit is milliseconds versus a multi-second model round-trip and is a major spend reducer where queries repeat.
  • LLM gateway centralise routing, caching, failover, and cost tracking in one layer. This is the role of Ikidna’s Bifrost gateway, which also enforces Guardrails and provider abstraction.

Two Empirical Anchors

Section titled “Two Empirical Anchors”
  • Model upgrade beats token budget on hard tasks route hard work to a better model rather than giving a weaker one more tokens.
  • True multi-agent fan-out costs ~15× route to fan-out only for provably breadth-first work (see Coordination Model).

Measuring Routing

Section titled “Measuring Routing”

Routing decisions and their outcomes must be logged to the Execution Ledger: route distribution, latency and cost by route and complexity class, and quality outcomes by route. False-route cases (a hard task sent to a cheap model and failed, or an easy task over-served) are reviewed and fed back into policy routing is itself a continuously evaluated subsystem (see System Evaluation Harness).

Section titled “Related Concepts”