Skip to content
Kiberon LabsAgentic Engineering Ebook

Controls & Autonomy

Every agent system answers one question, whether its designers ask it or not: how much behaviour is delegated to agents, and how much is constrained by structure, guardrails, and human review? In practice the answer sits on a sliding scale between full control and full autonomy, and the design work is deciding where each kind of action sits on it.

At the full-control end, agent invocation is limited, cheap, and heavily constrained: deterministic behaviour, explicit workflows, reproducibility, a human in the loop for important decisions. The goal is not creativity but reliable execution inside known boundaries. At the full-autonomy end, agents decide more of their own structure, sequencing, and interaction; that buys adaptation and discovery on ambiguous work, at the cost of predictability, traceability, and spend. Autonomy pushes more decisions through token-priced model calls; control moves reproducible work into workflows and deterministic services where it is cheaper and safer.

Autonomy is per-action, not per-agent

Section titled “Autonomy is per-action, not per-agent”

The crucial refinement: autonomy is not a global setting on an agent. It is a property of each action, tiered by risk. A single agent run should execute routine, reversible actions freely while gating the consequential, irreversible ones. The design question is where the gates sit, not how autonomous the agent is overall.

What determines an action’s tier? Five axes cover practice: reversibility (undoable in minutes, or permanent), blast radius (a scratch branch, or every customer), cost, sensitivity (internal code, customer data, credentials and payments), and precedent (routine, or the first time this class of action has run). The first three compound multiplicatively; a useful shorthand is that required supervision scales with blast radius times irreversibility times novelty. Recurring, reversible, narrow actions earn telemetry-gated autonomy; novel, irreversible, broad ones demand a human.

A working tiering for software delivery:

TierExamplesControl
Low (reversible, low blast radius)read code, run tests, search, draft a plan, write to a scratch branchno approval; run free
Medium (consequential but recoverable)open a PR, create issues, push to a feature branchpost-execution review; surface for inspection
High (irreversible or production-touching)merge to main, force-push, deploy, run migrations, edit CI or secrets, delete branchesmandatory pre-execution approval gate

Gate patterns

Section titled “Gate patterns”

Three patterns, chosen per action class:

  • Pre-execution approval: pause before a high-tier action and wait for a human or policy to approve. Used for irreversible and production actions. A good approval request states the action, the context, the risk and its mitigation, and, for anything high-tier, the rollback plan; a gate without a rollback plan is asking the approver to sign blind.
  • Post-execution review: act, then surface the result for inspection before the pipeline advances. Used for medium-tier actions like opening a PR.
  • Escalation triggers: run free, but halt automatically on a risk signal: untrusted input just ingested, confidence below threshold, an irreversible operation detected, anomalous behaviour. Used to make low-tier autonomy safe.

Every gate that waits on a human must carry a timeout and SLA action (escalate, auto-reject, or re-route) so a pipeline cannot stall forever; see Orchestrator durable execution.

Gates also fail in the human direction. Gate fatigue is the operational killer: approve enough routine requests and approval becomes a reflex, at which point the gate protects nothing. The remedy is tiering discipline itself: gate only genuine risk, auto-proceed everything reversible and observable, and treat a gate that approves at near-100 percent as a candidate for demotion to post-execution review.

Guardrails must be structural

Section titled “Guardrails must be structural”

The hard lesson from the field: high-consequence constraints expressed as prompt instructions (“don’t touch production”, “ask before deploying”) do not hold under adversarial input or long-horizon pressure. The Replit incident, an agent that ran destructive commands against production despite an instruction not to, then fabricated data to hide it (see Verification), is the canonical failure.

Controls must therefore be enforced by architecture, not by the model: environment separation so an agent physically cannot reach prod, a planning-only mode that produces a plan without execution capability, action gates the agent cannot self-grant, and the egress and credential constraints in Security & Prompt Injection. The Execution Ledger provides the evidence trail for every gated and ungated action alike. Operational guardrails on the run itself remain valuable (see Guardrails): maximum reasoning steps, time and token budgets, retry limits, and HITL escalation.

Progressive autonomy, with numbers

Section titled “Progressive autonomy, with numbers”

Autonomy should be earned. Grant an agent class more freedom on a task type as it demonstrates success on that type, and pull it back when failure or regression rates rise.

“Demonstrates success” should be measured, not felt. The practice this book endorses, adapting published reliability-gating work (Rabanser et al., arXiv:2602.16666): define the two operating modes explicitly and hold them to different bars. Augmented operation (human in the loop) is justifiable at moderate consistency with zero high-severity violations; automated operation (no human in the loop) demands near-perfect consistency across repeated runs and paraphrased inputs, zero violations of any severity that matters, and calibrated confidence, so the agent’s own uncertainty is usable as an escalation trigger. The precise thresholds matter less than their existence: a written, per-task-class bar converts “we trust it now” from a feeling into a deployment decision, and the Evaluation Harness is what measures against it.

The inverse failure has a name worth knowing: trust creep. Gates get relaxed because the last twenty runs were clean, autonomy granted on a streak rather than a risk tier. Safeguard depth must track blast radius, not recent luck; a clean streak on low-tier actions is not evidence about the high-tier ones. Swarm Operations covers the fleet-scale version.

Section titled “Recommended direction”

The objective is not to choose one extreme permanently, but to segment work by control and risk profile. High-risk, regulated, high-cost, or well-understood processes sit closer to the control end. Ambiguous, exploratory, design-heavy processes tolerate more autonomy when the value of discovery outweighs the extra cost and uncertainty.

Over time, the system should move work away from full agency as reproducible boundaries become clear. Once a task or interaction pattern is well understood, it is progressively constrained, formalised, and optimised so that less of it depends on open-ended agent reasoning. Autonomy is the mechanism for discovering the process; control is the mechanism for operationalising it. That movement is itself a self-improvement goal of the factory (see Minimal Bootstrap).

Section titled “Related chapters”