Skip to content

Swarm Operations

The earlier chapters of this part establish what a swarm is and how its agents coordinate. This chapter is about keeping one running: the operational patterns that appear, in system after system, once agent count crosses a few thresholds. None of them are hypothetical; each has been validated in public production systems, and each maps cleanly onto primitives this book has already established.

Operating patterns arrive in bands, and deploying them too early is as costly as too late:

  • One to three agents: direct human monitoring works. No infrastructure beyond version control.
  • Three to ten: a merge strategy becomes mandatory, plus shared conventions and CI gates. Useful work per agent starts falling as coordination overhead grows.
  • Ten to twenty: automated quality gates, attribution logging, and supervision layers pay for themselves.
  • Twenty and beyond: full orchestration, health monitoring, and typed messaging are load-bearing; reported operating costs run on the order of a hundred dollars an hour, and a human can meaningfully watch only a few agents directly.

The transition is a phase change, not a slope: workshop habits (watching sessions, reading every diff) simply stop functioning past the second band. The signals you have crossed a threshold are concrete: merge conflicts between agents, failures nobody noticed, a human review queue that never drains.

Watching agents is itself work, and it decomposes into tiers that mirror the Controls philosophy of spending judgment where it matters:

  1. Mechanical watchdogs (seconds): heartbeats, token flow, time and resource bounds. No reasoning, just liveness. This tier never thinks.
  2. Triage (fast, cheap model): is output coherent, is progress real, classify the failure, restart if safe.
  3. Supervision (mid-tier model): per-task quality, rework decisions.
  4. Patrol or human (slow, expensive): strategic review, threshold tuning, spec correction.

Worker state deserves the same discipline. A worker is working, stalled (distinguish thinking from looping), or a zombie: alive, producing output, meaningless, usually because its context degraded past usefulness. The operational rule that follows from Working With Models: terminate and respawn at high context utilisation, because a restart is cheaper than zombie work. And idle is not a state; a finished worker cleans up (push branch, emit completion message, destroy sandbox, terminate) rather than lingering.

Coordination is typed, and writes stay serialised

Section titled “Coordination is typed, and writes stay serialised”

Past a handful of concurrent writers, ad-hoc coordination corrupts. The stable answer is a small set of typed messages (task complete, merge ready, rework request, help request, handoff, status update) over a durable substrate, with idempotent delivery. This is the Execution Ledger doing coordination duty, and the anti-pattern it forbids is retroactive batch status updates, which destroy the causal ordering that debugging and audit depend on. Status changes are recorded at the moment they happen or the record is fiction.

Integration is where the coordination model’s rule (reads parallelise, writes serialise) becomes machinery: a merge queue admitting one change at a time, with escalating conflict resolution. Clean merge first; mechanical auto-resolution next; an agent attempts semantic resolution after that; and when a conflict is beyond resolving, the change is re-implemented against current main rather than force-merged. Quality gates sit at every integration point because reliability multiplies: twenty workers at 95 percent per-task reliability produce a steady stream of defects, and trust without verification scales the defects with the fleet.

Sessions are ephemeral by design; identity should not be. Persistent agent identity (a stable name, an append-only work history, accumulated domain expertise) survives session restarts and even model upgrades, and enables three things ad-hoc fleets cannot do: routing work to the agent whose record fits it, performance management from actual outcomes (gate pass rates, rework frequency, cost per task), and accountability in the ledger. The expertise itself accrues under the update discipline described in Skills and the Knowledge Base: executing agents read expertise but never write it; a single improvement pass, after execution, folds what the fleet learned back in. Learning and doing stay separated for the same reason writes serialise.

Two operator anti-patterns recur everywhere. Trust creep: gates get relaxed because the last twenty runs were clean; autonomy granted on a streak rather than on the risk tier of the action. Safeguard depth must track blast radius, not recent luck, which is the Controls argument restated operationally. Design starvation: keeping agents busy with vague issues because idle capacity feels wasteful; the swarm then manufactures plausible, wrong work. The upstream fix is Ticket Enrichment, and the operator’s real job distribution ends up weighted toward design and specification, not monitoring.