Runtime Context Management

Description

The foundational Context model is about authoring quality density, hierarchy, curation, evolution. This page covers the complementary problem: keeping context coherent and high-signal at runtime, while an agent is executing, over horizons that exceed a single context window.

These are different disciplines. Curation decides what is true and how it is written. Runtime management decides what is in the window right now, what gets evicted, what gets summarised, and how a task survives across the boundary where the window fills up. A system can have perfectly curated context and still fail because it loaded all of it at once.

Context Rot: The Root Constraint

The advertised context window is not a usable working set. As the number of tokens grows, the model’s ability to recall any individual fact degrades even on frontier models, even well under the stated limit. This is context rot, and it makes context a finite, depletable resource rather than free capacity.

The governing goal, restated from Context’s density constraint but applied to the live window: the smallest set of high-signal tokens that supports the next action. More context is not monotonically better; past a point it is worse.

Source: Anthropic Effective Context Engineering

The six Runtime Levers

These are distinct mechanisms and should be reasoned about separately:

Offloading. Keep work out of the window. Write intermediate results, plans, and progress to files (a NOTES.md / progress file, the repo itself, the Execution Ledger); hold only pointers in context. The filesystem is the agent’s unbounded memory; the window holds pointers plus current focus.
Just-in-time retrieval. Store lightweight identifiers (file paths, URLs, queries, symbol names) and load contents at the moment of need rather than front-loading. A hybrid a little context up front, the rest pulled during execution works best in practice.
Isolation (sub-agents). Give a focused subtask a clean context window and have it return only a condensed summary (1–2k tokens). This is the safe, read-only use of parallelism from the Coordination Model: sub-agents answer questions; they do not write shared code.
Compaction (reduction). When the window approaches its limit, summarise the history and reinitialise. The rule that matters: maximise recall first, then precision preserve architectural decisions, open problems, and unresolved bugs; discard redundant tool output. The cheapest first move is simply clearing stale tool results.
Caching. Prompt-prefix caching is an orthogonal cost/latency lever, not a context-quality lever. Stable prefixes (system prompt, org context) should be cache-friendly, but caching does not reduce rot.
Discard: If compaction is too complicated to do, or there is low confidence in preserving the signal, instead prefer to discard the context if it is already of low quality and restart.

Compaction Is Hard and Lossy

Both Cognition and Anthropic flag compaction as the hard, under-solved part “more art than science.” Naive summarisation silently drops the exact decision you will need twenty steps later. Cognition’s prescription is a dedicated compression model (ideally fine-tuned) whose only job is to distill history into key details, events, and decisions treated as a first-class component, not an afterthought.

This is a differentiation opportunity, because nobody has solved it cleanly. A decision-preserving, recall-first compaction strategy and an eval that measures compaction quality specifically (see System Evaluation Harness) is genuinely valuable infrastructure. The likely solution, however, is to get another LLM to attempt to compact the original one, one with a larger context window and chunking. The obvious side effect of this is an increase in latency while waiting for compaction, cost implications, and the nonzero probability that the compaction still results in a low-quality result .

Sources: Cognition Don’t Build Multi-Agents · Anthropic Effective Context Engineering

Long-Horizon Means Short Sessions

The most important operational consequence: long-horizon work is many short, well-scoped sessions not one long-running session. The evidence is consistent:

Devin’s performance degrades past roughly 10 Agent Compute Units; large projects are split into small isolated sessions.
Anthropic’s long-running-harness work used one feature at a time, a git commit per feature, and a progress file reviewed at session start to survive context resets.
Long-horizon coding benchmarks measure quality decay across many iterative turns.

The pattern is checkpoint → fresh context → resume from notes/git, not “keep the agent thinking for hours.” This connects to the Orchestrator’s durable execution model (resume from the failure point) and to the Coordination Model’s scope-small principle.

Sources: Cognition How Cognition Uses Devin to Build Devin · Anthropic Effective Harnesses for Long-Running Agents

Structured Note-Taking as Working Memory

External, agent-written notes are what buy multi-hour coherence not a bigger window. A progress file that records what is done, what is failing, what was decided, and what is next lets an agent recover its own state after a context reset, and lets a successor agent pick up without re-deriving everything. This is the runtime counterpart of the persistent Knowledge Base: the KB is durable org/project knowledge; the progress file is per-task working memory.

Initializer-vs-worker is a useful split here: the same harness and prompt behave differently on first run (scaffold init.sh, write the failing-requirements feature list, make the first commits) versus resume (read the progress file and git history, continue the next failing requirement).

Relationship to Authored Context

Authored context (Context, Layers)	Runtime context (this page)
What is true; how it is written	What is in the window now
Density, hierarchy, curation, versioning	Rot, compaction, offloading, eviction
Owned by teams / context engines	Owned by the agent loop and Orchestrator
Resolved at dispatch via Layers Declare/Diff/Inject	Managed during execution across window boundaries

The two compose: dispatch resolution decides the initial load; runtime management keeps it healthy as the task runs.

Context authoring-side density and curation.
Context Layers dispatch-time resolution.
Coordination Model sub-agent isolation and scope-small.
Knowledge Base durable cross-session memory.
Orchestrator durable execution and resume.
System Evaluation Harness measuring compaction and retrieval quality.