Working With Probabilistic Models

Language models are probabilistic components. The same input can produce different reasoning paths, tool choices, and answers. Reliable agent systems account for that variance instead of treating model output like deterministic program output.

Errors compound

A model that completes one step correctly 95 percent of the time completes ten dependent steps correctly about 60 percent of the time. Across one hundred dependent steps, the probability falls below one percent. This simple arithmetic explains why long autonomous runs can fail even when every individual action looks reasonable.

Per-step reliability matters more as a workflow grows. The system should therefore favour short, verified sessions, as developed in Runtime Context Management, and machine-checkable outcomes, as developed in Verification.

Two practical consequences:

• Control variance where it matters. Sampling randomness compounds across steps. Run orchestration and long workflows at temperature zero or near it; isolate creative generation, where variance is the point, from reliability paths.
• Prefer shorter chains. Every removed dependency improves end-to-end reliability. Break long work into bounded stages with explicit handoffs and verification.

The limitations catalogue

Model failures are not random; they are systematic, which means each has an engineering workaround. The recurring ones:

• Arithmetic: delegate calculations to a tool. Confidence does not correlate reliably with correctness.
• Hallucination: risk rises when the model synthesises from partially relevant retrievals. Search, capture evidence, cite it, and keep inference visibly separate from source material.
• Context degradation: capability falls as the window fills, long before the hard limit. Treat roughly 40 to 60 percent utilisation as the working ceiling and act (compact, or start a fresh session) before quality visibly sags. The full discipline is in Runtime Context Management.
• Instruction drift: instructions fade over long sessions. Anchor critical constraints at the start and end of the prompt and reinforce them periodically.
• Format failures: even with explicit schemas, expect a residual failure rate. Validate structured output programmatically and regenerate on failure rather than parsing hopefully.
• Version instability: the same prompt can behave differently across model versions. Pin exact versions in production and run behavioural regression tests before upgrading.

These limitations are predictable enough to engineer around. They also determine how a team should choose and evaluate models.