Notes from building agent systems
Evaluation harnesses, orchestration patterns, memory design and the failures behind them — written up while the details are still fresh.
Agent Fleets as Discovery Instruments: The Evaluation Problem Behind Patch the Planet
Fleets of agents change the evaluation question from precision per agent to yield per compute-hour under a false-positive budget. Most agent benchmarks do not measure that.
Tracing Non-Determinism: What Concordance and DeepEval Reveal About Agent Evaluation
Inconsistent agent decisions are usually a missing-input problem. Evaluation should record and replay the full decision context, not score outputs after the fact.
Eval Sets Rot: Versioning the Harness, Not Just the Model
Self-improving agents that memorise their tests reveal a pipeline defect: eval sets are data artifacts with the same lifecycle obligations as code.
The Benchmarks Are Fine; the Harness Is the Variable You Did Not Version
A leaderboard score is a joint outcome of model, tool schemas, retry policy and runtime limits. Publish the harness or the number means nothing.
The 8.3x Usage Gap Is a Workflow Fit Problem, Not a Licensing Problem
Wide deployment against narrow usage points to agents attached to workflows they were never scoped for. The fix is workflow decomposition before model selection.
Agent Evals Are Incident Replays: Building a Regression Corpus From Production Traces
Agent evaluation suites should be grown from captured production trajectories and incident replays, not authored from imagined tasks, because the failures that matter are discovered, not designed.
What Actually Breaks When You Scale an LLM Agent Swarm
Multi-agent systems fail less from model capability than from coordination mechanics. A look at shared state, handoff protocols, and the trade-offs of parallelism.
Interrogating the Model Is a Review Method, Not a Personality Trait
Treating agent output as a claim to be interrogated, not a result to be accepted, is a repeatable evaluation practice teams can institutionalise.
Benchmark Contamination Is a Data Pipeline Bug: Versioning Eval Sets Like Code
Agent evaluation failures blamed on model weakness are often eval-set hygiene failures. Eval corpora need the same provenance, versioning and access controls as training data.