Notes from building agent systems
Evaluation harnesses, orchestration patterns, memory design and the failures behind them — written up while the details are still fresh.
Telemetry Cost Is an Agent Design Variable, Not an Infrastructure Bill
Agent workloads multiply observability spend because every step emits traces, logs and metrics. The fix is a sampling and retention policy designed alongside the agent.
Decision-Scoring Models Do Not Remove the Policy Register
Scoring models compress policy into weights, but they do not replace the versioned decision register that makes scoring governable.
Evidence Models and Decision Models Are Different Artifacts
Deciding and judging are separate design roles. When one model does both, evaluations inherit the actor's blind spots and stop being trustworthy.
Decision Models Are a Latency Budget, Not a Governance Layer
Small decision models cut per-step latency, but they resolve routing, not authorisation. Conflating the two erases the audit trail that incidents require.
Decision Models Are Not Policy Engines: What a 2B Router Can and Cannot Enforce
Cheap decision models like Strands Decider and Clef are routing primitives, not control surfaces. Treating a fast classifier as an enforcement point moves the failure mode rather than removing it.
On-Device Arbitration: What an Agent Should Never Send Upstream
The choice of which agent steps leave the handset is a design-time data-classification and latency trade-off, not a runtime privacy filter.
Prediction-Market Agents Need a Resolution Ledger, Not a Confidence Score
When an agent places a position, its probabilistic output becomes a settlement obligation. The artefact that matters is a per-claim resolution ledger with provenance, not a calibrated number.
The Human Concierge Is a Fallback Path, Not a Feature
Routing agent failures to a human only works when the escalation contract is explicit: what state is handed over, what the operator may change, and how the correction returns to the agent.
Co-Scientist Agents Need a Claim Ledger, Not a Chat Log
Conversational transcripts are not a scientific record. Research agents should emit structured, checkable claims with evidence links and explicit confidence.