A six-part engineering guide: from individual prompts to specification-driven probabilistic software.
Read: Prompt engineering is becoming programming →Scale: multi-agent systems and feedback control
Distributed-systems problems, evaluator loops as feedback controllers, persistent state, and routing work to the cheapest reliable executor.
Multiple agents create distributed-systems problems
The moment two workers run at once, forty years of distributed-systems engineering applies.
Each worker owns a clear task or resource.
Structured messages, not uncontrolled shared context.
Every action belongs to a traceable task.
Follow a request across all workers, tools and state.
No worker may run indefinitely.
Stop repeatedly failing tools or services.
A failing worker must not collapse the system.
Limit new work when the system is overloaded.
Multiple evaluations when one judgment is unreliable.
Define how competing edits merge or get rejected.
Hard rule: do not allow several workers to edit the same files simultaneously without ownership, locking or merge rules.
The moment work is split across several agents, you inherit the classic distributed-systems problems, including race conditions, conflicting edits, stale context and duplicated effort, but without the mature tooling that exists for conventional distributed systems. Throughput rarely scales with the number of agents, while the cost of merging and reconciling their work grows quickly. This is the pattern where impressive demos most often fail to survive real workloads.
Evaluator-optimizer loops are feedback controllers
Repeated switching between two solutions.
Fixing one criterion breaks another.
Small repairs cannot reach the desired result.
Passes the grader without satisfying the real goal.
The loop never declares completion.
A generate, evaluate and repair cycle is a feedback controller, and it fails the way controllers fail: oscillating between two solutions, overcorrecting so one fix breaks another criterion, settling into a local optimum, satisfying the grader without meeting the real goal, or simply never declaring completion. Each iteration costs money and elapsed time, so these failure modes are budget events, not curiosities. The controls are ordinary: a target, a cap, a budget and explicit stop conditions.
Long-running agents need more than conversation history
A long-running agent should resume from explicit state, not reconstruct its progress from a long chat transcript.
Long-running work must resume from explicit, stored state. When progress lives only in a growing transcript, cost rises with length, older decisions get crowded out and quietly forgotten, and an interruption loses work that has already been paid for. Checkpointed state makes a run resumable, auditable and cheap to continue, and it is the difference between a process that survives a deployment and one that has to start again.
Use the cheapest reliable executor
Not every message should become an autonomous agent task. Route each decision to the lowest layer that can carry it reliably.
Every decision should be routed to the cheapest layer that can carry it reliably: deterministic code first, then a small model, then a strong model, and a human only where judgement or accountability genuinely requires it. Sending everything to the largest available model is the most common reason a promising feature has per-request costs that cannot scale, and it also adds latency users notice. Routing is a design decision with a direct margin impact.
More articles
Design reliable UX for connected devices across hardware, apps, BLE, Wi-Fi, cloud, onboarding, errors, and recovery.
Read: UX for Connected Devices: Designing Experiences Across Hardware, Apps, Connectivity, and Cloud →Hardware companion apps: hard parts, partner skills, and questions that separate specialists from agencies.
Read: How to Choose the Right Companion App Partner for Your Hardware Product →