Designing Resilient Inter-Agent Communication Protocols in Cloud Environments

An agent is not a microservice, even when it talks over the same HTTP. A microservice call costs you ten milliseconds and a row in a database. An agent call costs you four seconds, a dollar of tokens, and a non-deterministic answer that might be wrong in a way no status code reports. So the layer between agents has to assume the thing on the other end is slow, expensive, and occasionally confidently incorrect.

The way I build this now: every message between agents is a durable event on a queue, and every agent reads its work from that queue, not from a request.

A planner doesn't call a researcher and wait. It writes a task.research.requested event with a task ID, drops it on the bus, and moves on. The researcher picks it up when it's ready, does the work, and writes back task.research.completed carrying the same task ID. The planner is subscribed to that. Nobody holds an open connection. Nobody is blocked for a four-second model round trip before they can do anything else.

This buys the property that matters most: the work survives the process. If the researcher dies mid-task (out of memory, rate-limited, redeployed during a release), the message is still on the queue. Another worker takes it. In a synchronous chain, that same crash takes down everything upstream that was waiting on it.

The design I reject

The obvious version is direct calls. Planner does POST /research, gets JSON back, calls the writer with it, returns. It reads top to bottom like a normal function, and that legibility is the whole reason it stays the default.

I reject it for one specific reason: failure in a synchronous chain is not isolated, it's cumulative.

Picture three agents in a row. Planner, researcher, writer. The researcher takes nine seconds because the model is under load. Your gateway timeout is thirty, fine. But the writer also takes nine seconds, and somewhere a load balancer idle timeout sits at fifteen. The connection dies between agent two and agent three. Now the researcher's output is gone, because the only place it ever lived was an HTTP response that just got severed. That was the dollar you already spent and a result that was actually correct.

Retry the chain and you pay for the research again. And the planning again. You don't resume from the failure point, because no failure point was recorded anywhere. The state of the work was the call stack, and the call stack evaporated.

That's the real trade. Synchronous calls give you simple code and lose all in-flight work on the first timeout. Event-driven gives you uglier code — correlation IDs, idempotency keys, a subscriber that reassembles state — and keeps the work.

What you actually build

The hard part isn't the queue. It's that agents retry, retries duplicate, and a duplicated agent action can be expensive or destructive. Charge a card twice, send two emails, run two paid generations. So idempotency stops being a nicety. Every message carries a task ID, and every consumer checks "have I already handled this ID" before it acts. When the side effect costs a dollar, that check is not optional.

The second rule does most of the remaining work: completion events are facts, not requests. research.completed means it is done and the result is stored somewhere durable, keyed by task ID. A consumer can replay it ten times and the system lands in the same place. The event is a record of something that happened, not an instruction to make something happen.

Get those two right and a crashed agent is a non-event. The message waits. Someone picks it up. The user never sees it.

We moved the planner, researcher, and writer onto the queue a week before a release that, mid-deploy, killed three workers in the space of a minute. Under the old setup that would have been a batch of lost research and a refund ticket. This time the messages just sat there until the new workers came up and drained them. I found out it had happened from a log line the next morning. Nobody had filed anything.