How one agent hands work to another: the A2A delegation protocol, the queue, and surfacing results

When you have one agent, “do the work” is a loop: read the task, act, reply. When you have an organization of agents, the interesting verb is delegate. A lead agent breaks a job into pieces and hands them to peers; those peers may delegate further; eventually results have to flow back up and actually change what the originating agent does next. That is agent-to-agent communication — A2A — and most of the hard engineering is not in the “send” but in everything around it: routing, queueing under load, and waking an agent when an answer it asked for hours ago finally arrives.

This is a write-up of how that machinery actually came together in molecule-core, including the bugs that taught us where the boundaries were.

The shape of a delegation

An agent delegates by calling a tool — delegate_task for synchronous work, delegate_task_async for fire-and-forget — with a target workspace ID and a task string. That call does not open a socket to the peer. It goes through the platform: the workspace server proxies the request as an A2A message (POST /workspaces/:id/a2a), the platform resolves the target, and the response travels back the same way. Centralizing on a proxy is what lets the control plane enforce who-may-talk-to-whom, log every exchange, and buffer when the target is busy.

That single indirection is also where a surprising amount of subtlety lives, because there are now two different on-the-wire shapes in play. A direct A2A message uses JSON-RPC: body.params.message.parts[].text going out, body.result.parts[].text coming back. But the delegation tool stores its own envelope — {"task": "...", "delegation_id": "..."} on the request and {"text": "...", "delegation_id": "..."} on the response. Early on the canvas only knew how to read the JSON-RPC form, so MCP-initiated delegations rendered as blank bubbles and “Delegation dispatched” with no body. The fix was to teach the extractors both shapes (core #167, core #171). The lesson that stuck: a delegation has a protocol envelope and a tool envelope, and every consumer — canvas, logger, result-poller — has to handle whichever one produced the row.

We also learned to never trust the response shape blindly. When the proxy returned {"error": "plain string"} instead of the expected object, a single data["error"].get("message") raised AttributeError and broke every delegation on that path (core #273). Defensive parsing at the boundary is not optional when the other side is another agent’s runtime.

The queue, and the priority of being busy

A peer is frequently mid-turn when work arrives. An Opus agent chewing through a large context can hold its run lock for a long time, so “deliver now” is often not possible. The platform’s answer is a durable a2a_queue: if the target can’t take the message synchronously, it gets enqueued with a TTL and the caller is told Queued rather than handed a fake failure.

Getting the queue to be the default path under contention took real iteration. The original dispatch logic short-circuited to a bare 503 whenever the target adapter declared provides_native_session=True — these runtimes manage their own session and bypass platform buffering. That looked fine until a production client’s cron fired every 30 minutes for six hours and bounced 503 the entire time because a single native session held the slot (core #1685). The fix was to route native-session targets through the same a2a_queue instead of failing closed, so a scheduled delegation waits its turn rather than starving. We followed up by raising the response-header budget so a queued-then-served turn isn’t misrecorded as a failure, and logging a successfully-queued attempt as queued/ok rather than a false error (core #1751).

There is a second axis to “priority” here: in-flight work must not be silently dropped. When a container restarts, native-session targets that bypass the queue could lose messages already accepted into the SDK session. We added a pre-restart drain signal so stopForRestart flushes in-flight A2A work before the container goes down (core #207). Combined, these say: a delegation is durable from the moment the platform accepts it, whether it’s parked in the queue or already inside a peer’s session.

The queue also has two delivery modes — poll and push — and the response parser has to distinguish them honestly. A push-mode queued response that silently defaulted to delivery_mode="poll" would mislead any caller branching on that field (core #356). Small field, real correctness bug.

Don’t delegate to yourself

The cheapest way to deadlock the whole machine is to delegate to your own workspace ID. The sending turn holds the run lock; the receive handler waits for that same lock; the request times out after 30 seconds and the cycle is wasted. We had been warning agents off this in the Dev Lead system prompt — by hand, in prose — which is exactly the kind of invariant a prompt cannot be trusted to enforce. So we added a platform guard that rejects self-delegation outright (core #291), excluded the workspace from its own peer list, and returned an agent-readable 400 instead of a mysterious hang (core #1624). A related echo bug — a workspace seeing its own a2a_receive row in its inbox — got the same treatment with a self-echo predicate (core #1348). The general principle: any structural invariant that can deadlock the runtime belongs in the platform, not the prompt.

Surfacing results back — the part everyone underestimates

Sending work is easy. The genuinely hard problem is the round trip for async delegation: the originating agent fired and forgot hours ago, its turn is long over, and now a result has landed. How does it find out?

The mechanism is a small relay. The platform’s heartbeat writes completed delegation rows to a DELEGATION_RESULTS_FILE and sends the agent a self-message to wake it. On its next turn, the executor calls read_delegation_results() to atomically consume that file and fold the results into context. The wiring between “heartbeat writes the file” and “executor reads it” is what closes the loop, and it had a real gap: the consumer helper existed but was never actually invoked by the A2A executor, so async delegations never auto-resumed and the workspace looked dead to the user (core #358).

Then we found the result rows weren’t always where the poller looked. Delegations dispatched via the proxy path got logged with activity_type='a2a_receive', but the heartbeat polled for method IN ('delegate','delegate_result') — so proxy-path results were invisible to the resume logic (core #483, core #501). A sibling bug: the lookup keyed on response_body->>'delegation_id', but the insert wrote delegation_id only into request_body, leaving response_body NULL and the lookup blind (core #998). Every one of these is the same category of failure — a producer and a consumer disagreeing about the column or the method name — and each one manifests as the most user-hostile symptom possible: an agent that asked for help and then just goes quiet.

Ordering matters too. The heartbeat writes the results file and sends a wake message, but the idle prompt sometimes arrived first, so the agent composed a stale tick before noticing the pending result. We added a guard so the idle loop skips its prompt when delegation results are pending (core #432), and later extracted that check into a directly-testable function (core #451).

The boundary lesson: a peer’s reply is untrusted input

The most important architectural realization came from treating delegation results as a trust boundary. A result coming back from a peer contains text that peer generated — and that text gets injected straight into the originating agent’s prompt. That is untrusted input crossing into a privileged context. Under OFFSEC-003 we hardened every exit point that returns peer-sourced content (tool_delegate_task, the sync polling path, check_task_status) so that summary, response_preview, and error fields are escaped and wrapped in explicit boundary markers before they reach the prompt (core #382, core #390, core #433). The work spanned many follow-ups because the discipline is unforgiving: there are several code paths that return peer text, and missing one re-opens the hole. We eventually added a backstop test that asserts every public A2A exit point wraps its output, so a new path can’t regress silently (core #539).

The P0 that proved the design

The clearest validation of “delegation is its own subsystem, not an HTTP detail” came when fleet-wide A2A went fully down. The cause was a regression that left executeDelegation’s context attached to the originating HTTP request — so when that request ended, the in-flight delegation’s context was cancelled out from under it (core #1446, core #1450). Delegation outlives the request that triggered it. Detaching the context was the fix, and it’s the one-sentence summary of the whole architecture: a handed-off task has a lifecycle independent of whoever handed it off, and every layer — proxy, queue, results relay — exists to honor that.

What we’d tell the next team

Delegation is three problems wearing one tool name. Routing wants a central proxy so the platform can enforce policy and log everything. Queueing wants durability and fairness so a busy peer parks work instead of dropping it. Surfacing wants a relay that wakes a long-finished agent and folds the answer back in — and that relay only works if every producer and consumer agree on the exact row, column, and method they’re keying on. Pin those contracts down with tests at the boundary, treat the peer’s reply as untrusted, and remember that the task lives longer than the request.