The canvas idle watchdog: how a blocking tool call starves the heartbeat and cancels a long turn

An agent running a bulk image migration hit a hard stop at five minutes. The canvas reported “tool chain lost” and the turn was cancelled mid-work, with no error in the agent’s own logs. The workspace looked healthy from the outside; the agent thought it was still running.

This post explains what happened and what changed.

The idle watchdog

The Molecule AI canvas sends agent turns over the A2A protocol. Each turn has a live broadcaster on the workspace server — a channel that streams activity events back to the canvas. The canvas attaches an idle watchdog (applyIdleTimeout) to that broadcaster: if it sees no events for idleTimeoutDuration, it concludes the turn has stalled and cancels it.

Under normal operation the watchdog never fires. The workspace runtime emits a WORKSPACE_HEARTBEAT every 30 seconds. As long as the heartbeat ticks, the watchdog resets and the turn survives indefinitely.

The prior idleTimeoutDuration was 5 minutes.

The starvation path

Python runtimes (including claude-code’s asyncio-based infrastructure) run the heartbeat as a coroutine on the main event loop. That’s fine for most tool calls, which yield control quickly. It’s not fine for a blocking call — one that executes synchronously and holds the event loop thread for minutes.

When a tool call blocks, asyncio can’t schedule other coroutines. The heartbeat coroutine is queued but never runs. From the workspace server’s perspective, the broadcaster goes silent. The watchdog clock ticks. At 300 seconds, the turn is cancelled.

The agent doesn’t receive an exception. Its blocking call eventually returns, but the canvas-side channel is already closed. The turn is gone.

The fix

PR #2727 raises the default idleTimeoutDuration from 5 minutes to 30 minutes. The 30-minute ceiling matches the agent-to-agent A2A ceiling, and it deploys via the standard tenant workspace redeploy — no runtime-template roll required.

The tunable A2A_IDLE_TIMEOUT_SECONDS environment variable was already present; this PR changes its default. If your deployment has a specific ceiling (shorter for interactive chat, longer for batch pipelines), override it explicitly.

The 30-minute default is a mitigation, not the complete fix. The correct fix is to run the heartbeat on an independent daemon thread so it’s never starved by a blocking tool call, regardless of how long that call takes. That runtime-side change is tracked in core#2723 and will land in a future runtime-template update. Until then, the 30-minute window gives agents enough headroom for practical multi-minute blocking work.

Why 5 minutes was the prior default

Five minutes was chosen when the typical long turn was an LLM-heavy reasoning chain — lots of short async hops, none of them blocking. For that workload the heartbeat stayed healthy and 5 minutes of true silence (no events, no heartbeat) was a reasonable signal that something had gone wrong.

The workload expanded: agents now routinely do things that block — bulk file processing, subprocess calls, synchronous SDK operations, long database queries. The heartbeat assumption (always ticking) broke for those callers, but the watchdog default never followed.

What this means for agent developers

If you’re building on Molecule AI, the 30-minute window is in the current platform image — no SDK change required.

If you’re writing tools that might block for more than a few seconds, the right long-term posture is to run those operations in a thread pool (asyncio.to_thread or loop.run_in_executor) so the event loop stays free for the heartbeat. That’s not always possible with third-party SDKs; the 30-minute default exists precisely for those cases.

The failure mode is otherwise opaque: the agent logs look normal, the turn is cancelled at the canvas layer, and the only signal is a canvas-side “tool chain lost” message. If you’ve seen unexplained turn cancellations on long tasks, this is likely the cause.

Related fixes in the same cluster

The stall reporting also surfaced a related canvas issue: the thinking indicator was frozen at 0s when the agent was busy on its own task (current-task path rather than the user-send path). Both the timer and the live tool-call feed gated on sending rather than the broader thinking condition. PR #2720 unified those gates so the indicator reflects real agent activity regardless of which path triggered it.

The server-side raise was the first half of the #2723 fix. PR #2731 completed it by aligning the canvas-side send timeout in useChatSend.ts to match. Both client and server now share the same 30-minute ceiling.

Sources: molecule-core PR #2727 (server-side idle watchdog raised to 30m), PR #2731 (client-side send timeout aligned to 30m), core#2723 (root cause + daemon-thread spec).