Engineering blog
Stories from building Molecule AI.
What we learned shipping a multi-runtime agent control plane — incidents, migrations, governance decisions, and the small refactors that turned out to matter.
-
Engineering
False-green CI: when a passing test was actually xfail
Harness replay counted xfail-marked tests as PASS, inflating pass rates until we fixed skip accounting.
By Molecules AI Engineering -
Engineering
Why our prod deploy halted on a single 502 (and how it learned to retry)
A transient HTTP 502 halted production auto-deploys; bounded retry got deploys moving again.
By Molecules AI Engineering -
Engineering
Reachability is not auth: a preflight story
Our LLM preflight mislabeled a healthy auth proxy as DEP-DOWN by conflating reachability with auth success.
By Molecules AI Engineering -
Engineering
The workspace the SSRF guard would not register
The SSRF guard blocked workspace registration for Docker gateway IPs; here's the precise fix.
By Molecules AI Engineering -
Engineering
Three A2A proxy reliability fixes: silent truncation, sender identity, and system-caller wedge
Three molecule-core A2A proxy fixes: larger body caps, verified canvas-user sender identity, and a system-caller source-ID wedge fix.
By Molecules AI Engineering -
Engineering
After the 202: reliable A2A status polling across workspace restarts (core#2437)
When a workspace returns 202 with a queue_id, callers must poll for the result. Here's the two-part fix that made A2A queue polling safe across restarts.
By Molecules AI Engineering -
Engineering
Mid-turn steering: how Hermes Agent's ACP adapter handles /steer and /queue
How /steer commands route through the Hermes ACP adapter: three execution paths and why the distinction matters.
By Molecules AI Engineering -
Engineering
Four canvas chat improvements: cross-device sync, free multi-send, session reset, and auto-grow
Canvas chat now syncs across devices, supports follow-ups during generation, and surfaces the agent's current operation.
By Molecules AI Engineering -
Engineering
The canvas idle watchdog: how a blocking tool call starves the heartbeat and cancels a long turn
Long synchronous agent tasks starved the asyncio heartbeat, tricking the idle watchdog into killing active turns.
By Molecules AI Engineering -
Engineering
Building the ACP adapter: how Hermes Agent became a Molecule workspace runtime
NousResearch's Hermes Agent now runs as a native workspace on Molecule AI. Here's the adapter we built — and what the Agent Client Protocol made simple.
By Molecules AI Engineering -
Engineering
Why every agent mode needs a gate-raising primitive: the create_approval incident
The org concierge needed management-mode approvals; we added governance before it became a standing admin credential.
By Molecules AI Engineering -
Engineering
molecule-cli: the command-line tool for managing your Molecule AI agent organization
molecule-cli brings workspace lifecycle, org admin, secrets, and runtime controls to the terminal.
By Molecules AI Engineering -
Engineering
molecule-sdk-python 0.2.1: inbound attachments for external workspaces
SDK 0.2.1 adds inbound attachment downloads for poll-mode external workspaces.
By Molecules AI Engineering -
Engineering
Why your agent never heard back: fixing More-Info request delivery (core#2606)
User replies to More-Info requests were stored but never dispatched back to the requesting agent.
By Molecules AI Engineering -
Multi-agent systems
Stop Paying Genius Rates for Junior Work: How I Run a 24/7 AI Engineering Team Across Four Providers
Treating every AI agent as one premium model doesn't scale — in cost or quality. Here's how I run a multi-model agent org with the right model for each role.
By Hongming Wang -
Engineering
Every agent message is untrusted input: hardening the A2A trust boundary (OFFSEC-003)
Why we treat agent-to-agent messages as untrusted input, and the boundary sanitizer we shipped to keep peer text from rewriting an agent's instructions.
By Molecules AI Engineering -
Engineering
How we gave our AI agent platform an autonomous SEO engineer that ships its own pull requests
We put moleculesai.app under continuous SEO ownership by an AI agent — a Molecule workspace that opens reviewed pull requests on a thirty-minute schedule.
By Molecules AI Engineering -
Engineering
Cutting the Cloudflare Tunnel: Direct-Connect Public IPs for Agent Workspaces
Why we moved agent workspaces off per-tenant Cloudflare tunnels toward direct-connect public IPs — fewer moving parts, fewer failure modes.
By Molecules AI Engineering -
Engineering
Our GitHub org vanished overnight: migrating code, CI, identities, and registry to self-hosted Gitea in a day
GitHub suspended our org with no warning. Here is how we moved code, CI, agent identities, and our image registry to self-hosted Gitea in a day.
By Molecules AI Engineering -
Engineering
Hierarchical agent memory: LOCAL, TEAM and GLOBAL scopes that follow the org tree
How we gave agent teams a memory model shaped by the org tree — workspace, team and org scopes, pgvector recall, and the isolation boundary we hardened.
By Molecules AI Engineering -
Engineering
How one agent hands work to another: the A2A delegation protocol, the queue, and surfacing results
How we built agent-to-agent delegation: the proxy protocol, a durable priority queue, and the path that wakes an agent when a result lands.
By Molecules AI Engineering -
Engineering
Keyless Gemini on Vertex AI: AWS role to GCP STS to a short-lived token, no API keys on disk
How a retired runtime experiment used Vertex AI with zero API keys on disk through Workload Identity Federation.
By Molecules AI Engineering -
Engineering
Making AI-Agent PRs Follow a Real SOP: A Merge Gate for Tiered Review and Per-Item Acks
How we turned an engineering SOP into a CI merge gate that scales review to a PR's blast radius and forces per-item acknowledgement.
By Molecules AI Engineering -
Engineering
One agent, one machine: hard isolation for multi-tenant AI agents
A historical account of Molecule's one-agent-per-EC2 architecture, its isolation boundary, and the hardening work it required before the off-AWS rebuild.
By Molecules AI Engineering -
Engineering
One source of truth for runtimes and models, and a drift gate that keeps the templates honest
How we collapsed a sprawl of hardcoded provider switches into one manifest, derived everything from it, and gated the copy so it can never drift.
By Molecules AI Engineering -
Engineering
Rebuilding CI on Gitea Actions After Leaving GitHub: Runner Labels, needs + if:always(), and Status Reaping
What actually broke when we moved CI from GitHub Actions to self-hosted Gitea Actions, and the gates we built to keep main honest.
By Molecules AI Engineering -
Engineering
The Migration That Crash-Looped Prod: Why Every Migration Must Be Idempotent
A control-plane migration runner re-applied every migration on every boot. One bare INSERT took prod down. Here is the fix and the rule.
By Molecules AI Engineering -
Engineering
The Org Chart Is the ACL: How CanCommunicate Derives Who Can Talk to Whom
How Molecule derives agent-to-agent reachability from the org tree, and the root-sibling fix that closed a cross-tenant leak.
By Molecules AI Engineering