https://www.moleculesai.app/blog/ Engineering stories from building Molecule AI — incidents, migrations, governance decisions, and lessons from shipping a multi-runtime agent control plane. en Mon, 15 Jun 2026 00:00:00 GMT https://www.moleculesai.app/blog/false-green-ci-harness-skip-counting/ https://www.moleculesai.app/blog/false-green-ci-harness-skip-counting/ Harness replay counted xfail-marked tests as PASS, inflating pass rates until we fixed skip accounting. Molecules AI Engineering Mon, 15 Jun 2026 00:00:00 GMT Engineering Reliability CI/CD Testing https://www.moleculesai.app/blog/prod-deploy-halted-on-502-bounded-retry/ https://www.moleculesai.app/blog/prod-deploy-halted-on-502-bounded-retry/ A transient HTTP 502 halted production auto-deploys; bounded retry got deploys moving again. Molecules AI Engineering Mon, 15 Jun 2026 00:00:00 GMT Engineering Reliability CI/CD Production https://www.moleculesai.app/blog/reachability-is-not-auth-preflight/ https://www.moleculesai.app/blog/reachability-is-not-auth-preflight/ Our LLM preflight mislabeled a healthy auth proxy as DEP-DOWN by conflating reachability with auth success. Molecules AI Engineering Mon, 15 Jun 2026 00:00:00 GMT Engineering Reliability CI/CD Production https://www.moleculesai.app/blog/ssrf-guard-would-not-register-workspace/ https://www.moleculesai.app/blog/ssrf-guard-would-not-register-workspace/ The SSRF guard blocked workspace registration for Docker gateway IPs; here's the precise fix. Molecules AI Engineering Sun, 14 Jun 2026 00:00:00 GMT Engineering Security Infrastructure Multi-agent systems https://www.moleculesai.app/blog/hermes-acp-adapter/ https://www.moleculesai.app/blog/hermes-acp-adapter/ NousResearch's Hermes Agent now runs as a native workspace on Molecule AI. Here's the adapter we built — and what the Agent Client Protocol made simple. Molecules AI Engineering Sat, 13 Jun 2026 00:00:00 GMT Engineering Open source Multi-agent systems Dogfooding https://www.moleculesai.app/blog/a2a-proxy-reliability-fixes/ https://www.moleculesai.app/blog/a2a-proxy-reliability-fixes/ Three molecule-core A2A proxy fixes: larger body caps, verified canvas-user sender identity, and a system-caller source-ID wedge fix. Molecules AI Engineering Sat, 13 Jun 2026 00:00:00 GMT Engineering Platform Multi-agent systems https://www.moleculesai.app/blog/a2a-queued-response-polling-core2437/ https://www.moleculesai.app/blog/a2a-queued-response-polling-core2437/ When a workspace returns 202 with a queue_id, callers must poll for the result. Here's the two-part fix that made A2A queue polling safe across restarts. Molecules AI Engineering Sat, 13 Jun 2026 00:00:00 GMT Engineering Platform Multi-agent systems https://www.moleculesai.app/blog/acp-steer-queue-commands/ https://www.moleculesai.app/blog/acp-steer-queue-commands/ How /steer commands route through the Hermes ACP adapter: three execution paths and why the distinction matters. Molecules AI Engineering Sat, 13 Jun 2026 00:00:00 GMT Engineering Multi-agent systems Dogfooding https://www.moleculesai.app/blog/canvas-chat-ux-improvements/ https://www.moleculesai.app/blog/canvas-chat-ux-improvements/ Canvas chat now syncs across devices, supports follow-ups during generation, and surfaces the agent's current operation. Molecules AI Engineering Sat, 13 Jun 2026 00:00:00 GMT Engineering Platform Product https://www.moleculesai.app/blog/canvas-idle-watchdog-long-blocking-turns/ https://www.moleculesai.app/blog/canvas-idle-watchdog-long-blocking-turns/ Long synchronous agent tasks starved the asyncio heartbeat, tricking the idle watchdog into killing active turns. Molecules AI Engineering Sat, 13 Jun 2026 00:00:00 GMT Engineering Platform Reliability https://www.moleculesai.app/blog/mcp-create-approval-governance/ https://www.moleculesai.app/blog/mcp-create-approval-governance/ The org concierge needed management-mode approvals; we added governance before it became a standing admin credential. Molecules AI Engineering Sat, 13 Jun 2026 00:00:00 GMT Engineering Platform Multi-agent systems https://www.moleculesai.app/blog/molecule-cli-workspace-management/ https://www.moleculesai.app/blog/molecule-cli-workspace-management/ molecule-cli brings workspace lifecycle, org admin, secrets, and runtime controls to the terminal. Molecules AI Engineering Sat, 13 Jun 2026 00:00:00 GMT Engineering Platform Dogfooding https://www.moleculesai.app/blog/molecule-sdk-python-attachments/ https://www.moleculesai.app/blog/molecule-sdk-python-attachments/ SDK 0.2.1 adds inbound attachment downloads for poll-mode external workspaces. Molecules AI Engineering Sat, 13 Jun 2026 00:00:00 GMT Engineering Platform SDK https://www.moleculesai.app/blog/more-info-request-delivery-fix/ https://www.moleculesai.app/blog/more-info-request-delivery-fix/ User replies to More-Info requests were stored but never dispatched back to the requesting agent. Molecules AI Engineering Sat, 13 Jun 2026 00:00:00 GMT Engineering Platform Multi-agent systems https://www.moleculesai.app/blog/multi-model-agent-org/ https://www.moleculesai.app/blog/multi-model-agent-org/ Treating every AI agent as one premium model doesn't scale — in cost or quality. Here's how I run a multi-model agent org with the right model for each role. Hongming Wang Mon, 08 Jun 2026 00:00:00 GMT Multi-agent systems AI engineering Cost optimization Dogfooding https://www.moleculesai.app/blog/one-ssot-for-runtimes-and-models/ https://www.moleculesai.app/blog/one-ssot-for-runtimes-and-models/ How we collapsed a sprawl of hardcoded provider switches into one manifest, derived everything from it, and gated the copy so it can never drift. Molecules AI Engineering Tue, 02 Jun 2026 00:00:00 GMT Engineering Architecture CI/CD https://www.moleculesai.app/blog/autonomous-seo-agent/ https://www.moleculesai.app/blog/autonomous-seo-agent/ We put moleculesai.app under continuous SEO ownership by an AI agent — a Molecule workspace that opens reviewed pull requests on a thirty-minute schedule. Molecules AI Engineering Tue, 02 Jun 2026 00:00:00 GMT Engineering Multi-agent systems Dogfooding https://www.moleculesai.app/blog/agent-to-agent-trust-boundary/ https://www.moleculesai.app/blog/agent-to-agent-trust-boundary/ Why we treat agent-to-agent messages as untrusted input, and the boundary sanitizer we shipped to keep peer text from rewriting an agent's instructions. Molecules AI Engineering Tue, 02 Jun 2026 00:00:00 GMT Engineering Security Agents https://www.moleculesai.app/blog/cutting-the-cloudflare-tunnel/ https://www.moleculesai.app/blog/cutting-the-cloudflare-tunnel/ Why we moved agent workspaces off per-tenant Cloudflare tunnels toward direct-connect public IPs — fewer moving parts, fewer failure modes. Molecules AI Engineering Tue, 02 Jun 2026 00:00:00 GMT Engineering Infrastructure Networking https://www.moleculesai.app/blog/gitea-migration-in-a-day/ https://www.moleculesai.app/blog/gitea-migration-in-a-day/ GitHub suspended our org with no warning. Here is how we moved code, CI, agent identities, and our image registry to self-hosted Gitea in a day. Molecules AI Engineering Tue, 02 Jun 2026 00:00:00 GMT Engineering Infrastructure CI/CD Incident https://www.moleculesai.app/blog/hierarchical-agent-memory/ https://www.moleculesai.app/blog/hierarchical-agent-memory/ How we gave agent teams a memory model shaped by the org tree — workspace, team and org scopes, pgvector recall, and the isolation boundary we hardened. Molecules AI Engineering Tue, 02 Jun 2026 00:00:00 GMT Engineering Agents Architecture https://www.moleculesai.app/blog/how-agents-delegate-work/ https://www.moleculesai.app/blog/how-agents-delegate-work/ How we built agent-to-agent delegation: the proxy protocol, a durable priority queue, and the path that wakes an agent when a result lands. Molecules AI Engineering Tue, 02 Jun 2026 00:00:00 GMT Engineering Architecture Distributed Systems https://www.moleculesai.app/blog/keyless-gemini-on-vertex/ https://www.moleculesai.app/blog/keyless-gemini-on-vertex/ How we run Gemini 2.5 Pro on Vertex AI for the google-adk runtime with zero API keys on disk, using Workload Identity Federation. Molecules AI Engineering Tue, 02 Jun 2026 00:00:00 GMT Engineering Security Infrastructure https://www.moleculesai.app/blog/making-ai-agents-follow-an-sop/ https://www.moleculesai.app/blog/making-ai-agents-follow-an-sop/ How we turned an engineering SOP into a CI merge gate that scales review to a PR's blast radius and forces per-item acknowledgement. Molecules AI Engineering Tue, 02 Jun 2026 00:00:00 GMT Engineering CI/CD AI Agents Governance https://www.moleculesai.app/blog/one-agent-one-machine/ https://www.moleculesai.app/blog/one-agent-one-machine/ Why every Molecule workspace runs as one agent on its own dedicated EC2 — no shared filesystem, env, or secrets — and how we hardened that boundary. Molecules AI Engineering Tue, 02 Jun 2026 00:00:00 GMT Engineering Security Architecture Infrastructure https://www.moleculesai.app/blog/rebuilding-ci-on-gitea-actions/ https://www.moleculesai.app/blog/rebuilding-ci-on-gitea-actions/ What actually broke when we moved CI from GitHub Actions to self-hosted Gitea Actions, and the gates we built to keep main honest. Molecules AI Engineering Tue, 02 Jun 2026 00:00:00 GMT Engineering CI/CD Infrastructure https://www.moleculesai.app/blog/the-migration-that-crash-looped-prod/ https://www.moleculesai.app/blog/the-migration-that-crash-looped-prod/ A control-plane migration runner re-applied every migration on every boot. One bare INSERT took prod down. Here is the fix and the rule. Molecules AI Engineering Tue, 02 Jun 2026 00:00:00 GMT Engineering Databases Reliability Postgres https://www.moleculesai.app/blog/the-org-chart-is-the-acl/ https://www.moleculesai.app/blog/the-org-chart-is-the-acl/ How Molecule derives agent-to-agent reachability from the org tree, and the root-sibling fix that closed a cross-tenant leak. Molecules AI Engineering Tue, 02 Jun 2026 00:00:00 GMT Engineering Security Architecture