One agent, one machine: hard isolation for multi-tenant AI agents

When you let an AI agent run shell commands, clone repositories, install dependencies, and hold credentials, you have built something that looks a lot less like a SaaS feature and a lot more like a remote-code-execution surface that you are inviting people to use on purpose. The interesting question stops being “what can the agent do” and becomes “what is the blast radius when it does something you did not anticipate.”

Our answer, formalized in the workspace-placement RFC (core #1819), is the bluntest one we could defend: every Molecule org runs as a fully isolated tenant on its own dedicated machine, and every agent workspace is a container on that machine. No shared filesystem between tenants. No shared process environment. No shared secret store. The isolation boundary is a host, not a namespace.

This post is about why we landed there, what the boundary actually is, and the specific places we had to harden it once we started treating it as a security property instead of a deployment convenience.

The shape we rejected

The default multi-tenant shape is a shared cluster: one big pool of compute, tenants separated by logical boundaries — namespaces, row-level scoping, a tenant_id column threaded through every query. It is efficient and it is what most platforms do.

It is also the wrong default when the workload is an autonomous agent. A logical boundary only holds if every code path that touches tenant data remembers to check the boundary. That is a fine assumption for a CRUD app where you wrote every query. It is a bad assumption when the thing executing inside the boundary is a model that can write and run arbitrary code, read arbitrary files, and reach arbitrary endpoints. The agent is not adversarial, but it is unpredictable, and “unpredictable code with credentials” is exactly the threat model that logical isolation is weakest against. One traversal bug, one over-broad query, one credential that was scoped to the pool instead of the tenant, and the boundary is gone.

So we inverted the default. Instead of one machine hosting many tenants behind logical fences, one tenant gets one machine. If you want to reach another org’s data from inside a workspace, there is no query to get wrong and no namespace to escape — the data is on a different host you have no route to. That contract has been implicit since we rebuilt the platform after the GitHub-org suspension on 2026-05-06; core #1819 wrote it down so we would stop re-deriving it.

What the boundary actually is

The architecture has three layers, and the isolation lives in the relationship between them.

The control plane is the only multi-tenant component. It owns the org registry, billing, and provisioning, and it is the thing that talks to AWS. No agent code runs there and no tenant ever gets a shell on it.

A tenant EC2 is provisioned per org. It is the isolation unit. The control plane launches it, attaches an instance role scoped to that tenant’s resources, and hands it exactly the configuration it needs — nothing about any other tenant exists on the box.

A workspace is a container on the tenant host running a single agent runtime — Claude Code, LangGraph, an ADK adapter, whatever the org configured. Workspaces within one org share a host (they belong to the same trust domain — the same customer), but they are still separate containers with separate mounts.

The thing to internalize is that the strong boundary is between orgs, and it is physical. The boundary between workspaces in the same org is softer by design, because they are the same tenant. We do not pretend otherwise.

Three things we will not share

Once you commit to host-level isolation, the failure modes move. They stop being “did the namespace check fire” and become “did something leak across the host boundary by accident.” Three categories turned out to need explicit hardening.

Filesystem

No tenant filesystem is shared with another tenant — that one is free, it falls out of separate hosts. The subtler leak is within a tenant’s own data layer. Our memory subsystem originally co-mingled tenant data in a way that made the schema the only fence. core #1742 moved the v2 memory plugin’s tables under a dedicated memory_plugin schema so the storage boundary is structural, not a convention every query has to honor. Same philosophy, one layer down: make the boundary something the system enforces rather than something each call site remembers.

Environment

Configuration and secrets used to ride in the EC2 user-data as inline gzipped heredocs. That is shared-channel thinking — everything the box needs, smuggled through one blob — and it bit us with a hard 12288-byte ceiling (AWS’s 16 KiB user-data limit minus bootstrap overhead) that silently truncated larger configs. cp #358 moved workspace config delivery off user-data and into AWS Secrets Manager, fetched per tenant at boot under that tenant’s instance role. Persona credentials moved the same way in cp #76: each tenant’s identity env is delivered from Secrets Manager at launch, not baked into a shared image or a bind-mount that only existed in local dev. The environment a workspace sees is assembled for that tenant, from that tenant’s secret namespace, and reaches no further.

Secrets

The rule we converged on is that a workspace gets the credentials it needs and nothing platform-wide. The cautionary tale is the seed allow-list. We used to auto-seed new tenants with a fixed bundle of tokens, and one of them was a GITHUB_TOKEN that was broader than any single tenant should hold. cp #204 removed it as the sole violation of our tenant-token policy (RFC#523), and cp #219 eventually retired the auto-seed path entirely in favor of explicit per-secret writes. The lesson generalizes: a convenience that hands every tenant the same credential is a shared-secret leak wearing a helpful hat. Per-tenant secret scoping is the only version that composes with per-tenant hosts.

Hardening the seams

Physical isolation removes the easy cross-tenant paths but it does not remove all of them, and the remaining seams are where we spend our review attention.

The auth boundary on the tenant itself mattered more than the host boundary in one case. The workspace server had a same-origin shortcut that trusted the Referer/Origin headers to authorize sensitive actions like listing and deleting secrets. Those headers are forgeable, so the shortcut was a bypass. core #1924 removed the forgeable path and replaced it with a control-plane-verified member session, plus regression coverage for the forged-header case. Host isolation does not save you from an auth bug inside the host’s trust domain — it only bounds it to one tenant. Both boundaries have to hold.

Provisioning correctness is the other seam, because the isolation property is only as good as the lifecycle that creates and destroys these hosts. A split-write during deprovisioning once left nine production EC2s orphaned over three days — marked removed in the database but still running, still holding their instance roles. core #2 added an orphan sweeper that re-issues the stop on a tick for any workspace marked removed with a live instance id. An isolation model built on dedicated machines lives or dies on the machine lifecycle being exactly-once; a leaked host is a leaked boundary.

One more clarification we had to make explicit: host sizing and host access are different axes. We briefly derived EC2 instance size from the access tier, which conflated “how much root access does this agent have on its box” with “how big should the box be.” cp #173 decoupled them. Worth stating because the access tier is real — a high-tier workspace genuinely has root on its dedicated host, which is only a defensible thing to offer because the host is dedicated. The isolation model is what makes deep access a feature instead of a liability.

The tradeoff, stated honestly

One machine per org is not free. It costs more than bin-packing tenants onto shared compute, provisioning is slower than scheduling a pod, and the orphan-sweeper class of bug is the tax you pay for managing real hosts instead of logical records. We think it is the right trade for this workload, but it is a trade, not a free lunch, and pretending otherwise would be the marketing fluff this post is trying to avoid.

What we get for the cost is a boundary we can reason about without trusting every line of agent-adjacent code to remember a check. When the unit of isolation is a host, “can org A read org B’s data” has a one-word answer — no route — instead of an audit of every query in the system. For a platform whose entire job is running other people’s autonomous agents, a boundary you can state in one sentence is worth paying for.

The lesson we keep relearning: choose the isolation boundary your weakest assumption can’t break. For unpredictable code holding credentials, that boundary is the machine.