Cutting the Cloudflare Tunnel: Direct-Connect Public IPs for Agent Workspaces

Historical infrastructure note (July 2026): This article records the system as it existed when published. The operator-host and AWS deployment described below were retired in the off-AWS rebuild. Current access uses domain-based endpoints; inspect the relevant repository’s current workflows and runbook before operating the platform. Treat the remaining details as engineering history, not a current runbook.

Every agent workspace we run is an EC2 instance with an HTTP server on :8000 (the A2A endpoint) and an SSH path used by our control plane for filesystem and config operations. For a long time the public face of each workspace was a per-tenant Cloudflare Tunnel: a cloudflared daemon inside the tenant box dialed out to Cloudflare’s edge, and <slug>.moleculesai.app resolved to a CNAME pointing at that tunnel. No inbound ports, edge TLS for free, DDoS absorption at the edge. On paper it is the textbook way to expose a private box.

In practice the tunnel became the single most operationally expensive component in the workspace lifecycle. This post is about why, what we tried, and where we landed.

The shape of the problem

The tunnel was not one component. It was a distributed state machine spread across three systems that all had to agree: our control-plane database (which org owns which tunnel), the Cloudflare API (the tunnel object and its DNS CNAME), and the cloudflared process inside a tenant EC2 instance that may or may not be running. Every lifecycle transition — provision, deprovision, billing-suspend, billing-resume — had to mutate all three in the right order, and any drift produced a user-visible Cloudflare error page.

The failure modes were not theoretical. A few that actually shipped fixes:

• Name collisions on fast re-provision. Delete a tenant, recreate the same slug within a couple of seconds, and Cloudflare would still hold the old tunnel — provisioning failed three times in a row on exactly this race before we scoped tunnel names by org UUID and made tunnel-create failure abort the whole provision (cp#65).
• Active-connection delete rejection. Our GDPR cascade called DeleteTunnel a few seconds after TerminateInstance, but cloudflared was often still registered as connected, so Cloudflare returned error 1022 and the cascade stalled. The fix was to make the cascade idempotent so a retry-to-resume wasn’t required (cp#45).
• Billing-suspend dead-ends. Suspending an org stopped its EC2, which killed the tunnel origin, which left <slug>.moleculesai.app serving a raw Cloudflare “Tunnel error” (1033) with no path for the customer to even add credits. We had to repoint suspended subdomains to an add-credits page (cp#344) and then self-heal the tunnel on resume, because the tunnel had sometimes been deleted while the org was suspended (cp#346).

That last one is the clearest illustration of the core issue: a customer’s billing state, an EC2 power state, a cloudflared registration state, and a Cloudflare DNS record all had to be reconciled to deliver a single HTTP 200. When we built a live against-real-staging test for the resume self-heal, it immediately found two more production defects that fakes had never surfaced (cp#386), and follow-ups kept landing for query encoding and multi-record DNS matches (cp#434).

The drift tax

Beyond runtime, the tunnel imposed a steady CI and maintenance tax. Because the control plane terminated the tunnel and proxied platform routes itself, it kept a platformPaths allowlist describing which URL prefixes belonged to the platform versus the tenant. That list had to stay byte-aligned with the routes molecule-core actually registered — and a dedicated cf-tunnel-drift-gate job reddened every control-plane PR whenever the two drifted.

We paid that gate repeatedly: adding /uploads/* to platformPaths after a new core endpoint shipped (cp#406, cp#232), fixing cross-repo checkout auth in the gate itself (cp#26), and rewriting the cloudflared systemd-unit validator to be a pure-Go INI parse so it ran anywhere (cp#33). The cloudflared binary also had to be baked into the CI runner image, which created its own coordination problems across repos (cp#22). None of this delivered customer value; it was the cost of keeping three systems in agreement.

What we tried first

The honest first phase was not “rip it out.” It was hardening. We genuinely tried to make the tunnel safe.

The most serious work was on the cross-environment boundary. On 2026-05-08 a token-scope problem let one environment mutate another environment’s Cloudflare resources — the kind of blast radius that keeps you up at night. The structural fix was three independent layers: scope orphan-tunnel cleanup so it can’t touch the wrong env, add per-environment prefixes, and a boot-time token-scope check (cp#36). We followed it with an audit-claim gate so a control plane only deletes a tunnel or DNS record its own database created a claim for (cp#126).

We also tightened the network boundary the tunnel implied. The workspace security group had once allowed 0.0.0.0/0 on tcp/8000; we narrowed it to Cloudflare’s announced egress ranges so only edge-proxied connections could reach the A2A endpoint (cp#125). And we cleaned up a legacy fly-replay routing assumption left over from a Fly.io era — every production tenant runs on EC2, so the router now HTTP-proxies straight to the tunnel in EC2 mode (cp#257).

Each of these was a good change. But step back and the pattern is unmistakable: a long tail of fixes whose root cause was always the same — too many independent systems had to be correct, in order, for a request to land. Hardening reduces the probability of each individual fault; it does not remove the faults from the fault tree.

What we shipped

The decision was to remove the tunnel from the data path and let workspace traffic terminate against a direct-connect public IP, with the control plane and edge handling TLS, routing, and rate limiting at the platform boundary rather than via a per-tenant outbound daemon.

Concretely, the direction is:

• No cloudflared inside the tenant. That deletes the EC2-power ↔ tunnel-registration ↔ DNS reconciliation entirely. An instance that is up is reachable; an instance that is down is down. One state, not four.
• The instance’s address is a first-class, rate-limited fact. Per-tenant instance details, including the public IP, are already exposed through /cp/orgs/:slug/instance — and that route is rate-limited to prevent enumeration (cp#185). The address is data the control plane owns, not a side effect of a daemon’s registration.
• The boundary moves to the security group and the edge. Instead of “only Cloudflare egress IPs may reach :8000,” reachability is governed by an explicit, control-plane-managed security group — the same place we already auto-resolve the EIC endpoint SG so the control plane’s own SSH path into the workspace works (cp#167). The boundary becomes one SG rule set, owned and tested in one place, rather than an allowlist that has to track a third party’s published ranges.

The latency argument follows for free. The old path was client → Cloudflare edge → cloudflared outbound tunnel → tenant :8000, with the tunnel hop pinned to wherever cloudflared happened to connect. Removing the daemon removes a hop and removes the variance that came from tunnel re-registration after every restart — the same re-registration that caused the billing-resume Unauthorized: Tunnel not found loops in the first place.

The lesson

The tunnel was not a bad technology. It was the wrong abstraction for a system where the unit of deployment is a per-tenant EC2 instance whose lifecycle we already fully control. Cloudflare Tunnel earns its keep when you genuinely cannot expose an inbound address — a box behind NAT, a laptop, an on-prem network. We were paying that premium for instances we provision, address, power-cycle, and firewall ourselves.

The recurring tell, across two months of PRs, was that almost every incident traced back to two systems disagreeing about state: the database said one thing, Cloudflare said another, and the customer saw a 1033 or a 1022. When the fix to a class of bugs is “make the reconciliation idempotent,” “add an audit-claim gate,” “scope the token,” “self-heal on resume” — each individually correct — the meta-signal is that you have one too many sources of truth in the path. The most reliable component is the one that isn’t there.

We are not anti-edge. TLS, DDoS posture, and WAF still belong at a managed boundary, and the platform router still terminates and rate-limits platform routes. What changed is that the workspace data path no longer depends on a per-tenant outbound daemon staying registered through every power and billing transition. Fewer moving parts, fewer state machines to reconcile, lower and more predictable latency — and a CI pipeline that no longer reds on a drift gate guarding an allowlist we no longer need to maintain.