The workspace the SSRF guard would not register

The real-image Local Provision Lifecycle E2E job had been red for a while, and the reason was the kind of bug you cannot find by reading the code. The test’s workspace container — running in a sidecar next to the control plane on the act_runner host — was dutifully registering itself with the control plane every few seconds. The control plane kept replying 400 url_validate_failed. The workspace’s URL was http://127.0.0.1:<host-mapped-port>, which the workspace was sure was right. The control plane’s SSRF guard was sure it was not.

This is the write-up of what was actually happening, why the obvious “weaken the guard” fix was wrong, and the small change to the workspace provisioner that made the harness green without weakening the security boundary. Tracked internally as core#2851; fixes shipped in PR #2860 and PR #2878.

The shape of the bug

The failure showed up in the E2E logs as three things in sequence, each one confusing on its own and obvious in hindsight together:

1. The first registration attempt from the real workspace returned 400 url_validate_failed … hostname "30e9e720fbc2" cannot be resolved. The hostname 30e9e720fbc2 is a Docker container short-ID — the workspace’s own hostname inside its container, the literal string hostname returns.
2. A retry loop in the harness burned through ten attempts, each one registering and getting a 400. The first failure was the most informative; the next nine just added noise.
3. When the harness’s A2A proxy finally got a “successful” registration (via the host-mapped 127.0.0.1:<port> path), it forwarded a delegation to the workspace. The forward hit EOF / context-deadline. The MiniMax round-trip came back with an empty result envelope. The test asserted “non-empty text”, got empty text, and failed with a “proxy returned a result envelope” assertion.

The symptom is “an empty LLM result”, but the root cause is “the platform cannot reach the address the workspace told it to reach.” The SSRF guard is doing its job. The problem is the address.

What the SSRF guard is, and why it was right

The platform’s SSRF guard lives in workspace-server/internal/handlers/a2a_proxy.go (around line 886) and calls into a dedicated ssrf.go helper. Its job is to refuse to proxy any request whose target URL points at a host that is not publicly routable. The premise is sound: a peer agent’s “A2A delegate” tool is a way for any compromised or confused workspace to make the control plane itself fetch an arbitrary internal URL. If the platform will not make outbound requests to 127.0.0.1, 10.0.0.0/8, 169.254.0.0/16, or any of the other reserved ranges, then a malicious delegate cannot pivot through the platform to hit the platform’s own metadata service, the workspace-server’s local Postgres, or anything else on the loopback.

That is the right behavior. We are not weakening it.

The bug is that the workspace server — the very thing the SSRF guard is supposed to protect — is the one trying to register a 127.0.0.1 URL in the first place. In production, the workspace runs on a different host from the platform, and its registered URL is something like https://ws-<org-hash>-<workspace-id>.moleculesai.app:<port> — a real, publicly-routable hostname. The SSRF guard passes that fine.

In the local-provision lifecycle E2E, the workspace and the platform both run on the same act_runner host. The workspace is reachable from the platform as 127.0.0.1:<host-mapped-port>. That is the only valid way for the two to talk in that topology. The workspace’s auto-detected URL.Host was the container hostname 30e9e720fbc2; the harness re-wrote the host to 127.0.0.1 for its own bookkeeping; the platform’s SSRF guard looked at the registered URL, saw 127.0.0.1, and rejected it as “not publicly routable.”

Three layers all individually correct, collectively broken.

The obvious fix, and why we did not ship it

The first reflex on a team with a CI failure is “make the test pass.” The two cheapest-looking options:

• Add 127.0.0.1 to the SSRF guard’s allowlist in the lifecycle-real test path. This is the kind of test-only carve-out that survives in production forever. A future maintainer reads “allowlist 127.0.0.1 for the E2E” and ships it.
• Set MOLECULE_ENV=development and let the guard know we are in dev mode. We did ship a dev-mode bypass; it just was not firing here, because the workspace registration happens before the workspace realizes it is in dev mode. A bypass that fires “before” the mode check is no bypass at all.

Both of these are anti-fixes. They trade a real CI red for a real security guard. The guard’s whole reason to exist is to refuse these URLs.

What we did instead: advertise a host-reachable URL

The actual fix is one sentence: let the workspace advertise a URL whose host the platform can resolve and reach, instead of the URL whose host happens to be the workspace’s own container identity. The platform does not care that the URL is 127.0.0.1; it cares that the hostname resolves to a host the platform can route to. In a Docker-on-Docker E2E, that host is the act_runner’s Docker bridge gateway — conventionally 172.17.0.1 in a default-network setup, exposed to the workflow as PLATFORM_HOST_IP.

PR #2860 added a new env var to the provisioner: MOLECULE_WORKSPACE_ADVERTISE_HOST. If set, the provisioner’s workspaceAdvertiseURL() reads it and uses it as the host portion of the registered URL. If unset, behavior is unchanged (localhost). The local-provision lifecycle workflow exports MOLECULE_WORKSPACE_ADVERTISE_HOST=${PLATFORM_HOST_IP} in the lifecycle-real job, so the workspace registers http://172.17.0.1:<host-port> — a host the platform container can resolve and reach. The SSRF guard looks at 172.17.0.1, sees that it is a routable address (even if it is the local bridge), and passes it.

PR #2878 closes the loop on the harness side. Two small changes:

• .gitea/workflows/local-provision-e2e.yml exports MOLECULE_WORKSPACE_ADVERTISE_HOST=${PLATFORM_HOST_IP} in both the lifecycle-stub and lifecycle-real jobs. Same shape, same variable, so a future change to the provisioner default does not produce a real-vs-stub skew.
• tests/e2e/test_local_provision_lifecycle_e2e.sh adds a fail-fast DNS resolve check on the advertised URL host. If a future test ever advertises a hostname the platform cannot resolve, the harness fails at registration time with “advertised host does not resolve” — not fifteen minutes later as an empty LLM result. The opaque “empty MiniMax envelope” failure mode is gone.

A follow-up commit in the same series kept the lifecycle-real job in host-network mode (MOLECULE_IN_DOCKER=false) for the A2A proxy path, so the proxy uses the host-mapped 127.0.0.1:<port> URL directly instead of trying to resolve a ws-<id>:8000-style container hostname. That is an implementation choice, not a security trade: the URL the proxy stores is the URL the platform uses to receive registrations, and the A2A proxy sits inside the platform container where 127.0.0.1 means “the host network the workspace is bridged onto.” That is the right URL for that path. The two URLs are different on purpose: one is “where I am reachable from” (advertise), the other is “where I send from” (proxy).

The lesson, stated generally

Defensive code rejects inputs that look like the bad case. The bad case here is “an agent registers a URL that the platform can be tricked into fetching on its behalf.” The guard looks at the URL’s host and asks: is this a public address? A 127.0.0.1 answer is “no” — almost always correctly. The fix is not to teach the guard that 127.0.0.1 is sometimes OK. The fix is to make sure the inputs the guard is asked to evaluate are always ones where the answer is the same in the test environment and in production: a real, host-routable address.

The general form: when a defensive check rejects an input that the system itself produced, the right move is almost never to weaken the check. It is to change how the system produces the input so the input is not in the rejected class. In this case, “the system produces a URL the guard will reject” was true because the test environment was the only place the workspace would ever advertise its own container identity. The fix is a one-line env var that lets the test environment advertise the same kind of URL production advertises. The guard is unchanged. The CI is green. No test-only carve-out survives into production.

What the test looks like now

The Local Provision Lifecycle E2E (real image + MiniMax LLM) job runs the harness, the harness exports MOLECULE_WORKSPACE_ADVERTISE_HOST=${PLATFORM_HOST_IP}, the workspace registers http://${PLATFORM_HOST_IP}:<host-port>, the platform’s SSRF guard accepts the URL (it is a public-looking address in the docker-bridge range), the A2A proxy routes a delegation to the workspace, the workspace runs a real LLM round-trip, and the harness asserts a non-empty result. Job is green, with a real LLM response, not an empty envelope.

The job is still advisory — touching the local-provision lifecycle E2E without breaking stub-mode or the host-network path is the kind of change that earns a two-reviewer gate rather than a self-merge. But the next time a real lifecycle-real run goes red with url_validate_failed, the failure will be at the new fail-fast DNS check, with the unresolvable host printed in the error message, not buried under an opaque empty LLM envelope.