Keyless Gemini on Vertex AI: AWS role to GCP STS to a short-lived token, no API keys on disk

Historical infrastructure note (July 2026): This article records the system as it existed when published. The operator-host and AWS deployment described below were retired in the off-AWS rebuild. Current access uses domain-based endpoints; inspect the relevant repository’s current workflows and runbook before operating the platform. Treat the remaining details as engineering history, not a current runbook.

Every agent workspace we provision runs on an AWS EC2 instance. Most of our LLM runtimes get their credentials the obvious way: at provision time the control plane seeds a tenant’s secrets table, and the workspace boots with a provider token in its environment. That is how cp#109 wires CLAUDE_CODE_OAUTH_TOKEN, ANTHROPIC_API_KEY, OPENAI_API_KEY, and GEMINI_API_KEY into a fresh tenant so agent calls do not 401 at the provider’s edge.

When we brought up an experimental runtime to serve Gemini 2.5 Pro, that pattern hit a wall. This post preserves the credential architecture from that now-retired experiment; the runtime is not part of the current Molecule SDK contract.

The constraint: the org disables API-key auth for Google AI

The straightforward path would have been a GEMINI_API_KEY in the workspace environment, the same shape as every other provider. We could not use it. The Google Cloud org we operate under disables API-key auth for Google AI at multiple layers, and the console mandates Application Default Credentials (ADC). This is not a preference we could toggle off — it is an org policy boundary, and the right move was to design with it rather than around it.

ADC normally means one of two things: a service-account JSON key file, or a metadata server on a GCP instance. We had neither in a useful form. Our compute is on AWS, not GCP, so there is no GCP metadata server to ask. And a service-account JSON key is exactly the thing we did not want — a long-lived, high-value secret sitting on a tenant’s data volume, surviving container swaps, waiting to leak. We have enough memory of token-burn incidents to treat any static credential on a workspace disk as a liability, not a convenience.

So the real problem statement: how does a process on an AWS EC2 instance authenticate to Vertex AI, satisfy an ADC-only org policy, and never hold a durable Google secret?

Workload Identity Federation: trade the identity you already have

The answer is Workload Identity Federation (WIF). The insight is that the EC2 instance already has a verifiable identity — its AWS IAM role, expressed as a signed request to AWS STS that says “this caller is arn:aws:sts::...:assumed-role/...”. WIF lets GCP trust that AWS-issued identity directly. We configure a workload identity pool and an AWS provider on the GCP side, and GCP’s Security Token Service agrees to exchange a proof of the AWS identity for a short-lived GCP federated token. That federated token then impersonates a Vertex-scoped service account, and the resulting access token is what actually talks to the Vertex AI endpoint.

No JSON key is ever generated. No Google secret is ever written to the workspace. The only thing on disk is an ADC configuration file that describes the exchange — it names the pool, the provider, and the service account to impersonate, and it tells the Google auth library how to obtain the AWS proof at runtime. The credential material itself lives for minutes and only in memory.

The chain, end to end:

1. The workspace process needs to call Vertex. The Google auth library reads the ADC config and sees it is an external_account (federated) credential, not a key file.
2. The library obtains a signed proof of the instance’s AWS IAM identity from the AWS side.
3. It presents that proof to GCP STS, which validates it against the configured workload identity pool and AWS provider, and returns a federated token.
4. That token impersonates the Vertex service account, yielding a short-lived OAuth access token scoped to Vertex AI.
5. The Vertex AI call to Gemini 2.5 Pro goes out with that token. When it expires, the library runs the exchange again.

The trust boundary is the GCP-side pool configuration: GCP only honors proofs from the specific AWS account and role we registered. Tightening that mapping is the security work — the disk holds no secret to steal, so the question shifts from “protect the key” to “who is allowed to perform the exchange.”

What we actually shipped

The code half landed in cp#416: keyless Vertex AI ADC via Workload Identity Federation for the experimental runtime, explicitly framed as the org-policy-compliant path because the org disables API-key auth for Google AI and the console mandates ADC. The provisioner was responsible for the credential wiring — writing the ADC config into the workspace so the auth library could perform the exchange at call time.

The configuration half landed in cp#421, which added a production vertex block to our envs SSOT pointing at the molecule-vertex project in us-central1. A deliberate design choice there: the VertexConfig to NewVertexADCConfig path is all-or-nothing. Either the full set of fields is present and ADC is configured, or it is absent and the feature does not half-activate. Partial configuration is the kind of thing that produces a workspace that looks healthy and silently fails its first Gemini call, and we would rather fail closed at boot than debug a confusing 401 later.

On the provider-registration side, cp#424 mapped the experimental runtime to the google vendor (the Gemini family, model gemini-2.5-pro). Then cp#426 registered a first-class vertex provider for the keyless path and gave the experiment two arms — a keyless vertex arm as the default, mirroring the existing anthropic-oauth / anthropic-api and openai-subscription / openai-api splits we already maintain for other vendors. Having two explicit arms matters: it means “keyless Vertex” and “Gemini via API key” are distinct, named serving configs rather than one ambiguous google entry that the adapter has to guess about.

At the time, core#2003 made the experiment selectable through the manifest, provisioner gate, and canvas. That historical state is no longer current: the SDK runtime contract has since retired the experiment.

The sharp edges

A few things were not obvious until we hit them.

Two google serving configs, two base_url shapes. The keyless Vertex path and the legacy Gemini-API-key path do not hit the same endpoint. cp#432 fixed the Gemini API-key base_url (alongside an unrelated kimi-coding /v1 fix), and the lesson generalizes: the moment a vendor has two auth arms, every serving field — base_url included — has to be correct per arm, not per vendor. A single google row would have papered over exactly the difference that breaks at request time. The two-arm split from cp#426 is what made these fixes localizable instead of a guessing game.

Fail loud on runtime-seed mismatch. A recurring theme in our provisioner work is refusing silent fallbacks. When a workspace names a runtime whose template is not in the tenant cache, the historical behavior was to fall back to the default claude-code template — which could make a workspace run a different engine than its configuration declared. core#2028 made that case fail loud. For a keyless path this is doubly important: a silent fallback would also abandon the WIF wiring, and you would never know the careful credential design was bypassed.

ADC config is configuration, not a secret — treat it accordingly anyway. It is tempting to relax once there is no key on disk. But the ADC config still names your project, pool, provider, and the service account it impersonates. It is a map of the trust relationship. We keep it scoped to the workspace and treat the GCP-side pool restriction — which AWS account and role may perform the exchange — as the real control. The disk artifact is inert without that mapping; the mapping is where the rigor goes.

The lesson

The reflex when a provider needs credentials is to find a token and put it somewhere the process can read it. WIF inverts that. Instead of injecting a new secret, you teach the destination to trust an identity the workload already has and already proves continuously — here, the EC2 instance’s AWS IAM role. The credential that actually authenticates the Vertex call lives for minutes and never touches the disk, so the entire class of “long-lived key leaks off a workspace volume” risk simply does not apply.

It is more moving parts up front: a pool, a provider mapping, an impersonation grant, an all-or-nothing config path. But every one of those parts is declarative configuration you can audit, not a secret you have to rotate, guard, and pray never lands in a log. For a fleet of agent workspaces that get swapped, snapshotted, and re-provisioned constantly, “no durable secret to lose” turned out to be worth the extra wiring — and it was the only path that satisfied the org policy in the first place.