Why our prod deploy halted on a single 502 (and how it learned to retry)

Historical infrastructure note (July 2026): This article records the system as it existed when published. The operator-host and AWS deployment described below were retired in the off-AWS rebuild. Current access uses domain-based endpoints; inspect the relevant repository’s current workflows and runbook before operating the platform. Treat the remaining details as engineering history, not a current runbook.

Our production auto-deploy workflow was green-CI, green-superseded, and then hard-failed in the canary redeploy with a one-line log message: redeploy scoped call failed for hongming: HTTP 502. The whole fleet rollout — every tenant, every canary, every promote — stopped right there. One unclassified 502 from the control plane, and a hundred tenants did not get the new workspace server image.

The CP was not actually down. It was a transient upstream flake — a brief SSM hiccup while the deploy endpoint was resolving infrastructure state. The 502 was real, but it was the kind of error a deploy helper should treat as try again rather than give up. This is the write-up of how we classified those failures, added bounded retry with backoff, and surfaced the actual error body so the next operator who sees this log line knows what the CP was really saying. Tracked internally as core#2859; fix shipped in PR #2862.

The shape of the failure

A production auto-deploy is a long pipeline. It waits for green CI on the SHA, waits for any earlier SHA to be superseded, builds the image, and then steps through every tenant slug calling the control plane’s /cp/admin/tenants/redeploy-fleet endpoint to ask for a canary redeploy. That last step is the one that failed. The CP returned HTTP 502 for the hongming canary, and the helper raised. The workflow halted. The new image never made it past the canary.

A 502 from an API gateway is one of three things:

• An upstream the gateway depends on is down (an internal service, a database, a queue). Often transient.
• A bad gateway config (the gateway cannot route a request). Often persistent, but the symptom is the same.
• A genuinely broken request (the gateway gave up after N internal retries). Usually transient, but the only way to know is to look at the upstream logs.

The cost of treating any of these as “permanent” is high. The cost of treating any of these as “transient and worth a retry” is bounded — you wait a few seconds and try again, and the second try is usually the one that succeeds. The asymmetry favors retry. The thing you do not want is an unbounded retry loop, and the thing you also do not want is to retry on a class of errors that is genuinely permanent (404, 410, 422).

The right shape is: retry transient gateway errors with bounded backoff; surface non-transient errors immediately with the actual error body.

The fix, in three parts

PR #2862 changes the .gitea/scripts/prod-auto-deploy.py helper along exactly those lines. Three changes, each one a different kind of fix:

1. Classify the error class. redeploy_scoped() now distinguishes “transient gateway error” (HTTP 502, 503, 504) from “non-transient error” (everything else, including 4xx like 401/403/422 and 5xx like 500). The retry loop runs only against the transient class. A 401 from the CP — wrong auth, wrong path, wrong body — is raised immediately. A 422 — invalid request shape — is raised immediately. The retry budget is reserved for the cases where retrying has a meaningful probability of succeeding.

2. Retry with bounded backoff. Three attempts total, with 5s / 10s / 20s sleeps before the second, third, and final attempt. The retry budget is small enough that a fleet-wide deploy still completes in under a minute even when the CP is degraded, and large enough that a one-off transient upstream hiccup can clear. The final attempt, if it fails, raises — there is no fourth sleep, no terminal wait. The operator sees the failure when there is a failure to see.

The annotation ::warning:: is emitted on each retry so the GitHub Actions log makes the retry behavior visible. The pattern is small but durable: a deploy that retries silently is a deploy where the operator cannot tell whether the canary was actually retried or not.

3. Surface the actual error body. The old _raise_for_redeploy_result() raised a RuntimeError whose message was the HTTP status code. The new version parses the CP’s JSON error response and includes the error, message, and (truncated) JSON body in the exception text. The operator reading redeploy scoped call failed for hongming: HTTP 502: error=upstream_timeout, message=SSM call to /api/production/state timed out after 30s learns, in one line, what the CP was actually saying. That is the difference between a 2AM page that ends in “restart the gateway” and a 2AM page that ends in “SSM is wedged, page the platform team.”

What this is not

It is worth saying what this change is not.

It is not a fix to the CP. The CP’s /cp/admin/tenants/redeploy-fleet endpoint may still return 502 sometimes; that is a separate concern owned by the control plane team, and the RCA for the specific 502 in this incident lived in the CP logs, not in our deploy helper. Treating the deploy helper as the layer that retries is a deliberate choice: the helper is the layer that gets to decide “is this a retry-class error or a raise-class error,” because the helper is the only place that knows whether it has retry budget left. The CP cannot know that — every individual call from the helper looks the same to it.

It is not a “retry forever” wrapper. Three attempts, then raise. The deploy helper is not a service mesh; it is a workflow step. Workflow steps that retry forever block subsequent steps. The whole pipeline — image build, canary, verify, promote, full rollout — is structured around the assumption that each step is bounded. Bounded retry preserves that assumption.

It is not a “retry on every non-2xx” wrapper. 4xx errors are raised immediately. A 401 means the helper itself is misconfigured; retrying will not fix that. A 422 means the request body is wrong; retrying will not fix that. The retry class is 502, 503, 504 only, and the rationale for each is “this status means the gateway or the upstream the gateway depends on had a transient problem; the next call has a meaningful chance of succeeding.”

The test coverage that earned the change

The PR added four pytest cases to .gitea/scripts/tests/test_prod_auto_deploy.py:

• test_retry_success — a 502 followed by a 200 succeeds without raising, after one backoff sleep.
• test_retry_exhausted — three 502s in a row raises the RuntimeError with the CP error body in the message.
• test_no_retry_on_non_transient — a 500 raises immediately, no backoff, no retry.
• test_error_body_surfaced — a 502 with a JSON body of {"error": "upstream_timeout", "message": "..."} produces a RuntimeError whose message contains both error=upstream_timeout and the original message.

These tests are small but they encode the four properties that matter: retry is allowed on the transient class; retry stops at the budget; retry is forbidden on the non-transient class; and the operator gets the actual error body, not just a status code. The full suite was 46 passed locally at PR time. (For context on the Gitea Actions runner pool that runs this suite, see our Gitea migration write-up.)

The general lesson, stated for any deploy helper

A deploy helper is the layer between your CI pipeline and your infrastructure. The shape that holds up over time is: classify the response, retry the transient class with bounded backoff, surface the non-transient class with the body the operator needs to debug. A deploy helper that hard-fails on a single 502 is a deploy helper that will halt your fleet on a single blip. A deploy helper that retries forever is a deploy helper that will hang your fleet when the upstream is genuinely down. The right shape is in the middle, and the right middle is small.

If you take only one thing from this post, take this: a transient 5xx in a deploy path is almost always the right thing to retry, but it is never the right thing to retry forever, and it is never the right thing to surface as just a status code. The deploy helper is the layer that knows the difference. Make it do that job.