Reachability is not auth: a preflight story

For several days, every CI run that touched a staging lane reported DEP-DOWN:staging-llm in the preflight step. The staging LLM was not down. The proxy was up, healthy, and serving requests to the agents that actually authenticated. The preflight itself was the bug: it sent an unauthenticated probe and treated every non-200 — including the 401 a healthy auth-required proxy correctly returns — as evidence that the service was unreachable. The lane classification fired a fleet-wide staging-down alert, which an on-call paged on, and the operator was greeted with a fully-working LLM proxy and a red preflight that contradicted it.

This is the write-up of what the preflight was supposed to do, why it was doing the wrong thing, the one-line semantics fix, and the general lesson about health checks that conflate reachability with successful auth. Tracked internally as core#76; fix shipped in PR #2866.

What a preflight is, and what this one was for

A completion-gated e2e lane is an expensive operation. We spawn a real workspace, boot a real LLM, run a real round-trip, and assert on the result. Doing that on every commit is wasteful when the dependency it depends on is broken. The preflight is a fast probe at the top of the lane: confirm the staging LLM proxy is reachable before we spend ten minutes booting a workspace only to fail at the LLM call. If the preflight fails, the lane is skipped with a clear DEP-DOWN:staging-llm status (coordinated with the control plane lane classification, so fleet-wide alerts dedupe cleanly). Operators and the redgate-reporter can then see the outage and dedupe; we do not run ten-minute jobs that will deterministically fail.

The preflight sends a minimal POST to the LLM proxy’s chat-completions endpoint, with a body of one user message (“pong”) and max_tokens: 1. It does not include a credential. The preflight only needs to know: is the proxy answering HTTP? The full LLM round-trip — the part that needs auth — is the lane’s job, not the preflight’s.

The preflight script has, in its own header comment, the explicit statement of this contract: “#76 root cause, 2026-06-13: the preflight sends an UNAUTHENTICATED probe. A healthy staging LLM proxy that requires auth correctly returns 401. Previously any non-200 status (including 401) was classified as DEP-DOWN, causing fleet-wide false staging-down incidents.” That is the bug. The comment was correct, the implementation drifted from it.

The shape of the bug

The preflight’s classification logic looked roughly like this:

http_code=$(curl -s -o "$tmp" -w "%{http_code}" ...)
if [ "$http_code" != "200" ]; then
  echo "::error::DEP-DOWN:staging-llm preflight failed: proxy=$proxy http_code=$http_code body=$(head -c 500 $tmp)"
  return 70
fi
return 0

The “what counts as UP” check is http_code == "200". A 401 from a healthy auth-required proxy does not match. The check treats “the proxy refused my unauthenticated request” the same as “the proxy is down, slow, or returning 5xx.” The classification is wrong, and the consequence is wrong: a fleet of CI lanes is reported as staging-down even though the staging LLM is up, serving authenticated traffic, and healthy.

This is a category error, not a code typo. Reachability is a property of the proxy. Auth success is a property of the request against the proxy. The preflight’s job is to probe the first property, but its check is on the second. A 401 means the proxy is up and that the request did not carry a valid credential. The preflight’s “I have no credential” is fine; the preflight’s “this means the proxy is down” is not.

The one-line fix

PR #2866 changes the classification to:

if [ "$http_code" = "000" ] || [[ "$http_code" == 5* ]]; then
  echo "::error::DEP-DOWN:staging-llm preflight failed: ..."
  return 70
fi
return 0

Two changes, both small. First, the preflight now classifies transport failures (http_code=000, meaning curl could not connect) and 5xx server errors as DEP-DOWN. Second, every other HTTP response — 200, 201, 204, 301, 401, 403, 404, 410, 418, 429 — is treated as OK. The proxy is up. We are done.

The header comment was updated to match the implementation: “the preflight only needs to prove the proxy is REACHABLE: any HTTP response (including 401/403/404) means the proxy is UP. Only transport failures or 5xx server errors classify as DEP-DOWN.” A future reader sees the comment first, the code second, and they agree.

The pre-existing unit-test cases were extended with a 401-reachable case (http_code=401 returns 0, not 70). The 5xx case continues to return 70. The unreachable case continues to return 70. All 5 tests pass.

Why this is the right shape

The preflight is a check on reachability. Reachability is a binary at the level a CI lane cares about: the TCP connection is open, the proxy is processing HTTP, the proxy is responding. The exact status code matters only insofar as it lets us distinguish “the proxy answered me” from “the proxy did not answer me.” For the purposes of the lane — “is it worth booting a workspace to talk to this thing” — the right answer is “the proxy answered me, even if it said no.” A 401 is a yes: yes, the proxy is up, yes, it is processing HTTP, yes, it would have accepted this request if it carried a valid credential.

The pre-existing optional Authorization header path was also removed. The preflight does not need to authenticate; it does not need to prove the credential works. It needs to prove the proxy is up. Adding optional auth was scope creep: a future maintainer might have wondered why the preflight sometimes succeeded with a 200 and sometimes failed with a 401, and chased a phantom auth bug that was not there.

The general lesson, stated for any health check

A health check that probes a service should probe the property it is trying to verify, and it should be explicit about which property that is. A check that wants to know “is the service reachable” should accept any HTTP response as evidence of reachability, and reserve the failure classification for transport-level failures (connection refused, timeout) and 5xx server errors. A check that wants to know “is the service healthy enough to serve a successful request” should authenticate, send a real request, and assert on a real success status.

Conflating the two — using one probe to verify both reachability and auth — looks like a simplification but is actually a class of false-positive. A healthy auth-required service will always return 401 to an unauthenticated probe. If your probe classifies 401 as failure, your monitoring will say the service is down whenever the service is up and requires auth. The fix is to pick: reachability or auth-success, not both. The preflight needs reachability; the lane that boots a workspace and runs a real round-trip is where auth-success lives.

If you take one thing from this post, take this: a 401 from a probe that did not authenticate is not evidence that the probed service is down. It is evidence that the probed service is up and requires auth. Health-check semantics that treat 401 as a down-state will lie to you.