False-green CI: when a passing test was actually xfail

A CI run was green across the board. Replays executed. Counts came back: 12 passed, 0 failed, 0 skipped. The summary line at the bottom of the workflow page said “all green.” The PR was approved. The merge happened. Two days later, the thing the replay was supposed to verify had broken in production, and the green CI from two days ago did not predict it.

The replay had not actually exercised the thing it was supposed to exercise. It had exited 0 — the “success” exit code — but it had printed an __XFAIL__ marker on its way out, meaning the test was expected to fail and was being skipped pending a fix to a tracked bug. The harness counted that as a PASS. The harness was wrong, the count was wrong, the green was wrong, and the merge was made on a count that meant nothing.

This is the write-up of how that happened, how a few lines in the runner fixed it, and the general lesson about counting test outcomes honestly. Tracked internally as part of the broader false-green audit; fix shipped in PR #2872.

The shape of the false-green

A harness replay is a small script that exercises a runtime path the platform supports — A2A ping, A2A pong, peer discovery, MCP request/response, and so on. The harness runs each replay in a loop, captures the exit code, and reports a summary: “passed N, failed M.” A replay that exits 0 counts as a PASS. A replay that exits non-zero counts as a FAIL.

canary-smoke-a2a-pong.sh was one of the replays. It was waiting on a tracked issue — call it #2863 — and was marked with an __XFAIL__ marker so the runner knew the test was expected to fail until #2863 was fixed. The replay’s job, in the meantime, was to confirm the harness itself still wires up correctly — not to confirm the A2A pong actually works. So the script exited 0 with an __XFAIL__ marker and a one-line explanation pointing at #2863.

The harness’s exit-code classification was: if [ $? -eq 0 ]; then PASS; else FAIL; fi. There was no third branch. There was no check for __SKIP__ or __XFAIL__ in the replay’s stdout. The replay exited 0, so the harness called it a PASS.

The number of “passed” replays in the summary was therefore 12, not 11-and-1-skip. The number of “skipped” was 0, not 1. The summary line at the bottom said “all green” because every replay either passed or had been classified as passed. Nothing in the runner was actively lying — every individual check was technically correct — but the aggregate count was off by one, in the direction that made the run look healthier than it was.

Why this is a hard bug to find

A false-green is hard to find because the failure is at the counting layer, not the execution layer. Every individual replay ran. Every individual exit code was checked. The summary line was assembled from the per-replay classifications. The only place the bug lives is in the classification rule that maps an __XFAIL__ replay to a count. A reviewer reading the runner’s source code and the summary line would see 12 PASS outcomes and 12 replay runs, and call it consistent.

The runner is also the layer that the on-call is least likely to dig into. When a CI run is green, the natural response is “nothing to look at.” When a CI run is red, the natural response is to look at the failing replay and figure out what is wrong. The runner is the silent middle: it is the infrastructure the testing infrastructure relies on, and the rare false-green is the failure mode that escapes the normal review path.

The audit that caught this one was triggered by a separate observation: the green CI count for the harness was high relative to the recent rate of code changes touching the harness. A green rate that is suspiciously close to 100% on a layer that has open bugs is itself a signal. False-greens look like reliability, but they can also look like an over-eager exit-code-zero classification.

The fix, in three parts

PR #2872 changes the harness runner in three small ways. None of them are individually hard; together they fix the false-green.

1. Capture stdout and look for skip markers. Each replay’s stdout is captured to a temp file as the replay runs. After the replay exits, the runner greps the captured output for __SKIP__ and __XFAIL__ markers. If a marker is present, the replay is classified as SKIP, regardless of exit code. The exit-code-zero-from-an-xfail-replay is no longer a PASS; it is a SKIP with the same exit-code-zero path. A future skip marker is just a grep.

2. Update the summary line. The runner’s summary line now reports passed=N failed=M skipped=K explicitly. The aggregate “all green” line is no longer a sum of pass + skip; it is gated on failed == 0 only. A run with 11 passed, 0 failed, 1 skipped is now honestly summarized — the SKIP is named, not absorbed into PASS.

3. Switch the marker convention. The canary-smoke-a2a-pong.sh replay was emitting __XFAIL__, which is the xfail-marker convention used elsewhere in the codebase. The PR switches it to __SKIP__ — same effect on the runner classification, but the marker name is now self-describing from the test’s perspective (this test is being skipped, not expected to fail). The xfail reason and the reference to the blocking issue stay in the human-readable output, so the audit trail is preserved.

The verification block in the PR is small but real: bash -n passes for both modified scripts, and a standalone loop-logic test confirms PASS / SKIP / FAIL classification. No replay semantics changed; the runner now honestly reports xfails as skips instead of false-greens.

For the broader merge-gate story — when a checklist becomes a load-bearing control rather than a decoration — see our SOP-as-merge-gate post.

The general lesson, stated for any test runner

A test runner that classifies outcomes on exit code alone is one marker convention away from a false-green. The __XFAIL__ convention is in use across the codebase precisely because humans reading the output know what it means — the test was expected to fail pending a fix. If the runner does not also know what it means, the human-reader convention diverges from the runner’s classification, and the aggregate count is wrong.

The general form: a test runner’s classification of outcomes should match the human-reader convention used in the test’s output, and any marker the output emits should be reflected in the aggregate count. If your tests emit __SKIP__ and __XFAIL__, the runner’s summary line should report a skip-count. If your tests emit a JUnit XML <skipped/> element, the runner’s summary should report skipped tests. The marker is the contract; the runner is the consumer; the summary is the visible signal.

A second, related lesson: a CI run that is suspiciously close to 100% green on a layer with open bugs is itself a signal worth auditing. Green rate is a property of the runner’s classification rule, not a property of the code. A classification rule that is too generous can produce a green rate that is uninformative. False-greens look like reliability. They can also look like the testing infrastructure is not actually testing anything.

If you take one thing from this post, take this: a CI run that is green is a claim about the system. A test runner that calls an __XFAIL__ exit-zero a PASS is a runner that does not honor the convention the test is using to communicate intent. Honor the convention, and the green becomes informative again.