Rebuilding CI on Gitea Actions After Leaving GitHub: Runner Labels, needs + if:always(), and Status Reaping

When our GitHub org was suspended, we did not get to plan the migration. CI for the control plane and molecule-core had to come back up on self-hosted Gitea Actions in a hurry, and the workflows we carried over looked syntactically identical to what GitHub ran. They were not. Gitea Actions runs the act_runner, which implements a deliberate subset of the GitHub Actions surface, and the gaps showed up one red build at a time.

This is a write-up of what broke, why, and the small set of gates we eventually built so that main tells the truth about whether the code is good.

The labels that can never be satisfied

The first failures were the cheapest to diagnose and the most total. Our GitHub workflows targeted runs-on: [self-hosted, macos, arm64] and, in molecule-controlplane, that label set appeared eleven times across six files. None of those labels exist on the new pool. Our act_runner containers register ubuntu-latest (and docker), so every one of those jobs queued forever against zero eligible runners. That is not a flake — there is no runner that will ever match, so the job never starts and never posts a status. We flipped them all to ubuntu-latest in cp#13.

The deeper lesson arrived later (see our Gitea migration write-up for the full migration timeline). Our runner pool is heterogeneous: some runners have /var/run/docker.sock mounted and some do not. A publish job that needs to build an image but only asks for ubuntu-latest lands on a random runner and fails roughly half the time — a coin flip, not a bug in the code. The instinct is to add a docker label to the job. We tried exactly that and had to revert it within the hour in core#606: the docker label was not actually registered on any runner, so runs-on: [ubuntu-latest, docker] reproduced the original “zero eligible runners” deadlock — strictly worse than a coin flip. Labels are a contract with the runner registry. Adding a label to a job without first registering it on a runner converts an intermittent failure into a permanent hang.

act_runner is not GitHub Actions

A pile of smaller incompatibilities followed, each one a place where a workflow that is legal on GitHub is quietly wrong on Gitea:

• Case-sensitive owner slugs. Gitea resolves uses: and repository: references case-sensitively. Our mixed-case Molecule-AI/... references that GitHub tolerated failed at zero seconds against the canonical lowercase molecule-ai/. Fixed in core#17.
• Artifact protocol version. actions/upload-artifact@v4+ speaks the newer artifact protocol that our act_runner does not implement; the upload step emits a GHESNotSupportedError. We pinned the artifact actions back to @v3 in core#89.
• Workflow directory. Gitea reads .gitea/workflows/, not .github/workflows/. A publish workflow that lived only under .github/ was simply dead — it never fired once. Porting it surfaced in core#211.
• The on: parser is stricter. The single sharpest example: Gitea 1.22.6 mis-parses workflow_dispatch.inputs, flattening the sub-keys and then ignoring the entire workflow for all events with a one-line ignore invalid workflow ... unknown on type in gitea.log. The workflow looked present and correct in the repo and ran on nothing. We found the real root cause and dropped the offending inputs block in core#353.

The meta-lesson: when a Gitea workflow “doesn’t run,” do not assume a trigger or a permissions problem. Read gitea.log first. A silently-ignored workflow posts no status, which is indistinguishable from “never triggered” unless you look at the server.

Host-network runners collide

Our act_runner is configured with container.network: host operator-wide — applied to both job and service containers. That makes service containers convenient to reach, but it means two parallel runs of the same workflow bind the same Postgres or Redis port on the same host and one of them dies on a port collision. This is invisible on GitHub, where each job gets its own network namespace. The fix was to make the integration workflows parallel-safe by not assuming an exclusive port — done for the handlers-Postgres job in core#98 and the e2e-api job in core#100.

The aggregate gate: from a 20-minute poll to native needs

Branch protection wants a single required context that means “all the real checks passed.” With path filters on individual jobs, you cannot just require each job by name — a job that is skipped by its paths: filter never posts, and the PR blocks forever waiting for a status that will never come.

Our first answer was an aggregator sentinel job, all-required, that polled the commit-status API until the upstream checks resolved or a timeout fired (cp#89). It worked, but it was ugly: the poller holds a runner for up to twenty minutes while it waits, and on Gitea 1.22.6 the upstream integration job would sometimes sit in Blocked by required conditions and never recover, starving the poller until its timeout (cp#321).

When prod Gitea moved to 1.26.2, needs: plus if: always() finally behaved the way the spec describes. We converted all-required to a native dependency aggregator — needs: [changes, build, integration, arch-lint, golangci-lint, ...] with if: always() so it evaluates even when an upstream job is skipped — and it holds no runner while the upstream jobs run (cp#427). The if: always() is load-bearing: without it the aggregator inherits the default success() condition and is itself skipped the moment any dependency is skipped, which silently turns the required gate green. A skipped gate that reads as success is the most dangerous failure mode in this whole story, because nothing looks wrong.

Status reaping: when main goes red for a fake reason

The longest saga was about commit-status context strings, and it is worth telling because the bug moved every time we thought we had it.

Gitea 1.22.6 emits the status context for any run on the default-branch HEAD as <workflow> / <job> (push) — appending (push) regardless of the actual trigger. So a schedule- or workflow_dispatch-triggered run that fails paints main red under a fake (push) context that branch protection then treats as a real push failure. We built a status-reaper cron to compensate these stale-suffix statuses (core#589).

Then we learned how many ways a compensating cron can be wrong:

• A workflow-level concurrency block with cancel-in-progress: false does not queue a colliding tick on 1.22.6 — it cancels it. So the reaper kept cancelling itself (core#618).
• A scheduled run posts failure to whatever SHA was HEAD when it completed, which by the next tick is usually a stale commit because main has moved on. Reaping only the current HEAD missed everything; we widened the window to sweep recent commits (core#633, core#650).
• The actual reason compensation had been unreachable since rev1: the Gitea combined-status response uses two different keys. The top-level aggregate is combined.state, but each per-entry item in combined.statuses[] uses the key status, not state. Our per-entry loop read state on objects that only had status, so it never matched anything to compensate. Fixed in core#652.

The honest lesson from the reaper is that it is a workaround for a server quirk, and a workaround that itself emits statuses and consumes runners can make the problem worse before it makes it better — at one point we disabled it entirely while machinery was saturated. The right long-term fix was the platform upgrade, not a smarter cron.

Don’t let exit codes lie about your code

One last category: workflows that exit non-zero for reasons that have nothing to do with whether the pushed code is good. Our publish-runtime-autobump fires on every push touching workspace/ and exits non-zero when there is nothing to bump, or a tag already exists, or a dispatch token is absent. None of those mean “the code is broken,” but each flipped main red. Wrapping the right step in continue-on-error (core#524) restored the invariant we actually care about: a red main should mean the code is broken, not that a publisher had nothing to do.

What we’d tell the next team

Migrating CI off GitHub is not a YAML find-and-replace. The runner pool, the status API, and the on: parser are all subtly different surfaces, and the failures cluster in two buckets: jobs that can’t run (unsatisfiable labels, ignored workflows) and statuses that lie (fake (push) suffixes, skipped-reads-as-success aggregators, non-code exit codes). The fixes that held were the boring structural ones — register labels before you require them, make the aggregate gate native with if: always(), and treat a red main as a contract you defend. The clever crons were the parts we kept having to revisit.