Our GitHub org vanished overnight: migrating code, CI, identities, and registry to self-hosted Gitea in a day

Historical infrastructure note (July 2026): This article records the system as it existed when published. The operator-host and AWS deployment described below were retired in the off-AWS rebuild. Current access uses domain-based endpoints; inspect the relevant repository’s current workflows and runbook before operating the platform. Treat the remaining details as engineering history, not a current runbook.

On the morning of 2026-05-06 our GitHub organization, Molecule-AI, stopped responding. Not a partial outage: every URL under the org returned a suspension page. Repositories, the container registry we published runtime images to, the GitHub App that minted per-workspace tokens, and the CI that all of it hung off — gone at the same instant, with no advance notice.

We are an AI-agent orchestration platform. A control plane provisions tenant workspaces, and agents inside those workspaces clone code, build images, open pull requests, and review each other. When the org went dark, the whole supply chain went with it. This is the write-up of how we got back to a working state on self-hosted Gitea inside a day, what actually broke, and what we would do differently.

The blast radius was wider than “where the code lives”

The first instinct after a suspension is “push a mirror somewhere and carry on.” That instinct is wrong, because the source repository is only one of four things a platform like ours leans on GitHub for. We had four coupled dependencies, and each had to be re-homed:

• Source of truth — the monorepos molecule-core and molecule-controlplane, plus a constellation of runtime-adapter mirrors.
• CI/CD — GitHub Actions workflows, the merge primitives behind them, and the cross-repo uses: graph.
• Image registry — runtime and tenant images were published to GHCR (ghcr.io), and every workspace pulled from there at boot.
• Identity — a GitHub App minted installation tokens that workspaces used to authenticate git and gh.

We stood up Gitea on our own operator host (it was already running for an unrelated experiment) and pointed a new vanity domain, git.moleculesai.app, at it. The raw git push --mirror of the repos was the easy 20 minutes. The remaining day was spent on the three dependencies that a mirror does not fix.

CI was silently a no-op — and the silence was the danger

The mirror copied the repos faithfully, including the .github/workflows/ directory. The problem: Gitea Actions only reads .gitea/workflows/. So every workflow came across intact and ran exactly zero times. CI was not red — it was absent, which is worse, because branch protection that requires a check it never receives looks the same as a repo with no protection at all.

The first structural fix was simply renaming the directory (cp #108). That made workflows fire — and immediately surfaced a long tail of GitHub-isms that do not translate to Gitea’s runner (act_runner):

• Owner slug case sensitivity. GitHub treated Molecule-AI/ and molecule-ai/ as the same owner. Gitea does not. Every cross-repo uses: and repository: reference with the mixed-case slug failed to resolve at zero seconds.
• Artifact action protocol. actions/upload-artifact@v4+ speaks a newer artifact protocol that our act_runner version did not implement. We pinned the artifact actions back to @v3 (core #18, core #89).
• No GitHub REST/GraphQL surface. A surprising amount of our automation shelled out to gh pr ... and gh api. Gitea exposes only /api/v1/ — no GraphQL, no /api/v3, no workflow-runs endpoint. Those calls returned 404/405 and failed, sometimes loudly, sometimes not. We rewrote the branch-sync automation that used gh pr create plus auto-merge to drive Gitea’s REST API directly (core #66), and ported the publish cascade from GitHub’s repository-dispatch endpoint to Gitea’s equivalent (core #20).

The meta-lesson here is uncomfortable: a green pipeline on a self-hosted runner is not evidence of a healthy pipeline. A skipped or never-fired job can report success, and required checks that are simply never emitted leave a gate wide open. We later wrote down the recurring quirks — runner network isolation, step-level vs job-level continue-on-error, the (push) status-context suffix — as a standing runbook (core #457) so the next person did not have to rediscover them under pressure.

The registry move was the one that broke running workspaces

Code and CI breaking is an engineering inconvenience. The registry breaking meant live tenants could not boot new workspaces, because the workspace boot path pulled images from ghcr.io, which now 403’d.

We mirrored runtime and tenant images to Amazon ECR and threaded an MOLECULE_IMAGE_REGISTRY override through the provisioner so the registry was no longer hardcoded. Two things made this sharper than a find-and-replace:

First, the boot path itself needed an auth step. EC2 instances had ECR read access via their instance role, but the user-data never ran aws ecr get-login-password before docker pull — a no-op against GHCR, a hard failure against ECR. That fix had to land on both the workspace and tenant user-data builders, and the tenant one was initially missed.

Second, not every runtime had an explicit image pin. When a pin was absent, the resolver fell through to a legacy “clone the adapter from github.com and build” path — which now 401’d. We backfilled runtime_image_pins rows for the affected runtimes against their freshly mirrored ECR digests (cp #7) so the resolver never reached the dead clone path. We also swept ghcr.io and raw.githubusercontent.com references out of shell scripts that the workflow-only audit had missed (core #16).

URL references hid in places grep had to be taught to look

The obvious github.com/Molecule-AI/... URLs in docs and configs were a bulk substitution (core #40, cp #30). The non-obvious ones cost more:

• Go module paths. go.mod, go.sum, and import statements referenced the suspended org. We moved these onto a vanity import host, go.moleculesai.app, and stood up a go-import responder so go build could resolve modules without caring where the repo physically lives (cp #32).
• Indirect URLs. A script that built a clone URL from a variable (https://github.com/${repo}.git) had no Molecule-AI literal in it, so the first migration regex sailed right past it. It only surfaced when a Docker build failed deep in a workflow with could not read Username for https://github.com.
• Operator-facing copy. Onboarding instructions emitted to external users still printed GitHub URLs, breaking silently for anyone who followed them.

The pattern across all of these: the dangerous references are the ones that are computed, not literal. A static-string sweep is necessary but never sufficient.

Identity was the part we chose to redesign, not just port

Before the suspension, workspaces authenticated to GitHub using tokens minted by a GitHub App. We could have rebuilt the same shape on Gitea. We deliberately did not.

The App model had concentrated trust: a single installation identity, in practice anchored to a founder credential, signing for every workspace’s git activity. On the way over we made per-workspace identity the default — each workspace’s git author name and email are set from the persona credentials the provisioner already injects, so a commit from a given team’s workspace is attributed to that team’s Gitea identity rather than one shared principal (core #156). When the platform-side App config is absent, the endpoint now returns an explicit 501 that the workspace credential helper can distinguish from a transient error (core #407), and the credential chain gained a static fallback so a future control-plane outage cannot simultaneously delete git auth from every running workspace.

This is the one place where the forced migration was a net positive. A vendor outage is a brutal but honest audit of where you have a single point of failure. Ours pointed straight at a shared identity, and we will not be recreating that fingerprint.

What we would tell our past selves

A few things held up, and a few we earned the hard way:

• Re-home all four dependencies, not just the repo. Source, CI, registry, and identity each have their own failure mode. Mirroring the repo and declaring victory leaves three live landmines.
• Treat “the registry is down” as a production incident, not a tooling one. It is the dependency that stops live customers, so it should be first in the recovery order.
• Distrust green. On a freshly self-hosted runner, absent and skipped checks masquerade as success. Verify that required contexts are actually emitted, then write the quirks down.
• Sweep computed references, not just literals. The URLs that hurt were assembled at runtime and invisible to a naive grep.
• Let the outage retire your single points of failure. We came out of it with per-workspace identities and a registry override we should have had all along.

We did the bulk of the recovery in a day. The cleanup tail — porting the long list of workflows, hardening the gates, reconciling drift between branches — ran for weeks after. That ratio feels about right: getting back online is a sprint, and earning back the guarantees you used to take for granted is the real work.