Making AI-Agent PRs Follow a Real SOP: A Merge Gate for Tiered Review and Per-Item Acks

When most of your pull requests are opened by AI agents, “follow the SOP” stops being a cultural norm you can enforce in a standup. An agent will happily check every box in a PR template, write a confident summary, and request a merge — whether or not it actually ran the tests it claims to have run. A human reviewer skims a tidy checklist and clicks approve. The process exists on paper; nothing on the machine enforces it.

We hit this wall on the control plane. Our written SOP described tiers of risk, who must review what, and a per-item checklist every change had to satisfy. None of it was load-bearing. So we did the obvious-in-hindsight thing: we compiled the SOP into a merge gate.

The problem: a checklist is not a control

Our SOP has two halves. First, a risk tier (tier:low, tier:medium, tier:high) that scales how much review a change needs — a test-only canvas tweak is not the same as touching a provisioning hot path. Second, a per-item checklist every PR body must carry: comprehensive testing performed, local-Postgres E2E run (or a justified N/A), staging-smoke status, root-cause-not-symptom, no-backwards-compat-break.

The failure mode was specific. Agents would emit a fully-ticked checklist as prose. The boxes were decoration. A tier:high change touching the deploy chain could land with the same two meta-checks as a typo fix, because branch protection on main only required a secret scan plus one tier check — the real CI gates (Go build, canvas, E2E, Postgres integration) were not even required contexts. We learned that the hard way when broken merges slipped past the thin protection and we had to file an emergency hardening RFC and add an aggregator sentinel (core#553, core#656).

The lesson that framed everything after: a checklist that nothing reads is a comment, not a control.

What we shipped: two gates, composed

We built two CI workflows that emit commit-status checks, then made those contexts required in branch protection.

1. sop-tier-check reads the PR’s tier:* label and resolves which teams must approve. The interesting part is that approvals are AND-composed per tier (core#225): a tier:high change does not pass on a single thumbs-up — it requires approvals from every mandated team for that tier. The gate queries Gitea’s reviews API, maps reviewer logins to canonical teams, and only reports success when the full set is satisfied. It started life as a soft-launch workflow-only change with no protection update (core#144), so we could watch it run green before it could block anyone.

2. sop-checklist parses the PR body and verifies each SOP item is genuinely addressed — and that the per-item acknowledgements come from the right peers (core#688). Crucially, an unfilled checklist section is a hard failure, not a diagnostic warning, and the failing status names exactly which section is empty (core#797). An agent can no longer paste a ticked-box block and call it done; the gate reads the actual body and the actual reviews.

Both feed a single aggregator, all-required, which is the one context branch protection actually gates on. That indirection matters: it lets the set of underlying jobs be path-aware (a docs-only PR need not run Go tests) while the required context stays stable. Without the aggregator, every individual job had to be a required context, and any rename or path-filter quietly broke the gate.

The security boundary we had to get right

A merge gate runs with repository secrets — it needs a token to query teams and reviews. The naive way to wire this on Gitea Actions is the pull_request event, which loads the workflow definition from the PR’s own HEAD. That means a PR could modify the gate’s own script and run it with the repo’s secrets in context. That is an exfiltration path, full stop.

We moved every gate to pull_request_target, which loads the workflow and scripts from the base branch (main), not from the PR (core#146). The gate that judges a PR is always the trusted, already-merged version of the gate — never the version the PR ships. We later had to re-apply the same base-SHA checkout discipline to a sibling automation after a regression reintroduced PR-HEAD checkout (core#556). The principle: code that wields secrets must come from a trusted ref, never from the change under review.

We pair this with a force-merge audit (core#150) that emits a structured event whenever a PR is merged with a required check unsatisfied — so a privileged override is observable after the fact, not silent.

What actually broke (and what it taught us)

The gates themselves became a source of incidents, and every one was a lesson about CI runtime reality rather than about the SOP.

• Fail-open is harder than it looks. We wanted the gate to degrade gracefully if its token was missing — better to fail open than to wedge every PR on an infrastructure hiccup. But set -euo pipefail meant a jq call on an empty/401 response exited the script before the fail-open block could run (core#635). And job-level continue-on-error is silently ignored by our runner — only step-level works (core#411). Defensive behavior has to be reachable on the actual failure path, not declared in the happy path.

• The runner is not your laptop. A jq-install step that fetched a binary from an external host hung on the runner’s restricted network and blocked every PR, forcing a revert (core#402) and then an apt-first reordering (core#428). Assumptions about network egress are assumptions about the gate’s availability.

• Status semantics bite. A blocked job can report success if it is skipped; the combined commit state can read failure for reasons that should not gate a merge. We learned to key off the specific required context and its timing, not the aggregate state, and to dedupe per-context status correctly (core#995).

• The gate must re-evaluate. Approvals and labels land after the first gate run. We added labeled/unlabeled and review-comment triggers so a tier label or a late approval refires the check instead of leaving a stale failure (core#818, core#449).

Letting the SOP bend without breaking

A rigid gate that cannot express “this item genuinely does not apply” trains people to lie to it. So we added structured escape hatches that are recorded, not hidden.

A peer can declare a checklist item not-applicable with a /sop-n/a <gate> [reason] comment; the gate posts an na-declarations status naming the gates marked N/A, and downstream review logic respects it (core#915). And when the SOP’s approval matrix itself was wrong — certain items demanded acks from personas whose tokens were dead, deadlocking every PR — we fixed the policy, widening ack eligibility to a risk-classed two-eyes rule rather than weakening the mechanism (core#1554). The gate is strict; the policy it enforces is allowed to evolve, in the open, through a PR.

Where it landed

The same shape now runs on both molecule-core and molecule-controlplane. The control plane piloted the aggregator-and-drift pattern first (cp#89, cp#112) before we fanned it out — proving a CI hardening pattern on one repo before org-wide rollout turned out to be its own small SOP worth keeping.

The result is not that agents became more honest. It is that honesty stopped being required. A PR’s blast radius now determines how many real teams must approve it; a checklist item is satisfied only when the body and the reviews actually say so; and the code that judges all of this always comes from a ref the PR cannot touch. The SOP is no longer a document we hope people read. It is a status check that has to go green.