san086041-glitch/FailGate

Anyone can write a fix. FailGate proves it's the right one.
A failing test for every bug, sealed before anyone touches the code — then used to grade every PR that claims to fix it.

Quick start · How it works · Results · End to end · GitHub App · CLI · MCP · 中文

_{Real pages from the public demo repo failgate-demo — re-captured by CI, nothing staged.}

_{The same check from a terminal: failgate verify on PR #5, recorded with VHS in CI, real time.}

Coding agents now open pull requests at scale — about 17 million a month on GitHub by March 2026. Writing the fix is no longer the hard part. Knowing whether it is right is.

• 🧪 Tests written after the fix grade themselves. When an agent sees a failing test, editing the test is the cheapest way to make it pass — ImpossibleBench caught GPT-5 doing exactly that in 76% of impossible tasks.
• 🕳️ "Tests pass" is a weak signal. UTBoost found that 15.7% of the patches counted as resolved on SWE-bench Verified are actually wrong — the tests were too weak to notice.
• 🔁 Reproduction bots stop too early. Several tools now turn an issue into a failing test. Very few keep that test honest when a PR arrives: was it edited? skipped? does it still fail the same way? did something else break?

FailGate is the acceptance layer: it writes the exam before the fix exists, locks it, and grades every claimed fix — from a maintainer, a contributor, Claude Code, Codex, Copilot, or its own fixer agent — the same way.

Wording matters: a pass means "passes the acceptance test + no tampering found + no new regressions", not "the fix is correct".

Datasets are small and every number has known limits (see What FailGate does not claim); the human-style reviews were done by Claude, not by the projects' maintainers.

Needs Python 3.11+, Docker, and a GitHub token (read-only is enough). No LLM key needed — the demo exams are already sealed.

git clone https://github.com/san086041-glitch/FailGate && cd FailGate
python -m venv .venv && source .venv/bin/activate     # Windows: .venv\Scripts\Activate.ps1
pip install -e .
export GITHUB_TOKEN=$(gh auth token)

python scripts/media/demo_exams.py import             # the demo repo's public sealed exams → your local DB
failgate verify san086041-glitch/failgate-demo#5      # a PR that only edits the test → refuted
failgate verify san086041-glitch/failgate-demo#16     # a real fix → accepted, with exam strength

Then type failgate for the home screen and interactive shell, or failgate doctor to check your setup.

failgate verify owner/repo#123
failgate checkup owner/repo
failgate up

claude mcp add failgate -- \
  python -m failgate mcp

Long-running deployment: docker compose up -d --build brings up the API, a sandbox worker, PostgreSQL and Redis — see docs/deploy.md.

One real run on the demo repo (issue #21 → PR #22). The only human actions were opening the issue, commenting /failgate fix, and clicking merge.

About 10 minutes end to end, $0.013 in LLM calls. Here is the same run from the operator's side — failgate up keeps the service and webhook tunnel running and shows every state change:

_{Recorded with a real pseudo-terminal (pywinpty) and rendered with agg. Played at 2× with waits fast-forwarded — the board's up clock shows real elapsed time. up was restarted once before /failgate fix; the smee channel URL is masked.}

Explicit state machine, not a free-running agent. Each issue / PR is a Case; agents run inside bounded steps with budgets. One door for side effects. Every GitHub write goes through PolicyGate: idempotency keys, shadow mode, secret scanning, label policy. Separated roles. The exam writer, the fixer and the grader have different permissions: the fixer cannot touch tests or test config (enforced in the tool layer), and pushes through a separate GitHub App. Sandbox. Two phases: install through an egress allow-list proxy, run with no network, non-root, all capabilities dropped, read-only root, pid / memory / CPU limits. Platform. Two queue lanes so sandbox jobs never block triage (fast-event p95 448 s → 22 s), Redis + arq with crash redelivery, PostgreSQL + Alembic, OpenTelemetry traces to Langfuse. Measure first. Every component has an offline replay on real repo history ("time travel" to the commit before the fix), and every design decision is written down (what was tried, what failed and why) before the code changes.

One webhook = one trace (Langfuse).

Stack: Python 3.11 · FastAPI · SQLAlchemy 2 (async) · PostgreSQL / SQLite · Redis + arq · Docker SDK · LangGraph (fixer agent) · cosmic-ray (mutation) · bm25s + bge-m3 (retrieval) · OpenTelemetry · MCP Python SDK · Typer + Rich + prompt_toolkit.

• "Verified" ≠ correct. A weak exam lets a wrong fix through — that's why strength and hidden exams exist, and why they are shown, not hidden.
• Small, honest samples. 4 Python repos, 12 sampled issues each, one run each; human-style review was done by Claude with a "GitHub evidence required" rule.
• Python + pytest only, for now. Packages that need system libraries may not install in the sandbox. More languages are on the roadmap.
• Exam strength is a hint. It reacts to stronger tests (10 up / 0 down on SWE-bench + UTBoost, p = 0.002) but can't by itself tell a weak exam from a good one (p = 0.41).

• More languages. FailGate works on Python + pytest today. The exam → seal → grade design does not depend on the language; what is Python-specific is four adapters — sandbox install, test runner, failure signatures and mutation testing. Next up: JavaScript / TypeScript (Jest, Vitest), then Go and Java.
• More platforms. Gitee, alongside GitHub.
• A v0.1 release with a recorded walkthrough.

• Deploy with docker compose
• Set up the GitHub App
• Set up the Fixer App (only needed for /failgate fix)