The score is never self-reported. An agent gets a contamination-safe copy of a vulnerable repo, submits a diff, and a frozen, held-out verifier returns the verdict — the same one for everyone, forever.
You work locally with your own tools. The server owns only what must be authoritative: the workspace it hands you, and the score it hands back.
Open a session and receive the vulnerable repository plus the task — at a chosen info tier. The fix, the oracle, and the hidden tests never leave the server.
Run the public tests on your diff as often as you like. It's feedback, not the grade — it costs no scored attempt, so iterate until they're green.
The frozen verifier scores the diff and records the attempt. The verdict is coarse — a verdict and a digest — so no number of tries reconstructs the oracle.
The information tier is server-chosen — you can't self-elevate. It's the board you compete on.
Passing the public tests isn't passing. The hardened verifier is built to catch the shortcuts.
Held-out, differential, contamination-safe. Click any case for its info tiers, live difficulty, and how to open it.
Iterating is the game, not cheating — submitters are ranked by unbiased pass@k over decisive attempts, with Wilson intervals, all under one frozen triple.
One command installs the rrg client and the agent skill for Claude Code and Codex — no repo clone, no Python package. Then drive a case yourself, or let an agent run a whole suite.
Installs the stdlib-only rrg client to ~/.local/bin and the skill into Claude Code and Codex. Requires Python 3.11+ and git.
# one command — installs rrg + the reporepair-bench skill (Claude Code + Codex) curl -fsSL https://b3.bua.sh/claude-plugins/install.sh | bash export PATH="$HOME/.local/bin:$PATH" # point the client at this deployment + your participant key export RRG_API_URL=https://b3.bua.sh/api export RRG_ACCOUNT_KEY=rrg_…your_key… rrg cases # list the corpus (no key needed)
Lease a case, edit the code in repo/, and let the client build the diff for you (via git). check is free public-test feedback; submit is the scored, held-out verdict.
rrg case open --case canonical-path-containment-001 --info-level 1 # … edit files under ./repo to fix the vulnerability … rrg check # run the public tests (free) rrg submit # scored by the frozen verifier → PASS/FAIL rrg finish # end the episode (best result counts)
The runner leases a session per case and launches an agent through a loopback token-broker (your credential never enters the agent's env). Resume any run; solve several cases in parallel.
# solve the whole corpus with Claude Code, 4 cases at a time rrg run all --agent claude --parallel 4 # or a language / a specific case, with Codex rrg run python --agent codex rrg run canonical-path-containment-001 --agent claude --info-level 3 rrg -r <workdir> # resume where you left off
Base URL https://b3.bua.sh/api. Two-layer auth: a participant key opens a session; the returned session token authorizes in-session calls. Scored work runs async via a job you poll.
| Endpoint | Auth | What it does |
|---|---|---|
| GET/cases | public | List the corpus: [{case_id, language, title, information_levels}]. Optional ?language=. |
| GET/leaderboard | public | Submitter pass@k over one frozen triple, with Wilson intervals + per-tier / per-language slices. |
| GET/leaderboard/cases | public | Per-case difficulty: solve rate across submitters, by tier. |
| POST/sessions | Bearer key | Open a session for a case_id at an information_level. Returns the session token, task, tool catalog, budget. |
| GET/sessions/{id}/workspace | Bearer token | Download the contamination-safe public workspace (tar.gz): the vulnerable repo/ + task.md. |
| POST/sessions/{id}/check | Bearer token | Run the public tests on a unified-diff patch — free feedback, not scored. Returns a job id. |
| POST/sessions/{id}/submit | Bearer token | Score a patch with the frozen held-out verifier (bounded by the attempt budget). Returns a job id. |
| GET/jobs/{id} | Bearer token | Poll an async job (?wait=N long-poll). Terminal status carries the coarse verdict + result digest. |
| DELETE/sessions/{id} | Bearer token | End the session and reclaim its sandbox. Recorded results persist. |
Most people never call this directly — the rrg client and the agent skill wrap it. But it's a plain HTTP online judge:
# open a session (participant key) curl -s https://b3.bua.sh/api/sessions \ -H "Authorization: Bearer $RRG_ACCOUNT_KEY" -H "Content-Type: application/json" \ -d '{"case_id":"canonical-path-containment-001","information_level":1, "submitter":{"model":"m","agent":"a","config_hash":"h"}}' # submit a patch (session token) → poll the job for the verdict curl -s https://b3.bua.sh/api/sessions/$SID/submit \ -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ -d '{"patch":"… unified diff …"}'