member-console/docs/agent-runner.md

---
title: "Agent Runner"
audience: [developer]
summary: "How models from other families review this repository: what contains them, how a task is shaped, and what a finding must carry before it counts."
---

# Agent runner

A repeatable method for putting this repository in front of models that did not
write it. Two lanes use it. The **security lane** audits a named slice of the
code against a threat model. The **ideation lane** answers a design brief with
a proposal and a mockup. Both run the same containment and the same rules about
what comes back.

The runner is [`scripts/agent-runner/`](../scripts/agent-runner/README.md),
which documents the commands and the environment variables. This page states
the method.

## The problem this solves

The model that built a flow cannot audit it. It knows which inputs it already
validated and which branch it already considered, so the gap it left is the gap
it cannot see. A model from another family lacks that memory, and it also
carries different priors: published evaluations put the same frontier model far
apart on different bug classes, so the family that misses a class is not the
family that catches it.

The cost of that instrument is over-reporting. Models reading code unaided
invent vulnerabilities, cite lines that do not say what they claim, and rate
hardening advice as critical. Every rule below exists to keep that cost
bounded.

## Containment

The model never sees the working tree. The runner unpacks `git archive HEAD`
into a disposable copy, mounts that copy as the container's working directory,
and deletes it at teardown. The model may edit anything it likes; the edits go
nowhere, and uncommitted work is not exposed.

The model container sits on a Docker network marked `internal`, which has no
route off the machine. Its only exit is a proxy sidecar that denies every host
except an allowlist of the model APIs. The arrangement is fail-closed: a
misconfigured proxy costs the run its network instead of leaking the code.

One secret reaches a container: the sign-in file of the coding agent that lane
runs. For opencode that is the auth file holding the model-provider keys,
mounted read-only. For Codex CLI it is a copy of the ChatGPT token in a
throwaway home directory, writable because codex rewrites the file when it
refreshes the token, and removed with the lane. No other credential, and no
live database, identity provider or payment surface, is in the run at all.

## Shaping a task

A task names a **small set of files**, states a **one-sentence threat model**,
and asks **falsifiable questions** that each end in a yes, a no, or an explicit
"undetermined" with the code that settles it. The questions are the floor: the
model then hunts freely inside the slice, in whatever category it finds
something.

A task is not a bug-class checklist. A checklist inherits the reviewing model's
blind spots, because a class the model scores badly on stays badly scored when
it is listed, and it anchors the model away from whatever the list omits. One
benchmark scored a frontier model at F1 0.000 on authentication failures while
the same model scored 0.689 on SSRF; a checklist would have hidden that gap
behind a line item marked done.

The system context is supplied verbatim rather than left to the model to
assemble. Models are poor at gathering their own context, so the auth model,
the request pipeline in its actual middleware order, and the trust boundaries
are written out as fact, with file and line references the model can verify,
and prepended to every task. Prescriptive about slice and stakes; open about
bug class.

That shape fits a model that cannot hold the repository: the slice bounds what
it must read, the questions guarantee coverage, and a fresh container per slice
keeps its context clean. For a model that can hold the repository and plan its
own coverage, the same shape costs something. The system context is the
builder's map, and it names which couplings are deliberate, so the auditor
inherits the builder's judgment about the one place a fresh reader might
disagree. Slice boundaries are where cross-cutting bugs hide: a finding that
needs the exemption list and a webhook receiver belongs to neither slice. The
**open brief** drops both: no map, no file list, no questions, one run over
the whole repository. `open/bare.md` says only what to do; `open/stakes.md`
adds what the system is and from which positions it is attacked. Both keep
`REPORT.md` whole, because the evidence toll constrains the report, not the
search, and a looser search makes the toll matter more. The two shapes are run
side by side and their findings combined, under the union rule below.

A third shape delegates both search and report to a vendor's own audit agent:
the **Codex Security** task runs OpenAI's plugin over the same disposable copy,
under the same containment, and the plugin writes its own artifacts (a
findings file with a schema, a coverage file, a generated report). It is a
different instrument, not a variant of ours: it validates and proposes fixes,
and its report contract is not `REPORT.md`, so a task runs one or the other,
never both. Its output enters the same adjudication as every other candidate
list.

The ideation lane shapes a task the same way: the brief names the surface, what
survives any redesign, and the constraints the repository enforces, and it does
not name candidate solutions. A brief that lists the shapes it expects gets
those shapes back.

## Method rules

- **Union, never intersection.** Findings from several models are combined, not
  filtered by agreement. Coverage is the reason to run more than one model, and
  consensus filtering throws that coverage away: a model has reported a real,
  later-confirmed finding on one pass and reported nothing on the same slice on
  the next, so requiring a second reporter would have discarded it.
- **No self-review inside a run.** A model never re-checks or argues with its
  own findings in the same context. Self-refinement loops measurably raise the
  count of reported critical vulnerabilities without raising the count of real
  ones. Adjudication is a separate pass by a different reader, against the
  source.
- **Every claim is spot-checked before it counts.** A finding is a claim about
  this repository, and it is checked against this repository, at the cited
  path and line, before it enters any report or ledger. Claims fail this check
  in practice: one model reported a committed `node_modules` directory that
  does not exist here.
- **Candidates are not findings.** What a run produces is a candidate list.
  Only what survives adjudication and the spot check becomes an issue on
  the forge tracker.

## The evidence toll

A finding that does not pay all four is deleted before the report is written.

1. An exact `path:line-range` **and a verbatim quote** of the code. A model
   that cannot quote the line has not found the bug.
2. The **full path from attacker-controlled input to the dangerous operation**,
   naming each function it passes through. "Probably reachable" is not a path.
3. The **preconditions**: what the attacker must already hold, such as no
   session, a member session, an operator session, a webhook signing key, or
   control of DNS.
4. A **failure scenario** with concrete inputs and state, and the wrong outcome
   they produce.

Severity and difficulty are rated on two independent axes rather than combined
into a CVSS score, because the one number hides which of the two a reader
disagrees with. Hardening and defence-in-depth are reported in a separate
section from vulnerabilities, so advice cannot inflate the vulnerability count.

## Security audits: the scanners run first

The deterministic scanners run before any model reads the code, offline, over
the same snapshot; the model then triages their hits as one of its tasks. That
order is deliberate. A model adjudicating scanner output removes most of the
scanner's false positives at a small cost in recall, while a model reading code
unaided over-reports. [`testing.md`](testing.md) names the scanners and fixes
the order.

Scanner output is a lead, not a finding. A hit confirmed in the source becomes
a candidate and pays the evidence toll like any other; a hit the model judges
false is reported as false, with the reason.

## Definition of done for a review

1. Every task in the lane has run for every model in the set, and the run
   manifest records the audited commit, the model ids and the per-lane budget.
2. Each candidate finding has been adjudicated against the source by a reader
   who did not produce it.
3. Each surviving finding has been spot-checked at its cited path and line.
4. Survivors are filed as issues on the forge tracker; the rest are
   dropped, and a finding dropped for being unreproducible is recorded as
   such rather than silently deleted.
5. `teardown.sh` has run, and its residue check is clean.