Verification

An agent saying "tests pass" is a claim. Verification turns it into evidence: checked-in tests tied to each criterion, raw reports matched to the commit, and your CI as a second opinion.

Every finished objective should leave behind tests that keep guarding it after it ships. Planwright asks agents to write those tests, tags each one with the criterion or invariant it guards, and checks what the agent reports against what your CI reports for the same commit. When the evidence is missing, contradictory or shrinking, acceptance waits for a human, and the human's decision goes on the audit chain.

Planwright never runs your code. Agents run tests on their own machines under a set protocol, and your CI provides the independent result.

Each criterion gets a stable id, and the test that proves it carries a marker naming the objective and that id.

When an agent first claims an objective, each acceptance criterion gets an id (ac_1, ac_2, …) that never changes, and a marker that adds the objective's id prefix. The agent writes a checked-in test whose title includes the marker, for example "refuses a blank note [AC:3f2a91c0:ac_3]", and submits the raw Playwright JSON or JUnit XML report. Every objective has an ac_1, so a result counts only toward the objective its marker names, even when one report holds the whole suite. Criteria are tagged as a behavior (what the change does) or an error case (what it must do when something goes wrong, like an expired token or bad input), so failure paths get tests too. A criterion the agent can't test is listed as such and left for the reviewer to check by hand.

Older tests tagged with a bare marker like [AC:ac_3] count only for the objective that first reported them. Tests for objectives that shipped before markers existed can be linked later: an agent adds the qualified marker, and the link waits for approval on the Verification page. An objective's own tests need no approval: once the objective is accepted, the tests its run reported are linked automatically, and the audit log records which run reported each one.

Two sources, kept apart, so a reviewer can see when they disagree.

If the two disagree, acceptance is blocked at Enforce. A reviewer who has looked at both can accept anyway with a reason, which is audited. A CI re-run that agrees with the agent clears the block without an override.

Statements every change must keep true, each guarded by tests and scoped to the code it covers.

An invariant is one line, such as "Audit records are never updated or deleted", with a permanent key like audit-append-only. Tests whose titles carry [INV:audit-append-only] guard it, once a human approves the link. Its scope is a set of path globs. When a run's diff touches a file in scope, the invariant's guarding tests must have passed in that run. An invariant with no scope is never checked against diffs. Agents propose new invariants, drawn from your context files or from the criteria of finished objectives: the agent that built an objective judges each of its criteria when it hands the work off, and criteria worded like rules come back in later sessions too. A human approves every proposal before it applies.

A failing invariant can't be overridden at acceptance. The only way past it is an exemption that waives that invariant for the objective.

One level per project decides whether verification work happens and whether it blocks acceptance.

The workspace sets the default and a project can override it. Projects created before levels existed start at Off and get a one-time prompt to turn verification on. Any member except a viewer can change a level, and every change is audited.

Switching a project from Off to Observe starts a one-time backlog for your agents, then changes how each new objective is verified. The Levels & exemptions tab shows how big that backlog is for your project.

There's no time estimate for finishing the backlog: it depends on how often your agents run and how quickly your team reviews. Start at Observe to see what verification finds before moving to Enforce, where it blocks acceptance.

Lower the level for one objective, or waive specific invariants, with a reason.

Some objectives can't meet the project's level, like a spike or a change to code with no test harness yet. A member can set an exemption on the objective directly, and it applies at once. An agent can request one, but the stricter level stays in force until a human approves the request. An exemption only ever lowers the level or waives invariants, never raises them, and every request and decision is audited with who made it.

The test commands agents are handed, checked into your repo and approved by a human.

verify.yml tells Planwright how to run a given set of tests and where the reports land. At claim, Planwright fills in the markers the run needs, the objective's criteria plus any invariants it touches, and hands the agent the exact commands. The agent runs them on a clean checkout of its commit and submits the raw reports. Selected tests missing from the reports need the reviewer's approval note.

version: 1
setup: pnpm install --frozen-lockfile
run: pnpm vitest run -t "{grep}" --reporter=junit --outputFile=reports/junit.xml
reports:
  - path: reports/junit.xml
    format: junit

setup

Optional. Runs once before the tests.

run

Required. Must contain {grep} (a regex of the markers) or {markers} (a space-separated list).

files

Optional. Runs whole test files: must contain {files}, where Planwright puts the test files the test-impact index selected. Write its report to its own path under reports.

reports

Required. Paths and formats: junit, playwright, lcov, cobertura or jacoco. At least one must be junit or playwright.

coverage

Optional. Measures one test file for the test-impact index: must contain {file}, and writes a report listed under reports as lcov, cobertura or jacoco.

coverage_noise

Optional, default 0.5. A source file covered by more than this share of the measured test files (a db client, config, a logger) is left out of every mapping. Applies once five test files are measured.

full

Optional. The whole suite, used only to generate a CI workflow. Never handed to an agent.

Planwright only uses the latest approved version. An agent can draft or change the file as part of an objective, and accepting that objective approves it. A human's commit is approved when it lands on the default branch. Changing an approved file needs an approval note at acceptance. If a report from the commands finds no tests, or a command fails to start, agents are told the commands may be out of date and to fix them.

Which source files each test file covers, so a change runs every test that can catch a break in it, tagged or not.

With a coverage command in verify.yml, agents measure test files one at a time and Planwright stores the source files each one covered, with the commit it was measured on and the run that measured it. A run measures only the test files its change adds or modifies, the files holding its selected tests that haven't been measured yet, and mappings its diff makes due: the test file changed, or the diff touches a file the mapping covers or one in the same directory. Never the whole suite. At record_diff, every test file whose mapping covers a changed file joins the run's selection and runs through the files command. The review drawer lists each added test file with where its coverage came from, this run or an earlier one, and the changed files no recorded coverage includes, without asking anyone to run the full suite for them. Each invariant also gets a derived scope, the coverage of its [INV:] test files, and diffs are matched against it as well as its hand-written globs.

The coverage that narrows a selection is measured by agents, so the reviewer always sees whether this same run measured it, an earlier run did, or it came from a seed. In an existing repo the index starts empty and fills in as agents work. To fill it on day one, Suite health offers a seed script built from verify.yml and your known test files: you run it with your Planwright token, it measures every test file once and uploads each report, and a test file whose coverage command fails is reported and skipped. It's effectively a full-suite run, so it's yours to run; agents are never asked to.

An agent can pass a suite by trimming it. Planwright catches existing tests that were deleted, skipped or renamed.

Planwright reads the run's whole diff from GitHub rather than trusting the agent's own account of it, and flags existing tests the run removed or skipped. Moving a test to another file isn't a removal. A rename blocks only when the old title guarded a criterion or an invariant, since the link goes with it. At Enforce, flagged tests block acceptance until the reviewer approves them with a note, so the suite only shrinks on purpose.

One page per project for what the tests prove and the invariants every change must keep.

The review drawer shows the same evidence per objective. It shows test evidence only when something needs attention, alongside Code Signals.