Review the Evidence, Not the Agent’s Confidence

An agent saying a task is complete is not evidence; reviewers need diffs, tests, command results, assumptions, unresolved risks, and a reproducible path.

Claim Versus Evidence: Confidence Is Cheap

Treat the agent’s humanlike sentence as a claim; treat artifacts as evidence. Confidence scores, success messages, and polite confirmations are useful for triage, not for approval. While designing my Git-based agent workflow I treated each completion statement as a pointer to artifacts the reviewer must inspect. The difference between claim versus evidence is procedural: a claim asserts, evidence demonstrates and is verifiable by a second party.

Minimum Evidence Bundle: Define The Acceptance Gate

A reviewer’s checklist must be explicit about the minimum evidence bundle required to move a change forward. Don’t accept a statement of completion without this bundle attached to the commit or PR.

Minimum evidence bundle (what to require):

• A clear diff (git patch/PR) scoped to the intent, with a one-line summary and a short rationale.
• Repro steps that a reviewer can run in a clean environment (commands, expected outputs, and environment variables).
• Focused tests: unit, integration, or smoke tests that fail on the previous commit and pass on the change.
• Command results and logs from the agent’s run (stdout/stderr, exit codes, timestamps), redacted for secrets.
• A list of assumptions and known unresolved risks (what the agent could not verify).
• A rollback or mitigation path and the blast-radius estimate.

Require these artifacts as structured checklist items on the PR. If any item is missing, the submission is incomplete, not merely “auto-approved.”

Failed And Skipped Checks: Silence Is A Signal

An agent will often skip checks when it can’t access an environment or when a test is flaky. Capture skipped and failed checks explicitly. Make the agent produce a short table of checks with these columns: name, status (pass/failed/skipped), artifact location, and why a skip happened.

Treat failed and skipped checks as first-class signals:

• Failed: attach logs and the smallest reproducer that triggers the failure.
• Skipped: attach the reason and the least-privilege way for a reviewer to run it locally (e.g., a disposable credential stub or a mocked endpoint).

Do not accept silence. If a run lacks any check outputs, fail the review with a reproducibility request.

Reproduction By Another Reviewer: The Final Gate

The critical test of evidence is reproduction by another reviewer: can someone with the documented steps obtain the same observable results in a reasonably independent environment? Define a simple decision test:

Reproduction decision test:

1. Clone the repository fresh.
2. Checkout the change ref or PR branch.
3. Follow the provided reproduce steps verbatim in a disposable environment (container or VM).
4. Run the referenced tests and commands.
5. Compare outputs to the agent-provided command results.

If step 3–5 succeed with only the documented inputs, accept. If they do not, require the submitter (or the agent) to fix the bundle. Reproduction by another reviewer is not optional for changes that touch stateful or security-sensitive systems.

Tradeoffs, Costs, And Failure Modes

Requiring a full evidence bundle increases reviewer time and the surface area of artifacts to store. That cost is intentional: it converts an implicit trust decision into an explicit verification process. When this policy is wrong:

• Prototypes and exploratory experiments where speed matters and no production state is affected. Use a tagged “lab” workflow with lighter evidence requirements.
• Very small cosmetic edits where the blast radius is nil; keep the checklist minimal.

Common failure modes and mitigations:

• Non-determinism: capture random seeds, versions, and environment hashes; prefer hermetic runs or recorded fixtures.
• Forged outputs: require raw logs and the exact command lines used; use ephemeral run IDs or signed CI artifacts when available.
• Hidden privileges: record the identity and permissions the agent used; prefer least-privilege runners and short-lived tokens for reproduction.

Costs: storage for artifacts, CI compute to re-run reproduction steps, and reviewer time. Treat those as engineering investments—an investment that prevents larger rollbacks and firefighting later.

A Practical Checklist (Reusable Artifact)

Use this checklist as the PR acceptance gate. Mark each item pass/fail and include links where appropriate.

• Diff and one-line rationale included.
• Reproduce steps present and validated locally by author.
• Tests that fail on baseline and pass on change (attach test output).
• Agent run logs (stdout/stderr), exit codes, and run-id included.
• Failed and skipped checks enumerated with reasons.
• Assumptions and unresolved risks documented.
• Rollback or mitigation plan and blast-radius note.
• Reproduction by another reviewer confirmed (or an explanation why reproduction is impractical).

Use the checklist as an automated PR template so that an agent can’t skip enumeration. If an agent submits a PR, it must populate the checklist fields programmatically.

Operational Practices To Make This Work

• Instrument CI to produce canonical artifacts: a known path for logs, a reproducible container image tag, and a stable CLI wrapper for the reproduce steps.
• Make tests small and targeted; a reviewer should not re-run a full end-to-end suite to validate a trivial fix.
• Enforce least privilege: reproduction steps should not require production secrets. If they do, provide a mocked harness and an explicit test that demonstrates equivalence.
• Record provenance: which agent run produced the artifacts, which agent config and prompt were used, and the git ref of the agent’s workspace.

These practices narrow the blast radius and make failures actionable.

Grounded takeaway

An agent’s message is the start of a review, not the proof. Require a minimum evidence bundle, surface failed and skipped checks, and insist on reproduction by another reviewer before accepting changes that affect state, security, or availability. The extra upfront discipline costs time but reduces unknowns and rollback complexity. If you want a starter template, use the checklist above and enforce it in your PR workflow; for questions or a sample PR template, see /contact.