Skip to main content

Verify the agent's report

The agent says "committed", "file updated", or "all tests pass". Treat each of these as a claim about something outside the chat, and confirm it against the artifact itself: the repository, the file system, the run panel. The check takes less than a minute per claim; skipping it costs a review cycle later.

This page collects the checks in one place. Start with the three commands, use the IDE panels when you prefer to stay in the editor, then go through the checklist before you accept the change.

Three claims, three commands​

Run each command yourself, in the project root and on the branch the agent was working on. Do not copy the output from the chat: a code block that the agent pasted "to confirm" is model output, not a tool result.

Claim: "Committed"​

pwd
git branch --show-current
git log -1 --stat
git status --short

Expected: the working directory is the project you asked about; the branch is the one you expected; the hash and the files in git log -1 --stat match the agent's message; git status --short prints nothing. For a "pushed" claim, add git fetch && git log origin/<branch> -1 and compare the hashes.

If git status lists modified files, the commit did not happen, whatever the message said. If the agent quoted a hash, git cat-file -e <hash> tells you in one line whether git has ever seen it.

Claim: "Created or updated the file"​

ls -la path/to/File.kt
git diff --stat -- path/to/File.kt

Expected: the file exists, its timestamp is later than the start of the task, and the diff shows the content change you asked for. A file the agent "wrote" that appears in no diff and has an old timestamp was not written.

Claim: "Build passed" or "tests passed"​

A build claim needs an exit code and a timestamp later than the last edit. A test claim needs the number of tests, the test classes, the duration, and confirmation that the run happened after the change.

./gradlew test --rerun-tasks    # or: mvn -q test

Expected: a non-zero test count and a summary line from the runner itself. "All 38 passed" with no run attached is decoration, and "0 tests, exit 0" is a number too: a filter that matches nothing exits successfully.

The same checks in the IDE​

A JetBrains IDE keeps the same facts in its own panels, and none of these panels is written by the model.

  1. File edits: Agent Changes. Open Agent Changes. It lists the files the agent modified in the active chat, with a diff per file. Compare this list with the files the agent named, item by item. A file that is missing from the list was not edited in this chat. Agent Changes covers file edits only: it does not roll back the effects of an executed command, a migration, or a request to an external system (see Permissions and safety).
  2. Builds and tests: the run panel. When the agent runs a build or a test through an IDE run configuration, the result comes back as console output, test results, and compilation errors, and you see the same run in the IDE's run panel with counts and timing. Check that the last run is later than the last edit. If the only run predates the edit, the "tests green" claim is stale.
  3. Commits: the Git tool window. Explyt does not create commits for you. The Commit message generation tool drafts the description in the IDE's commit dialog; the commit itself is a git action, and git commands the agent runs go through the terminal like any other command. Open the Git tool window (Log tab) and compare the top commit and its file list with the agent's message, or run the commands from the previous section.
  4. Behavior: Debug mode. For a claim about behavior ("this fix removes the NPE") ask the agent to reproduce the scenario in Debug mode and inspect the values at a breakpoint. A single debugger run confirms the fix only for that scenario; keep the related tests in the loop.

Explyt's own documentation states the rule this page is built on: do not treat a chat response as a finished result on its own; ask the agent to run the appropriate configuration and report what exactly was verified.

Six traps where the check itself is fake​

Once you start verifying, the agent, or your own habits, will offer shortcuts. Watch for these six.

  1. A transcript is not a fact. The agent describes a tool call in prose or inside its reasoning, and the call never runs. The description reads exactly like a result. Only a tool result or your own command counts.
  2. The tool returned success for zero actions. A test filter that matches nothing exits 0. A Windows copy can fail without a non-zero exit. A compile-only check gets reported as "verified". Look at the count and the duration, not only at the exit code.
  3. The summary says "done" while the last step is pending. Session recaps and task trackers are generated from the model's picture of the work, which can be older than the last tool result. Trust the last raw tool output over the recap.
  4. The confirmation lives inside the agent's own message. "Here is git log to confirm:" followed by a code block is a claim, not evidence. Rerun the command.
  5. Cached green and the wrong directory. Gradle and Maven report success from cache when inputs did not change; multi-module repositories and git worktrees make it easy to verify a different checkout. Print pwd first; if the build says UP-TO-DATE for the module you expected to change, the change may not be there.
  6. Partial success reported as full. Three of five files written, two rejected by a permission rule, and the message says "updated the files". Compare the claimed list with the diff list item by item. A commit on a branch nobody asked for belongs here too: git log --all --oneline -5 catches it.

Checklist before you accept the change​

The first part covers the side effects the agent reported. The second part covers the change itself: a commit can be real and still wrong.

## Side effects the agent reported

- [ ] Working directory and branch printed before any check.
- [ ] `git log -1 --stat` shows the claimed hash and the claimed files.
- [ ] `git status --short` is clean after a "committed" claim.
- [ ] Every file the agent named appears in Agent Changes or `git diff --stat`.
- [ ] Build status comes from the run panel or the build tool's exit code, with a timestamp after the last edit.
- [ ] Test summary includes a non-zero count and a duration, from a run after the edit.
- [ ] Any command output pasted in the chat was rerun by me or matched against the raw tool result.
- [ ] The claimed file list and the actual diff list were compared item by item.

## The change itself

- [ ] The new or changed test fails on the code before the fix, where it is meant to prove a regression.
- [ ] The project compiles after the change, not only the edited file.
- [ ] The tests that cover the affected code were run after the last edit, not only the new test.
- [ ] The diff serves one task: no unrelated files, no unexplained dependency or configuration changes.

Whether the fix hits the root cause, and whether the tests mean anything, are review questions. For a team-level version of that review, see the Team acceptance rule.

Make the agent report evidence instead of prose​

You can move part of this work back to the agent with a project Rule or a section in AGENTS.md. The rule changes what "done" is allowed to mean: after any state-changing action the agent must run the corresponding check and include its raw output.

## Reporting side effects

After any action that changes state outside the chat (git commit, git push,
file create/delete, build, test run), do not describe the result in prose.
Run the corresponding check and include its raw output:

- commit  -> `git log -1 --stat` and `git status --short`
- file    -> `git diff --stat -- <path>` (or `ls -la <path>` for new files)
- build   -> the build tool's final status line and exit code
- tests   -> the test runner summary with counts and duration

If the check fails, returns an error, or runs zero tests, say so first,
before any summary. Never report a step as completed if its tool call
did not return success.

With this rule a missing check becomes visible: if the block is absent from the agent's turn, the step did not run. The rule helps; it does not replace your own git log.