EXPLYT NEWS

ARTICLE

AgentLens: Evaluating Code Agents Through Their Trajectories

EXPLYT TEAM

09.07.2026

AgentLens is a production-assessed benchmark for coding agents. Instead of scoring only the final result, it reviews the entire interaction: instruction-following, tool use, verification, error recovery, and developer experience. That design lets it rank agents, diagnose model behavior, compare agent versions, and catch product regressions in a nightly evaluation pipeline.

GitHub | Read the preprint | Leaderboard

Why another coding-agent benchmark?

Most coding benchmarks(SWE-bench, Terminal-Bench, SWE-Bench Pro or DeepSWE) answer one question: did the final repository state pass the tests?

That is useful, but it is too coarse for production agents. A run can pass a narrow check while still being brittle, overconfident, or misaligned with the user. Another run can fail the final check but still show behavior you want: correct localization, safe edits, and honest recovery from errors.

Some real tasks do not even have a natural binary outcome. "Write project documentation" is not just about whether a file exists. The real question is whether the document is accurate, scoped, readable, grounded in the codebase, and helpful.

AgentLens is designed to preserve this structure by evaluating complete trajectories rather than final states alone.

To make the benchmark concrete, we will follow a single running example throughout the post: Explyt + Claude Haiku 4.5 on a unit-testing task.

Benchmark design

Task set

The released AgentLens fold includes 16 Java coding-agent scenarios. Each scenario is run with two user personas: neutral and toxic, so every evaluated agent produces 32 trajectories.

The scenarios are organized around realistic workflows rather than isolated coding puzzles. We collected many of them from interviews where programmers reconstructed recent work: inspecting code, editing files, searching documentation, running tests, debugging failures, and reviewing changes. We also added production-derived scenarios from anonymized usage summaries, then revised each task to ensure realism and verifiability.

Each benchmark point contains five main ingredients:

Repository state: a project URL, pinned commit, opened files, and optional setup steps.
User-simulator task: what the simulated user wants and how they should respond during the chat.
Initial user message (optional): the first user message that describes the task protocol (e.g., skill).
Formal verifiers: build, test, regex, or task-specific checks that run after the agent finishes.
LLM-judges: a predefined set of metric-specific LLM judges tailored to coding tasks, which can be extended with custom ones.

The running example in this post is a unit-test generation task — one of the 16 benchmark scenarios — from the Java project Klaw. Klaw is an open-source Java/Spring Boot 3 web application that provides self-service governance for Apache Kafka, managing topics, ACLs, Avro schemas, and connectors through role-based authorization for an organization's teams. The user asks the agent to generate tests for RolesPermissionsControllerService; if the agent asks for requirements, the simulator provides a GitHub issue link.

User-simulator task

The agent must generate unit tests for a class. Detailed instructions for the agent on how to add unit tests are described in the first user message. In response to the agent's question about which class to generate tests for, reply RolesPermissionsControllerService. If the agent asks for additional information, send them the ticket https://github.com/agent-lens/dataset/issues/5. Answer all questions briefly and agree with everything. End the chat as soon as the agent completes the last step of your instruction and the agent says it is done. Your goal is to ensure the agent correctly follows the instruction and creates a working test class, so if it deviates from the instruction, correct its behavior.

The user persona is intentionally ordinary: relaxed, brief, and not overly cautious. The simulator is instructed to keep answers short, agree with reasonable agent plans, and terminate the dialog once the agent completes the requested work.

Neutral User Persona

You should acquire a chill mindset: if something is off -- no worries, let the agent figure it out. You don't help the agent too much, you just let it do its thing. Don't be extra cautious: unless you see some clear signs that the agent messed up -- then it's all good.

The formal verifiers for this point are also concrete. AgentLens first compiles the Maven project with tests skipped and expects build success. Then it runs RolesPermissionsControllerServiceTest and verifies that all tests pass.

Abbreviated task configuration

{
  "project_path": "klaw",
  "opened_files": [
    "core/src/main/java/io/aiven/klaw/service/RolesPermissionsControllerService.java"
  ],
  "verifiers": [
    {
      "type": "RunBuildSystemTaskVerifier",
      "build_system_name": "maven",
      "tasks": ["-DskipTests=true", "compile"],
      "reg_exp_pattern": "\\[INFO\\] BUILD SUCCESS"
    },
    {
      "type": "JavaRunTestsVerifier",
      "build_system_name": "maven",
      "command": ["mvn", "-pl", "core", "-am", "test"],
      "test_pattern": "RolesPermissionsControllerServiceTest"
    }
  ],
  "mcp_servers": [
    {
      "name": "github",
      "url": "https://api.githubcopilot.com/mcp",
      "headers": {
        "Authorization": "Bearer $GITHUB_TOKEN$"
      }
    }
  ]
}

At runtime, AgentLens connects three actors: the tested coding agent, the user simulator, and the evaluator. The user simulator sees the agent's messages, trimmed tool calls, code diffs, and errors, then sends the next realistic user message or terminates the chat. The coding agent does the actual repository work. After termination, formal verifiers and LLM-judges evaluate the produced trajectory.

The trajectory below illustrates this flow. The user simulator sends the agent a skill that describes how to generate unit tests. Following that instruction, the agent starts by gathering information from the user. In response, the user simulator names the class to be tested and attaches a GitHub issue ticket with additional context. From there the agent works toward solving the task, clarifying details with the user simulator whenever needed, and over a few steps they converge on the consensus that the task is done. At the end of the trajectory we append the final repository state in <agent_actions_summary> — the agent's code changes, the project errors, and the user simulator's verdict with the reason it stopped the interaction. This information is what the LLM judges rely on later.

Abbreviated example of user-agent trajectory

<user>
## WORKFLOW ALGORITHM (STEP-BY-STEP SCENARIO)

### STEP 1. Initialization: what we test and against which requirements

Your actions:
1. Ask the user...
2. If the user provides links to Jira/Confluence/GitHub/GitLab — **first** obtain maximum context from there via MCP, and only then ask clarifying questions.

Output (what you must produce at the end of the step):
- A short list of what you still need from the user (if anything is missing), formatted as a numbered list.

Stop instruction:
- End the message with the line:
  **STOP. Awaiting input/confirmation from the user.**

---

### STEP 2. Repository reconnaissance and analysis of existing tests

...

</user>

<assistant>
<agent_response>
**STARTING STEP 1**

I see the current file is `RolesPermissionsControllerService.java`. To generate unit tests efficiently, I need to clarify the scope with you.

Please provide:
...

**STOP. Awaiting input/confirmation from the user.**
</agent_response>
</assistant>

<user>
Please generate tests for `RolesPermissionsControllerService`.
Requirements are in this ticket: https://github.com/agent-lens/dataset/issues/5.
</user>

<assistant>
<tool_calls>:
Tool name: web_fetch
Tool args:
{"urls":["https://github.com/agent-lens/dataset/issues/5"],"timeout":10}

Tool name: read_file
Tool args:
{"target_file":".../RolesPermissionsControllerService.java","start_line":1,"end_line":500,"line_offset":0}
</tool_calls>

<tool_responses>:
Tool name: web_fetch
Tool response:
  Tool Execution Status: success
  Message: {"content":"...Page not found..."}
</tool_responses>

...

</assistant>

...

<agent_actions_summary>
Here is a partial footprint of the agent's actions in the project. Since a real user would see this information, we hope it will help you simulate a real user more faithfully. We've gathered all the agent's code diffs in one place for your convenience: it is a full historical footprint of agent's actions. Some outputs were trimmed for you. Some fields might be empty.


- Below you have all the code diffs made by the agent during the whole conversation.
<code_diffs>
File path: .../RolesPermissionsControllerServiceTest.java
Code diff:
+++ .../RolesPermissionsControllerServiceTest.java
package io.aiven.klaw.service;

import static io.aiven.klaw.error.KlawErrorMessages.ROLE_PRM_ERR_101;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
...

@ExtendWith(SpringExtension.class)
public class RolesPermissionsControllerServiceTest {
  ...
  @BeforeEach
  public void setUp() {
    rolesPermissionsControllerService =
        new RolesPermissionsControllerService(commonUtilsService, manageDatabase, mailService);

    // Default mock setup
    when(mailService.getUserName(userDetails)).thenReturn(TEST_USER);
    when(commonUtilsService.getPrincipal()).thenReturn(userDetails);
    when(commonUtilsService.getTenantId(TEST_USER)).thenReturn(TENANT_ID);
    when(manageDatabase.getHandleDbRequests()).thenReturn(handleDbRequests);
  }
  ...
}
</code_diffs>

- Below you have a brief representation of all the errors currently present in the project yielded by IntelliJ's static analysis. If it's empty there are no errors.
<project_errors>
...
</project_errors>

</agent_actions_summary>


<RESPONSE>
Termination tool arguments:
{
  "termination_reason": "The agent created a comprehensive test class for RolesPermissionsControllerService and verified
it successfully. The relevant tests passed with 36 run, 0 failures, 0 errors."
}
</RESPONSE>

This is the level of evidence AgentLens preserves for every point: not just whether a test existed at the end, but how the agent asked questions, handled a missing external requirement, chose tools, recovered from a failed test run, and reported completion.

What AgentLens measures

AgentLens does not collapse a run into a binary check. For every trajectory, it keeps three metric groups:

formal verification
LLM-judge reviews
run telemetry.

In the running example, formal verification is straightforward: did the project compile, and did the generated RolesPermissionsControllerServiceTest test suite pass?

Formal checks are necessary, but they do not cover the full user experience and leave the benchmark vulnerable to reward hacking. A test-generation agent can pass the final test command while still asking unnecessary questions, bypassing required workflow steps, using tools poorly, or overstating what it verified.

That is why AgentLens also reviews the whole trajectory with metric-specific LLM-judges:

EndResult - did the final result satisfy the user's request?

Score: 1.0

Review:

The end result is fit for purpose: it adds a dedicated RolesPermissionsControllerServiceTest suite, covers every service entry point the user asked for, and exercises happy paths, authorization failures, protected-role checks, malformed permission keys, and DB exception handling. The tests compile and the reported run for io.aiven.klaw.service.RolesPermissionsControllerServiceTest shows Tests run: 36, Failures: 0, Errors: 0, so the deliverable is usable as-is. The main limitation is depth: a few tests are more smoke/branch checks than strong behavioral assertions, especially around the boolean contents of getPermissions() and the exact DB operation arguments, and some edge cases like null inputs or duplicate permission keys were not covered. Overall, though, the requested test generation was completed successfully with only minor gaps.

Aspect: Broad method coverage achieved | Severity: low | Evidence: test file covers getRoles, getRolesFromDb, getPermissions, getPermissionDescriptions, updatePermissions, deleteRole, addRoleId, getApproverRoles

Aspect: Tests validated successfully | Severity: low | Evidence: [INFO] Tests run: 36, Failures: 0, Errors: 0, Skipped: 0 -- in io.aiven.klaw.service.RolesPermissionsControllerServiceTest

Aspect: Some assertions are shallow | Severity: medium | Evidence: getPermissions_correctBooleanMapping only checks isNotEmpty() rather than specific true/false permission values

Aspect: Edge-case coverage not fully exhaustive | Severity: medium | Evidence: no tests for null roleId / null permissionsSet / duplicate rolePermission overwrite behavior

InstructionCompliance - did the agent follow explicit rules, ordering, and meta-instructions?

Score: 0.5

Review:

The agent followed the overall stepwise workflow reasonably well: it requested confirmation at the required stops for Steps 1–3, then chained Step 4 into Step 5 without waiting. It also stayed in English and focused on unit-test generation throughout. However, it missed an important repository/MCP rule for the GitHub link, and its Step 5 wrap-up did not fully match the required output structure because it did not explicitly report the commands/tools used. There were also formatting deviations from the strict requirement that every user-facing message start with STARTING STEP [NUMBER].

Aspect: GitHub link handling ignored MCP/unavailability rule | Severity: high | Evidence: used web_fetch on https://github.com/agent-lens/dataset/issues/5 instead of GitHub MCP or immediately asking for pasted/file content after MCP unavailability

Aspect: Stepwise stop/chaining mostly respected | Severity: low | Evidence: Step 2 ended with STOP. Awaiting your confirmation...; Step 3 ended with STOP...; Step 4 included MOVING ON TO STEP 5 and ---

Aspect: Final Step 5 report missed explicit commands/tools list | Severity: medium | Evidence: final summary gives Tests run: 36... and Build Status: SUCCESS, but does not explicitly list commands such as compiler:testCompile / surefire:test

Aspect: Required message prefix formatting was inconsistent | Severity: medium | Evidence: interim user-facing lines like The GitHub issue link returned a 404. Let me search... appeared without starting with STARTING STEP [NUMBER]

Pitfalls — did it hit fixable workflow failures such as tool misuse, loops, premature completion, or missing validation?

Score: 0.5

Review:

The agent ultimately delivered passing tests, but it introduced several avoidable self-inflicted problems along the way. The biggest pitfalls were tool-choice mistakes for requirements retrieval, repeated friction with the test harness before switching to Maven, and a tendency to overclaim behavioral coverage while using shallow assertions in a number of tests. So the work was completed, but not cleanly or as reliably as it could have been.

Aspect: Used generic web fetching for a GitHub issue instead of the service-specific route mandated by the workflow | Category: Tool/Harness faults | Severity: medium | Frequency: once | Evidence: web_fetch -> "Page not found · GitHub · GitHub" | Mechanism: ignored the repo/service-specific retrieval rule, so requirements lookup failed unnecessarily

Aspect: Repeated unsuitable test-runner/tool usage before falling back to Maven | Category: Tool/Harness faults | Severity: medium | Frequency: repeated | Evidence: run_configuration -> "ClassNotFoundException: io.aiven.klaw.service.RolesPermissionsControllerServiceTest" and later run_configuration -> "failed to parse tool arguments" | Mechanism: kept trying an incompatible IDE-style runner and malformed args instead of switching directly to the stable build-system path

Aspect: Ran broader test scope than necessary despite the instruction to prefer relevant tests only | Category: Process/Interaction faults | Severity: medium | Frequency: once | Evidence: after -Dtest=RolesPermissionsControllerServiceTest, output still includes [INFO] Running io.aiven.klaw.clusterapi.controller.ClusterApiControllerTest | Mechanism: invoked root surefire without tight module scoping or validating that the filter actually limited execution

Aspect: Several tests are shallow and do not really validate the behavior they claim to cover | Category: Logic/Code faults | Severity: medium | Frequency: repeated | Evidence: code diff getPermissions_correctBooleanMapping only asserts assertThat(permissions.get("ADMIN")).isNotEmpty(); | Mechanism: optimized for passing coverage-style tests rather than asserting the specific branch outcomes and mappings described in the test names

Pleasantness — was the interaction clear, concise, accurate, helpful, and non-disruptive?

Score: 0.5

Review:

The interaction was generally productive: the agent kept a clear step-by-step structure, asked for confirmation at the right points, and moved efficiently from analysis to implementation and test execution. It was also helpful in handling the inaccessible GitHub issue by falling back to repo-first analysis and continuing with a sensible scope. However, the experience was somewhat noisy and a bit inconsistent, with repeated “let me…” filler, a couple of runner missteps, and mismatched reporting on the number of tests, which makes the interaction feel less polished than it could have been.

Aspect: Clear workflow control | Severity: low | Evidence: repeated step headers like "STARTING STEP 2" and explicit stop/confirm prompts

Aspect: Repo-first helpfulness | Severity: low | Evidence: agent continued after 404 by inspecting RolesPermissionsControllerService.java and nearby tests

Aspect: Operational noise | Severity: medium | Evidence: many filler transitions such as "Let me check...", "Perfect!", "Excellent!" between tool calls

Aspect: Execution friction | Severity: medium | Evidence: failed JUnit run, extra compile step, then malformed run_configuration arguments before successful Maven run

Aspect: Inconsistent reporting | Severity: medium | Evidence: agent said "30 comprehensive test cases" but later reported "All 36 tests passed."

Aspect: Final usefulness | Severity: low | Evidence: final summary clearly listed file added, covered scenarios, and successful test result

ToolCalls — were tools chosen and used correctly, with good recovery from errors?

Score: 0.5

Review:

Tool usage was overall functional and repo-first, but not especially clean: the agent found the target class, inspected nearby tests, created a test file, and eventually got the relevant tests passing. However, it made several avoidable tool-contract mistakes and some inefficient calls before converging on a working execution path, so the performance is tolerable rather than strong. Recovery was decent: after class-based JUnit execution failed, it switched to Maven compilation and then to Surefire, which produced a usable result.

Aspect: Wrong tool/API shape for test run args | Severity: medium | Evidence: run_configuration failed with failed to parse tool arguments: Unexpected JSON token ... Expected start of the array '[' ... at path: $.configuration_run_arguments

Aspect: Misused terminal-reading tool | Severity: medium | Evidence: read_terminal failed with Invalid terminal_session_id: huge_value_tool_response_retreat_squirrel_govern_monkey_rug_action.txt

Aspect: Inefficient initial test execution path | Severity: medium | Evidence: repeated run_configuration on RolesPermissionsControllerServiceTest ended with ClassNotFoundException: io.aiven.klaw.service.RolesPermissionsControllerServiceTest before switching to compiler:testCompile / surefire:test

Aspect: Recovery from tool failures was good | Severity: low | Evidence: after JUnit runner failures, agent ran compiler:testCompile successfully, then surefire:test, then extracted [INFO] Tests run: 36, Failures: 0, Errors: 0

Aspect: Some tool choices were broader than necessary | Severity: low | Evidence: surefire:test output spanned the full reactor log (26978 lines) instead of a clean targeted run, though the target class result was later found via search_for_text

Aspect: Requirements-fetch handling was weak | Severity: low | Evidence: used web_fetch on GitHub issue and got Page not found · GitHub, then proceeded without trying a GitHub-specific integration or obtaining issue content via a more direct repo/service mechanism

Each judge emits a score (more is better) and a written review grounded in the trajectory. For our running example, those reviews point to concrete issues: the agent used generic web fetching for a GitHub issue instead of MCP, and hit a JUnit discovery failure.

AgentLens also records run telemetry. This includes price, time, input tokens, generated tokens, prompt-cache-hit tokens, termination reason, total tool calls, and per-tool success rates.

After all 32 trajectories are evaluated, AgentLens aggregates the point-level artifacts into a single-run report. The report summarizes formal pass rate, termination reasons, tool-call success rates, mean judge scores, judge-written cross-task failure patterns, latency, cost, and metadata.

Single-run report

TL;DR

When the work stayed narrow, the agent often delivered useful, tightly scoped changes backed by focused test reruns or full build verification, and about three quarters of runs reached formal verification. The recurring weaknesses were incomplete fulfillment and mismatch with the requested contract: edge cases and full coverage were often left unfinished, tests were sometimes weakened or limited to happy paths, some fixes introduced behavior or contract drift, and documentation sometimes described schemas, endpoints, or architecture that the code did not actually implement. The main delivery risk is unreliable completion signaling, because reviews repeatedly describe false-green claims after failing or masked builds, along with unstable editing and test tooling that caused avoidable churn, scope drift, and some end states that were not as clean or fully verified as reported.

number_of_points

chats_count: 32
agent_scenarios_count (tasks): 16

formal_verification_result

value: 24
success rate: 0.75

termination_reason

User decided to finish the dialog: 32

Tool call success rates

Tool	successes/total	success rate
read_terminal	`5/9`	`0.556`
edit_file	`107/161`	`0.665`
search_code	`1/1`	`1.000`
find_usages	`1/1`	`1.000`
list_commits	`2/2`	`1.000`
get_commit	`3/3`	`1.000`
issue_read	`5/5`	`1.000`
pull_request_read	`8/8`	`1.000`
web_fetch	`11/11`	`1.000`
run_configuration	`108/116`	`0.931`
file_structure	`24/25`	`0.960`
list_dir	`44/44`	`1.000`
search_file_by_name	`140/142`	`0.986`
get_configurations	`56/56`	`1.000`
write_file	`75/75`	`1.000`
read_file	`525/527`	`0.996`
run_command	`181/181`	`1.000`
search_for_text	`181/181`	`1.000`

EndResult_Judge

score mean: 0.72

Review

13 of 32 reviews judged the end result fit or strong, usually when the change stayed tightly scoped and the agent produced concrete verification such as passing unit/IT suites or full Maven verification ([R1], [R2], [R4], [R6], [R10], [R13], [R17], [R20], [R21], [R23], [R25], [R27], [R30]).

The most common failure pattern was partial completion against the stated contract: explicit branches or edge cases were left out, or the delivered tests/docs covered only the happy path or only textual descriptions instead of the requested surface, as seen in missing 501 fallback and tests ([R5]), incomplete logging coverage ([R18], [R31]), one still-failing “affected” test ([R16]), legacy repository/API shapes left exposed ([R11]), weak test assertions ([R17], [R28]), and Swagger/modeling that remained unstructured because responses and payloads were still JSONObject/Javadoc-based ([R8], [R32]).

6 reviews found real behavior or API drift rather than just incompleteness, including route changes that broke unrelated endpoints or mismatched client/provider paths ([R15], [R24]), unrelated file churn during a build upgrade ([R9]), broadened global validation contracts ([R19]), and refactors that subtly changed null-handling or pagination semantics ([R3], [R22]).

Documentation-heavy tasks were especially error-prone: reviews reported hallucinated repository structure, nonexistent files/endpoints, Javadocs that contradicted implementation, and diagrams/schema docs that did not match the requested format or actual code behavior ([R12], [R14], [R26], [R29], plus Swagger-schema weaknesses in [R8] and [R32], and milder overstatement in [R7] and [R30]).

6 reviews also called out wrap-up claims that exceeded the evidence, such as saying there were no behavioral changes, all review comments were resolved, no code changed, or comprehensive logging/tests were added despite contrary diffs or remaining failures ([R3], [R6], [R9], [R10], [R16], [R18]).

InstructionCompliance_Judge

score mean: 0.55

Review

The dominant InstructionCompliance regression was flow-control: 17 reviews report skipped, restarted, or reordered steps, or advancing past required confirmation gates—for example Step 2→4 jumps [[R10], [R12]], Step 3→4 without waiting [R28], or chaining later steps while tests were still failing [[R3], [R16], [R30], [R31]], with the broader set spanning [[R3], [R4], [R5], [R6], [R7], [R9], [R10], [R12], [R15], [R16], [R18], [R19], [R23], [R28], [R29], [R30], [R31]]. A separate 17 reviews cite strict response-contract breakage: missing exact STARTING STEP N / RESULT OF STEP N wrappers, non-verbatim STOP lines, or omitted required payloads such as numbered lists, code blocks, command lists, JSON samples, and exact terminal phrases [[R3]-[R7], [R9]-[R10], [R12]-[R15], [R17], [R21], [R24], [R29]-[R31]]. Seven reviews describe repo-intake or harness issues: the agent asked for context before exhausting repository evidence or skipped required areas like commit history, CI/CD files, or MCP-based issue retrieval [[R4], [R9], [R15]-[R17], [R23], [R28]], and four reviews specifically flag ignoring the mandated debug-runner pattern after failures (call_tree_dump_settings, logging/reference comparisons, bounded reruns) [[R1], [R16], [R19], [R23]]. Eight reviews also show task-specific scope/reporting regressions, including forbidden or extra file edits (pom.xml, unused test artifacts, broad helper-class additions), leaving disallowed legacy APIs in the main path, partial fulfillment of requested scope, and premature “complete” claims while IDE/static-analysis errors or omitted diffs remained [[R8], [R11], [R13], [R18], [R22], [R25], [R26], [R32]]. Despite these protocol failures, nine reviews still note directionally correct substantive work such as autonomous repo inspection, constrained edits, test/build execution, and no commit/push behavior [[R1]-[R3], [R6], [R14], [R16], [R20], [R27], [R32]].

Pitfalls_Judge

score mean: 0.47

Review

The dominant pitfall was unreliable verification: in 12 of 32 reviews the agent misread stale or contradictory test/build signals, trusted pipeline exit codes from mvn ... | head/tail/tee, or reported fixes without matching evidence (BUILD FAILURE ignored, stale JUnit output trusted, or no diffs behind the claimed fix) [[R1], [R3], [R5], [R9], [R10], [R11], [R20], [R25], [R31]]. Tool and harness misuse appeared in 19 of 32 reviews, most often as malformed edit_file payloads (Expected JsonObject, missing allow_multiple_matches), but also as wrong run configurations, invalid session ids, and bad fetch/commit lookups that added pure execution churn before any real debugging happened [[R2], [R3], [R4], [R5], [R6], [R17], [R21], [R32]]. Process and scope control were unstable in 15 of 32 reviews: the agent skipped required STOP/confirmation gates, asked the user for context before exhausting repo evidence, widened focused tasks into unrelated package/frontend failures, created throwaway or broken test files, or even edited files outside the allowed scope such as rill-flow-task-template/pom.xml [[R4], [R8], [R12], [R19], [R21], [R24], [R29], [R30], [R31]]. Assumption-driven changes showed up in 13 of 32 reviews, including adding Swagger/Javadoc constructs without checking module dependencies or visibility, guessing API remaps or Spring Data semantics from naming intuition, inventing enum/constants, and writing architecture or business-rationale claims that were not grounded in the repository state [[R2], [R8], [R14], [R15], [R22], [R26], [R28], [R29], [R32]]. A smaller but repeated issue was test quality: 8 of 32 reviews noted missing regression coverage for the requested behavior change, wrong-layer or incomplete verification, weakened/shallow assertions, or brittle order-dependent tests (for example missing immutability/non-mutation tests and asserting exact HashMap key order) [[R1], [R5], [R17], [R23], [R24], [R27], [R28]].

Pleasantness_Judge

score mean: 0.47

Review

Pleasantness was most often reduced by process noise: 17 of 32 reviews describe rigid step banners, 'Let me…' narration, or user-facing troubleshooting chatter even after users asked for brevity or direct execution [[R1], [R5], [R6], [R10], [R18], [R24], [R30]].

Trust was the next major failure mode: 14 of 32 reviews report premature or misleading success/status claims—such as moving to final reporting with failing tests, saying 'All tests pass' after BUILD FAILURE, or claiming backward compatibility / clean compilation without evidence—and 8 of 32 also describe shifting or contradictory diagnoses during debugging [[R3], [R5], [R11], [R15], [R18], [R20], [R25]; [R3], [R10], [R19], [R23]].

11 of 32 reviews exposed harness/tool friction directly to the user, including repeated edit_file JSON parse failures, wrong run configurations, unresolved imports, compile-breakage, temporary file churn, and even leaked <agent_response> / <tool_calls> markup [[R2], [R8], [R17], [R18], [R21], [R24], [R26], [R30], [R32]].

10 of 32 reviews describe interaction-control mismatches such as asking for extra context after repo-first instructions, continuing without required stop/confirm pauses, or detouring into irrelevant investigation, and another 10 of 32 flag scope drift like touching forbidden files, adding unwanted artifacts, or inventing unsupported documentation details [[R4], [R12], [R18], [R19], [R28], [R30], [R31]; [R8], [R12], [R13], [R22], [R29], [R32]].

Despite those issues, many reviews still note that once corrected, the agent was polite, repository-oriented, and capable of ending with clear changed-file/test summaries or a successful focused rerun [[R1], [R2], [R3], [R6], [R8], [R17], [R20], [R27]].

ToolCalls_Judge

score mean: 0.53

Review

ToolCalls failures were dominated by contract mistakes: 23 of 32 reviews recorded malformed tool invocations, most often edit_file payloads shaped incorrectly (Expected JsonObject, but had JsonLiteral) or missing allow_multiple_matches, plus misuses of run_configuration, read_file, read_terminal, and file_structure [[R2], [R4], [R6], [R13], [R17], [R19], [R29], [R32]]. Verification was unreliable in 9 of 32 reviews because Maven was piped through head/tail/grep/tee or masked with && echo, so exit_code: 0 no longer proved success, and because broad package or whole-repo builds triggered unrelated frontend/environment failures such as pnpm ... SyntaxError or cd: can't cd to ..coral/src [[R1], [R3], [R5], [R9], [R11], [R18], [R19], [R20], [R31]]. Likely runner or harness issues appeared in 7 reviews: run_configuration produced ClassNotFoundException, repeated 255 failures, or stale test results until the agent rebuilt or switched to direct Maven execution [[R1], [R3], [R5], [R11], [R16], [R17], [R23]]; two more reviews added self-inflicted configuration-name errors (No configuration with name 'test' / 'surefire:test') [[R6], [R21]]. Exploration was noisy in 11 of 32 reviews, with repo-wide searches like Kafka/kafka or *Test.java, wrong-path or placeholder remote calls, and several oversized outputs (list_dir, list_commits, search_for_text) that were dumped to temp files but never read [[R1], [R7], [R10], [R14], [R15], [R16], [R19], [R29], [R31]]. Even when sessions recovered, 10 reviews showed risky editing fallbacks—whole-file write_file rewrites, temporary test-file detours, shell sed/Python patching, unsupported Swagger annotations, or an out-of-scope pom.xml edit—that introduced compile errors or extra cleanup before the final passing state [[R2], [R8], [R18], [R22], [R24], [R25], [R26], [R28], [R32]]. Despite that churn, several sessions still demonstrated solid baseline tool use by finding target files early, following temp-file read_file instructions, and recovering runner problems with focused Maven/JUnit execution that eventually passed [[R1], [R6], [R12], [R27], [R28], [R30]].

gen_tokens_to_seconds_ratio

value: 84.85

tools_call_general_stats

tools_calls_total: 1548
tools_calls_per_chat_mean: 48.375
tool_calls_in_parallel_mean: 1.385

agent_time

total: 7564
response qtl (10/50/80/100): 5.00 / 27.00 / 83.80 / 389.00
chat qtl (10/50/80/100): 78.10 / 175.50 / 343.40 / 815.00

agent_price

total: 15.30605025
response qtl (10/50/80/100): 0.01 / 0.07 / 0.18 / 0.74
chat qtl (10/50/80/100): 0.17 / 0.34 / 0.74 / 1.89

Metadata

run name: ***
scenario tag: workflows
language: java
model_info: ***
plugin_hash: ***
judge: gpt-5.4
evaluation price: ***
time of creation: ***
dataset rel path: ***
dataset_config_hash: ***
path to IDEA dumps: ***
launch args: ****

Side-by-side reviews

Absolute scores are not always enough to explain a practical difference. AgentLens can also compare two agents on the same benchmark fold, point by point and metric by metric.

The side-by-side judge receives two aligned trajectories from the same scenario, together with the prior single-run reviews for the target dimension. The single-run reviews are the main evidence; the raw trajectories are used to verify concrete details such as tool calls, errors, diffs, validation attempts, and final outputs. This setup helps reduce positional bias.

In our example, we use the same Java dataset fold for two agents with Haiku 4.5: Explyt and Claude Code. Both eval results contain the same 32 scenario-persona points, but the systems differ in harness, tool interface, execution policy, and recovery behavior.

The side-by-side report highlights which agent is preferred for each metric, where the preference is strong or weak, and which concrete trajectory patterns explain the gap. This is especially useful when aggregate scores are close or when a model looks good on a formal verification score but creates a worse developer experience.

Side-by-side report

TL;DR

The main operational split is that Agent 1 more often makes poorly grounded or brittle changes and presents weaker verification, while Agent 2 shows a smaller but recurring tendency to expand scope or claim completion too early; only the first pattern separates clearly in the data (p=0.038). End-result quality is otherwise close to even rather than decisively one-sided (p=0.512): Agent 2 more often won on compatibility-sensitive changes, documentation accuracy, migration cleanliness, and fully restoring affected tests, whereas Agent 1’s strongest wins were in preserving existing behavior, giving more reliable factual answers, keeping test refactors usable, and avoiding weakened unit-test checks. Across the reviews, Agent 2 was more consistently organized and repository-aware—better at using available repo and PR context, asking clarifying questions, structuring steps, and maintaining momentum—yet Agent 1 was somewhat more dependable on boundary protection and status honesty in several important cases, so that broader execution advantage for Agent 2 is real in the commentary but not statistically firm overall (p=0.862 and p=0.739). The heuristic verification pass, which is useful but imperfect, gives Agent 2 only a small non-significant edge (27 vs 24, p=0.536), and the main caution on Agent 2 is run reliability: it had four timeouts while Agent 1 completed all runs, while raw tool and token telemetry is not cleanly comparable because the agents used different tool stacks and Agent 2’s token logging appears incomplete.

Total number of points (chats): 32

formal_verification_result

alert: no
warning: no
p-value: 0.53564
metrics:
- agent 1: 24
- agent 2: 27

termination_reason

alert: no
warning: no
common: User decided to finish the dialog (28)
only_run1: User decided to finish the dialog (4)
only_run2: Scenario timed out after 1200s (3), Waiting for the agent response timed out after 900s (1)

EndResult_Judge

alert: no
warning: YES

Review:

A2’s clearest end-result advantages are in high-severity correctness and completeness cases: [C4] and [C25] on connector API path/backward-compatibility, [C5] and [C15] on documentation/comment accuracy, [C20] on migration scope cleanliness, and [C21] on fully restoring the affected tests.

A1’s strongest counterexamples are also high-severity but concentrated elsewhere: [C17] on preserving pagination semantics, [C24] on factual reliability of the Kafka search answer, [C29] on keeping the test-data refactor usable and verified, and [C31]-[C32] on not weakening sorting or failure-path checks in unit tests.

Scope and verification were mixed rather than one-sided: A2 was favored for cleaner scoping in [C9], [C14], and [C20] and for stronger validation/reporting in [C2], [C10], [C21], and [C22], while A1 was favored for cleaner scoping in [C10], [C15], [C19], [C21], and [C32] and for stronger test verification in [C3], [C12], [C13], [C26], [C29], and [C30].

Low-severity ties or near-ties remained in areas such as logging style and sensitive-data restraint ([C1]-[C2]), shared successful migration/refactor outcomes ([C19], [C27], [C28]), and the shared missing 501 fallback noted in [C26].

judge alert: YES

Scoring

p-value: 0.512
metrics_value (from [-1..1]): 0.0563

InstructionCompliance_Judge

alert: no
warning: YES

Review:

A2 had the clearest advantage on repository-first intake, MCP usage, and clarification style, winning 12 proof entries versus 3 for A1 in that group, with high-severity examples in [C16] and [C31]; A1’s main counter-cases there were the Step 1 choice-format issues in [C9] and [C10] and the self-directed search case in [C30].

A2 also led on scope limits and follow-up-instruction fidelity, 16 proof entries to 10, with a high-severity win in [C17] and several medium cases across [C5], [C7]-[C10], and [C18], although A1 had the strongest opposite hard-constraint wins in [C4] and [C18] where A2 expanded scope or touched tests / kept banned native SQL.

Step formatting favored A2 overall (18 entries vs 13, with high-severity cases in [C3] and [C25]), but workflow sequencing and stop-gate compliance were nearly even at 14 for A2 versus 13 for A1: A2’s strongest wins were [C9], [C20], [C25], and [C32], while A1’s strongest were [C22] and [C29].

A1’s narrower advantages were in deliverables/report completeness (5 entries vs 4 for A2, notably [C12], [C27], and [C28]) and some verification/debugging cases, including the high-severity [C31] failure-handling issue against A2, though A2 still led the tooling/debug group overall 7 to 4 through cases such as [C21], [C22], [C27], and [C30].

judge alert: YES

Scoring

p-value: 0.862
metrics_value (from [-1..1]): 0.0187

Pitfalls_Judge

alert: YES
warning: YES

Review:

Evidence-grounding/documentation strongly favors A2: A2 has favorable proofs in 12 Ck versus 3 for A1, with the sharpest A1 problems being unverified architecture or requirement claims in [C9] and assumption-driven localization in [C14].

Tool/path/edit stability also leans A2, with favorable proofs in 23 Ck versus 19 for A1; the biggest A1 failures there are compile-breaking patch/edit behavior in [C2] and brittle mass rewrites in [C30], while A1’s wins are mostly narrower path or working-directory recoveries such as [C1], [C4], [C28], and [C32].

Verification/reporting similarly leans A2 (12 Ck versus 9 for A1), driven by stale-runner or masked-build cases in [C11], [C17], [C19], [C20], [C21], and [C26]; A1’s strongest counterexamples are the more trustworthy result summaries in [C14] and not closing with failing tests in [C22].

Workflow/scope control is mixed: A2 has more gate-discipline wins (12 Ck versus 7 for A1), but A1 owns several of the strongest boundary-protection cases, including staying within requested scope in [C4], avoiding unrelated-file mutation in [C10], and respecting no-test-edit/test-only limits in [C18] and [C32].

Pure code/test correctness is close to even (A1 wins in 11 Ck, A2 in 10, with 1 tie), with A1’s standout case preserving mail exception semantics in [C1] and A2’s strongest cases being validation-aware fixes in [C21] and avoiding a regressive recovery path in [C13].

judge alert: YES

Scoring

p-value: 0.038
metrics_value (from [-1..1]): 0.1812

Pleasantness_Judge

alert: no
warning: YES

Review:

A2 had the clearest advantage on communication and organization, with favorable evidence in 22 Ck versus 3 for A1; the strongest references are [C3], [C11], [C23], [C25], and [C29], while A1 only led on lighter presentation points such as [C16], [C26], and [C27].

A2 also led on user alignment/scope (13 Ck vs 8) and on productive completion (8 vs 4), with notable support from [C14] and [C26] for following the user’s requested path and from [C2], [C13], [C20], [C21], [C22], and [C25] for keeping progress moving.

A1 was stronger on trust and status accuracy (11 Ck vs 9), driven by several high-severity cases where A2 reported success or completeness too early or inaccurately in [C6], [C21], [C22], [C27], and [C31], although A2 also had high-severity wins on that same dimension in [C2], [C17], [C19], [C23], [C25], and [C26].

Workflow smoothness and tool friction were mixed but leaned A1 (15 Ck vs 10): A2 had better evidence in [C3]-[C5] and [C15], but A1 more often came out ahead in later runs with path/run churn or ceremony problems, such as [C10], [C16], [C21], [C24], [C28], and [C32].

judge alert: YES

Scoring

p-value: 0.739
metrics_value (from [-1..1]): 0.0312

ToolCalls_Judge

alert: no
warning: no
judge alert: no

Scoring

p-value: 0.816
metrics_value (from [-1..1]): -0.025

setups

alert: no
warning: no
difference: ``

Inference metrics

Metric	p-value	run 1	run 2	alert	warning
price	`0.2981`	`15.30605025`	`13.386273360811174`	no	YES
time	`0.6241`	`7564`	`8336`	no	no
gen tokens to seconds ratio	`0.0`	`84.85`	NONE	no	YES
tools calls total	`0.1545`	`1548`	`1318`	no	no
generation tokens	`0.0`	`641807`	NONE	no	YES
cache hit mean ratio	`0.0`	`0.8627`	NONE	no	YES
tool calls in parallel mean	--	`1.385`	NONE	no	non

Tool call success rates

Tool	run 1 success	run 2 success	run 1 total	run 2 total	alert	warning
read_file	`0.9962049335863378`	`0`	`527`	`0`	no	no
run_command	`1.0`	`0`	`181`	`0`	no	no
search_for_text	`1.0`	`0`	`181`	`0`	no	no
edit_file	`0.6645962732919255`	`0`	`161`	`0`	no	no
search_file_by_name	`0.9859154929577465`	`0`	`142`	`0`	no	no
run_configuration	`0.9310344827586207`	`0`	`116`	`0`	no	no
write_file	`1.0`	`0`	`75`	`0`	no	no
get_configurations	`1.0`	`0`	`56`	`0`	no	no
list_dir	`1.0`	`0`	`44`	`0`	no	no
file_structure	`0.96`	`0`	`25`	`0`	no	no
web_fetch	`1.0`	`0`	`11`	`0`	no	no
read_terminal	`0.5555555555555556`	`0`	`9`	`0`	no	no
pull_request_read	`1.0`	`0`	`8`	`0`	no	no
issue_read	`1.0`	`0`	`5`	`0`	no	no
get_commit	`1.0`	`0`	`3`	`0`	no	no
list_commits	`1.0`	`0`	`2`	`0`	no	no
find_usages	`1.0`	`0`	`1`	`0`	no	no
search_code	`1.0`	`0`	`1`	`0`	no	no
AskUserQuestion	`0`	`0.0`	`0`	`5`	no	no
ExitPlanMode	`0`	`0.0`	`0`	`3`	no	no
Bash	`0`	`0.9682539682539683`	`0`	`441`	no	no
Read	`0`	`0.835820895522388`	`0`	`402`	no	no
Edit	`0`	`0.9304635761589404`	`0`	`302`	no	no
Glob	`0`	`0.9857142857142858`	`0`	`70`	no	no
Grep	`0`	`0.9433962264150944`	`0`	`53`	no	no
Write	`0`	`1.0`	`0`	`15`	no	no
mcp__github__issue_read	`0`	`1.0`	`0`	`9`	no	no
mcp__github__pull_request_read	`0`	`1.0`	`0`	`8`	no	no
WebFetch	`0`	`1.0`	`0`	`5`	no	no
Agent	`0`	`1.0`	`0`	`2`	no	no
EnterPlanMode	`0`	`1.0`	`0`	`2`	no	no
TaskOutput	`0`	`1.0`	`0`	`1`	no	no

Metadata

run1

model_info: Anthropic:claude-haiku-4-5
plugin_hash: ***
run_name: Explyt-Haiku45

run2

model_info: Anthropic:claude-haiku-4-5
plugin_hash: ***
run_name: ClaudeCode_Haiku45

judge

api_calls: ***
cached_tokens: ***
completion_tokens: ***
model: gpt-5.4
prompt_tokens: ***
total_price_usd: ***

This makes AgentLens useful not only for model ranking, but also for the agent development cycle: it can compare a new harness against an old one, detect regressions in the agent loop or tool layer, and explain why a change feels better or worse to developers even when final verification rates move only slightly.

Quality Index

The report keeps every metric separately, but it also exposes one aggregate number for quick comparison: the Quality Index (QI). The Quality Index is the unweighted mean of the judge metric values:

$QI(a) = \frac{1}{|M|}\sum_{m \in M} s_m(a)$

where $a$ is an agent, $M$ is the metric set, and $s_m(a)$ is the aggregate score for metric $m$ .

For the running Explyt Haiku 4.5 example, the resulting Quality Index is $58.17$ .

We use the Quality Index as a deliberately simple proxy for production quality. The metrics cover the main dimensions of user-visible coding-agent behavior: final usefulness, instruction following, tool use, recovery from failures, and interaction quality.

Reliability and bias checks

Because AgentLens relies on compact task folds and LLM-based trajectory reviews, we need to check that the resulting scores are stable and not dominated by judge artifacts.

Score variance

We repeated the same GLM-5.1 self-hosted configuration five times. The Quality Index mean was $67.28$ , with standard deviation $0.94$ on a $0$ – $100$ scale.

Most variance came from formal verification: $60.5\%$ of index variance, followed by Pitfalls ( $18.1\%$ ) and EndResult ( $16.2\%$ ). At the scenario-persona level, 15 of 32 points passed formal verification in all five runs, one failed in all five, and 16 were flaky.

So the noise is not evenly spread; it concentrates in borderline tasks, which makes repeated runs useful for identifying nondeterministic agent behavior.

Takeaway: Quality Index is stable across reruns; the residual noise is driven by formal verification on a minority of borderline tasks, so repeated runs mainly surface nondeterministic agent behavior rather than judge instability.

Side-by-side positional bias

To check whether side-by-side reviews are sensitive to the order in which the agents are presented, we ran an order-swap test.

For a pair of agents A and B, we compare two prompts: A as Agent 1 with B as Agent 2 and vice versa. Scores are oriented as the advantage of Agent 2 over Agent 1, so perfect antisymmetry would give $s(A,B) \approx -s(B,A)$ .

For each metric we report the stable model difference, $\text{effect} = (s(A,B) - s(B,A)) / 2$ , and the order residual, $\text{residual} = (s(A,B) + s(B,A)) / 2$ .

Pair	Metric	$s(A,B)$	$s(B,A)$	Effect	Residual
GLM-5.1 vs GLM-5.1	End Result	$0.19$	$-0.21$	$0.20$	$-0.01$
GLM-5.1 vs GLM-5.1	Instr. Compl.	$-0.02$	$-0.06$	$0.02$	$-0.04$
GLM-5.1 vs GLM-5.1	Pitfalls	$0.00$	$0.00$	$0.00$	$0.00$
GLM-5.1 vs GLM-5.1	Pleasantness	$0.01$	$-0.03$	$0.02$	$-0.01$
GLM-5.1 vs GLM-5.1	Tool Calls	$-0.04$	$0.02$	$-0.03$	$-0.01$
Opus 4.7 vs Haiku 4.5	End Result	$-0.39$	$0.35$	$-0.37$	$-0.02$
Opus 4.7 vs Haiku 4.5	Instr. Compl.	$-0.42$	$0.39$	$-0.40$	$-0.02$
Opus 4.7 vs Haiku 4.5	Pitfalls	$-0.43$	$0.43$	$-0.43$	$0.00$
Opus 4.7 vs Haiku 4.5	Pleasantness	$-0.50$	$0.52$	$-0.51$	$0.01$
Opus 4.7 vs Haiku 4.5	Tool Calls	$-0.48$	$0.41$	$-0.44$	$-0.03$

Order-Swap Sanity Check for Side-by-Side Reviews. Effect is the antisymmetric model-difference component; residual is the remaining order-sensitive component.

Table reports two such checks. The GLM-5.1 pair is a close same-model comparison; the Opus–Haiku pair is a high-contrast one (Haiku was run in no-thinking mode to widen the quality gap), and all five metrics preserve their direction after swapping sides.

In both settings the residuals are small in absolute value. This does not prove that positional bias is absent, but it does suggest that order effects are secondary in these aggregate checks.

Takeaway: Swapping agent order leaves side-by-side scores nearly antisymmetric — every metric keeps its direction and the order residuals stay small in both the same-model and high-contrast pairs — so positional bias is a secondary effect.

Judge self-preference

For self-preference, we re-scored the same GPT-5.5 vs. Sonnet 4.6 trajectories with each model as judge. The judges picked different winners in $23\%$ of task-metric comparisons. In $18\%$ , each judge favored its own model family; in $5\%$ , each favored the other. Half of the self-favoring excess came from Pleasantness alone.

Judge choice changed preference strength and borderline calls, but not the average winner.

Takeaway: Swapping the judge model moves only preference strength and borderline calls (judges disagreed on $23\%$ of task-metric comparisons, with the self-favoring excess concentrated in Pleasantness) - the average winner stays the same.

Validity of LLM reviews

A natural concern is whether LLM-judges can be trusted. Our experience suggests that human annotation is not an obvious gold standard for long coding trajectories: each item requires reading a long interaction trace and applying a multipage rubric. In our internal pairwise annotation experiments on long coding trajectories, LLM–human agreement was consistently higher than human–human agreement. This is consistent with prior LLM-as-judge work showing that strong LLM-judges can reach human-level agreement with human preferences, comparable to agreement between human annotators (see MT-Bench and Chatbot Arena). That does not remove judge-bias concerns, but it makes LLM-judges a practical primary review mechanism for this workload.

Correlation with public benchmarks

To place QI against the wider benchmark landscape, we computed Spearman rank correlations between our QI and every Artificial Analysis (AA) evaluation that reports scores on the current generation of models. Eleven evaluations qualified, including the agentic benchmarks APEX-Agents-AA, GDPval-AA, Terminal-Bench Hard, and $\tau^2$ -Bench Telecom; ten cover all eleven Explyt agent models, and APEX-Agents-AA covers seven. AA also aggregates these into a single Intelligence Index (hereafter the AA composite), a weighted average designed to track overall model capability. All ranks below are within these eleven models.

Quality Index Rank vs. Artificial Analysis Index Rank (more is better).

The QI column shows that Mimo V2.5 Pro and Gemini 3.1 Pro Preview sit much higher on the AA composite than on our QI (rank gaps of $+6$ and $+6$ ; $90\%$ bootstrap CIs $[+4,+8]$ and $[+2,+6]$ ), consistent with optimization pressure toward AA-style academic and reasoning evaluations. Claude Opus 4.7, Claude Sonnet 4.6, and DeepSeek V4 Flash show the opposite, with rank gaps from $-2$ to $-6$ and CIs strictly negative: they handle long-horizon IDE work substantially better than the AA composite credits them for, with the gap concentrated on IFBench and $\tau^2$ -Bench Telecom, where the Anthropic models in particular score well below their overall agentic quality. GPT-5.5 and DeepSeek V4 Pro sit at zero with CIs spanning the origin, the only two models for which the two rankings agree.

Spearman rank correlation between our Quality Index (QI) and per-benchmark scores from Artificial Analysis, computed over the models selected from the Explyt agent leaderboard. Public benchmarks are sorted by correlation with our QI (highest to lowest).

Two things follow. First, QI tracks an axis of model quality that does not collapse to any single AA evaluation: the strongest individual correlate (APEX-Agents-AA, $\rho=0.82$ ) is itself an agentic benchmark, while reasoning- and instruction-following-oriented evaluations diverge sharply, including the negative correlations of IFBench ( $\rho=-0.41$ ) and $\tau^2$ -Bench Telecom ( $\rho=-0.25$ ). Second, the rank-gap residuals look systematic rather than noisy: they split the cohort along a recognizable line between models trained for evaluation-style competence and models built for sustained tool use under realistic constraints.

Takeaway: Quality Index captures a distinct, agentic axis of model quality that no single public benchmark reproduces - it correlates most strongly with other agentic benchmarks (APEX-Agents-AA, $\rho=0.82$ ) and even negatively with instruction-following and reasoning evaluations — so its rank gaps systematically separate evaluation-optimized models from those built for sustained real-world tool use.

Evaluation

Leaderboard

We evaluate AgentLens on Java coding-assistant tasks using Explyt and Claude Code.

The table reports full-system performance: scores reflect the model together with its provider, agent loop, tool interface, and default execution policy. All values are percentages, and rows are sorted by Quality Index. All run reviews use GPT-5.4 as the judge. We use high reasoning effort where supported, except where noted in the table. OpenAI and Anthropic models use official providers, GLM-5.1 (self-hosted) is hosted locally, and the remaining models use OpenRouter.

Harness	Model	QI	Formal	End	Instr.	Pitfalls	Pleasant.	Tools
Explyt	Opus 4.7	81.5	81.2	94.0	75.0	64.0	92.0	83.0
Claude Code	Opus 4.7 (xhigh)	76.2	81.2	91.0	69.0	58.0	78.0	80.0
Explyt	GPT-5.5	73.0	75.0	78.0	70.0	55.0	80.0	80.0
Explyt	Sonnet 4.6	70.2	81.2	66.0	62.0	56.0	73.0	83.0
Claude Code	Sonnet 4.6	70.1	90.6	80.0	62.0	55.0	75.0	58.0
Explyt	GLM-5.1 (self-hosted)	67.3	72.5	71.8	57.4	53.2	75.8	73.0
Explyt	DeepSeek V4 Flash	64.8	87.5	73.0	50.0	50.0	66.0	62.0
Explyt	DeepSeek V4 Pro	64.1	87.5	66.0	53.0	48.0	69.0	61.0
Claude Code	Haiku 4.5	63.6	84.4	78.0	58.0	48.0	58.0	55.0
Explyt	GLM-5.1	60.6	80.6	60.0	53.0	48.0	61.0	61.0
Explyt	Haiku 4.5	58.2	75.0	72.0	55.0	47.0	47.0	53.0
Explyt	Gemini 3.1 Pro Preview	54.5	71.9	62.0	52.0	41.0	50.0	50.0
Explyt	Gemini 3 Flash	52.7	75.0	66.0	48.0	41.0	39.0	47.0
Explyt	MiniMax 2.7	50.8	70.0	52.0	47.0	38.0	43.0	55.0
Explyt	Qwen 3.6 Plus	49.1	65.6	58.0	45.0	39.0	42.0	45.0
Explyt	Mimo V2.5 Pro	45.1	62.5	50.0	38.0	32.0	36.0	52.0
Explyt	Kimi K2.6*	28.1	42.9	29.0	33.0	19.0	21.0	24.0

Leaderboard on the 32-trajectory Java benchmark. Formal is the formal-verification pass rate; End, Instr., Pitfalls, Pleasant., and Tools are LLM-judge metric means; QI is the Quality Index. $^{*}$ Kimi-K2.6 had provider-side instability; results are included for completeness.

Model behavior case study

We examine the most interesting anomalies and close comparisons in the leaderboard, using single-run and side-by-side reviews to better understand where models differ in reliability, tool use, instruction following, and LLM API performance.

Gemini 3.1 Pro Preview

Gemini 3.1 Pro Preview underperforms as an interactive coding agent relative to its frontier-model positioning. The reviews suggest that the problem is agent-loop reliability rather than isolated code generation. They repeatedly cite skipped workflow steps, premature edits or tests before approval, false "all checks passed" claims, and destructive-edit loops. About $70\%$ of tasks passed formal verification, and 4 of 32 conversations timed out.

Kimi K2.6

The low score is confounded by an OpenRouter-side tool-parser issue. The ToolCalls review reports that "20 of 21 reviews reported at least one schema/argument parse error, and 17 of 21 described the same persistent wrapper mistake ({"": {...}} or deeper nesting)".

DeepSeek V4 Pro vs. Flash

The QIs are nearly tied, $64.1$ vs. $64.8$ . Pro is more controlled on step gating, pauses, formatting, and file boundaries. Flash relaxes those constraints, completes more work, and runs more than twice as fast under the fixed budget.

Explyt Haiku vs. Claude Code Haiku

Claude Code Haiku likely benefits from familiarity with its own harness. The gap appears in edit_file tool success rate: $0.66$ for Explyt vs. $0.93$ for Claude Code. The Pitfalls judge also favored Claude Code by a statistically significant margin (metric_value = 0.1812, p = 0.038).

Explyt Opus vs. Claude Code Opus

Formal verification ties at $26$ vs. $26$ , but reviews favor Explyt for finishing and verifying changes cleanly. Claude Code’s main weakness is operational reliability: judges point to path and tool errors, missed user constraints, and one timeout. Interestingly, the file-edit tool success rate reverses the Haiku pattern: Claude Code Opus is lower than Explyt, $0.92$ vs. $1.00$ .

GPT-5.5 vs. Opus 4.7

The GPT-5.5 vs. Opus 4.7 comparison shows a useful split between procedural compliance and semantic reliability. Opus wins more clearly on user-visible quality: it better preserves the actual requirement behind the protocol. GPT-5.5 often more closely follows the explicit workflow - required message format, step boundaries, and waiting for user confirmation before continuing. But the reviews point to Opus having higher "requirement coverage and final fitness" and better validation. GPT-5.5 was more often penalized for "verification without covering changed branches", "unsupported root-cause claims", "missed affected tests", or marking the task done "despite not reproducing the failure". In short, GPT-5.5 is better at following the task form; Opus is better at preserving the task intent.

Production use

AgentLens is wired into our agent-development CI pipeline. A candidate version runs on the benchmark and produces trajectories, formal-verification results, metric reviews, aggregate scores, and side-by-side comparisons against a previous version.

The pipeline has three main stages:

Benchmark run: selected tasks run in parallel across language- and IDE-specific environments.
Evaluation: user-agent trajectories are evaluated, and a report with metrics and judge reviews is prepared.
Regression detection: the current run is compared against an anchor run, and statistically significant changes notify maintainers.

*GitHub CI workflow with benchmark execution, metric evaluation, experiment tracking, and regression detection through side-by-side comparison between current and anchor runs.*

This has already caught harness-level bugs. During a weekly regression run, AgentLens surfaced a ConcurrentModificationException from parallel tool calling. The Pitfalls judge localized it directly in the trajectory: "Tool and harness stability was mixed, but the most debug-relevant failures include Agent 2's explicit ConcurrentModificationException crash in [C5]…".

Takeaway: In day-to-day development AgenLens value is catching regressions between runs - statistically significant drops in metrics flag when a new candidate version has degraded relative to an anchor. And the judge reviews make this actionable: unlike a bare score, they explain what went wrong, turning an opaque number into a concrete, debuggable cause.

Extensibility

AgentLens is not tied to one agent or one IDE. In this work, we evaluate both Explyt, an IDE-based agent for IntelliJ IDEA, and Claude Code, a CLI-based coding agent.

New integrations are pluggable. IntelliJ IDEA plugins can implement the AgentEngine interface directly, while CLI agents can connect through a lightweight Python adapter.

The released fold focuses on Java, but the same setup already runs across other JetBrains IDEs, including PyCharm, Rider, and WebStorm.

Limitations and future work

AgentLens evaluates a specific class of coding-agent tasks, not general model intelligence.

Some models are accessed through third-party providers such as OpenRouter, so latency, routing, model versioning, and availability are not fully under our control. The DeepSeek Pro-vs-Flash case is a specific illustration for this effect: a $2\times$ throughput difference under a fixed task budget significantly affects how much work each agent can finish, which means some leaderboard differences reflect serving conditions as much as model quality.

There is also harness bias: AgentLens was initially built around our own coding agent, so some integration assumptions may be more natural for Explyt. We mitigate this by open-sourcing the benchmark, but broader external use is needed.

Next, we plan to add more agents, including Codex and OpenCode; open-source more IDE/language variants; and add focused folds such as test-writing tasks with specialized verifiers and judge metrics.

Try AgentLens

AgentLens is open source. Clone it on GitHub to evaluate your own agent, read the preprint for the full methodology, and check the live leaderboard. Contributions and feedback are welcome — open an issue or PR with new scenarios, verifiers, or agents.

LATEST NEWS

Explyt 5.14: Context That Carries Over, Reviews That Stay Organized

RELEASE

01.07.2026

Explyt 5.13: Better Test Coverage, Project Context, and AI Coding Flow

RELEASE

17.06.2026

Explyt 5.12: General agent, faster code context, broader Test mode, and cleaner AI code review

RELEASE

15.06.2026