Tokens Show Activity, Not Impact: 9 Metrics for AI Coding Agents

Businesses need numbers. They use them to make decisions about bonuses, team composition, and tool purchases. Developers quickly spot the problem: a metric can rise while the amount of useful work stays the same.

Token consumption went up, so the agent must be getting more use. The number of requests increased, so the team must be active. More lines reached the commit, so AI must have helped more. Each connection looks convincing in a report until someone asks which task was completed, how long verification took, and what happened to code quality.

During Open Mic discussions, developers described how companies measure work with AI. Their approaches ranged from tokens and request counts to the volume of accepted code. These figures show activity, but they do not answer the main question: did the engineer work faster, and did the team get a better result?

Metrics influence behavior. Put token consumption into a KPI and employees will find ways to consume more. Reward generated lines and people will favor tasks that produce large diffs. Look only at commits and part of the work may not appear until after the reporting period closes.

A useful methodology connects agent activity to an accepted engineering result. The nine metrics below answer separate questions about adoption, usage, time, quality, cost, and developer sentiment.

Why tokens do not measure effectiveness

Token consumption depends on context size, model choice, instruction length, tool calls, and retries. One long request may replace several short ones. A large context may be necessary to investigate a bug but unnecessary for generating a test.

Tokens measure model load and provider cost. They do not measure the value of a change. A developer may spend many tokens on failed attempts and then solve the task manually. Another engineer may send a short request, verify the result, and close a defect within minutes.

The problem grows when a counter becomes a target. A team may cut time spent on careful verification or create unnecessary requests to improve a chart. The report gains activity while the product gains nothing.

The same limitation applies to request counts and lines of code. A request records an interaction with the agent. A line records text in a diff. Neither tells you whether checks passed, the change survived review, or a regression appeared.

Keep tokens in the usage and budget section. To assess engineering value, connect them to the task, time, and quality of the accepted result.

Define the unit of accepted work

Before measurement begins, the team must define its unit of output. It may be a task or change that reaches a previously agreed state:

• the code enters a commit and passes required checks;
• the pull request passes review and is accepted;
• a test confirms the fix;
• the change reaches a test environment.

The criterion depends on the process, but it must remain consistent for tasks in the same category. Test generation, bug fixing, refactoring, and error investigation should not be collapsed into one line of a report.

Calculate the full cost of an accepted change as follows:

Full cost of an accepted change =
  cost of all model calls, including rejected attempts
  + active developer time × fully loaded hourly cost

If the company does not convert working time into money, keep minutes and model cost separate. Hours and currency cannot be added into one figure without an agreed conversion.

Nine metrics for evaluating an AI coding agent

1. Active participant rate

What it measures: whether the team has started using the agent.

Active participant rate =
  participants with at least one recorded interaction during the period
  / all participants

Show absolute numbers alongside the percentage and week-over-week movement. Low activity may result from access problems, missing training, vacations, or a lack of suitable tasks. This metric shows reach, not quality of use.

2. Acceptance rate and accepted lines

What it measures: how much generated code reaches a commit.

Acceptance rate =
  committed AI-generated lines
  / all lines generated by the agent

The metric shows how much agent output enters the project and how much code developers are willing to accept. It is not a code quality score.

Acceptance is recorded at commit time. If a developer is still verifying a change, the daily figure may be understated. Report commit lag alongside acceptance rate or data near the reporting boundary will be incomplete.

Lines are not equal to value. A small fix may remove a serious defect, while a large template-heavy diff may require extensive rework. Compare this metric only across similar tasks and equal observation windows.

3. Results accepted without substantial rework

What it measures: how often developers keep the main logic produced by the agent.

Rate accepted without substantial rework =
  tasks accepted without changes to the main logic
  / all tasks assigned to the agent

Define the boundary for substantial rework in advance. Rewriting the main logic or changing the architectural approach counts as substantial. This criterion is closer to accepted-result quality than a line count.

4. Active developer time

What it measures: how much time an engineer spends on the task.

Include task definition, guiding the agent, reading the diff, running checks, and manual corrections. Track model execution time separately because the developer may work on another task while the agent runs.

Compare median active time within the same category. Without a baseline or control sample, you cannot claim that the agent accelerated development. The comparison must account for task complexity and the completion criterion.

5. Human verification time

What it measures: how much effort is required to verify agent output.

This includes reading changes, running tests, checking behavior, and discussing the change during review.

Verification share of active time =
  human verification time
  / total active developer time for the task

Fast generation produces no time saving if the result takes a long time to verify. A higher verification share may indicate a difficult scenario, weak automated checks, or low confidence in the output. Investigate the cause through task categories and participant comments.

6. Retries

What it measures: how many cycles are required before the result can be accepted.

Count cases where the developer clarifies the request or restarts the agent's work. Record a reason for each retry: missing context, misunderstood requirements, a failed test, a tool error, or an architecturally unsuitable solution.

Retry count connects token consumption to a specific problem. Without reason classification, high consumption remains a signal that is difficult to act on.

7. Defects after acceptance

What it measures: what happens after a change is accepted.

For measured tasks, track review returns, reopened work, regressions, and incidents. Compare defects per accepted task within the same category and observation period.

Do not attribute every defect to the agent. Link it to a specific change and use an agreed cause classification. Otherwise, the metric becomes a way to assign blame rather than analyze process quality.

8. Cost per accepted task

What it measures: the cost of model calls for each completed result.

Model cost per accepted task =
  cost of all calls in the sample, including rejected attempts
  / accepted tasks in the same sample

Compare one agent configuration and one task category. Differences between teams may come from model choice, context size, retries, and the type of work. Report cost next to time and quality rather than using it alone.

9. Developer rating and intent to reuse

What it measures: confidence in the result and willingness to use the workflow again.

After a task, ask how confident the engineer is in the result, whether constant supervision was required, and whether they would use the scenario again. At the end of the period, measure satisfaction, willingness to recommend, and intent to continue separately.

Show the number of respondents and response rate. An average recommendation score is not NPS. NPS requires the shares of promoters and detractors among respondents.

Do not combine subjective responses with measured time into one score. They reveal barriers absent from telemetry, including low confidence, difficult verification, and an awkward workflow.

How to compare results

Begin with a few categories that can produce enough observations. Do not combine local fixes, test generation, refactoring, and exploratory tasks.

Before measurement starts, define:

1. The completion criterion for each category.
2. Required checks.
3. The boundary for substantial rework.
4. The rule for measuring active time.
5. The window in which acceptance rate catches up with commits.
6. The defect observation period.
7. The threshold for ending an agent cycle.

Use medians and report sample size. If historical data exists, classify those tasks using the same rules. Historical comparisons are weaker than parallel ones because people, requirements, the codebase, and CI speed change.

Treat the figures from the most active users as a hypothesis. If one engineer has a higher acceptance rate, test that person's workflows on similar tasks with other participants. This helps separate agent practices from developer experience, task choice, and project characteristics.

Do not set individual quotas for requests, tokens, or accepted lines. People may choose safer tasks, increase interactions, or delay commits to improve their reports. Measurement should help the team identify useful workflows and process bottlenecks.

A task record for measurement

## Task

- Anonymized identifier:
- Category:
- Mode: without agent / with agent
- Model and configuration:
- Completion criterion:
- Required checks:

## Time

- Work started:
- Work completed:
- Active developer time:
- Model execution time:
- Verification and review:
- Manual corrections:

## Model usage

- Request count:
- Input tokens:
- Output tokens:
- Monetary cost:
- Retry count:
- Reasons for retries:

## Acceptance

- Commit date and time:
- Generated lines:
- Lines accepted at commit:
- Result not committed yet: yes / no
- Degree of rework:

## Quality

- Checks passed:
- Review returns:
- Task reopened:
- Defects after acceptance:
- Observation period:

## Developer assessment

- Constant supervision required: yes / no
- Confidence in the result after verification: 0-10
- Would use this scenario again: yes / no
- Comment:

The record must not contain the participant's name, an internal project name, private code fragments, or other sensitive information. An anonymized identifier and task category are sufficient for analysis.

What belongs on the dashboard

Divide the metrics into four groups:

• Adoption: active participants and weekly movement.
• Usage: requests, tokens, acceptance rate, and accepted lines.
• Outcome: active time, rework, checks, and defects by task category.
• Sentiment: developer ratings with respondent counts.

Tokens belong in usage and budget. Acceptance rate shows how much generated code reached a commit after accounting for reporting delay. Both figures are useful, but neither tells you whether the team became faster.

The answer appears only after connecting activity to tasks: how many comparable changes were accepted, how much active time the work and verification required, whether required checks passed, whether defects appeared, and how much each accepted task cost.

Conclusion

A metric is useful when it informs a decision without encouraging people to optimize the counter. Tokens, requests, and lines show activity. Time, rework, checks, defects, cost per accepted task, and developer assessment show what happened to the engineering result.

Do not collapse the nine metrics into one ranking. Together, they give the team a set of questions for evaluating agent adoption, identifying weak workflows, and comparing similar tasks. Only after this analysis can the company discuss effectiveness rather than the size of its token counter.