The hidden team cost of vibe coding and a merge gate that tests understanding.
A familiar pattern is appearing in AI-assisted teams.
One developer uses an agent as a force multiplier. They know the system, check the assumptions, and use the model to remove mechanical work.
Another developer uses the same agent as an outsourcing service. They paste in a task, accept a large diff, point to green tests, and move on. When the implementation reaches review, the strongest engineers have to reconstruct the architecture, find the missing edge cases, and explain what the author cannot.
The first developer gets leverage. The second creates work for everyone else.
Team leads often describe the result in less diplomatic terms: the people who understand the code are now doing even more work, while the people who do not understand it can produce more code than before.
There is a new name for part of the problem: comprehension debt. It is the gap between the code a team owns and the code its engineers can explain, modify, and defend without asking the agent to think for them.
The uncomfortable question is whether generative AI can widen that gap—not only by producing bad code, but by allowing low-effort work to look productive for long enough that the cost lands on someone else.
The evidence: coding agents can produce output without understanding
The strongest evidence for this argument comes from studies that actually put programmers in front of code.
1. AI-assisted developers completed the task, but understood less
In “How AI Assistance Impacts the Formation of Coding Skills,” J. H. Shen and Alexander Tamkin recruited 52 mostly junior software engineers who had used Python at least weekly for more than a year and were unfamiliar with Trio, an asynchronous Python library. Participants were randomly assigned to complete coding tasks with a chat-based AI assistant or with web search and documentation alone.
This was a small experiment on learning an unfamiliar library. The assessment took place shortly after the task; it does not establish long-term skill loss or quantify review work in production teams.
The AI group completed the coding task about two minutes faster on average, but that difference was not statistically significant. On the subsequent 27-point knowledge quiz, the AI group averaged 50%, compared with 67% for the hand-coding group: a 17% lower score, with Cohen’s d = 0.738, p = 0.01. The largest gap appeared in debugging questions—the exact capability needed to supervise code that an agent has written.
The researchers also found a behavioral split inside the AI group:
- low-scoring participants tended to delegate code generation or debugging to the assistant and averaged below 40% on the quiz;
- higher-scoring participants used AI for conceptual questions, explanations, or follow-up checks, with reported interaction patterns averaging roughly 65–86% on the quiz.
This is not a claim that every AI user learns less. It is evidence that how a developer uses the assistant can separate output from skill formation.
2. Coding agents can make the initial result better while comprehension gets worse
A second study, “(Im)Paired Programming: Coding Agents Improve Productivity but Harm Understanding,” tested 54 computer-science students building a web-based tic-tac-toe variant. Participants used either an agent that directly edited their code or a chatbot that provided higher-level help while leaving more implementation work to the participant.
The agent group completed the initial task more accurately, but scored substantially lower on comprehension questions about its own code. In a follow-up extension task performed without an agent, the initial accuracy advantage did not transfer into a clear advantage. The study also links lower comprehension to low-effort interaction patterns such as copying task criteria into prompts and auto-accepting edits.
The researchers report an important asymmetry: novices using an agent could produce code as accurately as more experienced participants, while showing worse comprehension. Traditional programming background still predicted understanding, even when the initial output looked similar.
This is close to the team-lead problem. An agent can compress the visible difference between a strong and weak programmer at the moment of delivery, while preserving the difference in who can explain, debug, and extend the result.
3. A realistic codebase reveals different reliance patterns
The 2026 AIRELI study, Grasping AI Reliance in Program Comprehension and Coding, used a controlled study with 21 participants working on two realistic web-application changes. Its qualitative analysis identified three reliance patterns:
- self-sufficient developers, who owned understanding, planning, and execution;
- understanding-gated developers, who formed the model and plan themselves but used AI for execution;
- AI-steered developers, who outsourced system understanding, planning, and implementation.
The AI-steered participants could complete the first task quickly, but showed shallow comprehension and struggled when the next task had to be done without the assistant. The authors explicitly describe this as an exploratory, small-sample study—not proof of a long-term productivity decline. Its value is diagnostic: aggregate task-completion numbers can hide a dangerous difference in how people reached the result.
What each source establishes
| Claim | Evidence | What it does not establish |
|---|---|---|
| AI assistance can reduce immediate mastery of a new library | Shen–Tamkin randomized experiment: 50% versus 67% quiz score | Long-term skill decline or production-team performance |
| Heavy delegation is associated with weaker comprehension | Shen–Tamkin qualitative interaction analysis; low-scoring patterns averaged below 40% | A causal effect of each interaction pattern; the subgroups were small |
| An editing agent can improve initial output while comprehension falls | (Im)Paired Programming: 54 CS students | Prevalence among professional mid-level engineers |
| Developers differ in which parts of a task they delegate | AIRELI: exploratory study of 21 participants | Population rates or long-term causal effects |
| Verifying AI output takes effort | Sonar developer survey | Who performs that work or whether it shifts from mid-level to senior engineers |
Together, these findings give a team lead a reason to check understanding separately from task completion. They do not measure how review work is redistributed inside a team.
Verification is already part of the workload
A 2026 Sonar survey provides a related, though different, signal. Among 1,149 professional developers surveyed, 95% reported spending at least some effort reviewing, testing, or correcting AI output. 38% said reviewing AI-generated code requires more effort than reviewing code written by colleagues, while 61% agreed that AI often produces code that looks correct but is not reliable.
The study does not prove that weak developers create more work for strong developers. It does show that verification is already a substantial part of AI-assisted development.
| Survey finding | Respondents | How to read it |
|---|---|---|
| Spend at least some effort reviewing, testing, or correcting AI output | 95% | Verification is common; this says nothing about hours spent |
| Say AI-generated code takes more review effort than colleagues' code | 38% | A comparison of perceived effort, not a measured time increase |
| Agree that AI often produces code that looks correct but is unreliable | 61% | A reported concern, not a measured defect rate |
Source: Sonar, State of Code Developer Survey 2026; see the linked report above. Fieldwork ran in October 2025. Each row answers a different question; the percentages should not be compared as one scale.
The team-level risk is easy to describe:
| Low-effort agent use | Cost transferred to the team |
|---|---|
| “The agent said this is the root cause” | A senior engineer reconstructs the actual cause |
| “The tests are green” | A reviewer checks whether the tests assert anything meaningful |
| “It only changes this service” | Someone searches for hidden callers and consumers |
| “Claude wrote the architecture” | A tech lead has to explain decisions the author cannot defend |
| “The agent can fix the next bug” | Development stops when the agent reaches an unfamiliar boundary |
The issue is not that every developer must write every line manually. The issue is that someone must own the model of the system.
Comprehension debt is a team liability
Technical debt is the cost of taking a shortcut in the code.
Comprehension debt is the cost of taking a shortcut in understanding.
It appears when an engineer accepts a decision they did not make and cannot reconstruct later. The code may be perfectly readable. The missing asset is the mental model:
- what the component guarantees;
- what it assumes;
- who depends on it;
- what happens when the happy path fails;
- why the implementation has this shape;
- which alternatives were rejected.
Comprehension debt is especially dangerous because it can look like productivity in the short term. The developer closes more tickets. The team sees more diffs. The dashboard records more completed work.
The debt becomes visible during the next incident, migration, or refactoring. The agent is asked to fix a bug. It cannot infer the missing business context. The developer cannot either. A senior engineer is pulled in—not to design a new feature, but to reverse-engineer a system that the team has already accepted.
This is how AI-assisted development can create an uneven team dynamic:
Agent output increases
↓
Independent understanding does not increase at the same rate
↓
More code reaches review and production
↓
The most experienced engineers become the verification layer
↓
Senior-engineer time shifts from design to reconstruction and cleanup
The Shen–Tamkin interaction analysis points to a useful distinction: using AI is not the same as outsourcing understanding. Participants who asked conceptual questions or requested explanations showed stronger quiz results than those who delegated generation or debugging. These were observational comparisons within small subgroups, not randomized tests of each behavior.
What does not solve the problem by itself?
Teams usually respond by adding more automated checks. Those checks are valuable, but they answer different questions.
| Mechanism | Good at detecting | Still leaves unanswered |
|---|---|---|
| Compilation | Type and syntax errors | Whether the design preserves the business invariant |
| Tests | Selected behavior under selected conditions | Whether the test is meaningful or the right behavior was selected |
| IDE linter | Configured static-analysis findings | Whether the product intent is correct |
| Plan agent | Missing steps and implementation options | Whether the developer understands the final result |
| Diagram | Structure and relationships | Why the changed boundary is correct |
| AI reviewer | Plausible defects and inconsistencies | Whether the author can independently defend the diff |
| Explanation and quiz | Understanding of intent, invariants, impact, and failure modes | Correctness by itself; technical checks are still required |
A second model can find issues the first missed. The team still needs to identify who can explain the change, debug it, and make the next modification safely.
The best process combines both kinds of checks:
- deterministic checks for the artifact;
- a comprehension check for the person accepting responsibility.
The quiz skill: a forcing function before merge
A quiz should not be a generic programming exam. It should be generated from the actual change.
The workflow can be simple:
1. Explain the change in your own words
Before accepting the diff, the author gives a short explanation without reopening the agent transcript:
“This change moves normalization to the API boundary because three callers can pass equivalent identifiers in different formats. The invariant is that equivalent identifiers produce one cache key. The CLI path is intentionally unchanged.”
This sentence exposes more than “the agent updated validation.” It names the behavior, boundary, invariant, and scope.
2. Ask change-specific questions
| Question | What it verifies |
|---|---|
| What was the old behavior? | Baseline understanding |
| What is the new behavior? | Change understanding |
| Which invariant must remain true? | Design understanding |
| Which callers or consumers are affected? | Impact-boundary understanding |
| What happens on the important failure path? | Operational understanding |
| What did this diff intentionally not change? | Scope understanding |
| What evidence would falsify your explanation? | Verification understanding |
3. Treat a wrong answer as a merge pause
A wrong answer is a reason to investigate. The question may be ambiguous, the expected answer may be wrong, or the developer may understand the behavior but describe it differently. An AI-generated answer key must be checked against the code and the intended behavior.
The next step may be to:
- reduce the diff;
- inspect real symbol usages;
- run a focused test or configured application;
- inspect a debugger value and call stack;
- read the actual dependency source;
- add a missing test;
- ask the agent for a narrow explanation;
- update the design note;
- reject the implementation.
Let the author inspect the code, challenge an ambiguous question, and retry. Accept an explanation supported by evidence even if it differs from the agent's wording. Block the merge only when uncertainty about a critical behavior remains unresolved. Use failures to improve the change and the questions, not to rank engineers by verbal fluency.
4. Use risk-based gates
A documentation change does not need the same gate as a concurrency or billing change.
| Change | Suggested gate |
|---|---|
| Documentation or formatting | Confirm scope and no behavior change |
| Local implementation change | Explain intent, invariant, callers, and evidence |
| Multi-file refactoring | Explain usage boundaries and rejected alternatives |
| Security, data, concurrency, or public API change | Quiz failure modes, rollback, impact boundary, and evidence |
A good quiz is short. Three precise questions about a changed invariant are more useful than twenty questions about syntax.
How to prevent the strong engineers from becoming the permanent safety net
A team lead cannot solve this by telling people to “review more carefully.” That simply turns senior engineers into a manual verification service.
The process has to make ownership visible before the review is escalated.
Option 1: Require an author-owned change summary
The author writes:
Intent:
Invariant:
Affected callers:
Important failure path:
Evidence:
Known uncertainty:
The summary must be written by the author, not generated by the same agent that produced the code.
Option 2: Make the author demo one critical path
For a runtime-sensitive change, the author shows a configured run, a focused test, a debugger value, or a trace. The point is not to perform theatre. It is to connect the explanation to an observed fact.
Option 3: Use a review budget
Ask the author to identify the two decisions most likely to be wrong. Review those first. This discourages large, vague agent diffs and makes uncertainty explicit.
Option 4: Add a quiz gate
Use the change-specific questions and risk tiers above. The agent can draft questions, but a designated reviewer resolves disputes about the answer key. Unresolved uncertainty about a critical behavior blocks the merge; a different wording or an initially wrong answer does not automatically fail the developer.
Option 5: Track learning, not just throughput
Useful team signals include:
- how often a quiz reveals a missing assumption;
- how often a diff is reduced after the quiz;
- how often a missing test or caller is discovered;
- how much senior-engineer review time goes to reconstruction rather than design;
- whether the explanation remains useful during a later incident.
Start with a two-week pilot on one class of changes. Compare review time and issues found with your recent baseline, and record how much time the gate itself adds. Keep the protocol only if it reveals useful uncertainty without turning senior reviewers into full-time examiners.
Make the team policy explicit
Before the pilot, agree on a few operating rules:
| Policy decision | Practical default for the pilot |
|---|---|
| Which changes need a quiz? | Use the risk tiers above; skip formatting and documentation-only changes |
| Who owns the explanation? | Name an accountable engineer; pair authors can explain the change together |
| Where does it live? | Store the intent, evidence, unanswered questions, and final explanation in the PR |
| Who resolves a disputed question? | A designated reviewer checks the code and requirements, not just the agent's answer |
| What happens after a failed attempt? | The author investigates, adds evidence or changes the diff, then retries |
| Who can waive the gate? | The reviewer or team lead, with the reason and remaining risk recorded in the PR |
| How do we avoid bureaucracy? | Limit the pilot to three questions per eligible change and review its overhead |
If critical uncertainty persists, escalate it as a design or requirements question. Do not pass an unexplained patch to a senior engineer with the expectation that they will silently finish the author's work.
Where Explyt fits
An IDE-native agent can help with the evidence-gathering part of this workflow. In supported JetBrains IDEs, Explyt can work with project facts and IDE operations such as symbol usages, configured runs, debugger evidence, refactorings, reviewable changes, and IDE linter feedback.
Those capabilities are mechanisms, not guarantees. A green test can still be weak. A symbol search can miss dynamic behavior. A debugger run covers one execution. A review finding can be wrong. The developer remains responsible for the requirements, architecture, and decision to accept the diff.
A quiz workflow would add an explicit check of whether the responsible engineer can use that evidence to explain the change. The final decision still belongs to the team.
Try it on your next agent-generated diff
Choose one real change this week—preferably a multi-file refactoring, bug fix, or new integration.
- Let the agent implement it.
- Run the checks your team already requires.
- Close the agent transcript.
- Ask the author to explain the change in their own words.
- Ask three questions about the invariant, callers, and failure behavior.
- Investigate wrong or vague answers; pause the merge if a critical uncertainty remains unresolved.
- Record what the team learned.
Explyt can help you inspect the project facts behind the diff inside your JetBrains IDE. A quiz skill can help test whether the person accepting the change owns those facts.
The real question is not whether your team uses AI. It is whether the people who merge AI-generated code can still explain what they are responsible for.
Try the workflow with your next Explyt-assisted change—and tell us in the comments: how does your team prevent vibe-coded output from becoming someone else’s review and cleanup problem?
Related reading: The agent wrote the code, and you lost the model covers the developer-side cost. The CTO Ships at Comprehension Speed, Not Generation Speed covers the release-policy side.
Get started with Explyt
Reproduce, inspect and fix inside JetBrains IDEs, with the debugger, run configurations and IDE facts as evidence.
Sources
- J. H. Shen and Alexander Tamkin, How AI Assistance Impacts the Formation of Coding Skills, 2026. Randomized experiment on learning a new Python library; the AI group scored 17% lower on the follow-up knowledge quiz. The sample was relatively small and the study measured immediate comprehension, not long-term career outcomes.
- How AI Impacts Skill Formation, 2026 preprint. Full experimental details, interaction patterns, preregistration, and limitations for the Shen–Tamkin study.
- “(Im)Paired Programming: Coding Agents Improve Productivity but Harm Understanding, 2026 preprint. Study of 54 computer-science students comparing an editing agent with a more implementation-preserving chatbot workflow; initial task accuracy improved, while comprehension was lower for agent users.
- Grasping AI Reliance in Program Comprehension and Coding through the AIRELI Persona Taxonomy, Proceedings of the 2026 IEEE/ACM International Conference on Program Comprehension. Controlled study of 21 participants; exploratory qualitative reliance taxonomy, not a population estimate.
- Sonar, The current reality of AI coding: State of Code Developer Survey 2026. Survey of 1,149 professional developers; fieldwork ran in October 2025.
- Explyt, AI agent for JetBrains IDEs and feature matrix. Feature depth varies by IDE, language, and configuration; verify current support before relying on a specific workflow.
