Kibble Blog

How to monitor, evaluate and improve an engineering team in the AI era

By Nhan Duc Nguyen · 30 August 2026 · 8 min read

A year ago you could look at a team's commits, review turnaround and story points and have a rough idea of how it was doing. Then the agents arrived, the commit count tripled, and the number stopped meaning anything. This is a working set of measures for a CTO or AI lead whose team now ships with Claude Code, Codex and whatever comes next: what to monitor, how to evaluate it, and which reading is a trigger to change something. None of it needs anyone's prompts.

What broke

Every classic engineering metric is a proxy for effort, and agents decoupled effort from output. Lines of code, commits per week, PRs opened: an engineer with an agent produces several times more of each without being several times more useful. Story points measured human estimation, which is now the estimate for a human plus a tool the human is still learning. Even cycle time, the best of the old set, lost a leg: the time from ticket to merged PR is now dominated by review, because generating the change is minutes and reading it is not.

Two new costs also appeared with no metric attached. The first is money: a coding agent is a metered thing, priced per token, and a team of ten can quietly spend more on tokens than on its CI. The second is attention: every skill, command and MCP server an engineer installs adds text to every session's context, whether or not it is ever used, and nobody is counting it.

So the job is not to replace the old dashboard with a new one. It is to keep the part of it that still measures outcomes, add the two costs that are new, and read all three together, every week.

Monitor: three layers, one grain

The measures below share one grain, per engineer, per repo, per week, so they can be laid side by side. A number that only exists per team, or only per month, cannot be compared with one that exists per person and per day, and comparison is the whole method.

Layer 1: delivery, which you already have

• Merged changes per repo per week, counted as PRs, not commits. Commits are free now; a merged PR still had to pass a human.
• Review latency: time from PR open to first review, and from first review to merge. This is where the hours now go.
• Change failure rate and time to restore, the two DORA measures that survived intact, because an outage is an outage whoever wrote the code.

Leave story points alone for now. They will re-calibrate on their own once the team has a few months with the tools; forcing them to earlier only produces an argument about what a point means.

Layer 2: agent spend, per engineer, repo and model

This is the layer most teams do not have. The vendor invoice is one number per month, after the month, and it cannot say which engineer, which repo, which model, or what changed. The information exists, though: every coding agent writes a transcript on the machine it runs on, with token counts per turn, the model, and the working directory (where those files live). A collector that reads counts from those files and sends nothing else gives you:

• Spend per engineer, this week against last, at the vendor's list rates, with the biggest mover named. Keep it beside the billed figure from your Console and never add the two.
• Spend per repo, which is the number that turns a cost into a decision: a repo that costs three times the others per merged PR is a repo with a problem, and the problem is usually the codebase, not the engineer.
• Model mix: what share of tokens went to the expensive model and whether that share is moving.
• Session shape: turns per session, tool calls per turn, errors, compactions. The counts are enough to tell a long productive session from one that went in circles.

Layer 3: the skills the team actually uses

A team that has been using agents for six months has accumulated dozens of skills, slash commands and MCP servers across personal configs, repo checkouts and plugins. Each one has a description the model reads in every session to know it exists, paid whether or not it fires, and a body paid each time it does. Inventory them, by name only, and record three things per week:

• Installed where, and by how many people. Two teams with their own release command is a merge waiting to happen.
• Invocations, split into model-chosen and typed. A skill the model picks on its own is doing its job; one that only ever fires when someone types its name has a description the model cannot match.
• Always-on context: the tokens the installed set costs every session before any work starts, and how much of that belongs to skills that never fired.

Evaluate: read them together, weekly

Any one of these numbers on its own is a stick to beat someone with, which is why teams are right to distrust metrics. The evaluation is the ratios between layers, per repo, over a week:

Two rules make the reading safe to publish inside the company. Engineers see their own numbers at the same grain you do, on their own machines, before anyone else does, so nothing on the weekly page is a surprise to the person named on it. And the page shows counts, never content: no prompts, no diffs, no file paths. A team will accept being measured on cost and output; it will not accept being read.

Improve: the triggers

Most of the value is in a short list of standing rules that turn a reading into an action without a meeting.

1. Remove any skill installed for four weeks with zero invocations. It costs context in every session and earns nothing. If someone misses it, it is one command to bring back.
2. Merge duplicates. When two names resolve to the same job in two teams' configs, pick one, put it in the repo's shared .claude, and delete the others. The adopters count tells you whether the merge took.
3. Rewrite the typed-only ones. A skill that never fires on its own has a description the model does not recognise as matching the task. Rewrite the description, not the body, and watch the model-chosen count the following week.
4. Set a model default per repo, not per person. If a repo's work is mostly small edits, the cheaper model is the default there; the expensive one is opted into for a migration. The model mix per repo is the check that the default is holding.
5. Fix the repo the agent keeps fighting. A repo with high spend per merged PR and high errors per session needs an AGENTS.md, a build that runs in one command, and tests the agent can run itself. Do that once and the spend per PR drops for every engineer in it.
6. Put review capacity where the output went. When review latency climbs in a repo, that is where the next reviewer goes. It is the one number that tells you where the humans are needed.

What not to do

Do not set a spend budget per engineer. It makes the cautious engineer stop using the tool and does nothing to the one who was going to overspend anyway; a per-repo default model and a weekly reading of movers does the same job without the chilling effect. Do not rank people by spend, in either direction: the top spender is as likely to be doing the migration everyone else avoided as to be wasting money. And do not read transcripts to find out. The counts answer every question in this post, and a team that knows its prompts are private will use the tool the way you want it used.