Kibble Blog

How to monitor, evaluate and improve an engineering team in the AI era

By Kibble · 30 August 2026 · 8 min read

Evaluate engineering AI ROI by comparing delivery outcomes, actual agent costs and review effort over the same reporting window. Use per-repository trends to form questions, then check them with the team. This is a proposed operating method for a CTO or AI lead, not a validated causal model or a benchmark dataset.

What broke

More generated code does not establish better delivery. Keep outcome measures such as stability and delivery speed, and examine whether review effort or rework changes as agent use grows. DORA's software delivery metrics provide a starting point; no token count can substitute for those outcomes.

Two new costs also appeared with no metric attached. The first is money: a coding agent is a metered thing, priced per token, and a team of ten can quietly spend more on tokens than on its CI. The second is attention: skill descriptions and tool definitions can consume context before a task uses them. Loading behavior varies by agent and configuration; inventory size alone is not a measured bill.

So the job is not to replace the old dashboard with a new one. It is to keep the part of it that still measures outcomes, add the two costs that are new, and read all three together, every week.

Monitor: three layers, one grain

Use the same reporting window and a comparable application or repository scope. Keep delivery outcomes at the team or application level; aggregate cost cuts to that scope before comparing them. Per-engineer usage can help check coverage, but it is not an individual productivity score.

Layer 1: delivery, which you already have

• Merged changes per repo per week, counted as PRs, with the team's review policy and change sizes recorded. This measures flow, not value by itself.
• Review latency: time from PR open to first review, and from first review to merge. Check whether review is becoming a bottleneck.
• Change fail rate and failed deployment recovery time, two of DORA's current five delivery metrics. Read them alongside throughput and rework, at the application or service level.

Keep estimation conventions explicit. If the team changes how it assigns story points, mark the change so a before-and-after comparison does not mistake a new convention for a delivery improvement.

Layer 2: agent spend, per engineer, repo and model

Begin with the analytics your vendor provides, then add local estimates where they answer a missing question. Claude Code and Codex keep supported local session records (where those files live). A collector can give the team a shared view across agents and repositories:

• Spend per engineer, this week against last, at the vendor's list rates, with the biggest mover named. Keep it beside the vendor estimate from your Console and never add the two.
• Spend per repo, an estimate that can identify questions to investigate. A higher cost per merged PR may reflect task difficulty, change size, incomplete attribution or wasted work; the ratio alone cannot identify the cause.
• Model mix: what share of tokens went to the expensive model and whether that share is moving.
• Session shape: turns per session, tool calls per turn, errors and compactions. These counts can flag sessions worth discussing, but cannot prove useful output or a loop without additional context from the engineer.

Layer 3: the skills the team actually uses

Teams can accumulate skills, slash commands and MCP servers across personal configs, repo checkouts and plugins. Descriptions may be available before invocation, while bodies load when used. Claude Code's skill documentation describes exceptions for manual-only skills and context reuse. Kibble estimates description sizes; it does not measure a bill for every installed capability. Inventory them, by name only, and record three things per week:

• Installed by whom, within the reader's permitted scope. Matching command names can prompt a conversation about overlap, but Kibble does not send installation paths or source contents, so names alone cannot establish duplication.
• Invocations, split into model-chosen and typed. A typed-only skill may be intentionally manual. Check its purpose and configuration before treating that pattern as a discoverability problem.
• Always-on context: the estimated description size of the inventoried set, alongside observed Claude Code usage. Codex records explicit skill selections and recognized command submissions; automatic skill reads and skill cost attribution remain unobserved, so zero is not evidence of inactivity.

How do you evaluate engineering AI ROI?

Compare the cost of the tool with evidence of useful delivery and the work needed to review and maintain it. A change in a ratio is a prompt to investigate, not proof that AI caused the outcome. These are hypotheses to test with your team:

Two rules make the reading safe to publish inside the company. Engineers see their own numbers at the same grain you do, on their own machines, before anyone else does, so nothing on the weekly page is a surprise to the person named on it. And the page shows counts, never content: no prompts, no diffs, no file paths. A team will accept being measured on cost and output; it will not accept being read.

Improve: the triggers

Most of the value is in a short list of standing rules that turn a reading into an action without a meeting.

1. Review Claude Code skills with no observed use over a representative period. Check collection coverage, manual-only use and emergency workflows with the owner before removing one. Codex records explicit skill selections and recognized command submissions, but automatic skill reads remain unobserved, so its zeros cannot establish non-use. The cleanup guide gives a worked example.
2. Merge duplicates. When two names resolve to the same job in two teams' configs, pick one, put it in the repo's shared .claude, and delete the others. The adopters count tells you whether the merge took.
3. Check typed-only skills against their intended use. Keep deliberately manual workflows manual. For a skill intended to trigger automatically, review its description and settings, then observe another comparable period.
4. Set a model default per repo, not per person. If a repo's work is mostly small edits, the cheaper model is the default there; the expensive one is opted into for a migration. The model mix per repo is the check that the default is holding.
5. Fix the repo the agent keeps fighting. A repo with high spend per merged PR and high errors per session needs an AGENTS.md, a build that runs in one command, and tests the agent can run itself. Do that once and the spend per PR drops for every engineer in it.
6. Put review capacity where the output went. When review latency climbs in a repo, that is where the next reviewer goes. It is the one number that tells you where the humans are needed.

What not to do

Do not set a spend budget per engineer. It makes the cautious engineer stop using the tool and does nothing to the one who was going to overspend anyway; a per-repo default model and a weekly reading of movers does the same job without the chilling effect. Do not rank people by spend, in either direction: the top spender is as likely to be doing the migration everyone else avoided as to be wasting money. And do not read transcripts to find out. The counts answer every question in this post, and a team that knows its prompts are private will use the tool the way you want it used.