We Rolled Out an AI Coding Assistant… How Do We Prove It Helps?

This article is part of the XPRT. Magazine #21

We've been through this conversation more times than we can count. A company spends months rolling out an AI coding assistant. Licenses are paid. Training is done. Developers use it daily. Then the quarterly review comes around, and someone asks the question every rollout eventually triggers:

"We invested €X this year. Is it working?"

You open the dashboard. Usage is high. Satisfaction is high. PR throughput is up. Everything looks green. But look closer - every number on that screen answers the same question: Is the tool being used? None of them answer the harder one: Is it actually making engineering work better?

Faster delivery is part of "better." So is quality. So is the day-to-day experience of doing the work. And this isn't a new challenge. Every major shift in software engineering - Agile, DevOps, CI/CD - has raised the same question in the past: we changed how work gets done, but did we actually get better?

AI-assisted development is the latest version of that problem. It doesn't require a brand-new measurement philosophy - it requires that we make better use of the one we already have. In this article, we will explore existing frameworks in the context of AI.

Same tool, different measurement problem

Before we get into frameworks, one thing needs to be clarified: not every team has the same AI problems. Some teams still struggle with rollout. Others went further, but have no evidence that it helps. The difference usually has less to do with the tool itself than with where the team is in its journey.

We find it helpful to think about AI journey in four stages:

This isn't a linear path. Teams inside the same company can sit at different stages, and they can slide backward when a champion leaves, leadership attention fades, or the novelty wears off.

At Rollout you care about access, authentication and first use. At Enablement you care about training, confidence, and early wins. At Adoption the real measurement challenge begins - now you need engineering signals, not just usage signals. At Transformation you need to connect those changes to broader workflow and business outcomes.

The metrics that matter at one stage can be almost useless at the next. For example, we see teams can get stuck in two ways: they either keep reporting rollout numbers long after the real question has changed, or they start asking for ROI before the rollout has properly taken hold. The easiest way to see the difference is to look at two teams.

A tale of two (abstract) teams

Team A: too early to prove impact

Team A has 500 licenses, but only 18% of users are active weekly. Many developers never even authenticated. Usage is shallow - mostly basic autocomplete. Training uptake is inconsistent.

This isn't an ROI problem yet. It's a rollout and enablement problem. The question isn't "Is AI helping?" - it's "Are people using it at all?" The next step is fixing access, onboarding and support, not building a business impact dashboard.

Team X: strong usage, weak proof

Now meet Team X. They rolled out their AI coding assistant a year ago. Training is complete. Champions are in place. The AI policy is clear. Developers use the tool every day. On paper, everything looks great - the kind of dashboard people love to screenshot into leadership decks:

All green. Then the CTO looks at it and says:

"Impressive. But are we delivering better?"

And that's the harder question. Team X can show that the tool is active. They can point to real wins - less tedious work, happier developers, more PRs. What they can't prove yet is whether those gains are turning into better engineering outcomes.

Team X is at the Adoption stage, but their dashboard is still answering Rollout questions.

Start with your goal, not with a framework

When teams realize they need better measurement, the first question is usually "Which framework should we use?" The better question is: "What are we trying to understand or improve?"

For engineering organizations three broad goals usually show up first:

One framework is rarely enough. You may start with one lens, but AI adoption usually pushes you to combine several.

Four lenses for the same question

Existing frameworks didn't arrive together. They came out in sequence over a decade, built by a small, overlapping group of researchers. Each new framework addressed a limitation the previous had exposed.

And each one would tell Team X something different.

DORA: are we shipping better or just more?

DORA - DevOps Research and Assessment - emerged when DevOps was gaining momentum, but a basic question was still open: did these practices actually improve software delivery? Instead of trading opinions, the researchers studied thousands of teams over time to find out what high performers were doing differently. That work became the Accelerate book and the delivery metrics teams still use today.

The core insight remains one of the most useful in software delivery: the best teams don't choose between speed and stability. They do both.

DORA's four metrics - lead time for changes, deployment frequency, change failure rate, and recovery time - are still highly relevant in the AI era. In fact, DORA's more recent AI research makes the picture sharper: AI often acts as an amplifier, magnifying the strengths and weaknesses already present in your engineering system. More AI can mean more throughput, but it can also mean more instability.

For AI adoption, three DORA questions matter most:

For Team X, DORA would probably reveal something like this: coding got faster, but review became the bottleneck. The team is producing more change, but not necessarily turning it into faster, safer delivery overall.

But DORA only tells you what's happening in delivery. It says much less about what productivity actually means, or how developers experience their work.

SPACE: the dimension DORA can't see

DORA tells you how delivery is going. It doesn't tell you how developers are doing. SPACE, published in 2021 by a team spanning Microsoft Research, GitHub and the University of Victoria, argued that productivity isn't one number. It's at least five dimensions: Satisfaction and well-being, Performance, Activity, Communication and collaboration,Efficiency and flow.

For AI adoption one dimension matters more than the rest: satisfaction. Developers who feel less blocked, less drained by repetitive work and more able to focus stay longer, collaborate better, and take on harder problems. That shapes team stability - and team stability shapes delivery.

But SPACE also comes with a warning: satisfaction tells you AI is valued; it doesn't, by itself, tell you AI is working.

SPACE gave teams a vocabulary for productivity. What it didn't give them was a clear picture of what to actually fix.

DevEx: where did the friction move?

That gap led to DevEx, published in 2023 - a lens built around the daily reality of developer work: feedback loops, cognitive load and flow state.

In plain terms: How fast do I get useful feedback? How hard is it to understand what's going on? Can I actually get into deep work?

AI changes all three - and not always in the same direction.

The inner loop often gets better: faster results, quicker answers to local coding questions. But the outer loop can get heavier: more verification, slower reviews.

So Team X may genuinely feel faster while also creating more downstream review friction. The problem isn't always more friction or less friction - sometimes the friction just moves.

By now, Team X has three useful views. DORA shows whether delivery improved. SPACE captures how developers experience the work. DevEx shows where the pain points shifted. But it also leaves you juggling too many lenses. The CTO who asked "is it working?" doesn't want a framework tour, they want one clear answer.

DX Core 4: the executive summary

DX Core 4 solves that. In 2024, the team behind the DX platform in collaboration with the researchers behind the earlier frameworks set out to unify DORA, SPACE, and DevEx into a single model.

Speed and Quality come from DORA. Effectiveness draws on SPACE and DevEx. Impact connects engineering work to business outcomes - a dimension the earlier frameworks touched but never owned.

Same ingredients, different recipes

By this time, you can think of frameworks as recipes and metrics as ingredients. The same ingredients - cycle time, failure rate, satisfaction scores - can be rearranged into different recipes depending on your goal. Just look at the overlap: lead time appears in both DORA and SPACE. Feedback loops connect DORA and DevEx. This means that the data you collect for one framework can often support several others.

The causation gap

The frameworks got Team X this far: a clearer picture of delivery, developer experience and quality. What they still can't see is causation. The CTO doesn't just want to see the numbers go up - they want to know whether AI is what moved them.

Start with a hypothesis

"Improve productivity" is a direction, not a hypothesis. Team X needs something testable: "If developers use a documentation agent loaded with our organizational standards, style-related review comments will drop by 70%". That's specific enough to succeed or fail, and failing is just as valuable. Before running the experiment they define success criteria, pick a target team and record their baseline.

Run a real intervention

You can't run a laboratory experiment in a live organization. But you can get close. Team X introduces one specific, scoped change, keeps everything else as stable as possible and documents exactly what changed, when, and how, so someone else could repeat it later.

Triangulate

Don't rely on a single metric. Track delivery data, developer surveys and experiment-level outcomes together. When all three point the same way, you have something real. When they don't converge, you haven't failed the experiment - you've found where to look next.

Scale through repetition

One experiment is a data point. Similar results across multiple teams become a pattern. Each cycle sharpens the next hypothesis.

Use external benchmarks for context

"We save 4.25 hours per week" sounds impressive, but compared to what? Platforms like GetDX provide industry benchmarks. If the P75 for time saved is 3.46 hours, Team X knows their result is genuinely strong, not just better than a weak starting point.

Evidence, not proof

None of this is laboratory science. But clear baselines, multiple data sources and reproducibility across teams get you close. Before you run the experiment, decide what failure looks like. If nothing could disprove your hypothesis, you're not running an experiment, you're running a confirmation exercise.

Key takeaways