When a Successful AI Pilot Makes the Process Worse

At Meridian Components, a fictional industrial supplier, an AI assistant increased order-entry throughput by 40%. Users adopted it. Accuracy met the pilot threshold. The project team could show more orders entered per hour and less manual effort. Then delivery reliability fell.

The assistant had accelerated an existing process. Entering an order also reserved stock, even when the order still needed a credit or product-substitution decision. As more unresolved orders entered the system, they held inventory for longer. Some otherwise ready orders could no longer ship by the dates already promised to customers.

The software performed its assigned task. The scale decision missed the interaction between faster entry, exception capacity, and stock allocation.

Meridian is a fictional teaching case; its figures and outcomes are assumed. It shows how a useful tool can clash with inherited operating rules.

This article shows how to decide whether a successful AI pilot is ready to scale, using "SCOPE", a framework I have created for AI-enabled process redesign, tests whether task gains create customer value, compares credible alternatives, and identifies who can authorize the trade-off.

Before expanding an AI solution, it is important to identify the operating change that converts its gain into value, compare a credible conventional design, and assign authority over the full trade-off.

Task improvement alone does not establish system improvement. A pilot supplies evidence about the work tested; the scale decision must connect it to the outcome the organization wants and the consequences transferred elsewhere.

What the pilot actually established

Before the entry assistant was introduced, Meridian released 100 orders a day from order entry. About 28% needed an exception decision. The receiving teams could resolve around 30 exceptions a day. Average inflow was below that nominal capacity, but there was little room for variation, absence, or rework.

The assistant raised releases to 140 orders a day while an existing backlog supplied enough work. With the exception rate unchanged, about 39 exceptions arrived each day against the same nominal capacity of 30. Even if that capacity held, the queue would gain roughly nine cases a day on average.

That arithmetic establishes downstream overload. It does not, on its own, establish worse customer delivery. Some of those orders previously waited before entry. Moving a queue can expose a problem without making the whole system worse.

The additional mechanism matters. Meridian’s old reservation rule tied up stock when an order was entered, before its exceptions were cleared. Faster release increased reservations against unresolved orders. Stock that could have served ready orders became unavailable. The affected orders missed their original promise dates because of those reservations, not because a dashboard has merely started counting a different queue.

To investigate such a claim in a real business, measure from the customer’s original request, include the pre-entry backlog, and preserve the original promise date. Follow comparable order groups through to delivery. Check whether blocked reservations actually prevented shipments and whether demand mix, supply disruption, or another change explains the result. A growing queue is a diagnostic signal, not the customer outcome.

Meridian was already fragile. The assistant amplified a dependency that its operating rules had left unresolved. Demand growth or conventional automation could have exposed the same problem. The lesson applies to an AI investment without requiring a failure unique to AI.

The team therefore compares release limits, delayed reservation, additional exception capacity, and upstream prevention. It can also test whether retaining the entry tool’s time savings without increasing release creates enough value. Conventional redesign is a serious candidate.

Figure 1. The reservation rule connects upstream acceleration to customer harm. Queue growth alone does not establish that connection.

The error was treating increased local capacity as a sufficient reason to scale an unchanged operating process.

Task augmentation can be the right answer

There is good evidence for useful task-level AI. In a quasi-experimental study of staggered adoption among 5,172 customer-support agents, access to a generative AI assistant increased issues resolved per hour by 15% on average. Gains differed across workers, with less-experienced and lower-skilled agents benefiting particularly strongly. These are meaningful results from a specific firm and support setting. ^[1]

A randomized experiment involving 758 consultants found better performance on tasks within the tested model’s capability frontier and worse performance on a task outside it. The lesson concerns matching assistance to work; it does not demonstrate the queue mechanism in Meridian. ^[2]

A bounded tool can be a sensible investment when its value mechanism is credible, its effects on adjacent work are understood, and failures can be detected and contained. The improved activity need not be the throughput bottleneck. Reducing the cost of an administrative task, making a service more accessible, or easing employee workload may be worthwhile without increasing total output.

For a modest change, use a light assessment:

The depth of investigation should follow the consequence and uncertainty of the decision. A tool that helps an employee prepare an internal draft does not automatically require a cross-functional transformation program.

When the comparison supports adoption

Consider a separate fictional example: an internal support team answering routine questions about approved procedures. It wants to resolve requests correctly with less total effort. The team compares a maintained search-and-template workflow with an assistant that drafts answers linked to the same approved sources. Both receive the same source updates and staff training; the comparison includes preparation, checking, correction, escalation, and recurring support costs.

Suppose the evidence clears the agreed effort and answer-quality criteria for one routine request category, including follow-up on unresolved and reopened requests. These are teaching assumptions, not reported results. The support owner adopts AI drafting only for that category, with an analyst checking each answer before release and the conventional workflow available as fallback. Requests that require policy interpretation or lack adequate sources remain outside approval. The owner assigns the released capacity to clearing the existing support backlog and checks whether response times actually improve. Changed sources, repeated omissions, or a loss of the expected benefit reopen the decision.

Here, the comparison supports bounded AI adoption. The useful result is a better support service, and the operating commitment makes that benefit testable.

The operating decision behind the benefit

Hours saved are an intermediate result. Their value depends on what becomes possible and what the organization does next.

If the aim is cost reduction, saved minutes do not automatically reduce expenditure. A manager may need to reduce overtime, avoid an additional hire, or change how work is scheduled. If the aim is growth, there must be useful demand and enough capacity in the rest of the flow. If the aim is better service, someone must allocate the freed time to that service and measure the change.

Retaining slack can also be a deliberate benefit. A team may use it to absorb demand variation, train new colleagues, or handle difficult cases without sustained overtime. That choice should appear in the business case. Otherwise, a later manager may remove the capacity and unintentionally remove the expected resilience with it.

The calculation must include the work introduced by the AI solution and its operating changes: integration, checking, correction, support, evaluation, and knowledge maintenance. A task can become faster while its total cost remains unchanged because another team now performs the missing work.

For Meridian, the release capacity belonged on the dashboard. It belonged in the value calculation only alongside reliable completed orders, inventory effects, exception effort, and operating cost. The owner of the benefit needed authority to change release and reservation rules, not merely access to the assistant’s usage statistics.

This is the practical bridge from productivity to value: name the operating decision that realizes the benefit, its owner, its costs, and the evidence that it happened.

Start with an outcome or a capability, then establish the value

A useful starting point is the outcome-producing system:

Purpose and stakeholder value → system outcome → flow and leverage points → processes and decisions → tasks and tools.

For Meridian, the outcome is a reliable customer promise. The relevant system includes sales, credit, inventory, planning, fulfillment, and the rules connecting them. Faster entry is only one possible design choice. Discovery can also begin with a new capability. A tool may make a service possible that nobody previously considered feasible. Exploring that possibility is legitimate. The inquiry should move between capability and value until the team can state a testable operating proposition.

Figure 2. Discovery can run in either direction. The scale decision requires a credible value mechanism, evidence, and acceptable trade-offs.

The figure separates how an idea is discovered from how its expansion is justified. For either starting point, expansion depends on a credible route to value. That includes examining the operating rules the tool inherits.

Processes deserve close attention because they translate strategy, resources, policy, and knowledge into repeated action. They also carry historical assumptions about the cost of information, the location of authority, and what technology can do. Michael Hammer challenged the practice of computerizing inherited work long before generative AI. ^[3]

Meridian’s reservation rule is such an assumption. It may once have helped secure scarce stock for customers. That purpose deserves investigation. The redesign question is whether the same purpose can be served without preventing ready orders from shipping.

Each handover, check, and control should have an understood purpose. A forgotten rationale is a reason to investigate, not permission to delete. Equally, a familiar procedure should not be preserved simply because an AI system can now perform it cheaply.

Processes do not explain every business outcome. Strategy, demand, competition, supply, resources, and external events still matter. The value of process redesign is that it gives leaders a concrete surface on which to intervene and observe consequences.

What changes elsewhere when the tool succeeds

When order entry became faster, exception handling became the next limit on flow. This shift is called constraint migration, an idea from Theory of Constraints. ^[4]

Migration can accompany progress. A business that increases output will eventually encounter another limit. The managerial concern is whether the receiving system is prepared and whether the change introduces avoidable harm.

AI can make this question urgent where capacity changes quickly or generated output creates additional review and correction work. Faster document production may increase the load on reviewers. Automated intake may send more cases to scarce specialists. Several independently approved tools may draw on the same knowledge owners or support team.

The prediction depends on what changes: higher release can increase downstream demand; better-quality inputs can reduce it. If arrivals stay fixed, assess the proposed cost, service, or resilience benefit on its own terms.

Before scale, ask which assumption is doing the work. If the expected benefit depends on higher throughput, identify the likely next constraint and a signal that would reveal it. If it depends on lower cost, trace the costs that disappear and those that move. If the purpose is resilience, test what happens under disruption rather than judging the design only in an average week.

For Meridian, exception arrivals and resolution capacity are leading signals. Stock held by unresolved orders is a second signal connecting the queue to delivery risk. Neither replaces observation of delivery against the original promise.

A prediction should remain open to correction. Record an alternative explanation and decide what evidence would make the team change its mind.

The trade-off belongs in the scale decision

A scale decision needs more than a primary metric, but not every measure should carry the same authority.

Meridian wants reliable delivery. It also needs to understand expedite cost, correction effort, customer contacts, employee burden, and the distribution of service across customer groups. Some of these are countermetrics: they expose what the improvement costs. Others are hard guardrails, such as a prohibition on unauthorized credit decisions.

A countermetric can worsen within an agreed tolerance if the trade-off is justified. An increase in operating cost may be worth a sufficient improvement in reliability. A temporary training burden may be acceptable within a specified transition period. These decisions require evidence and explicit agreement about who bears the cost.

A hard guardrail is different. A good average delivery result cannot compensate for a prohibited action.

The leadership rule is therefore: A local gain cannot authorize scale when hard guardrails are breached or the evidence fails the agreed system-value and trade-off thresholds.

Set those thresholds before reading the pilot result. Specify the observation period, affected groups, acceptable transition effects, and decision authority. If circumstances require a revised threshold, document the reason and obtain fresh approval rather than quietly moving the goalposts.

This makes ownership concrete. Someone must be able to resolve a dispute between sales, finance, operations, and technology. A named outcome owner without authority over those trade-offs cannot fulfill the role.

Qualitative research on 23 Swiss companies, conducted from 2022 to 2025, identified two that the authors judged to have developed a scaled strategic GenAI capability. Their account emphasizes cross-functional process knowledge, reusable support, feedback, and continuous iteration. The study does not establish that those practices caused success, but it supports examining organizational capability alongside deployments. _^[5]

The SCOPE framework

SCOPE organizes these checks into five recurring decisions:

SCOPE integrates established ideas from process management, systems thinking, constraint management, human factors, experimentation, and governance. The integrated framework awaits field validation.

A SCOPE decision states why a local gain passes or fails the system test, what a non-AI design could achieve, where work or risk may move, and who will perform the feedback and exception work after launch.

End-to-end attention need not mean wholesale reengineering. A light cycle can confirm that augmentation is enough. A deeper cycle can still select a modest redesign. In Meridian, the team adopts the supported conventional operating changes and holds expansion of additional AI assistance because its incremental value remains uncertain. The larger promise-redesign option also waits for adequate data.

SCOPE at the next scale review

Begin with these three questions:

A bounded benefit with understood effects may justify straightforward augmentation. A harmful reservation rule, an overloaded shared team, or an unresolved authority conflict may require changes to the surrounding work before expansion.

The next article follows one decision through SCOPE. Article 3 examines authority, funding, and the work needed to sustain redesign. Article 4 is an optional implementation guide with a worked decision record.

At the next review, put the pilot dashboard beside the customer outcome and the benefit owner’s operating decision. The gap between them is where the real scale decision begins. Use SCOPE to work through it.

Connect your AI gains to business value

Let's talk about your AI initiative. If your AI pilot is delivering results but the wider business benefit remains unclear, at Xebia we can help you trace where work is slowing, identify the rules and dependencies behind the problem, and assess practical changes.

References

[1]  Erik Brynjolfsson, Danielle Li, and Lindsey Raymond, “Generative AI at Work,” The Quarterly Journal of Economics 140, no. 2 (2025): 889-942. One firm, tool, and occupational setting; these findings do not establish effects across all enterprise workflows.

[2]  Fabrizio Dell’Acqua et al., “Navigating the Jagged Technological Frontier,” Organization Science 37, no. 2 (2026): 403-423. Experimental consulting tasks and GPT-4; results concern the tested work and capability frontier.

[3]  Michael Hammer, “Reengineering Work: Don’t Automate, Obliterate,” Harvard Business Review, July-August 1990. Foundational argument, not evidence that SCOPE is effective.

[4]  Eliyahu M. Goldratt and Jeff Cox, The Goal, first published 1984. Constraint movement is not itself evidence of deteriorating performance.

[5]  Kevin Schmitt, Gregory Vial, and Ivo Blohm, “Create Generative AI Value at Scale,” MIT Sloan Management Review, Summer 2026. Qualitative findings, not a representative failure rate or causal comparison.