Blog

AI Gateway Traces: How to Find and Fix Wasted Tokens in Your AI Agents

An AI gateway sits between your agent and the LLM provider. Every request and every response passes through it, so it can record the whole story of a task.

November 19, 2026
10 minutes
‌
AI Gateway Traces

The fastest way to cut what your AI agents cost is to read what they actually do. An AI gateway records every step an agent takes. That record is called a trace, and it shows you where tokens go to waste. Often you can simply ask an LLM to read it for you.

This post is the hands-on follow-up to AI Token Economics: How to Cut AI Agent Costs Without Losing Quality, where we explained why you need visibility before you change anything. Here we look at what to do once the traces start coming in.

What an AI gateway trace shows

An AI gateway sits between your agent and the LLM provider. Every request and every response passes through it, so it can record the whole story of a task. That story is the trace: one step after the other, from the first instruction to the final result.

Take an agent that converts a SQL file to a different SQL dialect. It reads the file, works through it step by step and writes the result. The trace lists each of those steps.

This matters because wasteful behaviour rarely stops an agent from finishing its task. If you only look at the output, everything seems fine, and there is nothing to optimise. The trace is where you see how much work it took to get there.

‌
ai-gateway-diagram

How do you find wasted tokens in a trace?

Start simple: ask Claude to read a trace and give you honest feedback on how the run went. LLMs are good at spotting what happened in a sequence of steps. Once you have a handful of runs, the same problems show up again and again.

Here is a clear example. In one migration agent we looked at, each run wrote its own parser for an XML-heavy file format, used it to pull out the information, and then threw it away. In a trace, you see that straight away.

The fix is a tool or script that is written once and reused in every session. That lowers the number of tokens you spend on each run. The same goes for any website, API or internal tool your agent works with. A small, dedicated tool usually costs far fewer tokens than letting the agent work it out from scratch every time.

How many traces are enough

Fewer than you might think. Even around 10 sessions can show you the big problems, because they are repetitive. The agent builds the same throwaway tool, or reaches for the help command, almost every time. A few dozen or 100 traces make the picture clearer.

The last few percent of savings need more. At that point you want a golden data set, or two versions of the same prompt running side by side, so you can see which one uses fewer tokens while users stay happy. It works like an A/B test in data science.

When the agent uses a tool the wrong way

Look at how the agent handles the tools you gave it. It might pass the wrong arguments, ignore the tool, or keep calling its help command. When you see that, the tool is usually fine and the prompt is the problem. The manual you gave the agent is too vague.

The fix is to add short snippets that show exactly how to use the tool, so nothing is left to guess. For example: if you want to run this kind of task, call the tool in this way, with these parameters.

The prompt can also set limits. If an agent should convert one SQL file, tell it to read only that file. Without that instruction it may go on a witch hunt through the codebase, which costs a lot of tokens and rarely helps. The trace makes this easy to spot, because you can see the agent opening more than one file.

A longer prompt is not always a better prompt 

Adding additional instructions to a prompt usually helps an LLM, and it is the easy fix. But you pay for every token of that prompt, every time it is sent. A manual with examples for each parameter will help, but if a version half as long works just as well that one is better.

Pruning instructions is harder than adding them, and right now it is still a bit of black magic. You need to test what you can cut without losing quality.

Models change what a prompt needs, too. Claude Code is an example: its own prompt was cut back sharply as the models improved, because many of the "don't do this, do that" rules were no longer needed. Older models needed a lot of examples of how to use a tool. Newer ones often get by with little instruction.

‌
long-vs-short-prompt

Shortcuts for shorter prompts, and why they fall short

One popular shortcut is the "talk like a caveman" plugin. It makes prompts choppy, so you need fewer tokens to tell the LLM what you want. It was very popular, and it still is.

It is also a hack. It may give you a bit of efficiency, but it does not solve the actual problem, and it is not the direction we should be going.

There are better ways to reduce the number of tokens in a prompt. Ask what the prompt really needs to keep the same performance, cut the rest, and test the result. The traces show you where to look first.

Cached tokens: why long sessions get expensive

You pay for two kinds of tokens: new tokens and cached tokens. When you ask a question, it is new. The agent answers, and you ask a follow-up. Now your first question and the answer are sent again as cached tokens, together with the new question. This repeats with every turn.

In a normal coding or agent session, most of the tokens you pay for are cached. Cached tokens cost less than new ones, which is good. But you pay for them again on every turn, and they keep piling up.

That is why every new topic should get a new session. It saves money, because you stop paying for old tokens that have nothing to do with your current question. It can also improve the results, since the LLM cannot be confused by earlier output that is unrelated to what you are asking now. 

Does batching tasks save tokens?

Sometimes, but not without limits. Every time you start an agent, you send the initial prompt with the work instruction and the task you want it to perform. If you run two tasks in two separate sessions, you pay for that initial work instruction prompt twice. Batching allows you to send the prompt once, followed by two (or more) tasks you want the agent to perform. And hence, allowing you pay once for work instruction prompt instead of every time.

But there is a catch. In a longer session, everything the agent did for the first task stays in the context. It is sent again as cached tokens while the agent works on the second, third, etc task. This causes your saving to grow at first, but then shrinks as the cache builds up, and you start paying more for cached tokens than you would for the new initial prompt.

For independent tasks, such as separate files that share nothing, the cheapest approach is to batch a limited number, say five or ten files. Then end the session and start a new one for the next batch. If the tasks are linked and the next one needs what the previous one produced, you can also consider compacting the session instead. Compacting creates a summary of what happened so far, allowing the agent to learn from what happened before without needing the whole conversation.

Routing each question to the right model

Routing is another feature which people are experimenting with now. Not every question needs the same model or the same thinking level. The AI Gateway or router looks at the initial question of a user and decides where to send it. A simple question goes to a small model, a harder one to a mid-size model, and a very hard one to the most capable model. The code that sends the question, a chat agent or a customer support agent for example, does not need to change.

The risk is a wrong first choice. A conversation can start simple and turn complex. Switching to a stronger model then means sending the whole context again, and all of it counts as new tokens, making switching expensive. That is why most setups choose at the start and hope for the best for the rest of the session.

We looked at when a cheaper model is really cheaper in our earlier post.

‌
ai-gateway-routing

Where to start: match the trace to the fix

Look at what your traces show and match it to the fix. The first two rows are the easiest, and they give you most of the savings. Trimming prompts, batching and routing need more testing, ideally with a golden data set. 

What you see in the trace

What it usually means

What to do

The agent builds a tool and throws it away

There is no shared tool

Write the script once and reuse it

Tool errors, wrong arguments or calls to the help command

The prompt's manual is vague

Add short usage snippets to the prompt

The agent opens many files or searches the codebase

The prompt sets no scope

Tell it which file or area to work on

One long session covering unrelated topics

Old context is paid for again on every turn

Start a new session for each topic

Many similar, independent tasks, each in its own session

The prompt is paid for on every task

Batch five to ten, then start a new session

Simple questions sent to the most capable model

Every question gets the same model

Let the gateway route by difficulty

Conclusion: start with the traces

You do not need a perfect setup to start. Put an AI gateway in front of your agents, collect a few runs, and ask an LLM to read them. Fix the repeated tools and the vague manual first. Then work on prompt size, session habits, batching and routing, and check each change on a golden data set so you keep the quality. 

If you want help setting up an AI gateway or reading what your agents do, get in touch with our Data & AI team at Xebia.


Frequently Asked Questions

1. What is a trace in an AI gateway? 

A trace is the record of every step an agent took to complete a task, in order, from the first instruction to the final result. 

2. Should I start a new session for every topic? 

Yes. Each follow-up sends the earlier conversation again as cached tokens, so unrelated old context keeps costing you and can confuse the model. 

3. Does batching tasks always reduce token costs? 

No. Batching saves you from paying for the prompt on every task, but the cache grows with every file in the session. For independent tasks, batch around five to ten, then start a new session. 

4. Is a shorter prompt always better? 

A shorter prompt is cheaper, because you pay for every prompt token each time it is sent. Cut only what you can remove without losing quality, and test the change before you keep it. 

5. Can an AI gateway pick the right model for each question? 

Yes, it can send simple questions to smaller models and hard ones to more capable models. Switching in the middle of a session is expensive, because the full context is sent again as new tokens. 

Written by

Niels Zeilemaker

Global CTO, Data & AI

Niels Zeilemaker helps some of the world’s most recognized enterprises move from data ambition to real, scalable AI execution. With a rare blend of deep technical expertise and hands-on delivery experience, Niels works at the intersection of data platforms, analytics, Generative AI, and emerging agentic systems.

Contact

Let’s discuss how we can support your journey.

‌
‌
‌
‌
‌
‌
‌
‌
‌