Articles
The Hidden Life of Tokens in GitHub Copilot

Introduction
GitHub Copilot moved to usage-based billing on June 1, 2026. Instead of a fixed pool of premium requests, every plan now includes a monthly amount of GitHub AI Credits consumed based on actual token usage (input, output, and cached tokens) at the published rates for each model. This aligns GitHub Copilot with how every other major AI coding assistant already prices its service: you pay for what you use.
The reasoning is straightforward: agentic sessions, where a single user prompt can trigger dozens of model turns, file reads, test runs, and iterative edits across an entire repository, consume vastly more compute than a quick chat question. Subsidising that difference at a flat rate is not a sustainable model for anyone over the long run. Billing on actual usage creates a fair relationship between cost and value, and it shifts the incentive toward using Copilot thoughtfully rather than carelessly.
That shift is also an opportunity. When Copilot sends a request to a model, it sends a full prompt: system instructions, a catalog of available tools, and the entire conversation history so far. Model providers cache the leading portion of that prompt so that if the next request starts with the same bytes, those tokens can be served from cache instead of being reprocessed from scratch. Tokens served this way are called cached input tokens, and they are typically billed at up to ten times less than regular input tokens. A session where Copilot reuses 90% of its context on every turn therefore costs a fraction of one that re-sends the full prompt from scratch. The reason that these cached input tokens are so cheap is that the provider can skip all the expensive steps of recomputing the transformer layers for those tokens, and can often skip the full request validation and parsing steps as well. Then it only comes down to reusing the cached activations instead of recomputing them, which is where the big cost savings come from. The good news is that most of the behaviours that inflate token costs (switching models mid-session, injecting timestamps into instructions, churning through tool definitions) are fixable once you can see them.
The diagram below visualises that idea in an agentic session. Each turn appends to the same growing prompt. As long as the prefix stays stable, every turn after the first should report a cache hit on essentially all the older tokens. The drift (the blue bar) is the first chunk of bytes that changed between turns, which is where the cache broke.

How caching works in GitHub Copilot
Every model provider Copilot talks to (OpenAI, Anthropic, Gemini, GitHub Models, ...) implements prompt-prefix caching. VS Code does not call these providers directly. Requests route through GitHub's CAPI proxy, which forwards them to the provider and helps keep the shared prefix cache warm across users (see the FAQ on cache lifetime for details). Your request is a long byte sequence: system instructions | tool catalog | messages[0] | messages[1] | .... The provider hashes the leading bytes of that sequence and stores them in a per-tenant cache for a few minutes. If the next request shares the same leading bytes, those tokens are billed at a steep discount (often ~10× cheaper) and are reported back as cached input. The first byte that differs ends the cache hit: everything from that point on is billed at full price, and (for some providers) those uncached tokens are written into the cache as cache write to seed the next turn.
The three token counters that matter for this guide are therefore input (total), cached input (cheap, reused from cache), and cache write (written to cache, only reported by some providers). They come from the GenAI OpenTelemetry semantic convention; see https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/. The cost of caching per model can be found here: https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing.
The reason cached input tokens are cheaper becomes clearer when you look at what the model provider can skip. On the first pass, the model tokenizes the whole prompt, computes the attention state for every input token, and stores reusable key/value tensors. On the second pass, if the beginning of the prompt is still byte-for-byte the same, the provider can retrieve that cached state and only process the new suffix before generating the next response.

Why it matters more in agentic mode
In agent mode a single user prompt does not map to one LLM call. The tool-calling loop turns one prompt into a chain of model turns:

Every turn appends to the same growing prefix. As long as nothing in the existing prefix changes between turns, every turn after the first should report a cache hit on essentially all the older tokens. A healthy agentic session looks like this in the rail:

An unhealthy one looks like this:

The prompt signature: what is actually cache-keyed
The cache key is the full prompt read left to right. The first two segments are always fixed; the messages that follow accumulate as the conversation unfolds:

This is the full extent of what is hashed for caching: the system instructions, the tool catalog, and then every message in order. The first byte that differs between two consecutive requests ends the cache hit: everything from that point on is billed at full price.
Introducing the Cache Explorer in VS Code
⚠️ The cache explorer is still under heavy development. By the time you read this article, it may have additional features, different visuals, or a different workflow. The core concepts of cache hit rates, prompt segments, and drift analysis will likely remain the same, but the way they are presented in the UI may evolve.
The Cache Explorer (🪟 Ctrl+Shift+P / 🍏 Cmd+Shift+P > Developer: Open Agent Debug Logs > Choose a session > Cache Explorer) helps you see these behaviors. It shows you how Copilot uses context across agent turns, how much of a request was reused from cache, and exactly where new input started to matter. It is a practical tool for understanding how agentic AI builds requests over time: which parts stay stable, which parts change, and how those changes affect token usage, latency, and overall cost.
Use Cache Explorer when you want to:
- understand how a conversation evolves across turns
- learn how prompt caching works in practice
- see why one turn reused more context than another
- spot which part of a request changed between turns
- find opportunities to reduce input token usage
- understand the effect of system instructions, tool definitions, tool results, and user follow-ups on cache reuse
Rather than only showing token counts, Cache Explorer helps you connect token usage to the actual structure of the request. That makes it useful both for diagnosing unexpectedly high token costs and for building intuition about how to work more efficiently and more economically with GitHub Copilot.
Anatomy of the Cache Explorer view
Let's first take a look at the different sections of the view and what they show. We'll go into how to interpret the information and take action on it in the next section.

Rail (left)

- Lists every model turn (one LLM API call) recorded for this session in chronological order.
- Turns are grouped by user prompt as a collapsible group. Each group header shows the user prompt text and the count of turns spawned.
- Each turn row shows:
- A title (e.g.
panel/editAgent,backgroundTodoAgent, ...) - Cache percentage ([cache ...%], it turns red below 90%)
- The duration of the turn in milliseconds
- The time at which the turn was initiated (local time)
- The model used for the turn (e.g.
claude-sonnet-4.6,gpt-4o-mini, ...)
- A title (e.g.
- Clicking a row selects it as CURRENT. The previous turn becomes PREVIOUS.
- When a session has turns from more than one agent, an agent filter dropdown appears at the top of the rail. It defaults to
panel/editAgent(the main chat panel loop), hiding background and utility calls (title generation, subagent coordination, todo-list maintenance). Those extra turns would otherwise clutter the rail and drag down the aggregate cache hit percentages despite carrying negligible token cost. You can toggle individual agents on or off; the rail and the Session cache health card both update to reflect only the visible turns.
Session cache health
The Session cache health card sits directly above the per-turn Summary cards and is visible on any turn after the first. It aggregates the whole session, up to and including the turn you have selected, and reports on patterns that recur across multiple pairs.
The card shows (not all of these are always visible; some only appear when triggered):
- Token-weighted overall hit rate:
...% overall cache hit. Per-request percentages can be misleading because small utility calls (title generation, short summaries) drag down the average despite their negligible token cost. Weighting by actual input-token count shows what the session actually cost. - Sub-line:
... of ... input tokens served from cache across ... requests (token-weighted). This is the rationale behind the headline percentage: how many tokens were cached vs how many total input tokens there were across the whole session. - Healthy pairs count:
... of ... request pairs healthy, plus~... tokens recomputed avoidablywhen there were avoidable breaks. A "pair" is two consecutive turns in the rail (Previous and Current). A "healthy pair" is one where the cache hit rate was above 90% and there were no avoidable break categories detected in the Components section. This count gives you a quick sense of how many turns had good cache reuse versus how many had issues that could be fixed. - Category chips: one chip per break category (model, tools, system, options, history, expiration) showing an occurrence count and the total tokens lost to that category. These chips let you see at a glance whether the session's losses came from a single root cause (e.g., tool-catalog churn from an MCP server that reloads between turns) or from scattered, unrelated events.
- Session findings: aggregated insights rendered as the same severity-coded rows used in the per-turn Summary:

The key distinction is that the per-turn Summary answers "what broke on this specific turn?", while Session cache health answers "what has been breaking repeatedly across the whole session?" A recurring-invalidator finding here tells you there is a systemic root cause worth chasing; a one-time break in the per-turn Summary usually is not.
Note that the report is scoped to the filtered turns in the rail. Focusing the agent filter on panel/editAgent therefore excludes background and utility calls from the health computation, giving a cleaner picture of your primary conversation loop.
Summary (top of the detail pane)

When you select a turn in the rail, the summary updates to show you a side-by-side comparison of the selected turn (CURRENT) with the immediately preceding turn (PREVIOUS). This comparison is the heart of the Cache Explorer: it surfaces exactly what changed between these two turns and how that affected cache reuse.
The summary usually consists of three side-by-side cards:
- PREVIOUS REQUEST: All the headline numbers the model turn that came before the selected turn
- REQUEST: Same, but then for the selected turn
- CACHE PERFORMANCE: A summary of cache effectiveness, based on the comparison between the previous and current turns
The CACHE PERFORMANCE card has four sections:
...% cache hit- Which percentage of the current turn's input tokens were served from cache.... of ... input tokens reused- The rationale behind the cache hit percentage: how many tokens were cached vs how many total input tokens there were.- Where the cache broke - Will explain the reason for the cache break. Possible explanations, and an example reason for it to occur, include:
- "System instructions changed [...]" - A switch in the model.
- "Tool definitions changed [...]" - Enabling/disabling an MCP server.
- "At
messages[...]-... message body changed (... chars)" - Re-editing/regenerating an earlier message in the thread. - "Request options changed [...]" - Reasoning effort differs from the previous turn. Options are part of the provider's cache key, so changing them mid-session invalidates the prefix.
- "No prefix divergence detected." - The prompt is byte-identical to the previous request but the cache hit rate dropped anyway. Either the provider's cache entry expired after inactivity (typically ~5 minutes TTL), or the prompt is below the provider's minimum cacheable prefix size (~1,024–4,096 tokens depending on model).
- Diff summary - counts of identical / in-place changed / added / dropped messages.
The very first turn in a session only shows the REQUEST and FIRST REQUEST IN SESSION with the cache hit percentage. There is no prompt signature, no components, and no diff, because there is nothing earlier in the session to compare against.
If the request options (explained later) changed, there is also a yellow inline banner: Options changed: model (claude-sonnet-4.6 → gpt-4o-mini-2024-07-18) [...].

Prompt Signature

The two horizontal bars (Previous, Current) are drawn proportionally to the character count. The system and tools segments indicate the portion of the prompt that comes from system instructions (different per model) and tool definitions (depends on user selection). Other segments can be:
user(blue) for the actual prompt the user sentassistant(green) for the assistant's previous messages, including tool calls and reasoning tracestool(yellow) for tool result messagestool_search(white) for tool-search output messagesdrift(red outline) for any segment that differs between Previous and Current
Chunk breakdown (collapsible table)

Below the summary line sits a collapsible Chunk breakdown toggle. This is a tabular representation of the same segment data shown in the bars above; it exists because the visual bars merge small messages together and hide exact numbers, while the table shows every segment of the prompt (system, tools, and each individual message) with precise counts. Rows where the chunk drifted between Previous and Current are highlighted (dark red).
Below the bars: ... of ... chars reused · break at ... (or ... · no divergence detected) is a summary of what can be seen in the bars.
Request Options

This table shows the request-level settings (model, reasoning effort, response format, etc.) side by side for the previous and current turns. Rows where the value changed between turns are highlighted with a colored border, making it immediately obvious when an option flip caused a cache break. The model row is always shown because switching models is the single most aggressive cache invalidator. If you see it highlighted, that explains the reset.
Components

For every component that drifted between A and B you get an item in this section with:
- A status badge that helps identify the reason for the drift:
identical,content drift,length change,only in A,only in B. - An inline character-level diff including a character count.
How to use this view to reduce token usage
The Cache Explorer is diagnostic. The actions you take based on it always reduce to one of three patterns.
Pattern A - the prefix changed (system, tools, options)
These are the cheapest to fix and the most damaging when broken: a change here invalidates the cache for every token of every turn after it. Anything below in the Components will then show as drift even if the user's prompt didn't change.
Signals in the view:
- Components:
systemortoolsshows a non-identicalbadge. - Request Options table has rows highlighted as
changed(especiallymodel). - The "Where the cache broke" line cites system / tools / options.
Actions:
- Don't switch models mid-session. A
modelchange always re-keys the cache. Pick a model at the start of a conversation and stay on it. - Stabilise custom instructions. Don't include timestamps, working directory absolute paths with timestamps, ticket numbers, randomised UUIDs, or any other "changes every turn" text in your
.github/copilot-instructions.md, agent instructions, or workspace prompt files. Make a section's content depend only on the workspace, not the clock. - Keep the tool catalog stable. Every time you enable/disable an MCP server, toggle a tool set, or install/uninstall an extension that contributes language-model tools mid-conversation, the
toolscomponent changes and you take a full prefix re-hash. If you're experimenting with MCP servers, do it between sessions, not between turns of one session. - Don't change reasoning effort, thinking budget, or response format per turn. All of these appear as Request Options rows.
- Don't switch between custom agents mid-session. An agent definition (
.agent.md) can inject its own instructions into the system prompt and restrict or extend the available tool catalog. Switching agents mid-conversation changes both thesystemandtoolsprefix segments simultaneously, causing a full cache invalidation.
Pattern B - a specific message changed
Signals in the view:
- "Where the cache broke" line cites
messages[...] - ... message body changed (... chars). - The Prompt Signature break marker lands inside the messages region, not at the very left.
- Components:
messages[...]showscontent drift/length change.
Note that messages[0] is usually the user's first prompt, so a break there for a fresh conversation is unavoidable. The interesting case is a break at messages[2], messages[4], ... deep in an already-long conversation. Common causes:
- History truncation (context-window management). When the conversation grows too long for the model's context window, VS Code drops or compacts earlier messages to make room. The Cache Explorer reports this as
only in Aat the truncation point with the insight: "History slicing or compaction shortens the prefix - the cache can only match up to the cut, and everything after it is recomputed." You'll typically see one or more messages disappear from the middle of the conversation rather than from the end. - Reasoning or preamble stripping. Between turns, VS Code may strip the assistant's extended thinking/reasoning content or drop preamble text from historical messages to save tokens. This changes the byte length of the assistant message at that position. The diff engine's insight explicitly calls this out: "A re-serialized assistant turn - trimmed whitespace, dropped reasoning or preamble text, reformatted tool calls - silently invalidates everything after it." Look for a
length changebadge on an assistant message whose inline diff shows reasoning content on the Previous side that is absent on the Current side. - Re-edited or regenerated earlier turn. Editing or regenerating an earlier message in the chat thread changes the bytes at that position and invalidates everything after it. Prefer appending a new follow-up to rewriting an earlier turn when you want to steer the agent.
- Tool-call re-serialization drift. Assistant tool-call messages include their arguments as serialized JSON in the cache key. If those arguments get re-serialized with a different key order or formatting between turns, the bytes change even though the content is semantically identical. The diff engine does not normalize key order, so any such instability shows up as
content drifton the assistant message that issued the call.
Pattern C - nothing changed but the hit rate still dropped
Signal:
- Headline:
X% cache hit - likely cache expiration. - "Where the cache broke" section says no prefix divergence detected.
Provider prompt caches have a TTL (typically a few minutes, provider-dependent). If you walk away from a chat for half an hour and come back, the session-specific cache for that conversation is gone, though the shared system/tools prefix may still be warm from other users' activity (see FAQ on cache lifetime). There is nothing in the diff to fix.
Action:
- Send follow-ups within the provider's cache window when possible.
- For long-running agentic tasks, prefer keeping the agent actively iterating over long pauses.
Conclusion
GitHub Copilot's shift to usage-based billing makes it important to keep an eye on the prompt cache as an important enabler for cost reduction. Model providers bill cached input tokens at up to ten times less than regular tokens, so a stable prefix across an agentic session is the biggest lever on cost. Three things are worth taking away:
- Cached input tokens are billed at a steep discount (up to 10x cheaper), so keeping the prompt prefix stable across turns cuts costs more than anything else.
- The most common cache breakers are model switches mid-session and tool or MCP server toggles between turns.
- The Cache Explorer in VS Code ensures you have the visibility into cache hit rates and prefix stability so you can optimize your usage.
Frequently Asked Questions
Written by
Fokko Veegens
Hi, I’m Fokko from Voorburg, where I live with my wonderful girlfriend and our two lovely daughters. I’m passionate about outdoor activities like cycling, sailing, skiing, and snowboarding, as well as working on classic cars. My favorite moments, though, are those spent with my family. As a DevOps consultant, I specialize in blending People, Process, and Products to streamline software delivery and ensure top quality. I love tackling bottlenecks, working closely with teams, and focusing on delivering exceptional value to end-users through DevOps best practices.
Our Ideas
Explore More Articles
Contact













