Token Efficiency in Production: Lessons from a Multi-Agent Platform

In Why We Built a Native Agentic Harness for Data, our CTO and co-founder, Harsha Chintalapani made the case that the model is only one component of an agent. Around it sits a harness that manages context, decides which tools are available, executes the work, observes the result, and gives the model a chance to correct itself. A good harness is selective. It gives the model what the current task needs and leaves the rest out.

Organizations pointing AI agents at their data spend more tokens than the work requires. Without a context platform, an agent rediscovers schemas and business definitions each time it answers a question, and the team running it pays for that search in tokens, database trips, and hallucinations. Collate's agents start from the context our platform maintains: metadata, business meaning, and memory. The savings in this post came from changes to how the harness moves that context. We did not fine-tune a model to get them. Take the platform away and the same agents would spend more per turn. A later post in this series measures that gap.

Harsha's post explained why we built a harness for data. This one covers what it costs to run one. A production agent spends tokens on much more than its final answer. It routes work, passes context to specialized workers, reads tool results, repairs SQL, and revisits earlier conversation turns. Each step has a purpose. As the system grows, the way those steps share information can become a significant part of its operating cost.

Over August and September, Collate's engineering team traced those steps through Collate's AI Platform in production. Models, frameworks, and product capabilities are evolving quickly, and production traces gave us a more precise understanding of how context moves through our system. The main August changes improved reuse across planner stages and the Analytics Agent calls, reduced the context copied into workers, and stopped SQL repair from responding to scenarios that weren't recoverable. Later work strengthened tool-result boundaries and made caching more specific to the workload. Mean raw tokens per turn fell 68% and recorded spend per turn fell 67%, while output per turn barely moved. The savings came from how the harness moved context.

The strongest measured change in our August sample was a reduction in repeated cache writes. The broader lesson was that token efficiency needs to be an observable property of the whole execution path.

What we measured

A single token percentage hides most of what matters. We report the four categories separately so other engineering teams can test the same mechanisms against their own traces.

• We focused on samples spanning just a few days, covering +400 conversations and +1,000 turns, alongside the implementation changes made up until the time of writing this post.

Definitions:

• Raw tokens are uncached input tokens plus output tokens, cache-write tokens, and cache-read tokens. A turn can contain many model calls, so these figures measure aggregate model traffic across the turn, not a single request.

• Recorded USD is the model-provider charge captured in production traces under the pricing configuration used for this analysis. It excludes tool, warehouse, infrastructure, and observability costs and is not total platform operating cost.

• All figures are ratios of summed daily totals, not averages of daily ratios.

A turn can contain many model calls. Raw tokens include uncached input, output, cache reads, and cache writes.

• Comparing the results of two weeks, mean raw tokens per turn fell 68%, and recorded USD per turn fell 67%.

• The largest proportional reduction was in cache writes per turn: 864,000K → 179,000K, down 79% because the agents stopped wrongly storing the same context over and over. Output tokens per turn stayed close to 13,000K in both samples, meaning that the agents wrote as much as before.

• The main August changes improved reuse across planner stages and the Analytics Agent calls, reduced the context copied into workers, and stopped SQL repair from responding to scenarios that weren’t recoverable. Later work strengthened tool-result boundaries and made caching more specific to the workload.

-Recorded spend per conversation fell from $7.57 to $2.92. That is a separate cohort-level measure, not the per-turn figure multiplied by a fixed number of turns, because the two samples had different conversation shapes.

If you’re a data team sizing its own agent program, that last row is the one to pay attention to: our agents, at nearly unchanged output volume per turn, cost less than half as much per conversation once context reuse was designed in.

One note on scope. These figures measure the running cost of Collate's own agents. They are not the savings a customer's agents get from being grounded in Collate. That is a different measurement, and it is the subject of Part 2.

Over the next few lines, we’ll go over some of our lessons learnt.

Cache economics depend on reuse

The first useful change was to separate the token categories in our analysis. A cache write and a cache read have very different costs. For those teams starting out their cost analysis, make sure you have the necessary observability in place first. In our case, Koog AI and OTel make it quite easy to keep tabs of where time and money is going.

For the Claude cache pricing configuration relevant to this investigation, the input multipliers were:

Note that these are model-specific pricing assumptions; newer models and other providers can differ (Claude pricing documentation).

A write only pays off when the entry is reused within its TTL (time to live-how long something stays stored before it expires). For a five-minute entry, one write followed by one read costs 1.35 input-equivalent units, compared with 2.0 for two uncached requests. A write with no subsequent read costs 1.25 units for work that would otherwise have cost 1.0, with no reuse benefit at all..

That gave us three useful diagnostics:

1. The ratio of cache-read tokens to cache-write tokens,

2. The number of calls that write without reading,

3. And the share of model cost attributable to writes.

The point is to see the patterns and react to it. . Usage patterns differ from platform to platform, and in our case they also differ customer by customer. . Being able to quickly change configurations and control the TTLs of system context and conversation history independently lets us change configuration quickly based on alerts and observations.

To give a specific example, using the default 5min cache write, an 8.8-minute gap caused roughly 800K tokens to be rewritten, illustrating why user cadence matters to that choice.

Organize prompts by sharing scope

Our planner performs several kinds of work: routing, checking the result, deciding on remediation, and synthesizing the answer. Each stage had specialized instructions, but much of its context was the same.

Prefix caching makes the placement of those instructions consequential. The cache covers the content leading up to its boundary; changing an earlier block can invalidate reuse of later blocks. Basically, order matters, and when designing dynamic and multi-agent systems (meaning loading agents and their tools based on input request), how you organize the puzzle pieces has a cost.

We first moved shared request context ahead of agent-specific instructions. We then gave the six planner-side call paths one byte-identical system prompt and placed each stage's role instructions in its final task message. This kept the stable prefix and conversation history reusable while letting each call retain its own output contract.

The diagnostic evidence was concrete: earlier planner paths rewrote roughly 390–410K history tokens per call. After the change, gate, remediation, and synthesis agents could read the router's cache entry with zero cache writes in the validated trace.

We also removed incidental variability from cached regions. Again, dynamism and user-based context configurations adds complexity for the internal system, and debugging practices do not always go hand in hand with cost savings! In our case, when users update their AI Persona Context, we set a generated_at flag aimed to help us find the root of possible behavioral changes from the agent. However, we were not stripping this timestamp from the context the agent received. The cached prefix differed on every request, which eliminated reuse for that segment. The lesson learnt here? You’re not just building a system for engineers, you’re building a system for agents, and sometimes their needs are not the same.

The resulting rule is precise: put content into a cached segment according to the calls that can reuse it.

Give workers the context their task needs

Business context is central to Collate. A metric definition, an entity identifier, or a company-specific rule can determine whether an answer is useful. As AI Persona Context becomes richer (and heavier), distributing the entire document to every worker gets increasingly expensive.

In one measured case, the curated AI Persona Context document was approximately 500K characters, estimated at roughly 125K tokens. Its size depends on what users put into it., The planner needs the full document to decide what matters; most workers do not. So we changed who owns that context.

The planner reads the full curated document. When delegating, it copies the relevant definitions, expressions, and identifiers verbatim into the worker's task. Workers exclude the full document by default.

This creates a clear responsibility for the planner: it must author a task that contains the information the worker needs. The benefit is smaller worker prompts and less sensitivity to unrelated changes in a large persona document.

The SQL pipeline needed a similar treatment. Generation, repair, exploration, results analysis, and rewriting all used substantial schema and profile context. We built that context once per attempt and passed the same strings through a common segmented prompt builder. Shared schema comes first; invocation-specific context follows; the failed query, error, and current task come later.

Subsequent work applied the same rule to user preferences and permission-dependent samples. Those belong after the shared schema boundary, so a small change in personal context does not require a new write of the entire schema.

Recovery should match the failure

Recovery is valuable when a generated query can be repaired. It becomes expensive when the failure happens before the query reaches the warehouse or any scenario that’s beyond the Analytics Agent control.

An August diagnostic sample showed that two messages accounted for 47% of the credits across 18 messages while repeatedly encountering an unavailable Query Runner. One message consumed 44 model calls without returning rows. The repair path treated the connection failure as a SQL execution error and attempted to regenerate the statement. In a deterministic system, that misclassification is a minor bug. On an agentic system, the exception type alone does not distinguish a transport failure from a SQL failure a model could repair, and every retry is a paid model call.

We classified these infrastructure failures separately and prevented them from entering SQL-fix retries. We also memorized repeated schema work and exploration within a job, including an exploration that found nothing.

The distinction is useful across agent systems: a recovery call should have a plausible way to change the outcome. Transport recovery and SQL repair solve different problems, and need different signals.

Later work addressed another source of unnecessary tool calls. Workers needed a valid way to report that the task lacked required context. We aligned the worker prompts with the graph's existing text-only completion path, so a worker could return its structured finding instead of producing an invalid call to finish. A targeted evaluation reported a reduction from 32.7% to 0% in such calls for that scenario.

Apply optimizations to the workload that benefits

As we extended the work, several changes reinforced the need for workload-specific measurements:

The automation result is a replay estimate, separate from the August production comparison. It explains the scope of the implementation: longer loops can read earlier writes repeatedly, while short loops may end before the last write is ever reused.

Making the lessons durable

We turned the resulting constraints into decision records and regression checks: stable cached prefixes, identical planner system prompts, bounded cache-marker counts, shared SQL context, model-facing tool-result assertions, and a common path for one-off calls. Every important decision is now recorded by an architecture decision record (ADR) that the agent checks, so we can keep evolving without falling into the same gaps twice.

The data shows a substantial reduction in tokens and recorded spend per turn, concentrated in cache writes, alongside nearly unchanged output-token volume. It supports the direction of the work without establishing equal answer quality or assigning an exact share of savings to each change.

As agent capabilities expand, the useful engineering questions become increasingly specific: which context must this stage see, which parts will another call reuse, what can this retry repair, and what bytes are actually sent? Production traces helped us answer those questions, and the resulting contracts give us a way to keep answering them as the stack evolves.

Next: what Collate’s context saves your agents

This blog measured the cost of running our own agents and our learnings from operating our platform in production for our customers. Part 3 in this series: measures the other side: how Context that grounds AI queries through Collate improves the token efficiency of an organization's agents, compared with running the same agents without that grounding.