LLM Engineering
Your LLM Bill Is Too High. Here’s How to Fix It (Part 6)
Route, cache, measure, and escalate deliberately.
Previously in the series
The first five parts each attacked one piece of the LLM cost problem. We removed unnecessary model calls, turned model choice into a routing discipline, used pricing modes before rewriting the app, capped reasoning tokens, and structured prompts so caching can actually work. This final post pulls those ideas into one production playbook: how to route, measure, cache, escalate, and audit an LLM system without letting the expensive path become the default path.
LLM cost optimization is not one trick. It is a system design discipline. The teams that cut costs meaningfully do not simply switch models or shorten prompts. They redesign the workflow so cheap paths absorb most traffic and expensive models handle only the cases that deserve them.
The mental model is a pyramid. The bottom is deterministic code, cheap models, cached prompts, Batch jobs, and routine paths. The middle is mid-tier models and fallback paths. The tiny top is uncached frontier reasoning.
The healthy cost pyramid
Table 1 turns the pyramid into an operating target. It separates cheap default paths from managed model paths and expensive exception paths, so the team can decide where most production traffic should land before debating individual model prices.
Figure 1 shows the same idea visually. A production-ready system should be wide at the cheap and cached layers, narrow at the frontier layer, and explicit about when traffic is allowed to climb upward.
Measure before you optimize
A cost playbook fails if it only tracks monthly spend. The trace has to explain why a route was chosen, which tokens were billed, whether the user task succeeded, and whether retries or tools quietly erased the savings.
Route by complexity and confidence
Once you accept that not every query deserves your best model, routing becomes obvious. Simple requests go to cheap paths. Hard, ambiguous, or risky requests escalate. The important part is to make escalation conditional instead of letting the strongest model become the default.
Table 2 gives a concrete ticket-triage example. The default route favors rules, libraries, parsers, small classifiers, and cheap drafting models. The escalation column defines the exceptions: low confidence, missing fields, legal risk, refunds, security issues, or other cases where a bad answer costs more than the model bill.
Measure cost per successful task
Monthly spend is not a useful engineering metric by itself. Cost optimization starts with traces. The metric that matters is cost per successful task, because a cheap request that fails and gets retried three times is not actually cheap.
Table 3 lists the minimum fields worth logging. Together, they show whether the system is routing correctly, whether cache hits are visible, whether reasoning tokens are controlled, and whether the workflow is getting cheaper without damaging quality.
The full checklist
-
Can this task be handled without an LLM?
-
What is the cheapest model that passes the eval?
-
Can this run in Batch, Flex, or another lower-priority processing mode?
-
Are reasoning or thinking tokens capped and logged?
-
Does the router escalate only low-confidence or high-risk cases?
-
Are cached input tokens visible in traces?
-
Is stable prompt content placed before dynamic content?
-
Are tool definitions, schemas, system prompts, and reusable context cacheable?
-
Are you measuring cost per successful task, not just cost per token?
-
Are provider prices, discounts, and context-window thresholds stored as configuration so they can be updated without code rewrites?
Bottom line
The APIs for cost-efficient LLM systems already exist. The hard part is forcing the architecture to admit that most requests are boring. Once you do that, the bill drops.
Where to go from here
This closes the series, but it should not close the work. The useful next step is an audit: pick one real workflow, map every model call, mark the default route, measure cache hits, log reasoning tokens, and calculate cost per successful task. That is where the theory becomes leverage. Once you see the expensive path clearly, it becomes much harder to let it stay the default.