Liz Zhang

LLM Engineering

Your LLM Bill Is Too High. Here’s How to Fix It (Part 5)

By Liz Zhang · · 4 min read

← Back to blog

Stop paying full price for the same 20,000-token prefix over and over.

Previously in the series

Parts 1 through 4 covered the big ways teams accidentally burn money: calling a model when deterministic systems would work, defaulting to frontier models, ignoring cheaper processing tiers, and letting reasoning tokens run without a budget. The next lever is different. Prompt caching does not just trim a few tokens. Used correctly, it changes the economics of repeated agent workflows, long system prompts, tool definitions, policies, and examples.

Prompt caching is one of the most important LLM cost levers because it changes the economics of repeated work. Agents often send the same system prompt, tool definitions, policies, schemas, examples, and memory instructions repeatedly. Without caching, you pay to process those stable tokens again and again. With caching, the provider reuses the stable prefix and charges the cached-input rate.

That is not a small optimization. For long agent workflows, it can be the difference between a toy demo and an economically viable product.

The agent problem

Imagine an agent with a 40-step workflow, a 20K-token system prompt, and context that grows at each step. Without caching, that 20K stable prefix gets charged repeatedly. With caching, the system prompt and tools can hit the cached-input path.

The table below separates prompt content by how often it changes. The practical rule is simple: put rarely changing content first, keep it byte-for-byte stable, and push volatile content like request IDs and timestamps to the end.

A table is useful for planning, but the layout of the prompt is what actually determines cache reuse. Stable instructions, tools, and examples should be grouped before user-specific or retrieved content so the provider can match the prefix across requests.

In a multi-step agent, the impact compounds. The more steps that reuse the same stable prefix, the larger the gap becomes between paying full input cost every time and paying the cached-input rate on repeated content.

How it works on OpenAI

OpenAI applies prompt caching automatically to recent models when prompts are at least 1,024 tokens long. Cache hits require exact prefix matches, so static content such as instructions and examples should appear at the beginning of the prompt, while variable content should appear later. OpenAI also documents prompt_cache_key for improving routing when many requests share the same long prefix, and prompt_cache_retention for models that support extended retention. As of May 15, 2026, the standard OpenAI pricing table lists gpt-5.5 at $5.00 per 1M short-context input tokens and $0.50 per 1M cached short-context input tokens, while gpt-5.4 is $2.50 per 1M short-context input tokens and $0.25 per 1M cached short-context input tokens.

How it works on Anthropic

Anthropic uses cache_control. Current Claude API pricing documents two prompt caching modes: a 5-minute cache write at 1.25x the base input price and a 1-hour cache write at 2x the base input price. Cache reads cost 0.1x the base input price. Anthropic also now describes automatic caching as the recommended starting point for most use cases, with explicit cache breakpoints available when you need fine-grained control.

{ “type”: “text”, “text”: “Long stable system instructions, tools, policies, and examples...”, “cache_control”: { “type”: “ephemeral”, “ttl”: “5m” } }

The first write costs extra. The repeated reads are where the money is made.

Four caching rules that actually matter

  1. Use layered breakpoints. Cache global instructions separately from tools, tools separately from user context, and user context separately from retrieved context.

  2. Keep the prefix stable. Reordering sections, inserting timestamps, changing IDs, or adding dynamic content in the middle destroys cache reuse.

  3. Maintain cache locality. Do not randomly split identical workloads across routes if cache reuse matters.

  4. Do not blindly trust automatic caching. Explicit structure protects expensive stable prefixes when message history changes.

Bottom line

Prompt caching is not a minor trick. It is a structural cost lever. If your agent sends large repeated prompts and you are not designing for caching, you are overpaying by design.

Next in the series

Part 6 ties the whole system together. No-model paths, model routing, pricing lanes, reasoning budgets, and prompt caching only become powerful when they are measured as one workflow. The final post turns the series into a production playbook built around the metric that actually matters: cost per successful task.

References

OpenAI. (2026). Prompt caching. OpenAI API documentation. https://developers.openai.com/api/docs/guides/prompt-caching

OpenAI. (2026). Pricing. OpenAI API documentation. https://developers.openai.com/api/docs/pricing

Anthropic. (2026). Pricing. Claude API documentation. https://platform.claude.com/docs/en/about-claude/pricing