We Added 3 Lines of Code. AI Costs Dropped 57%.

Every chat message on our platform cost $0.049. Not the complicated ones. All of them, including "what time is it?".

The fix was three settings in a model config, and it took the per-request cost to $0.021 and the response time from 13 seconds to 3. Nothing about the output changed. Here is the trace that found it and the arithmetic that made it obvious.

All figures below were measured on Claude Sonnet 4.6 in March 2026. The rates have moved since, and the shape of the problem has not.

What was actually costing $0.049 a request?

We had recently enriched our observability with per-trace metadata, which is the only reason this was findable at all:

With that in place, a single trace read like this:

1Input usage:   27,885 tokens
2  └── System prompt:        ~4,000 tokens
3  └── Company context:      ~2,500 tokens
4  └── Skill instructions:   ~1,500 tokens
5  └── Tool definitions:       ~500 tokens
6  └── Conversation history: ~19,000 tokens
7  └── User message:            ~20 tokens
8
9Output:        479 tokens
10
11Total cost:    $0.049026

The user's actual question was 20 tokens out of 27,885. Everything else was context, and the largest, most repetitive parts of it, the system prompt and the tool definitions, were byte-identical on every single request.

We were paying to have the same instructions read back to the model a thousand times a day. At that volume the bill came to roughly $1,470 a month.

What does prompt caching actually do?

It is a server-side optimisation, and the important part is what it does not do:

This is not prompt trimming. Nothing is removed and nothing is summarised. The model still sees everything you sent. Anthropic's servers simply skip re-processing a prefix they already hold.

The pricing is what makes the decision trivial:

The first request pays a 25% premium to write the cache. Every request after that takes 90% off the cached prefix. Break-even is request number two.

The fix, in three settings

We were running Pydantic AI, so this was three keys in AnthropicModelSettings:

1from pydantic_ai.models.anthropic import AnthropicModelSettings
2
3model_settings = AnthropicModelSettings(
4    # ... existing settings ...
5    anthropic_cache_instructions=True,       # cache the system prompt
6    anthropic_cache_tool_definitions=True,   # cache the tool definitions
7    anthropic_cache_messages=True,           # cache the conversation history
8)

Pydantic AI handles cache-point allocation itself. You get four cache points per request and these three settings spend three of them.

What each one is worth depends entirely on how stable that part of your payload is:

The cache carried a five-minute TTL that refreshed on every hit, which means an active conversation keeps it warm indefinitely.

What changed

Three consecutive messages in one conversation, before and after:

The usage breakdown on message three is where the mechanism shows:

1Input usage:           12,062
2  ├── input (standard):      6,036
3  ├── input_cache_read:      5,772  ← 90% discount
4  └── input_cache_creation:    254  ← new content
5
6Output:                   241
7Total:                 12,303
8
9Cost: $0.021723

Those 5,772 cached tokens would have cost $0.01732 at the standard rate. From cache they cost $0.00173. That single line is $0.016 a request.

The latency drop was the part we had not predicted. We went looking for a cost fix and got a product improvement, because the model was no longer re-reading 20,000 tokens of unchanged context before it started answering.

What we would tell you to check first

Look before you optimise. We only found this because the traces attributed cost per company, per request, at the token level. A monthly invoice would have told us the bill was high and nothing about why.

Assume your system prompt is the bill. In most AI products the user's message is a rounding error next to the instructions, tools and context wrapped around it. That is especially true of a brand-aware system, where the standing context describing how a company sounds is exactly the part that never changes: it is why one agent carrying a brand's voice has a large, stable prefix to cache in the first place. The question is not how to write shorter prompts, it is which parts of the payload never change.

There is nothing to weigh here. Caching does not alter output, memory or behaviour. If your payload has a stable prefix and you send more than one request per conversation, the only cost of not enabling it is the bill.

FAQ

Does prompt caching change the model's output?
No. The model receives the same tokens either way, so the response, the memory and the quality are identical. Only the server-side processing of the repeated prefix is skipped.

When does prompt caching start paying for itself?
On the second request. The write costs 25% more, each read costs 90% less.

What should you cache and what should you leave alone?
Cache the system prompt, the tool definitions and the stable prefix of the conversation. Leave the user's new message alone; it can only ever be a write.

How long does the cache stay warm?
It carried a five-minute TTL that refreshed on every hit when we measured this in March 2026, so an active conversation keeps it warm and a burst of traffic hours later pays the write again.