Static vs Dynamic Prompt Caching

[RameshSangili]

Enjoyed this article? See more similar articles in Pega GenAI Cookbook - Recipes series

Static vs Dynamic Prompt Caching

What changes every call, what never changes, and how that distinction saves 90% of your input token cost in production.

Every prompt has two parts that behave completely differently at runtime. One part is identical on every single call — the instructions, the schema, the rules. The other part is unique to each call — the case data, the customer, the live inputs. Treating them the same way is the most common and most expensive mistake in production AI systems.

The Fundamental Distinction

When you send a prompt to a language model, the API receives two things: a system prompt containing your instructions, and a user message containing the actual input to process. These two blocks serve entirely different purposes — and they have entirely different caching characteristics.

The test is simple. Ask of every sentence in your prompt: would this sentence be different on the next call? If no — it is static. If yes — it is dynamic.

Static — never changes

• Role and task definition

• Evaluation criteria and rules

• Output schema and field definitions

• Word count and format constraints

• Null state instructions

• Validation rules

Identical whether processing Case E-00001 or Case E-99999. Same customer or different customer. Same branch or different branch.

Dynamic — changes every call

• Current case ID and CustomerID

• Related case IDs and dates

• Case related information (differs from Case to Case)

• Any live data from your system

Unique to every customer, case, and moment in time. Cannot be cached — and should not be.

The output is dynamic. The instructions are not. Output variance comes from different input data — not from different instructions. That distinction is what makes caching work.

Two Layers of Caching

Caching for prompts happens at two independent layers — and understanding both is essential for getting the full benefit.

Layer	How it works	Who controls it	What it saves
Pega layer	Static prompt stored as a literal string in Agent Rule. Never fetched, never computed.	You — set once, reference everywhere	Zero fetch cost, zero latency to retrieve
Model layer (API)	API caches the system prompt by content fingerprint after first call. Subsequent calls with identical content pay ~10% of normal token cost.	Anthropic — automatic, no configuration	~90% of static input token cost from call 2 onward

The model layer cache is a cost concern — the API recognises the identical content and charges a fraction of the normal rate. Both activate automatically when the static prompt never changes between calls.

The critical rule

Model layer caching works on a content fingerprint. If even one character changes — a space, a punctuation mark, a version tag — the fingerprint changes, the cache is invalidated, and you pay full price again. Store the static prompt in exactly one place in Pega and reference it everywhere. Never build it dynamically. Never append version strings at runtime.

What Happens Across 5 New Conversations

This is the question that trips most engineers: if each conversation is new, does the cache reset each time? No — and this is the most important thing to understand about prompt caching.

Caching is not per conversation, per session, or per Pega case. It is per API key plus prompt content. The cache lives at the model layer across all calls from your API key, regardless of which conversation, which user, or which case triggered them.

Note: The token costs shown in the timeline below (e.g. “~19 tokens”) are illustrative estimates based on a ~90% cache discount. Actual cached token rates depend on your provider, model tier, and whether caching is explicitly configured. Always verify current rates in your provider’s pricing documentation.

Conv 1

CACHE MISS First ever call — cache cold. Full processing. Cache now warm. Prompt A

193 tokens

Conv 2

CACHE HIT Same prompt — served from cache. New conversation, same result. Prompt A

~19 tokens

Conv 3

CACHE HIT Same prompt — served from cache. Prompt A

~19 tokens

Conv 4

CACHE HIT Same prompt — served from cache. Prompt A

~19 tokens

Conv 5

CACHE HIT Same prompt — served from cache. Prompt A

~19 tokens

All five conversations are new. All five send the full static prompt. From the second call onward, subsequent calls may be charged at a significantly reduced rate — depending on your provider, whether caching is active, and whether your prompt meets the minimum token threshold. The timeline above illustrates the pattern; actual cost reduction will vary by provider and configuration.

What Happens When the Prompt Changes at Conv 4

Understanding cache invalidation is as important as understanding cache hits. When Conversation 4 sends a different prompt — even one character different — the API sees a new content fingerprint it has never encountered. The old cache for Prompt A is not deleted; it simply becomes unreachable for new calls. A new cache entry for Prompt B is created.

Conv 1

CACHE MISS First call. Cache cold. Prompt A processed in full. Cache warmed. Prompt A

193 tokens

Conv 2

CACHE HIT Same Prompt A — served from cache. Prompt A

~19 tokens

Conv 3

CACHE HIT Same Prompt A — served from cache. Prompt A

~19 tokens

Prompt changed — Conv 4 sends Prompt B

---

Conv 4

MISS + REWARM New fingerprint. Full cost paid. New Prompt B cache created. Prompt B

193 tokens

Conv 5

CACHE HIT Prompt B hits its new cache. Prompt B

~19 tokens

Conv 6+

CACHE HIT Prompt B cache hit indefinitely. Prompt B

~19 tokens

You pay full price exactly once per unique prompt version — the cache warm-up call. Every call after that is ~10% until you change the prompt again. Prompt A’s cache entry expires quietly after roughly 5 minutes of inactivity. You do not need to manage it.

The Rule That Makes This Work

One prompt change = one cache miss = one full-price call. That is acceptable. What is not acceptable is accidental invalidation — whitespace differences between dev and prod environments, version strings appended at runtime, or the same prompt defined in multiple places that drift out of sync.

Store the static prompt in exactly one place. Reference it everywhere else. Treat it like a versioned constant — change it deliberately, infrequently, and with awareness that change = one miss.

The static prompt (system parameter — paste as literal)

Analyse inflight fraud cases for the same customer (CustomerID) to detect fraud patterns — not duplicates. Evaluate all related cases on: suspicious transaction behavior, coercion or pressure indicators, requested financial action, repeated attempts across locations, account takeover signals, and branch-to-branch or channel-to-channel escalation patterns. Use Create Date/Time and Location to describe patterns. Exclude current case from all comparisons. Never recommend merging into current case. Output ONLY valid JSON using no quotes on keys. No prose, no HTML, no markdown, no extra keys. Schema: {related_cases: [{case_id: F-#####, created: MM/DD/YYYY HH:MM AM/PM TZ, location: verbatim, pattern_classification: Same pattern or Different incident or Unclear, time_location_signal: max 8 words, key_pattern_indicator: max 6 words}], recommendation: {action: MERGE CURRENT INTO F-##### or KEEP CURRENT SEPARATE, reason: max 12 words, merge_justification: [signal 1, signal 2], merge_items_from_current: [element 1, element 2]}}. Rules: related_cases max 5 items. pattern_classification must be exactly one of: Same pattern, Different incident, Unclear. merge_justification 2 to 3 items present only if action is MERGE. merge_items_from_current 2 to 6 items present only if action is MERGE. reason omit if action is KEEP CURRENT SEPARATE. created format MM/DD/YYYY 12-hour time in incident-location timezone. case_id format F-##### only.

The dynamic user message (built at runtime) - Knowledge sources

Current case: E-12345 | Customer ID: ABC123456789

Related inflight reports:
Case ID: E-11111 | Created: 05/10/2025 09:14 AM CT | Location: Dallas Branch
  | Description: Customer requested wire transfer | Detail: Accompanied by unknown male, appeared nervous
Case ID: E-11222 | Created: 05/10/2025 02:47 PM CT | Location: Fort Worth Branch
  | Description: Same customer attempted wire transfer | Detail: Same unknown male present, same destination account

What This Saves at Scale

The numbers compound quickly. At modest production volume, the difference between cached and uncached static prompts is significant — not because the per-call saving is large, but because it applies to every single call.

Volume	Without caching	With caching	Monthly saving	Cache hit rate
1,000 calls/day	12.9M tokens/mo	7.5M tokens/mo	~5.4M tokens	~90%
10,000 calls/day	129M tokens/mo	72M tokens/mo	~57M tokens	~90%
100,000 calls/day	1.29B tokens/mo	720M tokens/mo	~570M tokens	~90%

These figures assume ~193 static tokens and ~300 dynamic tokens average per call. The dynamic portion is never cached — it is the actual work. The static portion after the first call costs approximately 19 tokens instead of 193 on providers where caching is active. That 174-token saving per call, multiplied by volume, is the total monthly saving shown above.

Disclaimer — Illustrative Figures

The savings figures in this table are illustrative estimates only. Actual savings depend on several factors that vary by provider, model, and configuration:

Caching must be active. On Claude, prompt caching requires an explicit cache_control flag and a minimum token threshold (1,024–4,096 tokens depending on model). Prompts below the threshold are not cached and receive no discount. On OpenAI and Gemini 2.5+, caching may be automatic but subject to internal thresholds and TTL policies.

Discount rates are provider-specific. Cache hit discounts vary by provider and model tier. Published figures range from ~50% to ~90% depending on who you ask and when. These rates are subject to change without notice.

Always verify current pricing, thresholds, and caching policies directly in your provider’s official documentation before making production cost estimates.

The real value of the static/dynamic split is not just cost. It is correctness. When your instructions live in one place, they cannot drift. When they cannot drift, your outputs do not silently degrade between deployments.

The Five Rules That Make Caching Work

Rule	What it means in practice
One source of truth	Store the static prompt in exactly on the Agent rule.
Never build static content dynamically	If you concatenate anything into the system prompt at runtime — a date, a version, a flag — that content changes between calls. The cache breaks. Static means genuinely static.
Static first, dynamic last	The API caches the prefix of the combined prompt. Static content in the system parameter, dynamic content in the user message. This structure maximises the cacheable portion.
One miss per version	Every prompt change costs exactly one full-price call to rewarm the cache. Accept this. It is not a problem — it is a known, bounded cost.
Same across environments	Dev and prod must use the identical static string. If they differ by even whitespace, they warm separate caches. Use a shared constant, not environment-specific values.

Prompt Caching Across LLM Providers

Prompt caching is not unique to Claude. OpenAI and Google Gemini both support it — but each provider implements it differently in activation, cost structure, discount depth, and cache lifetime. Understanding the differences matters when choosing a model for a production system like Pega.

Side-by-side comparison

Factor	Claude (Anthropic)	OpenAI (GPT-4o+)	Gemini (Google 1.5+)
Activation	Manual — explicit cache_control tag required	Automatic — zero setup	Explicit (1.5+) or implicit/auto (2.5+)
Cache hit cost	~10% of normal	50% of normal	25% of normal
Write penalty	+25% on first call	None	None
Cache TTL	~5 min inactivity	5–10 min / up to 24h extended	1 hr explicit / 3–5 min implicit
Minimum tokens	1,024 tokens	1,024 tokens	1,024 (Flash) to 4,096 (Pro)
Storage fee	None	None (in-memory)	Yes — per token per hour
Speed benefit	Up to 85% faster TTFT	Moderate	Moderate
Best for	High frequency, deep savings	Simplicity, longer TTL	Batch processing

What the write penalty actually means

Claude is the only provider that charges a write penalty — 25% extra on the first call that creates a cache entry. The API must serialize the model’s internal processed state (KV tensors from the attention layers) and store them in memory for rapid retrieval. That work has a real cost. But the penalty pays for itself within a single follow-up call.

// Write penalty break-even — Claude at $3 per 1M tokens
// Example: 1,000 token static prompt

Normal cost per call    = 1,000 x $0.000003        = $0.00300

Call 1 — cache WRITE    = 1,000 x $0.000003 x 1.25 = $0.00375  ← costs MORE
Call 2 — cache HIT      = 1,000 x $0.000003 x 0.10 = $0.00030  ← saving from here
Call 3+ — cache HIT     = 1,000 x $0.000003 x 0.10 = $0.00030

// 3 calls WITHOUT caching:  $0.00900
// 3 calls WITH caching:     $0.00435  — 52% saving across 3 calls
// Break-even: achieved at call 2. Penalty irrelevant from call 3 onward.

Why the 1,024 Token Floor Exists

Caching very short prompts costs more in cache management overhead than it saves in processing. The 1,024 token minimum is the point where caching becomes economically rational for the provider. Below that floor, the overhead of serializing and storing the KV tensors outweighs the token processing cost saved on subsequent calls.

This means aggressive prompt compression — which is always the right starting point — can actually push a prompt below the caching threshold. For very short optimised prompts, Pega-layer caching is the only layer available.

Choosing the right provider based on your caching pattern

Pattern	Best provider	Reason
High frequency, same prompt, long-running system	Claude	90% hit discount compounds massively. Write penalty paid once and irrelevant from call 2.
Simplicity is the priority	OpenAI	Zero setup, automatic, 50% off. No penalty, no configuration needed.
Batch processing, hours-long cache needed	Gemini explicit	1-hour TTL fits batch windows. 75% discount. Storage fee acceptable at batch scale.
Prompt changes frequently (daily or weekly)	OpenAI	No write penalty means each change costs nothing extra. Claude’s write fee adds up if prompt changes often.
Latency is critical, real-time user-facing	Claude	Up to 85% faster time to first token on cache hits — highest speed benefit of the three.

The concept of static versus dynamic is universal across every LLM provider. What differs is how you tell the model which part to cache, how much it costs, and how long it stays warm. The architectural discipline — one source of truth, static first, dynamic last — applies regardless of which model you use.

Enjoyed this article? See more similar articles in Pega Cookbook - Recipes series