Enjoyed this article? See more similar articles in 
Pega Gen AI Cookbook - Recipes series
Token Economy: How AI Really Thinks, Costs, and Scales
The currency that powers every AI interaction — and why it matters
What they are · How they cost · How to optimize
Every word you send to an AI model, every response it returns, every instruction it follows — all of it is measured, processed, and billed in tokens. Understanding tokens isn’t just technical trivia. It is the foundation of building AI systems that are fast, reliable, and cost-effective at scale.
What Exactly Is a Token?
A token is the basic unit of text that a language model reads and generates. It is not a word, not a character, and not a sentence — it sits somewhere in between, determined by a process called tokenization.
Tokenization splits text into fragments that the model’s vocabulary recognizes. Common words become single tokens. Rare or complex words are split into multiple sub-word pieces. Spaces, punctuation, and casing all affect the count.
~4 chars per token
¾ of a word per token
~750 words per 1,000 tokens
2–4× more tokens for code
// Example tokenization
"Hello, how are you?" → 5 tokens
"cat" → 1 token
"unbelievable" → 4 tokens
"tokenization" → 4 tokens
{"key": "value", "n": 1} → 9 tokens // JSON is expensive
// Python function header → ~3× vs prose
Models like GPT-4 and Claude use Byte Pair Encoding (BPE) — a statistical method that builds a vocabulary of the most common character sequences across massive training corpora. The result is a vocabulary of roughly 50,000–100,000 tokens that covers most of human language efficiently.
Tokens are to AI what clock cycles are to a CPU — the atomic unit of computational work, measured relentlessly at every layer of the stack.
Why Tokens Are Expensive
Token cost is not arbitrary pricing. It reflects genuine computational expense at two levels: raw GPU compute and architectural complexity.
Raw Compute
For every single token, the model executes billions of floating-point operations across hundreds of layers of a neural network. Input tokens are processed to build a rich internal representation. Output tokens are generated one at a time — each requiring a full forward pass through the network. This is fundamentally different from reading a file or executing a query.
The Attention Problem — O(n²) Scaling
Transformer models — the architecture behind all major language models — compute relationships between every token and every other token in the context window. This is called self-attention, and it scales quadratically.
// Attention complexity scales quadratically with context length
100 tokens → 10,000 attention operations
1,000 tokens → 1,000,000 attention operations
10,000 tokens → 100,000,000 attention operations
// Double the context = quadruple the cost
// 10× the context = 100× the attention compute
This is why long-context requests are disproportionately expensive — and why context management is one of the highest-leverage optimizations available.
Input vs Output Tokens
Output tokens are typically 2–4× more expensive than input tokens. Generating requires sequential computation that cannot be parallelized; reading input can be processed in parallel. Understanding this distinction shapes prompt design significantly.
| Token Type | Relative Cost | Primary Driver |
|---|---|---|
| Input (prompt) | Medium | Parallel processing of context |
| Output (completion) | High | Sequential autoregressive generation |
| Cached input | Low | KV-cache hit, minimal recompute |
| Long context (>32k) | Very High | Quadratic attention overhead |
How Tokens Impact Operations
Token consumption affects every dimension of an AI-powered system — cost, speed, quality, and reliability.
Cost
Token spend compounds rapidly in production. A single careless system prompt of 2,000 tokens, called 10,000 times per day, adds 20 million tokens of daily input before a user types a single character. Multiplied across an organization, unmanaged token usage becomes a significant operational expense.
Latency
Longer contexts take longer to process. More output tokens take longer to generate. In user-facing applications, every unnecessary token adds perceptible latency. Systems demanding real-time response — chat, code completion, live agents — are acutely sensitive to token bloat.
Quality Degradation
Counter-intuitively, more tokens do not mean better results. Research consistently shows that models lose focus in very long contexts — critical information buried in the middle of a long prompt receives less attention than information near the beginning or end. Verbose prompts can actively degrade output quality.
Context Window Limits
Every model has a maximum context window. Exceeding it causes truncation — often silently — which can corrupt reasoning, lose critical instructions, or cause unpredictable behavior. In multi-step workflows, context accumulates; without active management, it will eventually overflow.
The Hidden Cost in AI Systems
The most expensive token patterns in production are rarely obvious: redundant system prompts on every call, full conversation histories passed to every agent, unstructured outputs that pad JSON with prose, and uncached static instructions re-processed millions of times.
These patterns are invisible in development and catastrophic at scale.
How to Optimize Token Usage
1. Prompt Compression
The highest-leverage optimization. Eliminate filler, formality, and redundancy from system prompts. Prefer structured formats — instructions as JSON or bullet points rather than explanatory prose. Say what the model should do, not what it is.
✗ "You are a highly capable AI assistant with expertise in
reviewing documents. Your task is to carefully read the
provided document and extract the key points in a clear
and organized manner."
✓ "Role: Document Reviewer.
Task: Extract key points.
Output: JSON array of strings. Max 10 items."
// Savings: ~60 tokens → ~18 tokens per call
2. Output Format Control
Verbose output is a silent cost multiplier. Specify exact output schemas. Use structured formats — JSON with defined fields, not prose narratives. Add explicit length constraints to your prompts.
✗ "Please provide a detailed analysis..."
✓ "Respond ONLY in JSON. Schema: {summary: string,
issues: string[], severity: 'low'|'med'|'high'}
Max 150 tokens."
3. Prompt Caching
Static content — system prompts, reference documents, few-shot examples — can be cached at the API level. Cached tokens cost roughly 90% less than uncached tokens. Structure prompts so static content comes first and dynamic content comes last to maximize cache hit rates.
// Structure for maximum cache efficiency
[STATIC — cache this]
System role, rules, output format, examples
Reference documents, knowledge base
[DYNAMIC — always fresh]
Current task, user input, variable context
4. Model Routing
Not every task requires the most capable — and most expensive — model. Route tasks by complexity: use lightweight models for classification, extraction, and formatting; reserve frontier models for reasoning, judgment, and generation that genuinely requires it.
| Task Type | Recommended Tier | Rationale |
|---|---|---|
| Classification / routing | Small | Binary or categorical, low reasoning |
| Extraction / formatting | Small | Pattern matching, structured output |
| Summarization | Mid | Moderate reasoning, bounded output |
| Code generation | Mid | Structured, verifiable output |
| Complex reasoning | Large | Requires deep inference chains |
| Judgment / critique | Large | Nuance, calibration, edge cases |
5. Context Pruning
In multi-turn conversations and agent workflows, context accumulates. Rather than passing full histories, maintain rolling summaries — replace completed steps with compressed representations. Pass each component only the context it needs, not the entire thread.
✗ Agent receives: Full conversation + all prior outputs
+ complete plan + original task
✓ Agent receives: Its specific step
+ 3-line summary of prior context
+ its required dependencies only
6. Memory Over Context
The most scalable optimization for long-running systems: replace growing context windows with external memory. Store outcomes, facts, and preferences in a retrieval layer. At inference time, retrieve only what is relevant. This keeps every call’s context small and focused, regardless of how much history exists.
Dos and Don’ts
✓ Do
-
Use structured output formats (JSON, XML)
-
Cache static system prompts aggressively
-
Specify exact output length and schema
-
Route tasks to appropriately-sized models
-
Summarize context rather than append it
-
Measure token usage per call in production
-
Put static content before dynamic in prompts
-
Use external memory for long-running systems
-
Compress few-shot examples to minimum needed
-
Set explicit max_tokens on every API call
✗ Don’t
-
Pass full conversation history to every call
-
Use prose instructions when structure works
-
Ask for “detailed” output without bounds
-
Use frontier models for simple tasks
-
Repeat the same context in every agent call
-
Ignore token counts during development
-
Assume more tokens equals better results
-
Let context windows grow without pruning
-
Include pleasantries or filler in system prompts
-
Leave max_tokens unset in production
The discipline of token optimization is the discipline of precision thinking — knowing exactly what information a model needs, giving it nothing more, and trusting it to work within that constraint.
Where to Draw the Line on Tokens
There is no universal magic number for prompt or output length. The optimal token count is task-specific — but it is always findable, and always measurable. The goal is to reach the point where every token present earns its place, and removing any one of them would cost quality.
A prompt is optimized not when you can add nothing more, but when you can remove nothing further without losing fidelity.
The Three Lines That Matter
Line 1 — Minimum Viable Token (MVT). The fewest tokens that produce acceptable quality output. Find it by compressing until quality breaks, then stepping back one level. Everything above this line is headroom. Most first-draft prompts sit 40–60% above their MVT.
Line 2 — The Quality Plateau. The point at which adding more tokens stops improving output. Beyond this plateau, additional tokens add cost and latency with no quality return — and in long contexts, they dilute model attention, actively degrading results. The plateau is almost always lower than engineers expect.
Line 3 — The Degradation Point. For very long contexts, quality does not just plateau — it drops. The lost-in-the-middle effect is well-documented: models attend more strongly to content at the beginning and end of a context window. Critical instructions buried in the middle receive less weight. This means extremely long prompts can perform worse than compact ones.
// The quality curve — every prompt has this shape
Quality
│ plateau ────────── degradation ↘
│ /
│ /
│ /
│ /
│ /
└───────────────────────────────────────── Tokens
↑ ↑ ↑
MVT Plateau Degradation
(your floor) (your target) (your ceiling)
Benchmark Ranges by Task Type
These are practical ranges observed across production AI systems. They are starting points for calibration, not hard rules. Your optimal will sit somewhere within the range — use the sweep method to find exactly where.
| Task Type | Input Range | Output Range | Red Flag If Over |
|---|---|---|---|
| Classification / routing | 50–150 tokens | 1–20 tokens | 300 input / 50 output |
| Extraction / formatting | 100–250 tokens | 50–200 tokens | 500 input / 500 output |
| Summarization | 200–500 tokens | 100–400 tokens | 1,000 input / 800 output |
| Code generation | 150–400 tokens | 200–1,000 tokens | 800 input / 3,000 output |
| Reasoning / analysis | 200–600 tokens | 200–800 tokens | 1,200 input / 2,000 output |
| Agent orchestration | 400–1,500 tokens | 100–500 tokens | 3,000 input / 1,000 output |
| RAG / document tasks | 1,000–8,000 tokens | 200–600 tokens | 20,000 input / 1,500 output |
How to Find Your Optimal Line — The Sweep Method
Run your prompt at four compression levels against a representative sample input. Score output quality at each level. Plot where quality flattens. The lowest compression level that still achieves plateau-level quality is your optimal line.
Step 1 — Baseline
Run at 100% length. Record tokens + quality score.
Step 2 — Compress to 75%
Remove filler, pleasantries, verbose role descriptions.
Run. Score. Compare to baseline.
Step 3 — Compress to 50%
Keep only core role + task + output format.
Run. Score. Compare.
Step 4 — Compress to 25%
Role + single task sentence only.
Run. Score. Compare.
Step 5 — Find the cliff
The level just before quality drops = your optimal line.
Everything above that level = compressible waste.
Identifying an Optimized Prompt — The 5 Signals
| Signal | What to Check | Healthy Target |
|---|---|---|
| Token Efficiency Ratio | Quality score ÷ input token count | Higher ratio at lower token count = optimized |
| Output Consistency | Run same prompt 10 times — variance in structure | 85%+ similar structure across runs |
| No Output Padding | Useful content tokens ÷ total output tokens | Signal ratio above 0.85 |
| No Model Guessing | Output contains “I assume” / “It’s unclear” | Zero ambiguity phrases — prompt is complete |
| Cache Hit Rate | % of input tokens served from cache in production | 80%+ means prompt structure is well-separated |
The One Hard Rule
Output tokens should never routinely hit your max_tokens limit. If they do, either your limit is too low and you are silently truncating — or your prompt is not controlling output tightly enough. Either case is a production bug. Set max_tokens explicitly on every call, and monitor it as a health metric.
The Practical Rule of Thumb
Most first-draft prompts are 40–60% compressible without any quality loss. The tokens that survive compression are your true optimal floor. If you cannot justify why a specific token is in your prompt — remove it. Prove it was needed by watching quality drop without it.
Optimization Priority Matrix
| Technique | Impact | Effort | When to Apply |
|---|---|---|---|
| Prompt caching | Very High | Low | Any repeated static system prompts |
| Output format control | High | Low | All production prompts |
| Prompt compression | High | Medium | All prompts before production |
| Model routing | High | Medium | Multi-agent or high-volume systems |
| Context pruning | Medium | Medium | Multi-turn conversations, agents |
| External memory | Very High | High | Long-running or stateful systems |
| Rolling summaries | Medium | Medium | Complex multi-step workflows |
Prompt Design, Token Limits & Model Selection
Benchmarks and frameworks are useful. But in the heat of building, you need fast heuristics — rules you can apply in seconds without running a sweep. These are the ones that hold up consistently across task types, models, and production environments.
Prompt Design
The 3-Part Prompt Rule
Every effective prompt has exactly three parts: Role (who the model is), Task (what to do right now), and Output (exactly how to respond). If your prompt has more than these three elements, each additional element needs a clear justification. If it cannot justify its token cost — cut it.
Role · Task · Output — everything else is overhead.
// Prompt anatomy — the 3-part rule
ROLE → 1–2 sentences max. What the model is.
Not a biography. Not a capabilities list.
TASK → 1–3 sentences. What to do with this specific input.
Imperative voice. No hedging. No pleasantries.
OUTPUT → Format + schema + length constraint.
JSON schema, max tokens, field names.
If you do not specify it, the model invents it.
// Total target: under 200 tokens for most tasks
// Every token above 200 needs explicit justification
| Prompt Element | Rule of Thumb | Common Mistake |
|---|---|---|
| Role description | 1 sentence — title + domain | Writing a paragraph of capabilities |
| Instructions | Imperative voice, no filler words | “Please make sure to always…” |
| Examples (few-shot) | 1–3 max; only when task is ambiguous | Adding examples for every simple task |
| Output spec | Always include schema + max length | Asking for “a good response” with no bounds |
| Constraints | State what NOT to do only when critical | Long lists of negative constraints |
| Context | Only what this call uniquely needs | Repeating background on every call |
| Total length | Under 300 tokens for most tasks | 500+ token system prompts without audit |
Setting Token Limits
Token limits are not a safety net — they are a specification. Setting max_tokens forces the model to be concise and prevents runaway output costs. The rule is simple: set it to 1.3× your expected output length. This gives a 30% buffer for variation without allowing unconstrained generation.
// Token limit formula
max_tokens = expected_output_tokens × 1.3
// Examples
Classification → expected: 5 tokens → max_tokens: 10
JSON summary → expected: 120 tokens → max_tokens: 160
Code snippet → expected: 300 tokens → max_tokens: 400
Analysis → expected: 500 tokens → max_tokens: 650
// Warning: output routinely hitting max_tokens
// = limit too low OR prompt lacks output control
| Scenario | Rule | Why |
|---|---|---|
| Output hits limit regularly | Raise limit OR tighten output spec | Silent truncation corrupts results |
| Output is far below limit | Lower limit to expected × 1.3 | Unused headroom signals no real constraint |
| Variable output length tasks | Set limit by worst-case, not average | Occasional truncation is still a bug |
| Structured JSON output | Estimate schema size + 20% buffer | JSON must close correctly or parsing breaks |
| Chain-of-thought tasks | Set higher limit; filter reasoning from final output | Reasoning tokens are needed but not returned |
Choosing the Right Model
Model selection is a cost-quality trade-off, not a prestige choice. The rule is to use the smallest model that reliably achieves the required quality threshold for a given task. Larger models are slower, more expensive, and often overkill for structured or deterministic tasks.
The Model Selection Decision Tree
Is the output deterministic and structured? (classify, extract, reformat) → Use a small model.
Does it require moderate reasoning? (summarize, translate, generate from template) → Use a mid model.
Does it require judgment, nuance, or long reasoning chains? (critique, plan, complex code, multi-step analysis) → Use a large model.
Is it customer-facing and high-stakes? → Use the largest model regardless of task simplicity.
// Model tier decision — fast heuristic
yes/no, category, label, slot-fill → SMALL
rewrite, translate, short generate → SMALL–MID
summarize, moderate code, FAQ → MID
reasoning, analysis, long code → MID–LARGE
critique, strategy, ambiguous intent → LARGE
customer-facing + high stakes → LARGE always
// Cost ratio (approx): Small 1× · Mid 5× · Large 15–30×
// Routing classification to Large = 15–30× unnecessary cost
| Decision Factor | Points to Small model | Points to Large model |
|---|---|---|
| Output type | Label, category, structured JSON | Prose, reasoning, judgment |
| Correctness verifiability | Can be auto-verified | Requires human or model evaluation |
| Ambiguity tolerance | Task is fully specified | Task requires interpretation |
| Volume | High volume (>10k calls/day) | Low volume, high value per call |
| Latency requirement | Real-time (<500ms) | Async or batch acceptable |
| Failure cost | Low — easily retried or caught | High — customer-facing or irreversible |
| Context length needed | Under 2,000 tokens | Over 8,000 tokens |
The goal is not to use the best model. The goal is to use the right model — and to know the difference between the two for every task in your system.
Enjoyed this article? See more similar articles in Pega Cookbook - Recipes series