Understanding Token Pricing: Prompt Caching, Context Windows & Cost Optimization
Large Language Models (LLMs) do not charge per second of computation; they bill strictly by tokens processed (input) and tokens generated (output). Because generation is fundamentally sequential (autoregressive decoding), providers charge 3x to 5x higher rates for output tokens.
The most significant cost revolution in modern generative AI is Prompt Caching. Pioneered by Anthropic and rapidly adopted by OpenAI and Google, caching allows applications to cache static tokens (system instructions, product catalogs, codebases) in GPU high-bandwidth memory. Instead of re-billing those tokens at full price on every user turn, providers discount cached reads by 50% to 90%, reducing application burn rates by hundreds or thousands of dollars monthly.
Benchmark Pricing per 1 Million Tokens (USD):
- Claude 3.5 Sonnet: $3.00 Input • $0.30 Cached Read (90% off) • $15.00 Output.
- OpenAI GPT-4o: $2.50 Input • $1.25 Cached Input (50% off) • $10.00 Output.
- Google Gemini 1.5 Pro: $1.25 Input • $0.3125 Cached Input (75% off) • $5.00 Output.
- Claude 3.5 Haiku: $0.80 Input • $0.08 Cached Read • $4.00 Output.
- OpenAI GPT-4o mini: $0.15 Input • $0.075 Cached Input • $0.60 Output.
- Google Gemini 1.5 Flash: $0.075 Input • $0.01875 Cached Input • $0.30 Output.
- DeepSeek-V3 (Open Inference): $0.14 Input • $0.28 Output.
Frequently Asked Questions
How do I enable prompt caching in my code? ▼
On Anthropic Claude, you simply add "cache_control": {"type": "ephemeral"} to your system prompt or large message blocks. On OpenAI, caching is applied automatically behind the scenes for requests sharing prefixes longer than 1,024 tokens. On Google Gemini, you use the Context Caching API to create explicit cache objects with a defined time-to-live (TTL).
What is the difference between batch processing and real-time APIs? ▼
Both OpenAI and Anthropic offer Batch API endpoints that provide an automatic 50% discount on all token costs in exchange for non-real-time completion within 24 hours. Batch processing is ideal for overnight data labeling, document evaluation, and bulk generation.
When should I run open-weight models instead of closed APIs? ▼
If you require strict HIPAA, financial data compliance, air-gapped on-premises privacy, or your traffic volume exceeds 50 million tokens daily with consistent throughput, renting dedicated cloud GPUs (e.g. H100s or A100s) to host Llama 3.3 or DeepSeek-V3 via vLLM becomes cheaper and provides complete data sovereignty.