When you build an MVP with LLM APIs, paying $0.03 per request feels negligible. But as soon as your app grows — adding large system prompts, extensive agent tool declarations, and 5-chunk RAG retrieval to every prompt — your token consumption explodes linearly. At scale, an AI app processing 100k daily requests can easily burn $15,000/month on token overhead alone. In this article, I'll walk through the exact three-part caching and routing architecture we deployed to cut our LLM monthly spend by 70% while improving average response latency from 2.4 seconds down to 380 milliseconds.
TL;DR — 80% of token consumption in RAG and agent applications comes from repeating static system prompts and retrieved context. Caching at the provider and semantic layer turns linear cost curves sub-linear.
Technique 1: Anthropic & OpenAI Provider-Level Prompt Caching
Both Anthropic (Claude) and OpenAI support prompt caching, which allows static prefixes of your prompts — such as large system instructions, tool definitions, or background documents — to be cached on their infrastructure at up to a 90% discount on input tokens.
// anthropic-cache.ts — Setting ephemeral cache breakpoints
import Anthropic from '@anthropic-ai/sdk';
const anthropic = new Anthropic();
export async function queryClaudeWithCachedSystemPrompt(userQuery: string, systemDocs: string) {
const response = await anthropic.messages.create({
model: 'claude-3-5-sonnet-20241022',
max_tokens: 1024,
// Add cache_control to heavy system instructions
system: [
{
type: 'text',
text: `You are an expert AI software architect. Reference documentation:\n${systemDocs}`,
cache_control: { type: 'ephemeral' } // Caches system prompt for 5 minutes
}
],
messages: [
{ role: 'user', content: userQuery }
]
});
// Check cache hit performance metrics in production logs
console.log('Tokens:', {
input: response.usage.input_tokens,
cachedRead: (response.usage as any).cache_read_input_tokens || 0,
cachedCreation: (response.usage as any).cache_creation_input_tokens || 0
});
return response.content[0];
}By marking static system instructions and RAG references with ephemeral cache headers, repeated queries hit the provider's memory cache. Input token costs drop from $3.00 per million tokens to $0.30 per million tokens.
Technique 2: Semantic Vector Caching with Redis
Traditional exact-string key-value caches fail for AI apps because user questions are phrased differently ('How do I reset my password?' vs 'Where can I change my password?'). Semantic caching computes an embedding vector for incoming queries and checks if a semantically equivalent query was answered recently within a cosine similarity threshold (e.g. > 0.94).
// semantic-cache.ts — Query similarity caching with Redis Vector Search
import { createClient } from 'redis';
import { OpenAIEmbeddings } from '@langchain/openai';
const redis = createClient({ url: process.env.REDIS_URL });
const embeddings = new OpenAIEmbeddings();
export async function checkSemanticCache(userQuery: string): Promise<string | null> {
const queryEmbedding = await embeddings.embedQuery(userQuery);
// Search Redis HNSW Vector Index for similar past queries
const searchResult = await redis.ft.search(
'idx:semantic_cache',
`*=>[KNN 1 @vector $BLOB AS score]`,
{
PARAMS: { BLOB: Buffer.from(new Float32Array(queryEmbedding).buffer) },
SORTBY: 'score',
RETURN: ['response', 'score']
}
);
if (searchResult.total > 0) {
const topMatch = searchResult.documents[0].value;
const similarity = 1 - parseFloat(topMatch.score as string);
// If similarity > 94%, serve instantly from cache (0 LLM token cost)
if (similarity >= 0.94) {
return topMatch.response as string;
}
}
return null; // Cache miss -> proceed to LLM call
}Technique 3: Smart Tiered Model Routing
Not every user query requires Claude 3.5 Sonnet or GPT-4o. Simple task classification (e.g. intent detection, simple formatting, binary status checks) can be handled by Claude 3.5 Haiku, GPT-4o-mini, or a fast local model running on vLLM.
// model-router.ts — Dynamic complexity routing
export function routeQueryToModel(query: string, metadata: { requiresReasoning?: boolean }) {
const isSimpleIntent = query.length < 60 && !metadata.requiresReasoning;
if (isSimpleIntent) {
// 15x cheaper & 4x faster model for lightweight queries
return { model: 'claude-3-5-haiku-20241022', temperature: 0.1 };
}
// High-capability model reserved for complex architectural tasks
return { model: 'claude-3-5-sonnet-20241022', temperature: 0.3 };
}Production Impact & Results
After implementing this three-tier optimization strategy across our client backend deployments, here were the key performance metrics:
- Semantic Cache Hit Rate: ~32% of incoming user queries served directly from Redis (< 30ms latency, $0 token cost).
- Prompt Cache Hit Rate: 68% of remaining LLM calls hit Anthropic's ephemeral prompt cache (90% input token discount).
- Model Routing Savings: 40% of queries safely downgraded to Haiku / 4o-mini without quality loss.
- Overall Results: Monthly LLM spend dropped by 71.4%, and average end-to-end response latency decreased from 2,450ms to 380ms.
Frequently Asked Questions
Doesn't semantic caching risk serving outdated answers?
Set TTLs (Time-To-Live) on cached responses (e.g. 24-48 hours) and invalidate cache keys whenever underlying workspace data or documentation is updated.
How long does Anthropic prompt caching persist?
Anthropic prompt caches are ephemeral and refresh automatically for 5 minutes after every request hit. For active applications, the cache remains hot continuously.
What embedding model works best for semantic caching?
OpenAI text-embedding-3-small or fast local HuggingFace sentence-transformers (e.g. all-MiniLM-L6-v2) provide fast vector generation under 15ms.
Continue reading

Why "Vibe Coding" Feels Like Flying Until You Hit Production: A Brutally Honest Field Guide for 2026

Why Insurtech Still Runs on Java: What Building US Retirement Engines Taught Me About Architecture, Rust, and Why India Got It Right
Get updates directly in your Google Search feed
Add this publication as a preferred source to highlight articles in Top Stories and AI Overviews.
