How to Reduce LLM Token Costs in Your AI Agent Framework

Last month I checked my OpenAI bill and nearly choked on my coffee. $847. Not for a team. Not for a startup. Just me, running an AI agent framework that handles a handful of daily automation tasks. The agent was doing its job. The budget? It wasn’t.

That’s when I went down the token optimization rabbit hole. And here’s the thing nobody tells you upfront: most AI agent frameworks burn through 40-60% more tokens than they actually need. The waste isn’t in the model choice or the API pricing tier. It’s in the architecture. Fix that, and you’ll cut your bill in half before touching anything else.

Prompt Caching: The Single Biggest Lever

If you take one thing from this article, make it this. Both OpenAI and Anthropic now offer prompt caching that can slash your input token costs by up to 90%. Ninety percent. Not a typo.

OpenAI’s implementation is automatic. Any prompt hitting 1,024 tokens or more gets cached, and subsequent requests sharing the same prefix get routed to warmed-up machines. No code changes needed. Just structure your prompts so static parts — system instructions, tool definitions, few-shot examples — sit at the beginning, with dynamic user content at the end. Cache hits appear as cached_tokens in your usage stats. Latency drops up to 80% on top of the cost savings.

Anthropic gives you more control with explicit cache_control breakpoints, plus automatic caching via {"type": "ephemeral"} at the top level. Cache hits on Claude cost 90% less than base input. Default cache lifetime is 5 minutes, renewable with each use. A 1-hour extended duration is available at a small premium.

Real numbers from my setup: My agent framework sends roughly 80 requests daily across four pipelines. Before caching, daily input token cost averaged $12.40. After restructuring? $3.10. That’s a 75% drop. The system prompts were identical text across nearly every call, re-processed from scratch every single time.

Put your system prompt first. Keep it identical across requests. That’s the whole trick.

Context Window Management: Less Is More

Here’s a counterintuitive truth. Bigger context windows don’t produce better results. They often produce worse ones.

Anthropic’s engineering team calls it “context rot.” As token count grows, recall accuracy decreases. It’s a gradient, not a cliff. Every new token dilutes the model’s attention across n² pairwise relationships. Like reading a 400-page manual to answer a one-sentence question. You’ll get there. But you’ll waste a lot of energy.

Think of it as an “attention budget.” Every token depletes it. The goal isn’t more context. It’s the minimal set of high-signal tokens that gets the output you want.

What this means for your framework:

• Trim system prompts ruthlessly. If a sentence doesn’t guide behavior, cut it.
• Don’t dump full conversation histories into every turn. Summarize.
• Pass back only the relevant parts of tool outputs — not the full raw response.
• Use XML tags or Markdown headers to separate sections. Models parse structure faster.

I cut my system prompt from 2,800 tokens to 1,100. Same behavior. 60% fewer tokens per request. Multiply across 80 daily calls.

Model Routing: Don’t Use a Sledgehammer for a Thumbtack

This one feels obvious once you hear it. But almost nobody does it.

Not every task your agent handles needs GPT-5 or Claude Opus 4. Simple classification? A cheap model works fine. Entity extraction? Same thing. Summarizing a paragraph? You don’t need a model that costs $15 per million input tokens. Claude Haiku costs roughly 90% less than Opus. GPT-4o-mini is a fraction of GPT-5’s price. And for many tasks in an agent pipeline, the output quality difference is negligible.

So build a router. Classify each incoming task by complexity before picking a model. Here’s a simple taxonomy that works:

• Tier 1 (cheapest): Text classification, entity extraction, sentiment analysis, simple yes/no decisions, data formatting. Use Claude Haiku, GPT-4o-mini, or Gemini Flash.
• Tier 2 (mid-range): Summarization, content generation, moderate reasoning, tool selection. Use Claude Sonnet or GPT-4o.
• Tier 3 (premium): Complex multi-step reasoning, code generation with dependencies, ambiguous problem-solving, final output polish. Use Claude Opus or GPT-5.

In my framework, about 60% of all agent calls now go through Tier 1 models. Another 25% through Tier 2. Only 15% reach Tier 3. The cost distribution flipped from “everything at premium” to “premium only when it genuinely matters.” Total bill dropped another 40% on top of the caching savings.

Batching: The Underrated Optimization

Most agent frameworks process tasks one at a time. But many agent tasks are independent — or can be grouped.

OpenAI’s batch API gives you 50% off for non-real-time work with a 24-hour turnaround. Anthropic matches that. But batching isn’t just for the formal API. Within a single turn, combine multiple tool outputs into one prompt. Instead of three separate calls to summarize A, summarize B, then compare both — do it all in one. Fewer round trips. Less duplicated context. Lower cost.

Compression and Truncation Strategies

Anthropic now ships server-side compaction as a beta feature. You enable it with a context_management block, and when your conversation approaches the token limit, Claude automatically summarizes older context into a concise compaction block and continues from there. No client-side summarization code needed. It’s the cleanest solution for long-running agent sessions that accumulate massive context.

If you’re building your own, here’s what works:

• Sliding window with summary. Keep the last N messages in full. Everything older gets compressed into a running summary that occupies maybe 500 tokens.
• Selective truncation. Drop tool outputs that are no longer relevant. Drop thinking blocks (unless you’re using a model that needs them preserved).
• Token-aware message pruning. Before each API call, count your tokens. If you’re above 70% of the context window, prune the oldest messages first.
• Chunked RAG. Instead of stuffing entire documents into context, retrieve only the most relevant chunks. Cheaper and often more accurate.

My agent framework now caps conversation context at 32K tokens even though the models support 200K. Quality stays consistent. Costs stay predictable. And I’ve never once had an agent lose the plot mid-conversation.

Before and After: The Numbers

Let me give you the raw comparison from my own framework. Same agent tasks. Same volume. Same output quality — verified through manual spot-checking over two weeks.

• Daily token consumption: 1.2M input + 180K output → 340K input + 140K output (71% reduction in input, 22% reduction in output)
• Daily cost: $18.60 → $5.20 (72% reduction)
• Monthly cost: ~$558 → ~$156
• Average latency per request: 2.8s → 1.1s
• Cache hit rate: 0% → 64%

That’s $400 a month back in my pocket. For an individual developer running personal agents, that’s the difference between “this is a fun hobby” and “I need to shut this thing down.”

Tools and Frameworks That Help

A few tools worth your attention:

• LiteLLM — open-source proxy handling model routing, cost tracking, and fallbacks across 100+ LLM providers.
• LangSmith — shows exactly which calls burn the most tokens and where cache hit rates are low.
• Helicone — LLM monitoring with token-level analytics. Generous free tier.
• Portkey — AI gateway with built-in prompt caching and model fallback logic.
• OpenAI usage dashboard — underrated. Shows your cached_tokens ratio over time.

Practical Checklist

Here’s your Monday morning action plan. Takes about two hours. Saves you money forever.

1. Restructure every system prompt. Static content at the top. Dynamic content at the bottom. Identical prefix across all requests of the same type. Do this first. It’s the biggest win.
2. Enable prompt caching. For OpenAI, it’s automatic above 1,024 tokens — just verify it’s working in your usage dashboard. For Anthropic, add cache_control: {"type": "ephemeral"} to your requests.
3. Trim system prompts. Cut 40-60%. Your agent won’t miss the fluff. Test behavior on a small sample to confirm.
4. Build a model router. Start with three tiers. Route simple tasks to cheap models. Track accuracy. Adjust thresholds.
5. Cap your context window. Set a hard limit at 32K or 64K tokens. Implement sliding-window summarization for anything beyond that.
6. Batch non-real-time work. Use the batch API for tasks that can wait up to 24 hours. That 50% discount compounds fast.
7. Add token counting. Log tokens per request. Set up alerts when a single call exceeds 10K input tokens. You’ll be shocked how often it happens without you noticing.
8. Review weekly. Spend 15 minutes every Monday looking at your usage dashboards. Caching metrics, model distribution, outliers. Adjust and repeat.

The Bottom Line

Token optimization isn’t glamorous. It’s not the part of AI agent development anyone puts in a tweet. But it’s the difference between a framework that costs you dinner money and one that costs you rent. The techniques are straightforward. The tools exist. The only missing piece is doing it.

Start with caching. Then trim. Then route. The order matters, and so does actually checking your bill. Which, honestly, I still hate doing. But at least now I don’t choke on my coffee.