Agentic AI · Token Optimization Playbook · May 2026

Stop Burning Money on Invisible Tokens

18 technique groups across 6 lifecycle stages — all within your implementation control · any LLM · any provider

$8:$1

Compute vs. revenue per Anthropic user

40–60%

Of prod token budgets are pure waste

🎯 Signal per Token — Maximise signal per token, not the token window. Tokens are simultaneously: working memory · reasoning bandwidth · execution fuel · cost (output tokens cost 2–5× more than input). Budget them like CPU.

Stage 1 · Before the Request

Design time + governance

G1 · Prompt & System Prompt Design

Compress & Structure Prompts

15–40% input

Strip filler phrases; one-role-one-agent rule; lazy-load edge-case instructions only when needed

OSS: LLMLingua-2 · Selective Context · DSPy · Promptfoo

✅ Fully agnostic

G2 · Prompt Template Registry

Git-Backed Token Governance

Fleet-wide

Centralised registry with per-template token budgets enforced in CI/CD pipelines

OSS: Promptfoo · DSPy · Langfuse OSS

✅ Fully agnostic

G3 · Knowledge Strategy

RAG vs Fine-Tune + Doc Prep

Up to 100%

Preprocess docs at ingestion (strip HTML/boilerplate); decide RAG vs fine-tune per domain

OSS: Unstructured · Docling · MarkItDown · Apache Tika

⚠️ Fine-tune tooling is provider-specific

▼

Stage 2 · At the Gate

Request arrives — intercept before LLM call

G4 · Rules-Based Bypass

Skip the LLM Entirely

30–90%

Route deterministic requests (CRUD, lookups, eligibility checks) directly to backend APIs — zero tokens consumed

OSS: LiteLLM · Apache Camel · Bifrost

✅ Fully agnostic

G5 · Response & Step Caching

L1 Exact + L2 Semantic Cache

40–70%

L1 hash-based exact match; L2 vector similarity (≥0.88). Idempotent step cache for agent retry loops

OSS: GPTCache · Redis OSS · FAISS · Chroma · Temporal OSS

✅ Fully agnostic

G6 · Model Routing & Cascading

Cheapest Capable Model Wins

60–80%

Route 70% to cheap tier; escalate only when confidence <0.88. Static routing pre-call; cascading post-cheap-model

OSS: RouteLLM · LiteLLM

✅ Fully agnostic

▼

Stage 3 · Into the LLM

Context assembly — controlling what enters the prompt

G7 · Retrieval Optimisation

Inject Top-1, Not Top-5

50–80%

256–512 token chunks; reranker → inject top-1/2. Naïve RAG: 10K tokens. Optimised: ~400

OSS: LangChain · LlamaIndex · Haystack · pgvector · Qdrant

✅ Agnostic

G8 · Tool Definition Loading

Lazy Load + Prune Weekly

20–70%

Inject only tools relevant to current step. Audit: prune tools unused for 30+ days from agent context

OSS: LangGraph · MCP SDK · LangChain

✅ Agnostic

G9 · Context Schema & Handoffs

Compact Typed State Fields

20–70%

Use compact typed fields for inter-step state. Suppress reasoning traces from agent handoffs — pass answers only

OSS: LangGraph · Instructor · AutoGen

✅ Agnostic

G10 · Conversation & Memory

Sliding Window + Externalise

30–70%

Keep last 4–8 turns; summarise older turns with cheap model. Externalise agent state to Redis — never embed in prompt

OSS: Mem0 · Zep OSS · LangMem · LangChain4j

✅ Agnostic

▼

Stage 4 · Inside the LLM

Inference — controlling what the model generates

G11 · Output Length & Format

max_tokens + JSON Schema

30–60%

Always set max_tokens — treat it as mandatory. Return JSON not prose for machine-consumed outputs. Output costs 2–5× more

OSS: Instructor · Instructor Java · Guardrails AI

⚠️ Schema enforcement varies by provider

G12 · Reasoning Budget Control

Bound Chain-of-Thought

50–90%

Prompt-level: "answer only, no reasoning shown." Provider params where available (budget_tokens, reasoning_effort)

OSS: DSPy · Promptfoo

⚠️ Budget params are provider-specific

G13 · Batch & Compact Notation

TOON + Batch Bundling

20–98%

Bundle similar items into one call. TOON compact schema for repetitive data — up to 98% reduction vs verbose JSON

OSS: LiteLLM · Kafka · Ray · Temporal OSS · Instructor

✅ Fully agnostic

▼

Stage 5 · After the Response

Tool execution + result handling

G14 · Tool Call & Output Minimisation

Fewer Calls, Smaller Results

30–90% tool tokens

Combine related lookups into one coarse-grained tool. Parallelise inside the handler (not the loop). Project to only the 3–5 fields the agent uses — never return full 50-key objects

OSS: LangChain · LangGraph · LlamaIndex | Commercial: LangSmith · Portkey

✅ Fully agnostic

G15 · Server-Side Computation & MCP Offloading

Process in Code, Not in Context

40–80% data-heavy

Filter, sort, aggregate in your tool/MCP handler before results reach the LLM. Code execution costs zero tokens — processing raw data in-context is expensive and unnecessary

OSS: MCP SDK · LangGraph · Apache Camel | Commercial: Portkey · LangSmith

✅ Fully agnostic

▼

Stage 6 · Across the Loop

Orchestration + governance — runs continuously

G16 · Agent Architecture

Decompose + Async

20–60%

Break monolithic agents into specialised sub-agents (each carries only its own context). Async execution for I/O-heavy workflows

OSS: LangGraph · AutoGen · CrewAI OSS · Temporal OSS · Kafka

✅ Fully agnostic

G17 · Loop Control & Budget Propagation

Hard Limits + remaining_budget

10–30%

Set hard iteration caps. Stop when confidence drops. Pass remaining_budget token downstream so sub-agents self-constrain

OSS: LangGraph · AutoGen · LangChain

✅ Fully agnostic

G18 · Observability & Token FinOps

Measure Everything First

15–40% indirect

Tag every call (feature + team). Track ET metric + turn efficiency. Alert on budget overruns before the invoice arrives. Without this you optimise blind

OSS: Langfuse · OpenLLMetry · Helicone OSS · OpenTelemetry · Grafana

✅ Fully agnostic