Token Optimization
Playbook

Every token in every system prompt is billed on every call. Three compounding levers:

Compression and trimming

Remove filler phrases ("As an AI…", "Please note…"), collapse whitespace, eliminate boilerplate. For automated compression: LLMLingua-2 (Microsoft Research, MIT) runs as a Docker HTTP sidecar — call via HTTP from Java or Go before any LLM request. Reduces prompts 50–80% preserving semantic accuracy. Deploy in your AI Gateway, not individual service logic. Compress static prompts at build time; dynamic prompts at runtime. Do not compress user inputs.

Few-shot → Zero-shot

Each example adds 50–200 tokens per call. Audit all prompts with 3+ examples; replace with precise task description + JSON output schema. A/B test on 200 representative samples. If accuracy drops, add back one example only. ⚠️ Cheaper model tiers may still need 1–2 examples — test each tier separately.

System Prompt Architecture (role isolation + lazy instructions)

Three failure modes inflate system prompts fleet-wide:

• Role stacking: one agent carrying multiple roles → one-role-one-agent (see G16)
• Instruction duplication: every agent repeating shared safety/format rules → extract to a shared base layer; agent-specific content layered on top
• Always-on edge instructions: escalation paths, refusal language relevant to ~5% of calls but loaded 100% of the time → tag with intents; inject conditionally

Establish a layered composition model: base → role → task → dynamic context. Each layer independently versioned and token-budgeted (G2).

All other groups fix token waste reactively. This group prevents it by making token budgets a first-class property of every prompt template, enforced in CI/CD.

Each template stored in Git (YAML/JSON) with mandatory fields:

template_id: customer-support-classifier
version: 3.2.1
owner: platform-team
token_budget:
  system_prompt_max: 400
  total_input_max: 800
  output_max: 150
current_token_counts:
  system_prompt: 387    # measured at build time

CI/CD enforcement: pre-commit hook renders each template with representative inputs, counts tokens, fails the pipeline if budget exceeded. Token count history tracked per version — regressions visible in PR diff. Increasing a budget requires documented justification. Templates not accessed in 30 days flagged for deprecation.

Count tokens using your provider's tokenizer endpoint. For Go: tiktoken-go gives reasonable BPE approximations for GPT-family models only. ⚠️ tiktoken and tiktoken-go are OpenAI-specific — they will produce incorrect counts for Gemini and other non-BPE models. For a provider-agnostic fallback, use a character-based estimate (~4 chars ≈ 1 token) as a ceiling check before calling the provider endpoint.

Document Preprocessing

Strip content that tokenises expensively but carries no value: HTML tags, headers/footers, legal boilerplate, base64 data. Process at ingestion — cost paid once, not on every retrieval.

Raw doc → Extract text → Strip boilerplate → Collapse whitespace
        → Tables to CSV → If >4,000t → summarise with cheap model
        → Store clean text in vector DB

Apache Tika: ⚠️ Java Maven dependency. Unstructured OSS, Docling, MarkItDown: Python-based, available as Docker HTTP sidecars for Java/Go.

RAG vs. Fine-Tune Decision

Every RAG call injects 400–3,000 retrieval tokens per query. For stable, high-frequency domain knowledge (product taxonomy, internal policies, support playbooks), fine-tuning bakes knowledge into weights — reduced retrieval tokens per call permanently, though Fine-tuning is not a reliable replacement for knowledge retrieval and can lead to stale knowledge.

Break-even: for a domain receiving 1M queries/month with 1,000 retrieval tokens per cache-miss query, even a 50% cache hit rate from G5 still yields 500M retrieval tokens/month saved via fine-tuning. ⚠️ Re-train when knowledge changes. ⚠️ Keep RAG fallback for out-of-distribution queries.

Not every request needs an LLM. Route deterministic requests — order lookups, eligibility checks, CRUD, FAQ matches, structured data queries — directly to backend services before the LLM call. Zero tokens spent.

Patterns

• Lightweight rules engine or regex classifier identifies deterministic intents → backend API
• Invoke LLM only when rules engine cannot handle with sufficient confidence
• Database-first resolution before reaching for the LLM

Audit LLM call logs first — categorise by type, identify what is deterministic. Do not over-specify rules — incorrectly bypassing the LLM costs more in quality than it saves in tokens.

For Java: Apache Camel ⚠️ for custom routing logic.

31% of enterprise LLM queries are semantically similar to previous ones.

Response Caching (L1 + L2)

• L1 Exact-match: hash(normalised prompt) → Redis lookup. Sub-millisecond.
• L2 Semantic: embed query → vector similarity search against past queries. Threshold: 0.88–0.92.

Tiered strategy: L1 → L2 → L3 (LLM call + store result). Normalise before hashing. TTL: FAQ answers → days; real-time data → no cache.

Step-Level Idempotent Cache

Individual workflow steps treated as pure functions — results memoised by hash(step_name + step_inputs + template_version). On retry/re-run, unchanged steps skipped — no LLM call. High value for: workflows that fail partway, evaluation pipelines, batch pipelines sharing upstream steps.

Temporal provides deterministic workflow replay: completed Activity results are stored in workflow history and reused during replay, avoiding re-execution of already completed Activities. Activities themselves use at-least-once execution semantics and should be designed to be idempotent when side effects are involved.

For Java: Jedis/Lettuce for Redis. For Go: go-redis.

Two complementary strategies — use both:

Static Routing (pre-classify then route)

Classifier categorises each request before any model call → cheapest capable model.

RouteLLM (ICLR 2025): achieves 95% of GPT-4 quality routing only 14–26% of traffic to frontier models, though these results are benchmark-dependent and workload-specific

Cascading (live quality decision)

Send every request to cheap model first. If it returns {answer, confidence ≥ 0.88} → return immediately. Escalate only on low confidence. In practice 70–80% of queries never need escalation.

⚠️ 0.88 is a starting point only — model-generated confidence is not statistically calibrated. Validate against ground truth on your own workload before deploying; the right threshold varies significantly by task type.

Start with word-count + keyword heuristic classifier (zero latency). LiteLLM acts as provider-agnostic proxy — routing logic lives in the proxy, your code calls one endpoint.

Naive RAG retrieves top-5 large chunks (~10,000 context tokens/call). Optimised RAG delivers top-1 or top-2 (~400–800 tokens). Two compounding levers:

Chunk size, top-k, reranking

• Chunk size: 256–512 tokens, not 1,000–2,000
• Retrieve top-3 → rerank → inject top-1 or top-2
• Similarity threshold > 0.85 — pass "no relevant context found" if nothing qualifies
• Cross-encoder reranker is far cheaper than sending all chunks to the LLM
• Just-in-Time retrieval: fetch exactly when needed, not upfront in bulk
• NOTE : Optimal chunk size, top-k, and re-ranking thresholds are corpus-specific and require evaluation

Hybrid Dense + Sparse Retrieval (RRF fusion)

RRF score: 1/(k + rank_dense) + 1/(k + rank_sparse) where k=60. Start with alpha=0.5, tune on your dataset.

• Vespa / Qdrant: native hybrid (HNSW + BM25) via REST API
• pgvector: combine <=> with tsvector in SQL — no new infra if Postgres is running
• Weaviate: native hybrid with tunable alpha

Pre-compute all embeddings and sparse indexes at ingestion — never at query time. For Java: LangChain4j ⚠️ for orchestration.

All tool definitions injected into every system prompt by default. A large tool registry adds 2,000–5,000 tokens to every single LLM call. Inject only tools relevant to the current step.

Patterns

• Intent-based tool selection — classify task → inject relevant subset
• Step-aware injection — LangGraph per-node tool scoping
• MCP lazy loading — on-demand tool manifest exposure
• Automated pruning — cross-reference manifest against call logs weekly; flag any tool not called in 30 days

⚠️ Requires tools tagged with intent/category metadata in advance.

The architect's choice of how information flows between steps and agents is made once but affects every token for the system's lifetime. Prose accumulates 3–5× larger than structured context.

"Customer John Smith called about
order 1234 placed March 15th,
shipped 7 days ago without
confirmation. Requested escalation."

cust=C99|order=1234
|placed=2026-03-15
|status=shipped|days=7
|action=escalate

Inter-Agent Message Rules

• Typed fields only in state schema — no free-form text blobs
• Explicit field scoping — if one node needs it, pass as node-local variable, not shared state
• Reasoning suppression — consuming agent needs outcome only, not producing agent's reasoning
• Hint fields — include next_action upstream to reduce downstream re-derivation (saves an extra LLM call)

Design LangGraph TypedDict collaboratively before implementation — state field proliferation is hard to prune retroactively. Use Instructor to enforce typed LLM output at message boundaries.

Three compounding patterns preventing context from growing unbounded:

Sliding Window + Rolling Summary

A 20-turn conversation accumulates 15,000+ tokens/call. Fix: keep last 4–8 turns verbatim; summarise older turns with cheap model once; prepend as a single context message. One summarisation call recovers in 2–3 subsequent calls of avoided history tokens.

State Externalisation

Naive agents embed full state in system prompt every call. By step 7 of a 10-step workflow: 5,000+ tokens/call. Fix: store state externally; inject only instruction + minimal_context for the current step.

[Orchestrator] <--> [State Store (Redis / any KV)]
      │
  [LLM API]  — sees only current-step context (200–500 tokens)

Agent Knowledge Base (SKILLS.md pattern)

Knowledge-heavy agents embed all procedures in the system prompt permanently (1,000–3,000+ tokens/call). Fix: index skills as RAG chunks with metadata tags; retrieve only relevant skills using hybrid retrieval (G7).

Before: [Role: 500t] + [All skills: 2,000t] + [Tools: 800t] = 3,300 tokens/call
After:  [Role: 500t] + [Retrieved skills: ~300t] + [Tools: 800t] = ~1,600 tokens/call

Skills: atomic chunks (one skill = one chunk), tagged with category and conditions. Version in Git. Store history/state in Redis — never in-process memory.

Output tokens cost 2–5× more than input (model/provider dependent). Two always-on levers:

• Always set max_tokens — mandatory on every call. Without it, models return 3–10× more than needed. Start at 2× expected output, tune down.
• JSON schema instead of prose for any response parsed by code — eliminates preambles, disclaimers, padding.

Extended reasoning models generate large reasoning volumes billed at output token rates — often dwarfing the actual response.

Prompt-Level (fully agnostic)

• "Answer in 2 sentences or fewer."
• "Respond only with JSON. Do not include any explanation."
• "Give your final answer only. Do not show your work."

Provider-Specific Budget Parameters (⚠️ not portable)

Apply only where reasoning is not needed in the output. ⚠️ Over-constraining hurts accuracy on complex tasks — validate quality trade-offs on your workload. Measure reasoning vs. output split separately in observability (G18).

Batch Processing

Shared prompt overhead paid once per batch instead of once per item. Instead of 50 calls each paying for the same system prompt, send one: "Classify each of the following 50 tickets. Return a JSON array indexed by position."

Use message queue (Kafka) to accumulate items; flush on size or time threshold. Cap batch size by token budget: items × avg_size + shared_prefix. For Java: CompletableFuture. For Go: goroutines with semaphore.

TOON — Token-Optimized Object Notation

For repetitive structured data with uniform schema, verbose JSON is extremely expensive — every key name, brace, and quote is tokenised.

// Standard JSON (~85 tokens for 2 rows)
[{"order_id": 1001, "status": "shipped", "region": "APAC"},
 {"order_id": 1002, "status": "pending", "region": "EMEA"}]

// TOON (~12 tokens for 2 data rows)
orders: id|status|region
1001|shipped|APAC
1002|pending|EMEA

// TOON with code substitution (~8 tokens per data row)
orders(id,s,r): 1001,S,A|1002,P,E
# s: S=shipped P=pending  r: A=APAC E=EMEA

Schema definition is a one-time cost in the system prompt, amortised across thousands of data rows. For a batch of 500 rows, the ~20-token schema overhead adds ~0.04 tokens per row. Apply TOON only to the LLM-facing portion when the downstream system needs standard JSON.

⚠️ Requires uniform structure across items.

Two distinct compounding levers — reduce number of calls and size of each result:

Minimise Call Count

Each tool call = full LLM round-trip (output tokens + result input tokens + follow-up). Three lookups = 3× the tokens.

• Combine related lookups into a single coarse-grained tool
• Avoid confirmation round-trips — if result is deterministic, don't ask LLM to confirm
• Parallelise inside the tool handler (not the agent loop) — one LLM call, tool fans out concurrently

For Java: CompletableFuture.allOf() inside tool implementations. For Go: sync.WaitGroup.

Project Tool Outputs

If a tool returns 50 fields but agent uses 3, the other 47 are waste on every subsequent turn.

• Field projection in the tool handler — LLM never receives extra fields
• Truncate large text fields to a token budget before returning
• Summarise before return for large result sets
• Structured compaction — [1001, 1002, 1003] vs [{"id": 1001, ...}]

For Java: ToolResponseProjector class with field whitelist. For Go: middleware wrapper stripping unexposed keys. Use G18 observability to measure average tool result token size — high-average tools are top targets.

A common expensive pattern: loading raw data into the LLM context for filtering, sorting, or aggregation. Code execution costs nothing in tokens; in-context data processing is extremely expensive.

MCP servers are the ideal location: filter, sort, project data in the tool handler so the LLM never receives raw unprocessed data. Audit current tool handlers for raw dataset dumps. Compounds with G14.

For Java/Go: filtering/sorting in tool handler methods or DB queries — never delegate to the LLM.

Agent Decomposition

A monolithic agent carries all tools, instructions, and domain knowledge — the system prompt is maximally large. Decompose into specialised sub-agents: each carries only context relevant to its function. Additional benefits: each sub-agent uses the cheapest model tier for its complexity (G6); failures isolated; agents independently testable. ⚠️ Start with 2–3 sub-agents for highest-token tasks before decomposing everything. NOTE: Multi-agent systems can reduce context per agent, but they can also increase total token usage due to handoffs, orchestration overhead, and duplicated context. The benefit is workload-dependent.

Async / Event-Driven Execution

In a synchronous ReAct agent, the full accumulated context is held open and re-sent while waiting for tool I/O.

Synchronous:   Agent holds [full context] → calls tool → waits → injects result → LLM call
Async:         Agent emits event → suspends (context discarded, zero cost during I/O)
               Tool executes → fires result → Agent resumes with [minimal state + result only]

On async resume, the agent loads only current-step context from the external state store (G10) — 200–500 tokens vs. 5,000+ in the synchronous version.

Temporal is the most mature tool for durable async workflows: each Activity receives only its specific inputs on resume, not the full workflow history. ⚠️ Async adds operational complexity (distributed state, event ordering, idempotency) — apply to high-token, high-latency workflows first.

Loop Limits and Stop Conditions

Without explicit bounds, agents accumulate tokens at each iteration until context window exhaustion.

• Hard iteration limit — never allow > N steps; return partial result or escalate
• Confidence-based stop — below threshold, request clarification
• Turn Efficiency alert — turn count > 2× baseline = loop pathology, halt
• Timeout-based stop — cap total execution time

LangGraph: recursion_limit in graph config. AutoGen: max_turns. LangChain AgentExecutor: max_iterations.

Token Budget Propagation

In multi-agent pipelines, downstream agents default to unconstrained verbosity without a budget. Pass remaining_budget as an explicit field in every inter-agent message (as part of the G9 state schema):

{"task": "...", "token_budget_remaining": 800, "output_format": "compact"}

Budget-aware output instruction in each agent's system prompt: "If token_budget_remaining < 500, respond only with required JSON fields — no explanation."

In LangGraph: add remaining_budget to TypedDict state; decrement at each node using completion_tokens from API response. Implement in the orchestration layer, not individual agent code.

"Optimization without measurement is guesswork."

Measure per LLM call

• prompt_tokens + completion_tokens split
• Model tier used
• Feature + team labels (for chargeback)
• Cache hit/miss
• Tool call count per task
• Retry count (retries double spend)

Effective Token (ET) Metric

Normalised cost across tiers:

Key KPIs and Governance

• Turn Efficiency KPI: alert when workflow turn count > 2× baseline — indicates loop pathology (broken tool, unexpected fallback, retry storm)
• Automated tool governance: cross-reference tool manifest against call logs; flag any tool not called in 30 days for pruning

Implementation

OpenLLMetry — lowest effort, wraps existing LLM SDK via OpenTelemetry auto-instrumentation. Langfuse — adds prompt versioning, eval scoring, team dashboards. Emit token-usage.jsonl per workflow run for post-hoc analysis. Set per-feature and per-team daily budgets in Prometheus; alert on overrun.

When a loop pathology is detected, the framework must act — not just alert. Implement a configurable circuit breaker per workflow: on trigger (turn count > 2× baseline, or repeated identical call fingerprint detected via `hash(step_name + input)` appearing ≥ 3 times in a sliding window of 5 calls), execute one of: `partial_return` (halt and return accumulated result with a `forced_stop` flag), `human_interrupt` (halt and emit to a webhook/queue for human decision), or `hard_kill` (halt and discard). Configure the action per workflow type in your orchestration layer — interactive agents default to `partial_return`; high-stakes workflows to `human_interrupt`; batch pipelines to `hard_kill`.