LLM cost optimization: controlling tokens and model choice

Most of an LLM bill is input, not output. Every API call is priced per token, and input (everything you send) and output (everything the model generates) are billed at separate rates. A token is a chunk of text — roughly three-quarters of a word — and providers charge per million tokens processed.

The model is stateless, so it has no memory of the previous call. Each request carries the full payload again: system instructions, any few-shot examples, the documents your retrieval step pulled in, and the running conversation history. A chat assistant that keeps ten turns of context pays for those ten turns on turn eleven, and again on turn twelve, and so on as the thread grows. The answer the model writes back is often small next to that resent context.

This cost shape is inherited rather than chosen. Teams adding an AI feature to a live product wire a model into an existing flow, ship it, and only later discover that a long system prompt and a generous retrieval window set the per-request cost before any user types a word. The economics of production AI systems are decided mostly at the input boundary, which is also where most of the controllable savings sit.

Switching to a cheaper model lowers the per-token rate; it does nothing about how many tokens you send. If a request ships a 12,000-token system prompt and twenty retrieved chunks, the cheaper model still reads all of it on every call, so the bill drops by a fixed percentage and keeps climbing on the same curve.

There is a second catch. A smaller model that struggles with the task can produce longer, hedged answers or need a retry, and both claw back the saving you moved models to get. The durable wins come from sending less and paying less for the parts that repeat, then choosing the model — which is why model selection belongs later in the sequence, not first.

When the bill is climbing and no one can name the cause, the fastest first step is an audit of what each request actually sends and why. On live LLM features, our engineers have usually found the largest savings in caching and retrieval, not in model swaps.

Review your cost profile with an engineering team that ships cost-controlled LLM features.

Prompt caching removes the single biggest recurring charge: paying full price for the same prefix on every call. When a stable block repeats across requests — system instructions, tool definitions, a fixed reference document — the provider can bill it at a fraction of the normal input rate instead of reprocessing it each time.

The discounts are large and documented. Anthropic bills cached reads at one-tenth of the base input price, with the first write costing about 25% more than a normal input token. OpenAI takes a different route: its automatic 50% discount on cached input tokens applies with no code change on prompts over 1,024 tokens, though that discount does not extend to its Batch API, so the two savings don't stack.

Caching has a failure mode worth naming, because it is the usual reason a team turns it on and sees no saving. A read only happens when the same prefix comes back inside the cache lifetime — five minutes by default on Anthropic — and the cached block has to be byte-for-byte identical. Put a timestamp, a request ID, or the user's message before the cached content and every request misses, so you pay the write premium and never collect the read. Order the prompt static-first and variable-last, and place the cache boundary at the end of the part that never changes.

The cheapest token is the one you never send. Two habits inflate requests without adding value: over-broad retrieval on the input side and uncapped responses on the output side. Both are quick to fix and both show up directly on the bill.

Retrieval scope is the common offender. Pulling the top twenty chunks "to be safe" multiplies input on every call, and most answers need far fewer; narrowing the set usually cuts cost and improves accuracy at once, because the model sees less noise. That overlap between cost and quality is the same reason fixing RAG retrieval pays back twice — a tighter retriever is both cheaper and more reliable.

On the output side, set a maximum output length per endpoint and use stop sequences. A summarizer that can answer in 150 tokens should not be free to run to 800, and output tokens are priced above input per token, so a cap protects both spend and latency.

A few sources of input bloat recur across projects:

• A system prompt that grew with every patch and was never trimmed back.
• Few-shot examples kept long after the model stopped needing them to perform.
• Full conversation history resent on every turn where a short running summary would do.
• Retrieval fixed at a high top-k regardless of how simple the question is.

Plenty of requests don't need your most capable model, and plenty don't need an answer within the second. Two structural levers follow from that split: model routing and batch processing, each of which lowers cost by matching a request to the cheapest option that still meets its needs.

Routing sends routine, high-volume work — classification, extraction, short replies — to a small model and reserves the frontier model for genuinely hard cases, behind a classifier you can measure. Small models are priced well below frontier models per token, so moving even part of the traffic down a tier moves the bill. The failure mode is a weak router that sends hard queries to the small model and degrades answers silently, so measure routing decisions and keep a fallback to the larger model for low-confidence cases.

Batch processing suits any work that tolerates delay: overnight document processing, bulk classification, offline evaluation runs. AWS Bedrock batch inference runs at 50% of on-demand pricing, with typical completion inside 24 hours, and the same half-price pattern holds across major batch APIs. The constraint is obvious but easy to forget under pressure — batch is useless for anything a user is waiting on, so it fits back-office pipelines rather than interactive features.

Volume decides how much these levers return. For high-volume e-commerce features such as support assistants and product Q&A, routing and batching pay back fastest, because a small per-request saving multiplied across millions of calls is the whole game. For agentic features there is a sharper risk: a loop that retries can run up cost quickly, so hard caps on tokens-per-task and on spend stop a runaway run before it reaches the invoice — the subject of a companion article on agent reliability (timeouts, retries, and cost caps).

Every change above is a guess without per-request cost data. Before optimizing anything, instrument token usage broken down by feature, endpoint, and user, so you can see which calls dominate the bill and confirm whether a change actually helped rather than just moved the problem.

There is now a vendor-neutral way to capture this. The OpenTelemetry GenAI semantic conventions define a standard metric, gen_ai.client.token.usage, recorded as a histogram split by input and output, so token counts land in the same telemetry pipeline across providers. Pair that with cost dashboards and spend alerts, and fold it into the same observability and alerting setup you already run for latency and errors, so a sudden climb surfaces in days instead of at the next quarterly review.

The levers compare cleanly once the measurement is in place:

• Most of an LLM bill is input tokens — system prompt, retrieved context, and resent history — not the model's output.
• Prompt caching bills a repeated prefix at a fraction of the normal input rate, but only when the cached block stays identical and is reused within the cache lifetime.
• Tighter retrieval and output caps cut tokens on every call and often raise answer quality at the same time.
• Model routing and batch processing lower cost structurally by matching each request to the cheapest model and billing mode that still meets its latency and quality needs.
• Per-request token logging by feature is the prerequisite for every other change; without it, optimization is guesswork.

A rising LLM bill at flat usage almost always means two things at once: context is repeating on every call, and nobody is watching the number closely enough to see which feature is responsible.

The order of operations that works is measure first, then cache the repeated prefix, trim retrieval and cap output, route and batch what fits, and re-measure to confirm each change held. Model choice still matters, but it earns its place near the end of that list, once the per-request payload is under control.

If your LLM spend is rising faster than usage and the cause is unclear, our engineers can review your current token profile and the cost levers that fit your stack and latency needs — start a conversation about your LLM costs.