How LLM API Pricing Works: Tokens, Asymmetry, Waste

LLM providers like OpenAI, Anthropic, and Google bill through a pay-per-token API model: cost is calculated from the amount of text processed, not from subscriptions or server uptime. Every API call splits into two billing categories — input tokens (everything you send: prompts, chat history, system instructions) and output tokens (everything the model generates). Output tokens are priced roughly 4–5× higher than input tokens across providers, because sequential generation costs far more compute than parallel prompt reading. A token corresponds to roughly 3.5–4 English characters, so 1,000 tokens is about 750 words. Because billing counts every token, waste accumulates invisibly: bloated system prompts resent on every turn, unconstrained verbose outputs, uncached repeated inputs, and frontier models doing work a small model handles. The Asymmetry Audit below — one division — tells you which half of the bill to fix first.

In short:

• Billing unit: LLM APIs bill per token processed; a token is ~3.5–4 English characters, so 1,000 tokens ≈ 750 words — not the other way around.
• Double charge: Every call bills twice — input tokens (what you send, including chat history and system prompts) and output tokens (what the model generates).
• Cost asymmetry: Output tokens run roughly 4–5× the price of input tokens across major providers — the single most consequential fact in LLM cost design.
• Optimization rule: One calculation — output's share of your spend — determines whether to constrain generation or compress prompts first.

Introduction

Most LLM providers bill through a pay-per-token API economy: instead of flat subscription rates or server uptime, cost is a direct function of text processed and generated. That single design decision shapes everything about production AI economics — and most teams discover its implications on an invoice rather than in a design review. This piece covers the mechanics: what a token is, how the two halves of every API call are priced, where waste hides, and a one-line calculation that tells you which half of your bill to attack.

Why It Matters

Per-token billing means cost scales with behavior, not seats. Two products with identical user counts can have bills an order of magnitude apart depending on prompt design, history handling, and model selection. Understanding the pricing anatomy is the prerequisite for every optimization decision downstream — routing, caching, self-hosting break-evens all start from knowing what a token costs and which tokens dominate your spend.

Core Concepts

Token. The smallest unit of text a language model processes — a word, subword, or character sequence. In English, one token corresponds to roughly 3.5–4 characters, per Anthropic's glossary. Practical conversion: 1,000 tokens ≈ 750 words. (A common inversion — "1,000 words ≈ 750 tokens" — gets the ratio backwards and underestimates costs by ~44%.)

Context window. The total tokens a model can process in one call: your input plus its output. Everything inside it is billed.

The Two Halves of Every API Call

	Input tokens	Output tokens
What's counted	Prompt, user messages, conversation history, system instructions, tool definitions	Everything the model generates back
Also called	Prompt tokens	Completion tokens
Relative price	Baseline	Typically 4–5× input
Why	Prompt is processed in parallel	Generation is sequential — each token requires a full forward pass

The asymmetry is consistent across providers: Anthropic's published tiers pair rates like $3/$15 and $5/$25 per million tokens (anthropic.com/pricing); OpenAI's current rates follow the same shape (platform.openai.com/docs/pricing). Output is the premium product.

Two details in the diagram carry most of the cost consequences. First, the input side is a stack, and only its bottom layer is new: on every conversational turn, the system prompt, tool definitions, and the entire accumulated history are resent and re-billed — the user's new message is typically the smallest slice of what you pay to send. Second, the output block's price premium is physical, not commercial: reading a prompt is a parallel operation over all input tokens at once, while generation produces one token per forward pass, each pass conditioned on everything before it. Providers price the compute difference at roughly 4–5×.

Where Hidden Waste Accumulates

Because billing tracks every token, inefficient designs compound silently — and the compounding is geometric in conversations, because the input stack above is resent whole on every turn:

The chart is illustrative in scale but exact in shape: the dark base is the system prompt, identical and re-billed every turn; the light block is the history, which contains every previous turn's input and output. By turn ten, the tokens the user actually typed that turn are a rounding error on the bar.

Bloated system prompts. Long background instructions and context documents resent with every user turn. A 3,000-token system prompt in a 20-turn conversation bills 60,000 input tokens before the user has said anything new.

Verbose outputs. Unconstrained generation at 4–5× input pricing. Every filler sentence the model produces is billed at the premium rate; uncapped max_tokens and chatty response formats are direct margin leaks.

No caching. Paying full price to reprocess identical or near-identical inputs. Provider-side prompt caching discounts repeated prefixes; semantic caches skip the call entirely.

Over-powered models. Frontier models (Claude Opus-class, GPT-4o-class) doing categorization or formatting a small model (Haiku-class, mini-class) handles at 10–20× lower per-token rates.

Mature AI operations attack these through prompt compression, caching layers, and routing simple requests to cheaper models.

The Asymmetry Audit

Everyone lists the waste categories. Nobody tells you which to fix first. The 4–5× price asymmetry does — with one division.

Output share of spend:

S =	(T_out × P_out) ÷ (T_in × P_in + T_out × P_out)
T_in, T_out	Your monthly input and output token volumes (from your usage logs)
P_in, P_out	Your model's published per-token rates

The decision rule:

• S > 0.5 (output-dominated): Constrain generation first. Cap max_tokens, demand structured or terse output formats, cut conversational filler from response templates. Prompt work is second-order here.
• S < 0.5 (input-dominated): Attack input first. Prompt caching, system-prompt compression, history summarization, surgical RAG. Typical of chatbots dragging long histories and retrieval-heavy apps.
• Recompute after each fix — cutting one side flips which side dominates, and the priority flips with it.

The audit takes five minutes with request-level token logging in place, and it prevents the most common optimization mistake: weeks spent compressing prompts on a bill that generation verbosity is actually driving.

→ Run the Asymmetry Audit interactively — an animated walkthrough of the same call anatomy, turn-compounding chart, and decision flow above.

Limitations

• Rates move. Per-token prices and the input/output ratio are provider decisions; every number here should be re-checked against the pricing pages at decision time.
• The token-to-word ratio is language-dependent. Non-English text, code, and dense notation tokenize at different rates; the 3.5–4 character heuristic is English prose only.
• Cached and batch pricing complicate S. Prompt-cache hits and batch-API discounts bill at reduced input rates; compute S from actual spend, not list prices, once discounts apply.
• This covers API billing only. Self-hosted inference has a different cost structure entirely — fixed infrastructure rather than per-token rates.

Related Analysis

• Artificial Intelligence hub — the parent topic hub
• Building AI Systems That Actually Work — the systems pillar this cost discipline sits under
• Cutting LLM Costs: Tracking, Routing, and the Local Break-Even Line — the companion piece: what to actually do once you know which half of your bill is driving spend
• The AI Inference Cost Paradox — why falling per-token prices don't mean falling bills

References

1. Anthropic, Glossary — token definition (~3.5 English characters) — platform.claude.com/docs/en/about-claude/glossary
2. Anthropic, API pricing — anthropic.com/pricing
3. OpenAI, API pricing — platform.openai.com/docs/pricing

Final Thoughts

Per-token billing rewards teams that treat cost as an architecture property, not an accounting line. The mechanics are simple — two token types, one steep price asymmetry — but the asymmetry is the operative fact: it means the shape of your traffic, not just its volume, sets your bill. Run the Asymmetry Audit before any optimization sprint; it converts a list of best practices into an ordered plan.