The Real Cost of Agentic Loops at Scale

Two cost trends are moving through AI infrastructure at the same time, in opposite directions, and most conversations about agent economics only track one of them. Per-token prices are falling fast. Agentic workloads are pulling more tokens per task, faster still. Which one dominates your bill depends entirely on how many steps your loops actually run — and that's a number worth modeling before scaling a product, not after.

The Price Is Actually Falling

Start with the trend that gets less attention: model inference has gotten dramatically cheaper for the same capability. a16z's tracked analysis, which it calls "LLMflation," finds that for a model of equivalent performance, the cost is decreasing by roughly 10x every year — around a 1,000x decline over three years for GPT-3-equivalent capability. That is a real, structural deflation in the cost of intelligence, and it is the reason naive year-over-year cost comparisons for "the same task" are often misleading: the task got cheaper to run at a fixed capability level even as the frontier moved.

Anthropic's own current pricing illustrates where the market sits today: Claude Opus 4.8 runs $5 per million input tokens and $25 per million output tokens; Claude Sonnet 4.6 runs $3 and $15; Claude Haiku 4.5 runs $1 and $5. Anthropic publishes a worked example directly relevant to agent loops: a one-hour agent session consuming 50,000 input tokens and 15,000 output tokens on Opus-class pricing comes to about $0.71 — or roughly $0.53 with prompt caching enabled, since caching avoids re-paying for context that repeats across steps.

The Multiplication Is Winning

Set against that falling price is a rising quantity: agentic loops use far more tokens per task than a single exchange. A single chat turn typically runs on the order of 500 to 1,000 tokens. Agentic workloads — where an agent reasons, calls a tool, reads the result, and reasons again — commonly run 10 to 100 times that, and complex tasks involving many tool calls can push higher still, since each step re-sends the accumulated transcript alongside whatever new content it produces.

This is the same dynamic covered in the cost mechanics of agentic reasoning: a three-step loop costs roughly three times a one-shot answer, a five-step loop costs roughly five times, and the multiplier compounds with every additional step a problem turns out to need. The most extreme documented version of this multiplier comes from Anthropic's own account of building a multi-agent research system: their production multi-agent architecture used roughly 15 times the tokens of a normal chat interaction to gain a real, measured performance improvement on hard tasks — a concrete illustration that the step-count multiplier is not a theoretical worst case, it's what a well-built production system actually spends when the problem calls for it.

Falling per-token prices offset part of that multiplication — a 10x price drop absorbs a lot of a 10x token increase — but they don't automatically cancel it out, especially for loops that run well past five or ten steps, or that scale to thousands of queries a day.

What This Looks Like at Real Volume

The stakes of getting this wrong scale directly with how many queries a product runs. A handful of anecdotal industry accounts illustrate the shape of the risk, though it's worth being clear about how solid the sourcing is on each: one widely circulated account attributes a rapid AI infrastructure budget run-up to a large tech company's engineering organization, and another describes a healthcare company's inference bill rising nearly six-fold in six weeks after a retrieval fault — both reported secondhand by a vendor blog rather than confirmed in the companies' own public statements, so they're worth treating as illustrative of the failure pattern rather than as verified case studies.

The more defensible, vendor-neutral pattern behind those anecdotes is straightforward: an agentic loop that runs longer than intended — because of a bug, a missing iteration cap, or a task that turns out to be more open-ended than expected — doesn't fail loudly. It just keeps consuming tokens at whatever multiplier its step count implies, and the bill reflects that multiplier before anyone notices the loop should have stopped. This is the same failure mode covered in the agentic-reasoning concept as the infinite-loop problem, and the fix is the same one: a hard iteration cap, because cost containment for agents is fundamentally a step-count problem before it's a pricing problem.

The Actual Lever: Route, Don't Just Optimize

Given that per-token price is largely outside a builder's control and step count is largely dictated by the problem, the lever that's actually available is deciding which queries get routed into an agentic loop at all. A meaningful share of production traffic for most applications is simple lookups, classifications, or single-fact questions that never needed multiple steps in the first place — sending that traffic through an agent loop by default multiplies cost for zero benefit, since a single inference would have answered it just as well.

This routing principle is described by at least one AI infrastructure vendor as core to its own cost-optimization pitch — the specifics of any particular vendor's savings claim should be read as marketing rather than independently audited data, but the underlying logic doesn't depend on the vendor: a query that doesn't need multiple steps shouldn't be charged the multi-step price. The 10x annual price decline from a16z's data is a tailwind every builder gets for free; the 10-100x step multiplication is the part actually under a team's control, and routing is the mechanism for controlling it.

The Bottom Line

Per-token prices will likely keep falling — that trend has held for years and shows no clear sign of reversing. But agentic workloads are pulling costs the other direction fast enough that the net effect on any given product's bill is not obvious without doing the arithmetic for that product's actual step counts and volume. The teams that get burned are the ones that assumed falling prices would cover rising step counts by default. The teams that don't are the ones that treated routing — sending only the queries that need a loop into a loop — as the primary cost control, and treated the falling price curve as a bonus, not a plan.

Related reading: The Artificial Intelligence hub, How agents remember across steps, Tree-search and hierarchical agents in production