Why Cheaper Tokens Are Not Making Enterprise AI Agents Cheaper

Key Takeaways

• LLM pricing keeps dropping, yet Gartner expects AI inference costs for each agentic workflow to grow more than fivefold through 2028.
• Agentic AI can use 5 to 30 times more tokens per task than a regular chatbot, which quickly cancels out cheaper prices.
• 93% of respondents in a McKinsey survey say they have gone over their AI budgets, and 20–30% of AI spending often goes untracked.
• Only 37% of McKinsey survey respondents say AI has added to their profits (EBIT). The few AI high performers win by redesigning how work gets done, not by finding cheaper models.
• Enterprise AI ROI should be judged by the cost of each finished business result, not the cost per token.

Introduction

LLM pricing keeps falling. According to Jefferies research reported by the South China Morning Post, the average price to process one million tokens fell from $2.04 on May 31, 2026 to 1.16–1.18 between August 6 and 8, based on data from US research firm Silicon Data. A token is a small piece of text, about four characters long, and companies pay for AI by the token.

Yet companies running enterprise AI agents are watching their bills go up. Gartner calls this the Inference Paradox: each token gets cheaper, but the total cost of AI keeps rising. An AI agent does not answer once. It plans, uses tools, checks its own work and tries again when something fails. Each step uses tokens.

Here is why AI agent costs keep climbing, and what the small group of companies earning real profit from generative AI does differently.

Cheaper Tokens, Bigger Bills

Gartner predicts that by 2030, running a one-trillion-parameter model will cost AI providers over 90% less than in 2025. But it adds two warnings: providers will not pass all of those savings on to business customers, and companies are using more tokens faster than prices are falling. So Gartner expects overall AI inference costs (the cost of running AI models) to go up.

In August 2026, Gartner put a number on it: inference costs for each agentic workflow will grow more than fivefold through 2028. As Will Sommer, Sr. Director Analyst at Gartner, put it: "Product leaders cannot rely on more efficient token economics to rationalize AI costs."

A Simple Example

Cost per task equals tokens per task multiplied by the price per token. Say token prices fall about 40%, close to Silicon Data's May to August 2026 drop. At the same time, a company replaces a chatbot with an agent that uses 10 times more tokens, within Gartner's 5 to 30 times range. The cost of each task still rises to about six times its old level. For agentic AI, how many tokens the work needs matters more than the price of each one.

Why Enterprise AI Agents Use So Many More Tokens

A chatbot reads a question and answers it. According to Gartner, an agent must keep reasoning, negotiating and questioning itself until the job is done. The extra tokens go to three places:

• Rereading the same information: Agents keep resending prompts, earlier messages and tool results. A Stanford Digital Economy Lab study of AI coding agents found that this input drives most of the cost.
• Checking and retrying: McKinsey lists retries and agent loops among the hidden reasons AI spending grows.
• Many steps per request: Agentic workflows multiply the number of model calls in each interaction, as in other multistep AI agent workflows.

Costs are also hard to predict. In the Stanford study, token use for the same coding task differed by up to 30 times between runs. Now that providers charge by usage, McKinsey notes, LLM token costs have become an everyday operating expense.

The Hidden Costs Beyond LLM Token Costs

Tokens are only the easiest cost to see. McKinsey's May 2026 Enterprise AI FinOps survey found that AI spending grows nearly fourfold when companies move from a few projects to company-wide use. Many of these hidden costs of LLM systems are hard to spot, and 20–30% of AI spending often goes untracked.

Three other problems slow enterprise AI projects down:

• Agents where they are not needed: Gartner says many tasks sold as "agentic" do not need agents at all.
• Skills gaps: Leaders in Deloitte's survey named a lack of AI skills among workers as the biggest barrier.
• Weak controls: Deloitte found that only one in five companies has a mature way to govern autonomous AI agents.

As a result, Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027 because of rising costs, unclear business value or weak risk controls.

The Enterprise AI ROI Gap: More AI, Same Profits

Profits are not keeping up. In McKinsey's State of AI 2026 survey of 1,719 people, only 37% said AI had added to their company's EBIT (earnings before interest and taxes), about the same as a year earlier.

• Deloitte: In its survey of 3,235 leaders, carried out in August and September 2025, 66% reported productivity gains, but only 20% are growing revenue with AI.
• PwC: Just 12% of the 4,454 CEOs in its 2026 Global CEO Survey saw both cost and revenue benefits, and 56% saw no significant financial benefit.

Saved time only becomes profit when the work itself changes, which is why companies with AI-led processes pull ahead. Yet Deloitte found that 37% of organizations use AI at a surface level, with little or no change to how they work.

What AI High Performers Do Differently

McKinsey calls about 6% of its respondents AI high performers: at least 5% of their EBIT comes from AI.

Source: McKinsey, The State of AI in 2026

High performers rebuild workflows instead of adding AI to old ones. They also feel AI agent costs first: they report cost limits on AI coding agents about three times as often as other companies, a sign of heavier use.

A Practical Playbook for AI Cost Optimization

AI cost optimization does not mean spending less on AI. It means making every token worth paying for. McKinsey says companies that use AI carefully can save 20–30% on AI costs. Five steps help most:

1. Track cost per result, not cost per token. Measure what each finished task costs, such as one insurance claim processed.
2. Match the model to the task. Gartner recommends sending simple, frequent tasks to smaller, specialized models and saving the most powerful models for complex, high-value work.
3. Send less with each request. Prompt caching can cut repeated input-token costs by up to about 90% for RAG systems (tools that pull in company documents to answer questions).
4. Put limits on agents. Cap retries, tool calls and loops, and fix badly designed workflows.
5. Build one place to watch AI spending. An AI FinOps control plane tracks usage, assigns costs to teams, enforces budgets and sends each request to the right model.

‍