Your AI Bill Is a Data Problem
Enterprises are getting better at seeing what they spend on AI. Proving what they got for it is another matter.
During a recent SD Times Live! Supercast on the economics of AI, J.R. Storment, executive director of the Tokenomics Foundation, described a “whiplash” this year. Early on, token maxing was all the rage. By spring, he said, companies were pulling back and putting caps on spending, because there was no direct tie between the spend and results.
The foundation’s own survey on the state of tokenomics bears that out. Respondents said they were fairly confident they had visibility into their AI spending. “But then we flipped that around and said, well, how many of you can show value on that? And, you know, it was like crickets,” Storment said. Only 9% were really confident.
“It’s all vibes”
Shawn Alpay, director of data engineering at the Linux Foundation, admitted his own team is in the same boat. It’s one thing to say AI increased a team’s throughput of pull requests, he said, but what did that result in? And did the flood of AI-generated pull requests make reviewing and merging them harder?
Teams compare notes on new models, he said, but “it’s all vibes.” Few are quantifying whether a model change cut token consumption by some percentage and what that produced.
Part of the problem is attribution. Storment said survey respondents could see their total bill but couldn’t tie it to who was doing what, or allocate it to a cost center. They were also consuming AI from four to six different sources, each reporting data differently. “A token is not a token,” he said.
Alpay helps steward FOCUS, the FinOps Open Cost and Usage Specification, which unifies billing data across providers. But billing data can’t say which trace or span kicked off a given workload. So FOCUS will work with OpenTelemetry, a Cloud Native Computing Foundation project, during its 1.6 development cycle to create a telemetry schema that lets organizations associate OTel data with FOCUS cost data.
Storment also warned that the trendy fix, routing to cheaper models, has a catch. Switch away from a model where you’re running a lot of cached tokens, and “you break your cache, and suddenly you’re paying for uncached.” And some savings are as simple as education: feeding a model Markdown instead of a PDF, for example, uses far fewer tokens.
One correct answer in 400
Jerod Johnson, technology evangelist at CData Software, said the cost conversation is coming to a head because AI spend has moved from contained pilot spend to agents in production. Agents loop, call tools and decide for themselves how long to keep going. “What organizations are paying now depends on how agents behave more so than on how people use it,” he said.
CData’s team uses an internal test bed it calls CData Labs. In its most recent benchmark, 27 models ran the same enterprise tasks: a basic read, a composite read and an action, such as ranking accounts, identifying high-risk ones and inserting eligible accounts into a review queue. Without business context, only about one run in 400 succeeded, and that held for every model, including frontier models like Fable 5.1.
With context, 12 of the models went 15 for 15. The cheapest did it for less than a cent per correct answer; the most expensive cost about $1.17.
“The way we read that is that it tells you that your AI bill is mostly a data and architecture problem,” Johnson said. “If you get your data and your architecture right, the model choice just becomes a budgeting decision.”
Context times turns times model
Johnson framed cost as context times turns times model. “Speaking as a former math teacher,” he said, the word that matters is times. Double the context and quintuple the loops on the same model, and you have a tenfold problem.
In the company’s preliminary numbers from more than 800 runs, the median run without context took 29 turns, versus seven with context. Tool calls fell from roughly 40 to 8. And 37% of no-context runs hit the benchmark’s 40-turn cap, compared with 2% with context.
He said organizations can recover that spend by doing the work once, in one of two places:
- Before the request. Replicate and shape data in a warehouse or lakehouse so joins, filters and aggregations happen before the model sees anything. In CData’s testing, a derived view cut tokens from about 183,000 to about 40,000 per query, a 78% reduction, and cost from about 60 cents to about 15 cents. A simple cached copy cut token consumption by 89%. Johnson said live systems still make sense for up-to-the-minute data, write actions, data that can’t be copied for compliance reasons, and sources not yet replicated.
- Inside the request. Put a context layer in the AI gateway, so business definitions travel with each request. In his example, “What enterprise revenue is at risk this quarter?” depends on a fiscal calendar, finance’s approved definition of revenue, and the tacit knowledge of what “at risk” means, which lives in an organization’s best analysts. With that context served, one federated query answers the question. Johnson said a validated pattern in CData’s own work reduced 22 tool calls to 1, and a guarded write tool with server-side validation cut unauthorized writes to zero.
Measure cost per correct answer
The headline number from the benchmark: the full run without context cost $350 and produced one correct answer. With context, it cost $93 and produced 318 correct answers, or 29 cents each.
“A cheap run that fails is going to be the most expensive run that there is, because you haven’t gotten the correct answer yet,” Johnson said.
On whether to build cost governance tooling or rely on a cloud platform, Johnson advised starting with what a cloud or gateway provider already offers, such as spend by model, API key and project, plus budgets and alerts. But that won’t show spend per business question, turns per correct answer, or which data path served each answer. Explaining why a bill moved requires visibility inside the request path.
His questions for engineering leaders:
- What does a correct answer cost us?
- How many turns does an agent spend looking for answers, and how often does it hit the turn cap?
- Which questions are hitting live systems that a replica could answer?
- What is the cheapest model that passes the task?
Alpay’s closing advice was to keep encouraging teams to use the new tools while asking a basic question: can you quantify what you spent and what it resulted in? “Yes, you want to go along with the building, and you also want to trust but verify that was the best way to use that money,” he said.
To learn more, see CData’s Connect AI Gateway and the CData Labs research.
Register to watch the full Supercast on demand here.
The post Your AI Bill Is a Data Problem appeared first on SD Times.
Tech Developers
No comments