Stop Measuring AI Agents by Token Burn

Open most AI agent dashboards and you will find the same numbers: tokens consumed, API calls made, conversations handled, minutes of talk time. They are easy to count, so they get counted. None of them tells you whether the agent did its job.

Imagine running a sales team that way. The weekly report lists words spoken, emails sent, and hours logged, and never mentions pipeline. You would replace the report before you replaced the reps.

An AI employee is an AI agent with a job, and a job is measured on outcomes. This post makes the case for retiring token burn as a performance metric and replacing it with the scorecard you would give a person.

Tokens are an input, and the input is getting cheap

A token is a unit of text a model reads or writes. Token counts measure how much the agent processed, which is a cost line, not a result. And it is a shrinking cost line. Stanford's 2025 AI Index reported that the price of running a model at the GPT-3.5 level fell from $20 per million tokens in November 2022 to seven cents per million by October 2024, a reduction of more than 280 times in about 18 months.

When the input cost is falling that fast, a scorecard built on it tells you less every quarter. A metric that measures the cheapest part of the system cannot tell you whether the expensive part, your customer relationship, got better.

Activity is not output

The next layer up from tokens is activity: calls answered, messages sent, conversations completed. These feel closer to value, and they are still the wrong unit. A call answered is not an appointment booked. A message sent is not a lead recovered. Activity metrics describe effort. The business pays for results.

This is the gap most AI programs fall into. MIT's 2025 GenAI Divide study, drawing on 150 executive interviews, a survey of 350 employees, and 300 public deployments, found that 95 percent of enterprise generative AI pilots produced no measurable impact on profit and loss. Its explanation was not weak models but tools that never integrated into real workflows. A pilot with no workflow has no number to move.

The scorecard you already use for people

You do not need a new framework. You need the one you use for every hire. Five questions cover it.

• Outcomes. Did the work produce value? Appointments booked, leads qualified, requests resolved, revenue recovered.
• Speed. How fast did it happen? Time to first contact, time to resolution.
• Coverage. How much of the work got done? Contact rate, queue completion, after-hours coverage.
• Quality. Was it done right? Script adherence, disposition accuracy, customer satisfaction.
• Efficiency. What did it cost? Cost per outcome, handoff rate, hours returned to the team.

Notice where tokens land: inside Efficiency, as one input to cost per outcome. That is the right place for them. A denominator, not a headline.

What this looks like for one AI employee

Take the AI Lead Recovery Agent. Its job is to restart conversations with leads that went quiet: missed calls, abandoned forms, no-shows. An activity report would show attempts made and messages sent. The outcome report shows something different: time to first recovery attempt, recontact rate, appointments recovered, and pipeline recovered.

The second report answers the question the business asked when it hired the agent. If the recovered appointments do not cover the cost, the agent is not doing its job, however efficient its token use. If they do, nobody will ask about tokens.

Outcomes are where pricing is going anyway

The market is moving in the same direction. Gartner predicted in July 2025 that by 2030 at least 40 percent of enterprise SaaS spend will shift toward usage-, agent-, or outcome-based pricing. Deloitte's 2026 technology predictions, which cite that figure, add the catch: outcome-based pricing rests on business results, and results are harder to measure than consumption.

That is the whole argument in one sentence. Vendors would love to bill for tokens because tokens are easy to count. Buyers should insist on outcomes because outcomes are what they are paying for. The side that controls the measurement controls the economics.

We measure people badly too. Fix both.

Outcome measurement is not a solved problem for human teams either. Gallup reported in 2018 that only one in five employees strongly agreed their performance was managed in a way that motivated them to do outstanding work. The scorecard for an AI employee can be better than that from day one, because the agent writes every disposition and outcome to the record as it works. There is no self-reporting to argue about.

Build the scorecard once, with outcomes at the top and tokens at the bottom, and use it for every AI employee in the workforce. That consistency is what turns a collection of agents into something you can manage.

How Vida measures an AI workforce

Vida is where businesses hire, manage, and measure AI agents as digital employees. Every agent is built around an outcome you define, and the AI workforce management layer tracks what it attempted, completed, handed off, and cost, across all five KPI categories. Billing can follow the same logic, with outcome-based options where the workflow has a clear completed result.

Bring the number you wish your team could move. We will help define the role, the systems, and the KPIs, and prove the outcome in a paid pilot on live traffic. Design Your AI Agent.