Opinion: Tokens per Task Is the Mobile AI Metric You Are Not Measuring

You are on the last leg of a commute, dictating a long email reply through your phone's AI assistant. Halfway through the third paragraph, the assistant stops, shows a generic error, and the conversation resets to a blank state. You check the logs later and find the real story: the feature hit its token cap mid-task, and every retry from the user's phone re-sent the full conversation history, burning the remaining budget three times faster.

Here is my position: a blown token budget is a UX bug, not a cost problem. On mobile, a token budget is a hard runtime resource, exactly like battery or memory, and when it runs out the user sees a crash, not a bill. That is why you should measure your AI feature the way you measure frame rate — per completed interaction, not per API call.

Most teams I see are still reporting tokens per request, which is a metric that flatters the demo and hides the failure. A single user task can consume five requests, three retries, and two context re-sends before it succeeds or dies. If you only watch the per-request average, you will never see that the expensive part of your feature is the recovery path, not the happy path.

Why per-request metrics lie to you

A request that fails and retries costs two or three times its nominal price, and the retry often re-sends the entire conversation history. Context re-send is the silent killer: a ten-turn chat can spend more tokens on repeated system prompts and old messages than on the final answer.

The average per-request number also hides the tail. One task that burns 40,000 tokens because of a loop can sit next to a hundred tasks that cost 500 each, and the average will look healthy. Your token budget, though, does not care about averages; it cares about the total, and one runaway task can eat a day's allowance.

The metric that matters: tokens per completed task

Define a task as one user intent, from the moment the user starts speaking or typing to the moment the feature delivers a usable result. Count every token spent on that intent: the initial prompt, the context re-sends, the retries, the fallback calls, and the final completion.

That single number tells you more about your product than any latency chart. If tokens per task is high, your feature is either too chatty, too retry-happy, or too eager to re-send context. If it is low, you have room to add features without blowing the budget.

A minimal token meter for your mobile app

The instrumentation is deliberately small: a client wrapper that reports usage after every model call, and a server endpoint that aggregates by task name. Here is the client side, written for React Native but portable to any JavaScript environment:

// tokenMeter.js
const SERVER = 'https://your-server.example/usage';
const SESSION = 'session-' + Date.now();

export async function runTask(taskName, prompt, callModel) {
  const startedAt = Date.now();
  try {
    const result = await callModel(prompt);
    const usage = result.usage || { prompt_tokens: 0, completion_tokens: 0 };
    await report(taskName, usage.prompt_tokens, usage.completion_tokens, 'completed', startedAt);
    return result;
  } catch (err) {
    await report(taskName, 0, 0, 'failed', startedAt);
    throw err;
  }
}

async function report(taskName, promptTokens, completionTokens, status, startedAt) {
  try {
    await fetch(`${SERVER}/usage`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        session: SESSION,
        task: taskName,
        promptTokens,
        completionTokens,
        total: promptTokens + completionTokens,
        status,
        durationMs: Date.now() - startedAt,
      }),
    });
  } catch (err) {
    // Telemetry must never break the feature it is measuring.
  }
}

The server side is a small Express app that stores the measurements in a JSON file and returns a per-task summary. It deploys to any free server that runs Node, including the one mentioned below:

// usage-server.js — run with: node usage-server.js
const express = require('express');
const fs = require('fs');

const app = express();
app.use(express.json());

const FILE = './usage.json';

function read() {
  try { return JSON.parse(fs.readFileSync(FILE, 'utf8')); }
  catch { return { tasks: {} }; }
}

app.post('/usage', (req, res) => {
  const data = read();
  const { task, total, status } = req.body;
  data.tasks[task] = data.tasks[task] || { count: 0, totalTokens: 0, failures: 0 };
  data.tasks[task].count += 1;
  data.tasks[task].totalTokens += total;
  if (status === 'failed') data.tasks[task].failures += 1;
  fs.writeFileSync(FILE, JSON.stringify(data, null, 2));
  res.status(201).json({ ok: true });
});

app.get('/usage', (req, res) => {
  const data = read();
  const summary = Object.entries(data.tasks).map(([task, v]) => ({
    task,
    avgTokensPerTask: Math.round(v.totalTokens / Math.max(v.count, 1)),
    failureRate: Math.round((v.failures / Math.max(v.count, 1)) * 100),
  })).sort((a, b) => b.avgTokensPerTask - a.avgTokensPerTask);
  res.json(summary);
});

app.listen(process.env.PORT || 3000);

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that offers free model access and a free server slot, which makes this measurement setup cheap to run. As of late August 2026, the free tier includes a 10M-token allowance and a server you can deploy this aggregator to; check the official docs for current numbers, because free tiers change.

Read the report, then make the cheap fixes

Once you have a day of real usage, sort the summary by avgTokensPerTask and look at the top three rows. The fixes are usually cheap and mechanical:

The point of the table is that you do not need a fancy tracing system to find these problems. A JSON file and a sorted list are enough to tell you which task is eating your allowance and whether the fix worked after you ship it.

Limitations and who should skip this

• A JSON file store is fine for a solo developer or a small pilot, but it will not survive concurrent writes from a large user base.
• The measurements only capture what your code reports, so a model call that bypasses the wrapper will be invisible to the report.
• Tokens per task does not measure quality; a cheap task that produces a useless answer is still a failure, just a cheaper one.
• Skip this approach if you have no hard token cap, if your feature is a single one-shot request with no session, or if you cannot change the prompt, model, or retry logic after seeing the data.

Your users will never see the token meter, but they will feel it the moment the budget dies mid-conversation. Measure tokens per task, fix the top three rows of the report, and you will turn a silent crash into a feature that survives a full day of real use.

If you run this meter on your own mobile AI feature, I want to know your numbers — post your task name, average tokens per task, and whether the fix was a cache, a truncation, or a smaller model.