AI Observability Bill Shock — 200% Cost Increase

Got hit with a 200% increase on my OpenAI bill last month. Thought it was a rogue script or a bug in my code. It wasn't. It was just silent failures and inefficient prompt testing by the team that no one was tracking.

We were basically flying blind. A lot of startups are doing this right now with LLM APIs. You hook up OpenAI or Anthropic, throw some prompts at it, and look at the total bill at the end of the month. When it's $50, you don't care. When it spikes to $800 and you have zero attribution, it hurts.

I tried looking into the dashboard. It gives you an aggregate. That doesn't tell me if it was the customer service bot eating tokens or the new RAG feature. No idea who the "tenant" was driving the cost.

This isn't a rare thing either. Talked to a few other devs and everyone just accepts the billing page as truth and moves on. Tbh, it's a terrible way to run production.

If you are using LLMs, you need to track cost per model, per user, per day. And you need budget alerts before the end of the month.

I got tired of parsing raw logs and built LLMeter to solve this for myself. It tracks costs per tenant and sends budget alerts before you get hit with a surprise bill. It's open-source (AGPL-3.0) so you can self-host it or use the cloud version.

Stack is Next.js, Supabase, Inngest, and Vercel. Supports OpenAI, Anthropic, DeepSeek, and OpenRouter.

You can check it out here: LLMeter

Fwiw, even if you don't use LLMeter, start logging your token usage by user ID. You will thank yourself later when the usage scales.