I was building a product analytics tool.
The idea was simple: take product events and user feedback, run them through an LLM, and turn them into something useful — tags, summaries, sentiment analysis, and answers to questions like "Why did this user churn?"
Customers liked it. The models did their job.
The bill didn't.
Within a few months, my OpenAI bill was around $4,000–$5,000 a month, and it kept growing with every new customer.
Revenue was growing too, just not at the same rate.
Product analytics is often priced per seat. Inference is priced per token. As usage grows, those two curves don't always behave the way you want them to.
I needed to bring the cost down without making the product worse.
So I started trying things.
First, I tried the obvious stuff
1. Prompt caching
My system prompt was long and mostly identical across requests, so caching seemed like an easy win.
It helped, but not as much as I'd hoped.
The system instructions were only part of the input. Product events, session histories, and customer feedback changed with every request, so much of the expensive input couldn't benefit from prefix caching.
I kept caching where it made sense, but it wasn't going to solve the whole problem.
2. Renting GPUs and running Ollama
I tried running open-weight models like Llama and Qwen on GPU instances.
The per-token economics looked attractive. But once I factored in idle GPU time, redundancy, monitoring, quantization, and the time spent maintaining the setup, the savings weren't nearly as impressive.
My traffic wasn't consistent enough to keep the GPUs busy around the clock.
Self-hosting can make a lot of sense at sustained volume. For my workload, the economics weren't working out the way I'd expected.
3. Asking for volume discounts
Next, I looked into volume pricing through a reseller.
The conversations involved annual commitments in the $50,000–$60,000 range for discounts that could be around 10–20%, depending on the agreement.
That wasn't a commitment I wanted to make while the business and its usage were still growing unpredictably.
I didn't want to promise a certain amount of spend just to get a lower rate.
4. Switching to cheaper models
This was the obvious next step.
For simple tasks like tagging and classification, smaller models worked reasonably well.
But the more valuable features involved summarizing user behavior and explaining why customers might churn. That's where differences in output quality started to matter.
I could save money on individual requests, but I didn't want to introduce quality problems into the features customers were paying for.
I wasn't against cheaper models. I just didn't want to switch models blindly and hope nobody noticed.
Then I noticed something interesting
The same model families were available through different inference providers, sometimes at very different prices.
That got my attention.
What if I could keep using the models my application already depended on, but access them through a provider with better pricing?
I started exploring smaller inference providers and alternative API endpoints.
Some had competitive prices, but there were trade-offs. One offered only a handful of the models I needed. Another had compatibility issues. Rate limits and error behavior weren't always clear.
And I had another concern: how could I tell whether two endpoints were actually serving the model versions and configurations I expected?
A model name alone wasn't enough to answer that question.
Still, the price differences were interesting enough that I kept looking.
I wanted one endpoint instead of another collection of APIs
Managing multiple providers separately meant more credentials, integrations, dashboards, and failure scenarios to handle.
I wanted a single endpoint that would let me:
- Use the OpenAI SDK without rewriting my application.
- Access multiple supported models through one API.
- Monitor provider health and handle failures.
- Configure fallbacks when a provider wasn't available.
- See token usage and request costs without jumping between dashboards.
I kept looking, and that is how I found Osmexo. It was the first one that ticked the whole list: OpenAI-compatible, the models I was already using, provider health checks, fallbacks, and the cost of every request shown on the response.
The integration took a few minutes. For an application already using the OpenAI Python SDK, the change is two values:
from openai import OpenAI
client = OpenAI(
base_url="https://api.osmexo.com/v1",
api_key="YOUR_OSMEXO_API_KEY",
)
The rest of the application kept using the same SDK, the same model identifiers, and the same prompts.
That was the part I liked most: I could try a different inference path without rebuilding the application around a different client library.
But I still had one concern
Was I actually getting the model I expected?
Lower pricing is great, but it doesn't mean much if the output quality changes in ways that affect your product.
So before moving any real traffic, I tested it the only way I trusted: I took a few hundred of my own production prompts and sent each one twice, once to OpenAI with my own key and once through Osmexo, then compared the results side by side.
For a product analytics workload, that meant real tasks: classification, summaries, structured outputs, and churn explanations. The tags matched. The summaries were the same quality. Token counts matched, which is the hard thing to fake, because different models tokenise differently.
Osmexo also ships a small script that does a stricter version of this check with your own vendor key, on your own machine, and scores how alike the two sides are. I ran that too. It agreed.
It's also important to understand that identical model names don't automatically prove identical model versions or inference configurations. Even with the same underlying model, responses can vary because of sampling, serving implementations, and other settings.
I still wouldn't move all production traffic based on price alone. I started with one workload, compared results, and expanded only when the evidence supported it.
That evaluation habit is the most useful thing I took away from the whole exercise.
What changed for me
The biggest shift was realizing that reducing inference costs doesn't always require changing the model or running your own GPUs.
Sometimes, the opportunity is in how you access inference in the first place.
Today the same workload, on the same models, runs at roughly $2,000 a month instead of $5,000.
Caching still matters. Model selection still matters. Self-hosting can be the right choice for some workloads.
But comparing alternative inference providers is another option worth considering, especially when you want to keep your existing application architecture.
Using it also changed what I look at beyond token prices. Provider reliability, error handling, request tracing, and cost visibility turned out to matter as much as the rate, because the API is part of a product customers depend on.
I went looking for better economics. What I ended up with was a simpler stack as well: one key instead of one per vendor, one prepaid balance instead of several invoices, and one place that shows what every request cost.
Who this is for
If you're building a product and the LLM bill is one of your biggest line items, this is the situation it fits:
- Product analytics and data tools running extraction, classification or summaries across large volumes of records.
- Customer support and chat agents, where every turn resends the whole conversation.
- RAG and document pipelines, where retrieved context makes every prompt expensive.
- Coding assistants and agent workflows that issue dozens of calls per task.
- SaaS priced per seat, where revenue is fixed per user and cost is per request.
- Agencies and studios running AI features for several clients on separate vendor accounts.
If your bill is in the hundreds or thousands a month and growing faster than revenue, it is worth an afternoon: point one workload at it, compare the cost on the first response with your vendor invoice, and run the verifier against anything you don't trust.
And if you shelved an idea because the LLM API cost made the numbers impossible, this is a reason to pick it back up. At roughly 40 to 60 percent less per token, a feature that didn't pay for itself at list price may pay for itself now. The free credit on a new account is enough to find out.
You can take a look at Osmexo. New accounts start with free credit, so you can run your own prompts through it before deciding anything.
But I'm curious about something else, too.
What's actually worked for you when trying to reduce LLM costs?
Was it caching, switching models, running your own GPUs, negotiating pricing, or using a different inference provider?
I'd be interested to hear what worked — and what turned out not to be worth the effort.
