How I Monitored LLM Token Usage in a Go AI Service using OpenTelemetry (with MCP)

Every prompt you send to an LLM costs money—but do you actually know where those tokens are going? I instrumented an existing Go AI service with OpenTelemetry to track every token, then used MCP to dig through the telemetry and uncover what was really happening behind the scenes.

Why monitor your AI Service for Token Usage?

Unmonitored AI can instantly break your budget. A tech firm recently faced a massive $500 million Claude AI bill in just one month because nobody was tracking internal token usage.
Read more :

Company racks up $500M Claude AI bill in one month after putting no usage limits on employee licenses

A company faced a shocking $500 million bill for AI usage in just one month due to lack of spending limits on employee licenses. Explore the implications of unlimited access to AI tools and the need for governance in enterprise AI spending.

economictimes.indiatimes.com

Let's Use Datadog to solve this (Oh No Wait...)

Datadog can become expensive for AI apps because every model, user, and prompt adds more data to track. AI also creates lots of text, which costs more to store. It also uses Datadog-specific tools, so switching to another platform later can be hard and take a lot of work.

Here are few reasons why we choose OpenTelemetry :

1. Datadog "Custom Metrics Tax": Tracking GenAI apps creates too many unique tags, increasing Datadog custom metric costs ("Datadog Tax").
2. Vendor Lock-In: Datadog's LLM features rely on proprietary dd-trace-go instrumentation, making migration to another platform costly and code-intensive.
3. Self-Hosted Option: Self-hosting gives you full control over AI data ingestion, preventing expensive log indexing costs from large LLM payloads.

Adding OpenTelemetry to Your Existing Go AI Service

This blog focuses primarily on monitoring the data that's already being generated, rather than on instrumentation itself. If you haven't set up instrumentation yet, check out these resources first—they'll help you get started. (Or, honestly, you can just ask an AI assistant; it's pretty straightforward.)

Follow this : https://signoz.io/blog/golang-monitoring/

Why I Chose SigNoz for Observability

For this guide, we'll use SigNoz for observability. It's open source, supports self-hosting, and is a great choice if you're just getting started. Self-hosting also gives you more control over your data while keeping costs low.

BIG Mistake (ASKING AI FOR HELP WITH SIGNOZ)

SigNoz is a fast-growing open source project, which means the docs, installation steps, User interface, and best practices are constantly evolving. SigNoz's team is always finding better ways to make things easier, so when you're working with SigNoz, always start with the official documentation instead of blindly following what an AI says.

Otherwise, you might end up going down a rabbit hole, spend hours on things that have already changed, and eventually come back to write a blog post telling people "why you should've just read the docs."

...At least that's what happened to my friend. 😉

Installing Signoz on Linux

Follows this : https://signoz.io/docs/install/docker/

Play Around

Verify Metrics & Logs Ingestion

Create Dashboard

AI Token Usage Monitoring

Create Alert

Before you begin, make sure you've created a notification channel. I chose Slack—this time, setting it up was much simpler.
Follow this : https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/

Now, let's create the alert. I used llm_token_total, which measures the total number of tokens generated over time.

For this example, I configured the alert to trigger when llm_token_total > 1000 within a 1-minute window. This is just an example—it's better to monitor your token usage for a few days first, understand your typical usage patterns, and then set an alert threshold that aligns with your budget.

Result: Slack Alert After a Token Usage Spike

When the alert condition is met—for example, when your token budget is exceeded—a custom message is sent to Slack. This makes it easier to track usage and plan your budget more effectively.

Use MCP to explore further

I used the VS Code + Cline + free DeepSeek model setup because, well, it was free. But feel free to use whatever stack you prefer—pun absolutely intended. With the latest updates, connecting to an MCP server has become surprisingly straightforward, regardless of the tools you choose.

Bonus Challenge

If you actually made till here, then go further. Add some custom metrics, then write a blog on how you used OpenTelemetry and SigNoz to observe your AI stack. Be sure to add this blog as a reference to that!

Share what worked, what didn't, and any observability tricks you picked up along the way. Let's learn from each other.

Conclusion

Setting all of this up took barely 3–4 hours, which honestly isn't much considering the visibility and insights you gain in return. If you've made it this far, you've already done the hard part—so you might as well add OpenTelemetry to your project and start your observability journey.

And finally, a huge shoutout to SigNoz for continuing to invest in open source. If this guide helped you or you appreciate what they're building, consider giving their GitHub repository a star. It's a small gesture that goes a long way in supporting projects like these.

Collaboration

This blog was made with collaboration
Trushna from Trillion Defense &
Me (Bhavesh) from Trillion Defense

References:
https://signoz.io/docs/install/docker/
https://github.com/SigNoz/foundry/blob/main/docs/concepts/mcp-server.md
https://signoz.io/blog/golang-monitoring/

Let's Connect
Bhavesh
LinkedIn: https://www.linkedin.com/in/bhavesh-sp/
Github: https://github.com/BhaveshSP

Trushna
LinkedIn: https://www.linkedin.com/in/trushna-purohit/
Github: https://github.com/TrushnaCodes