Most Kubernetes clusters can't tell you what their GPUs cost

I built a small tool to answer one question: how many GPU-hours is a cluster paying for and not using?

Then I pointed it at real production clusters. Not one of them could answer the question out of the box and the reasons are the same everywhere, because they're all defaults nobody changes.

Here's what I found, in the order it bit me.

1. The utilization metric everyone reports doesn't mean what you think

Almost every GPU dashboard is built on DCGM_FI_DEV_GPU_UTIL. It's the obvious choice it's called utilization, it's a percentage, it goes up when things get busy.

It reports the fraction of time at least one kernel was resident on the device. Not whether that kernel did anything useful. A process that pins the GPU with a trivial loop reads 100% utilized while computing nothing at all.

The metrics that tell you about real work are DCGM_FI_PROF_SM_ACTIVE (streaming multiprocessor occupancy) and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE (tensor core activity). The gap between them is the interesting part: high engine activity with near-zero tensor activity means the GPU is busy doing something that isn't machine learning.

I've started calling that ghost work. It's invisible on every dashboard I've seen, because those dashboards are plotting the wrong number.

2. Your GPU metrics are probably attributed to the wrong pod

This one is worse, because it looks correct.

dcgm-exporter only emits the workload's pod labels when Kubernetes pod mapping is enabled DCGM_EXPORTER_KUBERNETES=true. Without it, Prometheus attaches the scrape target's own identity instead. Your GPU metrics come back tagged with pod="nvidia-dcgm-exporter-xxxxx", namespace="monitoring".

So the data looks complete. Every series has a namespace and a pod. Group by pod and you get a tidy chart. It's just that every GPU-hour in your cluster is attributed to the exporter that measured it.

The tell is cheap to check: do the pods in your GPU metrics appear anywhere in kube_pod_container_resource_requests? If the intersection is empty, your attribution is fiction. One-line check, and I'd bet on the outcome more often than not.

3. The interesting counters are off by default

The DCGM_FI_PROF_* metrics the ones that distinguish real work from ghost work require profiling to be enabled in the exporter's counter configuration. On the clusters I looked at, they weren't.

That means the failure mode most worth catching is invisible on a default install, and you won't get an error telling you so. The metric simply returns nothing, and whatever you built on top quietly reports less than the truth.

4. The one that nearly fooled me

This is the part I'd want to read, so it's the part I'll tell honestly.

When profiling counters are missing, you can still estimate utilization from power draw. An idle A100 pulls around 55W against a 400W TDP; a busy one is near the ceiling. It's coarse, but idle versus working is unambiguous.

I built that fallback with a generic 50–350W envelope for unknown GPU models. Then I ran it against a card that draws 15–130W.

The tool reported a GPU running at 5% utilization with 88% of its framebuffer resident textbook memory-parked, a model loaded and serving nothing. It put a confident $3,327/month on it. I believed it. I told someone it was the first real evidence the whole idea worked.

It wasn't real. The card was working fine. The generic envelope compressed its entire operating range into what looked like idle. When I fixed the fallback to prefer GR_ENGINE_ACTIVE a direct measurement rather than an inference from watts the finding evaporated.

The lesson isn't "watch your constants." It's that a cost tool that produces confident numbers from degraded inputs is worse than no tool at all. The output was going to be screenshotted and taken to someone who approves budgets. A wrong number in that meeting doesn't just fail; it burns the credibility you needed to fix the real problem.

Everything I built after that assumes the data is incomplete and says so out loud. It refuses to project a monthly cost from a window shorter than a day, because you haven't seen one diurnal cycle. It names every metric it couldn't find and which findings that suppresses. It labels utilization as estimated when it is. The report is less impressive and considerably more defensible.

What to check on your own cluster

Five minutes, read-only, no install:

# 1. Are profiling counters enabled?
curl -s localhost:9090/api/v1/query \
  --data-urlencode 'query=count({__name__=~"DCGM_FI_PROF_.*"})'

# 2. Is pod mapping on? Compare these two sets.
curl -s localhost:9090/api/v1/query \
  --data-urlencode 'query=count by (pod) (DCGM_FI_DEV_FB_USED)'
curl -s localhost:9090/api/v1/query \
  --data-urlencode 'query=count by (pod) (kube_pod_container_resource_requests{resource="nvidia_com_gpu"})'

If the first returns nothing, your profiling counters are off. If the pod names in the second and third queries don't overlap, your GPU metrics aren't attributed to the workloads consuming them.

Both are one-line config changes. Neither is on by default.

The tool

gpuwaste is open source and does nothing clever: it ingests exported metrics, joins utilization against allocation, and reports how many GPU-hours you paid for and didn't use, with a dollar figure at the bottom.

It's deliberately offline it reads a CSV you export rather than connecting to your cluster, because production metric endpoints sit behind network policies and mTLS and most people never get past that. It also ships a synthetic data generator, so you can see exactly what it does in thirty seconds without touching anything real.

Link in the comments. It's free and I'm not selling anything on the back of it.

What I'd genuinely like: if you run it and the numbers look wrong for your setup, tell me how. Every cluster I've tested has broken it in a new way, and that's been the most useful part.