It’s Sunday, 2:14 AM. The PagerDuty alert hits my phone with that specific, jarring frequency that makes your stomach drop before you’ve even opened your eyes. My Databricks billing alert wasn’t a standard "usage threshold reached" notification; it was the "you’ve hit 80% of your monthly cloud spend in 48 hours" panic text.
I sat up, opened the Databricks console, and stared at the Billing page. Our sql_warehouse_prod_v2 was burning DBU (Databricks Units) like it was a crypto-mining operation. We had shipped a new pipeline on Friday, it passed CI, the integration tests looked green, and the data landed on time. But while we were sleeping, the bill was growing faster than our debt in college.
What we saw
The symptom was simple: our bill went vertical. The dashboard showed a steady, flat line for the past three months, followed by a spike that looked like the edge of a cliff.
My first assumption was a runaway loop in a Python job. I checked the cluster logs for dbr 13.3 LTS. Nothing. Then I checked the spark_query_history. Nothing out of the ordinary—just the standard daily ingestion patterns.
The false lead was the "Auto-stop" setting. I looked at the UI for the Serverless SQL Warehouse and saw Auto-stop: 10 minutes. In my head, this was bulletproof. If the warehouse isn't doing anything, it shuts down. The math seemed solid: 10 minutes of idle time is negligible.
But I was looking at the wrong metric. I was looking at the cluster state, not the session state. The warehouse wasn't idling; it was being kept alive by a ghost.
Photo by Kisetsu Co on Unsplash
Root cause
The culprit was a hidden interaction between our BI tool (a standard Tableau integration) and the Databricks Serverless SQL Warehouse. We had migrated to Serverless because we wanted the "instant-on" experience.
In a traditional cluster, the warehouse would have eventually hit a resource contention limit or a timeout. But Serverless is designed to stay available. We had a dashboard connection string configured with Catalog and Schema settings that were hitting a system.information_schema query on a heartbeat interval.
Because we used a service principal with broad CAN USE permissions, the connection remained active. The Databricks Serverless SQL Warehouse interpreted these heartbeat pings as "active queries."
Crucially, the Auto-stop setting only triggers when the warehouse is truly idle. Because the heartbeat hit the SQL warehouse every 8 minutes, the 10-minute timer reset itself into infinity. The warehouse never hit its idle threshold. It wasn't "stuck" in a loop; it was being held hostage by a silent, low-latency heartbeat that didn't even show up in our main performance monitoring because it was sub-millisecond.
We were paying for a "Large" size warehouse—which runs at a significantly higher DBU rate—to serve a heartbeat ping that could have been handled by a "Starter" size or, better yet, a cached metadata call.
The fix
The immediate fix was to kill the connection from the BI tool side and force the warehouse to scale down. I manually set the Auto-stop to 1 minute to ensure it would die instantly once the connection was severed.
Then, we had to redefine the warehouse configuration. We abandoned the single large warehouse for BI and split the workloads. We created a "Serverless-Small" for the heartbeat-heavy dashboarding and kept the "Large" warehouse strictly for ad-hoc analyst queries and heavy ELT transformations.
We also updated our connection string in the BI tool to point to a specific Unity Catalog schema that didn't require the broad information_schema scanning that was triggering our phantom queries.
Finally, I implemented a Tag policy. Databricks allows you to add custom_tags to your SQL warehouses. I added CostCenter: Finance and Owner: DataEng. While this didn't stop the spending, it allowed me to isolate the DBU consumption at the warehouse level in the billing export CSVs within minutes, rather than waiting for the bill to aggregate.
What we changed so it never happens again
We stopped trusting the "Serverless is magic" marketing. Serverless means you don't manage the nodes, but you absolutely have to manage the session lifecycle.
First, we implemented a strict "Warehouse Sizing" policy. No production warehouse is allowed to be larger than "Medium" unless it has an explicit, documented exemption in the Terraform repo. If you want a "Large" or "X-Large" warehouse, you have to open a PR, and the CI pipeline runs a cost-estimate check using the Databricks Billing API to flag the daily run-rate.
Second, we moved away from generic service principals for BI tools. We now use scoped service principals with READ ONLY access to specific schemas. This prevents the BI tool from querying the wider information_schema or system catalogs that trigger those hidden, expensive backend processes.
Third, we set up a "Budget Alarm" using a Lambda function that polls the Databricks billing data every 6 hours. If the daily burn rate exceeds a 20% variance from the 7-day rolling average, it fires a high-priority alert into our Slack #ops-alerts channel.
Lastly, I learned to never, ever set an Auto-stop to anything longer than 5 minutes for non-critical workloads. In the world of Serverless, that 5-minute window is the difference between a productive team and a very uncomfortable conversation with your CFO.
You ship the code, the tests pass, and the data is correct. That’s the easy part. The hard part is ensuring that the infrastructure—the invisible, elastic, "serverless" part—doesn't treat your credit card like a bottomless well. Keep your warehouses small, your permissions tight, and your heartbeats monitored.
Cover photo by Kevin Ache on Unsplash.
