What AI Agents Taught Me About Building Databricks Apps

The agents wrote most of the code. Everything around it was still on me, and I think that's good news

I've been building software since 2013.

Most of that time on the frontend: React, components, the part of the product people actually touch.

Then AI agents started writing code. A lot of it, and faster than me.

And I had the same question I think a lot of us have right now: if the code writes itself, what am I here for?

My answer so far is to make the job bigger. I'm working toward what I call a "Super Engineer": software engineering, data engineering and AI engineering under the same belt.

I'm sharing it while it happens, wrong turns included :)

This is one of those turns. We built a production Databricks App with AI coding agents doing a big part of the work. The code showed up fast. Getting it to run correctly took way longer.

And every one of those extra hours went into knowledge the agents didn't have.

The agent types faster than I ever will. So what do I bring to the table now?

The problem: the agent writes the code, the platform decides if it runs

Databricks Apps let you host a web app right next to your data.

Users sign in with the identity they already have, every query can run as that user, and Unity Catalog decides what each person can see. For an internal tool on top of governed tables, that's pretty close to ideal.

Ask an agent for "a React app with an Express API that edits these tables" and you'll have working code in an afternoon.

Whether it runs is a different story. That part depends on configuration: a bundle file, user API scopes, a warehouse binding and three separate permission systems.

Most mistakes there are one line long. Many fail silently, or with an error that points somewhere else -> and an agent that reads the error and searches the code will confidently fix the wrong thing.

Our app: React plus Node/Express in one container, reading and writing Unity Catalog tables through a SQL warehouse, deployed with a Databricks Asset Bundle and promoted from dev to sqa to prd through CI.

One request, five hops, and a trap waiting at almost every one of them

The walkthrough: where it broke

Wiring user identity

The design is on-behalf-of-user (OBO) authentication.

The Databricks proxy forwards the signed-in user's token in X-Forwarded-Access-Token. The API opens a SQL session with it on every request. The app's own service principal gets CAN_USE on the warehouse and nothing else.

First deploy: blank user names, empty tables.

hm.

The code was fine. The sql scope was sitting under config: in the bundle, where it's silently ignored. It belongs at the app level:

resources:
  apps:
    my_app:
      name: ${var.deploy_app_name}
      source_code_path: ./
      user_api_scopes: [sql]   # app level, not under config:

Declaring the scope only requests it. The workspace has to allow it and have user-token passthrough enabled, and the only real proof is the deployed app:

databricks apps get <app>   # effective_user_api_scopes must include "sql"

Configuring environments

We promoted to sqa and saw dev data.

A committed app.yaml was overriding the bundle's per-target settings, so every environment was running against the dev catalog. We deleted it and kept every environment value in the bundle targets.

Then a bare warehouse id made every query return 404 (the driver needs /sql/1.0/warehouses/<id>). And a catalog typo (_qa instead of _sqa) gave us empty screens and no error at all.

These days the app logs its effective catalog, schema and warehouse at boot. That one line catches this kind of bug in seconds.

Deploying

Our first deployment pulled the code from a git branch and never started: "App process did not start within 10 minutes."

A git snapshot only has tracked files, and our build output was gitignored. So we switched to the bundle and built at start:

command: ["sh", "-c", "npm ci && npm run build && node server/dist/index.js"]

Next one: the pipeline went green and the app didn't change.

With Asset Bundles, bundle deploy uploads the code and bundle run starts the new version. BUT our CI template only ran the second step for resource keys that matched its naming convention. Ours didn't.

databricks bundle deploy -t dev
databricks bundle run <app-resource-key> -t dev

The npm install had its own surprises: a corporate registry pinned in .npmrc that Databricks can't reach, an omit=optional setting that dropped native binaries, and a missing package-lock.json.

Opening it to users

Some users still saw nothing.

Three permissions have to line up: app CAN_USE (who can open the app), warehouse CAN_USE (who can run queries), and Unity Catalog grants on each table, including USE CATALOG and USE SCHEMA. Each one fails on its own, and different teams usually own them.

Our group check was also rejecting the right people. The workspace SCIM /Me endpoint leaves out account-level groups synced from the identity provider. Running SELECT is_account_group_member(:group) with the user's own token fixed it.

And the first request after a quiet period sometimes failed with "Failed to open new session" while the warehouse woke up (30 seconds to five minutes on a classic one). Nothing had run yet, so retrying with backoff was safe. We added retries and a small "warehouse is starting" banner, and moving to a serverless warehouse removed most of those errors.

What the agents couldn't learn from a wiki

Each of these failures was easy to fix and took hours to find.

The agents hit every single one. And they kept hitting them after we wrote the fixes down. They only stopped when a check failed on the mistake.

That changed how I see my own role. I think the most valuable work in this project was knowing where the traps were, and turning that into something the agent has to pass.

The solution: how you can skip the hard path

Set up four things before your agent writes its first line.

1. Ten rules in AGENTS.md

Put short rules in an AGENTS.md (or CLAUDE.md) at the repo root, where the agent reads them first:

• Run every query as the user. Unity Catalog is the security boundary.
• user_api_scopes: [sql] goes at the app level. Check effective_user_api_scopes after each deploy.
• Three permissions must line up: app CAN_USE, warehouse CAN_USE, table grants.
• Deploy through the bundle, never a git snapshot.
• An app deploy is bundle deploy plus bundle run.
• No app.yaml. Environment values live in bundle targets.
• Normalize the warehouse path to /sql/1.0/warehouses/<id>.
• Retry session-open errors with backoff, and tell users the warehouse is waking up.
• Fake databases are for tests only.
• Every deploy failure becomes a check.

2. A preflight check before the first pipeline

Turn each trap into a lint rule in a script that exits non-zero on errors: user_api_scopes under config:, git_source, an app.yaml with environment values, a missing lockfile, a pinned registry, omit=optional, a personal /Workspace/Users/... root path, secrets in plain env values, two targets sharing one workspace.

The script should also list what it can't check (secret bindings, scope passthrough, the three grants), so the agent tells you who owns each one.

3. A template that already follows the rules

Generating the app from a reviewed spec is way more reliable than asking an agent to remember forty rules.

Ours asks for the app name, the catalog, warehouse and groups per environment, and the entities and fields. Then it generates an app with OBO sessions, group checks, cold-start retries, auditable writes, a bundle with dev, sqa and prd targets, grant scripts for the data owner, and tests that pass on day one.

4. A fixed order for "I see no data"

Check app access, then the OBO scope, the warehouse, the Unity Catalog grants, the data, and only then the UI.

Most fixes in the first four are admin actions, so the agent should pass them to the owner and leave grants alone.

Soooo, where does this leave me?

After this setup, our deploys passed on the first run, and every "no data" report went through the same steps. The agents stopped asking "why won't it start?" and started asking "which admin owns this grant?"

This project touched all three parts of the path I'm on: a software app, on governed data, built with AI agents.

The agents did write most of the code. What made it work was understanding the platform underneath and writing that down in a way the agents had to follow.

I find that really encouraging. It's work I can learn, and I believe you can too.

If you're feeling that same fear about AI right now, I get it. I think there's still a lot of room for us. The job just looks a bit different.

Huge thanks to the team I built this with - you know who you are :)

Let me know in the comments: what are you learning next? Or which Databricks trap got you the worst?

PS: everything from this article lives in bricksmith, the skill I built so my agents stop re-learning these traps. If it saves you a deploy or two, a ⭐ helps a lot :)
github.com/dcoferraz/bricksmith