Unity Catalog Schemas in Declarative Automation Bundles Are a Footgun

Why mode: development quietly renames your schemas, why the official escape hatch makes things worse for teams, and four safer patterns to structure your deployments instead.

Last week a teammate ran databricks bundle deploy against our dev workspace.

The command failed with an error about a schema that already existed.

She had not touched that schema. Nobody had. Her code was identical to mine, and my deploy had worked minutes earlier.

This guide walks you through why that happens.

We start with the pieces, we watch the failure happen step by step, and then I show you four patterns that avoid the problem completely. One of them needs no extra tooling at all.

Declarative Automation Bundles in one minute

A Declarative Automation Bundle, or DAB, is a set of YAML files in your repository that describes Databricks resources: jobs, pipelines, dashboards, schemas. You run databricks bundle deploy, and the CLI creates or updates those resources on your workspace.

It is infrastructure as code for the Databricks platform.

The YAML lives in your repo next to your source code. Data engineers use it to ship work without clicking through the workspace UI. Deployment is manual, you run the command when you want it to run.

Since CLI version 1.3.0, deploys run on the direct engine by default. The CLI does not use Terraform anymore. The engine is the part of the CLI that talks to the Databricks APIs and applies your changes.

Meet our example bundle

Everything in this guide uses one small bundle called sales-analytics. It lives in two files:

sales-analytics/
├── databricks.yml
└── resources/
    └── reporting.schema.yml

The top-level file names the bundle and holds the shared settings we will build up: variables and targets.

# databricks.yml
bundle:
  name: sales-analytics

The schema resource inside resources/ appears in the walkthrough section.

Two worlds with opposite rules

When you run databricks bundle deploy, the CLI records what it created in a deployment state file.

With an unmodified setup, that file lands in your user folder on the workspace:

/Workspace/Users/ana@gmail.com/.bundle/sales-analytics/dev/state/deployment.json

That location is a default, not a law. You can move it with the workspace.root_path setting, though most teams never change it. Production mode even validates that your paths do not point at one specific user.

The state file records what you deployed.

My state file sits under my own email.

Yours sits under yours.

Two developers deploying the same repo never share a state file.

Unity Catalog is Databricks' governance layer.

A schema in Unity Catalog is a named container for tables and views, and it lives in a catalog.

Here is the important part: a catalog and its schemas belong to the metastore, not to any person.

If a schema named reporting exists in catalog main, every user sees that same main.reporting.

There is exactly one of it.

So bundles mix two worlds with opposite rules:

• State files are private. One per developer.
• Schemas are global. One per metastore.

Most of the time these worlds never collide, because development mode adds a safety layer between them.

Let us look at that layer, because it is also where the footgun lives.

What each deployment mode does

A target is a named deployment environment, such as dev, uat, or prod. You pick one with the -t flag.

A well-built bundle has targets, and each target sets its mode:

# databricks.yml
targets:
  dev:
    mode: development
    default: true
  uat: {}
  prod:
    mode: production

The mode decides how a deploy behaves. Here is the full picture:

The schema prefix is not the pretty bracketed string. Schema names allow only letters, numbers, and underscores.

So the CLI removes brackets and spaces: [dev ana] becomes dev_ana_, and my schema reporting lands as dev_ana_reporting.

Second, the prefix comes from your identity, so every developer gets their own copy. That is isolation working as designed, and it only happens where humans deploy side by side.

So far so good... right?

The footgun

Your notebooks and pipelines usually expect a stable schema name. Maybe a query hardcodes main.reporting. After a dev-mode deploy, that schema is called dev_ana_reporting, and the query breaks.

This surprises everyone the first time. It looks like a bug. It is not: it is the safety layer doing its job.

The obvious next move is to search the docs, find an escape hatch, and switch it off:

# databricks.yml
experimental:
  skip_name_prefix_for_schema: true

The flag does exactly what it says. Jobs keep their prefix. Schemas lose theirs. Every developer now deploys a schema called plain reporting into catalog main, no matter who runs the deploy.

It feels harmless, but it is not!

Remember our two worlds: private state files, global schemas.

With the flag on, every developer's private state file claims the same public object.

Here is what happens next:

1. Ana deploys first. The CLI creates schema main.reporting. No prefix. Ana's state file records that she created it. The deploy succeeds.
2. Ben deploys ten minutes later. His state file is empty, so the CLI tries to create the schema fresh. It sends a create request for main.reporting.
3. Databricks refuses. The schema already exists, and Ben does not own it. On the direct engine the error names the resource and carries the API message, something like Error: cannot create resources.schemas.reporting: Schema 'main.reporting' already exists. Either way, his deploy fails.
4. Ben asks Ana to delete her deployment so he can test. Ana runs databricks bundle destroy -t dev. Bundles always force-delete schemas they manage, warning first that underlying data may be lost. Her state file says she owns main.reporting, so the CLI drops it, tables included. The schema vanishes for everyone.
5. Now Ben deploys successfully and owns the very schema Ana needs. When Ana redeploys, she hits the same wall Ben did.

The second hazard: toggling the flag

There is a second hazard that most people miss.

Inside a bundle, a resource's name is its identity. Change the effective name, and the CLI treats it as a brand-new resource.

So if you deploy with prefixes, then add the flag later, the CLI creates unprefixed schemas and deletes the prefixed ones.

Your data goes through that cycle too.

If you switch this flag on or off, the next deploy can delete schemas and their data. It is not a small settings change.

One reassurance before the patterns: this failure is a development-mode story.

Most teams deploy to uat and prod from a build system using one robot account, called a service principal. One robot account means no collision.

I lived through version one of this story... taking turns failing!

Do not reach for the flag. Reach for one of these four patterns instead.

Pattern 1: Accept the prefix and plan your code around it

The simplest pattern needs zero extra tooling. Leave everything as Databricks designed it, and make your code read the schema name at runtime instead of hardcoding it.

The trick is a prefix variable with a different value per target.

Dev derives it from your username, uat uses a fixed string, and prod uses none. The short_name substitution is your username without the domain, so ana@gmail.com becomes ana:

# databricks.yml
variables:
  catalog:
    description: Catalog for this deployment.
    default: main
  schema_prefix:
    description: Prefix for schema names in this environment.
    default: ""

targets:
  dev:
    mode: development
    default: true
    variables:
      schema_prefix: dev_${workspace.current_user.short_name}_
  uat:
    variables:
      schema_prefix: uat_
  prod:
    mode: production

Every consumer of the schema builds its name from that variable:

# resources/ingest.job.yml
resources:
  jobs:
    ingest:
      parameters:
        - name: source_catalog
          default: ${var.catalog}
        - name: source_schema
          default: ${var.schema_prefix}reporting

One YAML file, three outcomes. Ana deploying to dev passes her notebook dev_ana_reporting. The same job deployed to uat reads uat_reporting. In prod it reads plain reporting. Your notebook calls dbutils.widgets.get("source_schema") and never cares which stage triggered the run.

The schema resource itself keeps its plain name, name: reporting. The CLI adds its own prefix on top in dev.

Never put ${var.schema_prefix} inside the resource name. The CLI would stack its prefix on top of yours, and you would get a schema called dev_ana_dev_ana_reporting.

Positives

• Nothing new to build, host, or permission. It is pure YAML you already have.
• Full lifecycle stays in one place. Deploy, redeploy, and destroy all work.
• Isolation is automatic. Nobody can collide with anybody.
• Destroy cleans up everything, including the schemas.

Negatives

• Every consumer of the schema must take the name dynamically. Hardcoded queries break.
• You rely on the exact shape of the sanitised prefix, dev_<username>_. It is stable today, but it is a convention, not a contract.
• Prefixed names look odd in Catalog Explorer and confuse newcomers.
• Some external tools cannot accept dynamic names, which blocks this pattern.

Pick this pattern when your stack already passes configuration around, which good data pipelines do anyway.

Pattern 2: Declare schemas outside the bundle

Schemas often outlive the code that fills them. A natural split is to manage schemas somewhere else entirely: a Terraform module, a small provisioning script, or even a one-off manual setup for stable internal schemas.

# schemas.tf
resource "databricks_schema" "reporting" {
  catalog_name = "main"
  name         = "reporting"
}

Your product bundle keeps deploying jobs and pipelines, and simply points at main.reporting, which nobody's deploy mode can rename.

Set permissions on the schema in the same Terraform module with a databricks_grants resource, so access control travels with the schema.

Positives

• Stable, clean names in every environment. Code stays hardcoded and boring.
• The product bundle contains no schemas, so the prefix question never arises.
• Terraform brings plan previews and catches drift, which is when someone changes schemas outside the tool.

Negatives

• Two sources of truth. New joiners must learn where each thing lives.
• Manual or scripted setup drifts and does not scale across environments.
• Terraform adds state management, credentials, and a second tool to operate.
• Deleting the bundle leaves the schemas behind. Cleanup lives in the other tool now.

Pick this pattern when a platform team already owns governance as code, or when schemas are genuinely shared infrastructure rather than project output.

Pattern 3: Put schemas in their own bundle

A middle path keeps everything as bundles but splits ownership. Create a second, tiny bundle that declares only the UC objects. Its targets never set mode: development, so nothing ever gets prefixed:

# sales-analytics-schemas/databricks.yml
bundle:
  name: sales-analytics-schemas

targets:
  dev: {}
  uat: {}
  prod: {}

Deploy it once per environment from CI using a service principal. Product developers never touch it, and because exactly one identity deploys it, the collision story from earlier cannot happen. Even the skip flag would be safe here, though you will not need it.

Positives

• Stable names with full infrastructure-as-code discipline. Best of both worlds.
• One deployer identity removes the multi-developer hazard completely.
• Clear ownership boundary: platform owns schemas, teams own compute and code.

Negatives

• A second bundle means a second pipeline, repo or folder, and set of secrets.
• Ordering matters. Schemas must exist before product deploys reference them.
• Someone will eventually run it locally without noticing the missing dev mode.

Pick this pattern when you want IaC end to end and can afford a small platform-style repo.

Pattern 4: Give every developer a catalog

If developers need clean schema names and full self-service, flip the isolation up one level. Instead of letting dev mode rename schemas, give each developer their own catalog in dev, while uat and prod keep shared catalogs:

# databricks.yml
variables:
  catalog:
    description: Catalog for this deployment.
    default: reporting

targets:
  dev:
    mode: development
    default: true
    experimental:
      skip_name_prefix_for_schema: true
    variables:
      catalog: dev_${workspace.current_user.short_name}
  uat:
    variables:
      catalog: uat_reporting
  prod:
    mode: production

Wait, did I not just tell you to avoid that flag? Here it is safe, and the catalog is the reason.

The flag removes the schema prefix, so the schema deploys as plain reporting. Normally that invites collisions. In this pattern it cannot, because Ana deploys only into dev_ana and Ben only into dev_ben. The private catalog does the isolating that the prefix would otherwise do.

Positives

• Clean schema names and complete isolation at the same time.
• Deterministic per-person catalogs make debugging and cleanup easy.
• Works with existing code that expects fixed relative schema names.

Negatives

• Each developer needs CREATE CATALOG on the metastore, or a platform script must pre-provision catalogs.
• Catalogs accumulate. You need a naming convention and a cleanup habit.
• Grants must be set per catalog, which is more governance surface to maintain.

Pick this pattern for teams that iterate fast on data products and want sandbox-per-developer ergonomics.

Choosing between them

My default recommendation: start with Pattern 1 while the team is small, and graduate to Pattern 3 once a platform function exists.

Reserve skip_name_prefix_for_schema for two cases only: bundles that exactly one identity will ever deploy, such as a CI-owned pipeline, and the isolated personal catalogs in Pattern 4. Never toggle it on a bundle that already manages data without a migration plan.

One warning that applies to all four patterns: switching between them renames schemas, and in a bundle a rename means delete and create. Migrate your data before you change approach.

Wrapping up

Treat schemas as a boundary decision, not an accident of YAML layout.

Accept the prefix and parameterise, move schemas to Terraform, give them a dedicated single-deployer bundle, or isolate at the catalog level.

Any of the four beats an afternoon of two engineers taking turns breaking each other's workspace.

Not sure where your bundle stands today? Run databricks bundle plan -t dev and look for schema resources. Then decide which side of the boundary you want them on.