Declarative Automation Bundle: Bind an Existing Lakeflow Job

A matching YAML configuration file does not adopt the Lakeflow Job already in your workspace. Bind the bundle resource key to that Lakeflow Job, then let the first deployment update the right ID.

Suppose refresh-orders already runs in production every morning.

An engineer created it manually in the Databricks workspace user interface months ago. It has run history, notifications, a schedule, and a notebook that somebody still edits by hand.

We want the Lakeflow Job in Git. We do not want a second Lakeflow Job, a changed schedule, or a first deployment from somebody's laptop.

The risky step is not generating YAML configuration. It is the first databricks bundle deploy.

A Declarative Automation Bundle tracks deployed objects by ID in a state file in the Databricks workspace. The Databricks CLI does not match a Lakeflow Job by name. If the state file does not map the resource key to the existing Lakeflow Job ID, the Databricks CLI treats the Lakeflow Job as new.

This article adopts one existing scheduled Lakeflow Job into one production bundle target. It has only workspace notebook tasks and no paginated collection with more than 100 items.

I am not moving Lakeflow Jobs between bundles. I am not designing continuous integration, changing run_as, or hardening the Lakeflow Job. Those are separate changes after this adoption succeeds.

Use this only after the bundle passes its development or user acceptance testing suite and existing production release controls. The Lakeflow Job and target already meet Manuka standards.

The four values that must agree

Before I run a bundle command, I write down four values:

• the existing Lakeflow Job ID
• the bundle resource key
• the target name
• the identity that will deploy this target later

Our example uses these values:

Lakeflow Job ID: 6565621249
Resource key: refresh_orders
Target:       prod
Deployer:     the production release service principal

The resource key is the YAML configuration identifier under resources.jobs. It is not the Lakeflow Job display name.

# resources/refresh_orders.job.yml (fragment, not deployable alone)
resources:
  jobs:
    refresh_orders:
      name: refresh-orders

The key identifies the bundle declaration. name is the Lakeflow Job name engineers see in the Databricks workspace.

The target selects a workspace and its deployment state file. By default, the Declarative Automation Bundle uses this workspace root path:

/Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target}

For the refresh_team bundle in the prod target, a release service principal might use:

/Workspace/Users/deploy-prod-<UUID>/.bundle/refresh_team/prod

A developer named Ana uses a different path:

/Workspace/Users/ana@example.com/.bundle/refresh_team/prod

The deployer part changes, so the service principal and Ana use separate deployment state files.

That identity detail changes the adoption sequence. I bind with the same production service principal that will run future production deployments. A bind through a developer's profile records the mapping in that developer's state file. The continuous integration workflow then sees no mapping and can still plan a create.

Databricks recommends a service principal for production deployments. It also recommends granting write access to the production root path only to that principal. Do not use /Shared for that path.

Binding has one narrow effect. It records that refresh_orders refers to Lakeflow Job 6565621249 in this target's state file. It does not apply the YAML configuration to the Lakeflow Job until the next deployment.

The key, target, and ID each answer a different question:

Do not replace the ID with the Lakeflow Job name. A Declarative Automation Bundle stores resource IDs in the workspace state file. It does not use the display name to discover an existing Lakeflow Job.

The deploying identity also needs permission to manage the existing Lakeflow Job. CAN_MANAGE is the Lakeflow Job-level permission that allows an identity to edit the definition and schedule. Confirm that access before the pull request reaches the bind step.

Capture the Lakeflow Job before you turn it into code

I capture the remote definition first. This gives the pull request something concrete to compare with the generated files.

The identity that captures or generates the Lakeflow Job needs CAN_VIEW on the production Lakeflow Job.

JOB_ID=6565621249

databricks jobs get "$JOB_ID" \
  --include-trigger-state \
  --output json > refresh-orders.before.json

The command writes the Lakeflow Job definition and its trigger state to refresh-orders.before.json. Store it in the pull request summary or ticket, not in the repository if it contains sensitive configuration.

This walkthrough assumes the response contains every Lakeflow Job array on its first page. If jobs get returns next_page_token, I use a separate migration and capture every page.

Next, I run the generator from the bundle project root. I keep its generated files uncommitted until the review is complete.

databricks bundle generate job \
  --existing-job-id "$JOB_ID" \
  --key refresh_orders \
  --target prod

The generator creates a Lakeflow Job configuration under resources/. It also downloads its Databricks workspace notebook files under src/.

The generated YAML configuration is evidence. It is not a reason to deploy immediately.

Run as is the identity the Lakeflow Job uses for its tasks. It is separate from the bundle's deploying identity.

I compare the generated files with refresh-orders.before.json. I check the Lakeflow Job name, task keys, notebook paths, schedule, pause state, parameters, notifications, and Run as. I also check every file the Lakeflow Job references.

This is where I stop unrelated improvements. A new timeout, adjusted retry count, or better notification list makes the first deployment harder to reason about. I usually open follow-up pull requests for those changes.

If the Lakeflow Job uses Git source or a task type that the generator does not support, I stop this walkthrough here. I use a separate migration.

The review has one simple rule: the first bundle YAML configuration must describe the Lakeflow Job we already operate. It is not the place to make it better.

I keep refresh-orders.before.json outside the generated resources/ and src/ directories. The saved response records the Lakeflow Job at adoption.

Validate, bind, plan, then deploy

I commit the generated configuration and downloaded source files before binding. Git now holds the proposed definition, but the Databricks workspace Lakeflow Job remains unmanaged.

First, I validate the rendered production configuration:

databricks bundle validate --strict -t prod

Validation checks the bundle configuration. It does not prove that the YAML configuration describes the existing Lakeflow Job. The comparison in the previous section does that work.

Next, I bind the bundle resource key to the remote Lakeflow Job ID. The one-time adoption workflow uses --auto-approve.

Binding is non-interactive and records state only. It does not apply the YAML configuration to the Lakeflow Job.

databricks bundle deployment bind refresh_orders "$JOB_ID" \
  -t prod \
  --auto-approve

I use this one-time adoption exception because a useful plan needs the binding. An unbound plan can only show a create.

The workflow publishes the summary and plan. I review both before approving the deployment. This exception does not change the Lakeflow Job during binding.

Now I confirm the binding and inspect the first deployment plan:

databricks bundle summary -t prod --force-pull -o json
databricks bundle plan -t prod

The summary must map refresh_orders to Lakeflow Job 6565621249. The plan must not show create jobs.refresh_orders. If it does, I stop.

An update is not itself a failure. bundle generate job downloads notebooks and uses bundle paths. That can change a Lakeflow Job task's deployed notebook path.

I verify every planned update against the capture. An unexplained change to the task graph, trigger, Run as, parameters, or notifications stops the deployment.

I review and approve that plan before the workflow runs the first deployment:

databricks bundle deploy -t prod

Run this command through the release path. The first deploy changes the source of truth from workspace edits to the reviewed bundle configuration.

The deliberate pause point sits between bind and deploy. Binding records the association but does not update the Lakeflow Job. I can still stop, fix the YAML configuration, or remove the association.

If I bound the wrong key, ID, or target before the first deployment, I remove only the association:

databricks bundle deployment unbind refresh_orders -t prod

Unbind leaves the Lakeflow Job in the Databricks workspace running. It only removes the link from this bundle state. I do not deploy while the YAML configuration remains unbound, because the next deployment would create a new Lakeflow Job.

After the first deployment, unbind is not a configuration rollback. It does not restore the Lakeflow Job settings that the deploy applied.

To recover with a bundle change, I bind the existing Lakeflow Job again, inspect the plan, then deploy the corrected YAML configuration. I use refresh-orders.before.json as the pre-adoption reference.

What fails when I skip the bind

Consider the tempting shortcut. I generate YAML configuration, review it, and deploy without binding.

The resource key is present in the bundle. The Lakeflow Job name might even match exactly. Neither fact gives the Databricks CLI the existing Lakeflow Job ID.

The deployment state file has no mapping for refresh_orders. The Databricks CLI therefore plans to create the Lakeflow Job declared in YAML configuration.

That can leave two scheduled Lakeflow Jobs. One is the old Databricks workspace user interface-created Lakeflow Job. The other is the new bundle-created Lakeflow Job. Both can point at the same notebook and start work independently.

Binding replaces that uncertainty with an explicit mapping. It says that this key manages this Lakeflow Job ID in this target. The next deployment updates that Lakeflow Job according to the reviewed configuration.

The pattern I recommend

I use one adoption pull request for one existing Lakeflow Job in one target:

1. Capture the Lakeflow Job.
2. Generate and review the configuration without improving it.
3. Validate the target, bind the resource key, and inspect the plan.
4. Deploy through the same release identity.

Positives

• The existing Lakeflow Job keeps its identity instead of being recreated.
• The first plan can expose an unintended Lakeflow Job change before deployment.
• Future deployments use the production service principal's bundle state.

Negatives

• The adoption needs production release credentials before the first bind.
• Manual workspace edits become drift and a later deploy can overwrite them.
• Each target needs its own review and binding to its own existing Lakeflow Job ID.

The negatives are worth accepting. A production Lakeflow Job should have one declared owner and one reviewed deployment path.

Verify the handover

After deployment, I run these three read-only checks from the production release identity:

databricks jobs get "$JOB_ID" --output json
databricks bundle plan -t prod
databricks bundle summary -t prod --force-pull

The first command confirms that Lakeflow Job 6565621249 still exists. The plan should not propose creating a Lakeflow Job. The summary reads the remote state file and gives the reviewer the Databricks workspace link for the managed Lakeflow Job.

I watch the next scheduled run through the existing failure notification path. I expect one run from the same Lakeflow Job ID, using the same schedule. That is the first production measurement after the handover.

Do not use this article for a Lakeflow Spark Declarative Pipeline, Lakeflow Connect gateway, or Lakeflow Job already tracked by another bundle. Each has different state and recovery concerns.