Self-Hosted Databricks Alternative: Homelab Data Lakehouse (AI/ML Data Factory)

Self-Hosted Databricks Alternative: Homelab Data Lakehouse

Part of the homelab build series. Trino, Apache Iceberg, Project Nessie, dbt, Spark, MLflow
and OpenMetadata on RKE2, with highly available state on Ceph and CloudNativePG, secrets from
Vault and sign-in through Keycloak. Raw data comes in at one door, gets built and inspected
on the line, and leaves as tables and models I can trust. Nobody sends an invoice.

Introduction

A factory you rent by the hour, or one you own

Raw data is worth about as much as raw steel. Nobody wants it until someone has cut it,
joined it, checked it and stamped it.

Snowflake, Databricks and BigQuery will rent you an excellent factory for that, and the
meter runs for as long as the lights are on. I wanted the factory without the meter, so I
built one on three mini-PCs in my homelab, on the same RKE2 cluster as everything else in
this series. Every station is a real open-source tool doing a real job.

It's built to make one thing: machine-learning models trained on data I can name down to
the exact version. One is already serving, a football predictor that a scheduled pipeline
refits and re-tests every day.

Here is the floor, in the order the material moves:

• The loading dock: Ceph object storage. Everything that arrives lands here.
• The stock ledger: Project Nessie. Every table is an Apache Iceberg table, and Nessie records every version it has ever had. It's Git for data.
• The assembly line and quality control: dbt. Tables are built and tested on a branch, so a batch that fails inspection never reaches the shelf.
• The front counter: Trino. Anyone with access asks for what they need in plain SQL.
• The showroom: Superset. Charts and dashboards over the finished tables, for people who'd rather not write SQL. It asks for data in their name, never its own.
• The records office: OpenMetadata. It traces any table back to its raw material.
• The machine shop: Kubeflow. Notebooks, pipelines and hyperparameter searches, on GPUs when a job needs them.
• Finished goods: MLflow and KServe. Every run is tracked, and the model that passes its checks is served over HTTP.

The utilities were already in the building: Ceph, CloudNativePG, Vault, Keycloak, Istio and
ArgoCD.

It's highly available where it holds state. Postgres runs three instances, one in each
datacenter, the ledger runs two replicas and search runs three nodes. Trino, Superset,
OpenMetadata and MLflow run a single replica each, and I come back to that below. One bay is still
empty: a Hermes AI copilot that will read the lineage over OpenMetadata's MCP
interface, waiting on a DGX Spark that hasn't arrived.

If you know Databricks, you already know the layout:

It's an open-source equivalent, not Databricks, which is a product and a trademark, and
there's no affiliation. This is the second design: the first used Hive Metastore and
Airflow, and I say below why I replaced both.

The brief

The brief I set myself: an MLOps and big-data platform on the existing homelab RKE2 cluster
that a real data team could plausibly use. The homelab's existing backends (Rook-Ceph,
CloudNativePG, Vault, Keycloak) had to thread through every component, and each piece had
to land in GitOps and come back after a node reboot with nobody touching it.

What landed is two tiers. The lakehouse: Ceph RGW object storage, Apache Iceberg
tables, a Project Nessie catalog (state in CNPG nessie_db), Trino from Stackable's
coordinated 25.3.0 release, dbt-trino for tested marts, and OpenMetadata for
governance, with the Kubeflow spark-operator installed for the day the data outgrows
SQL. MLOps on top: MLflow Tracking and the Model
Registry, backed by CNPG mlflow_backend_db and the mlflow-artifacts bucket, behind
oauth2-proxy for the browser and reachable directly over in-cluster DNS from Kubeflow
notebooks.

The constraints that shaped every decision:

• No HDFS, no ZooKeeper. Ceph RGW is my S3, and nothing in the lakehouse needs ZK. The RAM saving alone justified leaving both out; the complexity avoided mattered more.
• No bundled databases. Every chart wants to ship its own MySQL or Postgres or Mongo. I ripped each one out of the lakehouse and MLflow charts and pointed them at CNPG. There is one Postgres cluster for this stack (postgres-mlops-cnpg-cluster, three instances across three DCs, each on its own zone-local volume) and it hosts the lakehouse and MLflow databases. (Kubeflow is the exception; it keeps its bundled MySQL, katib-mysql and Model Registry's Postgres for now.)
• No v1/Secret literals in Git. Every credential (application user passwords, OAuth2 client secrets, OIDC cookie keys) flows from Terraform into a Vault path, then into a VaultStaticSecret CR, then into the destination namespace. Kustomize may declare a placeholder Secret to keep references valid, but VSO always wins on the data values.
• Mirror production patterns, even when it's harder. Real OIDC, real mTLS, real RBAC. No "developer convenience" backdoors.

The rest of this article covers how each piece lands, what I cut, and which
traps cost the most time.

The architecture

If the word is new, the lakehouse is the third step in a short history. A warehouse
gives fast SQL over the tables you planned for, in a format you rent. A lake keeps any
file cheaply but has no table rules, so teams copy the data into a warehouse anyway and
end up with two versions of the truth. A lakehouse puts the table rules (Iceberg) and
the versions (Nessie) on the open files themselves, so every engine reads one copy.

Warehouse, lake, lakehouse: a warehouse sells storage and engine together, a lake plus a warehouse keeps two copies, and the lakehouse built here keeps one copy of open files in Ceph with Iceberg tables and Nessie versions read by Trino, dbt, Kubeflow and MLflow

Here is how this one is put together:

The data factory:** dbt, Trino, Nessie, Kubeflow, MLflow and KServe on the line, Superset and OpenMetadata above it behind Keycloak, and Ceph, CloudNativePG, Vault, Istio and Argo CD underneath

With five guided views: the query path, the daily pipeline, governance, train-and-serve,
and where secrets come from. Blog platforms can't embed the HTML, so host it and link it.

Kubeflow appears here as one station, the machine shop; how I changed it from upstream
is in the SSO section. Read the diagram bottom-up, because that's the order the
dependencies run:

• Foundation. Ceph RGW holds every byte of table data (s3://mlops-data-lake, TLS-only at s3.georgehomelab.com). One CloudNativePG cluster holds the lakehouse and MLflow stores (nessie_db, mlflow_backend_db, openmetadata_db). Vault + VSO hand out every credential.
• Catalog. Project Nessie answers "where does table X live, and at which commit?" over Iceberg REST, versioning the whole catalog like Git.
• Query and transform. Trino reads and writes the Iceberg tables; dbt builds and tests the marts through Trino every morning, on a Nessie branch dbt-audit that only merges into main if the tests pass.
• Govern. OpenMetadata 2.0.2 (search on its own OpenSearch 3.4.0) crawls Trino at 10:00 and ingests dbt's artifacts at 10:15. That's where the lineage comes from. People reach it through Keycloak (Google or GitHub).
• Consume. Superset 6.1 gives people charts, dashboards and a SQL editor over Trino. It signs people in through Keycloak and asks Trino to run each query as the person, so Trino's rules and its audit table see a name, never a shared login.
• Train and serve. MLflow tracks runs in Postgres and models in RGW; a KServe InferenceService serves the promoted model.

Everything in the foundation layer predates this work. Most of my effort went into
making sure the upper layers didn't reinvent any of it. The one deliberate exception is
OpenMetadata's own OpenSearch; the going-live section explains why that stopped being
optional.

Stackable operators, and what I dropped

Stackable ships a coordinated release of operators for Hadoop-ecosystem services. It's
tempting to take all of them. I only took the parts that fit:

The Stackable operators pin to 25.3.0 and release as a set, so I bump them as a set
(never Trino alone). Project Nessie and OpenMetadata are not Stackable products; they
deploy from their own Helm charts, the price of a best-of-breed catalog and governance layer.

The secret-operator needs explaining. The workload operators' admission webhooks
reference the SecretClass CRD, and without it workload CR creation fails. So I install
the operator but create no SecretClass of my own. The chart's default tls SecretClass
issues Trino's internal (autoTls) certificates, and the dbt and ingestion pods mount its
CA to verify them. Credentials never go through it; they're plain v1/Secrets synced by
VSO from Vault.

Stackable workloads also brought the first Istio trap. Under STRICT mTLS (the cluster
default), the API server's webhook calls can't reach the operator pods, because the API
server has no mesh client cert. The fix is a PeerAuthentication: PERMISSIVE in the
mlops namespace. Don't turn off injection; relax the mode for that namespace and
nothing else.

Storage: Rook-Ceph all the way down

The homelab already runs Rook-Ceph with four storage classes: ceph-block (RBD, RWO,
three copies, the default for notebook PVCs and most stateful pods), ceph-block-pg-local
(RBD, one copy, pinned to a datacenter, for Postgres only), ceph-filesystem (CephFS, RWX,
only when a workload really needs shared scratch) and ceph-bucket (RGW, S3), the data lake.

RGW is TLS-only. Clients use https://s3.georgehomelab.com,
which terminates on the Istio gateway with a public Let's Encrypt certificate (no private
CA to distribute), and path-style addressing still works. The old plain-HTTP
…rook-ceph.svc:80 endpoint is gone.

Buckets come from Rook's ObjectBucketClaim CR. The OBC controller provisions the bucket
on RGW and creates a per-bucket Secret with AWS_ACCESS_KEY_ID /
AWS_SECRET_ACCESS_KEY keys, named after the claim: Spark reads mlops-data-lake, MLflow
reads mlflow-artifacts, KFP reads kfp-artifacts. Nothing fans out from a single root key.

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: mlops-data-lake
  namespace: mlops
spec:
  bucketName: mlops-data-lake
  storageClassName: ceph-bucket

Apply that and the bucket, its owner user and the Secret all exist, ready for envFrom.

The Ceph dashboard → Object Gateway → Buckets: mlops-data-lake, mlflow-artifacts and kfp-artifacts, each with its own OBC-issued owner.

Databases: one CNPG cluster, many logical databases

The easy path is to leave every chart's postgresql.enabled: true on and accept the
extra StatefulSets. I didn't: backups, upgrades, monitoring and HA belong on one tier, not
nine. The homelab's CNPG pattern already runs Keycloak, SonarQube, Tekton and others, so
one more cluster meant copying an existing manifest (trimmed here; the real one adds
managed roles, affinity and resources):

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgres-mlops-cnpg-cluster
  namespace: mlops
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:17
  bootstrap:
    initdb:
      database: mlops_placeholder       # real DBs come from Database CRs
      owner: postgres
      secret:
        name: mlops-cnpg-bootstrap      # VSO-synced from Vault
  storage:
    storageClass: ceph-block-pg-local   # one Ceph copy, in the instance's own DC
    size: 40Gi
  postgresql:
    parameters:
      max_connections: "200"
  backup:
    retentionPolicy: "7d"
    barmanObjectStore:
      destinationPath: "s3://cnpg-backups/postgres-mlops-cnpg-cluster"   # Ceph RGW

Three instances, one per DC, automatic failover, and continuous WAL archival to RGW.

The storage class is worth a sentence. Postgres already keeps three copies of the data,
one per instance. On the default ceph-block, Ceph kept three copies of each of those,
nine in all, and every commit waited on a write to another datacenter. So Postgres now
sits on ceph-block-pg-local: one Ceph copy, in the same datacenter as the instance.
Postgres does the replicating and Ceph just holds the disk. A 4K synced write on a
database node went from about 34 ms to between 15 and 24 ms. The price is that losing a
drive loses that instance, and the operator has to clone a new one from the primary.
CNPG's Database CR (its per-logical-DB declaration) provisions one logical database per
app, each with its own owner role whose password lives at its own Vault path:

The lakehouse pivot added nessie_db and openmetadata_db with a short Database CR
each, and no new stateful tier. The old hive_metastore_db and airflow_metadata_db
(and their hive / airflow roles) are retained but no longer used: their Database
CRs set databaseReclaimPolicy: retain, so nothing drops them by accident.

Each app sees only its own DB, with its own creds. The superuser never lands on
an application Pod.

kubectl get database.postgresql.cnpg.io -n mlops: one Database per app on the single postgres-mlops-cnpg-cluster, all APPLIED true, including the retained (unused) hive_metastore_db and airflow_metadata_db.

Two Istio nuances are specific to CNPG. First, its istio-injected pods cycle their
sidecars whenever Istio config changes, so the READY column bounces between 1/2 and
2/2 and the operator reports READY < INSTANCES. Postgres is fine; it's sidecar churn.
Don't triage CNPG; bump the sidecar resources or ignore it.

Second, any istio-injected namespace must run at PSS=privileged, because istio-init
needs NET_ADMIN/NET_RAW. CNPG's initdb Job lives in such a namespace. If PSS is
baseline, the Job goes FailedCreate with no obvious cause. That's why the mlops
namespace is PSS=privileged.

Secrets: Vault all the way down

Vault is the single source of truth for credentials, and Terraform is the only thing that
writes to it. The chain for any credential: Terraform reads the value from a gitignored
.secrets.auto.tfvars and writes a Vault KV-v2 path; a VaultStaticSecret CR names
that path, the Vault role and the Secret to produce; VSO reconciles it hourly into a
plain v1/Secret (optionally reshaping keys with a template); the workload mounts it.

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: mlops-mlflow-db
  namespace: mlops
spec:
  vaultAuthRef: mlops-mlflow
  mount: secret
  type: kv-v2
  path: homelab/mlops/mlflow-db
  refreshAfter: 1h
  destination:
    name: mlops-mlflow-db
    create: true
    overwrite: true

Don't drop the overwrite: true. If the same Secret is referenced from both a kustomize
secretGenerator (with placeholder data) and a VSO sync, every kubectl apply -k briefly
stomps VSO's data with the kustomize placeholders. With overwrite: true on the VSS, VSO
puts the real values back on its next reconcile. Without it, oauth2-proxy gets
unauthorized_client from Keycloak and you spend two hours blaming the realm import.

Each app has its own ServiceAccount, its own Vault role (Kubernetes auth, policy scoped to
its own paths) and its own VSS, so no credential crosses an app boundary.

The Vault UI → secret/ → homelab/mlops/: the list of per-app paths (mlflow-db, nessie-db, openmetadata-db, …). ⚠️ Show the path list only; .

SSO: per-app oauth2-proxy with Keycloak upstream

Every UI in this stack sits behind Keycloak, using the standard
Authorization-Code-with-PKCE flow. For MLflow: the Istio gateway routes
mlflow.georgehomelab.com to its oauth2-proxy, which looks for the _oauth2_proxy_mlflow
cookie. Without it, the browser goes to
https://keycloak.georgehomelab.com/realms/mlflow/protocol/openid-connect/auth?..., signs in
with Google, and comes back to /oauth2/callback. oauth2-proxy validates the token, sets the
cookie, and from then on reverse-proxies to mlflow.mlops.svc.cluster.local:5000 with
X-Forwarded-Email added.

Each app gets its own realm too: MLflow's proxy uses realm mlflow, and OpenMetadata
(which speaks OIDC natively) uses realm openmetadata.

The decision that isn't obvious is running one oauth2-proxy per app rather than one
shared proxy. One shared proxy means one shared cookie. A leaked MLflow session becomes a
leaked session for OpenMetadata, Kiali, Hubble UI and every other gated app at once.
Per-app instances cost a little RAM (oauth2-proxy is ~30 MiB) and buy real session
isolation. Each overlay sets OAUTH2_PROXY_COOKIE_NAME (_oauth2_proxy_<app>) and
OAUTH2_PROXY_COOKIE_DOMAINS (its own host). The base values use a default that collides,
so without the override the first app to set a cookie "captures" it and later apps return

1. The per-app client secret follows the usual chain: Terraform writes Vault, a VaultStaticSecret syncs it into the namespace, and oauth2-proxy reads it via envFrom.

Two more lessons came out of this. In oauth2-proxy's proxy mode (which I use for
MLflow), the user email reaches the upstream as X-Forwarded-Email, not
X-Auth-Request-Email. The latter is a response header for auth_request-style
integrations. Get it wrong and the upstream app sees no user identity even though
authentication "worked."

And Keycloak doesn't substitute environment variables at realm import. The realm JSON
includes the oauth2-proxy client, but its secret is the literal placeholder string
${TOOL_OAUTH2_PROXY_CLIENT_SECRET}. Every new realm needs a post-import
kcadm update clients/<id> -s secret=<actual> patch with the same value that's in Vault.
post-deploy-fixes.sh automates it, but "Invalid client credentials" on a fresh realm
still catches you out the first time.

The Keycloak sign-in page for the mlflow realm, with the "Sign in with Google" and "GitHub" buttons. Every UI in this stack sends you to a page like this first.

Brownfield Kubeflow: the same pattern, applied to a stack that ships its own auth

Upstream Kubeflow ships its own login stack: Dex as the identity provider, with
oauth2-proxy acting as an Istio ext_authz check in front of it. I don't run Dex at all.

What changed, checked against the live cluster:

• Dex → Keycloak. Login goes Google → Keycloak realm kubeflow → oauth2-proxy (kubeflow-sso/oauth2-proxy-kubeflow, 2 replicas) in reverse-proxy mode, with one --upstream per app. The upstream per-app hosts: ['*'] VirtualServices are deleted.
• The user header. Every app reads X-Forwarded-Email instead of upstream's kubeflow-userid. That's nine separate places to patch, and all of them must agree.
• Authorization. The AuthorizationPolicies are path-based (paths: ["/*"]) instead of principal-based, because the mesh is PERMISSIVE. /notebook/* is gated by a CUSTOM ext_authz policy on the gateway.
• Istio. The cluster's shared Istio, with its own istio-system/kubeflow-gateway and a Let's Encrypt certificate.
• Storage. KFP artifacts go to an ObjectBucketClaim on Ceph RGW instead of the bundled SeaweedFS/MinIO, with the secret from Vault via VSO; every PVC is on ceph-block.
• GPUs. The NVIDIA gpu-operator runs three RTX A1000 cards, and notebooks claim one each.
• Still upstream. KFP and ML Metadata run on the bundled MySQL (a CNPG database exists for them but nothing uses it yet), Katib keeps katib-mysql, and Model Registry keeps its own Postgres. The bundled MySQL is at least backed up now: a daily CronJob dumps it to its own bucket on RGW.

The catalog: Project Nessie (Iceberg REST + Git-for-data)

The catalog is the small service everything else depends on. It remembers that "table events_curated
is an Iceberg table whose current metadata lives at s3://mlops-data-lake/warehouse/...,
with these columns, these snapshots, these partitions." Replacing the Hive Metastore with
Project Nessie was the main reason for the pivot.

Why Nessie over HMS:

• It's an Iceberg REST catalog, the modern standard. Trino, Spark and dbt all speak Iceberg REST; HMS's Thrift is the legacy path.
• Git-for-data. Nessie versions the entire catalog: branches, tags, commits, merges. You can cut a branch, run an ETL job, validate it, and merge atomically, which gives you multi-table transactions HMS can't do. On an ML platform that means reproducible data: tag the exact catalog state a model trained on. Both halves are in real use: dbt builds on a branch, and the football model trains on a tag (more on both below).
• Credential vending. Nessie holds the S3 creds and vends them to the engines when they load a table. (Without STS on Ceph, "vending" hands out the same bucket-owner key, and as the going-live section shows, vending alone isn't enough for Trino to create a table.)

Nessie deploys from its own Helm chart (charts.projectnessie.org), not Stackable. Its
version store is Postgres (CNPG nessie_db), and its warehouse is the same
mlops-data-lake bucket Spark writes to:

versionStoreType: JDBC2
jdbc:
  jdbcUrl: "jdbc:postgresql://postgres-mlops-cnpg-cluster-rw.mlops.svc:5432/nessie_db"
  secret:
    name: mlops-nessie-db
    username: username
    password: password
catalog:
  enabled: true
  iceberg:
    defaultWarehouse: warehouse
    warehouses:
      - name: warehouse
        location: s3://mlops-data-lake/warehouse
  storage:
    s3:
      defaultOptions:
        endpoint: https://s3.georgehomelab.com
        pathStyleAccess: true
        accessKeySecret:
          name: mlops-data-lake
          awsAccessKeyId: AWS_ACCESS_KEY_ID
          awsSecretAccessKey: AWS_SECRET_ACCESS_KEY

Trino reaches it at http://nessie.mlops.svc:19120/iceberg. Spark uses
org.apache.iceberg.nessie.NessieCatalog against .../api/v2 and can create and merge
branches from a job.

The Nessie UI on main: the commit log and the namespaces (raw, raw_football, marts_staging, marts_marts, marts_football) that dbt and Trino have created.

Why Nessie and not lakeFS? Nessie already gives the Iceberg tables Git-style
branches, tags and atomic multi-table commits; a second version-control layer over the
same tables would be two sources of truth for one question. lakeFS earns its place for
unstructured files (I have none yet), and its Iceberg REST catalog, SSO and RBAC are
paid-edition features. So I'll revisit it as a replacement, never an addition, if
Nessie stalls or unstructured data shows up.

Garbage collection. Versioning has a cost that's easy to miss: every dbt build and
pipeline publish replaces table snapshots, and the old files stay forever unless something
removes them. Nessie's separate GC tool runs as a weekly CronJob (Sundays, 13:00 UTC).

It marks the files still reachable (30 days of history per branch, and just the tagged
commit for each train-football-… tag) and treats the rest of each table's files as
candidates. It only looks under Iceberg table locations, so the landing Parquet is out of
its reach. It runs with --defer-deletes for now, recording candidates in nessie_db
until I've reviewed a few runs. The first walked 3 references and 131 commits and found 8
candidates, all old table metadata.

A daily tag-retention job sits in front of it, so the pins don't pile up either. It
keeps any tag a Model Registry version names, plus the last 30 days, and is also in
dry-run mode for now.

MLflow: tracking and model registry

MLflow is the simplest part of the stack to deploy and the one I use most day to day.
The chart is the official one from the MLflow repo, vendored at mlflow/charts/mlflow
(chart 0.1.0, appVersion 3.12.0). It takes a Postgres backend and an S3 artifact store.

values.yaml excerpt (trimmed):

mlflow:
  backendStoreUriFrom:
    secretKeyRef:
      name: mlops-mlflow-db          # VSO builds the `uri` key from Vault
      key: uri
  defaultArtifactRoot: mlflow-artifacts:/       # clients use the proxy scheme
  artifactsDestination: s3://mlflow-artifacts/  # the server does the S3 I/O

env:
  - name: MLFLOW_SERVER_ENABLE_JOB_EXECUTION
    value: "false"
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: mlflow-artifacts
        key: AWS_ACCESS_KEY_ID
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: mlflow-artifacts
        key: AWS_SECRET_ACCESS_KEY
  - name: MLFLOW_S3_ENDPOINT_URL
    value: https://s3.georgehomelab.com
  - name: MLFLOW_S3_IGNORE_TLS
    value: "false"

Artifacts are proxied: clients log to mlflow-artifacts:/ and the server writes to the
bucket with its own OBC credentials, so notebooks only need the tracking URI, not S3 keys.
The MLFLOW_S3_ENDPOINT_URL override points the server at RGW instead of AWS. The
endpoint has a public certificate, so MLFLOW_S3_IGNORE_TLS stays false: TLS is
genuinely verified. Path-style addressing works because the host is in the
CephObjectStore's hosting.dnsNames.

MLFLOW_SERVER_ENABLE_JOB_EXECUTION=false is there because MLflow 3.x's job subsystem is
on by default and starts eight huey consumers, which ate about 1.5Gi while completely idle
and kept OOM-killing the pod. I don't use server-side jobs, so turning it off beat raising
the limit.

The MLflow UI sits behind an oauth2-proxy in the mlflow-sso namespace, with the
per-app cookie scoping from the SSO section. Inside the cluster (a Kubeflow notebook
in a kubeflow-user-* ns), the tracking client talks to the service directly:

import mlflow
mlflow.set_tracking_uri("http://mlflow.mlops.svc.cluster.local:5000")
mlflow.set_experiment("homelab-iris-v1")
with mlflow.start_run():
    mlflow.log_param("lr", 1e-3)
    mlflow.log_metric("val_acc", 0.97)
    mlflow.sklearn.log_model(model, "model")

There's no SSO round-trip on the SDK path; any pod the Network Policy allow-lists can reach
the in-cluster service. Only the browser UI goes through Keycloak.

The MLflow UI → experiment football-match-outcome: the daily runs with wf_logloss and deploy_approved as columns.

Trino, dbt and OpenMetadata: query, transform, govern

v1 had an Airflow tier here, which the pivot deleted. Airflow isn't bad, but on a homelab
with one operator a persistent scheduler is a lot of surface area for what's really three
jobs a day. In its place is a query/transform/govern tier that does more and needs less
babysitting.

Trino: SQL over Iceberg. The Stackable TrinoCluster runs coordinator-only: workers
are scaled to 0 and node-scheduler.include-coordinator is true, so the coordinator
runs the splits itself. No app node has room for a 4Gi/1-CPU worker right now; workers
come back with more hardware. Trino queries Iceberg through the Nessie REST catalog. In
25.3.0 the first-class iceberg connector models the Hive metastore, so the REST wiring
goes through the generic connector.

Nessie vends credentials for reads, but Trino also needs its own S3 keys to create
tables (more on that in the going-live section). They come from the Rook bucket Secret via
valueFromSecret, never as literals:

spec:
  connector:
    generic:
      connectorName: iceberg
      properties:
        iceberg.catalog.type:
          value: "rest"
        iceberg.rest-catalog.uri:
          value: "http://nessie.mlops.svc:19120/iceberg/"
        iceberg.rest-catalog.vended-credentials-enabled:
          value: "true"
        s3.endpoint:
          value: "https://s3.georgehomelab.com"
        s3.path-style-access:
          value: "true"
        s3.aws-access-key:
          valueFromSecret:
            name: mlops-data-lake
            key: AWS_ACCESS_KEY_ID
        s3.aws-secret-key:
          valueFromSecret:
            name: mlops-data-lake
            key: AWS_SECRET_ACCESS_KEY

dbt: tested transformations. dbt-trino turns raw Iceberg tables into modeled,
tested, documented marts. It runs as a CronJob at 09:30 UTC. The project ships
as a kustomize-generated dbt-project ConfigMap (no git clone at runtime), and a
non-root container runs dbt build. dbt's not_null, unique, accepted_values and
accepted_range tests are the data-quality gate; there's no separate Soda/GE.

The dbt profile needs no secret at all. dbt proves who it is with a short-lived
Kubernetes token for its own service account, mounted into the pod and sent as a JWT,
and Trino's rules then decide what dbt-runner may read and write (how that works is
the subject of the RBAC article in this series). It also verifies Trino's certificate
against the Stackable tls CA, mounted per pod. After each run it publishes manifest.json, catalog.json and
run_results.json to a small CephFS volume for OpenMetadata.

dbt writes on a branch, then publishes (write-audit-publish). At first, each
dbt build landed in main as roughly 15 separate Nessie commits (__dbt_tmp renames,
__dbt_backup drops), so a query at the wrong moment could catch a half-built mart. Now
the 09:30 CronJob does three things:

• It resets a Nessie branch, dbt-audit, to main's head through Nessie's REST API v2.
• It runs dbt build --target audit through a second Trino catalog, audit, whose REST URI ends in .../iceberg/dbt-audit/. Nessie reads the ref from the URI path, so the same tables appear on a different branch. (It's audit and not dbt_audit because Kubernetes names can't contain _, and Stackable uses metadata.name as the Trino catalog name.)
• Only if the build passes does it merge dbt-audit into main.

main now gets one atomic "Merged dbt-audit" commit per build. A failure leaves main
untouched and the branch sitting there for me to inspect. A normal run ends PASS=23
(3 models and 20 tests) and merges cleanly.

spec:
  connector:
    generic:
      connectorName: iceberg
      properties:
        iceberg.catalog.type:
          value: "rest"
        iceberg.rest-catalog.uri:
          value: "http://nessie.mlops.svc:19120/iceberg/dbt-audit/"

A few details kept this from leaking into the rest of the stack. OpenMetadata's manifest
still comes from dbt docs generate --target prod, so lineage keeps its iceberg.*
names, and its Trino crawl only includes ^iceberg$, so the audit catalog never
shows up as a duplicate. The dbt runner got an Istio AuthorizationPolicy
rule that allows only /api/v2/trees*, not the Iceberg REST path that vends S3
credentials. And because adding a Trino catalog restarts the coordinator under Stackable,
I rolled it out between the scheduled runs, not during one.

Bronze, silver, gold. The tables fall into the three layers Databricks calls a
medallion architecture: bronze is data as it lands, silver is typed and cleaned, gold is
tested and ready to use. Here every layer is a set of Iceberg tables, and the gate
between them is the dbt-audit branch. OpenMetadata labels each table with its layer
(its Certification field), so a reader can filter the whole catalog down to Gold.

This is ELT rather than the warehouse's ETL: data is loaded first, almost as it arrives,
and transformed afterwards inside the lakehouse. Keeping the raw layer is what makes that
safe: when a dbt model is wrong, I fix the model and rebuild from bronze, without fetching
anything again.

Bronze, silver, gold: raw_football.matches and raw.events are bronze, stg_events is silver, the two marts are gold; dbt builds silver and gold on the dbt-audit branch, the football model trains on bronze at a pinned tag, and people read gold through Trino and Superset, with lineage in OpenMetadata.

Two details I'd rather show than smooth over. Football has no silver step: dbt builds
the gold mart straight from the raw table. And the football model trains on bronze,
at a pinned tag, because it needs the exact rows it saw, not a cleaned view that could
change underneath it. So raw_football.matches is Bronze and Tier 1 at the same time:
the layer says how refined a table is, and the tier says how much depends on it.

OpenMetadata → Explore → Data Assets: Table, Certification: Gold. The two gold marts with their badge, tier and owner.

OpenMetadata: governance, lineage and MCP. The top layer is a metadata knowledge
graph that crawls Trino and ingests dbt's artifacts to produce end-to-end lineage,
descriptions and data-quality results. It runs 2.0.2, with CNPG for its store, its
own 3-node OpenSearch 3.4.0 for search (operator-managed, on the database node pool),
and Keycloak realm openmetadata for OIDC; login redirects straight to Keycloak's
Google/GitHub page.

Ingestion runs as two CronJobs (om-ingest-trino at 10:00, om-ingest-dbt at
10:15), not the bundled Airflow (I'm not bringing back what I just removed). They
authenticate with the ingestion-bot token Terraform writes to Vault. OpenMetadata is in
this homelab at all because it ships an MCP server, which lets the Hermes agents query
certified, lineage-aware data in natural language. The rest of the stack is building
towards that.

OpenMetadata → marts_marts.daily_active_users → Data Quality tab: the dbt tests with their latest results.

Spark: installed, and not on the line yet

Every lakehouse diagram has Spark in the middle, so I should be straight about mine. The
Kubeflow spark-operator is installed and has run test jobs, but nothing on the daily line
uses it. My tables are small and Trino builds them in seconds, so Spark on that line would
be a forklift carrying one box. It joins in its own article, on a job that needs it:
real-time fraud detection with Kafka, once the cluster has more memory. It will read and
write the same Iceberg tables through Nessie, so nothing about the catalog or the storage
has to change.

SPARK JOB DEMO WITH SPARK CUSTOM IMAGE ON TEKTON AND HARBOR BUILT THROUGH SECURED CI/CD SUPPLY CHAIN

How reliable is it?

A factory that stops when one machine does isn't much of a factory. Here's what this one
survives, and how I know:

And the honest limits:

• Four stations are single replicas. Trino, Superset, OpenMetadata and MLflow each run one pod. If its node dies the pod is rescheduled and that station is down for a minute or two. Nothing is lost, because their state lives in Postgres and object storage.
• A demoted Postgres primary doesn't always come back by itself. After that same cold boot, the old primary couldn't rejoin: its history had diverged from the new primary's. The database ran on two instances until I deleted the stale one's volume and pod. The operator then cloned a fresh replica in the same datacenter and had it streaming within a few minutes. Failover is automatic; that repair was not.
• Each Postgres volume is one Ceph copy. That was a deliberate trade for write speed, covered in the databases section. Losing a drive loses an instance, not data.
• Snapshot cleanup is still deferred. The tag clean-up deletes for real now, but the catalog's garbage collector only records which files it would remove and deletes nothing yet.
• The daily training run can lose a race with the morning boot. If it starts before the cluster has settled, it fails and waits for the next day. About half of the recent scheduled runs went that way, which is next on my list to fix. A rerun by hand takes about six minutes.

kubectl get cluster.postgresql.cnpg.io -n mlops: postgres-mlops-cnpg-cluster with 3 instances, 3 ready, and its current primary.

The traps I hit

Everything above was designed before it ran. Once it ran, it met a cluster that powers
off at night and defaults that fail without an error. Three traps shaped the platform
enough to belong here. The rest, from an operator chart pointing at an image that was
never published to a catalog that would hand storage credentials to any pod that asked,
have their own article in this series.

Trino could query, but never create a table. Every CREATE TABLE failed:

Failed checking new table's location: s3://mlops-data-lake/warehouse/raw/...
  -> Failed to list location
  -> SdkClientException: Unable to load credentials from any of the providers in the chain

Trino checks a new table's location with its own credentials before Nessie gets
involved, and it had none. That gap alone is why dbt had never built anything. The fix
gives Trino the same bucket key Nessie already vends (so no new access).

The cron that never fired. dbt-build had been scheduled for 03:00 UTC for eleven
days and never ran once, with no error, event or Job. Thanos had the answer: the lab
powers off overnight, and there isn't a single metric sample at 03:00 in the retention
window. Every CronJob moved into daytime (09:30 dbt → 10:00 Trino crawl → 10:15 dbt
lineage), with startingDeadlineSeconds so a missed slot is skipped, not replayed at boot.

A quota on limits can block a model rollout. Every Kubeflow profile namespace gets a
ResourceQuota, generated by a Kyverno policy, and mine capped CPU limits at 8. A KServe
rollout runs the old and the new model server side by side until the new one is ready.
Each asks for 2.5 CPU of limit, and one of those is the Istio sidecar's. Add one running
notebook and the pipeline step itself and the total is 8.2. So the new revision was
refused, the deploy step waited fifteen minutes and timed out, and the run failed with a
perfectly good model sitting in the bucket. The symptom is a revision stuck on
FailedCreate with exceeded quota in its events and no pod to look at. I raised the
limit quota to 16 and left the request quota at 4, because requests are what reserve
real capacity. Then the change hit a second trap: ArgoCD reported the policy synced while
the live object was still on its first generation with the old value. Check
metadata.generation on a policy before believing a sync.

One day on the factory floor: every schedule in the lakehouse in the order it fires, from the morning power-on to the afternoon backups

End-to-end workflow

From a data scientist's seat, the platform looks like this:

1. Ingest (on a branch). A job writes new raw data as an Iceberg table on its own Nessie branch, checks it, and merges the branch into main in one commit, so nobody ever reads a half-loaded table. The football pipeline below ingests exactly this way. The other raw table, raw.events, is a seeded demo table with no scheduled ingest.
2. Transform and test. The 09:30 dbt-build CronJob runs raw.events → marts_staging.stg_events (view) → marts_marts.daily_active_users (Iceberg) on the dbt-audit branch, so a failed test never reaches main. The doubled-up schema names are dbt's default (custom schema appended to the target schema); overriding generate_schema_name would move the live tables and orphan their OpenMetadata history, so they stay.
3. Query. Anyone with Trino access runs ad-hoc SQL over the marts (SELECT * FROM iceberg.marts_marts.daily_active_users), something v1 couldn't do without Spark in the loop.
4. Train. A Kubeflow notebook reads the curated Iceberg table (via Trino or the Spark Nessie catalog), trains a model and logs to MLflow:

   mlflow.set_tracking_uri("http://mlflow.mlops.svc.cluster.local:5000")
   mlflow.sklearn.log_model(model, "model", registered_model_name="event-clf")

Because the data lives on a Nessie tag, the run is reproducible: months later you
can point back at the exact catalog state. That works through Nessie (pyiceberg reads
the table at the tag). Trino's catalogs are each pinned to one branch, so a SQL query
sees the current version, not last month's.

1. Register and govern. The best run is promoted in MLflow's Model Registry, and OpenMetadata's 10:00 and 10:15 CronJobs stitch the lineage together. Humans browse it today; the Hermes agents will query it over MCP once the copilot lands.
2. Serve. A KServe InferenceService serves the promoted model (the football match-outcome predictor is the first one live).

The worked example: a football model trained on a pinned snapshot

From match results to a served model: football-data.org to landing Parquet, published to an Iceberg table through a Nessie branch, then Katib, the promotion gate, KServe, MLflow and the predictions site

A KFP RecurringRun fires at 10:00 London time every day. A Job in mlops, the only
namespace with internet egress, pulls Premier League results into s3://mlops-data-lake
as Parquet. The lakehouse part is the publish step:

• It writes iceberg.raw_football.matches on a fresh Nessie branch.
• It audits that branch before anything else sees it: match ids are unique, finished matches have scores, every season is present, and the finished-match count never shrinks.
• It merges into main in one commit and tags the commit train-football-<UTC>.

Katib and the training step read exactly that snapshot, and MLflow and the Kubeflow Model
Registry record the tag, the commit and the Iceberg snapshot id beside the model. dbt's
football_matches model reads the same table.

The model is modest, and I'd distrust anything better: a walk-forward log-loss of 1.050
against 1.082 for always guessing the usual split, over 587 matches, and the right result
47% of the time. A new model has to pass a promotion gate, a smoke test and a canary at
10% then 50% of traffic before it serves. The full story, from the features to the rollout,
is its own article in this series.

epl-predict.georgehomelab.com: the matchday predictions with the scorecard above them, and the goals tables with their "does not beat that base rate" line.

This workflow is how I know the integration points actually work: S3 endpoints resolve,
Nessie answers Iceberg REST, dbt tests gate the marts, MLflow logs to Postgres and S3, and
a notebook in a different namespace reaches everything over in-cluster DNS.

Superset's SQL Lab is a Trino SQL client

Apache Superset is best known for charts and dashboards, but it also ships SQL Lab: a SQL editor in the browser. Point it at Trino and it becomes a SQL client for the whole lakehouse, with no desktop tool to install and no password to hand out.

Here it signs you in through Keycloak, then sends each query to Trino as you, not as a shared Superset account. Trino applies the same per-person rules it applies everywhere else, and the query lands in the audit table under your email.

Apache Superset client connected to Trino, running SELECT * FROM iceberg.marts_marts.daily_active_users and returning rows: the query v1 couldn't run without Spark.

What I deferred

A list, with reasons. Most of these deserve an article of their own.

• The Hermes data copilot. The AI-native part, an agent that reads the catalog, lineage and data over OpenMetadata's MCP and flags tables it can't trust, is waiting on the DGX Spark for local model inference.
• Generic MLflow → KServe promotion. One model is served today through its own pipeline. The general controller ("watch the registry for a promoted version, materialize an InferenceService") is still to write.
• Trino workers. Trino runs coordinator-only, which is plenty for tables this size. Workers come back with more hardware.
• Soda/Great Expectations. dbt tests cover data quality today; these stay deferred until they aren't enough.
• Feast feature store. It waits until a model needs online inference with sub-100ms feature retrieval.
• Streaming, and Spark with it. No broker runs today: Strimzi and Redpanda are both parked, and Spark has no job. They come back together in their own article on real-time fraud detection with Kafka and Spark, once the hardware is there.
• Multi-tenant MLflow and a Spark History Server. Not worth it for 1.x users; kubectl logs on the driver pod covers most debugging.

Conclusion

This build cost about three weekends of focused work, and most of the hours went on the
traps, not the happy paths.

What I got back is close enough to production that the lessons carry over: real OIDC with
per-app cookies, RBAC enforced in the data plane, continuous WAL archiving and
three-instance Postgres, Vault behind every secret, GitOps behind every change, and metrics
from day one.

What I'd tell anyone trying this on their own metal:

1. Say no to bundled defaults. Every chart wants to ship its own database, storage and auth, and every "yes" is technical debt.
2. One source of truth per concern. Vault for secrets, CNPG for relational state, Rook-Ceph for storage, Keycloak for identity. Two of any of these is an integration problem, not a feature.
3. The hard part is the seams, not the install. Nessie → Postgres → Vault, MLflow → RGW, oauth2-proxy → Keycloak → Google. That's where the documentation is thinnest.
4. Defer aggressively, then ship when the workload arrives. Trino was deferred until I needed ad-hoc SQL, and then it pulled Nessie, dbt and OpenMetadata in with it.
5. Don't be afraid to rip out something everything depends on. GitOps turned the Hive + Airflow swap into a series of reviewable commits instead of a rebuild.

The part I lean on most is Git-for-data. dbt publishes each build to main as one tested
commit, and the football model trains on a tagged snapshot that MLflow and the Model
Registry both remember. It stopped being a slide.

The ArgoCD UI filtered to mlops-* & kubeflow-*: the whole lakehouse (Trino, Nessie, dbt, OpenMetadata, MLflow, CNPG) Synced and Healthy.

What's Next?

The next milestones, and probably the next few articles:

• The Hermes copilot, once the DGX Spark lands: local inference, read-only verification first, proposed writes later.
• Streaming with Kafka and Spark: real-time fraud detection, with card transactions in and a model trained in Kubeflow out. This is the job that puts Spark on the line, and it waits for more hardware.
• Generic MLflow → KServe promotion, so any registered model can be served without its own bespoke pipeline.
• Turning the garbage collector on for real: Nessie GC without --defer-deletes, after a few weeks of clean logs. Tag retention already deletes.
• Unstructured data, probably arriving with the DGX Spark. That's the point where I revisit lakeFS, as a replacement for Nessie and not as a second layer on top of it.

If you want to see how those go, including the parts that break, follow along.

Follow me on LinkedIn for the next posts in this series.

Author: George Ezejiofor