12 Patterns Run Every Data Stack. Seniority Is Knowing When Each One Is a Mistake.

Suggested tags: dataengineering, architecture, bigdata, dataops

You've seen the chart — twelve boxes, twelve patterns, every data stack ever built somewhere inside it. Darshil Parmar's map of the territory is genuinely good. But a map doesn't tell you which roads are closed, which ones flood in the rainy season, and which ones look like highways but bill you like toll roads.

Here's my take on all 12, after 17 years of shipping them in production. For each one: what it's for, and — the part the chart can't show you — when it's a mistake.

Moving data: ETL, ELT, CDC

1. ETL — alive, and not from nostalgia

When PII must be masked before the data lands anywhere, transform-first isn't a preference. It's the only architecture that satisfies the constraint. Compliance keeps ETL employed.

The claim is executable. Here's the shape of it — masking in flight, so the raw value never touches storage:

import hashlib
import json

def mask_pii(record: dict, pii_fields: list[str], salt: str) -> dict:
    """Transform BEFORE load. The raw CPF/email never lands in the lake."""
    out = dict(record)
    for field in pii_fields:
        if field in out and out[field] is not None:
            digest = hashlib.sha256((salt + str(out[field])).encode()).hexdigest()
            out[field] = digest[:16]  # stable pseudonym, joinable, not reversible
    return out

raw = {"user_id": 42, "email": "maria@example.com", "cpf": "123.456.789-00", "amount": 89.90}
landed = mask_pii(raw, pii_fields=["email", "cpf"], salt="rotate-me-quarterly")
print(json.dumps(landed, indent=2))

{
  "user_id": 42,
  "email": "0a1f2b8c9d3e4f5a",
  "cpf": "7c6d5e4f3a2b1c0d",
  "amount": 89.9
}

Try doing that in ELT. You can't — by definition, the untransformed value already landed. If your regulator cares about where raw PII exists, not just who queries it, the L-before-T order is disqualifying.

When it's a mistake: when there's no landing constraint and you're paying for a transform cluster the warehouse could replace.

2. ELT — the modern default, with a hidden multiplier

Snowflake, BigQuery and Redshift turned the warehouse into the compute engine. Cheap storage flipped the letters. Load raw, transform in SQL, version it in dbt. It's the right default.

But watch the bill, because "transform later" quietly becomes "transform five times." Marketing derives sessions_clean from raw events. Product derives almost the same table. Finance derives it again with one extra filter. Nobody knows, because raw data is right there and deriving is one CREATE TABLE AS away.

The multiplier is just arithmetic:

raw_scan_tb = 2.0          # TB scanned per full transform over raw events
price_per_tb = 6.25        # on-demand scan pricing, adjust to your warehouse
runs_per_day = 4           # scheduled refreshes
teams_rederiving = 5       # teams independently transforming the same raw data

one_pipeline_month = raw_scan_tb * price_per_tb * runs_per_day * 30
print(f"One team, one month:   ${one_pipeline_month:,.0f}")
print(f"Five teams, same data: ${one_pipeline_month * teams_rederiving:,.0f}")

One team, one month:   $1,500
Five teams, same data: $7,500

Same raw data. Same business question. Five bills. The fix isn't abandoning ELT — it's a shared staging layer so the expensive scan of raw happens once, and everyone derives from that.

When it's a mistake: never as a pattern — only as an unowned one. ELT without a shared model layer is a subscription to redundant compute.

3. CDC — the most misdiagnosed pattern in the stack

Everyone asks for real-time replication. Most need hourly batch. The diagnosis question is not "how fresh is the data?" It's "how fresh is the decision?"

def cdc_or_batch(decision_latency_min: float, change_rate_per_hr: int) -> str:
    """If no decision changes faster than your batch window, CDC is cost, not capability."""
    if decision_latency_min < 15:
        return "CDC — a decision actually changes inside the batch window"
    if change_rate_per_hr > 500_000:
        return "CDC — batch extracts would hammer the source DB"
    return "Hourly batch — you're done, go home"

for case, args in {
    "Fraud hold on a transaction": (0.5, 50_000),
    "Executive sales dashboard":   (1440, 20_000),
    "Inventory sync to the app":   (10, 800_000),
}.items():
    print(f"{case:32s} -> {cdc_or_batch(*args)}")

Fraud hold on a transaction      -> CDC — a decision actually changes inside the batch window
Executive sales dashboard        -> Hourly batch — you're done, go home
Inventory sync to the app        -> CDC — a decision actually changes inside the batch window

And if the answer really is CDC — respect it. A Debezium agent feeding Kafka is production infrastructure: schema evolution handling, snapshot strategy, offset management, dead-letter queues, on-call. Not a weekend project. I've seen more incidents from casually-deployed CDC than from any other pattern on this list.

When it's a mistake: when it was prescribed by the word "real-time" in a slide instead of a decision that changes mid-hour.

Storing data: Lakehouse

4. Lakehouse — the truce after a decade of war

Lake vs warehouse was never a fair fight — it was two copies of the same data, two pipelines pretending to agree, and a reconciliation job apologizing between them. The lakehouse (Delta, Iceberg, Hudi — table formats with ACID over object storage) is the truce: one copy, one source of truth, warehouse semantics on lake economics.

I merged 6 data lakes into 1 on this pattern. The before/after:

The compute drop is the part people don't predict: most of that spend wasn't analysis. It was six pipelines keeping six copies loosely synchronized — work that produces no insight, only agreement. Delete the duplication and the compute deletes itself.

When it's a mistake: when your data fits comfortably in one warehouse and the "lakehouse migration" is résumé-driven. One Postgres replica and dbt beats a Delta lake you don't need.

Processing cadence: streaming, batch, Lambda, Kappa

5. Real-time streaming — powerful, and over-prescribed

Flink and Spark Structured Streaming earn their complexity when decisions change mid-minute: fraud holds, dynamic pricing, operational alerting. That complexity is real — watermarks, late data, exactly-once semantics, state stores that need care and feeding.

Meanwhile, a dashboard refreshed every 5 minutes is batch wearing a costume. If the consumer of the data is a human glancing at a screen between meetings, a 5-minute micro-batch delivers the identical experience at a fraction of the operational surface. The streaming job isn't serving the user — it's serving the architecture diagram.

When it's a mistake: whenever the freshest possible data and the freshest needed data differ by more than the batch window.

6. Batch — unglamorous, still moves the world

Most of the planet's data still moves at night, in batch, and that's fine. My favorite optimization win ever was pure batch: an 8-hour Spark pipeline tuned down to 47 minutes. No new tools. No migration. The fixes were the boring, compounding kind:

# The class of changes that took 8 hours to 47 minutes — none of them exotic.

spark.conf.set("spark.sql.shuffle.partitions", "auto")   # was a hardcoded 200 from a 2018 blog post
spark.conf.set("spark.sql.adaptive.enabled", "true")      # let AQE resize joins at runtime

# 1. Stop reading everything: partition pruning instead of full scans
df = spark.read.parquet("s3://events/").where("event_date = '2026-07-14'")

# 2. Stop shuffling the small side: broadcast the dimension table
from pyspark.sql.functions import broadcast
joined = df.join(broadcast(dim_stores), "store_id")

# 3. Stop recomputing shared branches: cache what three outputs reuse
base = joined.filter("status = 'valid'").cache()

Boring done well beats exciting. An 8-hour job at 47 minutes reruns inside a business day when it fails — which changes your on-call life more than any architecture choice on this page.

When it's a mistake: only when a decision genuinely can't wait for the window. Which is rarer than the vendor deck says.

9. Lambda — honest about the tension, brutal in practice

(Taking 9 and 10 here because they're one argument.) Lambda architecture admits the real tension: some answers need to be fast, all answers need to be complete. Its solution — a speed layer and a batch layer computing the same metric — means two codebases that must agree forever.

They won't. Here's the same "daily revenue" in both layers; spot the bug that ships twice:

-- Batch layer (SQL): timezone-aware, refunds excluded
SELECT DATE(created_at AT TIME ZONE 'America/Sao_Paulo') AS day,
       SUM(amount) AS revenue
FROM orders
WHERE status != 'refunded'
GROUP BY 1;

# Speed layer (streaming): UTC dates, refunds... forgotten
def on_event(order, state):
    day = order["created_at"][:10]        # UTC slice — already disagrees with batch
    state[day] = state.get(day, 0) + order["amount"]  # refunds counted — bug #2

Every definition change now has to land in two languages, two deploy pipelines, two teams' heads. Every bug ships twice, and the reconciliation meeting is where careers go to age.

When it's a mistake: almost always, today. Lambda made sense before table formats and modern stream processors; now it's mostly historical baggage with a Greek name.

10. Kappa — Lambda's apology

One stream, one codebase, replay the log when you need to reprocess. Elegant — if you can afford the Kafka retention and the reprocessing story:

daily_gb = 400            # ingest volume
retention_days = 90       # long enough to replay after finding a logic bug
replication = 3
storage_per_gb_month = 0.10

retained_gb = daily_gb * retention_days * replication
print(f"Retained in Kafka: {retained_gb/1000:.1f} TB")
print(f"Retention cost:    ${retained_gb * storage_per_gb_month:,.0f}/month")
print(f"Full replay time at 500 MB/s: {retained_gb/3 / 0.5 / 3600:.0f} hours")

Retained in Kafka: 108.0 TB
Retention cost:    $10,800/month
Full replay time at 500 MB/s: 20 hours

Many teams look at that and choose batch with a small streaming edge instead. That's not failure — that's arithmetic.

When it's a mistake: when the retention bill and replay window are bigger than the problem Lambda's dual codebase was causing you.

Knowing your data: catalog, lineage

7. Data cataloging — only works as a living system

A catalog nobody updates is a museum: pleasant to visit, wrong about everything current. The only catalogs I've seen survive are the ones populated by the pipelines themselves — metadata emitted at run time, not typed in by a data steward six weeks later. If updating the catalog is a human task, the catalog is already stale; you just haven't noticed yet.

When it's a mistake: as a standalone documentation project. As an automated side effect of orchestration, it's one of the highest-leverage things on this list.

11. Lineage — nobody budgets for it until a board deck is wrong

Then "where did this number come from" is suddenly worth more than the pipeline that produced it. Lineage is insurance: it looks like pure cost right up until the incident where it's the only thing that matters. The trick is the same as cataloging — derive it from execution (query logs, dbt manifests, OpenLineage events), never from documentation.

When it's a mistake: hand-maintained. Automated or absent — those are the only two honest states.

Organizing people: mesh

8. Data mesh — an org decision wearing an architecture costume

Mesh is not a technology. It's a claim about your company: that domain teams are ready to own data as a product — with SLAs, schemas as contracts, and someone on call when their dataset breaks a consumer. If that claim is true, mesh is liberating. If it's false, mesh is decentralized chaos with better naming, and the central data team still gets paged for everything while owning nothing.

When it's a mistake: when it's adopted as an architecture before it's true as an org chart.

Running it all: orchestration

12. Orchestration — the heartbeat nobody monitors

The DAG is your platform's heartbeat and its least monitored component. Everyone watches the pipelines. Few watch the scheduler running them — and a silently degraded scheduler fails every pipeline at once, quietly, by simply not starting them.

# The alert most platforms are missing: not "did the job fail?"
# but "did the scheduler fail to even try?"
from airflow.sensors.base import BaseSensorOperator
from datetime import datetime, timedelta, timezone

class SchedulerHeartbeatCheck(BaseSensorOperator):
    def poke(self, context):
        from airflow.jobs.job import Job
        latest = Job.most_recent_job(job_type="SchedulerJob")
        lag = datetime.now(timezone.utc) - latest.latest_heartbeat
        if lag > timedelta(minutes=5):
            raise RuntimeError(f"Scheduler heartbeat is {lag} old — nothing is being scheduled")
        return True

A DAG that fails pages someone. A DAG that never starts pages no one — until the morning dashboard is empty and twelve teams find out together.

When it's a mistake: it isn't. Unmonitored, it's just a mistake waiting for a quiet night.

What's missing from the chart

Notice: not one of these twelve is cloud-specific. The same patterns run on AWS or GCP, on Databricks or on bare Postgres and cron if you're stubborn enough. Tools rotate every five years. Patterns are the career.

The principle: every pattern here solves a problem and creates one. Architecture is choosing which problems you'd rather have.

Which of these did you adopt too early — and what did it cost you?

I'm Vinicius Fagundes — principal data engineer, independent consultant, and MBA lecturer in São Paulo. I write about data pipelines and the math that runs on top of them, and take on a few architecture projects per quarter through vf-insights.com.