Managed Data Lake: A Guide for 2027
Apache Iceberg is the standard table format for production data lakes in 2027. Every major engine reads and writes it natively. The catalog ecosystem standardized on REST. You own your data on commodity storage with no lock-in.
But Iceberg deliberately separates the table format from the system that keeps tables healthy. It gives you the primitives for maintenance — rewrite_data_files, expire_snapshots, remove_orphan_files, rewrite_manifests — but not the intelligence to decide when, how, and in what order to run them. Without that operational layer, every Iceberg table degrades over time: small files accumulate, snapshots bloat metadata, sort orders drift from query patterns, orphan files inflate storage costs, and query performance decays silently until something breaks visibly.
This operational gap is the central challenge of running a data lake at production scale. Netflix built four internal services to address it — Autotune for compaction strategy selection, Polaris for catalog management, janitors for garbage collection, Metacat for cross-service observability — each staffed by dedicated teams over multiple years. Google engineered automatic compaction and garbage collection directly into BigLake, so their managed Iceberg tables stay healthy regardless of write volume or query pattern changes.
In 2027, you do not need to replicate that investment. This guide covers what "managed" actually means for a data lake, the degradation mechanics that make it necessary, the control plane architecture that solves it, and the practical paths to getting there — whether you are running 50 tables or 5,000.
Why Lakes Degrade — The Mechanics
The degradation pattern is predictable and present in nearly every Iceberg lake running for more than three months without dedicated maintenance. Understanding these mechanics is necessary regardless of which management approach you choose.
The Small-File Problem
Every streaming writer — Flink, Spark Structured Streaming, Kafka Connect, RisingWave, CDC connectors — produces files sized by checkpoint intervals, not by optimal read patterns. A Flink job committing every 60 seconds across 100 active partitions creates 144,000 new files per day. Each of those files might be 1–5 MB. The target for efficient reads is 128–512 MB.
Every query pays per-file overhead: an S3 GET request ($0.0004 per 1,000), a Parquet footer parse, a task-scheduling slot in the engine. This means query compute scales with file count rather than with actual data volume. You are paying for how the data was written, not for how much data exists.
The insidious part is the rate. A batch table that loads once daily accumulates files slowly — degradation takes months to become visible. A streaming table crosses into problematic territory within days. A high-throughput streaming table (hundreds of partitions, sub-minute commits) can reach a degraded state within hours of going live if no background compaction runs alongside the writer. The longer you wait to compact, the heavier and more disruptive the operation becomes — it conflicts with active writers, consumes significant cluster resources, and takes hours instead of seconds.
Snapshot Accumulation
Every Iceberg commit creates a new snapshot — that is the mechanism that enables time travel and atomic rollback. A streaming table at 5-minute commits accumulates ~288 snapshots per day. Without expiration, counts reach tens of thousands within weeks.
Query planning reads the snapshot chain to resolve the current table state. Once you exceed roughly 1,000–2,000 retained snapshots, planning time degrades measurably. The metadata.json file grows — production environments have seen it reach 400+ MB. The symptom is slow EXPLAIN, slow first-row-time, and users reporting that "the query hangs for a few seconds before anything happens." The data scan itself is fine. It is the planning that has become expensive.
The correct retention window depends on your time-travel requirements. For most analytics workloads, 3–7 days of snapshots is sufficient. Regulated industries (financial, healthcare) may require 30–90 days — but should understand the planning-time trade-off and invest in more aggressive manifest rewriting to compensate. Whatever your retention policy, it must be actively enforced. Iceberg does not expire snapshots on its own.
Manifest Fragmentation
Manifests are Iceberg's lookup structure — they tell engines which data files exist in which partitions and what their column-level statistics are. Every commit appends at least one manifest entry. Over months of streaming writes, you accumulate hundreds or thousands of small manifest files.
The query planner must read all of them to build an execution plan. This adds seconds of latency before any data file is opened. When Iceberg 1.11 introduced server-side scan planning (shifting metadata traversal off query engines entirely), lean manifests became even more valuable — the planning server benefits proportionally from fewer, larger manifest files.
Most teams misdiagnose slow planning as an engine problem — they try tuning memory, scaling coordinators, or upgrading engine versions. It is almost always the manifests. Rewriting them into a smaller number of well-merged files is one of the highest-leverage operations available — and one of the least commonly automated.
Orphan Files
Failed Spark stages, interrupted compaction, aborted writes, schema-evolution retries, OOM kills during large rewrites, and partial commits all leave files on object storage that no Iceberg snapshot references. They are invisible to SELECT queries, invisible to Iceberg metadata, invisible to your data quality checks — but fully visible on your cloud bill.
You cannot find them by querying your tables. You must compare what exists in object storage against what Iceberg metadata references — a set difference operation that requires listing entire prefixes and cross-referencing with the catalog. This is why orphan accumulation goes undetected for months. Without a dedicated process that runs this comparison regularly, you have no way to know how much you are spending on dead data.
Production orphan sweeps routinely reclaim 20–40% of a lake's storage. For a 100 TB lake, that is 20–40 TB of billable storage serving no purpose. At $0.023/GB/month (S3 Standard), that is $460–$920 per month for one lake — just in orphan storage. Multiply by the number of accounts, regions, and environments your organization operates.
Sort-Order Drift
Bin-pack compaction — merging small files into larger ones — solves the file-count problem but does nothing for data organization. If your data is not physically sorted by the columns queries actually filter on, Parquet min/max statistics span wide ranges and engines cannot skip irrelevant row groups.
Consider a transactions table sorted by transaction_id (because it was the natural append key) but queried primarily by customer_id and event_date. Every file contains a wide range of customer IDs, so a predicate like WHERE customer_id = 'abc123' cannot skip any files based on statistics alone — the engine scans everything.
Sort the same data by customer_id, event_date and the statistics become tight. A query for one customer touches one or two files instead of hundreds. The engine eliminates 95%+ of scan volume before reading a single row. This is not an incremental improvement. It is a categorical difference in query performance.
The challenge is that sort orders set at table creation drift. The column that was the primary filter six months ago might not be today. New consumers join. Dashboard patterns shift. ML feature pipelines add access patterns the original designer never anticipated. Across hundreds of tables with different consumers, manual sort-order tuning is impractical — and sort orders that are wrong silently cost you compute on every query, every day, without generating any error or warning.
Delete File Accumulation
Merge-on-read tables accumulate position and equality delete files with every UPDATE or DELETE operation. This is the mechanism that makes updates fast at write time — rather than rewriting the entire file, Iceberg records "this row is deleted" in a separate file.
The cost lands on reads. Every query must reconcile live rows against all accumulated delete files for the partitions it touches. This reconciliation cost grows linearly with each write cycle. At high delete volumes — CDC workloads doing frequent upserts, GDPR compliance deleting records, slowly changing dimensions — the read-time overhead becomes significant. Queries that should complete in seconds spend most of their time reconciling deletes rather than reading actual data.
Iceberg 1.11's deletion vectors (backed by Roaring bitmaps) improve the mechanism by storing deletes more compactly and enabling faster reconciliation. But the fundamental accumulation problem remains: unless compaction regularly applies those deletes (physically rewriting files to exclude deleted rows and then removing the delete files), read performance degrades continuously.
The Organizational Cost
Beyond the technical debt, the team cost scales worst.
Maintenance responsibilities accumulate incrementally: compaction scripts, expiration logic, monitoring, alerting, per-table configuration for heterogeneous workloads, debugging writer conflicts, handling OOM failures, on-call rotation for overnight maintenance failures, documentation, and onboarding. Each table added to the lake adds marginal load to a team that is not growing proportionally.
Platform engineers who should be building data products, self-service tooling, or pipeline reliability improvements are instead maintaining an ever-growing library of operational scripts. The knowledge concentrates in a few heads — when those people leave, the lake degrades within weeks because nobody else knows what the DAGs do or why they are configured that way.
This cost does not show up on cloud bills but manifests in team velocity, retention risk, and the growing gap between what the data platform promises and what it delivers.
What "Managed" Actually Means
The term is overloaded. Vendors use it to mean proprietary platforms you migrate into. Cloud providers use it for single-service table maintenance. Some teams use it to mean a weekly compaction cron job. Others equate it with having a catalog.
A more useful definition: a managed data lake is one where every table stays healthy, every query hits well-organized data, and operational cost does not grow linearly with table count — without requiring a platform team that scales proportionally with the lake.
This definition has three important implications. First, it is about outcomes, not tools. Running compaction nightly does not make your lake managed if half your tables are still degraded. Second, it covers the full lifecycle — not just file compaction, but snapshot management, orphan cleanup, manifest optimization, sort alignment, policy enforcement, and observability. Third, it must scale sublinearly: managing 500 tables should not require 10x the effort of managing 50.
The Architecture
Your data lake already has three layers: commodity object storage (S3, GCS, ADLS), a table format and catalog (Iceberg plus Polaris/Glue/Nessie/Gravitino/Lakekeeper), and compute engines (Spark, Trino, Snowflake, Athena, DuckDB, Flink). These give you storage economics, structural reliability, and query flexibility.
What you are missing is the fourth layer: an operations control plane.
This is not the same as the catalog. The catalog (Polaris, Glue, BigLake Metastore) is a metadata control plane — it resolves table locations, handles access control, and provides the REST interface for multi-engine access. Every major catalog now implements the REST spec: Polaris (Apache top-level project since February 2026), Glue's REST adapter, BigLake Metastore, Nessie, Gravitino, Lakekeeper.
The operations control plane sits above the catalog. It uses metadata as input, but its job is fundamentally different: observe structural health across every table, classify what needs attention, decide what maintenance to run, execute it safely and in the correct sequence, adapt compaction to real query patterns across all engines, enforce governance at lake scale, route queries to optimal engines, and feed outcomes back into future decisions.
LakeOps is this operations control plane — a purpose-built product for Apache Iceberg that does nothing else.
It is not an engine, not storage, not a catalog. It connects to your existing catalogs (AWS Glue, Polaris, Gravitino, Nessie, Lakekeeper, S3 Tables) and engines (Spark, Trino, Flink, Snowflake, Athena, DuckDB), adds autonomous management across the entire lake, and requires no data movement or pipeline changes. Most teams have their first lake-wide health assessment within ten minutes of connecting a catalog.
The key architectural property: it does not replace anything. If you remove the control plane, everything continues working exactly as before — just without the automation. Your storage, your engines, your catalog, your data — all unchanged. The control plane adds intelligence to an existing stack. It does not create dependency.
The Maturity Spectrum
Teams operating Iceberg follow a predictable progression. Identifying where you sit determines what to invest in next.
Stage 1: No maintenance. Iceberg adopted for format-level benefits. Pipelines land data. Nobody explicitly owns table health. Performance degrades and storage costs rise gradually, but the symptoms are not connected to a root cause. Common in organizations that migrated to Iceberg within the last year.
Stage 2: Reactive fixes. After an incident — slow dashboard, cost anomaly, failed SLA — an engineer writes a one-off compaction script or snapshot expiration job. The immediate pain resolves, but only for the tables that broke visibly. The rest of the lake continues degrading silently. The cycle repeats every few months on different tables.
Stage 3: Scheduled automation. The team builds Airflow DAGs or cron jobs running maintenance on a fixed schedule. Compaction nightly. Snapshot expiration weekly. Orphan cleanup monthly. This works for stable batch workloads but has structural limitations at scale:
- Wastes compute on healthy tables that do not need maintenance
- Misses tables that degrade between cycles (a streaming table needing hourly compaction gets nightly)
- Cannot adapt to changing workloads — new query patterns do not trigger sort-order changes
- Runs operations as independent jobs without sequencing coordination, creating race conditions
- The maintenance infrastructure itself requires ongoing maintenance: updating DAGs, handling failures, adjusting schedules, onboarding new tables
Stage 4: Autonomous management. A dedicated system continuously observes every table, classifies health based on combined structural, storage, and query signals, sequences operations in the correct order, adapts compaction strategy to real query patterns from cross-engine telemetry, enforces policies lake-wide, and feeds outcomes back into decisions. Maintenance is signal-driven rather than schedule-driven. Tables that need attention get it immediately. Healthy tables cost zero compute.
Most production lakes sit between Stage 2 and Stage 3. The gap from Stage 3 to Stage 4 is where teams either build a dedicated platform (6–12 months, 2–3 senior engineers full-time, plus ongoing maintenance of the platform itself) or adopt an existing control plane. Stage 3 is where DIY efforts stall — building cross-engine telemetry collection, health classification logic, sequencing and conflict resolution, adaptive compaction strategy, and policy inheritance is a non-trivial distributed systems project that most teams underestimate.
How the Control Plane Works
A control plane is not a collection of scripts with a dashboard. It is a closed-loop system where each stage feeds the next. The difference between Stage 3 automation (scheduled scripts) and Stage 4 automation (autonomous management) is entirely in this loop: observe → decide → act → optimize → govern → route → enable → learn.
Observation: Unified Telemetry
Everything starts with signals. Three telemetry streams feed every decision the system makes:
First, structural metadata from Iceberg — file counts per partition, average file sizes, manifest depth, snapshot count and velocity, delete file ratios, current sort order configuration, partition layout, and schema history. This tells you the table's physical state — the raw structural facts that indicate health or degradation.
Second, object storage metrics — total bytes per prefix, orphan file counts, access patterns, request rates, storage class distribution. This reveals the cost reality that metadata alone does not capture. A table might look healthy in Iceberg metadata while harboring terabytes of orphaned files in its storage prefix.
Third, and most critically, cross-engine query telemetry — which columns appear in WHERE, JOIN, and GROUP BY clauses from every engine reading your tables. Trino knows its own queries. Spark knows its own. Snowflake knows its own. Athena knows its own. None of them knows what the others are doing on the same table. None of them can tell you what the combined access pattern looks like across your full consumer base.
A control plane collects this telemetry from every connected engine, correlates it per table, and uses the combined signal to drive optimization decisions that no single engine could make alone. This is the fundamental capability that separates an operations control plane from single-engine maintenance tools — it sees the whole picture.
This unified telemetry is what enables the system to classify every table as Healthy, Warning, or Critical — and to surface specific, actionable insights ranked by severity. You know exactly which of your 500 tables needs attention right now, what the problem is, what caused it, and what the fix should be.
Decision: Health-Driven Prioritization
Raw telemetry becomes actionable through health classification. Instead of blindly running compaction on every table overnight (Stage 3), the system targets precisely the tables that need attention — and leaves healthy tables alone.
A table's health score reflects the combined severity of its degradation signals: file count relative to target, manifest fragmentation ratio, snapshot depth relative to retention policy, sort-order alignment with actual query patterns, orphan volume, and delete file accumulation. Each dimension has independent thresholds, and the composite determines the classification.
When a table crosses from Healthy to Warning (file count exceeds target, sort order drifts from query patterns, manifest fragmentation passes threshold), maintenance triggers. When it returns to Healthy, operations pause. Critical tables get immediate attention regardless of scheduling. Warning tables are queued by priority. Healthy tables cost zero compute — no maintenance runs against them until conditions change.
This feedback loop eliminates the two failure modes of Stage 3 automation: wasting compute on healthy tables that do not need attention, and missing tables that degrade between fixed-schedule cycles. A streaming table that crosses into Warning two hours after the last scheduled maintenance run does not wait until tomorrow night. It gets addressed when the signal fires.
Execution: Sequenced Maintenance
The individual operations are well-understood — Iceberg provides the procedures. What matters is the order, the coordination, and the safety.
The correct sequence is: expire snapshots → remove orphan files → compact data files → rewrite manifests → refresh statistics. Each step depends on the previous:
- Expiration dereferences files and snapshots that are past their retention window
- Orphan cleanup then safely removes those dereferenced files from storage — only possible after expiration has run
- Compaction operates on live data only — not on files that are about to be expired — reducing wasted work
- Manifest rewriting consolidates the new manifest structure that compaction created
- Statistics refresh ensures the newly organized data has accurate min/max values for predicate pushdown
Running these in the wrong order wastes compute (compacting data that will be expired in the next step), creates safety risks (removing files that unexpired snapshots still reference), or produces suboptimal results (rewriting manifests before compaction means they immediately fragment again on the next compact).
At Stage 3, these operations typically run as independent cron jobs — compaction in one DAG, expiration in another, orphan cleanup in a third, each with its own schedule and no awareness of the others. This creates exactly the sequencing failures described above. A control plane runs them as a coordinated pipeline per table, every time, conflict-aware (never compacting partitions with active writers), with OCC-aware retry logic for optimistic concurrency conflicts.
Adaptive maintenance bundles this further: rather than configuring each operation's frequency separately, a single data-driven policy monitors commit frequency, file accumulation rate, query load, and seasonal patterns — then runs the right operations at the right frequency, automatically. A high-velocity streaming table gets frequent bin-pack compaction to keep pace with the write rate. A stable batch table gets occasional deep sort compaction when access patterns shift. A table that received a sudden burst of writes gets immediate attention. One policy toggle replaces dozens of per-table DAG configurations.
Optimization: Query-Aware Compaction
This is where the control plane delivers its highest-impact differentiation from scripts and scheduled jobs.
Basic bin-pack compaction normalizes file sizes — merging many small files into fewer large ones. That is necessary (it solves the per-file overhead problem) but insufficient (it does nothing about data organization). A table with perfectly sized files but random data ordering still forces engines to scan everything because min/max statistics span the full value range.
The real performance leverage is in sort compaction: physically rewriting data so records are ordered by the columns queries actually filter on. When data is correctly sorted, Parquet row-group min/max statistics become tight — each row group contains a narrow range of values for the sort columns. Engines read these statistics during planning and skip entire file groups whose ranges do not match the predicate. The difference is not incremental. It is the difference between scanning 100% of a table and scanning 5–15%.
The challenge at scale is knowing what to sort by — and keeping it current. Static sort orders set at table creation drift within weeks as query patterns evolve. A column that was the primary filter six months ago might be secondary today. New consumers arrive. Dashboard logic changes. ML pipelines add new access patterns. Across hundreds of tables with different consumers across different engines, manual sort-order tuning is impractical.
Cross-engine telemetry solves this. The control plane observes which columns are actually filtered, joined, and grouped across all engines hitting each table — Trino, Spark, Snowflake, Athena, DuckDB — then physically re-sorts data to match the real combined access pattern. As patterns change, the sort order adapts. The optimization is continuous and automatic, not a one-time configuration that decays.
LakeOps executes compaction on a purpose-built Rust/DataFusion engine — no JVM, no garbage collection pauses, no Spark clusters to provision or decommission. Compaction is event-driven (triggered by health signals, not cron schedules) and conflict-aware (never touching partitions with active writers, with built-in OCC retry). The engine completes bin-pack and sort operations in a fraction of the time Spark requires for the same workload, without the cluster management overhead.
Before committing to a sort-order change on a large table — where a wrong decision means rewriting terabytes of data for no benefit — layout simulations let you test strategies safely. Simulations create a real Iceberg branch from the latest snapshot, apply the proposed layout to that branch, replay actual query patterns from cross-engine telemetry against it, and compare results against the current baseline. You see projected file counts, data-skipping improvement percentage, and scan reduction for each candidate strategy — before writing a single byte to production data. The branch is discarded after analysis. This is the verification step that makes autonomous sort optimization trustworthy at scale.
Governance: Policies That Scale
At one table, configuration is a one-time decision. You pick a compaction strategy, set a retention window, choose a target file size, and move on. At 50 tables it becomes repetitive. At hundreds of tables across multiple catalogs and namespaces, it becomes governance entropy.
The symptoms: some tables have compaction configured, others use defaults nobody verified. Retention windows are inconsistent — production tables with 90-day retention next to staging tables with no expiration at all. New tables inherit nothing because nobody templated the configuration. An engineer leaving means their table-specific tuning knowledge leaves with them. Auditors ask "what is your data retention policy?" and nobody can answer precisely because the truth is scattered across table properties, DAG configs, and Terraform state files.
A control plane enforces declarative policies with hierarchical inheritance: compaction strategy, target file size, snapshot retention, cleanup safety windows, and maintenance frequency defined at organization, catalog, namespace, or table level. Everything cascades downward — set a policy at the namespace level and every table in that namespace inherits it. New tables automatically pick up the correct configuration. Exceptions override at the leaf level when specific tables need different treatment. Policies are versioned, auditable, and diff-able — changes are tracked, reviewable, and reversible.
Routing: Multi-Engine Cost Optimization
Running multiple engines on the same Iceberg tables is standard practice in 2027. The REST catalog spec means any engine can read any table. Most production lakehouses use at least three: Trino for interactive analytics, Spark for heavy ETL, and one of Snowflake/Athena/DuckDB for BI or ad-hoc queries. Each has radically different cost and latency profiles.
A point lookup costs fractions of a cent on DuckDB running locally and significantly more on Snowflake's warehouse compute. A multi-table join with complex aggregations completes in seconds on Trino but might take minutes on DuckDB. A 10 TB batch transformation is cheapest on Spark with spot instances. Without routing, applications hardcode engine connections — every query goes to one engine regardless of whether a cheaper or faster option exists for that specific query shape.
A control plane's routing layer classifies query shapes (point lookups, analytical aggregations, complex joins, full-table scans) and sends each to the optimal engine based on cost targets, latency requirements, and current table health. Applications connect to a stable SQL endpoint; backend engine selection adapts as conditions change, as table health improves, and as new engines are added to the pool.
Because the control plane already has cross-engine telemetry and table health data, routing decisions use the same signals that drive compaction — creating a virtuous cycle. Better-compacted tables unlock more routing options (a well-sorted table can serve point lookups from DuckDB that would be too slow on an unsorted table). Better routing data informs compaction priorities (tables with high cross-engine query volume get priority optimization).
AI Readiness: Agent Access at Scale
AI agents are becoming primary consumers of lakehouse data — not a future prediction, but a present reality in many production environments. They issue SQL iteratively, at high frequency, without human review of each query. They explore schemas, join across tables, aggregate at scale, and do it hundreds of times per analysis cycle. This creates requirements that traditional table maintenance does not address: structured schema discovery (agents need to understand table relationships programmatically), cost and safety guardrails (an agent with no limits will issue expensive full-table scans), and storage that self-optimizes for agent access patterns that may differ from human patterns.
LakeOps provides an agent-native MCP interface — any MCP-compatible agent (Claude, GPT, custom models) connects with zero integration code and auto-discovers available tools for schema discovery, health analysis, and query execution. Guardrails stack composably: ReadOnlyGuard prevents DDL and DML, CostEstimateGuard rejects queries that would scan more than a cost threshold, PIIMaskGuard scrubs sensitive columns before results reach the agent, HumanApprovalGuard gates high-stakes operations for human review. These are not all-or-nothing — you compose them per use case.
Agent query telemetry feeds back into the compaction engine alongside human telemetry — the lake self-optimizes for the combined workload. If agents start heavily filtering on a column that humans rarely use, the sort order adapts to serve both access patterns. The lake becomes agent-ready not by adding a separate AI layer, but by extending the same optimization loop to a new class of consumers.
Learning: The Closed Loop
The final stage is what transforms a management tool into a system that improves with time and scale. Every maintenance operation produces measurable outcomes: file count reduction, scan volume improvement, query latency change, storage reclaimed, compute saved. These outcomes are not just metrics to display — they feed back into future decisions.
Sort compaction that improved dashboard latency by 8x on one table informs strategy selection for structurally similar tables. Routing decisions that reduced costs without impacting latency reinforce the model for similar query shapes. Compaction schedules that were too aggressive for a table's write rate — causing frequent OCC conflicts — get automatically dialed back. Target file sizes that produced optimal performance on one storage class get recommended for similar tables.
This is why a control plane gets more effective as the lake grows. More tables under management means more signal about which strategies work for which table shapes. More engines means richer telemetry about cross-engine access patterns. More queries means more precise sort-order recommendations. The operational burden stays flat (or decreases) while the optimization quality increases. The system scales better than a team can — because it learns from the entire lake, not just the tables one engineer happened to look at.
Choosing Your Path
Three approaches to managed data lakes exist in 2027. Each suits different architectural choices and organizational realities. The right one depends on your engine diversity, cloud strategy, and team structure.
Vendor-Managed Platforms
Databricks (Predictive Optimization, Liquid Clustering) and Snowflake (automatic clustering, table maintenance) bundle operational management inside their platforms. If you run exclusively on one vendor's stack — all compute through their engine, all tables in their catalog — this is the lowest-friction path. Maintenance is handled within the ecosystem, tuned for their engine's access patterns, and requires no additional tooling.
The trade-off is architectural scope. Optimization runs only within that vendor's engine. If you later want to query those tables from Trino, DuckDB, or a different vendor — the core value proposition of choosing Iceberg in the first place — maintenance optimization will not account for those external access patterns. The vendor optimizes for its own queries. Cross-engine workloads are invisible to it.
Cloud-Native Services
AWS S3 Tables and Google BigLake provide managed Iceberg maintenance within their respective clouds. Compaction, cleanup, and optimization run automatically for tables registered with that service. No additional tooling needed for those tables.
The trade-off is scope. Maintenance applies to tables within that cloud service only. If your lake spans multiple catalogs, clouds, or includes self-managed Iceberg tables outside S3 Tables or BigLake, you need a separate solution for the rest. Cross-engine query-aware optimization is not typically available because the service lacks visibility into query workloads from external engines — it knows what Athena does but not what Trino or Spark are doing on the same tables.
Dedicated Control Plane
A platform like LakeOps operates across your entire stack — any catalog, any engine, any cloud — because it layers onto existing infrastructure rather than replacing it. It provides the full closed loop (observe → decide → act → optimize → govern → route → enable → learn) without constraining architectural choices.
The trade-off is that you add a service to your stack rather than relying on built-in capabilities. The benefit is that optimization accounts for your actual cross-engine workload, management applies lake-wide regardless of where tables live, policies are consistent across catalogs and clouds, and you retain full engine flexibility — add or remove engines without changing your management layer.
When to Choose Each
The decision framework is practical:
If you run 5–20 tables on a single engine from a single vendor, use that vendor's built-in management. The tooling is sufficient, it is already there, and the additional complexity of a separate service is not justified.
If you run 50+ tables across multiple engines, multiple catalogs, or multiple clouds, you need a layer that spans your stack. No single vendor or cloud service has visibility into your complete workload. Cross-engine telemetry, unified policies, and lake-wide governance require a purpose-built operations layer.
Many organizations run hybrid approaches — vendor-managed tables for workloads that live entirely within one ecosystem, and a dedicated control plane for the open multi-engine portion. The approaches coexist because they solve different scoping problems.
The Economics
Four cost vectors compound in an unmanaged lake. Left unchecked, each grows independently and they reinforce each other. They are also the four that a properly managed lake eliminates:
Storage drops 40–60% as orphan files, stale snapshots, and unreferenced data are continuously cleaned. This is not theoretical — it is the immediate, measurable result of running a proper orphan sweep and snapshot expiration. Production deployments routinely discover that a significant portion of their billable storage is serving no analytical purpose. The savings are recurring: every month you do not clean, the waste accumulates further.
Query compute drops as sort-aware compaction aligns data to real access patterns. When engines skip 80% of row groups via tight min/max statistics instead of scanning everything, compute consumption falls proportionally. Queries that scanned 100 GB to return 100 rows now scan 5 GB. The per-query savings compound across every query, every user, every dashboard refresh, every scheduled report — thousands of times per day across the organization.
Maintenance compute drops when a purpose-built Rust engine replaces Spark clusters for compaction workloads. No JVM startup overhead, no garbage collection pauses, no idle executor cores between compaction runs, no over-provisioning for peak load. Event-driven execution means compute runs only when needed (health signal fires) rather than on a fixed schedule (cluster running overnight regardless of whether any table needs attention).
Engineering time redirects from maintenance firefighting to building data products. Policies replace scripts. Observability replaces guesswork. Health classification replaces manual triage. On-call incidents for table degradation approach zero because degradation is prevented rather than repaired. The platform team stops being reactive (fixing things that broke) and becomes proactive (building capabilities that generate value).
The combined effect for teams running at production scale: the lake becomes cheaper to operate as it grows larger. More data, more tables, more engines — but costs grow sublinearly because the automation handles the marginal complexity without marginal engineering investment. This is the opposite of what happens without management, where cost and complexity grow linearly (or worse) with lake size.
Getting Started
The practical path, regardless of where you sit on the maturity spectrum:
Start with visibility. Connect your catalogs and see the health of every table before any maintenance runs. You will likely be surprised by how many tables are degraded — most teams discover Critical and Warning states on tables they thought were fine. The first insight alone (which tables are Critical, what orphan volume exists, where manifest bloat lives, which sort orders have drifted) changes how you prioritize the next sprint. This step costs nothing and requires no maintenance operations — it is purely diagnostic.
Fix the highest-impact tables first. You do not need to manage everything on day one. Identify the tables with the most file fragmentation, the highest query volume, or the largest orphan accumulation — ideally, tables that score high on all three. Run sequenced maintenance on those: expire, clean, compact, rewrite manifests. The ROI is immediate: dashboards that were slow become fast, storage bills drop measurably, and the team sees what managed looks like on real tables.
Establish policies early. Define your retention windows, target file sizes, compaction strategies, and cleanup schedules — even if you start with manual or semi-automatic execution. These decisions become the governance layer when you move to full automation. A clear, documented policy also prevents the entropy that accumulates when ten engineers configure tables differently based on individual judgment. Consistency across the lake matters more than perfection on any single table.
Expand to full automation. Once you trust the signals and see the impact on your initial tables, enable adaptive maintenance across the lake. Every table gets the right maintenance at the right frequency — coordinated, sequenced, conflict-aware, and fully observable. At this point, your platform team stops maintaining DAGs and scripts and starts building the data products and self-service capabilities that generate value for the organization.
Add routing and AI access last. Multi-engine routing and agent access depend on consistently healthy tables and predictable data layouts. They are the payoff of the operational foundation — not a starting point. A routing layer cannot optimize what it cannot predict. An AI agent cannot reliably query tables whose performance is unpredictable. Get the foundation right first; the advanced capabilities become straightforward once table health is reliable.
The constraint throughout: your data never leaves your storage. Your engines remain yours to choose and change. Your catalogs remain yours. The control plane adds operational intelligence without replacing anything in your stack — and if you ever remove it, everything continues working exactly as before, just without the automation.
Conclusion
Every production data lake faces the same operational reality: without an active system keeping tables healthy, Iceberg's open architecture degrades into slow queries, rising costs, and engineering time spent on maintenance instead of product work.
The problem is not the format. Iceberg provides exactly what it promises — reliable, open, engine-agnostic tables. The operational layer that continuously compacts, cleans, sorts, monitors, governs, and optimizes must exist separately. It must be built or adopted.
Netflix and Google proved the architecture works by building it internally over years with dedicated teams. In 2027, the same architecture is available without that investment. The approaches range from vendor-managed (Databricks, Snowflake) to cloud-native (S3 Tables, BigLake) to dedicated control planes — each fitting a different point on the architectural complexity spectrum.
The question is not whether your lake needs management — the degradation mechanics are inherent to how Iceberg works at scale. The question is which path fits your architecture, engine diversity, and team structure.
For teams running open, multi-engine Iceberg lakehouses at production scale, LakeOps is the control plane that closes the operational loop — across any engine, catalog, and cloud. Connect your existing infrastructure, define your policies, and let the lake manage itself.
Further learning:
