Redshift and Iceberg v3: field notes for mutable lakes

When I look at Amazon Redshift support for Apache Iceberg v3 tables, the interesting part is not simply "it now reads and writes v3". The architectural point is that three old pains in analytical lakes start becoming table contracts: schema evolution with explicit defaults, stable row identity for incremental pipelines, and compact deletes for workloads with frequent update/delete operations. In financial-grade environments, that directly touches reconciliation, erasure workflows, audit trails, scan cost, and coupling between engines.

The situation: the lake is no longer only append-only

For years, a relevant share of enterprise data lakes worked well because the implicit contract was simple: immutable Parquet files, predictable partitions, compaction jobs, and mostly append-only consumption. That model still holds for raw events, logs, and regulatory snapshots. It starts to creak when the platform becomes a shared data product across risk, fraud, customer operations, Open Finance, BI, credit models, and audit.

In those scenarios, the operational question is not "can I query S3 with SQL?". The question is whether I can modify a row without destroying performance, add a column without reprocessing terabytes, propagate only changes to downstream consumers, and prove what changed between snapshots. The Redshift announcement on August 31, 2026 matters because it brings Iceberg v3 features into a SQL engine many organizations already use as a consumption, transformation, and governance layer.

I would not read this as permission to turn every lake table into an OLTP database. Iceberg is still an analytical table format. The gain is reducing patches: CDC mirror tables, inconsistent technical columns, accumulated delete files, and pipelines that scan everything because there is no trustworthy change marker.

What actually changed

Redshift can now create or upgrade Iceberg v3 tables with the 'format-version' = '3' property, while keeping SQL DML syntax similar to v2.
Default column values reduce reprocessing during schema evolution, but require governance over the historical meaning of the default.
Row lineage exposes _row_id and _last_updated_sequence_number for incrementality, audit, and reconciliation, as long as consumers understand the contract.
Deletion vectors replace new positional delete files in v3 and help workloads with many deletes, updates, and MERGEs, especially when table maintenance is disciplined.
Compatibility is not universal: complex types and some Iceberg v3 types are not currently supported by Redshift, and downgrade from v3 to v2 is not supported.

Default values: schema evolution without rewriting the past

The most pragmatic use of default values appears when a large table needs a new operational column: source_system, risk_bucket, consent_status, schema_version, retention_class, or something similar. In Iceberg v2, many teams solved this with expensive backfills, views with coalesce, or different rules per consumer. In Iceberg v3 on Redshift, a column added with a default can return the initial value for older files without rewriting data, and new INSERTs can write the default when the column is omitted.

The trap is semantic. A default does not mean the historical data truly had that attribute at the original event time. In regulated environments, I separate technical defaults from business assertions. schema_version DEFAULT 3 is relatively safe. customer_consent DEFAULT true is dangerous if there is no historical basis for it. It is also worth noting that defaults are literals, not dynamic expressions; I would not rely on them for processing timestamps or contextual rules.

In practice, I would record each change in a short ADR, with effective date, reason, domain owner, and consumer impact. If the default changes metric interpretation, the publication needs a contract version and a coexistence window.

Row lineage: analytical CDC with less guessing

The feature I would watch most closely is row lineage. Redshift exposes _row_id and _last_updated_sequence_number pseudo-columns, which must be selected explicitly and are not included in SELECT *. That is a good decision: it avoids accidental leakage into dashboards, but lets technical pipelines create sequence-based checkpoints. For an incremental pipeline, the basic contract becomes simple: persist the last processed sequence number, read rows with a sequence greater than or equal to the planned watermark, write downstream results idempotently, and advance the checkpoint only after the downstream commit.

I would still not replace every CDC mechanism with this. In integrations with transactional systems, I still want source logs, partition ordering, and business events. Row lineage is strongest inside the lakehouse: deriving silver/gold tables, recalculating affected aggregates, feeding feature stores, rebuilding search indexes, and assembling analytical audit trails.

One operational detail matters in a v2-to-v3 upgrade: pre-upgrade data does not immediately get useful lineage; Redshift documentation says the values return null until the first write after the upgrade, when values are generated for the table. So the migration needs an explicit cutover line. I would treat that line as a platform event, not an invisible detail.

Iceberg v3 table pattern for controlled mutations

The design shows how I would isolate ingestion, table contract, incremental queries, and governance when adopting Iceberg v3 with Redshift.

🟦 Domínios produtores

Eventos de negócio Kafka/MSK ou batch (messaging)
Glue/EMR jobs validação e merge (compute)

🟧 Contrato lakehouse

AWS Glue Data Catalog metadados Iceberg (data)
Tabela Iceberg v3 format-version=3 (storage)
Deletion vectors Puffin + bitmap (storage)
Row lineage _row_id + sequence (data)

🟥 Governança

Lake Formation FGAC e grants (security)
KMS + S3 policies criptografia e perímetro (security)

🟩 Consumo e operação

Amazon Redshift Graviton provisioned/serverless (compute)
Checkpoint incremental última sequência processada (data)
CloudWatch/SLOs scan, latency, failures (compute)

Flows

events -> etl: normalizes and validates
etl -> table: INSERT/UPDATE/MERGE
catalog -> table: schema and snapshots
table -> dv: marks removed rows
table -> lineage: identifies changes
lf -> catalog: authorizes tables/columns/rows
kms -> table: protects S3 objects
redshift -> catalog: discovers table
redshift -> lineage: queries by sequence
redshift -> checkpoint: advances after commit
redshift -> obs: emits operational signals

Deletion vectors: fewer delete files, not zero maintenance

Deletion vectors address a familiar problem: updates and deletes on immutable files tend to generate auxiliary structures that hurt reads, planning, and compaction. In Iceberg v3, Redshift records removed positions in compact bitmaps stored in Puffin files, with at most one deletion vector per data file in a snapshot. That reduces positional delete file proliferation and makes regulatory deletes, customer-data corrections, and CDC MERGE workloads less costly to read and write.

But I would not sell this internally as free maintenance. If a table receives random deletes all day, the data files still physically contain invalidated records. Reads are better than with many small delete files, but there is still additional work. For high-churn tables, I would keep explicit policies for compaction, file rewrite, and snapshot expiration, using low-demand windows for maintenance.

A practical number I often use as an initial trigger is not absolute: when the share of logically deleted rows in hot partitions crosses 5% to 10%, or when P95 latency on queries over those partitions grows consistently, I review compaction. I am not presenting this as an AWS limit; it is an operational heuristic to start measuring before users feel it.

The architecture point: interoperability with clear boundaries

The value of Iceberg is allowing multiple engines to share a table without turning the lake into accidental coupling. Redshift, Athena, EMR, Glue, and other tools can participate, but I would only allow multiple writers where ownership is explicit. For critical tables, I choose one primary writer per domain and treat the others as readers or transformers with very specific permissions.

In Redshift, some details shape the design. The documentation states support for Lake Formation fine-grained access control on Iceberg tables, the use of Glue-generated column statistics for better performance, and different cost behavior depending on compute type: RG and Serverless use their own compute for S3 lake queries, while DC2 or RA3 use Redshift Spectrum. The pricing page confirms that Spectrum charges by bytes scanned, rounded to the next megabyte with a 10 MB minimum per query; in Serverless, external queries are part of the workgroup's RPU-hour consumption.

I would configure this as a platform product: S3 bucket with SSE-KMS, policies conditioned on aws:PrincipalArn and aws:SecureTransport, Lake Formation as the authorization plane, static roles for writes, sensitivity tags, and workload-specific budgets. The common mistake is treating an open format as absence of governance. It is the opposite: the more open the format, the clearer the contract must be.

The adoption playbook I would use

Choose a table with real mutation, not the most critical one: Look for a table with frequent MERGE or DELETE activity, known consumers, and an operational rollback path. Avoid starting with accounting ledgers, intraday risk, or tables with multiple writers and no clear owner.
Audit engines and types before the upgrade: Confirm that all relevant readers and writers understand Iceberg v3. In Redshift, do not plan v3 for tables that depend on struct, list, map, variant, geometry, geography, binary, uuid, time, or nanosecond timestamps, because the documentation lists those limitations.
Define the incrementality contract: Standardize checkpointing by _last_updated_sequence_number, a reread window to tolerate retries, an idempotency key in the target, and a metric comparing rows read with rows actually applied.
Run the upgrade as a platform change: The format ALTER is metadata-only, but the consequence is not trivial: there is no downgrade to v2, pre-upgrade data has specific lineage behavior, and old positional deletes remain valid until later writes merge them into deletion vectors.
Measure before and after with the same query set: Collect P50/P95/P99, bytes scanned, file count, MERGE duration, Lake Formation permission failures, planning time, and cost by domain. Without a baseline, any improvement becomes a story.

Start with the contract, not with ALTER TABLE: Before running ALTER TABLE ... SET TABLE PROPERTIES ('format-version' = '3'), write down three things: who may write, how incremental consumers advance checkpoints, and which metric triggers compaction. The command is short; the governance around it is what prevents incidents.

Anti-patterns I would avoid

Upgrading to v3 because it is new, without inventorying the engines that read the table. In a lakehouse, silent incompatibility across tools costs more than the upgrade.
Using default values to hide lack of data governance. A technical default helps evolution; a poorly defined business default creates historical error that looks like complete data.
Treating _last_updated_sequence_number as a universal substitute for domain events. It is excellent for analytical change, but it does not carry business causality by itself.
Allowing multiple writers without an ownership, retry, and idempotency protocol. The format supports interoperability; it does not solve organizational contention.
Ignoring column statistics and file maintenance. Redshift can optimize better when statistics exist; deletion vectors reduce pain, but they do not remove table hygiene.

Observability, security, and cost in production

I would put Iceberg v3 under the same observability discipline as any critical service. For each candidate table, I keep a dashboard with hourly write volume, rows updated/deleted, age of the latest snapshot, small-file count, MERGE duration, read P95 by query class, and scanned-bytes variation. In Redshift Serverless, I would also use RPU-hour usage limits by day, week, or month; the documentation allows actions such as logging, alerting, or turning off user queries when the limit is reached.

On security, I avoid relying only on IAM at the bucket. Lake Formation should express permissions by table, column, and where applicable row/cell; S3 and KMS should reinforce perimeter, encryption, and access by expected roles. Writing to Iceberg through Redshift should not use federated identity with SESSION, according to the documented consideration; I prefer a static platform role with least privilege to specific table paths and corresponding KMS keys.

For engineering leadership, the most important metric may be recovery time. If a faulty MERGE applies deletion vectors or bad defaults, how do I return to a previous snapshot, who approves it, how long does it take, and which consumers need reprocessing? Without that answer, adoption is not ready.

Questions I would ask in a design review

Should I convert all Iceberg v2 tables to v3?

No. I would prioritize tables with frequent schema evolution, recurring MERGE/DELETE activity, or incremental consumers. Stable append-only tables can remain on v2 until there is clear benefit and validated compatibility.

Does row lineage replace regulatory audit?

Not by itself. It improves analytical traceability and incrementality, but regulatory audit still needs decision trails, actor identity, approval, source evidence, and controlled retention.

Do deletion vectors eliminate compaction?

No. They reduce the operational cost of deletes compared with many positional delete files, but high-mutation tables still need maintenance policy, file rewrite, and snapshot expiration.

What is the largest upgrade risk?

Engine compatibility and data semantics. Redshift does not support downgrade from v3 to v2, and there are documented type limitations. I would validate readers, writers, Lake Formation grants, and critical queries on a copy before the change.

My curator note: I would adopt Iceberg v3 on Redshift first where there is measurable pain: analytical CDC, regulatory deletions, and contract evolution on large tables. The practical lesson is that open formats do not reduce the need for architecture; they move the discipline into metadata, ownership, and operations. If I cannot explain who writes, who compacts, who authorizes, and who reprocesses, I do not yet have a production design.

References

Verdict

My recommendation is to adopt Redshift with Iceberg v3 selectively and with discipline: use it for mutable analytical tables where default values, row lineage, and deletion vectors solve a real operational problem. Do not mass-upgrade. Inventory engines, validate supported types, define write ownership, monitor cost and performance, and treat the migration as a data contract change. For financial-grade platforms, this launch matters because it moves the lakehouse closer to an auditable and incremental model without abandoning S3, Glue Catalog, Lake Formation, and SQL; the value appears when the team combines the technical feature with production governance.

Rating: adopt selectively

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.