The lakehouse projects spent this week arguing about boundaries. Iceberg decided where conformance testing lives and started sketching the REST API shape that V4 tables will need. Polaris argued about what a committer owes a project when LLMs make pull requests cheap. Parquet pulled a feature apart because two proposals were reaching for the same mechanism. DataFusion and Iceberg Rust opened a joint thread about which repository should own their integration. Every one of those debates is a question about ownership, and the answers this week tell you a lot about how these communities plan to scale.
Apache Iceberg
The single biggest outcome of the week was the creation of a new repository. Neelesh Salian, working with Sung Yun and Andrei Tserakhau, called a vote to create apache/iceberg-verification, a standalone home for language-neutral conformance fixtures that every Iceberg implementation can run against. The vote passed with five binding +1s from Russell Spitzer, Sung Yun, Matt Topol, Daniel Weeks, and Amogh Jahagirdar, plus twenty-two non-binding votes. That is a wide turnout. The names on the non-binding list read like a roll call of the Rust, Python, Go, and Java maintainers, which is the point. Salian will now work with a PMC member to stand the repository up.
The reason this matters goes beyond tidiness. Iceberg has at least five serious implementations today across Java, Python, Rust, Go, and C++. Each one carries its own test fixtures and its own understanding of edge cases in the spec. When two implementations disagree about how to interpret a manifest list, users find out the hard way. A shared set of fixtures that every implementation reads from one place turns spec ambiguity into a failing test rather than a production surprise. The 29 messages in the vote thread also included a fair amount of discussion about what belongs in the first batch of fixtures, and the conversation is worth reading if you maintain a client.
The second major thread was about the REST catalog. Dhruv Arya's proposal on initial IRC changes needed for Iceberg V4 asked a pointed question: should V4 tables flow through the existing loadTable endpoint with new optional fields, or should the catalog spec add a versioned endpoint that returns V4 metadata with its richer structure? Amogh Jahagirdar argued that the first option is not really an option at all. Clients are not required to parse the format version before interpreting the rest of the metadata, so shoving V4 structures into the v1 response breaks the compatibility promise. He backed Daniel Weeks on a second endpoint that handles v1 through v4 tables, arguing that features like check constraints, default expressions, and generated columns give V4 metadata enough new semantics to justify a new versioned API.
Yufei Gu pushed back on one detail, noting that format-version is a required field inside TableMetadata, so a client can identify the version before reading the rest of the payload. Weeks replied that the two positions are close. The v1 loadTable stays frozen at the v1 through v3 shape, the v2 loadTable returns whatever structure a newer client can read, and a catalog that cannot serve a V4 table to a v1 client returns an explicit error rather than a payload the client cannot parse. Weeks also drew a line around scope. He wants the V4 changes bundled into the new endpoint, but he is not sold on folding partial metadata loading into the same work, since that feature still has vocal skeptics.
That thread is the practical start of V4 in the REST catalog. If you run a catalog service, the takeaway is that a new versioned loadTable endpoint is coming and the catalog is expected to advertise V4 support through its endpoint list. If you write clients, the takeaway is that V4 support means calling a new endpoint, not just parsing new fields.
V4 also moved on the deletes front. Huaxin Gao's vote to deprecate equality deletes in V4 and forbid new writes has a result, and Xiening Dai asked the obvious follow-up: when does the spec get updated? Gao said a spec PR is coming, and Renjie Liu added a late binding +1. This closes one of the longest-running debates in Iceberg. Equality deletes were always the cheap path for streaming writers and the expensive path for readers. V4 chooses readers.
A related cleanup thread came from Hongyue Zhang, who asked how maintenance actions should handle existing position deletes that carry row data. Position deletes with row data were deprecated in 1.11, and once the row schema is removed from the delete file builders, no new writes can produce them. The question is what rewrite-position-delete and rewrite-table-path do when they meet old files that still carry rows. Zhang favors failing with an explicit exception, which forces table owners to make a decision: upgrade to V3 and rewrite as deletion vectors, or stay on V2 and let data compaction fold the deletes into data files. Silently dropping the row column is easier but hides a migration step that operators should know about.
Compaction got a sharp new proposal from Heekyung Kim, who pointed out a blind spot in rewrite_data_files. The procedure selects files by size and delete count. A table whose files are all a healthy size but overlap heavily on the sort key never gets picked for sort or z-order compaction. Every run reports success, and clustering never improves. Kim's PR #17504 adds two pieces: a read-only compute_sort_order_stats procedure that reports per-partition overlap depth from manifest bounds alone, and an opt-in min-overlap-depth option that rewrites the files behind that depth. Anurag Mantripragada engaged on review, and Kim noted the core handler is a self-contained 283-line commit that can be evaluated on its own. This is the kind of fix that only comes from someone watching a real table refuse to get faster.
Hemanth Boyina proposed a small but useful spec change: add data-file size totals to manifest_list entries. Manifest lists already summarize file counts and row counts per manifest, but not bytes. Getting the total live size of a table today means opening and decompressing every manifest body. Three optional long fields, added_files_size_in_bytes and its existing and deleted siblings, give planners and monitoring tools a byte-level view at the manifest list layer with zero extra write cost. Boyina framed V4 as the right moment to close the gap.
Two spec votes wrapped up. Russell Spitzer announced the result for clarifying content file uniqueness in the table spec: passed with twelve +1s, binding votes from Steven Wu, Péter Váry, and Spitzer himself. Sung Yun cast a binding +1 on adding the variant type to the REST catalog spec, which has been open since late July. Gábor Kaszab bumped his vote on adding key-id to table and partition statistics and deprecating raw key-metadata. His framing is worth quoting in spirit: either raw key metadata in statistics is an existing feature and key-id is a nicer alternative, or raw key metadata is unsafe and key-id is the secure replacement. He believes the second, and he wants the community to say so explicitly.
The fine-grained read restrictions work continued in a dedicated sync. Prashant Singh posted notes from the August 18 session, and two decisions stand out. First, the group is inclined to forbid policies on both an ancestor and a descendant field in the same ancestry. Sung Yun surveyed the prior art: only Redshift supports overlapping policies with integer priorities, BigQuery is leaf-only, Unity Catalog and Snowflake are struct-column-only, and Trino, Hive, Impala, and Ranger have no support. The argument is interoperability, and the catalog is expected to deconflict or fail. Daniel Weeks followed up on the list to say he is fine putting the complexity on the catalog and disallowing overlap. Second, the group rejected capability negotiation. Laurent and Russell Spitzer pointed out that if clients advertise versions and two representations of a policy exist, a client can opt into the weaker one. TLS downgrade attacks are the precedent. The consequence, stated plainly in the notes, is that you upgrade clients before servers.
On the access delegation side, Singh replied to Weeks on the file-level access delegation proposal. The core Java work in PR #17457 adds HTTPInputFile and HTTPInputStream so FileIO can detect a presigned URL and use the right stream. Singh also floated a bulk signing API. Per-file remote signing was always a concern for very large tables because it can overwhelm the server. A bulk endpoint that returns presigned or remote-signed responses in one round trip takes the pressure off.
Releases moved on several fronts. Neelesh Salian's 1.12.0 release thread settled a few scope questions. The Hilbert curve PR is in and on track. Spark 4.2 support was pulled from the milestone because it touches a large surface, but Manu Zhang found a middle path: merge the source while excluding it from both the binary and source tarballs, following a suggestion from Szehon. Alexandre Dutra's REST changes, PRs 17709 and 17627, were added to the milestone. Cheng Pan flagged that the docs site still references old Flink and Spark versions that Iceberg no longer supports, which is a small thing that confuses newcomers.
PyIceberg is close to a big release. Alex Stephen proposed 0.12.0rc2 with view support, geometry and geography types, Python 3.14 support, a new file format API, and commit retry support. Aaron Niskode-Dossett of Etsy raised a gap in the same week: 0.12 supports vended credentials, but not automatic refresh of those credentials. He pointed at an open PR for S3 refresh and volunteered to add GCS support if the approach is accepted. Stephen reviewed it and wants it in. If you run long PyIceberg jobs against a REST catalog that vends short-lived credentials, this is the fix you have been waiting for.
The Terraform provider hit a snag. Sung Yun cast a binding -1 on the v0.1.0 RC2 vote after spotting that the iceberg-go dependency jumped from 0.5.0 to 0.6.0 between candidates, pulling extra modules into the binary without a matching LICENSE-binary update. He also noted the Go build info embeds the RC version because the binary was copied rather than rebuilt, which trips SBOM tooling. Yun opened issues to automate license detection for future releases. It is a small release, but the ASF release process does not have a small mode.
Danny Jones of Amazon proposed moving forward with iceberg-rust 0.11 and volunteered as release manager, with Shawn Chang backing him on committer-only steps. The last minor release took from June to August to ship, and Jones wants a smoother cycle this time. Matt Butrovich opened a tracking issue for blockers.
The Rust project also opened the week's most interesting cross-list conversation. Butrovich posted to both dev@iceberg and dev@datafusion about moving the DataFusion integration out of iceberg-rust into its own repository. The integration serves two purposes today: it is the engine that runs iceberg-rust's sqllogictest suite, and it is the TableProvider that DataFusion users rely on. The motivations to split are concrete. Feature PRs against the TableProvider go stale because few iceberg-rust committers use DataFusion. The project wants to stay engine-agnostic, and it already declined a Ballista integration on those grounds. Downstream projects like Comet get blocked waiting for iceberg-rust to bump its DataFusion and Arrow versions.
Shawn Chang agreed in principle but named the risk. His biggest worry is governance. If the integration leaves Apache, it can drift toward the shape of the Iceberg Java and Trino relationship, where the integration lives on the engine side and is maintained by engine people. That works for Java because Spark is the primary engine with a large Iceberg contributor base. It does not map well to Rust, where DataFusion is by far the most mature engine integration and the two communities overlap heavily. Chang wants the extracted repository to stay Apache-governed. Expect this to land somewhere between apache/iceberg-rust-datafusion and a DataFusion-side contrib crate, and expect the sqllogictest question to decide it.
Community news rounded out the week. Scott Haines announced a virtual Apache Iceberg meetup series and put out a call for speakers. Talks run 20 to 30 minutes on Google Meet, recordings go to the Apache Iceberg Meetup YouTube channel, and vendor pitches are explicitly out of scope. Colby Foss floated an SF Iceberg meetup in late September. Danica Fine reminded everyone to register for Lakehouse Day EU 2026. Varun Lakhyani wrapped up his GSoC 2026 project and said he plans to keep contributing, with Anurag Mantripragada replying to encourage him.
One new integration is worth watching. Gianluca Graziadei announced storm-iceberg, a new Apache Storm module that writes streaming tuples directly into Iceberg tables from inside a Storm topology, skipping Kafka Connect, Flink, and Spark entirely. It is append-only with at-least-once semantics, and it splits commit policy across two independent dimensions: writers roll on a target file size to fight small files, and a tick tuple commits eligible files on a timer to bound visibility latency. A write-ahead log drives recovery. The module is merged on Storm master and targets Storm 3.1.0. Graziadei asked for benchmarks against existing sinks using someone else's methodology, which is a refreshingly honest way to ask for help.
The _pos column question in the column-update proposal also got air time ahead of the August 25 sync. Marco Kroll argued that the dense null-filled representation already encodes position implicitly, so a separate _pos column is redundant for both debugging and detecting skipped rows. Leonid Lygin agreed that row counts are good enough for detecting gaps and questioned whether locating the exact gap is worth the storage. Nine messages in, the thread was leaning toward dropping the column and requiring that row order match the base file.
Apache Polaris
Yufei Gu opened the week's most important governance thread with a note on PR review and committership. His argument is short. LLMs make it easy to write code and open PRs. Polaris has seen a flood of them, which is a good problem, but the bottleneck is now reviewers. Gu said he will give sustained, high-quality review more weight than PR count when considering someone for committership. He named three reasons: the community needs to trust a committer to merge responsibly, more good reviewers raise everyone's quality, and a good committer knows when to ask someone with more context before merging. Jean-Baptiste Onofré replied in support. Every Apache project is going to have this conversation in 2026. Polaris is having it early and in public.
On the release front, Onofré proposed Polaris 1.8.0 for early September, keeping the monthly cadence. He wants the release to include a preview or beta of a new feature, naming Directories, Tags, OpenLineage, or Data Sharing as candidates. He also plans to link every open proposal to a GitHub issue so the proposal tracking view stops missing things. Yufei Gu +1'd the timeline. Onofré also noted he is back after three weeks off.
Tags are the most likely candidate to make that preview cut. EJ Wang opened a PR for the public API contract covering tag management, assignment and unassignment, direct and inherited reads, and reverse lookup. V1 scope is catalogs, namespaces, Iceberg and generic tables as whole objects, and top-level Iceberg table columns. Views, generic table columns, nested fields, multi-value assignments, and tag-based authorization are deferred. Wang plans four PRs in sequence: API contract, tag CRUD, assignment writes, then reads and reverse lookup. Grants on tag resources come in a separate follow-up, distinct from using tags to control access to tagged objects.
A configuration bug turned into a design question. Ayush Saxena raised the default value of DROP_WITH_PURGE_ENABLED, and Onofré confirmed the problem is real. With the purge guard enabled and PURGE_VIEW_METADATA_ON_DROP also true by default, views cannot be dropped at all out of the box. The drop internally requests purge, and the guard blocks it with a 403. Onofré called it a category error: the purge guard was designed to protect table storage, and views have no comparable storage risk. He favors scoping the guard to tables only and letting the view flag stand on its own. Four messages in, that looks like the direction.
Persistence had two threads. The relational JDBC schema name discussion ran six messages, with Yufei Gu arguing for a Polaris-owned property like polaris.persistence.relational.jdbc.schema-name rather than relying on Quarkus datasource properties. His point is that the Quarkus route works for PostgreSQL but not for the MySQL driver, and a Polaris-owned property keeps driver-specific details behind a stable interface. Alexandre Dutra came around on single versioned DDL scripts after a PR had to bump the H2 script version for no reason because a PostgreSQL-specific fix shared the version number. Versioned per-database scripts avoid that.
Federation raised a trust question. Jiajia Li of Alibaba asked whether federated catalogs should forward the remote's storage credentials. Today a federated catalog mints credentials from its own storage config. When the remote catalog owns the storage, there is no local config, and every route fails. Li built an opt-in per-catalog flag that forwards the remote's credentials instead, read off the loaded table's FileIO. The catch is that Polaris stops being the location policy point on that path, because it cannot validate locations against storage it did not configure. Li asked the list directly whether Polaris should take this on, noting the alternative is engines bypassing federation and talking to remote catalogs directly. Yufei Gu replied. This one deserves more eyes, because it decides whether Polaris federation is a proxy or a policy layer.
Two tooling notes. Sung Yun reported that ASF Infra created apache/terraform-provider-polaris and scaffolded it with the basics. Ajantha Bhat plans a release of iceberg-catalog-migrator 1.1.0 now that Polaris 1.6.0 and Iceberg 1.11.0 make view migration possible. Alexandre Dutra also replied on forwarding user-defined principal properties in PR #4405.
Apache Arrow
Arrow's week was quieter on the format side and busier on the people side. Antoine Pitrou announced Zehua Zou as a new committer, and eleven people replied with congratulations, including Gang Wu. That is the most active thread on the list this week, which tells you something about where Arrow's energy is right now.
The canonical BigDecimal extension type vote is not done. Micah Kornfield took another pass through the document and raised a process concern: the chosen representation does not seem to follow from clearly stated requirements. He asked Curt to bring his concerns into the doc so they can be closed. This is the same kind of requirements-first pushback that showed up on Parquet's decimal floating-point thread, and it is not a coincidence. Arrow and Parquet are both being asked to represent decimals wider than 38 digits, and both communities want the requirements nailed down before the bytes are.
Kosta Tarasov bumped the thread on the Variant extension spec being inconsistent with Parquet's shredding spec. He submitted a docs PR, and Andrew Lamb suggested it go through the mailing list first because the changes amount to a spec change. If you are building Variant support in an Arrow-native engine, this inconsistency is the kind of thing that produces subtle bugs when data crosses from Parquet into Arrow memory.
Two smaller threads are worth a mention. Nic Crane's proposal to limit concurrent open PRs for non-committers landed on three, with Pitrou agreeing. This is the same review-capacity problem Polaris is discussing, solved with a different lever. Erik Carstensen asked whether lazy memory mapping of Parquet columns is useful in practice. His library uses custom fault handling to expose an entire column through an mmap-like interface, loading row groups lazily as values are accessed, which lets you write plain vectorized NumPy over a whole column without handling row groups explicitly. He asked the honest question: is hiding row groups a feature, or do applications want them visible?
Ivan Ogasawara introduced ArxLang, an experimental programming language that uses Arrow C++ as the native foundation for its arrays, tensors, DataFrames, and RecordBatches, lowering to LLVM IR through llvmlite. Two GSoC contributors worked on the Arrow integration this summer. Jean-Baptiste Onofré said he is resuming Arrow Java 20.0.0 release prep after being away, and Ian Cook announced the Arrow community meeting for August 26 at 16:00 UTC.
Apache Parquet
Parquet had the sharpest format debate of the week, and it ended with a proposal being withdrawn in favor of a cleaner design. Alkis Evlogimenos withdrew his thread on self-references in FILE inheriting the inline codec and replaced it with a vote to remove self-references from the FILE logical type entirely.
The reasoning is worth following. FILE as merged lets a value set an offset and size with no URI, which addresses a byte range inside the containing file. That byte range is owned by the Parquet writer, and specifying it properly means giving it a compression block, an encryption module, an AAD identity, and its own size accounting. In other words, a second page mechanism reachable only through FILE. Evlogimenos and others concluded that mid-sized values are better served by a separate non-contiguous pages proposal, which will let outside values of the inline column live elsewhere in the file as ordinary pages that inherit is_compressed, encryption, and uncompressed_page_size for free. Non-contiguous pages composes with FILE instead of competing with it.
The vote PR makes four changes. Self-references are removed, so offset can only be set together with a URI. A URI is resolved uniformly as an external reference, even when it names the containing file, and Parquet applies no compression or encryption of its own to those bytes. The prohibition on modular encryption is dropped, since FILE group fields are ordinary columns. And inline is allowed alongside the locator fields, with both required to denote the same bytes so a reader can use either. The format has not shipped in a release, so this is the right time to make the cut.
Russell Spitzer voted +1 with a note on that last change. He questioned whether the spec should say the inline bytes and the located bytes must be identical, since nothing inside Parquet can enforce that. He suggested softer language: when a locator is present, a reader can use the locator or the inline bytes interchangeably. Nine messages in, that wording question was the main open item.
ALP, the adaptive lossless floating-point encoding, is getting its public launch. Kosta Tarasov, Andrew Lamb, and Prateek Gaur posted the ALP blog for review with a rendered preview covering motivation, performance results, a technical overview, and ecosystem adoption. Lamb also proposed moving the ALP spec to its own document page and an example file for implementations, with Vinoo Ganesh offering to help. Kevin Liu noted the ALP thread landed in his Gmail spam folder and has an INFRA ticket open about it, which is a reminder to check your filters if the list has felt quiet.
The extensible decimal floating-point type proposal continued with Thomas Kissinger replying to Costas. Both sides agree the type should not impose a permanent 38-digit ceiling and should cover at least the full finite decimal128 range. Kissinger flagged a mismatch with the generalized IEEE interchange layout, whose precision ladder moves from 34 to 43 digits around the SQL 38-digit boundary and from 70 to 79 around 76. Signed 128-bit and 256-bit integer significands support 38 and 76 digits directly, which lines up with how databases already store decimals. He also pointed to a parallel Spark SPIP and argued that both projects benefit if they converge on shared requirements before either finalizes a design.
Fokko Driesprong opened a thread for Parquet-Java 1.18.1 after regressions were found in 1.18.0, with a milestone to track what goes in. Julien Le Dem reminded everyone of the Parquet sync on August 26.
Apache DataFusion
DataFusion shipped. Andrew Lamb announced that 55.0.0 RC3 passed with 8 +1 votes, 7 binding, and completed the final release steps himself since Tim Saucer was out. The release is on dist.apache.org and crates.io. Lamb hit one small error in the process and filed an issue so the next release manager does not. Saucer, back the following week, proposed a 55.1.0 patch release within days and asked contributors to tag backport candidates on the tracking issue.
Andy Grove announced Manu Zhang as a new committer, and eleven people replied. Zhang's name also appears on the Iceberg 1.12 release thread and the iceberg-verification vote, which makes him one of a growing number of people active across both projects.
Lamb also opened a discussion on streaming support, pointing to a GitHub issue that sketches a design for streaming SQL in DataFusion. DataFusion has always been a batch engine that happens to be very good at incremental execution, and a first-class streaming story changes what people build on it. The thread is early, and the issue is where the design conversation is happening.
The DataFusion side of the iceberg-rust integration thread is the same message Butrovich sent to Iceberg, with Shawn Chang's reply mirrored. Reading both lists together is the only way to see the full picture, which is exactly why Butrovich cross-posted.
Apache Ossie (incubating)
Ossie, the open semantic layer specification, is pushing toward its first release. Jean-Baptiste Onofré revived the first-release thread and proposed starting with a source-only distribution to verify the build and release process. He also argued that converters should ship on independent release cycles rather than as part of a single project release, since a fix to one converter should not require re-releasing everything, and each converter needs its own artifacts anyway.
Yufei Gu asked whether the Python converters should consolidate into one project with shared build and test setup, a common converter interface, and a consistent CLI such as ossie-convert import databricks or ossie-convert export snowflake. Heavier dependencies like MetricFlow or sqlglot stay optional extras. The tradeoff he named is a shared release cadence, and he suggested vendor-maintained converters eventually move to their own repositories. Onofré replied, and the two threads together are really one question: what is the unit of release for a spec with many converters?
Markus Weimer reported that the Power BI converter work has started in earnest and asked for reviews on two foundational PRs before the converter itself lands, which he expects to review piecemeal. Ankit Tandon of RelationalAI posted notes from the Ontology working group sync. A GitHub discussion on how OSI is expected to be used surfaced a question from jakub-moravec about automated compatibility checking, drawing on experience with OpenLineage. A separate GitHub proposal on shared filters, shared dimensions, and metric references drew replies. Onofré welcomed Yong Zheng as a new committer and posted the draft September incubator report for review.
Cross-Project Themes
Three patterns connect the projects this week.
The first is review capacity as the scaling constraint. Yufei Gu said it directly on the Polaris list: LLMs make PRs cheap, so review is the bottleneck, and committership should reward review. Arrow reached for a blunter tool, capping non-committers at three open PRs. Iceberg's Terraform provider RC failed because a dependency bump slipped past license review, and Sung Yun's response was to automate the check. DataFusion's release manager hit a process error and filed it so the next person will not. Every project is discovering that the limiting factor on throughput has moved from writing code to verifying it, and each one is adjusting its process to match.
The second is ownership boundaries. Iceberg created a separate repository for conformance fixtures so no single implementation owns the spec's test suite. Iceberg Rust and DataFusion are negotiating which project owns their integration, and the governance question matters more than the code location. Parquet removed self-references from FILE because the byte range they described was really owned by a different, not-yet-written proposal. Polaris federation is asking whether Polaris owns location policy when a remote catalog owns the storage. Ossie is deciding whether converters are part of the spec or separate products. In every case the answer is being chosen to make the boundary explicit rather than convenient.
The third is V4 becoming real. The IRC endpoint discussion, the equality delete deprecation, the position-delete cleanup, the manifest list byte totals, and the _pos column debate are all V4 conversations. They are happening on the list now because the spec work is moving from principles to concrete wire formats and API shapes. If you have opinions about how V4 tables load over REST, this is the month to state them.
Decimals deserve a footnote. Arrow's BigDecimal extension vote and Parquet's decimal floating-point proposal are both stalled on the same request: state the requirements before choosing the representation. Micah Kornfield asked for it on Arrow. Costas and Thomas Kissinger are working through it on Parquet. Whatever lands should land in both formats at once, and the people involved know it.
Practitioner Notes
The threads above are community process, but several of them change what you should do with your own tables this quarter. Here is the practical reading.
If you write to Iceberg from a streaming engine, start planning your exit from equality deletes now. V4 forbids new equality delete writes, and V3 already gives you deletion vectors as the replacement for position deletes. Flink and Kafka Connect pipelines that lean on equality deletes for upserts will need a merge-on-read strategy built on deletion vectors or a periodic compaction that resolves upserts into data files. The vote closed this week. The spec PR is coming. The engines will follow over the next two or three release cycles, and the tables you create today will live long enough to meet V4.
If you have V2 tables with old position delete files that carry row data, audit them before you upgrade the Iceberg library. Hongyue Zhang's proposal has maintenance actions fail loudly when they meet those files rather than silently dropping the row column. That is the right behavior, but it means a rewrite_position_delete job that ran clean last month can start failing after an upgrade. The fix is either an upgrade to V3 with a rewrite to deletion vectors, or a data compaction pass on V2 that folds the deletes into data files. Both are cheap compared to discovering the problem during an incident.
If your sorted tables have stopped getting faster, Heekyung Kim's overlap diagnosis is worth trying even before the PR merges. The symptom is specific: rewrite_data_files reports success every run, file sizes look healthy, and query latency on the sort column does not improve. The cause is that size-based selection never picks size-healthy files no matter how badly they overlap on the sort key. You can approximate the compute_sort_order_stats procedure today by reading manifest lower and upper bounds for the sort column and counting overlaps per partition. If the overlap depth is high, force a sort compaction with a low target file size once, then let the new option take over when it ships.
If you run PyIceberg jobs longer than an hour against a REST catalog that vends credentials, watch the refresh PR. Until it lands, the workaround is to keep individual jobs short or to pass long-lived credentials outside the vending path. Neither is great, which is why Aaron Niskode-Dossett raised it.
If you build or operate a REST catalog, the V4 endpoint discussion is your homework. The direction is a versioned loadTable endpoint that serves V1 through V4 tables, an explicit error when a V1 client asks for a V4 table, and catalog support for V4 signaled through the endpoints list rather than a new capability flag. Start thinking about how your catalog stores V4 metadata structures like check constraints and default expressions, because those are the fields that make the new endpoint necessary.
If you run Polaris with views, check your purge settings. The default combination blocks view drops entirely. Until the fix ships, the workaround is to set DROP_WITH_PURGE_ENABLED to true or to set PURGE_VIEW_METADATA_ON_DROP to false, depending on which risk you prefer. Ayush Saxena's thread describes the exact failure mode.
If you federate Polaris to a remote catalog that owns its storage, read Jiajia Li's thread and form an opinion. The forwarding flag she built works, but it takes Polaris out of the location validation path for that catalog. That is acceptable if you trust the remote catalog as much as you trust Polaris, and dangerous if you do not. The community has not decided yet, and your use case is exactly the kind of input they need.
If you are implementing Parquet FILE or non-contiguous pages, wait for the vote to close before writing code. The self-reference removal changes what a valid FILE value looks like, and the non-contiguous pages proposal that replaces it has not been posted yet. Writing to the merged-but-unreleased spec today means rewriting next month.
If you are writing floating-point columns at scale, the ALP blog and example file give you what you need to test the encoding against your own data. ALP is already implemented in several readers and writers, and the example file exists specifically so implementations can verify they agree byte for byte. Run it against whichever engines you use before turning ALP on in production.
If you maintain any Iceberg client, plan to consume apache/iceberg-verification fixtures in CI as soon as the first batch lands. The whole value of the repository comes from every implementation running the same tests. A client that skips them is a client that finds out about spec disagreements from its users.
And if you contribute to any of these projects, take Yufei Gu's note seriously. Reviewing three PRs carefully is worth more to the project right now than opening three more, and the committers are starting to say so out loud.
Looking Ahead
Watch for the apache/iceberg-verification repository to appear and for the first fixture PRs. The Iceberg V4 IRC thread should produce a concrete PR against the REST spec once Weeks and Arya settle the oneOf question. PyIceberg 0.12.0 should ship if rc2 passes, and the vended credential refresh PR is a strong candidate for 0.12.1. The iceberg-rust 0.11 branch cut and the DataFusion integration decision will move together. Polaris 1.8.0 lands in early September, and the Tags API PR should go from draft to ready for review before then. Parquet's FILE self-reference vote should close, and the non-contiguous pages proposal it depends on should hit the list. DataFusion 55.1.0 is days away. And Ossie's first source release will be the podling's most important milestone to date.
Two dates on the calendar matter for anyone who wants to be in the room. The Arrow community meeting was August 26 at 16:00 UTC, and Ian Cook's notes usually land on the list within a day. The Parquet sync was the same day, and the FILE self-reference wording, the ALP blog, and the decimal floating-point requirements were all on the agenda. Julien Le Dem's reminder thread is where the notes will show up. The Iceberg community sync on August 25 covered the _pos column and the read restrictions work, and the recording link will follow the pattern of the August 18 session notes Prashant Singh posted.
If you only have time to read three threads from this week, read the iceberg-verification vote result for the list of implementations that showed up, read the IRC V4 endpoint thread for the shape of the next REST spec, and read Yufei Gu's committership note for the clearest statement yet of how an Apache data project plans to handle a world where code is cheap and judgment is not. Those three tell the story of the week better than any summary, including this one.
If this newsletter is useful and you want the longer-form version of how these projects fit together, I write books on Apache Iceberg, Apache Polaris, the lakehouse, and agentic AI on data. You can find all of them at books.alexmerced.com.
