This week the open lakehouse projects spent their energy on two kinds of plumbing. The first kind is data types. Parquet passed a new vector type, Arrow voted on a matching FILE extension type, and Iceberg contributors asked how those types should be signed, authorized, and queried. The second kind is trust. Polaris worked through authorization for view chains, row and column policies, and remote signing, while three projects rebuilt release candidates over license files, signing keys, and stray binaries.

Six dev lists carried more than 490 messages between November 3 and November 11. Ossie and Polaris were the busiest by far, with roughly 150 messages each. Below is what happened on each list, why it matters, and where the threads connect.

Apache Iceberg

V4 moves from design to execution

The biggest Iceberg news of the week was quiet but important. Daniel Weeks closed the vote on Default Value Expressions for v4 with 11 binding and 7 non-binding +1 votes. The vote thread itself drew late support from Amogh Jahagirdar, Yufei Gu, Junwang Zhao, Gianluca Graziadei, and Jean-Baptiste Onofré. Russell Spitzer summed up the mood in one line. He is waiting for the implementation pull requests.

Default value expressions let a column carry a computed default instead of a fixed literal. That matters for schema evolution, where new columns need sensible values for old rows. It also matters for engines, since every engine has to agree on what the default resolves to.

The second v4 vote of the week did not go as smoothly. Steven Wu, Amogh Jahagirdar, and Daniel Weeks cast binding votes on the proposal to add a _last_updated_time row lineage metadata column. Weeks attached a note. He wants the type changed from timestamptz to timestamp, since the value inherits from the snapshot timestamp and carries no real zone. Then Ryan Blue weighed in and said the proposal is close but needs more editing before it is ready. Expect another round on the spec text before this one lands.

Amogh Jahagirdar also changed how the community meets on v4. The old biweekly sync focused on the adaptive metadata tree. Separate calls covered column updates and data types. Now that most of those threads have shifted into execution mode, he replaced them with one weekly v4 metadata sync on Tuesdays from 9 to 10 a.m. Pacific. A single weekly call is a good sign. It means the spec work has enough momentum that a two-week feedback loop feels too slow.

Which file formats belong in v4?

Kevin Liu's thread asking whether v4 should stop writing ORC and Avro data files picked up a thoughtful reply from Zheng Hu. He agreed that feature parity across formats is a real maintenance cost. His suggestion was to give the file format API a set of compatibility flags. A table then rejects any feature its file format does not support. That approach keeps Iceberg open to many formats while letting the table spec move ahead of the slowest one.

This question is worth watching. Parquet already carries nearly all production Iceberg data. Writing new v4 features once, for one format, saves engine authors a lot of work. The flag approach offers a middle path that keeps the door open without slowing the spec.

Indexes, materialized views, and Puffin

Index support took a concrete step. Huaxin Gao posted a summary of the dedicated index sync. The group agreed to nest index routes under the table, at /v1/{prefix}/namespaces/{namespace}/tables/{table}/indexes, with index names unique per table and an optional endpoint to list indexes in a namespace. Péter Váry had argued for the nested layout earlier and said he preferred it because it matches how teams use indexes today. The group also agreed that derived columns, such as stored Hilbert values, belong in a separate effort. Váry pulled them out of the index spec and updated the spec PR, and he posted the meeting recording for anyone who missed it.

Jan Kaul announced that the materialized view spec is ready for broader review. The design defines a materialized view as an Iceberg view whose version points at a storage table. Readers that do not understand materialized views ignore the pointer and run the view query directly. Refresh state lives in the storage table's snapshot summary. It records the view version, the refresh start time, and the state of tracked source tables, so readers can decide whether the precomputed data is fresh enough. The working group built proof-of-concept implementations in Spark and Trino. Walaa Eldin Moustafa approved the PR and thanked the long list of people who shaped it.

Zheng Hu also nudged the thread on pluggable Puffin blob types and a compaction-aware index lifecycle. His request was simple. The design doc is long, and a short summary will help reviewers engage.

REST spec: signing, metrics, and FILE authorization

Two REST catalog threads touched the same theme: what a server promises when it answers a request.

Daniel Weeks bumped Sung Yun's proposal for a batch remote signing endpoint. He wants to wait for the FILE type implementation before settling the request shape. A main use of batch signing is file access, and files referenced by a FILE column can live outside the table's location in shared storage. In that case the server needs more context to decide whether to sign. Jean-Baptiste Onofré proposed an optional, spec-defined context on each batch element, carrying the data file location and the field ID. He argued for field IDs over column names, since names change on rename and get ambiguous in nested types. Sung Yun agreed and noted that this context is a claim from the client, not proof. The server uses it to inform its decision.

The metrics reporting endpoint thread reached a clear answer. Anand Kumar Sankaran asked what a catalog returns when it fails to store a metrics report. Yufei Gu wanted the spec to say "best effort" explicitly. Daniel Weeks disagreed. The endpoint already has defined behavior and return codes, and returning a success code after a failure is simply wrong. Whether a client retries is a client decision. Gu agreed to leave the spec unchanged, with the reading that the endpoint behaves like any other REST endpoint. This decision fed straight into a Polaris pull request, covered below.

Precision at the edges: nanosecond buckets

Roman Denysov found a gap while reviewing partition transform fixtures in the new iceberg-verification repository. Appendix B of the spec requires converting nanosecond timestamps to microseconds before bucket hashing, but it does not say how to round negative values. He laid out the example. The instant one nanosecond before the epoch lands in bucket 12 with floor division and bucket 76 with truncation. Iceberg Java, Go, and Rust all use floor division, and Rust recently changed to match Java.

Jean-Baptiste Onofré checked the Java code and recomputed the example. DateTimeUtil.nanosToMicros has used Math.floorDiv since nanosecond types arrived. He framed the change as a clarification, not a format change, and suggested placing the rounding rule directly in the existing note. Denysov opened a small spec PR. This is the kind of work that rarely makes headlines. It also keeps a pre-1970 row from landing in different partitions depending on which engine wrote it.

Releases: Rust 0.11.0 and the Terraform provider

Danny Jones announced Iceberg Rust 0.11.0 after the RC3 vote passed with 3 binding and 7 non-binding votes. The headline features are client-side encryption on the write path and incremental v3 support. That v3 work covers row lineage, nanosecond timestamps, delete vectors, and Variant in metadata, though Variant reads and writes are not there yet. The Rust implementation keeps closing the gap with Java, which matters for every engine built on DataFusion and for teams that want Iceberg without a JVM.

Matt Topol announced the Apache Iceberg Terraform Provider v0.2.0. The release adds an iceberg_tables data source, SigV4 request signing, OAuth2 improvements, and better schema evolution support. It ships through both the Terraform and OpenTofu registries. Infrastructure-as-code support for catalogs is a small thing that makes platform teams much happier.

Jackson 3 and the orphan files fix

Alex Dutra and Jean-Baptiste Onofré worked out a plan for migrating Iceberg to Jackson 3. The agreed order starts with a pure refactor. Iceberg introduces its own shim types over Jackson while still running on Jackson 2, then deprecates the Jackson-typed public APIs in 1.13. Only after that does Jackson 3 arrive, as an internal change. Thin adapter artifacts provide Jackson modules for REST serializers, so core does not depend on either version. A test-only switch runs the existing suites against both backends. The plan cleans up the public API whether or not anyone cares about Jackson 3.

The longest Iceberg thread of the week dealt with orphan file cleanup in cross-account AWS setups. Sebastian Daberdaku described a Glue catalog in one AWS account accessed from Spark in another. Every operation works except remove_orphan_files, which lists files with the cluster's own identity and fails. Russell Spitzer, Daniel Weeks, and Jean-Baptiste Onofré all pushed toward the same principle. Maintenance actions should use the table's FileIO, not a second Hadoop path. Weeks added that recursive listing does not scale with the object store layout. The right fix is to split the key space into prefixes and list them in parallel. Daberdaku reworked PR #17862 into a small change that uses the FileIO's Hadoop configuration when it has one, with docs pointing S3FileIO users to prefix_listing until parallel FileIO listing exists.

AI functions, Variant, and community health

Alex Stephen's proposal for AI functions over FILE and VECTOR types met careful pushback. Russell Spitzer asked what use case a shared function vocabulary serves when each engine has its own specifics. Szehon Ho explained that the Iceberg expression spec is small on purpose. It covers predicates and portable functions tied to data organization and skipping. Standardizing AI functions means agreeing on model identity, versioning, inference settings, and determinism. He prefers the existing function and UDF mechanisms unless a clear Iceberg-specific need appears.

Neelesh Salian refreshed the Variant tracking document with a "Next milestones" section and cancelled the December call for the holidays. The Variant sync resumes in January 2027. Hossein Torabi asked for a final review of the cache policy PR #14440.

Patrick McFadin joined the community health stats thread. He credited the CHAOSS project for the foundation of his analysis and said he is talking with ASF folks about offering it as infrastructure for every project. Jean-Baptiste Onofré offered to help through reporter.apache.org, with one condition. Sentiment analysis stays at the project level, never per person.

Apache Polaris

The road to 1.9.0

Polaris spent the week trying to ship 1.9.0, and the path had a few bumps. Jean-Baptiste Onofré hoped to include the Directories API in the release. Dmitri Bourlatchkov approved it, and Yufei Gu reviewed it twice. Pending comments remained, so Onofré moved PR #5927 to the 1.10.0 milestone and started the release.

He opened the 1.9.0 RC0 vote on November 10. Alex Dutra voted -1 within hours. The LICENSE file in the Spark bundle JARs was wrong in three ways. JSpecify was bundled but not listed, likely pulled in through the Iceberg 1.12 Spark runtime. Checker Framework annotations and JetBrains annotations were listed but not bundled. Onofré agreed, cancelled RC0, and did a full pass. The RC1 vote opened on November 11. Youngrae Kim posted the first +1 after running the verification script, checking all 468 signatures, and running catalog operations against an in-memory server.

Tracing that license issue led Dutra to a bigger question about the Spark plugin bundle. The plugin pulls in polaris-core with transitive dependencies turned off. The JAR ships all of core's classes and none of their runtime dependencies. Yufei Gu pointed out the Spark client uses only two classes from core. He suggested moving them into a small polaris-api-catalog-common module that both Spark and the server use. Dutra opened PR #6053 the next day.

Supply chain work: CVEs, VEX, and SBOMs

Two threads show Polaris investing in supply chain hygiene. Jean-Baptiste Onofré reviewed four open PRs that override transitive dependencies to clear CVE scanner findings. The list included Brotli4j, ZooKeeper, commons-configuration2, and Jetty. He checked each against the jars Polaris actually ships and how each library is used. None of the CVEs is exploitable in Polaris. His plan is to document that with VEX statements and close the override PRs instead of carrying version pins forever.

Robert Stupp followed with a proposal to publish SBOMs for Polaris release artifacts and switch license checks to Licensee. The first scope covers CycloneDX JSON sidecars for the binary distributions and each Spark bundle JAR, bound to the artifact's SHA-256 and signed like other release files. Stupp will target main after the 1.9.x branch is cut. Yufei Gu raised a fair caveat. Some dependencies inside the bundled Iceberg JAR do not appear in the SBOM, and consumers need to know that an unlisted component is not the same as an absent one.

NoSQL persistence grows up

Dmitri Bourlatchkov proposed removing the Beta label from NoSQL persistence and from the MongoDB implementation. He pointed to steady contributions from a diverse set of users. The response was unanimous, with +1 votes from Robert Stupp, Ayush Saxena, Russell Spitzer, Alex Dutra, Yong Zheng, Yufei Gu, Jean-Baptiste Onofré, and Ajantha Bhat. The new DynamoDB variants stay in beta for now. Bourlatchkov opened PR #6034.

The same day, Robert Stupp proposed Apache Cassandra as another NoSQL backend, with a draft implementation in PR #6024. Teams that already run Cassandra get to store catalog metadata on infrastructure they know. A catalog that runs on the database you already operate is a much easier sell to a platform team.

Authorization: view chains, row and column policies, and /v1/config

Polaris made real progress on fine-grained access control this week, and the threads connect.

Laurent Goujon proposed chain-aware authorization for views. The Iceberg REST spec has an optional referenced-by parameter on table, view, and credential loads. When an engine loads a table while evaluating a view, it passes the chain of views it walked through. This is the building block for definer views. Polaris accepts the parameter today but throws it away. Goujon's proposal parses, validates, and forwards the chain to authorizers. Each authorizer decides what to do with it. Prashant Singh noted the timing. He had just merged the final PR that makes the Iceberg Spark connector send referenced-by, after six months of related work with the Trino community. Jean-Baptiste Onofré backed part one and asked for a feature flag that operators can override per catalog.

Prashant Singh also resumed work on row-level and column-level security policies. The new design builds on the ReadRestrictions concept now in the Iceberg REST spec. Policy results become Iceberg expressions and actions that any engine can enforce. Goujon and Onofré both pointed out that RLS and referenced-by share the same open question. When does the server trust the engine? They want one definition of trust that covers both. Yufei Gu called it a highly requested feature.

On the endpoint side, Dmitri Bourlatchkov merged the soft-roll authorization change for GET /v1/config on November 7. The PR prepares stricter authorization behind a flag that stays off by default. Getting there took a frank exchange. Gu asked not to merge while review comments were open, and contributor Vignesh A resolved the grant issues. Vignesh then opened a thread to define which keys /v1/config sends to clients. Today the endpoint returns the full catalog properties bag. The proposal strips server-side polaris.config.* keys. Prithvi S pointed to new soft-delete settings as an example. Onofré added that a few legacy keys without that prefix need stripping too. Gu asked the group to fold in his earlier analysis of what Iceberg clients actually expect.

Remote signing comes back

Alex Dutra restarted the remote signing work with benchmarks and two designs. The first issues a signing token when a client loads a table, and the sign endpoint validates it. This works only with Iceberg 1.12 and later. The second authorizes each sign request, with a short-lived decision cache. He also proposed a new TABLE_REMOTE_SIGN privilege that grants nothing on its own. It must pair with read or write data privileges.

Tommy Bollhofer brought a user's view from an S3-compatible FlashBlade deployment where every client holds static storage keys today. Per-request authorization with a 30-second cache works fine for their short workloads. Yufei Gu came around to a simple TTL cache, on one condition. The maximum revocation delay becomes an explicit part of the security contract. He cited AWS Verified Permissions, which caches decisions for 120 seconds by default. Dutra agreed and plans to rerun his laptop benchmarks on Kubernetes in the cloud, where database calls cost more. He also described a compatibility mode for older clients. Sign requests without a token work for existing tables and fail with a clear error on create-table-as-select.

Credential vending everywhere

A cluster of threads extended credential vending past AWS defaults:

The one open disagreement sits in the typed storage config thread about skipping vending for Azure and GCS. Bourlatchkov sees vending as a property of the storage configuration. Gu wants it set at the catalog, namespace, or table level, since one storage configuration can serve many entities. Bourlatchkov asked for more voices before deciding.

Identity, transactions, and smaller fixes

Jean-Baptiste Onofré opened a thread on opaque access tokens with external OIDC. A Polaris user, a bank in France, has an identity provider that only issues opaque tokens. Polaris today assumes JWTs. Quarkus already validates opaque tokens through RFC 7662 introspection, so the change mostly stops Polaris from dropping that path. Bourlatchkov and Dutra approved.

Qiegang Long proposed conditional entity writes in the metastore SPI to fix a multi-table commit gap. Yufei Gu also asked to start with a design review for multi-table, multi-statement transactions before splitting work into small PRs. Youngrae Kim merged a fix so that invalid external catalog configs return 400 instead of 500. Vignesh A corrected his own recap of the metrics endpoint behavior after Gu pointed back to the Iceberg discussion. The answer is no 204 on a failed write.

On the housekeeping side, the team agreed to reformat all code with google-java-format v2 before branching 1.9, so cherry-picks stay clean. Alex Dutra simplified the CHANGELOG fragments layout to one file per change in changelog.d/. Onofré posted the November board report and agreed to clean up leftover Incubator dist directories that Rich Bowen spotted.

Apache Arrow

Who owns a Parquet type inside Arrow?

The most interesting Arrow thread of the week started as a routine format vote. Mandukhai Alimaa and Rok Mihevc called a vote to add arrow.parquet.file as a canonical extension type. The FILE column type is spreading across Parquet, Iceberg, Spark, and Delta Lake. Arrow needs a matching in-memory type so file references keep their meaning as they cross IPC streams, the C Data Interface, Flight, and language bindings. The proposal already had four committer reviews and closely tracks the Parquet logical type.

Weston Pace voted +0.5 and raised a governance question. Arrow already has arrow.parquet.variant, but Parquet-specific types fit the definition of community extension types. Parquet can define them without Arrow PMC approval, the same way GeoArrow does. He also disputed the claim that middlemen strip unknown extension metadata. IPC, the C Data Interface, and Flight should all preserve metadata they do not recognize. His proposal: the Parquet PMC publishes an extension-types.md in parquet-format, and Arrow links to it. Antoine Pitrou agreed.

Gang Wu took the other side. He has seen what happens when different bindings define their own extensions. Each one reads ARROW:schema differently, and interoperability falls apart. He hit this personally while implementing Iceberg v3 geospatial types in iceberg-cpp. Rok Mihevc noted that Parquet rejected an earlier extension-type proposal for duplicating logical types. He plans to propose that Parquet define how Arrow-compatible extension types serialize into Parquet metadata. Dewey Dunnington added that Rust and Go already handle geometry through GeoArrow types, and the C++ friction comes from process-global type registration. The vote is open, and the community has to decide whether to pause it.

This is a healthy argument. Arrow is the in-memory layer for nearly every open engine. If Arrow and Parquet disagree on who defines a type, every engine pays for it. Getting the ownership model right now, before VECTOR and GEOMETRY follow FILE, saves years of mapping bugs.

Releases and CI

Raúl Cumplido announced Arrow 26.0.1, a patch release with 20 resolved issues since 26.0.0. Jean-Baptiste Onofré froze Arrow Java to cut 20.0.0, with a 20.1.0 follow-up planned for the PRs still in flight. A user also asked about a pyarrow version 27 after a security scanner flagged 26.0.1 and named 27 as the fixed version.

Jeffrey Vo raised a CI deadline that affects Arrow, DataFusion, and many other ASF projects. GitHub plans to block the pull_request_target trigger by default on December 2, with brownouts on November 9 and November 16. A scan showed the trigger still in use across the arrow, arrow-java, arrow-site, and arrow-erlang repositories. Raúl Cumplido opened a tracking issue for the main repo. Vo also suggested the Boring Cyborg app for automatic PR labeling across Arrow repos, the same tool Airflow uses. Cumplido was happy to try it.

Apache Parquet

FINITE_VECTOR passes

Parquet now has a vector type. Rok Mihevc called the vote to add the FINITE_VECTOR logical type, and it passed with 7 binding and 11 non-binding +1 votes. The binding votes came from Ryan Blue, Gang Wu, Julien Le Dem, Daniel Weeks, Andrew Lamb, Fokko Driesprong, and Antoine Pitrou. Pitrou merged the format PR on November 10, and Andrew Lamb opened a PR to add the type to the website.

The type is narrow on purpose. It annotates the same structure Parquet uses for lists, so older readers fall back to reading a list. Every non-null vector has exactly the declared number of elements. Elements are required and finite, with no NaN, null, or infinity. No new physical layout ships with it, and nothing stops one later.

The path to passage taught a lesson. Antoine Pitrou first voted -1. The original text said writers must enforce the rules. Parquet does not require writers to validate STRING data as UTF-8, and he argued that one logical type should not carry different validation rules than the others. Mihevc updated the PR with feedback from Pitrou and Lamb, moved the vote to the new commit, and asked earlier voters to recheck. Pitrou then switched to +1.

Julien Le Dem made the meta point in a related thread on vector physical representation. Consensus on new types stalls easily. Narrowing a proposal to one clear, present use case is how a project keeps moving. FINITE_VECTOR is that narrowing. Embedding vectors for AI retrieval are finite and fixed-length, and now Parquet can say so in its metadata.

FILE type ambiguity

The FILE logical type, already merged and released, hit its first real-world question. Stefan Grafberger found that a value with both inline and uri set can resolve to different bytes in different engines. If someone overwrites the object at the URI, one reader returns the inline bytes and another returns the new object. Both stay spec compliant. He proposed that inline bytes win and locator fields act as provenance.

Divjot Arora asked whether changing read semantics on a released type is a compatibility break. Julien Le Dem said the project has to pick one correct behavior, even if that marks some readers as buggy. Gunnar Morling and Burak Yavuz argued the spec left resolution open on purpose. Yavuz sees inline as a cache for small values. Daniel Weeks gave the cleanest framing. The intent is that inline bytes always match the referenced bytes. If someone changes the file, the table sits in an error state. Readers should not have to validate, since that defeats the point of inline storage. A checksum exists for readers that want to compare. Morling also pinged reviewers on the FILE fixtures PR.

Parquet Java 1.18.2 ships after three candidates

Manu Zhang drove the Parquet Java 1.18.2 release, and it took three tries. Gang Wu caught two problems in RC0. The signing key was missing from the KEYS file. The source zip carried an entire Thrift 0.25.0 build tree, about 222 MB with compiled objects and an 80 MB executable. Zhang fixed the release script and cleaned the branch. Russell Spitzer caught a stale SCM tag in the POM for RC1 and fixed the automation. RC2 passed with binding votes from Gábor Szádovszky, Fokko Driesprong, and Gang Wu.

Then publishing failed. The release credential worked only for the dev SVN area, so Wu filed an ASF Infra ticket. Zhang announced 1.18.2 on November 11. The release fixes column-path handling in Bloom filters and dictionary lookups, improves writer failure handling and row-group buffer cleanup, fixes Protobuf bytes handling, and updates Jackson and Guava. Wu thanked Spitzer for the new release workflows, which made the process far easier than before.

On the same day, Julien Le Dem announced Russell Spitzer as a new Parquet committer. Spitzer is a long-time Iceberg maintainer. His Parquet commit bit strengthens the bridge between the table format and the file format underneath it.

Footer, layout, and encodings

Parquet's longer-term redesign kept moving. Julien Le Dem urged everyone to review the Parquet Modular Footer from Jiayi, calling it transformational and the last footer change for a while. Micah Kornfield liked the design and asked for better documentation of the footer proto.

Divjot Arora then shared a proposal for a Page-First Data Layout built on the modular footer. Today's row group structure works when columns have similar sizes. It breaks down when one column is huge and another is tiny. Arora's layout lets writers keep buffering small columns while cutting pages for large ones. Anyone who has stored embeddings next to small dimension columns knows this problem.

Encodings got attention too. Prateek Gaur posted his FastLanes benchmark breakdown with recommendations for frame-of-reference and delta variants. Andrew Lamb suggested naming the encoding FASTLANES, since the term already has mindshare, and including the Unified Transpose Layout for its extra speed. Lamb also announced that the Java implementation of ALP encoding for floating point data merged, thanking Gang Wu, Vinoo Ganesh, and Prateek Gaur.

Statistics, decimals, and versioning

Adrian Garcia Badaracco and Daniel Weeks debated whether distinct_count should be classified as inexact. Badaracco argued the field was never defined as exact. Weeks, who was involved when it arrived more than 13 years ago, said nobody understood it as an estimate. Redefining it is not a compatible change. They settled on a clean path. Badaracco will propose a new field for estimates.

The DECFLOAT proposal for decimal floating point made progress on semantics, with agreement on IEEE 754-2008 special values and precision from 1 to 43 digits. Thomas Kissinger restated strong objections to the BID encoding. Few languages support it, and no library he knows handles the 160-bit form needed for 34 or more digits. Julien Le Dem added a change taxonomy to the versioning doc after presenting it at the November 4 community sync. That sync drew attendees from Snowflake, Databricks, AWS, InfluxData, Spiral, and more.

Apache DataFusion

A rewrite of the SQL parser, and the LLM question

Andy Grove dropped a surprise on the list. Over a weekend he rebuilt sqlparser-rs from a clean slate using the design he wanted from the start. Each dialect gets one trait, and dialect-specific parsing lives in that trait. The single core parser full of dialect conditionals goes away. Related dialects share logic, so Hive and Databricks call into the Spark dialect. Grove reports roughly 10 times fewer memory allocations and roughly 10 times faster parsing.

Adrian Garcia Badaracco asked the obvious question. Is this mostly LLM-generated code? Grove said yes. The high-level design is his, from years of wishing he knew how to build it in Rust. He argued parser refactoring suits LLMs well, given the project's large test suite. Badaracco framed the community decision: will DataFusion accept a large block of LLM-generated code with the right guardrails, passing tests, and better benchmarks? Andrew Lamb reframed it again. The bigger question is who wants to maintain that codebase. If people commit to that, the origin of the code matters much less.

That framing is the right one. Every open source project now faces this question, and maintenance ownership is a better test than authorship.

Releases, agents, and CI

The regular release train kept running. Andrew Lamb announced the 56.0.0 release cycle, with Xudong cutting branch-56 ahead of a release candidate. The sqlparser-rs 0.64.0 vote passed with four binding votes, and Lamb published it to crates.io. Xuanwo's verification covered 1,917 passing tests.

Matt Butrovich shared AGENTS.md and skill updates across arrow-rs, iceberg-rust, and related projects. The goal is better code and reviews from coding agents, alongside the PR limits and CI gates that already manage volume. Jeffrey Vo posted the same pull_request_target call to action he raised on the Arrow list, noting that October's CI usage issues were painful enough to justify getting ahead of this one.

Andy Grove also opened an issue to capture the weekly sync discussion about the future of the datafusion-spark crate. A newcomer asked whether Ballista can add distributed processing to DuckDB. Andrew Lamb and Marko Milenković said no such plans exist. Milenković pointed out that Ballista and DataFusion both support Substrait, which allows offloading plans in theory, and asked why the user does not run DataFusion and Ballista directly.

Apache Ossie (incubating)

Preparing 0.3.0-incubating

Ossie, the Open Semantic Interchange project, is getting ready for its first release. Jean-Baptiste Onofré opened three PRs to stage release resources, update build tooling, and align converter dependencies. He later added another PR to the release scope. The stability thread settled the version at 0.3.0-incubating. Kyoung Min Kim showed that a schema constant is not the only version check. The Python SDK and several converters also hard-code version strings, so a patch release needs coordinated changes. Yufei Gu agreed and asked for documented compatibility rules.

Custom extensions become nonportable_extensions

The busiest decision thread reshaped how vendors store data Ossie cannot express yet. Mike Carlo proposed that custom extension data be a structured object instead of an opaque JSON string. The thread filled with measured evidence. The GASP SaaS metrics export writes 571 extension entries into one model, and payloads make up 42 percent of the file. Damian Waldron reported that 9 of the ThoughtSpot converter's 35 payload keys duplicate things the Ossie document already states. William Kelley gave a +1 from Hex. Aleksandr Kosachev said his ClickHouse executor will accept both forms.

Onofré drew the line on purpose. Ossie's goal is a portable core with agreed semantics, not a container for vendor behavior. Extensions exist to preserve vendor state for round trips while the core grows. Carlo's updated PR #522 renames the field to nonportable_extensions and drops the old name entirely. The name itself now tells readers the content is not portable.

A companion thread on vendor-specific validation landed a design. Each vendor's checker reads only its own payload and dialects. A vendor finding never changes whether a document is valid Ossie. Kyoung Min Kim defined the interface in PR #697 as a Python entry point group, and Waldron already wrote the ThoughtSpot checker on a branch.

Actions: giving agents verbs

The thread with the widest reach proposed adding Action as a concept in the Ossie ontology. Purvaja Narayanaswamy of Snowflake described the gap. The ontology names things, and metrics measure them, but nothing describes a business operation. An Action names the operation, the entity it changes, its inputs, criteria, effects, and the callable that runs it. She cited Schema.org's Action class as the open precedent. Piotr Sowiński shared experimental action support in LinkML.

Yufei Gu worried about scope. Once Actions arrive, do workflows, policies, and approvals follow? Onofré proposed a firm boundary that Narayanaswamy accepted. Ossie describes an operation. It never executes, authorizes, or orchestrates one. Ankit Tandon of RelationalAI put it well. Metrics give agents adjectives, the ontology gives them nouns, and actions are the missing verbs. He proposed an Actions working group. Julia Blyumen shared her own draft from a different angle.

Then Bei Li of Google broadened the discussion with a doc built with RelationalAI and shared earlier with Snowflake. Onofré responded with a reminder that matters for any ASF project. Design discussion belongs on the list, in the open, and working groups discuss but do not decide. Matthew Mullins of Coginiti asked where Actions sit in Ossie's layers, since the proposal moves from write-back over the data plane to REST and gRPC endpoints.

Semantics, tests, and new committers

The deepest technical debate asked how semantic engines interpret relationship definitions. Julian Hyde pressed on whether relationships are suggested joins or unenforced foreign keys. Chris Eubank explained the split between Layer 2 relational semantics and the more limited, declarative Layer 3. Implementers shared real data on fan-out and duplicate handling.

Testing infrastructure moved forward. The PPMC voted to approve DuckDB as a dependency for Hyde's query conformance test harness in PR #667, with Onofré confirming the MIT license and no bundling. Justin Talbot proposed the first use of the reserved AGG() function, letting model-level metrics evaluate dataset-level metrics. He was also welcomed as a new committer.

Smaller threads filled in the picture. John DaCosta opened PR #704 to define what Dataset.source guarantees, with ordered exact identifier parts and optional levels. William Kelley proposed case-insensitive vendor and dialect names with uppercase as canonical. Mahmoud Hassanen planned ClickHouse dialect support in the converters. Aleksandr Kosachev started a two-step plan for time zones. Yufei Gu proposed an opt-in trial of Orca, an AI review bot already tested in Polaris. Onofré supported it with conditions: comments only, no approvals, and clear AI labels. Julian Hyde created an @ApacheOssie account on X for the PPMC to manage, and the community wants a LinkedIn page too.

Cross-Project Themes

FILE and VECTOR are becoming lakehouse-wide types

Read the Parquet, Arrow, and Iceberg lists side by side and one story stands out. The open stack is adding first-class types for AI data, and every layer has to agree on them. Parquet passed FINITE_VECTOR and started fixing FILE semantics. Arrow voted on arrow.parquet.file so file references survive in memory. Iceberg contributors debated AI functions over FILE and VECTOR columns, and they paused batch remote signing until FILE authorization is clear.

The hard question is not the bytes. It is ownership. Weston Pace and Antoine Pitrou want each format to own its extension types. Gang Wu and Rok Mihevc want shared canonical definitions so bindings stop drifting. Daniel Weeks and Julien Le Dem are pulling the FILE type toward one correct reading. Whatever the answer, teams storing embeddings and unstructured file references in a lakehouse will soon get the same typed guarantees they already have for timestamps and decimals. That is the open lakehouse becoming a real home for AI data instead of a sidecar to it.

Trust is the shared hard problem

Polaris spent much of the week on one question: when does a catalog trust what an engine tells it? The referenced-by view chain is a claim from the client. Row and colu...