Trino Distributed SQL Engine · Iceberg Row-Level NDV & Optimizer
Core Java contributions to the Trino distributed SQL query engine (formerly PrestoSQL). Engineered statistics retention logic in IcebergMetadata.finishWrite so Theta sketch NDV (Number of Distinct Values) statistics survive row-level DELETE, UPDATE, and MERGE transactions without costly full-table re-ANALYZE runs, and contributed regression coverage for CREATE OR REPLACE MATERIALIZED VIEW.
Distributed Lakehouse Query Optimization
[Trino](https://github.com/trinodb/trino) is a high-performance distributed SQL query engine designed for fast analytic queries over petabyte-scale data lakes. When querying Apache Iceberg v2/v3 tables, Trino's Cost-Based Optimizer (CBO) relies on Puffin Theta sketch statistics (specifically Number of Distinct Values, `NDV`) to select optimal join distributions (broadcast vs partitioned) and join ordering.
Iceberg Transactional Statistics Architecture
The contribution modified `IcebergMetadata.finishWrite` within the transactional `commitUpdate` and `commitWriteTransaction` lifecycle to retain valid prior snapshot statistics across row-level modifications:
| Lakehouse Operation | Legacy Trino Iceberg Connector | Patched Trino Connector (#31546 / #31130) | Optimizer Impact |
|---|---|---|---|
| Row-Level `DELETE` / `UPDATE` | Drops existing Puffin NDV sketches | Retains latest valid snapshot NDV blobs | Preserves Cost-Based Optimizer join ordering |
| Position Delete Deltas | Forces fallback to heuristic join estimates | Keeps cardinality bounds monotonic | Avoids OOM broadcast join regressions |
| Materialized View Replace (#31130) | Unverified metadata state on replace | Full transactional regression suite (Merged) | Guaranteed view freshness invariants |
| Re-Analyze Overhead | Mandatory full table scan after DML | Zero post-DML scan required | 100% compute savings on DML workloads |
*Table 1: Trino Apache Iceberg Statistics & Materialized View Enhancements*
// Trino IcebergMetadata.java: Preserving Puffin NDV statistics on row-level writes
if ( writtenStatistics.isEmpty() && !table.statisticsFiles().isEmpty() ) {
// Retain existing NDV Puffin statistics across row-level position delete writes
StatisticsFile latestStats = table.statisticsFiles().getLast();
commitUpdate.updateStatistics().setStatistics(latestStats.snapshotId(), latestStats);
}Upstream Verification
- PR #31130 (Merged): Added comprehensive regression verification for `CREATE OR REPLACE MATERIALIZED VIEW` across Iceberg catalogs. - PR #31546 & PR #31544 Review: Validated `TestIcebergStatistics` across Iceberg format versions 2 and 3, ensuring compatibility with empty-snapshot predicate pushdown rules.