Rigor
Delta rules come from each operator's algebra (Z-sets, as in DBSP), not pattern matching: a view is maintained exactly, or classified up front for full refresh.
Z-sets · delta rulesIncremental view maintenance, democratized
Stop recomputing.
Refresh only what changed.
OpenIVM brings incremental materialized views to DuckDB and Spark. It compiles your views into incremental SQL, so inserts, updates and deletes propagate as deltas and a refresh costs what changed, not what exists.
Built at CWI Amsterdam · SIGMOD 2024 paper
ETL costs scale with your data. They should scale with your changes.
LLMs and tools like dbt generate more data pipelines than ever, and most of them recompute from scratch on every run, while Spark, the ETL engine behind most lakehouses, has no open incremental alternative.
OpenIVM is an open-source incremental view maintenance (IVM) compiler that makes those pipelines refresh only what changed, inside the engines you already run.
What an IVM engine needs, done properly.
State
View and auxiliary state live in ordinary tables, so it works on batch tables, not only streams, and survives restarts.
aux state · delta tablesDeletes
Every change is a signed tuple (+1 or −1), so inserts, updates and deletes share one algebra instead of special cases.
+1 / −1 multiplicitiesHow it works
The Z-set model, the architecture, and the exact SQL OpenIVM generates for a refresh.
Read the internals →Quickstart
Build the DuckDB extension, create a materialized view, and watch a refresh touch only the change.
Get started →Compare
How OpenIVM differs from pg_ivm, streaming databases, and warehouse materialized views.
See the comparison →One compiler, many engines.
DuckDB
available · source buildThe reference implementation: incremental materialized views in DuckDB, with automatic refresh, cascading pipelines, an adaptive cost model and DuckLake integration. Community extension planned for end of 2026.
github.com/ila/openivm →Apache Spark
in developmentIncremental refresh for Spark ETL and dbt models, including Spark-dialect view bodies such as VERSION AS OF.
Your engine
open standardOpenIVM emits SQL, not engine internals. If your engine speaks SQL and can store a table, it can host an incremental view.
Propose an integration →Open, reproducible numbers.
A live benchmark fed by ivm-bench: incremental refresh versus full recompute across engines, workloads and delta sizes. Results are published as a static Parquet file and rendered right in your browser.
OpenIVM: a SQL-to-SQL Compiler for Incremental Computations
SIGMOD Companion 2024 · doi:10.1145/3626246.3654743