TL;DR
Get monitors, keyboards and dev gear delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
Polars 2.0 changes lazy query execution to use its streaming engine by default and enables initial out-of-core processing that can spill supported operations to disk. The project also highlights SQL and optimizer improvements, a new Map dtype, and benchmark results it says favor Polars in most tests; those results come from the Polars team’s own setup.
Polars has released version 2.0, changing how lazy queries run by default: calling collect on a LazyFrame now uses the streaming engine. The release also enables initial spill-to-disk support, expands SQL capabilities and adds a Map data type, with the changes aimed at improving memory handling and widening the workloads the data-processing library can serve.
The default engine change has a behavior trade-off. Polars says its streaming engine does not preserve observable row order by default for some operations, including joins, group-bys and unpivots. Users who need that ordering can request it with maintain_order=True. The release describes this as a reason for the major version bump.
Polars 2.0 also enables out-of-core processing by default. The release says supported operations, including sorts, window functions and many expressions, can spill data to disk when memory use reaches about 80% of RAM. The default disk budget is 64GB. Joins and group-bys are not yet listed as supported spill-to-disk operations; the team says they are planned for later.
Other changes include expanded SQL support, optimizer and engine work, and a new Map dtype that represents Arrow MapType directly rather than reading it as a list of key-value structs. The team cites join reordering, common-subplan elimination and dynamic predicates or bloom filters among the performance improvements. It reports that Polars was fastest in all but one of its benchmarks, while noting a scaling overhead on a 192-thread machine for smaller queries.
More Queries Can Run Within Memory Limits
The default streaming engine and disk spilling address a practical constraint in data analysis: a query can fail or consume excessive resources when its working set exceeds available memory. In Polars 2.0, supported operations may use disk to finish rather than relying solely on RAM. That may make the library more useful for users working with larger-than-memory data, although the release does not claim every operation can spill.
Making SQL a first-class interface also matters to teams whose pipelines and analysts already use SQL. The project says its SQL coverage has grown and points to its benchmark results as evidence of performance. Those measurements are useful as a report from the developers, not an independent finding: outcomes depend on the tested queries, hardware, configuration and scoring method. The ordering change is another practical consideration for existing code, since users who relied on incidental row order may need to request it explicitly.
Polars 2.0 data processing library
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What Changed From Earlier Polars
The release frames 2.0 as a major version focused on execution defaults and stricter behavior, rather than solely as a feature-count milestone. The clearest compatibility change is that lazy collection now uses streaming by default, with order preservation opt-in for the listed operations. The new Map dtype also changes how Arrow map data is represented in Polars.
To compare SQL performance, the Polars team tested against DuckDB 1.5.6, DuckDB 2.0 alpha and DataFusion 54.0.0 using TPC-H and TPC-DS-derived Parquet data. Tests ran on two AWS machine types, with 16 and 192 virtual CPUs respectively. Each query was run five times in a hot setting, and the fastest run was used; query times were compared by sum and geometric mean. The team says Polars and both DuckDB versions completed all queries, while DataFusion timed out on TPC-DS query 72 (and once on query 67) and ran out of memory on TPC-H query 18 on the smaller machine. Those affected queries were excluded for all engines. Polars has published a repository for replicating the benchmark.
“Calling collect on a LazyFrame will now default to the streaming engine.”
— Polars, in its version 2.0 release report
As an affiliate, we earn on qualifying purchases.
Limits and Compatibility Questions
The release report does not give an independent replication of its benchmark claims, and its results are tied to the hardware, data generation, query selection and scoring rules it describes. Readers should not treat the reported ranking as a guarantee for other workloads. Polars also acknowledges that its 192-thread overhead can hurt small queries and says it hopes to address the issue in the next release; the report does not provide a firm delivery date.
Out-of-core support is partial. The report names sorts, window functions and many expressions as supported, but does not give a complete operation-by-operation compatibility list in the supplied material. It says joins and group-bys are planned, without a schedule. It is also not clear from the report how the approximate 80% RAM spill threshold or default 64GB disk budget should be adjusted for every environment. Users should check the release documentation for behavior relevant to their workloads.
As an affiliate, we earn on qualifying purchases.
Further Spill Support and Tuning
Polars says it plans to extend out-of-core support to joins and group-bys, which would cover two major operation types not yet included in the release’s stated spill support. The team also says it has diagnosed the scaling overhead seen on the 192-thread machine and hopes to fix it in the next release, though it has not supplied a date.
For now, users evaluating Polars 2.0 can test their own queries, confirm whether row order matters, and review which operations can spill under the new defaults. The project has made its benchmark repository available for replication; independent testing and further release notes may clarify how the reported results translate across machines and production workloads.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is the main change in Polars 2.0?
LazyFrame collection now defaults to the streaming engine. The release also enables initial spill-to-disk support and adds SQL and data-type improvements.
Does the streaming engine preserve row order?
Not by default for some operations, including joins, group-bys and unpivots, according to Polars. Users who need observable order can set maintain_order=True.
Which operations can spill to disk?
The release report lists sorts, window functions and many expressions. It says joins and group-bys are planned for future support, but provides no release schedule for them.
Are the Polars 2.0 benchmark results independent?
No independent testing is described in the supplied report. Polars ran the TPC-H and TPC-DS-derived comparisons itself and published its setup and a repository intended to help others replicate the results.
Source: hn
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
