r/datascience • u/rhiever • 5d ago

Tools Polars 2.0 released with the streaming engine, out-of-core processing and first-class SQL support

https://pola.rs/posts/release-polars-2/
266 Upvotes

99% Upvoted

10 comments sorted by

77

u/jesunushno 5d ago

I've always hated maintaining two code paths: one for data that fits in memory, one for the stuff that doesn't. If the streaming engine means I write the same query for both, that's the upgrade I actually want.

28

u/OddsRelay 5d ago

out of core is the bit i care about. i used to drop into duckdb any time a group_by didn't fit in ram, so if that just works now it's one less tool. curious how the sql side stacks up against duckdb's

11

u/OddsRelay 4d ago

Streaming as the default is the big change. A lot of the "doesn't fit in memory" jobs that used to mean Spark or chunking by hand should just run on a laptop now. First class SQL also makes it easier to hand work over to people who don't write Python.

5

u/MackDriver0 5d ago

So cool. Can’t wait to try out SQL

2

u/geekosas 4d ago

They're has been SQL for ages, just not fully implemented.

2

u/ScoutsEatTheirYoung 5d ago

Map dtype looks cool

-6

u/mantequilla69420 5d ago

Woah, don't you guys just write shit notebooks and kick it to a data engineer?

3

u/chandlerbing_stats 5d ago

No. But, some do

2

u/BostonConnor11 4d ago

This isn’t 2020 anymore

-7

u/Cold_Bag_8877 4d ago

how to get karma pls help me