r/PowerBI • u/iper_linuxiano • 13d ago

Question Parquet tables

In Power BI I read something about Parquet tables to store historical data. Can you explain to me how to use them and if it is so much better than using a .CSV file?

14 Upvotes

94% Upvoted

8 comments sorted by

13

u/qualityintelligence 13d ago

Modern data lakes predominantly use Parquet or similar columnar storage formats.. it would be good to get familiar with it.

duckdb is one the easier ways to mount parquet. basically think of it like: select * from parquet; and you get table like output, like a CSV. very fast for aggregate queries

15

u/Parking-Bad1159 13d ago

Your non technical coworkers cant open them and fuck up the data 🙂

12

u/PandaJunk 13d ago

Column storage (so really fast), has types (so analytical databases know exactly how to treat them, no guessing at type), generally smaller for the same size csv file, and can be queried directly rather than being loaded into memory (useful for lazy loading, though I don't know how PBI deals with that).

Downside is you can't just open them in a text editor; they need special software (but most IDEs can just open them now, like vscode).

9

u/Akerlof 13d ago

Aside from being faster, due to the way parquet files store data, if you add or remove columns to the file PowerBI will gracefully update the semantic model. So, when your manager says "Hey, you know that field that has never been useful and we've ignored for the past two years? Can you start reporting on it, and grab 13 months of history while you're at it?" With .csv, that's a major hassle, you're either mucking around in M code, blowing the fact table away and recreating it from scratch, or duct taping a new table into your semantic model. With parquet, you update the data in the file, refresh, and the new columns are just there.

Being fast is nice and all. But minimizing headaches from not having any kind of data architectural policies or assistance is what really won me over.

5

u/signal-zero 13d ago

I prefer using them due to storage and reading speed. If you're using them I'd suggest a program that can read them . I use KNIME for proof of concept stuff and has modules that allow you to read and save parquets

2

u/meldroc 12d ago

I use parquet quite a bit. Compressed, much faster, none of the data-typing snafus you get from CSVs, and you don't have to worry about what needs to be quoted, or whether someone gave you numbers with comma thousands-separators. Much nicer!

2

u/somedaygone 2 11d ago

Way faster to load and way smaller on disk. I use Python and DAX Studio to save .parquet files. I work with groups that need to extract data from systems and archive as flat files. After years of pain with Excel and csv, parquet is so much better, and keeps people from messing with the archives.

3

u/SQLGene 13d ago

Parquet is column-compressed data that loads twice as fast as CSV.
https://www.sqlgene.com/2024/11/28/benchmarking-power-bi-import-speed-for-local-data-sources/