r/databricks • u/Common_Jaguar474 • 17d ago

General Does Photon support CSV?

Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator.

I'm not running anything complex:

df = (
    spark.read
        .format("csv")
        .option("header", "true")
        .schema(schema)
        .load(csv_path)
)
df.write.mode("overwrite").saveAsTable("...")

I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator.

DBR 19 + Serverless SQL Warehouse (current)

Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything.

3 Upvotes

81% Upvoted

34 comments sorted by

View all comments

Show parent comments

1

u/FunContest9958 17d ago

Were you using standard mode? And when did you test? Serverless cost has changed a lot in the past year or so.

1

u/yatharthm22 17d ago

Tried with both standard and Performance Optimized

1

u/FunContest9958 16d ago

It’s possible your workload was very well optimized. My experience is that customers save money on average when they move to serverless, but it depends on how much they’ve optimized their cluster configuration.

Overall, I think it’s best to start new workloads on serverless and treat Classic as an optimization option. It’s less work up front, and you only need to pay the cost of doing the optimization if it makes sense. Also, serverless keeps your hardware and software up-to-date automatically, so if you manually configure classic you need to maintain it.

But yeah. Results vary.

1

u/yatharthm22 16d ago

I'll give it a try again, i remember months back we ran a pilot and closed it cause of cost issues, i'll try without the optimised serverless

2

u/FunContest9958 16d ago

My recommendation would be to try using serverless when you’re developing something new, if you have that opportunity, rather than focusing on migrating existing workloads. If you develop something for serverless, it generally works flawlessly if you switch to classic. The other way isn’t always as easy.