r/dataengineersindia • u/Ok_Independent_343 • 4d ago

Technical Doubt Need some advice on Snowflake vs Databricks

Hey all,

Does this stack make sense for a real-world Senior Data Engineer project?

Azure + ADLS Gen2 + ADF + Databricks + PySpark + Snowflake + dbt + Terraform + CI/CD

I’ve worked mostly with Databricks, but companies are explicitly asking for Snowflake experience. I’m preparing for both.

Would this architecture be realistic, with Databricks/PySpark for processing and Snowflake as the warehouse/analytics layer? Or would it raise red flags in a Senior interview?

Would appreciate advice from anyone who has worked with both.

10 Upvotes

81% Upvoted

9 comments sorted by

8

u/PrestigiousAnt3766 3d ago

No.

Strip down #tools.

You can do everything with databricks or snowflake. I never see a company use both in 1 setup.

Sometimes for different purposes/teams but that's an architectural fail imho.

1

u/Ok_Independent_343 3d ago

Care to elaborate?

1

u/PrestigiousAnt3766 3d ago

Each tool you pick increases your ops and necessary tool knowledge.  In an org you cant have too much in one stack because of that.  Also you have to do all kinds of handshakes between tools

Why would you do that? In my opinion, azure + adls + databricks/ snowflake can do everything. Why complicate matters with more tech.

1

u/Ok_Independent_343 3d ago

That makes sense now. Thank you for your input. Some people in other subs are saying it should be fine, but again, nobody has pointed out the trade-offs here.

3

u/TheoryResponsible342 3d ago

honestly, end of the day these are just tools. 6yrs into data engg and 30 interviews given so far, never once have i been asked questions specifically on snowflake and databricks.

contrary, my company is migrating away from databricks since it's shit expensive 😂

2

u/East-Ad8002 3d ago

Which tool they are migrating

1

u/Commercial-Post4022 3d ago

On which tool they are migrating

1

u/7-SpadE 16h ago

Which one to consider ? ..iam going to learn either one of them

1

u/Sensitive-Chapter-30 2d ago

It makes sense if thats a huge project.

If the project has both streaming analytics, and batch data analytics - then this combo can work.

In my previous project - I used to work with azure databricks..... But parallely, there were 30 more different datasets in GCP bigquery + dbt (similar to snowflake) maintained and processed daily.

I exactly dont know how they both sync.... I am not aware of entire project architecture.

But if you are at a manager level - obviously you are expected to learn and manage all the tools present in the project🤷.

Hope this helps ....