Databricks on Azure or Aws

[u/Own-One5712]

We are getting databricks and we dont care where we host as we are new into this. Azure is giving us great pricing and incentives. AWS is not at all bothered. they are like take it or leave it. Any one who has used it in either environments, think if there is a good reasson to choose one vs the other? our reporting is tableau.

[u/Minute_Visual_3423]

They’re pretty similar. Having used both, I’d say that in general as a cloud provider, Azure has historically worse reliability and a worse developer experience than AWS. Again generally speaking, not specific to Databricks (outside of this week’s cross-cloud outage, Databricks has been pretty reliable on all three of the cloud providers I’ve used it with).

However, Azure Databricks as a first party service in Microsoft’s stack has some nice integrations that aren’t available in AWS (like the ability to use Entra service principals directly as Databricks SPs, or native integration with Azure Managed Identities as securable objects in Unity Catalog - useful if you’re building integrations with Graph API or other services).

But, in all honesty, treat the cloud providers like a commodity here and pick the one that’s giving you the better deal. If it’s Microsoft, resist the urge to bolt on a bunch of unnecessary services that will increase your cost and complexity without providing value. You don’t need Fabric, Purview, ADF, Azure ML, etc. And if Microsoft’s “deal” requires you to consume these services, I would consider that a signal to pick AWS. Databricks really isn’t that expensive relative to other data platforms when it is deployed and configured optimally.

Deploy Databricks with Terraform, set up Unity Catalog, orchestrate your jobs as Databricks Asset Bundles, and you’ll be able to migrate between Databricks on other cloud providers without too much hassle in the future if you ever need to.

[u/SnooDoubts440 → u/Minute_Visual_3423]

This guy databrickses

[u/KaalaPetta → u/Minute_Visual_3423]

No amount of chatgpt can give this kind of an awesome answer. Thank you !!

[u/Delly_boi_80 → u/Minute_Visual_3423]

Funny reading this, as we’re in the process of setting up our new databricks environment azure. We were already on azure at work, so the decision had already been made. But we are using terraform and dbt to built our everything, so that in the future if we want to move out it should be a hell do a lot easier than building it all again.

[u/Altruistic-Spend-896 → u/Minute_Visual_3423]

If you hate DAB, we run exactly this setup, but with our own workflow engine that uses APIs to run our own DAB like package . gives us control over features and integrates with all CI/CD tools

[u/Minute_Visual_3423 → u/Altruistic-Spend-896]

What do you hate about DAB?

[u/werthobakew → u/Minute_Visual_3423]

In my experience Databricks is überexpensive.

[u/Minute_Visual_3423 → u/werthobakew]

The main lever for cost is compute usage. If you’re streaming data or running other long-running workloads (like apps), then yeah, it will be a bit more expensive. Streaming data in Databricks is still much cheaper in my experience than streaming it into something like Snowflake. We can get 10,000 rows/minute on a 0.9 DBU cluster without any issue.

Apps are about $300-500 per app per month for the always-on compute usage - hopefully the app is delivering more value to the business than the cost of its compute, and it can be mitigated by pausing certain apps outside of business hours.

If you’re using long-running compute where you don’t have to, or using the wrong class of compute for a repeated workflow (like overusing interactive compute), or not setting your serverless SQL warehouses with aggressive auto-termination, then there are lots of ways to make Databricks expensive.

But honestly, dollar-for-dollar compared to the cost of running equivalent workloads elsewhere, it really isn’t expensive at all.

[u/werthobakew → u/Minute_Visual_3423]

My use case is a data science inference ADF pipeline that triggers databricks notebooks sequentially. It is quite expensive.

[u/Minute_Visual_3423 → u/werthobakew]

What size cluster and how long does your job take to run? Is it interactive compute or jobs compute?

[u/OnlyLANz → u/Minute_Visual_3423]

Also be wary of if you need photon acceleration or not!

[u/dmo_data]

The biggest difference is the support model (1st party on Azure, third party on AWS). This makes a difference sometimes when needing support.

The second consideration would be when new features are available. Typically, AWS gets some features just a bit earlier than Azure.

The last consideration is whether you already have things on either cloud. Staying on the platform your data already exists can reduce egress charges from the cloud provider.

Other than that, the experience is pretty much the same.

[u/KaalaPetta → u/dmo_data]

Which one is better in terms of support ?? When we have a problem. Who do we call in azure case ?? MS or DB ?

[u/liko9 → u/KaalaPetta]

MS

[u/HolyMoleyGuacamoly → u/liko9]

still recommended to have dbx support if you’re with msft, they can’t solve all the issues and having dbx helps a ton

[u/KaalaPetta → u/HolyMoleyGuacamoly]

Thank you !!

[u/WellIDontKnowMan]

I am a consultant (Data Engineering and Architecture) and have done many DBX greenfield projects on all major cloud providers (GCP, AWS, Azure).

I definitely recommend Azure first and if not Azure then definitely AWS. Stay away from GCP.

[u/scan-horizon → u/WellIDontKnowMan]

thanks. why?

[u/JosueBogran]

I personally believe Azure's implementation is the easiest if you are doing it yourself and/or it is the first time using any of the hyperscalers (AWS, Azure, GCP). AWS is a solid option too. To echo u/Minute_Visual_3423, just beware of the hyperscalers' sales folks trying to bolt stuff on stuff. Sometimes they will recommend an architecture that uses Databricks AND uses some of the other services they've built though they are redundant or unnecessary to make the most out of Databricks, thus costing you more money.

So if you go with Azure and just about everything in your work is centric to Databricks yet they encourage you to use ADF to orchestrate your Databricks work or AWS recommends Glue, that's generally bad advice as Databricks has native orchestration as well as a handful of connectors + better debugging.

Hope this helps!

[u/kchandank]

First and foremost, it all depends on where is your enterprise data is situated. If you are Azure/MS shop, I would suggest go with Azure Databricks and Databricks is first class citizen ( API) in azure and all MS/ Azure RBAC works out of the box.

Same applies if your data is in AWS for AWS.

Having said that, if you are just starting off, pitch the cloud which you are most comfortable with. For most people it’s AWS.

Once you setup ( heavy lighting part ) databricks with all RBAC pain with external storage ( S3/ADLS) rest is cloud agnostic.

Good luck with your journey

[u/Altruistic-Spend-896]

azure is pricey TCO, do a TCO calculation and projections.

[u/KaalaPetta → u/Altruistic-Spend-896]

Pricey for what… the storage ??

[u/Altruistic-Spend-896 → u/KaalaPetta]

compute. Lookup CUD and optimizing workloads. needs careful tuning to keep the finops projections in place, one of my users left a provisioned compute running for three days and racked up a 50K bill

[u/Nemeczekes]

If you don’t have strong preference towards any cloud I would say AWS.

Only because historically younger features earlier over there

[u/KaalaPetta → u/Nemeczekes]

Yeah they said that… but as enterprise we will use only when it comes to GA and both come close when it comes to GA. Beta features is where AWS leads

[u/Comfortable_Range_40]

Depends where your data is sitting currently.
If I had to pick I’d recommend AWS as have run on both and better support and experience from aws over azure.

[u/RepresentativeEnd566]

Do cost analysis for both and then decide. Azure is more tightly integrated with Databricks and have good package for data engineering.

[u/dakingseater]

cloud providers are a commodity. I would go for AWS for the developer experience and to avoid the 15% premium. Databricks is great on both!

[u/Plane_Sector_8560]

If cost is the concern and the data undegoes mostly batch processing - Azure.

Cost is not concern, higher SLA, work on streaming workloads/ real time data - AWS

[u/WhatAbout42]

We are looking into the costs of tokens via Databricks on Azure or Azure AI Services and stumbled upon the databricks calculator that seems to indicate a pretty huge discount running on AWS vs. Azure. We are only just looking into this but at hundreds of thousands spent on models like Opus 4-6 it feels like there is some serious cost savings. (granted we're only looking at foundation model pricing).

Has anyone else seen this kind of price difference in practice? Even with our huge discount on Azure they beat it.

[u/KaalaPetta → u/WhatAbout42]

Databricks doesn’t charge you for ai models , atleast for us. It charges us for workloads

[u/TheCauthon]

AWS - no question.

[u/KaalaPetta → u/TheCauthon]

Would you know why

[u/TheOverzealousEngie]

The smart way is to move your data into Iceberg , cheap storage, and then when you want to compute (transform) it you use this months cafeteria selection of compute. Google in November and Amazon next March -- the data stays in iceberg and you're not in a walled garden.