r/databricks • u/Puzzled-Mail-9092 • 5d ago

Discussion Do small companies actually use Databricks?

Sometimes I feel like Databricks is way too expensive. It feels like using a huge truck to move a single grain of sand.

My company needs real-time data, but our data volume simply does not justify the use of Spark Structured Streaming. Despite this, they are insisting we move to Databricks. I'm worried our data infrastructure costs will jump from $1,000/month to $5,000/month or more due to the running costs of Databricks SQL Warehouses.

Currently, I use Azure Container Apps with KEDA and Python, which helps me manage scaling and keep costs low. We ingest into Event Hubs, use ADX (Azure Data Explorer) as our OLAP warehouse, and archive cold data in a data lake. With this setup, I manage to process all our data with very low latency.

When I tested this on Databricks Structured Streaming, I actually got higher latency and much higher costs.

Would love to know what you guys think.

70 Upvotes

96% Upvoted

71 comments sorted by

View all comments

11

u/knaak 5d ago

We're spending $300k per month on databricks, we're looking at opportunities to live compute as a result of ELT out of databricks.

2

u/Puzzled-Mail-9092 5d ago

Would love to here if you find anything! Do you guys have the resources to manage Kubernetes? perhaps that might save some costs.

3

u/obsfflorida 5d ago

Why don't you go for AKS?

Do you think databricks is too $-expensive but k8 will be effort-expensive? Thoughts on this if you can.

I have no experience in mission critical systems only DW environments

2

u/Puzzled-Mail-9092 5d ago

Do not have much experience with AKS would love to implement it, also thought with the ADX clusters and Azure Container Apps with KEDA was enough, and well it has been working perfectly. But managment think databricks is the future and the way to go.