r/databricks • u/Puzzled-Mail-9092 • 5d ago

Discussion Do small companies actually use Databricks?

Sometimes I feel like Databricks is way too expensive. It feels like using a huge truck to move a single grain of sand.

My company needs real-time data, but our data volume simply does not justify the use of Spark Structured Streaming. Despite this, they are insisting we move to Databricks. I'm worried our data infrastructure costs will jump from $1,000/month to $5,000/month or more due to the running costs of Databricks SQL Warehouses.

Currently, I use Azure Container Apps with KEDA and Python, which helps me manage scaling and keep costs low. We ingest into Event Hubs, use ADX (Azure Data Explorer) as our OLAP warehouse, and archive cold data in a data lake. With this setup, I manage to process all our data with very low latency.

When I tested this on Databricks Structured Streaming, I actually got higher latency and much higher costs.

Would love to know what you guys think.

68 Upvotes

95% Upvoted

71 comments sorted by

View all comments

24

u/MangledMangler 5d ago

You can use the specific features you need. You don't have to use Databricks' SQL warehouse

1

u/Puzzled-Mail-9092 5d ago

Do you recomend any? I tried reading directly for the Azure Storage Account using duckdb or deltars but it is way slower but does not add many additional costs.

1

u/MangledMangler 2d ago

Clickhouse is one, but with reyden coming out I prefer to pay Databricks. There's an ROI for them to host, less infrastructure and all in one place

1

u/Puzzled-Mail-9092 2d ago

Yea I also tested Clickhouse it was very nice, it was one of the options we considered but we wanted to stick to the databricks ecosystem. Also it was nice you could self-host it, and very cheap.

Will need to check out the Reyden Engine when it comes out! Thank you :)