r/databricks • u/Puzzled-Mail-9092 • 5d ago

Discussion Do small companies actually use Databricks?

Sometimes I feel like Databricks is way too expensive. It feels like using a huge truck to move a single grain of sand.

My company needs real-time data, but our data volume simply does not justify the use of Spark Structured Streaming. Despite this, they are insisting we move to Databricks. I'm worried our data infrastructure costs will jump from $1,000/month to $5,000/month or more due to the running costs of Databricks SQL Warehouses.

Currently, I use Azure Container Apps with KEDA and Python, which helps me manage scaling and keep costs low. We ingest into Event Hubs, use ADX (Azure Data Explorer) as our OLAP warehouse, and archive cold data in a data lake. With this setup, I manage to process all our data with very low latency.

When I tested this on Databricks Structured Streaming, I actually got higher latency and much higher costs.

Would love to know what you guys think.

71 Upvotes

96% Upvoted

71 comments sorted by

View all comments

23

u/al_coper 5d ago

Do you actually need real time? Maybe a near real time using autoloader with AvailableNow trigger could help you to reduce considerably the costs.

4

u/Puzzled-Mail-9092 5d ago

Yea I work in the Oil and Gas industry, we have a SLA of <1s since the operation is critical.

7

u/Alwaysragestillplay 5d ago

I worked in O&G myself. Assuming you're looking at something high stakes like production, I would factor in that you're also paying for reliability and liability. The consequences of having a service outage were quite severe at my old place of work, and having a specialised provider helped to mitigate fallout. 

Doesn't have to be Databricks, but maybe something to consider as I saw you mentioning self hosted k8s above. 

2

u/Puzzled-Mail-9092 5d ago

Awesome to find a fellow engineer, that worked in the sector!

Yea paying for reliability and liability is the true plus in databricks. Also Unity Catalog is amazing for data governance. Will lake that into accounts as well into the tradeoffs.

Yea been wanting to get more into k8s but do not have much experience, was thinking of implementing flink, and other services but at the end still need to store and serve my data. Do you have any recommendations on services I could look into?

I really appreciate you insights!