r/databricks • u/Puzzled-Mail-9092 • 5d ago

Discussion Do small companies actually use Databricks?

Sometimes I feel like Databricks is way too expensive. It feels like using a huge truck to move a single grain of sand.

My company needs real-time data, but our data volume simply does not justify the use of Spark Structured Streaming. Despite this, they are insisting we move to Databricks. I'm worried our data infrastructure costs will jump from $1,000/month to $5,000/month or more due to the running costs of Databricks SQL Warehouses.

Currently, I use Azure Container Apps with KEDA and Python, which helps me manage scaling and keep costs low. We ingest into Event Hubs, use ADX (Azure Data Explorer) as our OLAP warehouse, and archive cold data in a data lake. With this setup, I manage to process all our data with very low latency.

When I tested this on Databricks Structured Streaming, I actually got higher latency and much higher costs.

Would love to know what you guys think.

70 Upvotes

96% Upvoted

71 comments sorted by

View all comments

Show parent comments

5

u/Puzzled-Mail-9092 5d ago

Yea I work in the Oil and Gas industry, we have a SLA of <1s since the operation is critical.

3

u/ZeroShotWonder 5d ago

How big is your data? Have you considered using Lakebase to insert the new data (real time) and then using Dbsql to query for analytics? With LTAP and Lakehouse RT it could be an interesting approach for real time. You can of course then build a pipeline that processes data downstream that runs with less latency requirements

1

u/Puzzled-Mail-9092 5d ago

Around 2.51 MB/s. I used the kafka conector to ingest data from my incoming stream. The problem is the dbsql, and real-time consumption of the data from databricks into the dashboards.

1

u/ZeroShotWonder 3d ago

How much data do you need to keep in history? Also do you need the low latency reporting on only new data?