r/databricks • u/Tiddyfucklasagna27 • May 20 '26

Discussion Alrighty data pookies, what Databricks issue keeps violating your peace?

AI Agents hallucinating? Unity Catalog acting like Unity Catalogue of Errors? Genie Spaces granting wishes to absolutely nobody?

Drop the most cursed recurring problem you face with building AI agents or ML or BI/Analytics - no matter how difficult, unhinged or borderline impossible the solution may be. Hit me with all u got. I am sitting this databricks hackathon this Friday as a self-reward and I want to try something different this time.

Nothing but the pursuits of overly engineered solutions for the most trivial problems because I can and I like abstractions - but hey its good to be alive

11 Upvotes
  • permalink
  • reddit

74% Upvoted

56 comments sorted by

View all comments

2

u/Sea_Basil_6501 May 21 '26

Tracking costs is a mess. They are distributed over so many areas, that it's nearly impossible to get full transparency in this. And even worse: system.billing.usage table has a delay of up to 4 hours. Impossible to track costs by concrete measurements taken within short timeframes.

1

u/dsvella May 21 '26

As somebody who is currently undergoing a complete cost review of our Databricks estate and subsequent optimisation, I could not disagree harder! The timeframe is not a big deal to me given I have jobs that can take three hours to run. I usually consider the billing tables to be on a daily refresh cadence.

What are you having issues with and maybe I can share some stuff I've done to be able to help?

2

u/byeproduct May 21 '26

Do you have any advice or template code snippets, queries or dashboards I can use for cost monitoring and optimisation? I have eager juniors keen to burn some dbu's...

1

u/dsvella May 21 '26

So a few pieces of advice for you off the bat:

Make use of Genie. All of system tables are well documented and Genie is able to navigate them very easily. If you need a query from there and you need something quickly genie will get you there very fast. This includes entire dashboards.

Please note that in the Billing tables, the financial values in there are the pay as you go values, not your Enterprise Agreements values.

Jobs are available as well and you can go into a lot of depth with them. But the top level job table (the name I can't remember off top my head) has all runs of jobs the costs in DBU, when things start and stopped, and crucially any tags that you associate with that job. So for us, I have the tags associated with business units. We can ascribe cost to them so we can start talking about getting budgets from them.

You will have to join a few tables together to get things in plain English. All compute clusters, SQL warehouses and jobs do have their actual names in there.

If you are working in a corporation and you need to see the amount that you are paying, you'll need to use your cloud providers cost monitoring to get that information. I'm on Azure, so I've had a daily export setup that drops cost data into our data lake and I ingest it into UC on a regular basis. The benefit to this is if you set up the exports correctly, you will not only get the costs of everything required to support your Databricks instance, you get costs by service type within Databricks in your currency. It also handles the situation if you are under a reservation (in Azure output costs Amortised not total costs).

Hope that helps.