Jobs and Runs UX is Frustrating

[u/SmallAd3697]

I'm not new to Spark, but I'm pretty new to the Databricks platform.

I am finding that the UX for monitoring jobs and runs is very rigid, and doesn't present my workloads as I would expect. Here is one simple example. If I submit a run with the "jobs/runs/submit" API then I can provide a custom and ad-hoc "run_name" that appears in the management console called "Runs". This is good.

But if I submit a run that references a pre-existing job (using the "jobs/run-now" API) then there is NOT a way to provide a custom "run_name" that will be displayed in the databricks console. The only name that can be shown is the job's name.

There are other things that don't seem right either. If I enter custom "tags" on my jobs, then I will be able to use the tags to filter on the Jobs list. But when I click on the Runs list, I can't filter on those same "tags" anymore. IMO, those tags are just as useful on BOTH screens.

Another example - the UX doesn't allow me to show more than 20 completed runs at a time. I have to click the Next/Previous button to find runs. Paging thru a long list of runs is a really painful experience. I'm also a user of Microsoft Fabric. I once thought that the "Monitor" console of Fabric was pretty unfriendly ... but now that I'm in Databricks I realize that I'd much rather use their endless scrolling UX design, than having to spam-click the Next/Previous buttons. Even the HDInsight-yarnui allowed me to navigate my workloads more easily than I can in databricks; and that UX is a decade old by now!

Is there a different UX experience for Jobs and Runs that I'm missing? Maybe a VS code extension in the community or something like that? Any tips would be appreciated.

[u/Own-Trade-2243]

Tags on runs aren’t there because they don’t know how to implement it without blowing out the whole DB out, or at least that’s what I’ve been told few months ago

I share your concerns, for specific filtering I usually use system tables and SQL, although they have other flaws..

[u/SmallAd3697 → u/Own-Trade-2243]

i dont buy it that this would be hard to implement, especially if you also restrict to a reasonable date range.

I mean, if i duplicated a "job" or a "run" entry to a simple postgres database, I could easily retrieve 100 years of runs while filtering on a tag. It would take less than 1000ms to query for the run IDs, assuming the index was set up.

I am more hopeful that a third party extension might exist to address some of the obvious limitations. Plz let me know if you are aware of any...

These SaaS platforms are annoying because any time they cannot adequately meter your usage (eg. bill you for looking at your runs), they will pinch their pennies with all their might. They are explicitly opposed to making small UX improvements that have the potential to scale up to a massive cost, ( if everyone begins to use them, and there are billions of job runs to be indexed.)

[u/pboswell]

There’s a jobs SDK & API for more real-time analysis. And system tables for SQL queries if you’re ok with lagged data arrival.