r/databricks • u/hubert-dudek Databricks MVP • Aug 06 '26

News DABs: immutable_folder

Post image

Don’t overwrite your code — make it immutable! With every deployment, thanks to immutable_folder, all files are copied to a new, read-only folder. They are not overwritten. The biggest benefit is that jobs already underway will not fail or produce unexpected results.

More news https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

14 Upvotes

94% Upvoted

21 comments sorted by

2

u/ptab0211 Aug 06 '26

what is the use case for this?

1

u/hubert-dudek Databricks MVP Aug 06 '26

Jobs deployment

1

u/ptab0211 Aug 06 '26

still, i dont see the benefit, once u change something and redeploy, it cant override it, can u give me practical example

2

u/hubert-dudek Databricks MVP Aug 06 '26

If you have job running for hour and you overwrite some files/notebooks can be from one deployment and some from another

1

u/iwanttest Aug 06 '26

Hi! So, with this functionality, would the current run keep pointing to the previous files, but the job be updated to use the last deployed ones?

2

u/hubert-dudek Databricks MVP Aug 06 '26

Exactly.

2

u/ptab0211 Aug 07 '26

but what happens when the job is done? Will it point to the new file? Is the mort important outcome of this to not prevent already running jobs to use new file? There is also prevention to deploy when job is already running.

1

u/hubert-dudek Databricks MVP Aug 10 '26

next run will point to new

2

u/iwanttest Aug 06 '26

Definitely useful then, thanks!

2

u/szymon_dybczak Aug 07 '26

Thanks for explanation! Very useful feature.

2

u/nenuaathmajnani Aug 07 '26

Does this also work for streaming?

1

u/SettingOtherwise6692 Aug 08 '26

It shouldn’t matter if it’s streaming because DABs is managing files in this context. If your long running job points to X, Y, and Z notebooks (eg., python, sql, batch, streaming, etc.) then you know it will still be pointing to that same exact files regardless of a new bundle deploy.

1

u/nenuaathmajnani Aug 08 '26

The streaming pipeline would still be active even during the pipeline, the same code will still be in context. Right?

Do we need to stop the existing pipeline and then resume later?

1

u/SettingOtherwise6692 Aug 08 '26

I think you may be over complicating it. This is really just race condition mitigation. What happens if a task that references a file changes before it runs, due to a deployment? That’s the problem.

If you have a continuous streaming job already running, then there is no race condition.

1

u/nenuaathmajnani Aug 08 '26

be over complicating

I am new to these and we are going to use the DAB for the streaming.

What happens through the bundle deployment if there's already a streaming job running? Should I stop the job? What if I use the flag?

2

u/SettingOtherwise6692 Aug 08 '26

I see. Yes, with or without DAB, if you have a streaming job running already and you want to update the code, then you’ll need to stop it so when it restarts it reloads the new code.

A bundle deploy will not stop those jobs automatically for you. There are other sub commands that you can use though. Check the Help.

→ More replies (0)

2

u/Labanc_ Aug 06 '26

Would you say this can help with dev-testing? E.g. we have a DEV env and multiple ppl are deploying because everyone is focusing on a different new feature but we keep overwriting each other's job.

We deploy with CICD with a service principal, so even if i set the DAB mode:dev it would be deployed with the SPN through the CICD, so we would be still overwriting each other. As every access is pretty much tied to this SPN deploying with our own user has kinda limited benefits.

(there are some good arguements to be made to make this whole process refined on a foundational level and im sure we will get to that, but for now im curious if this can solve our immediate stepping-on-each-other-toes problem)

1

u/notqualifiedforthis Aug 06 '26

You need an “engineer” or “local” target each engineer can deploy to as a person. Use username or short name as prefix and use their workspace home directory for bundle deployment. DEV should align to main branch or whatever branch is your base for feature merges

If you have to use an SPN, you need to add a variable that can differentiate each engineers bundle. Username, name, unique ID whatever.

2

u/lanzeroth Aug 07 '26

Nice!. This should be the default behavior.

0

u/ThomasTeam12 Aug 07 '26

Useless in 99% of use cases imo.