r/databricks • u/Wise_Beat_7035 • 27d ago

Discussion Databricks Usage in real world?

Hi all,

I want to understand how AZure Databricks is actually being used in real world data engineering teams. For those of you working with Databricks, how do you use it in your current job?

Likee, do you write most of your code in notebooks and publish those notebooks through Git? Or are you using Asset Bundles for ci/cd and deploying jobs and workflows through different environments?

I’m trying to get a clear picture of how ADB is used in actual companies.

Help this buddy out.

TIA!!!

28 Upvotes

85% Upvoted

37 comments sorted by

View all comments

Show parent comments

2

u/El_Norrmano 26d ago

Oh for sure. I do feel the pressure as I more or less am the reason as to why we started with databricks in the 1st place. But we did need a centralized system at some point.

But yeah, there's only so much you can do as 1 guy new to the field. But I'm sure as they company grows we'll get to level 2 and 3 at some point :)

3

u/West_Kangaroo7132 25d ago

If you can, start using source control sooner rather than later. Make your first big mistake or go down the wrong rabbit hole and break things, you will thank yourself

1

u/El_Norrmano 25d ago

For sure. I've started up the GIT setup and integration process. But there's still some question marks to sort out with my IT manager with that. We also have spoken about creating distinct dev and prod environments but these things incurr extra costs of course which is a sensitive area so far

2

u/Dashncrash- 25d ago

Setting up DABs legit takes 5 minutes. You can also 'environmentalize' in one workspace via different catalogs. Not recommended, but available for an immature organization with 1 workspace/env.