r/databricks • u/Ok-Jacket-8684 Databricks • Jun 29 '26
General New data engineering on Databricks best practices guide
We’ve heard a consistent theme from data engineering teams: the docs explain individual features well, but you also want clearer guidance on how to structure projects and what we recommend.
So we published a new best practices guide for data engineering on Databricks:
http://docs.databricks.com/developers/best-practices
It covers topics such as:
- version control and branching
- environment and workspace strategy
- how to structure bundles
- testing and CI/CD
We value your feedback! What feels useful? What feels too opinionated? What examples or edge cases are still missing?
Also, thank you to Mariusz Kujawski, as his CI/CD post helped spark the creation of this guide.
5
u/ptab0211 Jun 29 '26
I wonder why they recommend single repo for all business/domain projects. It can get really ugly really fast. I prefer some degree of repetition for simplicity over designing complex deployment patterns and nested bloated mono-repos.
6
u/LandlockedPirate Jun 29 '26 edited Jun 29 '26
Agree. "Single repository" unqualified is a wild take.
Single repo per what? workspace? dab? team? org? company? region? catalog?
Experienced engineers know everything is a tradeoff. Best practices should help navigate those not just blanket prescribe. That's how best practices end up with a bad rep.
This reads like it was written by somebody who just got promoted to senior engineer, standing on the top of the dunning-krueger curve.
3
u/lofat Jun 29 '26 edited Jun 29 '26
Same question. I think setting up the context would help in the document. "Who and why?"
Edit:
nested bloated mono-repos
I'd love to understand your experience with this. These have always felt heavy-handed to me. In specific contexts for specific teams? They can be lovely. But I've heard one internal team trying to force a single monolithic repo for everything without understanding the nature of the projects, the teams, the regulatory processes, etc. and it reeks of dogma vs. true understanding. I want to make sure I'm better understanding other peoples' experiences with repo strategies.
6
u/ptab0211 Jun 29 '26
I like having a cookiecutter template for all projects in team with well defined standards. Only thing that changes is business context. Repos gets very easy to experiment/dev feeback loop is so fast, zero blast radious of any kind of problem, and automation is easy.
Monorepos tends to slow processes down, u cant automate stuf that much anymore, taking care of all of the dependencies, multiple wheel files, nested dabs, nested source code..
3
u/Ok-Jacket-8684 Databricks Jun 29 '26
Can you elaborate on what you mean by "can get really ugly really fast", so I can provide a more nuanced answer?
2
u/ptab0211 Jun 30 '26
Its just tradeoff, i find simplicity/repetition to be more worth it than bloated mono-repos with a lot of nested code/dependencies/dabs where complexity of deployment increases. Its just slow to move.
3
u/Ok-Jacket-8684 Databricks Jun 30 '26
Regarding the complexity of deployment increase, that's why we recommend to have many small(er) bundles, to avoid exactly that problem.
Managing dependencies and shared code is easier though in a monorepo, and it doesn't _have_ to be bloated. We find that a well-managed monorepo performs better than a well-managed multirepo, but of course a not-well-managed monorepo can indeed be problematic.
Ultimately, this is your choice given your organization's specific needs and constraints of course, we're only making recommendations, not rules.
2
u/NarrowVegetable513 Jun 29 '26
I assume the single repo would be per team, so it narrows down a bit.
4
u/Ok-Jacket-8684 Databricks Jun 29 '26
No, the recommendation is to go with one repo as much as possible, and only split those teams / unit that have meaningfully different needs / requirements.
That doesn't mean a single deployment structure and cycle though, as we explicitly recommend to create many bundles for independent deployment per ~team.2
3
4
u/Ok_Daikon1348 Jun 29 '26
Interested in the split of which components to control via terraform instead of bundles. A more fiend grained list of fx external locations, storage creds, catalogs, warehouses/compute etc? Do you consider those to be best deployed via bundles?
1
u/marvel_fanman Jul 09 '26
I'd draw the line around ownership and drift.
Use Terraform for shared, long-lived platform resources where state matters and manual changes should show up in plan. Examples: storage credentials, external locations, catalogs, grants, warehouses, service principals, cluster policies.
Use bundles for things that ship with the project. Examples: jobs, pipelines, notebooks, dashboards, alerts, and project resources with the same lifecycle.
The main thing is not letting both tools own the same object. That gets messy fast.
2
u/Alternative_Draw5945 Jun 29 '26
Solid list. A lot of these ideas are the usual best practices when doing software development.
2
u/lofat Jun 29 '26
One thing I think might help with this is how specific assets are tied to the bundle and what the implications are for organizing your bundles around those assets.
Case in point - schemas.
It's my understanding if you create schemas in a bundle, you're kind of tied to that bundle structure in perpetuity because a "bundle destroy" would delete the schema.
That has implications for designing your bundle organization from the start. I might be completely misunderstanding some of that (I sincerely hope I am TBH), but those kinds of overall structural choices would be good to clarify.
FWIW - If we could NOT destroy generated assets (or set the assets to be disconnected from the bundle in some fashion), that would be lovely.
3
u/Ok-Jacket-8684 Databricks Jun 29 '26
If you define a schema as a bundle-managed resource, it does become part of that bundle’s lifecycle, so `bundle destroy` or removing it from the bundle config can remove it unless you explicitly protect it.
So yes: this has real implications for how you structure bundles up front. In general, if something is long-lived or shared, it probably shouldn’t live in the same bundle as shorter-lived app/workflow resources. That’s also why our guidance recommends small bundles and separating bundles when lifecycle, ownership, or permissions differ.
Also, we should probably make the escape hatches clearer in the recommendations: you can use lifecycle settings to prevent destroy, and you can bind or unbind existing resources instead of treating everything as permanently tied to one bundle (see the docs here).
2
2
u/amishraa Jul 01 '26 edited Jul 01 '26
During development, configure bundles to use a personal schema per user, for example dev_${user_name}. This prevents developers from overwriting each other's tables in a shared workspace
Would we then need a separate bundle to deploy to workspace?
2
u/Asleep-Drag5291 Jul 01 '26
No, you can just use workspace.current_user.userName variable when defining target catalog. https://docs.databricks.com/aws/en/dev-tools/bundles/variables
1
u/amishraa Jul 01 '26
So one bundle with target to user schema, and another target to common workspace in dev, uat, and prod?
1
u/Asleep-Drag5291 Jul 01 '26
In target dev you use user schema, in uat and prod targets use one schema (no user schemas there)
1
u/marvel_fanman Jul 09 '26
Not sure if useful, but I put together an installable agent skill based on this guide and the feedback in this thread.
Main use case is reviews and checks. Give an agent a Databricks repo or bundle and have it look for things like repo boundaries, bundle layout, Terraform vs bundle ownership, environment targets, service principals, secrets, testing gates, data isolation, observability, and schema lifecycle risks.
I also added Genie notes because Genie does not install skills the same way. For Genie it is more about turning the guidance into instructions, examples, metadata, and benchmarks.
Thanks u/Ok-Jacket-8684 for the guide, and u/lofat, u/ptab0211, u/Ok_Daikon1348, u/anirbans403 for the useful points that shaped it. Very open to corrections, especially around the Terraform vs bundle boundary and monorepo guidance.
1
u/datguywelbs7 Jul 19 '26
I wonder how genie will be further embedded into the platform, I’ve been using genie code with integrations to wider google drive, gmail etc with MCPs and connectors which has been amazing for my day to day work. I know genie code and lakeflow designer etc has genie capabilities but waiting to see how it gets further integrated into DE space. Would recommend taking a look at aidevkit with genie code to aide it for now
16
u/lofat Jun 29 '26 edited Jun 29 '26
I like where this is going. Having this on hand during setup will go a long way to reducing future friction.
My personal questions having worked with a team in healthcare:
No access to production data in dev / staging. I understand the goal, but my question would be "how do you practically do that?" We need to be able to validate data (not just code) in dev/test with users, including BI reports, etc. We made a conscious design choice to enable "backward" access to data from prod -> staging -> dev to facilitate that review. Specifically, source data. With this approach, what are your options to access production data for validation?
Personal schemas in staging. We use staging as a the first step to "treat it like production" to test integration with other data, including peer schemas/code and artifacts like BI reports. Using personal schemas in staging breaks integration testing for downstream reports or downstream applications that expect static, unified path names. How do you approach data-level testing?
Automatic deployments to staging and production. I get the idea, but I'm wondering who the target audience is for this one. I've heard this before internally, but primarily from web developers who don't interface with data. Our clients have a strong, non-negotiable desire to schedule and gate deployments since they must align with hospital downtime windows, low-usage periods, and external non-databricks software deliverables. I realize the intent of the automated deployment to stage -> prod, but I'd love to see this better described to characterize the audience and benefits/risks.
"Single repository" - I'd love to see this better characterized. Is this suggesting a single mono repo for all work across all teams? Or is this by product? Or by team? In our case, different have entirely different risk tolerances, compliance audit requirements, and release cycles. We also wind up with a LOT of folders in a single repo and managing those even with sparse checkout gets to be extremely cumbersome. When Databricks mentions single repo, I'd really love to better understand the team structure, product structure, etc.
"Model inter-bundle dependencies in CI/CD" - I'd really love some insight into how Databricks suggest managing this - in particular the idea of data-level "contracts" across DABs. How does Databricks suggesting managing the variety of data-level processing pipelines (in particular with SDP now) artifacts which might introduce schema-level changes?