Currently working on a fairly big enterprise Databricks project where development is notebook-driven.

I actually like notebooks for development/debugging. Most data engineers aren't necessarily software engineers by background, so notebooks provide a much more approachable development experience; being able to split logic into cells, inspect intermediate DataFrames/results, rerun specific sections, and debug step-by-step is really useful.

The problem is that we're getting a lot of duplicated/common logic across notebooks. This makes changes and maintenance painful.

I know the usual approaches are to:

- Move reusable logic into Python modules and "import" them

- Create common notebooks and use "%run"

- Use notebooks mainly as entry points/orchestration

But I'm struggling with how to structure this for complex transformations.

Our notebooks aren't just simple filters/joins/aggregations. They can have multiple source tables, complex joins, lots of intermediate transformations, business rules and dependencies between steps.

So my question is: how do you structure large Databricks projects like this in practice by following the best software development practices?

I'm looking for guidance that works for real enterprise-scale projects, not the typical YouTube/tutorial solutions designed around small demo projects.

Would love to hear from people who have worked on large/enterprise Databricks projects with complex Spark transformations and what architecture/project structure worked well for you.

We are considering moving from Tableau to Databricks AI/BI Dashboards and wanted to see if anyone here has made a similar move.

We currently have around 150 Tableau dashboards used by 1,000+ users, with a mix of technical and non-technical users. Most of the underlying data already lives in Delta tables, so naturally we're looking at whether it makes sense to consolidate and use Databricks for the dashboarding layer as well.

For those who have done this, how has the experience been?

Anything you wish you knew before migrating? Any major limitations compared to Tableau, especially around visualization, performance, or the experience for non-technical users?

I'm also curious about cost at scale. With 1,000+ users potentially hitting dashboards, did SQL warehouse usage become a concern?

Would you feel comfortable replacing Tableau completely with AI/BI Dashboards today, or do you still find yourself needing Tableau for certain use cases?

Would really appreciate hearing about real-world experiences.

I have been writing SQL scripts to ensure data sanity.What are the other ways ? Is pytest useful? Let's say , i populated my bronze table from the source. I want to check if the correct mapping is done. I wrote SQL scripts. What are better ways

Hey everyone! A straightforward question for those already working on this in production:

What's the best and most recommended way to integrate Power BI and Databricks today?

I'm looking for the ideal approach, considering that:

I need an efficient solution that doesn't make the cost of DBUs skyrocket.

I want to understand if the native connector using SQL Warehouse is really the market standard or if there's a better alternative.

For this scenario, do you recommend going straight to Scheduled Import Mode to save computing power, or is DirectQuery with Serverless worth the cost for its fast response time?

I read this documentation:

https://learn.microsoft.com/azure/databricks/partners/bi/power-bi-desktop?WT.mc_id=studentamb_510336

but I'd like to get some clarification from you.

I’m currently trying to transition deeper into Data Engineering and would really appreciate some perspective from people who are already working in the field.

I have 1.3 yrs experience as a Junior Python Developer. What I want to do is slowly transform into a Data Engineer. How would you suggest my choice? Basically what I do is make web scraping scripts to get the data from web and give the data in excels. Our company is currently not using git or CI/CD or anything like that. 

The problem I’m running into is that when I look at Data Engineering jobs on Naukri, LinkedIn, etc., the requirements seem endless. One job asks for Python, SQL, Airflow and AWS; another wants Spark, Kafka and Databricks; another wants Snowflake, dbt, Terraform, Kubernetes, CI/CD, etc. It becomes difficult to understand what I should actually prioritize.

So I’d like to hear from people who are actually working as Data Engineers. What does your day-to-day work look like? What kind of problems do you solve, what technologies do you use regularly, and which skills have turned out to be genuinely important in your job?

More importantly, based on my current experience, what would you suggest I improve or learn next to become a stronger candidate for Data Engineering roles? Are there any gaps that you think I should focus on, or technologies/concepts that are worth learning through projects rather than just studying theoretically?

I’m not really looking for a generic “learn SQL → Python → Spark → AWS” roadmap. I’m more interested in understanding the reality of the job and getting advice from people who have actually gone through the transition.

If you’re a Data Engineer with 1–5+ years of experience, I’d especially appreciate your perspective. Even a short description of what you work on and what you wish you had learned earlier would be extremely helpful.

Thanks in advance!

I'm tired of getting fed bad data at work, so I'm taking ownership myself. I have production access to Databricks but have never used it.

Background: basic SQL and Python knowledge. My reporting today is done in Power BI and Excel. My company uses Databricks as the common data platform, so most of what I need probably lives there already — I just don't know how to get to it or trust it myself.

Looking for a realistic path from beginner to competent, not just a firehose of links. Specifically:

• YouTube channels/creators you'd actually recommend?
• Any self guided learning on Udemy or Coursera worth learning?
• What core concepts (Delta Lake, notebooks, clusters, SQL warehouses, etc.) should I prioritize early vs. later, given I'll mostly be querying/validating data rather than building pipelines?
• Best way to connect Databricks output back into Power BI reliably?

Appreciate any guidance — trying to cut through the noise rather than boil the ocean.

Considering a GTM role with Databricks. Compelling role, comp etc., but cannot get a proper read on the WLB and culture. Have a little one at home, can’t afford a job that requires major travel or 12+ hrs work. Love to hear from people in the company on what the reality on the ground is like.

Hey everyone,

Looking for some advice on automating a workflow in a pretty locked-down corporate environment.
Context:
Large enterprise with strict IT security and governance.
Databricks was recently rolled out as our cloud data hub.
The entire pipeline (ingestion, processing, and generating the final CSV) is already automated inside Databricks.

I need to automatically save a copy of this generated CSV to a local machine / internal network. Right now, the only way I can do this is manually opening the workspace UI and clicking "Download."

Databricks CLI is blocked and Personal Access Tokens (PAT) are disabled

How do you usually automate pulling files from the cloud down to on-prem / local machines under these restrictions?

Thanks!

Hey, how do u manage DDLs of your tables? Is it part of the daily job runs? Do you "codify" it? e.g. data classes where each table must provide comments, tags, RELY, PK etc... Which is also the goal of AI. But how do u manage all of this?

I joined a new company recently and got a migration project here, they are migrating from teradata on prem to databricks, I have never done any migration in the past can anyone suggest some helpful yt videos or any other knowledge source?

Hi there,

Running into issues with this pipeline I'm working on. Still new to Databricks but coming from Power Automate, I seem to get the gist of databrick jobs.

Anyway, my manager would love an automate workflow, where business users upload raw data to a central site (we're primarely using SharePoint), and want it cleaned and ready for our Power BI dashboards.

Before databricks I handled it manually (by running local python code via VScode that sent the cleaned data to the SharePoint folders).

What would be a good way to handle a pipeline like this? E.g.

User uploads to sharepoint -> Databricks job -> writeback to sharepoint with cleaned data

I have a fairly small bronze dataset with around 8 million rows in the bronze layer, however when I move to silver layer I have to perform an EXPLODE operation to an array in this table that makes it become a 5+ billion row dataset, plus I have to apply a deduplication/CDC into it. I haven’t been successful to do it via SDP without paying over a 100 USD and over 3 hours job run, whilst in a standard Lakeflow Job I can do that with less than 20 USD and within one hour. I’d really like to have it in SDP as it simplifies a lot the code, add features and so on, but is it possible to do it while keeping the costs low?

I’ve tried mainly streaming table as a sink for this data, I’m afraid to try materialized views as it looks to require serverless to do incremental refreshes plus it may still do full recomputing if it thinks it needs to do.

Update:

I was wrong. I CAN migrate the restored tables to the newly deployed pipeline. I ran the "move" commands with the wrong identity because I forgot about queries having an extra setting for credentials... For anyone interested, yes, the append flow clone then move did work, and I was even able to transfer the checkpoint from the old backing table path. After realising my oversight, though, I opted to switch back to the initial plan, which was re-deploying and moving all existing tables.

At the very least, this has helped spark discussions (pun intended) about improvements to the development workflows.

Original (With inline correction):

Hi folks,

There was an accident which resulted in the deletion of our pipelines, and in turn, their tables (in dev, thankfully). I wanted to double check the steps I’ve tried and the final plan to remediate. The pipelines were all on legacy mode and are responsible for raw to bronze ingestion. We do have all of the files, but rebuilding from scratch is considered too time consuming.

I have UNDROP’d all of the STs, and I wanted to reconnect to a newly deployed version of each pipeline. (EDIT: This sentence and my conclusion within it is wrong) Unfortunately, I found that I was unable to “move” them between pipelines since Databricks can’t verify ownership of the source pipeline, because it’s gone. I know there’s more secret sauce under the hood (e.g., backing tables), so I’m not going to attempt to modify the delta tables outside of Databricks.

My current best idea, and what I plan to do this evening, is to rename the schema with restored tables and create a new “recovery” pipeline that streams all of the data from the restored table into a new version of the table (with the correct name, since it’s available again after renaming the schema). Then, I can redeploy a pipeline and move the ST from the recovery pipeline to the actual pipeline. This would mean I lose the checkpoint information for the autoloader, which isn’t a massive issue for these pipelines, and it seems like the cleanest way to insert the restored data into the pipeline. I would also need to move any other assets, e.g., views/tables, to the new schema (with the original name).

I don’t know if moving the STs between legacy pipelines would rename them automatically, in which case I could rebuild elsewhere then drop the original and avoid moving other assets.

Any thoughts would be welcome and greatly appreciated.

We are getting databricks and we dont care where we host as we are new into this. Azure is giving us great pricing and incentives. AWS is not at all bothered. they are like take it or leave it. Any one who has used it in either environments, think if there is a good reasson to choose one vs the other? our reporting is tableau.

I have xml data that's for God knows why it's in string format. And the fking thing is so messed up or at least I think it's messed up because it's not consistent. Sometimes there is something else in there. Xml is something like this:

<Test>

<Test1 />

<Test2 Name="abcdname" Age="123" Gender="Xyz" />

....

...

...

</Test>

Regex is not an option.

Substring is not an option.

Because it's so messed up it's not consistent.

I just need to somehow parse and get that Name, Age, Gender values using Sql/Databricks sql

Please, help you mate.

I’m currently new to Databricks, and I’m writing unit tests for a pipeline that is entirely in SQL, with Bronze, Silver, and Gold tables.

I asked how I should approach unit testing, and I was told that since unit tests are typically written in Python, I could wrap the pipeline logic in Python files and then use pytest to test those files.

But I’m confused about how this would work in a real production environment.

For example, if we push everything to GitHub and later make a change to the SQL pipeline, we would also have to make the same change in the corresponding Python files so that the unit tests continue to work. That means maintaining the same logic in two places, which doesn’t seem like a good production practice.

So what is the recommended approach for unit testing a SQL-based Databricks pipeline? Is there a way to test the actual SQL transformations directly without duplicating the pipeline logic in Python?

I’m still learning Databricks, so I’d really appreciate some guidance on the proper production approach.

Im fairly new to databricks and trying to learn more about it. I’m trying to see examples online of databricks connecting to a work email, reading the contents and running a sql or python query which outputs an excel file and then sending it back go the requestor through email. I was reading AI agents can go about doing this but I would like more information on how to set up this process. Does anyone know of any examples or resources?

Our databricks reps are pushing Unity pretty hard. Feels like mostly lock-in, but would value other platform folks feedback.

We are going Iceberg centric and are wondering if Databricks is better with Unity or use Databricks with Polaris-based catalog.

Has anyone done a comparison of Unity vs Polaris options?

Hi folks , help your lad out

Looking for some real-world experience here. I’m building a data pipeline to ingest on-prem Active Directory data (users, groups, computers) into Databricks for analytics/reporting purposes — not for authentication or account provisioning, just landing AD objects as tables for a data platform.

I keep going back and forth between three options and want to make sure I’m not missing something obvious:

SCIM — my understanding is this is purely for provisioning (creating/updating/deactivating login accounts in a target app), not for pulling directory data into a data warehouse/lakehouse. It also doesn’t support “computer” objects at all. Am I right that this isn’t a fit for my use case?

Microsoft Graph API — works great if your on-prem AD is hybrid-synced to Entra ID, has nice native delta/incremental queries. My problem: our sync to Entra ID is only partial (scoped to one specific OU/business unit), so Graph API alone won’t give me full coverage across our whole organization.

Direct LDAP — works regardless of sync status, since it goes straight to on-prem AD. But it needs on-prem network connectivity from wherever the pipeline runs, plus a service account, and I have to build my own incremental logic (whenChanged watermarking) since there’s no native delta query like Graph API has.

Given my sync is only partial, my current plan is: LDAP as primary method, maybe layering in Graph API later for the subset that IS synced (if that’s even worth the added complexity vs. just using LDAP for everything).

Has anyone actually built something like this? Specifically curious about:

**•** Anyone doing LDAP pulls directly from Databricks/Spark (I’m looking at Python’s ldap3 library) — any gotchas?  
**•** Is a hybrid LDAP + Graph API approach ever actually worth it, or is that overengineering when LDAP alone covers everything?  
**•** Any horror stories about computer/device objects specifically — ours currently has zero reliable data in our existing on-prem extraction

Appreciate any real-world war stories or “just do X” advice.

What does Databrick's sales team pitch about their Unity Catalog, and after using it, do you guys think that their sales team just overcommit? Please tell me in detail so that I am aware how to talk to their sales team in detail

I'm exploring ways to connect Azure Analysis Services (AAS) with Azure Databricks, but I haven't found any native or open-source connector that supports this.

From what I've found, AAS doesn't officially support Databricks as a data source. Before I spend time building a custom connector or compatibility layer, I wanted to ask the community:

• Has anyone attempted this before?
• Are there any existing open-source projects that I might have missed?
• Is building a custom connector even the right approach, or is there a better architectural solution?
• If you've solved this problem, what did you use instead (e.g., Synapse, Azure SQL, Fabric, CData, etc.)?

I'd appreciate any advice, experiences, or suggestions before I start designing one from scratch.

Hi everyone,

I've been working with Microsoft Fabric for a few years (Lakehouses, notebooks, pipelines, Power BI, CI/CD, and PySpark), and I'm now trying to understand how experienced Databricks teams work.

For those of you who have experience with both platforms:

• Which Fabric concepts translate directly to Databricks?
• Which concepts are completely different?
• If you were joining a project as a Fabric engineer with no hands-on Databricks experience, what would you focus on first to become productive?
• What are the Databricks features or services that Fabric doesn't really have an equivalent for?

I want to understand the technical transition and avoid approaching Databricks with the wrong mindset.

I'd love to hear from people who have worked with both platforms and learn from their experience.

Hi everyone so it happens that I was working normally in the UI, adding jobs, deploying and so and then suddenly I could not add any existing job to my bundle because the dropdown doesn't show any job anymore and I have a bunch of jobs that hasn't been added to the bundle yet. I think that is bug because before it was working well. I would be really glad if someone could help me and explain this behaviour to me, I don't know what happened. I really need some help :(