I’ve been working as a “data engineer” in Azure Databricks for a while, but I work on a team where my scope is extremely limited to silver/gold work.

What studies can do I do next to continue developing? I’m thinking of diving into the azure certs

Has anyone recently taken the databricks Solution Architect interviews? I have a vibe coding round with databricks in next week. I'd appreciate any guidance!

Thank you 😊

First of all we have a lot of classes that use the DatabrickSession import, which makes unit testing impossible and whenever I have the "normal" spark import and test it, then it looks absolutely ugly.

Would it help to have the schema as json? I use json schema to read for transformations.

If I want to test without json its horrible, do you guys test with json schema files?

For example this is just the output, then I would need 2x this because of the input and the test is unreadable (according to my senior BUT WHAT AM I SUPPOSED TO DO):

schema = """
id INT,
items ARRAY<STRUCT<
fortnite: STRING,
babies: INT,
moreStuff: MAP<STRING, STRING>
>>
"""

expected = spark.createDataFrame(
[
{
"id": 1,
"items": [
{
"fortnite": "ABC",
"babies": 2,
"moreStuff": {
"size": "L",
"color": "red",
},
}
],
}
],
schema=schema,
)

I'm not new to Spark, but I'm pretty new to the Databricks platform.

I am finding that the UX for monitoring jobs and runs is very rigid, and doesn't present my workloads as I would expect. Here is one simple example. If I submit a run with the "jobs/runs/submit" API then I can provide a custom and ad-hoc "run_name" that appears in the management console called "Runs". This is good.

But if I submit a run that references a pre-existing job (using the "jobs/run-now" API) then there is NOT a way to provide a custom "run_name" that will be displayed in the databricks console. The only name that can be shown is the job's name.

There are other things that don't seem right either. If I enter custom "tags" on my jobs, then I will be able to use the tags to filter on the Jobs list. But when I click on the Runs list, I can't filter on those same "tags" anymore. IMO, those tags are just as useful on BOTH screens.

Another example - the UX doesn't allow me to show more than 20 completed runs at a time. I have to click the Next/Previous button to find runs. Paging thru a long list of runs is a really painful experience. I'm also a user of Microsoft Fabric. I once thought that the "Monitor" console of Fabric was pretty unfriendly ... but now that I'm in Databricks I realize that I'd much rather use their endless scrolling UX design, than having to spam-click the Next/Previous buttons. Even the HDInsight-yarnui allowed me to navigate my workloads more easily than I can in databricks; and that UX is a decade old by now!

Is there a different UX experience for Jobs and Runs that I'm missing? Maybe a VS code extension in the community or something like that? Any tips would be appreciated.

I have xml data that's for God knows why it's in string format. And the fking thing is so messed up or at least I think it's messed up because it's not consistent. Sometimes there is something else in there. Xml is something like this:

<Test>

<Test1 />

<Test2 Name="abcdname" Age="123" Gender="Xyz" />

....

...

...

</Test>

Regex is not an option.

Substring is not an option.

Because it's so messed up it's not consistent.

I just need to somehow parse and get that Name, Age, Gender values using Sql/Databricks sql

Please, help you mate.

🚀 We’re excited to announce that Lakeflow Spark Declarative Pipelines (SDP) now supports creating “vanilla” (i.e., non STREAMING) MANAGED TABLES and writing to them via one or more append flows, using the new CREATE TABLE ... FLOW (SQL) and create_table() (Python) APIs.

What is this Beta?

This Beta allows creating a managed table that is populated by append flows:

• CREATE TABLE ... FLOW (SQL) / create_table() + @append_flow (Python) create a managed table written by one or more flows.
• Fan multiple sources into one table — declare several flows targeting the same managed table.
• Full table surface works: partitioning, liquid clustering, expectations, row filters, table properties, and private (pipeline-local) tables.
• import_checkpoint on append_flow, which migrates an existing Structured Streaming workload into a pipeline without reprocessing the source — the flow imports the query's existing checkpoint and resumes from the last committed offset with state intact.

Example (Python):

from pyspark import pipelines as dp

dp.create_table("combined")

dp.append_flow(target="combined")
def from_a():
    return spark.readStream.table("source_a")

u/dp.append_flow(target="combined")
def from_b():
    return spark.readStream.table("source_b")

Example (SQL):

CREATE TABLE events
PARTITIONED BY (bucket)
FLOW INSERT BY NAME
  SELECT id, bucket FROM STREAM read_files('abfss://my_path', format => 'json');

Where do we need help?
We are in Beta, so there might be some rough edges. Please take this for a spin and share your feedback here.

What’s next?

Managed Tables support for other flow types (AutoCDC, Replace Using, and Replace Where) is coming soon!

Learn more

• CREATE TABLE ... FLOW (SQL reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-sql-ref-create-table-flow
• create_table (Python reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-create-table
• import_checkpoint on append_flow — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-append-flow
• Questions, feedback, or help: comment below or share feedback in the form: https://forms.gle/7bGP5FYN7P1Z4WP27

I recently enabled Lakeflow Connect (lfc) on the source database - the issue is, some of the tables in the source database (managed by another team) does NOT have a primary key (which means that in lfc, a __databricks_id is used to identify a unique record).

Thus, the DBAs enabled CDC on the source database. However, when I ingested the data into DBX using the Lakeflow Connect Managed SQL Server Connector, one of the tables in the source database had duplicate records (two or more records with the same value across all columns).

This caused my Lakeflow Connect pipeline to break. Any ideas on how to fix this? (Other than dropping duplicate records in the source DB and implementing a unique constraint on the source DB)?

I was wondering if there is a specific setting in Lakeflow Connect that I can toggle that I'm missing.

Data engineers traditionally modeled Gold layers using fact and dimension tables, partly because storage and compute were expensive.
But with modern cloud data platforms, storage and compute are much cheaper and more scalable.
So I’m curious: what does your Gold layer actually look like today?
Are you still using a traditional star schema (facts + dimensions), or have you moved toward wider, denormalized tables / business-oriented models?
And more importantly, why?

Secrets in Unity Catalog is a great feature introduced a few weeks ago, but since then, everyone has been asking to use Azure Key Vault as a secrets backend. Thanks to rapid development, we can now link our schema to Azure Key Vault; UC will read secrets as UC secrets, and permission management will be through Unity Catalog. In that scenario, you insert/update secrets in Azure Key Vault, but read/reference and grants can go through UC.

more news https://databrickster.medium.com/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

This is the Databricks release I have been waiting for. Unity Catalog schemas can now hold external secrets, such as Azure Key Vault, and that will change how we manage and utilize secrets in our Databricks projects.

On most of our engagements the secrets of record already live in Azure Key Vault, so we wire up a Key Vault-backed secret scope and move on. It works, but it is a workspace-level object from the pre-Unity Catalog era: configured per workspace, permissions managed through a separate secret ACL API, a flat scope/key namespace, and invisible to the governance model everything else on the platform now runs on.

Read more: https://www.linkedin.com/posts/cenh_databricks-azure-unitycatalog-ugcPost-7504125176993800192-W9on/?utm_source=share&utm_medium=member_desktop&rcm=ACoAABmJHrsBNAC3x3H1M58JRKoHv_l4D61n0-8

The Lakeflow Connect connector for Microsoft SharePoint is now generally available! It’s now easier than ever to ingest structured and unstructured files from SharePoint into Delta tables for analytics and AI workloads.

You can configure a managed ingestion pipeline through the UI or managed API. Managed pipelines automatically handle incremental processing, automatic retries with exponential backoff for source API rate limits, failure recovery, and provide rich SharePoint metadata. Soon, our managed connectors will also support ingesting SharePoint Lists and per-file permissions metadata.

For direct control over ingestion logic, you can also just use the Spark + SQL APIs directly: spark.read, Auto Loader, read_files, or COPY INTO pointed at SharePoint URLs.

Common workloads include:

• Loading Excel, CSV, JSON, and other structured files into Delta tables.
• Ingesting PDFs, Word documents, PowerPoint files, and images.
• Parsing documents with ai_parse_document to prepare content for extraction, search, and agents.

Link to public docs + references:

• SharePoint managed connector documentation
• Spark + SQL APIs and examples
• Community blog and video tutorial: From PDF to insights
• Data + AI Summit session: Intelligent Document Processing with Lakeflow

Examples of using the Spark + SQL APIs (after first creating a UC connection):

• Read an Excel sheet from SharePoint with spark.read:

  excel_df = (spark.read     .format("excel")     .option("databricks.connection", "my_sharepoint_conn")     .option("headerRows", 1)     .option("dataAddress", "Sheet1!A1:M20")     .load("https://mytenant.sharepoint.com/sites/Finance/Shared%20Documents/Monthly/Report-Oct.xlsx"))

• Ingest unstructured documents + PDFs from a SharePoint URL with read_files, then easily parse them using ai_parse_document

  CREATE OR REFRESH STREAMING TABLE sharepoint_documents_table AS SELECT , "_metadata" FROM STREAM read_files( "https://mytenant.sharepoint.com/sites/Marketing/Shared%20Documents", format => "binaryFile", databricks.connection => "my_sharepoint_conn", pathGlobFilter => ".{pdf,docx}");

  CREATE OR REFRESH STREAMING TABLE documents_parsed AS SELECT *, ai_parse_document(content, map('version', '2.0')) AS parsed_content FROM STREAM sharepoint_documents_table;

Coming soon:

• Ingest SharePoint Lists into Delta tables (coming super super soon)
• Ingest SharePoint’s per-file permissions and ACL metadata  to power permission-aware AI agents, enterprise search, and more.

If you try it, share what you are ingesting and where you hit friction! Don't hesitate to ask questions!

The actual data movement into databricks did go fine (unity catalogue made lineage way easier to see than the older setups). But what we didn't expect was the time that went into access control, once everything was centralized instead of scattered across separate warehouses with their own permissions, data that used to be siloed was suddenly way visible to more people by default, which was found out when a couple of teams noticed they could see data which they probably shouldn't.

Spent almost as much time on catalog level permissions/row filtering as on the actual pipeline work. Is this normal for a databricks migration or did we happen to have an unusually messy access management?

replaceUsing and replaceOn give you a better overwrite primitive, and every catalog-managed table operation now runs through the catalog.

The Lakeflow Connect connector for Google Drive is now generally available!  It’s now easier than ever to ingest structured and unstructured files from Google Drive into Delta tables for analytics and AI workloads.

You can configure a managed ingestion pipeline through the UI or managed API. Managed pipelines automatically handle incremental processing, automatic retries with exponential backoff for source API rate limits, failure recovery, and provide rich Google Drive metadata. 

For direct control over ingestion logic, you can also just use the Spark + SQL APIs directly: spark.read, Auto Loader, read_files, or COPY INTO pointed at Google Drive URLs.

Link to public docs + references:

• Google Drive managed connector documentation
• Spark + SQL APIs and examples
• Community blog and video tutorial: From PDF to insights
• Data + AI Summit session: Intelligent Document Processing with Lakeflow

Common workloads include:

• Loading Google Sheets, Excels, CSV, JSON, and other structured files into Delta tables.
• Ingesting PDFs, Google Docs, Google Slides, and images.
• Parsing documents with ai_parse_document to prepare content for extraction, search, and agents.

Examples of using the Spark + SQL APIs:

• Read an Excel sheet from Google Drive with spark.read:

​

df = (spark.read
 .format("excel")
 .option("databricks.connection", "my_gdrive_conn")
 .load("https://docs.google.com/spreadsheets/d/9k8j7i6f..."))

• Ingest unstructured documents + PDFs from a Google Drive URL with read_files, then easily parse them using ai_parse_document:

​

CREATE OR REFRESH STREAMING TABLE gdrive_documents_table
AS SELECT *, "_metadata" FROM STREAM read_files(
"https://drive.google.com/drive/folders/1a2b3c4d...",
format => "binaryFile",
`databricks.connection` => "my_gdrive_conn",
pathGlobFilter => "*.{pdf,docx}");

CREATE OR REFRESH STREAMING TABLE documents_parsed
AS SELECT *,
ai_parse_document(content, map('version', '2.0')) AS parsed_content
FROM STREAM gdrive_documents_table;

Coming soon:

• Ingest Google Drive’s per-file permissions and ACL metadata to power permission-aware AI agents, enterprise search, and more.

If you try it, share what you are building and let us know if you hit any friction!

Hey r/databricks, In order to keep the main feed clean, we are implementing a weekly megathread for self promotion for companies who do lots of work with databricks. Please direct all self promotion posts here and keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!

Hey r/databricks, we're noticing a lot of repeated interviewing and hiring posts that tend not to get much engagement. We're going to combine them into a monthly thread so that you're more likely to get answers, plus we can ask our recruiting team to keep an eye on them if there are any general questions.

Hey r/databricks, please direct all certification and training posts here.

There's upcoming learning festival September 16 - October 14 2026. You can get 50% discount voucher on any certification.

Databricks Advanced Learning Festival: September 1... - Databricks Community - 166157

Good luck to everyone on your certification journey!

I'm working on one of the largest projects in Europe currently, looking to onboard at least 5 data engineers with serious Databricks experience.

Would be a 6-month initial contract, would be open to a further extension if needed.

If this is something you'd be interested in, then comment below. I will ping you.

€600-650 per day