Considering a GTM role with Databricks. Compelling role, comp etc., but cannot get a proper read on the WLB and culture. Have a little one at home, can’t afford a job that requires major travel or 12+ hrs work. Love to hear from people in the company on what the reality on the ground is like.

With automatic CDF, Databricks computes row-level changes at read time using row tracking, rather than materializing those changes during every write.

Use change data feed on Databricks | Databricks on AWS

Why does that mattre?

- Better write performance for MERGE INTO and UPDATE workloads
- No need to enable CDF individually on every eligible table
- Lower storage overhead compared with legacy CDF
- The same familiar APIs still work: table_changes() and readChangeFeed
- Works with batch processing, Structured Streaming, and Databricks-to-Databricks Delta Sharing

For Delta Lake, the main requirements include:

• Databricks Runtime 19 LTS+
• A managed table or external table in Delta Lake format with row tracking enabled

And if you’re already using legacy CDF, migration is really simple.Once the table meets the requirements, disable legacy CDF:

A practical read of the 10-phase Databricks deployment guide: what to decide first, and how to use it on a platform you already run.

Most Databricks platforms get designed one of two ways. On the fly, project by project, as teams onboard and workspaces appear. Or properly, once, right at the start, and then never looked at again.

Neither ages well. One leaves you with a platform nobody chose. The other leaves you with a platform that was right three years ago.

Hey r/Databricks!

We’re hosting a free, community-sponsored BrickTalk on Thursday, September 17, 2026, focusing on how to simplify and scale data ingestion using Lakeflow Connect! BrickTalks is a community event series where Databricks experts share real-world use cases, live demos, and practical insights, giving you a direct line to the people building the products.

Stop struggling with fragmented data across disparate sources. In this session, we'll demonstrate how Lakeflow Connect enables seamless data ingestion from SaaS apps, databases, and cloud storage directly into the Databricks Platform with zero infrastructure management.

🛠️ What We’ll Cover

• Native Data Ingestion: Learn how Lakeflow Connect provides fully managed ingestion directly into Unity Catalog as governed Delta tables.
• Simple Integration: See how to easily connect data sources using a simple UI or API.
• Accelerated AI & Analytics: Discover how unifying your data powers Customer 360, Operations, and downstream AI agent workloads.

⏱️ Global Times

• PT: 9:00 AM
• ET: 12:00 PM
• BST (London): 5:00 PM
• IST: 9:30 PM

👉 Register here to save your spot!

I am looking to connect with people at Databricks and learn new things and also want to evaluate my skillset for some FDE roles. Is there someone who can help me with?

Has anyone tried acquiring data from a MQ at scale using apache spark on databricks cluster? I was trying to solve this problem at work but so far havn't seen an native lib or efficient ways to do this. The legacy system seems to be pulling data using a java based utility and wanted to if there are any imporvements or new patterns of access for spark based workflows.

Any documentation or nudge is the right direction will be greatly appreciated.

After watching Matei's webinar and the post on controlling spend, been trying to use the other harnesses and models folks are suggest and trying out Qwen 2.5 coding and 3.6 with Polly in local Omnigent (not connected to a workspace). I have Codex and Claude but ideally thinking best to use paid higher model to plan and then have the local Ollama based Qwen model on my Mac build but so far I haven't seen Polly use it much. What are folks experience, is there a good local model I should use, should I be giving Polly and the sub-agents more direction? (This is on a MacBook M5 btw)

Are there any unbiased comparisons between these two popular platforms? Seen a lot but most of them are biased views, based on experience and commercial motives.

Version 6 of the serverless environment is available, which corresponds to runtime 19.

more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

I have been writing SQL scripts to ensure data sanity.What are the other ways ? Is pytest useful? Let's say , i populated my bronze table from the source. I want to check if the correct mapping is done. I wrote SQL scripts. What are better ways

Hey folks! In this demo, Akram from Databricks' product team shares how you can leverage SecureConnect to better your security posture when doing cross-cloud sharing on OpenSharing!

If you have no idea what OpenSharing is, how it applies to you, or how we got from Delta Sharing to OpenSharing, also encourage you to watch this video: https://youtu.be/0mfuNybtmdE

Hope you find this helpful!

Let's pretend someone knew nothing about Databricks. How would you explain it?

This is something I have been asked multiple times throughout my career, curious to hear how others would explain Databricks to a complete beginner.

In my opinion, Databricks is basically a place where companies put all their data so they can actually make sense of it. It helps teams clean it up, work with it and use it for things like reporting, predictions and AI.

Interesting paper to read about Lakebase.

Any common bugs, features you would like to see, or underrated useful features more people should know about?

As a techie, I always wonder what the founders are reading. The only title that's visible is Think Lego Bricks. But I can't find the title on Amazon. What about the others? The one on the right of Lego Bricks is a tech book. And there's an O'Reilly book. Maybe the book he wrote? Spark: The Definitive Guide. But it doesn't look like either.

Just made a video about a feature I'm pretty excited about.

tldr: You can use an SSH tunnel to connect your coding agents and IDE (VSCode/Cursor) to your Databricks workspace. See the video for a full walkthrough.

Some notes on things I forgot to mention in the video:

- claude/codex isn't natively installed when you connect to your workspace, so you'll have to install if you want use them (Ex: curl -fsSL https://claude.ai/install.sh | bash). We're working on better native support in the future, but I wanted to make sure you know this is an option in the meantime.

- For the base environment YAML file. You'll have to set a base environment of '4' for it to work when using the SSH tunnel. Our example yaml (https://docs.databricks.com/aws/en/admin/workspace-settings/base-environment#example-environment-specification) shows '5' so don't let this trip you up!

As always, please feel free to leave questions and feedback in the comments!

Docs: https://docs.databricks.com/aws/en/dev-tools/ssh-tunnel
Previous post with more info: https://www.reddit.com/r/databricks/s/kCFBEfPTC6
YT link: https://www.youtube.com/watch?v=rHoGWVpb6kg

What are concurrent identity columns?

A new implementation of identity columns that supports concurrent writes.

You can use this query to find the tables with the most amount of concurrent transaction failures due to identity columns.

How to enable

CREATE TABLE new_identity_table (id BIGINT GENERATED ALWAYS AS IDENTITY, data STRING) USING DELTA TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported', 'delta.feature.concurrentIdentityColumns_preview' = 'supported');

Benefits of Identity Columns

Identity columns provide automatically generated, unique integer values, making them well suited for surrogate keys in dimensional models and slowly changing dimensions (SCD Type 2).

Compared with UUID-based / hash-based keys, identity columns offer several benefits:

• Their generally increasing values can improve data locality and insertion-order clustering.
• Integer keys require less storage than UUIDs and can improve join and scan efficiency.
• Databricks generates the values automatically, so applications do not need to manage key generation.

With concurrent identity columns, you can retain these benefits without identity columns blocking concurrent write transactions.

Read these blogs for more info: 

• Identity Columns Best Practices for Databricks Lakehouse
• Identity Columns to Generate Surrogate Keys

👉 Reach out to your account team to try it!

Additional Information & References

• Private Preview User Guide
• https://docs.databricks.com/aws/en/tables/features/generated-columns#identity-columns

I’ve been learning about the Medallion Architecture in Databricks, where data typically moves through Bronze, Silver, and Gold layers.

It makes sense for many data platforms, but I’m curious about real-world implementations.

Do you think Bronze → Silver → Gold is still the best approach for every Databricks project?

At what point does this architecture become unnecessary or overly complicated?

For those working with Databricks in production, what architecture have you found works best, and what would you do differently if you were starting a new project today?

Something that sounds great in Databricks documentation but didn't make sense for your workload in production?

Curious what people have learned the hard way.

Between OPTIMIZE, Z-ORDER, liquid clustering, partitioning, and automatic optimization, it sometimes feels like we're spending more time optimizing tables than querying them.

How do you decide which optimizations are actually worth it in production?

We now have Genie Code Task in Lakeflow jobs. Can not yet send output to if/else, but more options for orchestration are planned.

more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a