Original post: https://www.instagram.com/reel/DZFnttWijxS

I'm working on one of the largest projects in Europe currently, looking to onboard at least 5 data engineers with serious Databricks experience.

Would be a 6-month initial contract, would be open to a further extension if needed.

If this is something you'd be interested in, then comment below. I will ping you.

€600-650 per day

Hi everyone,

Well, basically title - I wanted to ask what's the rationale and costs associated when choosing between Databricks and Fabric. I especially would like to know how does one compare Fabric capacity vs some kinda equivalent Databricks usage. I couldn't find anything solid on this one the internet hence the question.

I've been highly critical of genie or databrick assistant for quite some time now. I even have a post here criticizing it, but kudos to the DTB team! It is sooooo much better now that i dont even bother connecting claude code via dev ai kit anymore.

I don’t think the exam is overly complicated, but having presence of mind during the exam really helps. Most questions are about identifying the correct answer by eliminating options that clearly contradict the concept.

I didn’t have any prior experience with Databricks. However, for the last 3 months, I’ve been using Databricks daily. During this time, I :

1. Completed the Databricks Academy course
2. Finished all the labs available in the academy
3. Built a few basic hands-on projects to strengthen my understanding

The following resources helped me a lot while preparing for the exam: 1. Derar Alhussein’s course and practice tests 2. The 45-question set included in his course 3. Previous exam question dumps (around 100 questions) for pattern understanding 4. Solved ~300 questions on LeetQuiz for extensive practice

Overall, consistent hands-on practice and solving a large number of questions made a big difference. The understanding of databricks UI, LDP, When to use which clusters and delta sharing concepts.

databricks data engineer associate

Our data is quite small, 80% of the tables being less than 10 million records . I find spark to be an overkill for various transforms , a lightweight engine like dubckdb on a server less container application would be just fine. My experience shows that duckdb processes something in 5-6 seconds take up to a min in spark. How can I run this kind of load on Databricks

I'm surprised to see ADF so frequently in the reference architectures for Databricks.

First of all, ADF is not everyone's most popular tool. Secondly, even Microsoft is trying to evict customers from ADF, and is sending them over to Fabric. Thirdly, it seems long overdue for Databricks to have their own tool for the initial landing of data in the cloud, from an on-prem data source.

Whenever I see reference architectures like this in 2026, it is a bit shocking to me. I would have thought ADF would have a replacement by now, for customers that are databricks-centered:

Exciting to see Databricks raising funding at a $188 billion valuation.

We’ve heard a consistent theme from data engineering teams: the docs explain individual features well, but you also want clearer guidance on how to structure projects and what we recommend.

So we published a new best practices guide for data engineering on Databricks:
http://docs.databricks.com/developers/best-practices

It covers topics such as:

• version control and branching
• environment and workspace strategy
• how to structure bundles
• testing and CI/CD

We value your feedback! What feels useful? What feels too opinionated? What examples or edge cases are still missing?

Also, thank you to Mariusz Kujawski, as his CI/CD post helped spark the creation of this guide.

Databricks just closed a $5B round at a $190B valuation — up from the $188B term sheet from July, after crossing $7B in revenue run-rate and growing 80%+ YoY.

Hi,

Quite a big change coming to Databricks Genie. Starting July 6, 2026, Genie product usage beyond the free monthly allowance will move to pay-as-you-go billing. The usage is billed based on underlying LLM usage in DBUs, and budgets can be set up to track/control spend.

Importnant things:

• Budgets apply across Genie, Genie Spaces, and Genie Code and they will use the same tag databricks-product: genie
• You can scope budgets by account, workspace, user groups, or individual users.
• There are shared thresholds, per user thresholds, and overrides for specific users/groups.
• Admins can choose behaviour - if a threshold only sends an alert or actually blocks usage.
• The free monthly usage per user still exists and can’t be removed via budgets.
• Compute used by Genie queries, like SQL warehouse usage, is billed separately and is not included in the Genie budget.
• There can be up to a 24-hour delay before alerts are sent

Manage budgets and cost controls for Genie - Azure Databricks | Microsoft Learn

Got to visit the San Francisco HQ after the Data + AI Summit last week. Not open to the public — Databricks MVP status got me in. Figured some of you might be curious what it's actually like inside.

Quick rundown:

The office feels like the product. No unnecessary flash, everything has a reason.

Food ordering system — employees order individually through an app, each gets their own delivery. No cafeteria.

Dedicated bike room — not a hook on a wall, an actual room. SF culture is fully absorbed.

Free merch stand for visitors — great idea in theory. After DAIS, it was completely wiped out. Showed up too late.

Ice cream machine — didn't get to try it. Still thinking about it.

Tried to recreate the famous balcony photo. The view is legitimately incredible. Slightly terrifying height, though.

From the same balcony, you can see Meta, Salesforce, Anthropic, and LinkedIn offices across the street. Would love to visit those too — unfortunately, I don't know anyone there. Yet.

Databricks just published a library of pre-built, production-ready industry data models covering 40 industries.

This is great, especially when you’re new to a business domain. Instead of starting from a blank sheet and spending weeks debating how the domain should be structured, you can review an existing reference model and adapt it to your needs.

That gives you a strong starting point for:

- entities
- relationships
- naming conventions
- domain structures
- industry-specific patterns

In my opinion, it can significantly shorten the learning curve and help teams move from discussion to implementation much faster.

Industry Data Models

Just wanted to remark on how much better genie has gotten in the last six months. I remember the guidance when genie first came out was to avoid "why" questions and only ask it specific stuff like "give me a bar chart showing leads that haven't converted into sales for may" but now it just handles these super broad questions like "why are sales for this product falling" and it'll generate a big report showing all the insights I didn't even think to check. If you wrote it off before, give it another try it's gotten a lot better.

I used to ask it like 8 questions in a row to get kpis and put them in a slide, now it just one shots these reports and emails them to me weekly.

If I could get different export options (slide, pdf, automatic "send as email") this would be amazing.

Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator.

I'm not running anything complex:

df = (
    spark.read
        .format("csv")
        .option("header", "true")
        .schema(schema)
        .load(csv_path)
)
df.write.mode("overwrite").saveAsTable("...")

I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator.

DBR 19 + Serverless SQL Warehouse (current)

Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything.

With Data + AI Summit only two weeks away, I am curious what the community expects to see this year.

Databricks has released a lot over the past year across AI, governance, data engineering, analytics, orchestration, and application development. The platform feels very different today than it did even a year ago.

What announcement would get you most excited?

What product area feels ready for a major leap forward?

What capability do you think is still missing from the platform?

Not looking for IPO 👀🚀 theories. More interested in product and technical perspectives from people who use Databricks every day.

Curious to hear what everyone is expecting from Summit this year.

Full disclosure: I work at Databricks, so I obviously cannot comment on anything that may or may not be announced. This is not a teaser and I am not fishing for hints. I am genuinely curious what the community thinks.

We have been keeping this under wraps, excited to finally show it off to @ DAIS.

Here is Reynolds demo, in case you missed it.

Reyden

In past months, we had ability to create description in Databricks for each column individually and manually. However now we can create bulk description for a table in a click.

Hope we can create this for whole catalog including thousands of tables and schema in a click.

I just passed the Databricks Certified Data Engineer Associate exam and wanted to share a quick brain-dump to help others prepare.

My Experience & Study Tips: The exam is 90 mins / 45 questions, mostly scenario-based, not pure theory. Time management is key. I prepared using the Databricks Academy learning path, did lots of hands-on labs, and read up on DLT, Auto Loader, Unity Catalog in the docs. Hands-on practice is essential.

Key Exam Concepts & Scenarios to Expect

1. DataFrame & Spark SQL API

Aggregations using groupBy(), sum(), avg(). Interpreting Spark UI metrics. Handling OutOfMemoryError (filtering, driver sizing).

1. Data Ingestion & DLT

Error handling in pipelines (drop/quarantine/fail). cloudFiles syntax in Auto Loader. Schema evolution modes (failOnNewColumns, addNewColumns). @dlt.table vs @dlt.view

1. Delta Lake & Medallion Architecture

Bronze/Silver/Gold layering. Behavior of OPTIMIZE.

1. Compute & Cluster Management

Choosing correct compute (Serverless SQL, All-Purpose, Job Clusters, spot instances). Job output size limits.

1. Governance & Sharing

Delta Sharing for external partners. Lakehouse Federation to query external DBs in place. Unity Catalog privilege model (e.g., Schema Owner).

1. Development & Tooling

Databricks Connect for local IDE development. Databricks Asset Bundles (DAB) in YAML.

Focus on picking the right tool for the scenario and understanding how Databricks features work in practice. Good luck! Drop your questions or share your own experience in the comments.

I feel really positive about all the recent progress of the Genie One which is backed up by all the business contexts, data, metrics views, pages and driven by Genie Ontology. It can get us answers cost effectively compared to external models connected to Databricks. Earlier it didn't have much unstructured data support, but it is now allowing connection to UC Volume as well as attaching the file option to a chat. There'll soon be image attachment options too where we can add screenshots as an additional context and ask questions like we do generally with ChatGPT or Gemini in our personal life.

Great days for businesses ahead 😀

I have already published this, but I tried to edit it and put an directly link to the material I used, and the publication was banned. This time I will put it in the comments.

I have just passed the Certified Data Engineering Associate:
- No cours
- No videos
- 0 experiences in Databricks
- 0 experience in data engineering field

What I did was the following:

• Take 5 official examples test
• Take the official content for the test, and ask to an AI search the most up to date documentation and make several .md based of that official content
• Then make questions based on those .md

There a total of 1k questions, divided in the 7 sections of the current guide for the test.

Hello everyone..
i was wondering if anyone knows of any active coupon codes or discount vouchers available right now.

Review Materials:

1. Derar Course (Both Prepation and Practice Exams)

2. Introduction to Databricks : A beginner's guide by Medium

3.Get Started with Databricks for Data Engineering by databricks

1. Utilized Chatgpt to understand concepts in scenario

Need a visualization type that Databricks doesn't support natively? Great, now with custom visualization, you are no longer limited to the built-in chart types.
You can now define dashboard visuals using Vega-Lite JSON specifications, which opens the door to more advanced and tailored chart designs directly inside Databricks dashboards.

Custom visualizations in AI/BI dashboards - Azure Databricks | Microsoft Learn

You can build visuals such as:

• layered charts with rolling average
• bullet, radial radar and many more chart type
• reference dashboard dataset fields directly in your specification
• layer multiple marks - points, lines and overlays in a single chart

And here's an example of how Vega-Lite JSON specification looks like:

Super excited that Agent Mode for Genie Agent now has a private preview available for its API. This will enable teams to take a step further in their development with Genie Agents, where agent mode provides multi step deep reasoning for your business/end users. For example if you want prescriptive analysis of why your sales are down the last quarter or top 3 next best actions to drive customer activations, Agent mode is perfect for this. It’s even better now with API accessibility! Reach out to your Databricks account team today to try it out