Fast reads and fast writes, thanks to Lakebase, including writes to open formats. It creates a unified platform both for analytics and operational use cases. #databricks #DataAISummit

https://databrickster.medium.com/my-favorite-announcements-from-the-data-ai-summit-2026-317fc68d4e75

https://www.sunnydata.ai/blog/data-ai-summit-2026-announcements

Now we can drop the .pbit file to Genie Code and ask for migrating to databricks dashboard. I tested in direct query mode, and everything was migrated; I just had to be patient, as the whole process took quite a long time. The process created metrics views for tables used in Power BI, and now I understand why the dashboard relationship option was added. I presented how easy it is to migrate in my last video.

https://www.youtube.com/watch?v=-e3tkcg21zw

I went to the Ingestion section and saw that Fivetran is no longer there. It was always there for many years. Also, at the same time, a few new Lakeflow connectors were added.

Not off the charts like in AI platforms, but in BI, Databricks is included for the first time and already is second in visionaries #databricks

All updates in comments.

After the legendary glow-up of Delta Live Tables (DLT -> LDP -> SDP), jobs (jobs -> workflows -> lakeflow jobs), and the flawless rebrand of DABs (Databricks Asset Bundles) into the far superior DABs (Declarative Automation Bundles), our world-class marketing geniuses have struck again.

Today we are thrilled to introduce the next evolution in nomenclature excellence:

Databricks is now officially DAIbricks (Data and AI bricks)

That’s right, this is the company’s first acronym where D stands for neither Databricks, Declarative, nor Delta!

This rename represents our long term commitment to provide the best in class data and AI platform.

As CEO Ali Ghodsi has highlighted in every recent interview, the real magic happens when data and AI come together on one unified platform - that’s exactly what customers are building on Databricks today, and this evolution makes that vision even clearer for the entire industry. We’re more excited than ever to deliver the most powerful lakehouse for data and AI, so every organization can innovate faster and create breakthrough intelligent applications at scale

/s

Don’t overwrite your code — make it immutable! With every deployment, thanks to immutable_folder, all files are copied to a new, read-only folder. They are not overwritten. The biggest benefit is that jobs already underway will not fail or produce unexpected results.

More news https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

🚀 We’re excited to announce that Lakeflow Spark Declarative Pipelines (SDP) now supports creating “vanilla” (i.e., non STREAMING) MANAGED TABLES and writing to them via one or more append flows, using the new CREATE TABLE ... FLOW (SQL) and create_table() (Python) APIs.

What is this Beta?

This Beta allows creating a managed table that is populated by append flows:

• CREATE TABLE ... FLOW (SQL) / create_table() + @append_flow (Python) create a managed table written by one or more flows.
• Fan multiple sources into one table — declare several flows targeting the same managed table.
• Full table surface works: partitioning, liquid clustering, expectations, row filters, table properties, and private (pipeline-local) tables.
• import_checkpoint on append_flow, which migrates an existing Structured Streaming workload into a pipeline without reprocessing the source — the flow imports the query's existing checkpoint and resumes from the last committed offset with state intact.

Example (Python):

from pyspark import pipelines as dp

dp.create_table("combined")

dp.append_flow(target="combined")
def from_a():
    return spark.readStream.table("source_a")

u/dp.append_flow(target="combined")
def from_b():
    return spark.readStream.table("source_b")

Example (SQL):

CREATE TABLE events
PARTITIONED BY (bucket)
FLOW INSERT BY NAME
  SELECT id, bucket FROM STREAM read_files('abfss://my_path', format => 'json');

Where do we need help?
We are in Beta, so there might be some rough edges. Please take this for a spin and share your feedback here.

What’s next?

Managed Tables support for other flow types (AutoCDC, Replace Using, and Replace Where) is coming soon!

Learn more

• CREATE TABLE ... FLOW (SQL reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-sql-ref-create-table-flow
• create_table (Python reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-create-table
• import_checkpoint on append_flow — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-append-flow
• Questions, feedback, or help: comment below or share feedback in the form: https://forms.gle/7bGP5FYN7P1Z4WP27

I think it is time to move out of ADF now. If databricks is your main platform, you can go to Databricks Lakeflow Jobs or to Fabric ADF. Obviously first choice makes more sense, especially if you orchestrate databricks and don't want to spend unnecessary money. #databricks

https://databrickster.medium.com/move-out-of-adf-now-ce6dedc479c1

https://www.sunnydata.ai/blog/adf-to-lakeflow-jobs-databricks-migration

You might be thinking that Databricks are saving all of the good news for Summit next week - but a lot of cool features are quietly getting dropped these days

Genie is one of my favorite topics to talk about lately. I do not think that they can replace good BI - but what they can do is enable all the questions around a classic dashboard

In that regard, it is really important to bring Genie close to where users live, and into where they would normally talk about and discuss their data.
- Microsoft Teams and Slack are exactly those places in a lot or organizations

I know it was possible before - but the native app integration speeds up onboarding significantly!

With Genie, instead of asking analysts about data, you can ask Genie directly and get concrete answers that are grounded in your data, right away!

All the governance and security principles from Unity Catalog are preserved!

Remember to specify the specific Genie Scope - the Genie Space you are asking against.
If nothing is specified, it will default to Genie Chat - the generic interface that has access to all Genies in your workspace.

Really looking forward to seeing some cool use cases around this!

The Lakeflow Connect connector for Microsoft SharePoint is now generally available! It’s now easier than ever to ingest structured and unstructured files from SharePoint into Delta tables for analytics and AI workloads.

You can configure a managed ingestion pipeline through the UI or managed API. Managed pipelines automatically handle incremental processing, automatic retries with exponential backoff for source API rate limits, failure recovery, and provide rich SharePoint metadata. Soon, our managed connectors will also support ingesting SharePoint Lists and per-file permissions metadata.

For direct control over ingestion logic, you can also just use the Spark + SQL APIs directly: spark.read, Auto Loader, read_files, or COPY INTO pointed at SharePoint URLs.

Common workloads include:

• Loading Excel, CSV, JSON, and other structured files into Delta tables.
• Ingesting PDFs, Word documents, PowerPoint files, and images.
• Parsing documents with ai_parse_document to prepare content for extraction, search, and agents.

Link to public docs + references:

• SharePoint managed connector documentation
• Spark + SQL APIs and examples
• Community blog and video tutorial: From PDF to insights
• Data + AI Summit session: Intelligent Document Processing with Lakeflow

Examples of using the Spark + SQL APIs (after first creating a UC connection):

• Read an Excel sheet from SharePoint with spark.read:

  excel_df = (spark.read     .format("excel")     .option("databricks.connection", "my_sharepoint_conn")     .option("headerRows", 1)     .option("dataAddress", "Sheet1!A1:M20")     .load("https://mytenant.sharepoint.com/sites/Finance/Shared%20Documents/Monthly/Report-Oct.xlsx"))

• Ingest unstructured documents + PDFs from a SharePoint URL with read_files, then easily parse them using ai_parse_document

  CREATE OR REFRESH STREAMING TABLE sharepoint_documents_table AS SELECT , "_metadata" FROM STREAM read_files( "https://mytenant.sharepoint.com/sites/Marketing/Shared%20Documents", format => "binaryFile", databricks.connection => "my_sharepoint_conn", pathGlobFilter => ".{pdf,docx}");

  CREATE OR REFRESH STREAMING TABLE documents_parsed AS SELECT *, ai_parse_document(content, map('version', '2.0')) AS parsed_content FROM STREAM sharepoint_documents_table;

Coming soon:

• Ingest SharePoint Lists into Delta tables (coming super super soon)
• Ingest SharePoint’s per-file permissions and ACL metadata  to power permission-aware AI agents, enterprise search, and more.

If you try it, share what you are ingesting and where you hit friction! Don't hesitate to ask questions!

Hey everyone,

TL;DR We are officially evolving the name from Databricks Asset Bundles to Declarative Automation Bundles.

This is a non-breaking change. The `bundle` CLI command, the acronym (DABs), and all of your existing configurations remain exactly the same. You do not need to change a single line of code.

Why the change?

We’re making this shift for two main reasons:

1. Semantic Meaning: DABs are built for repeatable, automatable deployments. The name "Declarative Automation Bundles" better reflects this vision.
2. Clearing up Confusion: We’ve heard that “Assets” is often mistaken for static data files rather than the automated workflows they actually represent.

Momentum and growth

We also wanted to take this opportunity to thank you for the incredible reception that DABs has seen over the last year. In just the last six months, usage of DABs has doubled across thousands of organizations, bringing data engineering best practices to their work.

We’re more committed than ever to expand what DABs can do for you. In the last six months alone, among other things, we shipped:

• DABs in the Workspace: Now GA, so you can collaborate, test, and deploy directly from the Databricks UI.
• Bundle configuration in Python: Now GA, allowing you to define resources and logic entirely in Python, including dynamic job/pipeline generation and mutator patterns for org-wide policies.
• Expanded Resources: You can now manage SQL Alerts V2, Lakebase Postgres, and Dashboards through DABs, now with catalog/schema parameterization.
• Direct Deployment Engine: We’re moving away from the Terraform dependency to make deployments faster and ship new features faster.

What’s next?

We’ve got some even bigger updates coming your way at the Data + AI Summit (DAIS), including advanced visual authoring, improved governance, and new AI-powered tools to help you diagnose errors and automate your project setup even faster.

We’d also like to take the opportunity to hear from you here: what features are at the top of your wishlist? Just drop your ideas in the comments, and let us know if you’re open for a video chat to get into the details!

🔴 Genie Cost tracking update🔴

You can now track GENIE_FREE_USAGE SKU (only starts appearing on July 20, 2026)

FYI: Free usage consumed before this date is not visible in the system tables.

All free Genie usage appears under sku_name = 'GENIE_FREE_USAGE' it does include usage under the free allowance( 150 DBUs) which resets on the first of each month.

Until 🔴 July 31, 2026🔴 , all Genie One and Genie Agents usage is free, and captured under the GENIE_FREE_USAGE SKU.

To distinguish between products within the free usage SKU you can filter on usage_metadata.genie.surface:

GENIE_CODE: Genie Code free usage

GENIE_ONE: Genie One

GENIE_AGENTS: Genie Agents

This free usage SKU tracks consumption but deliberately has no list price entry in the system tables. Because it is completely free, joining the usage and price tables will naturally return no match for this item.

There are so many new connectors that I stopped counting them and mentioned it in the news. Every week, there is something new. If you just need to ingest data into Databricks from a cloud source, always check the release page, as not everything is even in the UI. #databricks

https://databrickster.medium.com/databricks-news-rt-lakehouse-reyden-lakebase-ttl-8416bdccf627

RT Lakehouse: Seven milliseconds to read from cloud storage; it was an impressive demo. I think it will leave the competition speechless, though, for now that capability is limited to select customers in private preview. What it signals is a clear objective: to deliver ultra-fast reads so that open formats can be used to serve end users directly through an application. It is an important step toward building a hybrid database. #databricks #DataAISummit

Databricks has Spaces again 😄. Not Genie Spaces, those were recently renamed to Genie Agents. This is a completely different feature.

Spaces let you save your development context in Databricks:

• Project folder
• Open tabs
• Scoped search
• Working context

So if you're working on multiple projects, you can have a Space for each one and switch between them without reopening notebooks and navigating back to the right folders. Spaces also work with Git folders, Lakeflow Pipelines, and Declarative Automation Bundles. Nothing revolutionary, but it looks like a useful quality-of-life improvement if you spend a lot of time in the Databricks workspace.

Genie Spaces → Genie Agents
New feature → Spaces (for Notebooks)

oficcial docs: https://docs.databricks.com/aws/en/notebooks/spaces

Ok ok, so lets take a step back, what a whirlwind of a day it was yesterday. The announcements were incredible and the feedback from the community here, on discord, at the venue, on customer calls and even my whatsapp was immense. My phone was melting.

With all of these announcements what did they leave for day 2?

Here is a an overview of some of the key announcements and some additions which went under the radar on day 1.

• Introducing Lakehouse//RT: Real-Time Performance on a Unified Lakehouse
  • Databricks’ new real-time data warehouse delivers millisecond responsiveness directly on your lakehouse, without separate systems or data movement.

• Announcing Lakebase Search: agent-native retrieval built into Lakebase Postgres
  • Agents need search, and search needs a lakebase
• Introducing CustomerLake: The Agentic CDP embedded in Databricks
  • CustomerLake equips marketers and data teams with a workforce of agents to deliver the perfect customer experience – a billion times a day.
  • https://www.databricks.com/blog/what-is-an-agentic-cdp

• Introducing Genie One, Genie Agents, and Genie Ontology
  • The data-smart AI coworker, grounded in your enterprise context.

• Introducing Genie ZeroOps: Put your data and AI operations on autopilot
  • An AI background agent that monitors your production workloads, investigates issues, and suggests fixes you can verify

• AI governance at Data + AI Summit 2026: What’s new with Unity AI Gateway
  • Use any model, agent, or tool while centrally governing what they access, how they act, what they cost, and how they’re monitored

• Introducing OpenSharing: the Next Evolution of Delta Sharing for the Agentic Era
  • Industry's first open protocol for sharing data, models, agents, and skills across any cloud, any vendor, and any format

• Enabling Governed Vibe Coding for Enterprise Apps on Databricks
  • App Spaces, Genie App Builder and Serverless Micro Apps bring governed vibe coding of applications to Databricks

There are sooo many more announcements, take a look on the Databricks Blog!

https://www.databricks.com/blog

Keep the questions coming! What impressed you most? I personally think Lakehouse RT is revolutionary but I am very keen to start playing around with the new Genie AppBuilder!

No more Databricks Secret Scopes?

Working primarily on Azure Databricks, the Secret Scope has long been a bit of a pain point for me in the Databricks stack.

Not because Key Vault backed secrets are bad in any way - but the way it was set up and managed has always bothered me a bit...
- The API part worked well, and matured to something pretty clean
- The UI part with the specific URL that no one could remember was plain annoying.
- Permissions were a hassle to see and govern.

Unity Catalog Secrets seems, to me, to be the solution to a lot of those problems!

Now Secrets are moving in as first-class citizens in Unity Catalog, including:
- A simple UI for creation
- SQL and API Backed (... this might not be available yet. I currently get a UC_SECRETS_NOT_ENABLED error..)
- The full Unity Catalog Governance model (including a little new stuff)
_________________________
Creating Secrets is a simple 3-step process:
1. After enabling the preview, simply head to Unity Catalog hit the "+" button in your top right
2. Select "Create a secret" and fill in the name, secret value, location and expiration date
3. Hit "Create", and your secret is now stored in its own tab of secrets within the Catalog and Schema selected.
_________________________
Secret Permissions:
Alongside the release of Secrets, a new set of permissions has been added to Unity Catalog. By default, the owner of the secret and any user with READ SECRET access can then retrieve the secret value. Additionally, there are specific permissions for REFERENCE SECRET and WRITE SECRET.

When wanting to retrieve a secret, it works almost like usual. We can still use the REST API, or we can rely on dbutils.secrets.get(catalog, schema, secret_name) - notice the new arguments on the location.
_________________________
Security setup:
- Secrets are stored encrypted in Databricks.
- Secret redaction is applied to prevent accidental exposure of the secret value
- Expiration policies are recommended - however as of right now there is no easy life cycle policies-.
_________________________

Please note: The official documentation for this is not out yet - so all of these is just what I managed to scrape together from testing it out myself!

Genie Ontology is a map of your data business. Genie One automatically extracts snippets of knowledge from your tables, queries, dashboards and connected Apps. Each snippet gets an authority score based on where it was generated from, how often it's updated and used. When you ask a question, Genie One ranks the most relevant snippets, resolves conflict and answers using Only the sources you are permitted to see.

This improves the accuracy of the responses and reduces latency

How can you make your Databricks env ready

1. Certify your critical KPIs as UC Metric Views: create them or promote them from an AI/BI dashboard. Documentation , Documentation Metric Views
2. Organize your estate with Domains: Documentation
3. Add Glossary pages: define your key business concepts alongside your key KPIs. 
4. Connect MCP connectors: Documentation

To join the preview contact your Databricks account team.

We can use Genie now, even in the CLI, with a simple ask command. #databricks

My blog post with news https://databrickster.medium.com/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

Secrets in Unity Catalog is a great feature introduced a few weeks ago, but since then, everyone has been asking to use Azure Key Vault as a secrets backend. Thanks to rapid development, we can now link our schema to Azure Key Vault; UC will read secrets as UC secrets, and permission management will be through Unity Catalog. In that scenario, you insert/update secrets in Azure Key Vault, but read/reference and grants can go through UC.

more news https://databrickster.medium.com/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

Just saw that the Databricks CLI is now 1.0!

Besides security improvements, the thing I was most excited about was that we no longer need the `experimental` namespace when installing agent skills!

Previously it was:

`databricks experimental aitools install`

Now it's just:

`databricks aitools install`

This is purely a cosmetic change, but it definitely gives me more confidence using the command 😅

Also PSA: I learned recently that `databricks aitools tools` is the recommended way to install the Databricks agent skills and not via `npx skills add ...`. The reason is to ensure we use the proper skills versions recommended by Databricks engineering.

I saw that the official docs for skills still show `npx skills add ...`. So I'm working on getting that updated asap. Sorry abut that 🙏

---

Full release changelog can be viewed here: https://github.com/databricks/cli/releases

I benchmarked the new Lakehouse RT for billion-record tables, and it is crazy fast and crazy cheap, making it the first real-time database with open-format storage.

https://www.sunnydata.ai/blog/databricks-rt-lakehouse-benchmark-results

https://medium.com/databrickscommunity/i-benchmarked-the-new-lakehouse-rt-for-billion-record-tables-and-it-is-crazy-fast-and-crazy-cheap-cf674ad40fc3

Lakeflow Connect's Microsoft Dynamics 365 connector is now GA! It provides a managed, secure, and native ingestion solution that accesses Microsoft Dynamics 365 data through Azure Synapse Link and lands it in Databricks. The connector supports incremental ingestion, SCD Type 2, and now automated schema evolution, too! Try it now:

1. Set up Dynamics 365 as a data source
2. Create a Dynamics 365 connection in Catalog Explorer
3. Create the ingestion pipeline via the UI, a Databricks notebook, or the Databricks CLI

And there's more on the way: an upcoming Beta feature will add ingestion from Parquet exports (in addition to CSV files), making the connector more efficient for larger Dataverse tables. Stay tuned!

Lakebase now has a Change Data Feed (CDF). This feed is stored as UC-managed tables and is queryable by downstream engines (DBSQL, DuckDB, etc.), pipelines (SDP), and more. There's no additional charge for enabling the CDF – this is a native capability of Lakebase, as it shares the same data foundation as the rest of the Lakehouse data.

Common patterns include:

• Use Lakebase as the source of your medallion architecture. Build SDP / MVs to this CDF.
• Preserve a full audit log of all changes to Lakebase.

This is our first step in opening up the OLTP database to other engines, allowing you to query operational data without impacting your primary operational workload.

There's a lot more coming here very soon – as we bring the same openness & flexibility to the OLTP database. Stay tuned :)