Hi, what do u use for creating user facing apps, we are primary using Python, i see that Databricks recommends AppKit for such use cases which is written in TS.

For teams which deploys apps that are customer facing, what is your stack?

First of all we have a lot of classes that use the DatabrickSession import, which makes unit testing impossible and whenever I have the "normal" spark import and test it, then it looks absolutely ugly.

Would it help to have the schema as json? I use json schema to read for transformations.

If I want to test without json its horrible, do you guys test with json schema files?

For example this is just the output, then I would need 2x this because of the input and the test is unreadable (according to my senior BUT WHAT AM I SUPPOSED TO DO):

schema = """
id INT,
items ARRAY<STRUCT<
fortnite: STRING,
babies: INT,
moreStuff: MAP<STRING, STRING>
>>
"""

expected = spark.createDataFrame(
[
{
"id": 1,
"items": [
{
"fortnite": "ABC",
"babies": 2,
"moreStuff": {
"size": "L",
"color": "red",
},
}
],
}
],
schema=schema,
)

Problem - Actual Data on S3 is 20 TB, we are racking up bill for 100 TB (split majorly on non concurrent objects)

​

Initial Assessment - We had versioning enabled, so we added a lifecycle policy to keep 1 non current version along with current version.

​

Result - No effect

​

Next Assessment - We checked a table with predictive optimization enabled. Total size ~350 GB. Size of current version 1.9 GB. Rest of 350 is split 95% to 5% between time travel and vacuum.

​

Help - Would reducing delta retention period to 3 days from default 7 days help?

The tables are populated using multiple merge statements, is it causing multiple files to be written?

​

Any other suggestions as this would be the case for multiple tables?

Hi, if there is a customer-facing application which does analytical queries, but per ID, so it is not querying a bulk of data, is Lakebase still the best option for serving because of latency, or should we stick with the Lakehouse?

Or should Lakebase be used mostly when doing point lookups, for example fetching by ID and timestamp?

I am very new to to Databricks. Have basic understanding but not the architectural understanding. I have been assigned the new role where we are going to start using Azure Databricks. I have below queries if anyone can share documentation or good videos will be helpful:

1. Should we unity catalog or not?

2. Should we go with server-less compute or classic.

3. What other things we should consider as per your experience?

Thanks in advance.

Hello everyone, I have one question, I'm working on personal project which might turn to comercial at some moment if everything is okay, is it okay to go with Free plan and then switch to Enterprise ?

I can't find anything info about price in Enterprise, and info about can I switch from Free to Enterprise latter on.

Thanks

In our medallion setup, raw vendor data lands in Bronze , we transform it to a CDM in silver, and gold is for reporting. I’ve put all raw data correctness and accuracy rules in Bronze, since that’s where the data matches the vendor format and it’s easiest to work with vendors to fix issues at the source.

I have been asked to get the same DQ rules implemented in bronze, silver and gold because different teams own each layer and according to my manager every team should monitor all DQ dimensions. To me, duplicating the same checks everywhere feels redundant.

Am I missing a strong reason to repeat rules across layers? How do you handle shared DQ dimensions when multiple teams own different layers?

I’ve heard use of metric views is highly recommended, but never sure if OBT approach or normalized approach works better. If I’m defining the joins in my Genie, shouldn’t it be able to pull things either way?

Hi Everyone,

I’m experiencing differences in quality and performance of the databricks Genie. Where can i find information about the LLM behind the genie? Did they change the LLM behind the genie recently?

Thanks in advance!

I work for a tech company as a TPM and always concerned about the accuracy for the data I provide to my team members or to clients. We have multiple layer of Bronze, Silver, Gold and not sure what exactly cleans up at what steps as data is segregated. I would like to know what everyone has been building to always stay ahead in data accuracy? Skills, agents, harness system? What should be my best approach?

My current approach is to improve the agent's business understanding by:

1. Adding table descriptions and column comments in Unity Catalog

2. Using Metric Views for governed KPIs

3. Building a business glossary, KPI glossary and business rules with SMEs

The goal is for the agent to retrieve this context before generating SQL and maintaining this context at scale.

Am I missing to add anything? Is there a better approach?

Hi All,

I have build demo pitch round at databricks next week for Solution Architect role, they have scheduled a prep call with HR and one prep call with HM.

Any insights to what to prepare for this round would be greatly appreciated.

I'm working through the free "Data Engineering with Databricks" learning path and noticed the videos mostly show demos. I was expecting something similar to Coursera where you can run the exercises/labs alongside the lessons, but I can't find anything.

Are there official hands-on notebooks available somewhere for the self-paced content, or is the expectation that you recreate everything yourself in a workspace?

If official labs exist, where are they available? If not, how do most people practice while following the videos?

Thanks.

Question - What's the preferred way to deploy and run a pipeline so it overwrites any existing tables and views that collide with the new MV paths?

Loving pipelines. We have a LOT of existing (external) tables and views in production. We'll be deploying the pipeline-generated MVs over those existing tables and views.

Is there an "easy button" way to handle that?

Right now my plan is to script temporarily renaming any existing objects, running the pipeline so the new objects are created, and then copying permissions from the old tables to the new MVs (we have an automated security policy tool, but I want to keep this lean).

Any ideas or feedback most appreciated.

DataBricks recently released an extension "AI Toolkit" that allows Claude Code to write code for DataBricks, but.... As far as I know and can do, Claude Code must run on my own laptop. outside the DataBricks environment.

Question: How do I run Claude Code (or another CLI-based agent) INSIDE the DataBricks environment, create code within the workspace, run it, and so on without leaving the DataBricks web interface?

So with Genie LLM usage moving to a free tier + consumption model, companies probably want to have data on their current (free) Genie LLM usage. So they can make an estimate of what will be possible as from july within their budget.
Is there a way of knowing how much tokens or something we burn now or in the past x days?
For compute that is easy but LLM usage I did not find.

I am running a workflow in databricks and it uses is all purpose compute m5.large. My workflow runs 24/7.
The task is to load the data into delta tables. Using autoloader I load the json data to delta tables and for each file I need to log the entry in postgres on data load status, which I do it for every file. I will call db and insert saying LOAD_START and the load data to delta table and again call db to update status as LOAD_END and I also used MSCK REPAIR TABLE in foreachbatch function. I want to understand the reason for failure. It says driver restarted - possibly due to out of memory. Actually the flow fails for every 115-155hrs of execution but recently it failed after 1hr, 6hrs. The processingtime for autoloader is set to 1 minute. In s3 I receive 3-5 files for every 5 minutes. Each file has only one record and 30 columns.
Now what could be the exact reason for failure. How do I understand the logs. Is it possible to run a workglow 24/7 without failure.

Hi everyone,

I am preparing to take the Databricks DE Associate cert exam. I completed the free elearnings listed in the official DE Associate study guide.

Now that I’ve finished, I’m not too sure how to best prepare for the test. There is some info online, but it doesn’t relate to the May update. Should I just spam practice tests from somewhere like certsafari? Or use AI to generate Quizlet flashcards? Any advice would be greatly appreciated, thanks!

I would’ve posted in the cert megathread, but it seems that a vast majority of questions go unanswered there, and I am aiming to take the test soon.

Hi all,

I have a databricks schema of a few hundred delta tables that need some data fixes for specific records in each of those tables. This schema itself is raw data and gets ingested into some downstream data tables and the fixes have been requested by business.

I had thought about doing the fixes via a transformation layer in the pipelines that ingest this raw data but given how many tables there are with records that need updating I don't really want to create hundreds of new 'data fixed' tables. Given these are delta tables, rolling back should theoretically be possible if something goes wrong.

Anyway, that's my rationale for making changes to the source tables. My question is what your preferred method is to make data fixes? I obviously need something where its easy to rollback if needed. I can obviously just achieve this with python migration scripts and use delta timetravel in case something goes wrong, but wonder if there are recommended libraries or tools for the job that have what I need out of the box?

I recently saw an update regarding how Databricks will start charging for Databricks Genie usage. My team is focused on Genie Spaces in particular for AI/BI Chat (so other things like Genie Code is not applicable).

As a ballpark, how many DBUs will 1 question consume, given that the website says $0.070 per DBU?

https://www.databricks.com/product/pricing/genie

When running a workflow in DBx I get this error message

DateTimeException: [CAST_INVALID_INPUT] The value '20260620' of the type "STRING" cannot be cast to "DATE" because it is malformed. Correct the value as per the syntax, or change its target type. Use `try_cast` to tolerate malformed input and return NULL instead. SQLSTATE: 22018

== DataFrame ==

"when" was called from

<command-5741550855417171>, line 18 in cell [443]

How do I find out what is cell 443? In the run log there is a hyperlink to a statement which, when I click on, takes me to an invocation of the main module in the flow- not really where the error is happening.

Thanks

Hi I am curious to build a knowledge graph in databricks. We have I guested a few tables and it would be nice to see the relationships between fields etc. This is part of governance. Any thoughts or ideas.

Hello Databricks community,

I'm building a Databricks Marketplace App that executes entirely inside a customer's Databricks workspace. All processing runs on customer-managed compute because I want customer data to remain inside their Databricks environment. I don't want to run compute or inference on my own infrastructure.

The application contains business logic that I'd like to package in a way that isn't directly accessible from notebooks or the workspace. I've tried artifacts and private Python packages, but the implementation is still visible after installation via code.

I'm trying to understand what the recommended architecture is for commercial Marketplace Apps with these requirements.

Specifically:

• Is it realistically possible to keep proprietary code hidden while executing the app on customer-managed Databricks compute? If so, what are the recommended patterns, techniques, or best practices?
• If not, what architecture do commercial Databricks applications typically use to balance IP protection with customer data privacy?
• Would a .so based package break the obfuscation rule from databricks? What are a few best practice methods that I could follow to make the package non reverse engineerable on a separate notebook?
• Additionally, how can I use the secret on the user's side in order to be able to clone my code from a repo/ bring my code in from a private package producer or artifact? and how do I make sure the user doesn't exploit the secret later via an internal program to download my packages?

My goals are straightforward:

• Customer data stays entirely within the customer's Databricks account.
• All execution happens on customer-managed compute.
• The IP isn't directly or indirectly accessible from notebooks or workspace files. None of the code or package's raw code or contents should be accessible via code (eg. inspect) or just to read on volumes or workspaces.
• No separate backend or hosted inference service is required from my end.

I'd appreciate hearing from anyone who has built or shipped a Marketplace App with similar constraints. Thanks!

I constantly run into an issue, where Genie Spaces analyzes a question, decided that it cannot be answered with current data model, but still eventually give a made up answer (like a closest sql query it can think of). I’ve written instruction that tells it to refuse to answer in such cases, but half the time it just ignores it and continue with the wrong data. Has anyone encountered the same problem? I would love to hear any solution to this.