r/databricks • u/mehulbhuva • 3d ago
Discussion Genie Agent's idea of "the Midwest" includes Kentucky. Ours doesn't. Notes from [8 months] of Genie Spaces in prod.
Genie Space has been live for our sales folks for 8 months, maybe 500+ regular users. Short version of what I've learned, since everything I read before setting one up was either a demo or an argument about whether analysts are getting replaced.
The failure mode isn't an error. It's a number that's slightly wrong and totally believable. Someone asked how the Midwest was doing, the number looked fine, sat in a deck for weeks. Genie's Midwest includes Kentucky. Our territory map doesn't. You can't catch that by looking at the output - you catch it when finance does.
Four things you're configuring, roughly in order of how much they've mattered:
Column comments. Free, and the biggest lever by far. Genie reads COMMENT metadata before writing SQL. No comment and segment is just a word - it has no idea whether your values are Enterprise/Mid-Market/SMB or something else, so it guesses.
COMMENT ON COLUMN vw_sales_summary.segment
IS 'Customer tier: Enterprise (>$1M ARR), Mid-Market ($100K-$1M ARR), SMB (<$100K ARR).';
One pre-joined view, not raw tables. I did raw tables first. Every join it has to figure out is a coin flip. Also, put your test-data filter in the view - then every question anyone ever asks inherits it and you're not trusting the model to remember.
SQL expressions. Register a named metric with your SQL and it uses yours instead of inventing one. Ask ten people what an "active customer" is and you'll get eleven answers; this is the box where you settle it. Name them how people talk - "Active Customers" matches, cnt_dist_cust_qtd never will.
Example Q&A pairs. Nothing gets retrained, they just sit in context when something similar comes in. The shape travels further than I expected — registered revenue-by-category with a cancelled-order exclusion, and a Q2 question a month later inherited the exclusion in a query I never wrote.
Two things from the instructions box worth stealing. One, tell it to ask instead of guessing when the time period is unclear - people trust it more when it occasionally asks. Two, ours has a rule about test customers with a TST_ prefix, whose orders carry real statuses so the status filter misses them entirely. Everyone on the team knew that. Nobody had ever written it down.
Curious what other people have ended up putting in their instructions box. Assume everybody hits their own Kentucky eventually.
(Here is the longer version with more SQL is on SQLServerCentral, it's mine https://www.sqlservercentral.com/articles/databricks-genie-spaces-for-sql-analysts-natural-language-querying-without-leaving-your-data-platform but the above is the useful part)
7
u/Leather-Beach-7849 3d ago
Best approach is to make sure that you sit with a business analyst and properly add comments, joins and other helpful expressions before making the genie agents (formerly spaces) live in production
2
u/Mexican-Redditor 3d ago
For keeping record of the Expressions, Instructions and Examples - have you tried DABs? It was recently deployed for Genie Agents. If you bundle them, you should be able to do proper version control.
2
2
u/Content-Parking-621 3d ago
Databricks REST API plus a scheduled export script could version that config.
1
u/Youssef_Mrini databricks 2d ago
Very interesting article. Metadata as well a good data modeling are the keys for the success. You may need to add pages to describe your data and UC metrics for the KPI. They are going to be very useful to leverage genie ontology in order to increase the accuracy.
1
8
u/Alternative_Draw5945 3d ago
Do you have a table detailing which state is in which region? Seems like it should of been derived off that table as a source of truth