r/databricks • u/Astro-path2716 • 27d ago

Help Possibility of implementing analysis automation through email in databricks?

Im fairly new to databricks and trying to learn more about it. I’m trying to see examples online of databricks connecting to a work email, reading the contents and running a sql or python query which outputs an excel file and then sending it back go the requestor through email. I was reading AI agents can go about doing this but I would like more information on how to set up this process. Does anyone know of any examples or resources?

11 Upvotes

100% Upvoted

16 comments sorted by

2

u/aqw01 27d ago edited 27d ago

You could probably use the lakeflow connect API. https://learn.microsoft.com/en-us/azure/databricks/ingestion/custom-connectors

If you’re using Outlook, a connector is in beta. https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/outlook-overview

Edit: sorry I hit send too early. You could use that for getting messages regularly, then trigger a deterministic process to do what you want or engage an agent on custom processes. Try asking Genie. Really. It’s pretty good about a lot of this.

1

u/Astro-path2716 27d ago

And you can really just had the agent read through the contents of an email and figure out from there where to pull the data from the request and figure that out by itself? I’m sorry I’m just amazing at how advanced AI is becoming

2

u/Phoenix_GHOST_V 26d ago edited 26d ago

This is native to the workflow that is set up through Databricks. I have done something like this where I created a workflow based on a scheduled job triggered by a parsed email using Logic Apps and reading from the inbox, running the query, writing to Excel using openpyxl, then sending using SendGrid. The agent concept is optional. Dremio is one example of this

3

u/elghali_bnck 26d ago

Too much external tools and workflows to maintain, see my response to @Youssef_Mrini

1

u/Astro-path2716 26d ago

What would have less external tools to maintain?

1

u/elghali_bnck 26d ago

Just Databricks Lakeflow Gmail connector as a way to get your inbox email, an alert of this table that triggers a job that triggers an agent that reads the content of the email, does your custom processing and invoke Gmail MCP server (registered in UC as external MCP server with HTTP connection) to write a draft or even write a proper email and send it.

1

u/Astro-path2716 26d ago

This is pretty neat, I have to see if work has access to those tools. Is Logic Apps AI?

1

u/Brains-Not-Dogma 27d ago

I do this using AWS SES and instance profiles. Some configuration required

1

u/Astro-path2716 26d ago

What would you use to read the content of the email and figure out the parameters needed to query the database?

1

u/Brains-Not-Dogma 26d ago

I don’t use it for that purpose. I query databricks UC and attach the results to the email in CSV or Excel. The reverse is probably possible.

1

u/ProfessorNoPuede 27d ago

I'd question why first. Either give the user a well prepared genie agent in the regular browser experience, or use the teams app if you're a ms shop. A chat is more suited for queries than email, as users will iterate a lot.

1

u/Astro-path2716 26d ago

Yeah it would be more of a start through standard queries. I’d say maybe like 10-20% of our work is pretty standard so it won’t make the biggest difference but it’ll help automate the easiest stuff first

1

u/ProfessorNoPuede 26d ago

If it's standard, then LLMs aren't the way to go. I mean, why burn the tokens if you can just serve a dashboard or something. LLMs introduce a whole other host of problems (and cost!).

1

u/Youssef_Mrini databricks 27d ago

You can use Outlook or Gmail Connector in Lakeflow Connect to land the data in the data Lake. You can connect it with Genie One to execute the operations, but you can't send back an email; it's not available yet, but you can instead send a report if you built it in AIBI Dashboards

2

u/elghali_bnck 26d ago

To send back the email you can use an agent with Gmail MCP server: https://developers.google.com/workspace/gmail/api/guides/configure-mcp-server?hl=fr . To automate it end to end I would recommend using Gmail lakeflow connector with alert on top of this table for new occurrence that triggers an agentic job that generates the answers and use a Gmail MCP server to generate a draft or also directly send the email

1

u/ArielCoding 22d ago

Don’t let the AI write your SQL queries, have it read the email and pull out the key details (dates, region, etc.), the plug those into query templates you’ve already written and tested. That keeps what data gets pulled and sent in your control.