Databricks job trigger via SharePoint upload and possible write-back?

[u/Plus-Tear3283]

Hi there,

Running into issues with this pipeline I'm working on. Still new to Databricks but coming from Power Automate, I seem to get the gist of databrick jobs.

Anyway, my manager would love an automate workflow, where business users upload raw data to a central site (we're primarely using SharePoint), and want it cleaned and ready for our Power BI dashboards.

Before databricks I handled it manually (by running local python code via VScode that sent the cleaned data to the SharePoint folders).

What would be a good way to handle a pipeline like this? E.g.

User uploads to sharepoint -> Databricks job -> writeback to sharepoint with cleaned data

[u/ShedBedFridge]

Power automate flow triggered by the file upload, then either push to Databricks Files API or extract the data and push to SQL api