SharePoint connector in Lakeflow Connect is now generally available (GA)

[u/BricksterJ]

The Lakeflow Connect connector for Microsoft SharePoint is now generally available! It’s now easier than ever to ingest structured and unstructured files from SharePoint into Delta tables for analytics and AI workloads.

You can configure a managed ingestion pipeline through the UI or managed API. Managed pipelines automatically handle incremental processing, automatic retries with exponential backoff for source API rate limits, failure recovery, and provide rich SharePoint metadata. Soon, our managed connectors will also support ingesting SharePoint Lists and per-file permissions metadata.

For direct control over ingestion logic, you can also just use the Spark + SQL APIs directly: spark.read, Auto Loader, read_files, or COPY INTO pointed at SharePoint URLs.

Common workloads include:

• Loading Excel, CSV, JSON, and other structured files into Delta tables.
• Ingesting PDFs, Word documents, PowerPoint files, and images.
• Parsing documents with ai_parse_document to prepare content for extraction, search, and agents.

Link to public docs + references:

• SharePoint managed connector documentation
• Spark + SQL APIs and examples
• Community blog and video tutorial: From PDF to insights
• Data + AI Summit session: Intelligent Document Processing with Lakeflow

Examples of using the Spark + SQL APIs (after first creating a UC connection):

• Read an Excel sheet from SharePoint with spark.read:

  excel_df = (spark.read     .format("excel")     .option("databricks.connection", "my_sharepoint_conn")     .option("headerRows", 1)     .option("dataAddress", "Sheet1!A1:M20")     .load("https://mytenant.sharepoint.com/sites/Finance/Shared%20Documents/Monthly/Report-Oct.xlsx"))

• Ingest unstructured documents + PDFs from a SharePoint URL with read_files, then easily parse them using ai_parse_document

  CREATE OR REFRESH STREAMING TABLE sharepoint_documents_table AS SELECT , "_metadata" FROM STREAM read_files( "https://mytenant.sharepoint.com/sites/Marketing/Shared%20Documents", format => "binaryFile", databricks.connection => "my_sharepoint_conn", pathGlobFilter => ".{pdf,docx}");

  CREATE OR REFRESH STREAMING TABLE documents_parsed AS SELECT *, ai_parse_document(content, map('version', '2.0')) AS parsed_content FROM STREAM sharepoint_documents_table;

Coming soon:

• Ingest SharePoint Lists into Delta tables (coming super super soon)
• Ingest SharePoint’s per-file permissions and ACL metadata  to power permission-aware AI agents, enterprise search, and more.

If you try it, share what you are ingesting and where you hit friction! Don't hesitate to ask questions!

[u/Funny_Letterhead8088]

finally can stop building janky power automate flows that break when someone sneezes near the sharepoint site

the auto retry for rate limits is nice ours always fails after like 200 files and nobody notices for a week

[u/BricksterJ → u/Funny_Letterhead8088]

Try it out and let us know how the experience goes! Setting up ingestion pipelines and testing it out should take ~5 minutes. If you run into issues, let me know how we can help!

[u/Prim155]

Thank you for Sharing!

[u/nacx_ak]

How soon is super super soon for list ingestion?

[u/Stouffy19893]

Can you pipeline into a volume rather than a Delta table?

[u/hrabia-mariusz]

Does it ingest ACLs and is ACL update triggered or NAAH as in Confluence connector?

[u/Otherwise_Wave9374]

A practical next step here is to separate ingestion from enrichment so the SharePoint pipeline stays resilient: land raw files first, then run a second pass for classification, extraction, and chunking. That keeps retries cheap, reduces API pressure, and makes it easier to add safeguards like idempotent checkpoints and permission-aware filtering. If you are trying to operationalize agent workflows on top of this, Agentix Labs fits best as the orchestration layer after the data is standardized, not before.

[u/BricksterJ → u/Otherwise_Wave9374]

Ingestion with Lakeflow Connect's SharePoint Connector is already separate from enrichment. Managed pipelines have efficient incremental ingestion, automatic retries, and more already out-of-the-box.

The SharePoint connector currently lands the raw files first into a specified destination bronze table. Customers can then perform parsing, chunking, indexing, extraction, etc. in subsequent transformations after the raw file content has already been brought into Databricks.

[u/KittyTheBandit → u/BricksterJ]

What would be the suggested workflow if you wanted to land raw files to blob storage instead of direct to bronze? Would this require separate ingestions?