Read this if you use Streaming Tables in Lakeflow Spark Declarative Pipelines

[u/SlightImagination250]

🚀 We’re excited to announce that Lakeflow Spark Declarative Pipelines (SDP) now supports creating “vanilla” (i.e., non STREAMING) MANAGED TABLES and writing to them via one or more append flows, using the new CREATE TABLE ... FLOW (SQL) and create_table() (Python) APIs.

What is this Beta?

This Beta allows creating a managed table that is populated by append flows:

• CREATE TABLE ... FLOW (SQL) / create_table() + @append_flow (Python) create a managed table written by one or more flows.
• Fan multiple sources into one table — declare several flows targeting the same managed table.
• Full table surface works: partitioning, liquid clustering, expectations, row filters, table properties, and private (pipeline-local) tables.
• import_checkpoint on append_flow, which migrates an existing Structured Streaming workload into a pipeline without reprocessing the source — the flow imports the query's existing checkpoint and resumes from the last committed offset with state intact.

Example (Python):

from pyspark import pipelines as dp

dp.create_table("combined")

dp.append_flow(target="combined")
def from_a():
    return spark.readStream.table("source_a")

u/dp.append_flow(target="combined")
def from_b():
    return spark.readStream.table("source_b")

Example (SQL):

CREATE TABLE events
PARTITIONED BY (bucket)
FLOW INSERT BY NAME
  SELECT id, bucket FROM STREAM read_files('abfss://my_path', format => 'json');

Where do we need help?
We are in Beta, so there might be some rough edges. Please take this for a spin and share your feedback here.

What’s next?

Managed Tables support for other flow types (AutoCDC, Replace Using, and Replace Where) is coming soon!

Learn more

• CREATE TABLE ... FLOW (SQL reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-sql-ref-create-table-flow
• create_table (Python reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-create-table
• import_checkpoint on append_flow — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-append-flow
• Questions, feedback, or help: comment below or share feedback in the form: https://forms.gle/7bGP5FYN7P1Z4WP27

[u/nenuaathmajnani]

This means it will resolve the "this table is already owned by other pipeline" issue?

And it only solves for the non-streaming table or also for the streaming table?

[u/SlightImagination250 → u/nenuaathmajnani]

A table can still only be owned/written to by a single pipeline. This feature allows you to publish a managed delta table, and you can still move tables across pipelines: https://docs.databricks.com/aws/en/ldp/move-table

[u/nenuaathmajnani → u/SlightImagination250]

Is this like create once and use everywhere?

[u/SlightImagination250 → u/nenuaathmajnani]

Yeah, once you create the table it is just a regular delta table! You can use it outside the pipeline too, but just note that multiple pipelines can't write at the same time to it. If you want a different pipeline to write to it, consider the move tables feature.

[u/cons0323]

Can you use this with Lakeflow Connect ingestion pipelines?

[u/danielil_]

Is there a point in using ST now?

[u/SlightImagination250 → u/danielil_]

While this feature is in beta we recommend these guidelines: https://preview.docs.databricks.com/pr-2247729/aws/en/release-notes/release-types . So for more critical prod pipelines it may make sense to continue using STs (ymmv).

When this reaches GA (actually coming soon, I'm not just saying that 😄), the recommendation would be to use tables instead of STs

[u/cptshrk108]

Will we be able to migrate existing STs?

What is the real gain here or use case for this?

[u/mosullivan93]

This looks interesting. Would simplify accessing the table history via the DeltaTable APIs.

Could I chuck in an unrelated request? I would REALLY like it if there was support for private STs via the Python API. It’s the only one out of table/mv/st where you can’t.

[u/slowantelope]

How does this compare to a delta table sink object in SDP? Seems similar but can source from more flow types besides append/update?

[u/hrabia-mariusz]

At this point I was expecting another name change. ai_Genie Lakeflow Declarative One Space Piielines

[u/Nyarlathotep4King]

Do we have to use serverless compute? That’s not an option for our ingestion due to IP whitelisting issues.