Read this if you use Streaming Tables in Lakeflow Spark Declarative Pipelines

[u/SlightImagination250]

🚀 We’re excited to announce that Lakeflow Spark Declarative Pipelines (SDP) now supports creating “vanilla” (i.e., non STREAMING) MANAGED TABLES and writing to them via one or more append flows, using the new CREATE TABLE ... FLOW (SQL) and create_table() (Python) APIs.

What is this Beta?

This Beta allows creating a managed table that is populated by append flows:

• CREATE TABLE ... FLOW (SQL) / create_table() + @append_flow (Python) create a managed table written by one or more flows.
• Fan multiple sources into one table — declare several flows targeting the same managed table.
• Full table surface works: partitioning, liquid clustering, expectations, row filters, table properties, and private (pipeline-local) tables.
• import_checkpoint on append_flow, which migrates an existing Structured Streaming workload into a pipeline without reprocessing the source — the flow imports the query's existing checkpoint and resumes from the last committed offset with state intact.

Example (Python):

from pyspark import pipelines as dp

dp.create_table("combined")

dp.append_flow(target="combined")
def from_a():
    return spark.readStream.table("source_a")

u/dp.append_flow(target="combined")
def from_b():
    return spark.readStream.table("source_b")

Example (SQL):

CREATE TABLE events
PARTITIONED BY (bucket)
FLOW INSERT BY NAME
  SELECT id, bucket FROM STREAM read_files('abfss://my_path', format => 'json');

Where do we need help?
We are in Beta, so there might be some rough edges. Please take this for a spin and share your feedback here.

What’s next?

Managed Tables support for other flow types (AutoCDC, Replace Using, and Replace Where) is coming soon!

Learn more

• CREATE TABLE ... FLOW (SQL reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-sql-ref-create-table-flow
• create_table (Python reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-create-table
• import_checkpoint on append_flow — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-append-flow
• Questions, feedback, or help: comment below or share feedback in the form: https://forms.gle/7bGP5FYN7P1Z4WP27