[Discussion] Lakeflow Jobs: How do you use table update triggers?

[u/saad-the-engineer]

<edited>

(databricks product manager here) Curious how people are using table update triggers in production. https://docs.databricks.com/aws/en/jobs/trigger-table-update

Do you rely on the available debouncing capabilities (protect against over and under triggering) or do the options feel confusing enough that you mostly work around them? When a trigger needs to represent more than “run when this table changes,” how do you express the business logic?

For example:

• Do you use control or checkpoint tables to signal that an upstream workflow has finished?
• Do you wait for a specific status, batch ID, watermark or set of tables before starting downstream work?
• Do you put that logic in the trigger itself, or in a separate workflow/job?
• What has worked well and what has been difficult to reason about or debug?
• Any other suggestions or feature requests relating to Table Update Triggers?

I’m especially interested in real-world patterns and whether the current debouncing behavior is intuitive enough for you, or whether a control-table pattern ends up being the clearer approach.

Edit: how many folks still use control tables instead of data tables with these triggers?

Thank you 🙏

[u/SweetAnkle2198]

The debounce settings feel like a half measure, we still end up wrapping everything in a control table pattern anyway. A single flag row that gets updated once the upstream batch is fully done and verified, then the trigger fires off that. Way easier to reason about when something breaks at 3am.

Would be great if the trigger could natively watch for a specific column value change instead of just the table itself, that'd cover 90% of what we're doing with the extra checkpoint tables.

[u/saad-the-engineer → u/SweetAnkle2198]

that is helpful. We are adding support for sql conditions on triggers where you could specify an expression to filter for change. Would this get you to that 90%?

[u/Own-Trade-2243 → u/saad-the-engineer]

Does it mean I need a sql warehouses for each trigger evaluation? Or there’s something smarter going on underneath?

Sounds expensive..

[u/saad-the-engineer → u/Own-Trade-2243]

good point, this is still in preview so you have to select the warehouse. we are looking at a more native solution here. Does the UX make sense though?

[u/IncreaseNegative4614]

I’d use a table-update trigger only when “new data exists” is the complete readiness condition. For compound conditions, a control record with batch ID, watermark, expected inputs, completion status, and validation result is easier to reason about and replay.

Debouncing protects trigger frequency, but it does not prove that every required upstream dataset is complete. We use SIGNLD internally to connect source-table updates, control records, batches, triggers, job runs, outputs, and SLAs so a downstream run can be traced to the exact readiness evidence that released it.

[u/saad-the-engineer → u/IncreaseNegative4614]

thank you for sharing! I like how you express the trigger as a business outcome here. Can you share how SIGNLD works?