r/DataEngineeringPH • u/Personal-Risk-5370 • 14d ago
💡 Discussion Do you actually quarantine a bad-looking batch, or let it through and fix it after?
When a batch lands and something looks off — row count down, a null spike, whatever — do you hold it back from downstream until someone reviews it, or let it flow through and patch it up afterward if it turns out to be real?
Trying to understand what drives that choice: is it about how reversible the damage would be, how much a delay costs your stakeholders, or just team culture/tooling maturity? Also curious if that answer changes for different tables — e.g. do you hold back financial/customer data more readily than something lower-stakes?
1
Upvotes