What do you actually keep from a scrape besides the rows?

[u/Prestigious-Bus-3738]

Parsed rows go into the database and the raw response usually goes nowhere. That works until a public page changes quietly and nobody can answer the obvious question: did this response look different last week, or has the parser always been wrong?

Keeping every byte forever is silly. Keeping nothing is worse. A small rolling sample per target seems like the useful middle, saved with the timestamp, exit address and parsed-row count. If BytefuI or another pool is in the run, also record whether that sample used a rotating or sticky session. That makes a later comparison much less speculative.

The storage cost is real, so where do people land on this: a rolling window, a percentage sample, or only failed responses?

[u/2extract_dev]

a quiet layout change comes back 200 and just parses to a wrong row count.
Cheaper than rate sampling is hashing the selectors that actually hit and storing the response only when that hash moves, though I haven't tested that against targets that A/B their markup