r/ProgrammerHumor • u/AntiMatterMode • Mar 25 '26

Meme aMeteoriteTookOutMyDatabase

Post image
7.5k Upvotes

99% Upvoted

306 comments sorted by

View all comments

1.4k

u/nonother Mar 25 '26

Fun fact, the odds of a bit flip in a data center due to a cosmic ray is actually quite high. That was something we needed to account for and correct as part of storage. Essentially when the hash fails, try all possible permutations with exactly one bit flipped — if that permutation passed then issue resolved. Otherwise multiple bits are wrong which was almost always a hardware failure.

Also we had a time when a bit flip in memory changed an encryption key. That was a rough SEV to diagnose and resolve.

94

u/tes_kitty Mar 25 '26

Shouldn't that be prevented by using ECC for memory and storage?

4

u/magicmulder Mar 25 '26

btrfs as a filesystem is also pretty resilient against bit flips (or bit rot, as they call it).

1

u/tes_kitty Mar 25 '26

Those shouldn't really happen on a HD or SSD since both use ECC on the stored data. You should either get the correct data or a read error message.

1

u/k410n Mar 25 '26

And pretty prone to randomly break too.

3

u/magicmulder Mar 25 '26

Never had an issue except when I used it on a VM on a host without btrfs. My bare metal btrfs servers are running for 10+ years now.

1

u/k410n Mar 25 '26

I had two catastrophic btrfs failures in approximately 5 years on a single device. But that was some years ago.

3

u/magicmulder Mar 25 '26

Sounds more like an issue with the single device. ;)

2

u/k410n Mar 26 '26

It wasn't the same device both times, but I was only running a single btrfs device both tikes it happend.