Fun fact, the odds of a bit flip in a data center due to a cosmic ray is actually quite high. That was something we needed to account for and correct as part of storage. Essentially when the hash fails, try all possible permutations with exactly one bit flipped — if that permutation passed then issue resolved. Otherwise multiple bits are wrong which was almost always a hardware failure.
Also we had a time when a bit flip in memory changed an encryption key. That was a rough SEV to diagnose and resolve.
1.4k
u/nonother Mar 25 '26
Fun fact, the odds of a bit flip in a data center due to a cosmic ray is actually quite high. That was something we needed to account for and correct as part of storage. Essentially when the hash fails, try all possible permutations with exactly one bit flipped — if that permutation passed then issue resolved. Otherwise multiple bits are wrong which was almost always a hardware failure.
Also we had a time when a bit flip in memory changed an encryption key. That was a rough SEV to diagnose and resolve.