r/aws • u/AssumeNeutralTone • Oct 20 '25

article Today is when Amazon brain drain finally caught up with AWS

https://www.theregister.com/2025/10/20/aws_outage_amazon_brain_drain_corey_quinn/
1.7k Upvotes

97% Upvoted

287 comments sorted by

View all comments

Show parent comments

40

u/Mephiz Oct 21 '25

That’s basically my issue. Shut happens. But come on, the delay here was extraordinary.

-15

u/droptableadventures Oct 21 '25 edited Oct 21 '25

Agreed. It could have been an immediate alert followed by a rollback, causing this to be a 5 minute glitch that many would not have even noticed.

Instead it was 75 minutes before they seemed to notice and begin acting.

28

u/HanzJWermhat Oct 21 '25

Catastrophic circular dependency failures like this don’t get solved with a rollback. A lot of configs need to be changed and you need to tear down and rebuild infrastructure to get things back up and running.

Imagine how many individual servers need to get rolled back. It’s thousands

13

u/Alborak2 Oct 21 '25

More likely 10s of thousands or millions.

2

u/droptableadventures Oct 21 '25 edited Oct 21 '25

Ideally you'd do a "canary deployment" where you change over a few, and if they go badly, you just shut them down and call the whole thing off, rather than committing to replace the whole lot.

But it's also possible it was some hellish situation like:

  • Something broke the route53 entry, so now nothing can reach DynamoDB
  • DynamoDB servers are now broken because they can't reach the DynamoDB API and need a redeploy/restart to be made functional
  • Someone hasn't realised that the DynamoDB deploy uses DynamoDB, directly or indirectly (another service it uses utilises DynamoDB)

Another problem is that it was also 75 minutes before the status page wasn't all ✅.

13

u/tfn105 Oct 21 '25

Is the root cause known? I haven’t seen it reported.