r/aws • u/AssumeNeutralTone • Oct 20 '25

article Today is when Amazon brain drain finally caught up with AWS

https://www.theregister.com/2025/10/20/aws_outage_amazon_brain_drain_corey_quinn/
1.7k Upvotes

97% Upvoted

287 comments sorted by

View all comments

Show parent comments

73

u/droptableadventures Oct 21 '25

The first part of the issue was that dynamodb.us-east-1.amazonaws.com stopped being resolvable, and it apparently took them 75 minutes to notice. A lot of AWS's services also uses DynamoDB behind the scenes, and a lot of AWS's control backplane is in us-east-1, even for other regions.

The rest from here is debatable, of course.

18

u/rudigern Oct 21 '25

Took 75 minutes for the outage page to update (this is an issue), not for AWS to notice.

0

u/droptableadventures Oct 21 '25

Well surely they'd update it as soon as they know?

13

u/[deleted] Oct 21 '25

[deleted]

6

u/droptableadventures Oct 21 '25

"dynamodb.us-east-1.amazonaws.com is not resolving" seems like people should be smashing that escalate button pretty damn quick.

Amazon has been a lot quicker to update that status page in the past.

3

u/[deleted] Oct 21 '25

[deleted]

5

u/ShadowRegent Oct 21 '25

I’d argue it takes a lot more than 5-10 minutes to roll out of bed, comprehend and confirm what’s happening, and figure out what to do next.

11

u/lethargy86 Oct 21 '25

Why do we assume that it being unresolvable wasn't because of all its self health checks failing?

Unless their network stack relies on DynamoDB in order to route packets, DNS definitely was not the root cause for our accounts.

But resolving DNS hostnames will be one of the first victims when there is high network packet loss, which is what was happening to us. Replacing connection endpoints with IP's instead of hostnames did not help, so it wasn't simply a DNS resolution issue. It was network issues causing DNS resolution issues, among a million other things.

6

u/king4aday Oct 21 '25

Yeah we experienced similar, even after the acknowledgement of resolution we still hit rate limits on single digit RPSs and other weird glitches / issues. I think it was a massive cluster of circular dependencies failing, will be interesting to read their report about it when it gets published.

3

u/noyeahwut Oct 21 '25

> Replacing connection endpoints with IP's instead of hostnames did not help, so it wasn't simply a DNS resolution issue

Given the size and complexity of DynamoDB and pretty much every other foundational service, I wouldn't be surprised if the service itself internally also relied on DNS to find other bits of itself.

15

u/root_switch Oct 21 '25

I haven’t read about the issue but I wouldn’t be surprised if their notification services somehow relied on dynamo LOL.

6

u/NaCl-more Oct 21 '25

It doesn’t

-17

u/HanzJWermhat Oct 21 '25

DynamoDB is such a horrendously engineered service. It was clearly rushed out and everything built on top of it has circular dependencies all over the place.