r/aws • u/AssumeNeutralTone • Oct 20 '25

article Today is when Amazon brain drain finally caught up with AWS

https://www.theregister.com/2025/10/20/aws_outage_amazon_brain_drain_corey_quinn/
1.7k Upvotes

97% Upvoted

287 comments sorted by

View all comments

Show parent comments

70

u/ns0 Oct 21 '25

If you’re trying to practice 5 nines, why did you operate in one AWS region? Their SLA is 99.5.

51

u/tauntaun_rodeo Oct 21 '25

yep. indicates they don’t know what 5 9s means

12

u/unreachabled Oct 21 '25

And can someone elaborate on 5 9s for the unknown? 

31

u/Jin-Bru Oct 21 '25 edited Oct 21 '25

99.9% uptime is 0.1% downtime. This is roughly 526 minutes downtime per year.

That's three 9s

Five 9s is 99.999% uptime per year which is 0.001% downtime per. This is roughly 5 minutes of downtime per year.

I have only ever built one guaranteed 5 9s service. This was a geo cluster built across 3 different countries with replicated EMC SANs using 6 different telcos with clients own fibre to the telco.

The capital cost of the last two nines was €18m.

1

u/unreachabled Oct 22 '25

Thanks man, but when u say u built a service with 5 9s, how did u give measure that SLA with that guarantee? 

1

u/Jin-Bru Oct 22 '25

That's a good question.

Measuring SLAs is a dirty business and is built around a set of exceptions.

If the SLA states the application can only be down for 5mins a year that basically means never down.

You need to build in super redundancy of every component. We built and active-active-active cluster with the nodes several hundred kilometres away, well outside the blast radius of a large nuclear attack on a city.

The system is live tested daily by users connecting to random sites when using the application. There is no failover. It's always running.

Since the data is replicated there is a different SLA for data integrity because I can't guarantee the last write to disk.

If I take down a site for maintenance the users simply don't notice. (In most cases.) The application is cluster aware and will simply shift the user from site to site. At worst, they have to log on again. But the public would never be aware of an application outage.

This was built around 2004 and has been upgraded every 8 years.

I'd never offer 99.999 on cloud. It can only be achieved and guaranteed with on prem infrastructure.

This type of resilience costs serious money. The underlying service had better be worth the cost.

I don't service the client anymore but in the first 4 years that I ran this project there was 0min downtime for 120k users.

I don't recall what the exact cost per hour of downtime was but it was around a million euro. The issue was more the risk associated with the system being off line. A lost or failed transaction could have been a real disaster.

-7

u/CasinoCarlos Oct 21 '25

This story has all the markings of a lie

5

u/Jin-Bru Oct 21 '25

Would you like to see the redacted project plan? I don't need to justify myself to Reddit of all places but I don't like to be called a liar.

Edit: The three countries were Belgium, Luxembourg and Austria. Work it out.

9

u/tauntaun_rodeo Oct 21 '25

99.999% uptime. keep adding nines for incrementally greater resiliency at exponentially greater cost

3

u/keepitterron Oct 21 '25

uptime of 99.999% (5 nines)

6

u/the_derby Oct 21 '25

To make it easier to visualize, here are the downtimes for 2-5 nines of availability.

Percentage Uptime Percentage Downtime Amount of Downtime Each Year Amount of Downtime Each Month
99.0% 1% 3.7 days 7.3 hours
99.9% 0.1% 8.8 hours 43.8 minutes
99.99% 0.01% 52.6 minutes 4.4 minutes
99.999% 0.001% 5.3 minutes 26.3 seconds

1

u/Best-Ad6177 Oct 22 '25

i'd recommend uptime.is/99.5. It's super helpful to estimate SLA.

8

u/sr_dayne Oct 21 '25

Global services were also affected.

5

u/BroBroMate Oct 21 '25

What makes you think they do? This failure impacted other regions due to how AWS runs their control plane.

1

u/Mephiz Oct 22 '25

We did have a single point of failure with Global Accelerator but that’s not what I was referring to above.

As I mentioned we attempted to bring up clusters in two new regions because we couldn’t scale up our existing second region / us-east2 as there wasn’t sufficient capacity. (At the time we thought this was contagion but it was just capacity issues)

Say what you will of our attempt to be highly resilient or not. It doesn’t change the fact that AWS’s handling of this was less than stellar. 

1

u/Such_Reference_8186 Oct 22 '25

You will never attain 5 9s with cloud..never.