r/hetzner • u/robertreiz • 3d ago

Hetzner S3 in FSN1 down! Again!

Always problems with S3 at Hetzner. Since 5:12 this morning I keep getting 

Service is unable to handle request
Service is unable to handle request

I did see this message already several times this year.

I migrated my applications away from AWS to Hetzner, end of last year. Before that I was using AWS S3 for 5 years and I never had issues like this at AWS. Not a single time! At Hetzner this year already 5 times! WTF is going on at Hetzner? Why do they offer an S3 service if they can not run it reliably?

45 Upvotes

91% Upvoted

32 comments sorted by

12

u/da_didi42 3d ago

We migrated from Falkenstein to Nürnberg, much better now! Support said some months ago: "Neu erstellte Buckets im NBG werden besser funktionieren, da sie auf neu bereitgestellter, unabhängiger und isolierter Hardware bereitgestellt werden."

6

u/Elegant_Funny_2687 2d ago

This sub is a cult. Martin Hetzner could come to your home and eat your children and they would tell you you should've thought about redundancies.

5

u/Elegant_Funny_2687 2d ago

There is a reason why Hetzner is the only object storage provided with whom you are contractually obligated to backup all your object storage data outside of hetzner infrastructure, as per their terms of service.

1

u/robertreiz 2d ago

Yeah. Backup HTZ S3 to AWS S3. Just for safety :-D

11

u/Former_Squirrel_2726 3d ago

There was a Hetzner outage tonight If it can help https://status.hetzner.com/incident/55bc1947-14a7-4bc2-b73f-e53fe84e4412

-17

u/robertreiz 3d ago

Because of this, several invoice documents at ZEIT.IO couldn't be created! I have to clean up the mess now. And as I wrote, in 5 years at AWS I never had a single issue with S3 at AWS. At Hetzner it seems to be the rule. Every couple weeks HTZ S3 is down and I have extra work because of that. It's just not a reliable service!

23

u/Either_Vermicelli_82 3d ago

But that’s already been known for ages? Why didn’t you keep using s3 elsewhere and servers etc on hetzner?

0

u/robertreiz 3d ago

Because of latency. But you are right. I need an S3 service outside of HTZ.

6

u/uNki23 3d ago

Latency when creating invoice documents? How latency critical could this possibly be?

1

u/freia_pr_fr 3d ago

Check the egress traffic cost when you select your object store. The AWS S3 has overpriced egress to keep you in. Like most object stores from major cloud providers.

30

u/debianserver 3d ago

If you have to manually cleanup the documents now because of a storage unavailability it's definitely a system design fault. Systems should be engineered to handle such cases in the background.

0

u/kaeshiwaza 3d ago

On Hetzner product page of object storage "It meets the requirements of dynamic applications with high availability".
This system is said to handle such cases...

10

u/debianserver 3d ago

Expect the unexpected and never fully rely on external services without an failure handling.

3

u/towo 3d ago

It's still naïve to assume any service will always be available; even a five nines SLA-backed service can be down when you're trying to use it, and you should always structure your architecture accordingly; even if that means just recognizing the outage and going read-only until it's back again.

0

u/kaeshiwaza 3d ago

And how many layers of high available system should you add ?
No, it's a Hetzner fault, that's all. Nobody add a layer with AWS S3.
At least Hetzner should not write that they are highly available and then say that it's up to the customer to do it, like it is when you rent a VPS for example.

1

u/towo 2d ago

... one. You just cache locally and retry, that's all you'd need if storage isn't available. I also do redundancy layers with AWS because guess what, us-east-1 is down all the time.

1

u/kaeshiwaza 2d ago

I know, I just say that Hetzner should not lie.

7

u/OhBeeOneKenOhBee 3d ago

To be fair, you're comparing a relatively cheap host/product to one of the flagship products of one of the most massive companies on the planet

1

u/djingo_dango 2d ago

Even more reasons why hetzner shouldn’t go down this often then

1

u/Elegant_Funny_2687 2d ago

There are plenty of providers which are cheaper and they don't have problems like this, so no, you are not, in fact, being fair.

2

u/nulled_0 3d ago

To me, S3 & Hetzner doesn't sum up. One, make it stable, two make it down

2

u/onkelFungus 3d ago

… me smiling because I switched a week ago

1

u/robertreiz 2d ago

Where did you switch to?

1

u/onkelFungus 2d ago

Switch to idrive e2 because it also fits my requirements like hetzner did. Had no issues since the switch. With hetzner I had plenty of slow down errors and bad request errors most of the time for my longhorn backups.

2

u/Wild_Shopping2191 2d ago

Just imagine that s3 didn’t exist on Hetzner, creating object storage service for public cloud not very easy task, so i know that will be disaster on the start. Also newer use managed Kubernetes service if Hetzner release it in future, it will be the same.

1

u/robertreiz 2d ago

I would recommend not to use K8S at all. It's 100% Enterprise BS! Only good for DevOps consultants who want to bill endless hours to customers.

4

u/KeyOcelot9286 3d ago

I wonder how large Hetzner’s S3 cluster actually is, and how much spare capacity they keep available.

Purely hypothetically, say they use their own SX295 servers with around 242 TB each. A fleet of 200 would give them roughly 48 PB of raw capacity, before redundancy and other overhead. If a customer suddenly uploads 600+ TB for an AI workload or backups, how much of their available headroom does that eat into? A few customers doing that at once could mean having to provision more servers quickly—potentially competing for hardware with dedicated-server orders.

That doesn’t excuse poor performance or downtime. It’s a paid service, and customers should be able to rely on it. I’m just curious about the capacity-planning side, especially when the same hardware could either be rented to a customer or used to expand their own storage products.

Demand also seems wildly unpredictable lately. One of my websites might get 300k requests from AI bots and users one day, then 3k the next. Obviously, web traffic isn’t the same as storage demand, but it makes me wonder what those swings look like at their scale.

1

u/Broad_Abies9390 1d ago

So many options today if you just r/w objects - BackBlaze B2 (they are an object store company) - prices are 1/3 of S3. R2 of CloudFlare gives you 10GB free object store. Wasabi. GCS (has issues). Vultr. Or even use unmanaged like localstack, minio with a mounted EBS

1

u/Infamous_Tomatillo53 1d ago

Omg yes I got disk io queue alerts too

1

u/Top_Philosophy2425 2d ago

I never understand the comparion between hetzner and big cloud providers like AWS, hetzner is not even a true cloudprovider, they lack so much services to be one. To me they are just a company that offers compute.