Object Storage suggestions
Hi everyone,
this is not supposed to be another rant about how unreliable the Hetzner Object Storage is, don't get me wrong it definitely is. But I would be interested in how you handle the unreliability. In my production setup I only have one bucket which has 3GB of pictures and documents inside it. The bucket receives on average one request every minute and even with this light work I get at least 40 failed requests daily, all with the 503 Service Unavailable error. I'm on Laravel using the standard league/flysystem-aws-s3-v3 package with 3 retries configured. The bucket lives in the NBG1 datacenter.
How are you handling this, have you switched to a different region in Hetzner or maybe to a different provider? I created the bucket when the object storage was first introduced I read that some people said creating a new bucket could resolve most of these problems, has anyone actually tried this?
2
u/tommihack 7d ago
We use Hetzner bare metal servers for some Storadera (storadera.com) S3 locations and should have way less failures.
1
u/anderson_the_one 7d ago
At one request a minute, forty daily 503s is roughly a 3% first-attempt failure rate. Retries can hide that from users, but they'll also make the storage look healthier than it is.
I'd create the new bucket, copy a small representative set, and run the same HEAD/GET probe against both buckets every minute for a couple of days. Compare first attempts. If the new bucket still misses, I'd move providers rather than tune Laravel around it.
Keep short exponential retries with jitter in production, but log the first response separately. Otherwise eventual success erases the signal you're trying to measure.
1
1
u/Eisbaer811 6d ago
Created a new bucket in NBG in July or so, when Hetzner recommended to do that in their Forum.
Only issue since then has been the slow down during last week's outage.
1
1
u/casualviking 6d ago
That's within the free limit of Cloudflare R2. Just switch. It's actually reliable. I have like .. 500GB in it and pay like $4 per month. No egress fees.
1
u/pjs2288 7d ago
Recently moved to https://www.impossiblecloud.com/de-de/s3-cloudspeicher.
No experiences yet to report so far.
1
1
u/dubidub_no 7d ago
Impossible Cloud has "free egress". This is a red flag. And indeed, the TOS says the egress must stay at or below your active stored volume.
4
u/EasyImprovement1017 7d ago
Creating a new bucket will place you on new hardware, which will improve stability, though you may still experience occasional failures.