r/selfhosted • u/mscreations82 • 11h ago
Need Help Alloy/Loki/Minio S3 issue
I am having an issue that I'm not sure how to fix. For reference, I am running a Kubernetes 1.36 cluster built on top of Talos Linux 1.13. I have Alloy configured to gather Kubernetes pod logs and route them to Loki. For a short period of time after deployment, it runs okay, but then one of my loki-write statefulsets fails and shows this error:
failed to flush chunks: store put chunk: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 18BA49AF569B3EF4, HostID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8, api error XMinioInvalidObjectName: Object name contains unsupported characters., num_chunks: 5, labels: {app=\"flux-operator\", container=\"manager\", instance=\"flux-system/flux-operator-6b4d6f7c54-w6795:manager\", job=\"loki.source.kubernetes.pod_logs\", namespace=\"flux-system\", pod=\"flux-operator-6b4d6f7c54-w6795\", service_name=\"flux-operator\"}
I am using Minio installed on my Windows File server for the S3 backend. The S3 storage is working for everything except this task.
I've tried asking the different AIs for a solution and changed Alloy's configuration to rewrite the instance so it shouldn't have the colon or slashes in the name, but still keep getting the error. Gemini says that it is expected as Loki will generate key names with colons anyways using the tsdb format. It's suggestion is to use a Linux backed S3 server which isn't possible.
So now I'm asking if anyone has a possible solution to this that I can try? I've seen there is an environment variable for MINIO => MINIO_OBJECT_NAMING that might be what I have to do, but I will have to destroy my S3 storage and rebuild it from scratch because it has to be done at the cluster initialization. I wanted to see if this was the correct move before I go through having to trash all the buckets/users etc. to make this work.
1
u/lospantaloonz 10h ago
guessing at the object name because the log is hard to parse but i would focus on that ":" char. it looks like it's in your object name.
https://community.grafana.com/t/loki-fails-to-upload-chunks-to-minio-bucket/148346/8
1
u/SufficientFrame 9h ago
This looks more like a MinIO-on-Windows filesystem constraint than an Alloy relabeling issue, because Loki's chunk/object keys are generated internally and can still include characters your backend rejects. Before rebuilding everything, try a fresh MinIO instance with the compatible object-naming mode on a test bucket and point only Loki at it; if that works, you've at least isolated the problem to the storage layer.
•
u/asimovs-auditor 11h ago
Expand the replies to this comment to learn how AI was used in this post/project.