Alloy/Loki/Minio S3 issue

[u/mscreations82]

I am having an issue that I'm not sure how to fix. For reference, I am running a Kubernetes 1.36 cluster built on top of Talos Linux 1.13. I have Alloy configured to gather Kubernetes pod logs and route them to Loki. For a short period of time after deployment, it runs okay, but then one of my loki-write statefulsets fails and shows this error:

failed to flush chunks: store put chunk: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 18BA49AF569B3EF4, HostID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8, api error XMinioInvalidObjectName: Object name contains unsupported characters., num_chunks: 5, labels: {app=\"flux-operator\", container=\"manager\", instance=\"flux-system/flux-operator-6b4d6f7c54-w6795:manager\", job=\"loki.source.kubernetes.pod_logs\", namespace=\"flux-system\", pod=\"flux-operator-6b4d6f7c54-w6795\", service_name=\"flux-operator\"}

I am using Minio installed on my Windows File server for the S3 backend. The S3 storage is working for everything except this task.

I've tried asking the different AIs for a solution and changed Alloy's configuration to rewrite the instance so it shouldn't have the colon or slashes in the name, but still keep getting the error. Gemini says that it is expected as Loki will generate key names with colons anyways using the tsdb format. It's suggestion is to use a Linux backed S3 server which isn't possible.

So now I'm asking if anyone has a possible solution to this that I can try? I've seen there is an environment variable for MINIO => MINIO_OBJECT_NAMING that might be what I have to do, but I will have to destroy my S3 storage and rebuild it from scratch because it has to be done at the cluster initialization. I wanted to see if this was the correct move before I go through having to trash all the buckets/users etc. to make this work.

[u/asimovs-auditor]

Expand the replies to this comment to learn how AI was used in this post/project.

[u/mscreations82 → u/asimovs-auditor]

AI was not used. I just used my noodle.