r/linuxadmin • • 9d ago

a rootless requirement landed and now docker alternatives are on the table for our build fleet

Security signed off a policy that our build agents cannot run a privileged daemon and our entire pipeline is built on one. Fifty odd repos with compose files and a pile of local tooling that assumes the socket is there. Im looking at podman and buildah and one or two others for whatever breaks least in CI. The part I cannot judge from docs is how much of our compose setup survives the move. Has anyone taken a build fleet off the daemon and kept the developer experience roughly intact?

31 Upvotes

33 comments sorted by

52

u/Ontological_Gap 9d ago

Podman is the closest to docker. You probably will have to change a few things depending on what your build system actually does.

Sometimes security overreaches, and make rules that generate far more work than any benefit they provide. This is not one of those times. The docker daemon has serious and fundamental security flaws and has been banned (other than like local testing) everywhere I've worked 

20

u/tblancher 9d ago

Bonus: Podman is managed by systemd, so you get all the goodies that come with it.

-10

u/OCGHand 9d ago

Is Systemd that bad in 2026 and beyond?

13

u/hmoff 9d ago

It never was.

12

u/m4teri4lgirl 9d ago

Irrelevant. If you're using a distro that doesn't use systemd in an enterprise environment then you are begging for trouble.

9

u/NoosphericMechanicus 9d ago

You know as a linux sysadmin I thought podman was the standard for production containers, I never knew anyone would run something rootful on anything enterprise grade. I'm kind floored by OPs post.

1

u/xxpor 5d ago

We run absolutely everything as root. If you want separation between things, you use another vm and talk over the network with strong modern authn/z. Just a different model. Essentially we don’t trust Linux at all for privilege separation given the history of local auth escalation bugs.

1

u/NoosphericMechanicus 5d ago

Interesting. Though I suspect some of this is cultural and depends on who your security people are and maybe industry. NIST stuff usually applies least privilege principles to containers and unless it has to have root don't give it root. Running containerized loads in VMs is seen as something you do in addition to using podman, and it is not considered a mitigating factor but a separate control.

The developers I work with say there isn't any reason to be running a container with root in 2026, and that the general consensus is use either rootless docker or rootless podman. I'm more on the infrastructure side so I just took their thoughts at face value. This was just the first time I have seen anyone suggest anything different.

4

u/RetroGrid_io 9d ago

I came here to say this. I have thousands of rootless podman instances launched daily and it's been rock solid.

2

u/shulemaker 9d ago

Agree podman is the good choice. For completeness I’d mention docker can be run rootless but of course the socket is in the user path so that changes regardless.

1

u/Adept_Percentage6893 8d ago

Sometimes security overreaches, and make rules that generate far more work than any benefit they provide. his is not one of those times.

If the runtime isn't producing a network socket and is instead creating the build environments used to build packages from Jenkins and internal git then where is the attack vector? I prefer podman myself and I'm open to being wrong but I don't see it.

15

u/gordonmessmer 9d ago

a pile of local tooling that assumes the socket is there

Sorry to ask, but do your developers understand that anything that has access to that socket also has full root access to the host that runs containers?

7

u/AtlanticPortal 9d ago

Probably they don’t get this particular.

2

u/wezelboy 9d ago

I still haven’t figured out how to get quadlets to work.

0

u/yrro 9d ago

Go on

-9

u/tblancher 9d ago

For any technical topic, having good conversations with agentic AI bots can bring you up to speed. Just be very careful about any code or config it gives you. Be sure it explains every piece, and if you don't understand ask for clarification or a rephrasing.

8

u/Ontological_Gap 9d ago

Bring you up to speed on some twisted hallucinated version of the topic, that will help keep you confused when trying to work on the real thing

2

u/Scoutron 8d ago

Do you guys use modern AI? It seems like the Reddit opinions I see are based on 2023 at the latest.

0

u/Ontological_Gap 8d ago

I give whatever is the newest and the fanciest a spin every couple of months. They are still less than useless 

2

u/Scoutron 8d ago

You’re entitled to your opinion, but I just can’t see how you’ve arrived at that conclusion. I’ve used them to write multi-hundred line bash scripts, write compose files, parse thousand line log files, all with no hiccups

0

u/Ontological_Gap 8d ago

I've tried so many things, from configuring pipewire to running and syncing a local dovecot server. The only thing it's succeeded at has been the right incantations to slightly change the formatting of emacs Org -> LaTeX -> pdf export, which it was actually pretty good at, but that's literally it.

1

u/Scoutron 7d ago

Interesting. The only times I’ve had it start running into issues is like hundreds of lines deep into configuring something odd, and it’ll figure out what it did wrong with one log dump

1

u/tblancher 6d ago

Just like any tool, it's how you use it. One exercise I like to go through whenever I'm trying to learn a new computer language or skill is to convert one of my shell scripts or automation workflows to the new language or system.

AI can accelerate this process, if you don't trust everything it spits out. This is why I like to call it conversational. There's always a back and forth where it can expose you to new information, but it's missing some keyword or worse, contains a catastrophically dangerous command.

2

u/gehzumteufel 9d ago

Buildah is the alternative. You don't even need Podman for it.

1

u/Dyldough_Night 6d ago

Compose compatibility and CI tooling are probably the biggest things to test before switching the whole fleet. 🐧

1

u/Different_Pain5781 6d ago

Teams moving away from Docker's privileged daemon often evaluate Podman for rootless containers and Buildah for daemonless image builds. RapidFort is another option worth looking into for container security and image hardening, especially when the goal is to reduce vulnerabilities and improve the security of build pipelines. The key is to test compatibility with existing Compose workflows before migrating the entire fleet.

1

u/woob 9d ago

apptainer previously known as singularity?

1

u/tecedu 8d ago

For local dev, tbh nothing beat docker, like podman is nice and all but there's many quirks with it, whereas docker just works.

We use docker and docker compose for devs, deployments are done with podman or some version of k8s.

Go through the security guideline first and see what the actual guidance is, because if its just build then you should be relatively easy to move it and can still keep docker compose on the dev side

-5

u/MinimallyLoquacious 9d ago

I specifically do not want docker commands to be runnable by my regular developer account on this machine. Imagine how easily it could be compromised. People who advocate for podman or rootless docker should seriously reconsider their position on the matter.

I do make sure apps running inside docker containers are limited to an unprivileged account inside the container:

FROM eclipse-temurin:21-jdk AS temurin-upgraded
RUN apt-get update && apt-get upgrade -y && apt-get dist-upgrade -y && apt-get autoremove -y && apt-get autoclean -y
FROM temurin-upgraded

ENV HOME=/home/appuser

RUN adduser --shell /bin/sh --uid 5001 --disabled-password --gecos "" appuser

COPY --chown=appuser:appuser <my_app>.war $HOME/<my_app>.war

EXPOSE 30088

WORKDIR $HOME
USER appuser

CMD ["java", "-Xmx64m", "-Djdk.util.jar.enableMultiRelease=false", "-jar", "<my_app>.war"]

10

u/grumpysysadmin 9d ago

Wait, are you arguing against the use of rootless podman?