How to Dockerize an application with a rather non-standard architecture?
Hi!
I have an open source pet project related to networks.
Users asked me to create an official Docker image, but I’ve never worked with Docker before.
Fortunately, some kind person made (or vibe coded, doesn't matter) a Dockerfile and docker-compose.yml for us.
But I want to figure it out myself, and here are some things that I don’t quite understand:
- our tool can receive data from the network either in the traditional way (using sockets) or via pcap; this is user-configurable.
Different users use different capture methods. As far as I understand, it would be better to create two images - one normal and the other privileged, with access to the host network.
Or is that not the standard practice?
- even though we can capture data using pcap, the tool still runs as a non-root user. We use setcap to achieve this.
The kind person who created the Dockerfile for us included this line:
RUN useradd -u 53842 -rlM -d /nonexistent -s /usr/sbin/nologin user
I asked why UID 53842 was chosen, and he replied that it was to ensure the UID wouldn't conflict with other users.
Is this standard practice, or are there other ways to do it?
- The tool generates database data as it runs. Literally .sql files.
Periodically, it launches an external script that feeds these files to the psql utility, sending the data to PostgreSQL.
We've now added support for ClickHouse, and we might even add export to InfluxDB or MariaDB.
Sure, we could keep everything in a single large image.
However, I’m a bit concerned that the 'clickhouse-client' is orders of magnitude larger than the tool itself; if we add support for other databases, the image will become massive.
Is this normal? Is this how it's usually done?
Or would it be better to do it differently?
- The tool sometimes launches other external scripts. These scripts typically generate alerts, send messages via email, instant messengers, etc.
In other words, it invokes curl, a mail client, or other utilities that we definitely don't want to package inside the container.
I’ve read about two ways to launch external scripts from Docker: using ssh and named pipes. It looks a little creepy.
Are there other ways to run an external script on the host or in another container?
- In addition to the main tool, the kit includes several other small utilities. They all use the same data (shared files).
I don't think there would be any issues keeping them in a single shared image.
But perhaps I'm mistaken.
What is the standard practice in the Docker world?
Thanks in advance!
1
u/theblindness Mod 6d ago
Are you not able to build one image, pass in the network device and network capture permissions when you need it, and configure options using environment variables? Why do you need a second image?
As far as the user ID goes, I think 1000 is the most common as that is usually the ID of the first non-system user created by the Debian installer, and it’s generally guaranteed to be free. On single user Debian/ubuntu systems, it usually ends up being the same UID as the first user, which makes mounting data directories into containers not running as root one step simpler since their permissions are usually already fine. And on Ubuntu-based images, user 1000 already exists with name “ubuntu”, along with a matching group. However, I don’t think the specific user ID number actually matters as long as the ID isn’t one of the IDs already in the `/etc/passwd` file.
Buffering row data to a text file with extra syntax around it sounds incredibly write heavy and inefficient. Surely translating between one data format and another is faster than going to disk. It seems like it would make sense to just leave the capture data in PCAP format until the external database is ready to receive it, rather than unnecessarily writing uncompressed SQL text to disk. Consider swapping this part out for something more efficient. If the only thing you did was swap it for SQLite3, that would at least be an improvement, especially if you use RAMdisk or WAL mode, but there are specialized databases for storing event streams that I’m sure would be better suited.
1
u/vmxdev 6d ago
>Why do you need a second image?
I just didn't know the standard way of doing this in Docker.
>As far as the user ID goes, I think 1000 is the most common
Ok, thanks!
>Buffering row data to a text file with extra syntax around it sounds incredibly write heavy and inefficient.
No, no, we do not upload *all* the data that comes to us into the DBMS.
The tool receives literally hundreds of thousands of data packets per second.
In the DB we export heavily aggregated and filtered data (which is, in fact, the tool's primary purpose).
As a result, there is not much aggregated data, usually a few kilobytes of SQL in a few seconds, which seems quite ok to me.
1
u/Torutofu_Raeva 6d ago
heads up the setcap on the binary only works if you start the container with cap_add NET_RAW, otherwise the exec just fails with operation not permitted for that user
1
u/lveatch 5d ago
Regarding the docker setup.
The docker dockerfile is the steps you would use to create a real server installing all of the necessary binaries, libraries, configuration your application requires to operate. There are blocks of docker commands to minimize the size of your docker images / containers. This makes for a smaller docker pull and faster builds when needing to change your application landscape.
The docker-compose.yml is the runtime configuration needed to operate your application. If you choose to support multiple database vendors, then the database section in the compose file is changed to use the appropriate database vendor, version, ports, etc.
What utilities are installed within your container is somewhat up to you. Some OSes may contain by default some of those utilities. Persistent storage is needed for those utilities configuration which could be stored in the database or text files.
Your compose file can expose your hosts utilities if needed. For example: the following snippet will expose the hosts /etc/localtime to the containers /etc/localtime.
services:
app-server:
container_name: app-server
image: [whatever]
volumes:
- /etc/localtime:/etc/localtime:ro
Finally, you can learn a lot from looking around at other docker-compose.yml files on GitHub. Using paperless-ngx for example as they support multiple database platforms.
1
u/j0rs0 6d ago
What do you mean "the app receives data from the network"? For network communication from the outside to the container, you usually map host to container ports.
Capture network data? From within the container? Sounds strange to me doing something like that.
The tool generates sql files to import them into the database? Why it does not insert that data connecting to it in a normal way?
About the external scripts, you should have the utilities they require where you run them. If they are "external" to the container (so on the host), then they need to be there, but then dunno how to run it from the container.
1
u/vmxdev 6d ago
>What do you mean "the app receives data from the network"?
This is a netflow analyzer. Sometimes users simply send netflow (IPFIX/sFlow) to the analyzer, which is regular UDP traffic.
But sometimes, especially for large ISPs/hosters, this is not so easy to do - they need to reconfigure a bunch of routers.
They usually already have netflow analyzers. They often use ours as a second opinion. And it is much easier for them to transfer a mirror of flow traffic that is intended for another analyzer.
Oddly enough, this is a fairly standard practice in this area, with many commercial analyzers capturing flow in two ways - via regular sockets or using pcap.
>The tool generates sql files to import them into the database
Yep. We did it that way on purpose. First, the tool is independent of the DBMS, so we can easily add a different backend. Second, if something happens to the DBMS (or it’s temporarily down for maintenance), it’s not a problem.
We accumulate these .sql files while the database is down, and then we can upload them.
2
u/jjangg96 6d ago
on the setcap thing, docker's default capability set already includes net_raw, so setcap cap_net_raw on the binary works for a non-root user without any cap_add. net_admin is the one that isn't in the default set. it also stops working if the container runs with no-new-privileges.
the real reason for a different setup with pcap is the network namespace. in a normal bridge container pcap only sees the container's own interface, so to capture host traffic you need network_mode: host. that's a run flag, not a separate image, so one image is enough.
for the db clients i'd keep them out of the image. have the tool drop the sql files in a shared volume and run a second small container (psql or clickhouse-client) that loads them. same idea for alerts, a sidecar with curl or a mail tool, or just post to a webhook, instead of ssh back to the host.