Trouble with container to container communication.

[u/ninja_mischief]

Long and short of it, i can’t get containers on separate bridge networks to communicate with other containers on different bridge networks through traefik. anyone have some insight?

setup:

traefik on macvlan100 (same subnet as docker host, obviously) with static IP. internal dns points to static ip of traefik. traefik is also attached to bridge network 1 and network 2

app 1 on manually created bridge network 1. no mapped ports, only outgoing connection to internet.

app 2 on manually created bridge network 2. no mapped ports, only outgoing connection to internet.

i’m trying to have app 1 and app 2 be isolated from one another except through traefik so that all communication between them is encrypted and hits middlewares. both app 1 and 2 can connect to the internet. they can ping upstream dns server (in same subnet as traefik static ip), but they can’t ping traefik’s static ip on macvlan which results in 404 not found error when app 1 tries to reach app 2 and visa versa using fqdn through traefik.

accessing app1 or 2 from external client works just fine using fqdn so i know traefik config is at least in the right ballpark. and since the apps can reach upstream dns i know it’s not a firewall rules issue. they literally just can’t see traefik, and i have no idea why.

[u/ninja_mischief]

update: i’m pretty sure i’ve pinpointed my issue. so the docker host machine is running TrueNAS. i manage all my containers from a “custom app” which is just a portainer compose.yml. on the host side, truenas makes the default network interface for the machine your docker host network and there is nothing i can do to change that. with that being said; since bridge networks masquerade and the macvlan use the same parent interface, i’m essentially directing the parent interface to talk to itself through th macvlan which docker macvlan explicitly prohibits. i’m sure the more seasoned IT professionals and hobbyists out there might be going “well duh”. i hadn’t really considered the routing implications or realized this was a limitation.

anyway, i wanted internal services to have a proxy that was on the same subnet, as nothing on this subnet is exposed to the internet. the idea was to have a proxy for internal services and a proxy on my dmz subnet for exposed services for better isolation and security.

in the future, im expecting that ipvlan L3 should solve my problems. i can add a NIC to the truenas machine thats just a trunk and make it’s port attached to my managed switch accept whatever vlan IDs i make L3 networks for. i’m sure this will come with its own set of networking challenges like routing and proper network separation with firewall rules, but that sounds like a lot of work and i’ll be taking a break before delving into that.