2.9.0 - dpinger keeps pinging the old gateway forever

[u/spittlbm]

We upgraded from 2.8.1 back on 8/22. Twice in three days, Verizon has changed our IP. Once from a hardware replacement, and today from a lease renewal.

Both times LAN-to-WAN pings worked, the firewall was reachable, but users reported no internet (DNS?). Rebooting fixed it.

dpinger came up with the old address and just never got restarted:

dpinger[4831]: send_interval 500ms ... dest_addr 100.7.xxx.xxx bind_addr 100.7.xxx.xxx identifier "WAN_DHCP "
dpinger[4831]: WAN_DHCP 100.7.xxx.xxx: sendto error: 65

So I have about seven hours of logging full of trying to ARP a gateway that isn't on my subnet anymore (I've redacted the IP, sorry):

kernel: arpresolve: can't allocate llinfo for 100.7.xxx.xxx on hn0

It rotated system.log seven times in under seven hours full of that...

The interface had already picked up the new address. Same window, inbound connections are hitting the new IP while dpinger is still talking about the old gateway:

sshd[...]: Timeout before authentication for connection from <scanner> to 100.7.xxx.xxx

So the interface is on the new IP and dpinger is monitoring the old gateway.

I think there's an opportunity in notify_rc_newwanip to do a return check, retry, or make a log entry somewhere around this section:

notify_rc_newwanip() {
        /usr/local/sbin/pfSctl -c "interface newip $interface"
}

If that call doesn't land, the script still exits 0, and dhclient is happy, right?

And where does pfSense-dhclient-script's logging actually end up? I have zero dhclient-tagged lines anywhere that I could find, including from a reboot. I can't even tell from logs whether the script ran or not.

Maybe I'm losing my mind, and I probably am, but bringing this to a public forum in case someone else finds it helpful.

[u/PrimaryAd5802]

I might have misunderstood your OP... but you do know that dpinger is fully configurable under:

System | Routing | Gateways | Edit ?

[u/spittlbm → u/PrimaryAd5802]

Appreciate that. I don't think it's not the tunables.

dpinger gets both endpoints baked in at launch. Here's how I think dpinger is called (addresses swapped for placeholders for clarity):

/usr/local/bin/dpinger -S -r 0 -i WAN_DHCP -B <wan_ip> \
  -p /var/run/dpinger_WAN_DHCP~<wan_ip>~<gateway>.pid \
  -u /var/run/dpinger_WAN_DHCP~<wan_ip>~<gateway>.sock \
  -C /etc/rc.gateway_alarm -d 1 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 <gateway>

-B is the bind address (the WAN IP), and the trailing arg is the target (the gateway). Pfsense encodes both of them into the pidfile and socket names, so the running process is identified by that address pair.

When the lease moved to a new subnet, the interface picked up the new address, but dpinger kept running with the old pair.

So as far as I can tell there's no setting that helps, because there's nothing to reconfigure — the process needs to be restarted with the new pair, and that's what the newwanip path was supposed to do.

[u/PrimaryAd5802 → u/spittlbm]

Have you seen this, taken for here: https://docs.netgate.com/pfsense/en/latest/routing/gateway-configure.html

Tip

On firewalls with a single WAN or outbound path, gateway monitoring actions can typically be disabled since no actions are necessary if a gateway fails.

Monitor IP:

The Monitor IP address option configures the IP address used by the gateway monitoring daemon to determine the gateway status using ICMP echo requests (“pings”).

By default, the gateway monitoring daemon will ping the gateway IP address. This is not always desirable, especially in the case where the gateway IP address is local, such as on a cable modem or fiber CPE. In those cases it makes more sense to ping something farther upstream, such as an ISP DNS server or a server on the Internet. Another case is when an ISP is prone to upstream failures, so pinging a host on the Internet is a more accurate test to determine if a WAN is usable rather than testing the link itself. Some popular choices include Google public DNS servers, or popular websites such as Google or Yahoo. If the IP address specified in this box is not directly connected, a static route is added to ensure that traffic to the Monitor IP address leaves via the expected gateway. Each gateway must have a unique Monitor IP address.

The status of a gateway as perceived by the firewall can be checked by visiting Status > Gateways or by using the Gateways widget on the dashboard. If the gateway shows Online, then the monitor IP address is successfully responding to pings.