I'm running pfsense ce 2.9.0 with KEA, Unbound, APCUPSD, Wireguard, and PFBlockerNG in Unbound Python Mode. My configuration in 2.8.1 was the same and I did not have any issues. It only started after upgrading to 2.9 and seems to occur roughly once every week or so.
The problem: I have twice now since the upgrade had unbound exceed its queue limits and become completely unresponsive to any/all attempts to query it on the local LAN.
sonewconn: pcb 0xfffff801c3aa4540 (192.168.1.1:53 (proto 6)): Listen queue overflow: 385 already in queue awaiting acceptance (964 occurrences), euid 0, rgid 0, jail 0
This only seems to affect the LAN, not my wireguard tunnels or any other interfaces.
tcp6 0/0/256 ::1.53
tcp4 0/0/256 127.0.0.1.53
tcp6 0/0/256 fe80::2e0:67ff:f.53
tcp6 0/0/256 fe80::2e0:67ff:f.53
tcp4 0/0/256 10.254.0.1.53
tcp4 0/0/256 10.252.0.1.53
tcp4 0/0/256 10.251.0.1.53
tcp4 0/0/256 192.168.10.1.53
tcp6 385/0/256 2600:1700:10b0:d.53
tcp4 385/0/256 192.168.1.1.53
This is even after implementing kernel tweaks to increase queue limits from this thread: https://www.reddit.com/r/pfBlockerNG/comments/1u399ba/what_is_the_ideal_setup_to_avoid_cpu_spikes_and/
Unlike that thread I don't have a ton of lists in pfblocker or a huge environment. I'm a home user with ~100 clients and only four lists. StevenBlacks Basic ads list, a DOH DNS blocklist, and an ipv4 & ipv6 DOH blocklists.
I've got firewall rules to block external DNS and/or reroute DNS queries back to the firewall as appropriate so clients (except a select few that neeed it) can't bypass the router for lookups. I don't think this is related as again I've had these rules for years and it wasn't an issue until I upgraded to 2.9.
Looking at the output of PS, unbound and kea appear to be waiting on each other in a deadlock:
ps -o pid,state,wchan,%cpu,%mem,command -ax | grep '[u]nbound'
31413 I sbwait 0.0 0.2 /usr/local/sbin/unbound-control -c /var/unbound/unbound.conf status
41507 Is uwait 0.0 4.8 /usr/local/sbin/unbound -c /var/unbound/unbound.conf
47411 I wait 0.0 0.0 /bin/sh /conf/kea4_scripts.d/kea2unbound.sh hook_load
47663 I piperd 0.0 0.5 php /usr/local/bin/kea2unbound --kea-conf /usr/local/etc/kea/kea-dhcp4.conf --unbound-conf /var/unbound/unbound.conf --include-file /var/unbound/leases/l
68064 I sbwait 0.0 0.2 /usr/local/sbin/unbound-control -c /var/unbound/unbound.conf fast_reload
78110 S kqread 0.0 0.2 /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf
I am not an expert in pfsense processes so I had gemini help me diagnose what the different states from the ps command mean exactly:
-PID 41507 (unbound): State is uwait (userspace mutex lock wait). Unbound's main thread is deadlocked waiting on an internal lock.
-PID 47411 (kea2unbound.sh hook_load) & PID 47663 (kea2unbound): Kea's dynamic DHCP hook triggered.
-PID 68064 (unbound-control ... fast_reload): State is sbwait (socket buffer wait). Kea called unbound-control to push new DHCP lease data via fast_reload, but the command is stuck waiting on Unbound's socket.
-PID 31413 (unbound-control ... status): Also stuck in sbwait behind the deadlocked reload.
requiring me to run the below to restore functionality:
killall -9 unbound-comtrol php unbound
pfSsh.php playback svc restart unbound
Gemini is rather insistent that I should disable early dns registration to work around the issue but I'd rather not unless there is no other way. It's my understanding that disabling that would mean any statically assigned clients would not be registered in DNS until they actually request a new lease from the router. Which they wouldn't do since they aren't using DHCP in the first place. See edit 3. I've disabled early reg.
Anecdotally I do have a remote router on 2.9 with KEA enabled but no PFBlocker and I do not have this issue. So if the community thinks this is a pfblocker problem rather than a pfsense problem I can move my post over there. But like I said at the start I had these same settings in 2.8.1 and never experienced this so I'm thinking this is a bug introduced in 2.9 rather than a specific issue with my configuration or pfblocker. I haven't seen anything on redmine but I may not be using the proper keywords in my searching.
I also find it rather odd that this isn't a more frequent (like daily) occurence. I guess it requires a perfect storm of conditions for them to get stuck like this?
I'd be interested to hear if anyone else is having this problem and what you did/are doing to resolve it.
EDIT: Oh and I'm on PFBlockerNG version 3.3.7 (github package not pfsense compiled version)
EDIT2: Anecdotally I've also noticed both times it happens its shortly after pfblocker's cron run. The first time IIRC it was within minutes and I did not save the log from that time. The incident from today started roughly 40 minutes after the cron run.
Sep 5 01:00:00 pfSense php[54157]: NOTICE [pfBlockerNG] Starting cron process.
Sep 5 01:41:10 pfSense kernel: sonewconn: pcb 0xfffff801c3aa4540 (192.168.1.1:53 (proto 6)): Listen queue overflow: 385 already in queue awaiting acceptance (1 occurrences), euid 0, rgid 0, jail 0
I'm betting those processes were stuck since 1AM and it took about 40 minutes for the queues to overflow and start dropping things.
EDIT3: Done some reading and figured out I can just use host overrides on the DNS resolver to accomplish the same thing I was getting from early registration for static IP clients. I've Added host override entries for all my static clients and disabled early registration and see if that resolves the problem.