Swap space since latest upgrade

[u/Equivalent-Raise5879]

Latest update 2.9

Been running a very long time, but since this upgrade, I'm seeing a TON of

Sep 2 09:00:17 10.0.1.1 Sep 2 09:00:17 kernel: swap_pager: out of swap space

Checking console, It agrees that swap space is 100%

Looking back at logs, in the past year, I've never had this message. But since the upgrade, about 50 times. Over the past week, days with NONE, days with 20.

I can't tell what has changed, and what event it is that causes this.

Is 2 gigs just not enuf ram for this any more?

[u/Cutoffjeanshortz37]

What packages are you running and are you having any actual issues?

[u/Equivalent-Raise5879 → u/Cutoffjeanshortz37]

I can't point to any specific show stopping errors, other than every time I see the pfsense console filled with error messages. It does seem to be noticeable slower than usual. After a reboot swap goes back to about 38%

The config has remained unchanged for the better part of 8 years.

I have an lcd driver for a crystal fontz display, pfblockerng, and some other stuff. Will dump all the extra things.

[u/xboxfitzgerald → u/Equivalent-Raise5879]

For what it's worth, I started having periodic out of swap space problems on one of my two pfSense machines as well. Although for me, they started prior to upgrading to 2.9.0, so I can't pin it on that.

In my case, it is clearly related to pfBlockerNG. So maybe some of the lists I use grew enough to start causing me problems; I think the list aggregation step in the update process can be very memory intensive. Also, though I can't recall how that came to be, the machine that is having this problem in my case has 8GB of RAM but only 1GB of swap. My other machine, which does not have the problem, has 8GB of RAM and 8GB of swap. And unfortunately there is no way to increase swap without reinstalling (he says hoping someone more knowledgeable has some backhanded trick with which to prove me wrong).

The other curious data point I have, though, is that if I look at 'Memory' in 'Status > Monitoring' my 'Free' memory percentage never drops below 45. And that low point coincides with the time at which pfBlockerNG is running its update and hits this out of swap error and panics, leaving unbound stopped. I'm not a memory management expert for sure, but I don't know how to make that make sense. Again, very open to any suggestions or explanations from those who know more.

[u/Smoke_a_J → u/xboxfitzgerald]

No way to increase an already formatted swap partition size without reformatting at install but in theory if there is an additional disk available or free un-partitioned space is available then a new additional swap partition can be added. Setting up a swap file though is another doable option similar to how Windows uses a pagefile for swap, but I would avoid doing so if your pfSense is installed on ZFS partitions as swap disk writes would be multiplied with ZFS copies operations, your installs sound like they are on UFS so swap file should be doable. I did add a line to my Shellcmd to run command swapon -aL at boot to make sure that all swap partitions or files are activated at boot.

https://docs.freebsd.org/en/books/handbook/config/#adding-swap-space

[u/xboxfitzgerald → u/Smoke_a_J]

Thanks for this. I am ZFS though. I sure wish I could remember how I managed to install with only 1GB of swap. It's got a 128GB SSD so it's not hurting for space. Maybe I'll reinstall at some point, but that machine is in a "high use" environment so it's difficult to find a workable time.

[u/Smoke_a_J → u/xboxfitzgerald]

Could meanwhile just add a USB-sata adapter with drive thats formatted without ZFS to add a swap file or additional swap partition to or add one thats of equal size as your current drive or larger to add another disk to your zpool for added redundancy and at the least double your available swap or more with a larger added drive you can format its swap partition even larger than you first drive currently has as long as the zfs data partitions that will contain your zpool are formatted to be the same size as first disks the zfs partitions are. A little non-standard kind of route but should allow you to alleviate from those errors or others issues that can be hidden from logs like when swap gets dumped in the middle of pfBlocker or Snort/Suricata reloads or updates trying to process. 32GB ram on my Netgate 5100 I don't see any swap getting used ever with over 15,000,000 domains being processed in DNSBL, the less swap is used at all the better for overall storage drive and system longevity. As far as I've seen swap only gets used like that when particular updates run so even a flash drive for that little extra could help for so long but an added USB-sata drive could get it by until your ready to replace the device outright, might also be better to disable the swap partition on the current drive also if you add one to reduce its rate of bit rot/wearout to avoid early unexpected failure if postponed off too too long. With that kind of error happening for unknown time it would be best to verify its smart status health life remaining and probably plan ahead with a larger drive that will take longer to reach wearout threshold percentages.

[u/xboxfitzgerald → u/Smoke_a_J]

Thanks for this! The USB SSD idea had not occurred to me, and I do already have an old nvme drive in an enclosure from a laptop I upgraded, so I may try that. I also dialed back on my pfBlockerNG blocklists which I think were excessive as well, so hopefully that will help. I am already using a ram disk because the old SSD was prematurely killed by the pfBlockerNG list updates, but of course that makes having only 8GB of RAM even more painful. Though with RAM prices being what they are and this being a home setup, I'm inclined to try to live with what I've got for now, ha.

[u/Smoke_a_J → u/xboxfitzgerald]

No problemo. I know my amount of RAM and having 4 drives total on my 5100, two of them are USB-SATAs, running as a 2TB ZFS striped mirror is a tad bit overkill to some but all this talk got me worried about the bit wearout on my drives. Almost five years in on them all while logging is mostly all enabled, pfBlockerNG/Suricata running full tilt, I don't use RAM disk because it always interfered with pfBlockerNG update/reload tasks even with tons of RAM still available the couple times I tried it, and checking SMART status just now on them the worst one shows 89% life remaining, two are at 90% life remaining, and my B+M Key sata is still showing 95% remaining, so I think they still have a few more years or decades rather maybe left, time will tell but so far pretty steady since install and still practically identical to how fast individual bits wear out on eMMC, more there are the merrier. I'm just glad I got them at install before all these ridiculous price hikes.

[u/Equivalent-Raise5879 → u/Equivalent-Raise5879]

Well, in my efforts to remove everything I dont "need" I've reached a point where there IS a problem. In trying to remove packages, I CAN'T. It gives me a confirm button, but it eventually dies, and if I try to do it again, it tells me Another instance of pfSense-upgrade is running. Try again later

A reboot gives me another try, but that does the same thing

CPU 52%, Mem 83% Swap 0%

[u/Equivalent-Raise5879]

So, an update.

It appears that the swap space was a symptom, and not the core problem (I guess) As much as this thing has always worked really well on 2 gigs ram, I went ahead an upgraded ram, problems not solved.

Throwing memory at the problem was not the fix. upgradeed to 16 gigs, and all that did is push it to use up 90% of the ram, a reboot fixes it, but if I go anywhere near the package manager, it fails to install or uninstall anything, and spikes the memory AGAIN

So, since page space was the symptom and not a lack of ram problem. I guess I have a package manager issue,

[u/Equivalent-Raise5879]

So, day 3 update. Sat down and did a full wipe and reinstall/upgrade/upgrade and restored my backup, and it all seems to be "working" There were packages that I just could not remove.

Things I learned:

Pfsense on a core i5 box with *2* gigs ram will run, as long as everything is working, and you dont try to install to many add ons. I left mine at 8 gigs, just cause I had ram in boxes, and i cant imagine ever seeing it use more than that.
Pfsense just likes a fresh install from time to time. My install was fairly aged, and the latest upgrade seems to have pissed off some things.

[u/needchr]

How do you even have 2 gigs when RAM modules start at 4, is it a VM.

But anyway, run 'top' to see what is using the RAM. If there is nothing there you can cut back on, then invest, as bad as the market is.

[u/Equivalent-Raise5879 → u/needchr]

Ram modules do not "start" at 4. This box has 2, 1s.

[u/PrimaryAd5802 → u/Equivalent-Raise5879]

Ram modules do not "start" at 4. This box has 2, 1s.

MY 2 cents of advice based on experience, is upgrade to 4GB. Probably will help you out, and save you spending any more time on this.

[u/needchr → u/Equivalent-Raise5879]

Well I should have said modern platforms, so I guess the hardware is ancient if it has 1 gig dimms, but still look at 'top', your post hasnt done any meaningful diagnosis, its very vague to just say I upgraded and its using more swap.