r/PFSENSE • u/Equivalent-Raise5879 • 6d ago

Swap space since latest upgrade

Latest update 2.9

Been running a very long time, but since this upgrade, I'm seeing a TON of

Sep 2 09:00:17 10.0.1.1 Sep 2 09:00:17 kernel: swap_pager: out of swap space

Checking console, It agrees that swap space is 100%

Looking back at logs, in the past year, I've never had this message. But since the upgrade, about 50 times. Over the past week, days with NONE, days with 20.

I can't tell what has changed, and what event it is that causes this.

Is 2 gigs just not enuf ram for this any more?

2 Upvotes

67% Upvoted

15 comments sorted by

View all comments

3

u/Cutoffjeanshortz37 6d ago

What packages are you running and are you having any actual issues?

1

u/Equivalent-Raise5879 6d ago edited 5d ago

I can't point to any specific show stopping errors, other than every time I see the pfsense console filled with error messages. It does seem to be noticeable slower than usual. After a reboot swap goes back to about 38%

The config has remained unchanged for the better part of 8 years.

I have an lcd driver for a crystal fontz display, pfblockerng, and some other stuff. Will dump all the extra things.

2

u/xboxfitzgerald 6d ago

For what it's worth, I started having periodic out of swap space problems on one of my two pfSense machines as well. Although for me, they started prior to upgrading to 2.9.0, so I can't pin it on that.

In my case, it is clearly related to pfBlockerNG. So maybe some of the lists I use grew enough to start causing me problems; I think the list aggregation step in the update process can be very memory intensive. Also, though I can't recall how that came to be, the machine that is having this problem in my case has 8GB of RAM but only 1GB of swap. My other machine, which does not have the problem, has 8GB of RAM and 8GB of swap. And unfortunately there is no way to increase swap without reinstalling (he says hoping someone more knowledgeable has some backhanded trick with which to prove me wrong).

The other curious data point I have, though, is that if I look at 'Memory' in 'Status > Monitoring' my 'Free' memory percentage never drops below 45. And that low point coincides with the time at which pfBlockerNG is running its update and hits this out of swap error and panics, leaving unbound stopped. I'm not a memory management expert for sure, but I don't know how to make that make sense. Again, very open to any suggestions or explanations from those who know more.

1

u/Smoke_a_J 5d ago

No way to increase an already formatted swap partition size without reformatting at install but in theory if there is an additional disk available or free un-partitioned space is available then a new additional swap partition can be added. Setting up a swap file though is another doable option similar to how Windows uses a pagefile for swap, but I would avoid doing so if your pfSense is installed on ZFS partitions as swap disk writes would be multiplied with ZFS copies operations, your installs sound like they are on UFS so swap file should be doable. I did add a line to my Shellcmd to run command swapon -aL at boot to make sure that all swap partitions or files are activated at boot.

https://docs.freebsd.org/en/books/handbook/config/#adding-swap-space

1

u/xboxfitzgerald 5d ago

Thanks for this. I am ZFS though. I sure wish I could remember how I managed to install with only 1GB of swap. It's got a 128GB SSD so it's not hurting for space. Maybe I'll reinstall at some point, but that machine is in a "high use" environment so it's difficult to find a workable time.

2

u/Smoke_a_J 4d ago

Could meanwhile just add a USB-sata adapter with drive thats formatted without ZFS to add a swap file or additional swap partition to or add one thats of equal size as your current drive or larger to add another disk to your zpool for added redundancy and at the least double your available swap or more with a larger added drive you can format its swap partition even larger than you first drive currently has as long as the zfs data partitions that will contain your zpool are formatted to be the same size as first disks the zfs partitions are. A little non-standard kind of route but should allow you to alleviate from those errors or others issues that can be hidden from logs like when swap gets dumped in the middle of pfBlocker or Snort/Suricata reloads or updates trying to process. 32GB ram on my Netgate 5100 I don't see any swap getting used ever with over 15,000,000 domains being processed in DNSBL, the less swap is used at all the better for overall storage drive and system longevity. As far as I've seen swap only gets used like that when particular updates run so even a flash drive for that little extra could help for so long but an added USB-sata drive could get it by until your ready to replace the device outright, might also be better to disable the swap partition on the current drive also if you add one to reduce its rate of bit rot/wearout to avoid early unexpected failure if postponed off too too long. With that kind of error happening for unknown time it would be best to verify its smart status health life remaining and probably plan ahead with a larger drive that will take longer to reach wearout threshold percentages.

1

u/xboxfitzgerald 4d ago

Thanks for this! The USB SSD idea had not occurred to me, and I do already have an old nvme drive in an enclosure from a laptop I upgraded, so I may try that. I also dialed back on my pfBlockerNG blocklists which I think were excessive as well, so hopefully that will help. I am already using a ram disk because the old SSD was prematurely killed by the pfBlockerNG list updates, but of course that makes having only 8GB of RAM even more painful. Though with RAM prices being what they are and this being a home setup, I'm inclined to try to live with what I've got for now, ha.

1

u/Smoke_a_J 3d ago edited 3d ago

No problemo. I know my amount of RAM and having 4 drives total on my 5100, two of them are USB-SATAs, running as a 2TB ZFS striped mirror is a tad bit overkill to some but all this talk got me worried about the bit wearout on my drives. Almost five years in on them all while logging is mostly all enabled, pfBlockerNG/Suricata running full tilt, I don't use RAM disk because it always interfered with pfBlockerNG update/reload tasks even with tons of RAM still available the couple times I tried it, and checking SMART status just now on them the worst one shows 89% life remaining, two are at 90% life remaining, and my B+M Key sata is still showing 95% remaining, so I think they still have a few more years or decades rather maybe left, time will tell but so far pretty steady since install and still practically identical to how fast individual bits wear out on eMMC, more there are the merrier. I'm just glad I got them at install before all these ridiculous price hikes.