Date: Sat, 11 Dec 2021 18:04:17 +0100 From: Marcel Bischoff <marcel@herrbischoff.com> To: tech-lists <tech-lists@zyxst.net> Cc: freebsd-questions@freebsd.org, freebsd-pf@freebsd.org Subject: Re: pf cannot allocate memory after a time Message-ID: <E31F66A5-5014-4A53-B5BD-B0F38A6F18EB@herrbischoff.com> In-Reply-To: <YbTOficBUC8vhklu@ceres.zyxst.net> References: <YbTOficBUC8vhklu@ceres.zyxst.net>
next in thread | previous in thread | raw e-mail | index | archive | help
I've stumbled into this issue some time ago as well. The usual remedies = of raising states and table-entries did not help. I could resolve this with a combination of increasing the process memory = limit and lowering ZFS memory usage. I'm guessing that you do use ZFS, = since you have an 8 GB Pi, if not please disregard. FreeBSD limits the memory available per process to 512 MB by default. It = appears that large PF tables cause issues with this default. Raising = that limit to 2 GB seems to have done the trick. To attempt this, add = the "kern.maxdsiz" tunable to /boot/loader.conf and reboot, like so: kern.maxdsiz=3D"2147483648" You can verify this value has taken by examining the output of "ulimit = -d" before and after. See "man tuning" for reference. Sometimes PF still tripped up when replacing one large table with = another (via pf-badhosts). Usually this happened when the generated list = to replace grew to several MB in size. I've read somewhere that PF needs = double the memory to replace a table since for an instant it needs to = hold both tables in memory, which makes sense to me. You can verify that = this is the case by first dropping and then recreating the pf-badhost = table with all entries. That usually works, while replacing (what = pf-badhosts does) usually does not. In those cases I found the ZFS ARC cache to be the culprit. By default = this grabs most of the available memory to cache file access and frees = it on application request. Again, PF appears to require the memory = quicker than ZFS appears to be able to free it. Usually the default ARC = settings are fine but in this case you may want to tune them for testing = purposes, again in /boot/loader.conf and reboot: vfs.zfs.arc_max=3D"2000M" vfs.zfs.arc_min=3D"500M" See https://docs.freebsd.org/en/books/handbook/zfs/#zfs-advanced for = reference. These are just the observations I've made and the solutions I've found. = My reasoning may be sound or completely off. Best, Marcel > On 11. Dec 2021, at 17:14, tech-lists <tech-lists@zyxst.net> wrote: >=20 > Hi, >=20 > context: main-n251261-25d0ccbe101 on arm64.aarch64 (raspberry = pi4b/8GB) >=20 > I'm trying to use pf with pf-badhosts = (https://geoghegan.ca/pub/pf-badhost/latest/install/freebsd.txt) and am = seeing what *seems like* a pf problem which has been reported elsewhere = in different contexts ( e.g. = https://forums.freebsd.org/threads/cannot-define-table-cannot-allocate-mem= ory-since-upgrade-to-13-0.80822/ ) >=20 > from pfctl -sa: > [...] > LIMITS: > states hard limit 100000 > src-nodes hard limit 10000 > frags hard limit 5000 > table-entries hard limit 25400000 [*] > [...] >=20 > [*] the pf-badhosts guide quotes 400000 for this value; I bumped it to = 25400000 in order to "give pf more memory" >=20 > The problem is that if pf tables either get reloaded or if the machine = is running for say over 24 hrs, pf throws errors. This works if the = machine is rebooted but pf isn't switched on: >=20 > [...] > # doas -u _pfbadhost pf-badhost -O freebsd = Password: >=20 > pf-badhost 1512 - - Using experimental "aggy" aggregator... >=20 > 6105 addresses added. > 6235 addresses deleted. >=20 > pf-badhost 1580 - - > IPv4 addresses in table: 619200750 > [...] >=20 > running pfctl -e -f /etc/pf.conf loads and runs. A day or so later, = I'll see this in the logs, after pf-badhost runs its update: >=20 > [...] > pf-badhost 15202 - - Using experimental "aggy" aggregator... >=20 > pfctl: Cannot allocate memory. >=20 > pf-badhost 15256 - - ERROR: '/etc/pf-badhost.txt' contains invalid = data! Reverting changes and bailing out... > [...] >=20 > There's plenty of memory. I've tried running this with one term on top = -P open and there's always 1-2GB available (free) as well as 12GB of = swap which is unused. >=20 > If I try pfctl -Fa -f /etc/pf.conf and log back in and then run = pf-badhost manually: >=20 > [...] > # doas -u _pfbadhost pf-badhost -O freebsd [...] >=20 > not only the pfbadhost table doesn't load but nothing loads: >=20 > [...] > # pfctl -e -f /etc/pf.conf > /etc/pf.conf:18: cannot define table pfbadhost: Cannot allocate memory > /etc/pf.conf:23: cannot define table rfc6890: Cannot allocate memory > /etc/pf.conf:26: cannot define table gooDNS6: Cannot allocate memory > /etc/pf.conf:27: cannot define table friends: Cannot allocate memory > pfctl: Syntax error in config file: pf rules not loaded > [...] >=20 > The only solution is a reboot. How to fix? Do I need to increase = src-nodes/frags? >=20 > thanks, > --=20 > J.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E31F66A5-5014-4A53-B5BD-B0F38A6F18EB>