Date: Tue, 26 Oct 2021 10:13:26 -0700 From: Chris <bsd-lists@bsdforge.com> To: Marcel Bischoff <marcel@herrbischoff.com> Cc: freebsd-pf@freebsd.org Subject: Re: "pfctl: Cannot allocate memory" issue with a large table Message-ID: <fd8751a44b140fb927db1c4009456eff@bsdforge.com> In-Reply-To: <YXRXm4yCW9kblseH@herrbischoff.com> References: <YXRXm4yCW9kblseH@herrbischoff.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--=_b89be43f18cf44680f7a50132dbece09 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; format=flowed On 2021-10-23 11:42, Marcel Bischoff wrote: > Hi all, > > for some time now I'm using the excellent (in my opinion) pf-badhost script > [https://geoghegan.ca/pfbadhost.html] to create default blocklists for some > servers. When using IPv6 and/or geoblocking with it, I often run into the > "pfctl: > Cannot allocate memory" error upon replacing the table contents. > > The list contains about 300k+ lines with IPs and CIDRs. It is properly > aggregated > so the net blocks are compacted as far as possible into CIDR notation. Only > single > IPs are listed without a "/32" or "/128" suffix. > > /etc/pf.conf contains > >> set limit table-entries 1000000 > > /boot/loader.conf contains > >> net.pf.request_maxcount=1000000 >> kern.maxdsiz="2147483648" > > /etc/sysctl.conf contains > >> net.pf.request_maxcount=1000000 > > "pfctl -s memory" shows the limit is active: > >> states hard limit 100000 >> src-nodes hard limit 10000 >> frags hard limit 5000 >> table-entries hard limit 1000000 > > During my research I found out that replacing a pf table temporarily needs > double > the memory as both the old and new states are held before the old is > discarded. > This makes entirely sense to me. What I don't understand is why the error > still > occurs despite the proper limit being set. > > Does anyone have an idea how I can resolve this? It is entirely possible > this > happens due to me not entirely understanding how memory allocation in pf > works. > However, I haven't found anything particularly applicable either in the > Handbook > or the "pf.conf" man page. Have you reached your STATE limit? OTOH you might try adding the IPs from the list individually. Something like: iplist=" w.x.y.z a.b.c.d ... g.h.i.j " for block in $iplist do pfctl -T add -t <your-table-name-here> $block done I'm managing about a half dozen tables with a combined number of a over quarter of a billion addresses, and don't have a problem. Even on a servers with as little as 8GB RAM. HTH --Chris > > Best, > Marcel --=_b89be43f18cf44680f7a50132dbece09 Content-Transfer-Encoding: 7bit Content-Type: application/pgp-keys; name=0xBDE49540.asc Content-Disposition: attachment; filename=0xBDE49540.asc; size=5028 -----BEGIN PGP PUBLIC KEY BLOCK----- mQENBGDTzGEBCADHlXdS4V57s2soaEK2wi3o9rr9zo7to/giBSxCpFYJxOnPkL5A 2ibbvflrL8sWvAczx47wgDS7iIhzICBBRdnXtcFGnoeeriV27LSn+PcpnIB+DaWZ xe+6TDC0Z0JUJ7qDTjUBFzhnQGYlrVvc4WbnWTjJaB1LEwgIX8JqX5S3SX0/oXgs +OtqDuENZ4/a5te5xPnspTv/5NJHjqYGxjHP0Vw0KjRKS1AoJ1SBPSMQV5373AX9 5NzFS+CjqeQhjfHFPeRajQ8t4T6eqhKA7LtKMO1egeAwNehk9ZoEqEBT2+ojuKUd oSuzqvhhx+eUIYLFqoPSzMKR+YbStzergsbnABEBAAG0KUNocmlzIEh1dGNoaW5z b24gPGNocmlzaEB1bHRpbWF0ZWRucy5uZXQ+iQFrBBABCABVBgsJBwgDAgQVCAoC AxYCAQIZAQIbAwIeARgYaGtwczovL2tleXMub3BlbnBncC5vcmcWIQQGJAsyyBlk cuwsSYsYdR58veSVQAUCYNQl+wUJA8LAmgAKCRAYdR58veSVQN3NB/sFTeXrZeDk ml/dshET8QbkOPgXlnibk8+Mauf+y9LjS9WT7R8EmqhK7T7aw115JQ1RWTM6kpQM jyDBjYF7piJEpNKI9YDeSnODKir1fWQqm9+wd68wAKGvV4m8kg9uOHCvXG4J++MG zDFH+PuGVxKirFnaz46DpS0Zw7wTtjNiNFvCooYov3IeYGfqcchd3hwBuXgWLexZ vI8JW7lL9oXl7B/wcbSxg9rwy6/QLYGg6sEtYRcFYyvQWefSMJaLWjU/pZN2iSxM lXm55iZv1BXHupfeD1ldRiGs6ejrcpa8+U1ju291WbLzcIsU8IDljeW9/WB2dLFT hJmY1wRk158AtB5DaHJpcyA8YnNkLWxpc3RzQGJzZGZvcmdlLmNvbT6JAWgEEAEI AFIGCwkHCAMCBBUICgIDFgIBAhsDAh4BGBhoa3BzOi8va2V5cy5vcGVucGdwLm9y ZxYhBAYkCzLIGWRy7CxJixh1Hny95JVABQJg1CX7BQkDwsCaAAoJEBh1Hny95JVA aI0H/AlJAOfc5TcMKa479Itw31mwccKb+u0DPN9Gkm/RfWIBjeqqozxCM8G8jVFr dt/J6KmBO3dQtRZHlXdD57RAfDDl5Vm3uws0s+UIFOxMiua/YxyuDcKLsE8Bjkzx z+vuJ8f6cg4WlygPr3bo3l81AOuU/wOsTrNkQvVJxgATlooATSVxs0yNn2uoso9f nhMGUYsmT4c35JYh0k6Lq7Z2LS+ELipMTQ7M7iCWSP1O/zSEvPD4NBo52xCvjLka KcL4fRl7UN+6ouwGr5aUn83tztE/IR0AK45gFvL5yxI4g/zm1t3j2+hhhW1pBU8w uQWkD2DyLTWy7xs1uVF5m1ojHp60H0NocmlzIDxrbm90QHRhY29tYXdpcmVsZXNz Lm5ldD6JAWgEEAEIAFIGCwkHCAMCBBUICgIDFgIBAhsDAh4BGBhoa3BzOi8va2V5 cy5vcGVucGdwLm9yZxYhBAYkCzLIGWRy7CxJixh1Hny95JVABQJg1CX7BQkDwsCa AAoJEBh1Hny95JVA5m8H/iENaTD4j5QHfaHfiDIdxGx36GnETyRK0vAzr2b6pzG+ 7VHNCm4ZfuMsXDJ1ZD8fjTipvg0f4w31xCQI0NgNdAqudBqE075Jwcr9pE9j8VN1 Nvejto01cgLHODbLPhokrkFz1K023VjCdy5RaVuCZ6ajTif7Kq+BEOE8TumYx4ly zdhnh/9ICohqfVvEMh347wI36D7HuezHB773hOsHdqTy9T+0Qu0Vu+wud45MUy1f vRF11OkJFtKL0bh4yMSGVY1xte1Mt/qC6rd43TDtAW3ekw1o/exh764kp7XXQsmP wwe4Y040PZafcygJlEW9bBtjjxKnzDTvqeb5dMi6d7a0GENocmlzIDxvaWRldkBz dW5vcy5pbmZvPokBaAQQAQgAUgYLCQcIAwIEFQgKAgMWAgECGwMCHgEYGGhrcHM6 Ly9rZXlzLm9wZW5wZ3Aub3JnFiEEBiQLMsgZZHLsLEmLGHUefL3klUAFAmDUJfsF CQPCwJoACgkQGHUefL3klUB74wf8DSvT36bYZp7oqZ+35HNhTekJ2dbTzUhauF0S +Z9R1AGnNnINgua75CyQGdNCIgcZxo4qG9sePl7SllQ9i0qhmiw0mzmvky8bAZQV V/2Coc1C/81b+PI19VczYrbZC20jApsnbAIkKZgSh9XQoiLd3meY7G2lX2k6CXYL xSeBEh+N3BU8vLxExm82U71Qzm43u0kA1TlbTSqpBvg/tfAzTCsYQLSlB6b4ZL2W D6U7b7ZYF5oZNonVNWSHxpjUN3Evkta9xWS2+cgYQdlP1/ku5w5ZWwzmYG7awh0J /YuSNIp6Ks6D/PSBduu6XbH+FJHaXmq+ZCKpNBh5EKH+GhOfq7QfQ2hyaXMgPHBv cnRtYXN0ZXJAYnNkZm9yZ2UuY29tPokBaAQQAQgAUgYLCQcIAwIEFQgKAgMWAgEC GwMCHgEYGGhrcHM6Ly9rZXlzLm9wZW5wZ3Aub3JnFiEEBiQLMsgZZHLsLEmLGHUe fL3klUAFAmDUJfwFCQPCwJoACgkQGHUefL3klUC3GggAo4Y+hslaoV7Namp7qWYZ Vei4ZwPfsYW7/HtmFORSGV8C8xR+LSkwzN1Hc7Qxvwv+DXuk7Hzd1Ag/xe8XhbNG /NMrXENY/8ym9TRbxtrBIhQyhkyShSUT+N+g16GRNZKuNL2MOIHc/RCS/YyyaTtu TzIxFbP7Gb2LO1LiiZsFVOGirHfxyiww7CAm3HXY2K4smOiKs6swZMpStVy3dd6A BcB1LPGs3ywDglFfKCRbVmjsPgsi61r4kUBVO6ML7lAmPDXLXOa+7iAtBN479QxC MVeH3Y3SMrvu61Vyf1xL79rIznU3u8C34zfxqsoIV0zCZe2YDLbFfLhZYqatYYEo e7QjImNocmlzLmgiIDxjaHJpcy5oQHVsdGltYXRlZG5zLm5ldD6JAWgEEAEIAFIG CwkHCAMCBBUICgIDFgIBAhsDAh4BGBhoa3BzOi8va2V5cy5vcGVucGdwLm9yZxYh BAYkCzLIGWRy7CxJixh1Hny95JVABQJg1CX8BQkDwsCaAAoJEBh1Hny95JVAkUEH /jkzYrRh7muqoebwEgVeULzPbAs/nYJm9SMME2ypB2FS8kusO7lE+33UJO7PhHkJ 0nJ+tPfP8UV+fCzVjKjabzpvUGuiMWKRZEK9xNoxwi/epOrRw87msHA2LPqEob+F sVh09Nc58s75koUgSYp5h0FjsLK0+fwsQ6PtTfpY5W6JJVJRQnMwGKk5czrukBSM 79kJvphgul2xuzqo5K7rM98dL75AwCJmJZnbyXpUJIhtY/G01nURupBiQGgNixYs Zeo6OR669TFrMRWxueXtlHD0WaX7JNSlR5uyzpVaDCH0Kxa6ozmZtD+a6dAXg630 zbLGHg51JIm38Uvi1i47Jaa0KCJILlIuIENvbW11bmljYXRpb25zIiA8ZG5zQGRu c3dhdGNoLmNvbT6JAWgEEAEIAFIGCwkHCAMCBBUICgIDFgIBAhsDAh4BGBhoa3Bz Oi8va2V5cy5vcGVucGdwLm9yZxYhBAYkCzLIGWRy7CxJixh1Hny95JVABQJg1CX8 BQkDwsCaAAoJEBh1Hny95JVAABoH/iOWA+9BKxLIAIFgW2nxTFDrGvbxXL/mVSFt SOInKX8UqqfLCcikfpWLsj2D7mg5rKFMCu+31UYYlnrXl4YY1qruq0vh41L72qNy yHYol+xW4BSbZXf2q2ph7+lnPsFoodw7acVun5F8M8NH0roo5AOSbgRlK69ZFIcq fDEJdtk4oul7pqGArdeTCCdrSaeR3zrRN8P0PDOkGKSdlpeOE6XHnbbmAPZIhr/9 KsSpX1BGyipda3k5kOB4TsGVo+cRJMkK+GMpsZ+lJ7ZzRbjHbC+b52TiAIjMtXCK 3A3LrDUeMoJwvRKoO1tzquF6HqHJSg0ArZOvAB3BHlwUyUtA/o25AQ0EYNPMYQEI ANFpucNRdYEOubTNluoK97N9JmDb0WRXPPow+3XfBom6ZBSrWqNBgqDbjxSsLB00 QXbA8EB5W/Oolp/0epwEtgNAxyKVPowE/un+rY1PqvGjeAR4gBhY9Za1Lg1Q3vnR /WzsY7RIQCqhWUbfdGn1u6r/EgTBVrwUp4U/3ggfSz/PcUt4pUhlgxfYvjSjOgEZ wbqaQIwWud11FKMARNAUJzvJL/fDGeKLMvgRUwynIDGzCq7e67hhEEo5jwkZ0gEl 8RxXHKFuYkbb/q7rpdifXYYT6QCFlEZhiRbtH5Us7kgKuRD2XUFEQnN4U/rxuydH 4XOP6iOhiZfYnK/y9HBeRCMAEQEAAYkBPAQYAQgAJgIbDBYhBAYkCzLIGWRy7CxJ ixh1Hny95JVABQJg1CYkBQkDwsDDAAoJEBh1Hny95JVApBsH/iEg2ANRkHByfXB+ sH3PMf2Jsg5NSuj8OiNeKKGGIKCJkSAPjtv5rvKLNcvIcTR5Vnhr0e6AteFcK2te iFWDmj0QuFoQNvIOHQ3nHBPSpai2Ubq12nvYfg4bYK28AMi4xPMssgQ8awFgAI2V k9okq5XwC0Cc1MGhupEWYYSaFLIDQvFvRRSw1Lyc/W3SKa4d2dgesIPnB/rdv0Zq u8ftsSmurKxA2hQeNIcn06Ew7AbWUIjFX/bDXJlg/3Sj/spU2ur23TmaADBKhT5P DvfdaFTkk0SBfpN1j2S0DNXBHSrWvRp15zZmU4hwELiUY/H2/j/XpOGV3Q0i2iob 1hJ30C8= =aMQi -----END PGP PUBLIC KEY BLOCK----- --=_b89be43f18cf44680f7a50132dbece09--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?fd8751a44b140fb927db1c4009456eff>