From nobody Thu Dec 23 14:39:03 2021 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B2F3618FC420; Thu, 23 Dec 2021 14:39:06 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JKXrV2Nspz4qsY; Thu, 23 Dec 2021 14:39:06 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "R3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 18074CD39; Thu, 23 Dec 2021 14:39:06 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id 6610243B3D; Thu, 23 Dec 2021 15:39:04 +0100 (CET) From: Kristof Provost <kp@FreeBSD.org> To: freebsd-arm <freebsd-arm@freebsd.org> Cc: freebsd-questions@freebsd.org, freebsd-pf@freebsd.org Subject: Re: pf cannot allocate memory after a time Date: Thu, 23 Dec 2021 15:39:03 +0100 X-Mailer: MailMate (1.14r5852) Message-ID: <1CEFA3DA-2D46-45B3-ADE6-881A94A4D2E2@FreeBSD.org> In-Reply-To: <YbTOficBUC8vhklu@ceres.zyxst.net> References: <YbTOficBUC8vhklu@ceres.zyxst.net> List-Id: Porting FreeBSD to ARM processors <freebsd-arm.freebsd.org> List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: <mailto:freebsd-arm+help@freebsd.org> List-Post: <mailto:freebsd-arm@freebsd.org> List-Subscribe: <mailto:freebsd-arm+subscribe@freebsd.org> List-Unsubscribe: <mailto:freebsd-arm+unsubscribe@freebsd.org> Sender: owner-freebsd-arm@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="=_MailMate_44FD825A-A4A3-4FF5-A5A9-8E209A467CEF_=" Content-Transfer-Encoding: 8bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1640270346; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rcGL2FhY3AT4HxuXziDCXzcBVhWzlgT0Lmq/cgo4RzE=; b=s16NLDqvWo8BTDsooyxtW25c70mGuxXlVHjP+2YvlP8YI+vId94Lsa5V9QLlijtWlq8eMT GFpisSOG1kSa2s4HMMsMxjfjRUsN6c38Ovl1XqowiW/q3a7Bdb8PAk9nTth4NVd0Nz7WOv Om47LAlM2MOE6SSbAVnEc03jUpM067IrFyzAdj+8p8KCScLqoIE9gmZJywdYqEWi6IgNCu Ak8GoAdldNLdG7fT67ka0PDJyzVrcWvDounefxtt/BMh9V6/kDvNtgsqEKZSzDkZsN+PEv zh7ofz7srtm5A8dyFqdIf8x1yGkqmOgyY4N6+/VcIyfnxxBDRXuDHgNfqi1vTw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1640270346; a=rsa-sha256; cv=none; b=kNhj1SrIw4hngntKe4te345sCvz+l/A3bu5JYoVi0rNJLvs4+yrEXEsLwbGZ0K042uROdV vygMVPwRzB3JngflAMw/wlJKqUcilR2TLJZ/EWMmsILL2QlcEvxTbmz2l1DZg5Sn1Roqsu WofbOYkCcKCwQcRy86+3KweQ3NBcrm7rqmaEKFMlfubeLIX8cUdzb7QowME0JsVwOGUboX hNORHy5pZRfYoQponh6JmDkduLSUOeAynspiuCutAER+heVMY11ZfBxr0hsU2Wr5EsfJlF jBsISKjS/cKBza4FNu9ciRwZPWsNRhGE05k7+Q+ZAfv8f2rikZ6/Pn4xNxoB7g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N Status: O Content-Length: 4286 Lines: 104 --=_MailMate_44FD825A-A4A3-4FF5-A5A9-8E209A467CEF_= Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 11 Dec 2021, at 17:14, tech-lists wrote: > context: main-n251261-25d0ccbe101 on arm64.aarch64 (raspberry > pi4b/8GB) … > pfctl: Cannot allocate memory. > <long story snipped> See also https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 Tl;dr: pfctl fails to add addresses to a table because an uma_zalloc() call fails). vmstat -z output: ITEM SIZE LIMIT USED FREE REQ FAILSLEEP XDOMAIN pf table entries: 160, 400000, 15196, 1604, 36039, 9, 0, 0 We can clearly see the memory allocation failure there, but no obvious reason why. It appears that there’s plenty of free memory, and we’re also clearly far away from the configured limit. Nevertheless the uma_zalloc() call returns NULL. I’ve tried to dtrace that, but ran into impossible results from dtrace (the uma_zalloc_args:return probe did not fire, but the pfr_create_kentry:return did. pfr_create_kentry() unconditionally calls uma_zalloc_args(), so that shouldn’t be possible). Right now the suspicion is that there’s something strange going on with the arm64 allocator (because it’s not been seen on amd64 so far), but I’m generally uncertain of everything, other than it’s not actually pf’s fault that it’s not getting memory. Best regards, Kristof --=_MailMate_44FD825A-A4A3-4FF5-A5A9-8E209A467CEF_= Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <!DOCTYPE html> <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/xhtml; charset=3Dutf-8"= > </head> <body><div style=3D"font-family: sans-serif;"><div class=3D"markdown" sty= le=3D"white-space: normal;"> <p dir=3D"auto">On 11 Dec 2021, at 17:14, tech-lists wrote:</p> </div><div class=3D"plaintext" style=3D"white-space: normal;"><blockquote= style=3D"margin: 0 0 5px; padding-left: 5px; border-left: 2px solid #136= BCE; color: #136BCE;"><p dir=3D"auto">context: main-n251261-25d0ccbe101 o= n arm64.aarch64 (raspberry pi4b/8GB)</p> </blockquote></div> <div class=3D"markdown" style=3D"white-space: normal;"> <p dir=3D"auto">=E2=80=A6</p> </div><div class=3D"plaintext" style=3D"white-space: normal;"><blockquote= style=3D"margin: 0 0 5px; padding-left: 5px; border-left: 2px solid #136= BCE; color: #136BCE;"><p dir=3D"auto">pfctl: Cannot allocate memory.</p> <br></blockquote></div> <div class=3D"markdown" style=3D"white-space: normal;"> <p dir=3D"auto"><long story snipped></p> <p dir=3D"auto">See also <a href=3D"https://bugs.freebsd.org/bugzilla/sho= w_bug.cgi?id=3D260406">https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D= 260406</a></p> <p dir=3D"auto">Tl;dr: pfctl fails to add addresses to a table because an= uma_zalloc() call fails).</p> <p dir=3D"auto">vmstat -z output:</p> <pre style=3D"margin-left: 15px; margin-right: 15px; padding: 5px; border= : thin solid gray; overflow-x: auto; max-width: 90vw; background-color: #= E4E4E4;"><code>ITEM SIZE LIMIT USED FREE = REQ FAILSLEEP XDOMAIN pf table entries: 160, 400000, 15196, 1604, 36039, 9, 0,= 0 </code></pre> <p dir=3D"auto">We can clearly see the memory allocation failure there, b= ut no obvious reason why.<br> It appears that there=E2=80=99s plenty of free memory, and we=E2=80=99re = also clearly far away from the configured limit. Nevertheless the uma_zal= loc() call returns NULL.</p> <p dir=3D"auto">I=E2=80=99ve tried to dtrace that, but ran into impossibl= e results from dtrace (the uma_zalloc_args:return probe did not fire, but= the pfr_create_kentry:return did. pfr_create_kentry() unconditionally ca= lls uma_zalloc_args(), so that shouldn=E2=80=99t be possible).</p> <p dir=3D"auto">Right now the suspicion is that there=E2=80=99s something= strange going on with the arm64 allocator (because it=E2=80=99s not been= seen on amd64 so far), but I=E2=80=99m generally uncertain of everything= , other than it=E2=80=99s not actually pf=E2=80=99s fault that it=E2=80=99= s not getting memory.</p> <p dir=3D"auto">Best regards,<br> Kristof</p> </div></div></body> </html> --=_MailMate_44FD825A-A4A3-4FF5-A5A9-8E209A467CEF_=--