Date: Mon, 02 May 2022 11:02:53 +0200 From: Kristof Provost <kp@FreeBSD.org> To: qroxana <qroxana@protonmail.com> Cc: freebsd-current@freebsd.org, freebsd-arm@freebsd.org Subject: Re: Kernel panic on armv7 when PF is enabled Message-ID: <D190268C-3BA3-455D-B1D8-A3876A054434@FreeBSD.org> In-Reply-To: <t46qj3HiJIHpjcIlYQ8t7s7UFcPl3dHxJCc6ovjAF8teNjGunlJnGsbkpM2hLWrPQpRRCzCGnPGg1RuK7hbRMxr9BlHD03xvcsvqxNuM2DU=@protonmail.com> References: <t46qj3HiJIHpjcIlYQ8t7s7UFcPl3dHxJCc6ovjAF8teNjGunlJnGsbkpM2hLWrPQpRRCzCGnPGg1RuK7hbRMxr9BlHD03xvcsvqxNuM2DU=@protonmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--=_MailMate_43ACE16C-8FBB-4703-A9C0-5E8ED6BC4AE9_=
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
On 1 May 2022, at 5:13, qroxana wrote:
> After git bisecting the panic started since this commit.
>
> commit 78bc3d5e1712bc1649aa5574d2b8d153f9665113
>
> Author: Kristof Provost <
> kp@FreeBSD.org
>>
>
> Date: Mon Feb 14 20:09:54 2022 +0100
>
> vlan: allow net.link.vlan.mtag_pcp to be set per vnet
>
> The primary reason for this change is to facilitate testing.
>
> MFC after: 1 week
>
> sys/net/if_ethersubr.c | 9 +++++----
>
> sys/net/if_vlan.c | 5 +++--
>
> 2 files changed, 8 insertions(+), 6 deletions(-)
>
> The armv7 board boots from a NFS root,
>
> it can boot without any problem if PF is disabled.
>
> Any helps?
>
> add host ::1: gateway lo0 fib 0: route already in table
> add net fe80::: gateway ::1
> add net ff02::: gateway ::1
> add net ::ffff:0.0.0.0: gateway ::1
> add net ::0.0.0.0: gateway ::1
> Enabling pf.
> Kernel page fault with the following non-sleepable locks held:
> shared rm pf rulesets (pf rulesets) r = 0 (0xe3099430) locked @
> /usr/src/sys/netpfil/pf/pf.c:6493
> exclusive rw tcpinp (tcpinp) r = 0 (0xdb748d88) locked @
> /usr/src/sys/netinet/tcp_usrreq.c:1008
> stack backtrace:
> #0 0xc0355cac at witness_debugger+0x7c
> #1 0xc0356ef0 at witness_warn+0x3fc
> #2 0xc05ec048 at abort_handler+0x1d8
> #3 0xc05cb5ac at exception_exit+0
> #4 0xe3083c10 at pf_syncookie_validate+0x60
> #5 0xe30496a8 at pf_test+0x518
> #6 0xe306d768 at pf_check_out+0x30
> #7 0xc0415b44 at pfil_run_hooks+0xbc
> #8 0xc0445cfc at ip_output+0xce8
> #9 0xc045bc9c at tcp_default_output+0x20ac
> #10 0xc0471eb4 at tcp_usr_send+0x1ac
> #11 0xc0389464 at sosend_generic+0x490
> #12 0xc0389790 at sosend+0x64
> #13 0xc0502888 at clnt_vc_call+0x560
> #14 0xc05009d8 at clnt_reconnect_call+0x170
> #15 0xc01e7b14 at newnfs_request+0xb20
> #16 0xc0230218 at nfscl_request+0x60
> #17 0xc020d9bc at nfsrpc_getattr+0xb0
> Fatal kernel mode data abort: 'Alignment Fault' on read
> trapframe: 0xdf1f1c90
> FSR=00000001, FAR=d7840264, spsr=40000013
> r0 =6a228eda, r1 =dac0d785, r2 =d7840264, r3 =db5527c0
> r4 =df1f1e00, r5 =dac0d75f, r6 =00000018, r7 =d9422c00
> r8 =c093e5e4, r9 =00000001, r10=df1f1f5c, r11=df1f1d38
> r12=e3098dd0, ssp=df1f1d20, slr=e3083bdc, pc =e3083c10
>
>
The commit you point at is entirely unrelated to the code where the
panic occurred, so I’m pretty sure something went wrong in your
bisect.
The backtrace would suggest the issue occurs in the
pf_syncookie_validate() function, and likely in the line `if
(atomic_load_64(&V_pf_status.syncookies_inflight[cookie.flags.oddeven])
== 0)`
The obvious way for that to panic would be to call it without the
curvnet context set, but pf_test() uses it earlier, so that’s going to
be fine.
Given that this is unique to armv7 I’d recommend talking to the armv7
maintainer about 64 bit atomic operations.
You can probably avoid the atomic load with this patch (and not enabling
syncookie support):
diff --git a/sys/netpfil/pf/pf_syncookies.c
b/sys/netpfil/pf/pf_syncookies.c
index 5230502be30c..c86d469d3cef 100644
--- a/sys/netpfil/pf/pf_syncookies.c
+++ b/sys/netpfil/pf/pf_syncookies.c
@@ -313,6 +313,9 @@ pf_syncookie_validate(struct pf_pdesc *pd)
ack = ntohl(pd->hdr.tcp.th_ack) - 1;
cookie.cookie = (ack & 0xff) ^ (ack >> 24);
+ if (V_pf_status.syncookies_mode == PF_SYNCOOKIES_NEVER)
+ return (0);
+
/* we don't know oddeven before setting the cookie (union) */
if
(atomic_load_64(&V_pf_status.syncookies_inflight[cookie.flags.oddeven])
== 0)
That shouldn’t be required though.
Br,
Kristof
--=_MailMate_43ACE16C-8FBB-4703-A9C0-5E8ED6BC4AE9_=
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE html>
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/xhtml; charset=3Dutf-8"=
>
</head>
<body><div style=3D"font-family: sans-serif;"><div class=3D"markdown" sty=
le=3D"white-space: normal;">
<p dir=3D"auto">On 1 May 2022, at 5:13, qroxana wrote:</p>
</div><div class=3D"plaintext" style=3D"white-space: normal;"><blockquote=
style=3D"margin: 0 0 5px; padding-left: 5px; border-left: 2px solid #136=
BCE; color: #136BCE;"><p dir=3D"auto">After git bisecting the panic start=
ed since this commit.</p>
<p dir=3D"auto">commit 78bc3d5e1712bc1649aa5574d2b8d153f9665113</p>
<p dir=3D"auto">Author: Kristof Provost <
<br>
kp@FreeBSD.org</p>
<blockquote style=3D"margin: 0 0 5px; padding-left: 5px; border-left: 2px=
solid #136BCE; border-left-color: #4B89CF; color: #4B89CF;"></blockquote=
><p dir=3D"auto">Date: Mon Feb 14 20:09:54 2022 +0100</p>
<p dir=3D"auto">vlan: allow net.link.vlan.mtag_pcp to be set per vnet</p>=
<p dir=3D"auto">The primary reason for this change is to facilitate testi=
ng.</p>
<p dir=3D"auto">MFC after: 1 week</p>
<p dir=3D"auto">sys/net/if_ethersubr.c | 9 +++++----</p>
<p dir=3D"auto">sys/net/if_vlan.c | 5 +++--</p>
<p dir=3D"auto">2 files changed, 8 insertions(+), 6 deletions(-)</p>
<p dir=3D"auto">The armv7 board boots from a NFS root,</p>
<p dir=3D"auto">it can boot without any problem if PF is disabled.</p>
<p dir=3D"auto">Any helps?</p>
<p dir=3D"auto">add host ::1: gateway lo0 fib 0: route already in table
<br>
add net fe80::: gateway ::1
<br>
add net ff02::: gateway ::1
<br>
add net ::ffff:0.0.0.0: gateway ::1
<br>
add net ::0.0.0.0: gateway ::1
<br>
Enabling pf.
<br>
Kernel page fault with the following non-sleepable locks held:
<br>
shared rm pf rulesets (pf rulesets) r =3D 0 (0xe3099430) locked @ /usr/sr=
c/sys/netpfil/pf/pf.c:6493
<br>
exclusive rw tcpinp (tcpinp) r =3D 0 (0xdb748d88) locked @ /usr/src/sys/n=
etinet/tcp_usrreq.c:1008
<br>
stack backtrace:
<br>
#0 0xc0355cac at witness_debugger+0x7c
<br>
#1 0xc0356ef0 at witness_warn+0x3fc
<br>
#2 0xc05ec048 at abort_handler+0x1d8
<br>
#3 0xc05cb5ac at exception_exit+0
<br>
#4 0xe3083c10 at pf_syncookie_validate+0x60
<br>
#5 0xe30496a8 at pf_test+0x518
<br>
#6 0xe306d768 at pf_check_out+0x30
<br>
#7 0xc0415b44 at pfil_run_hooks+0xbc
<br>
#8 0xc0445cfc at ip_output+0xce8
<br>
#9 0xc045bc9c at tcp_default_output+0x20ac
<br>
#10 0xc0471eb4 at tcp_usr_send+0x1ac
<br>
#11 0xc0389464 at sosend_generic+0x490
<br>
#12 0xc0389790 at sosend+0x64
<br>
#13 0xc0502888 at clnt_vc_call+0x560
<br>
#14 0xc05009d8 at clnt_reconnect_call+0x170
<br>
#15 0xc01e7b14 at newnfs_request+0xb20
<br>
#16 0xc0230218 at nfscl_request+0x60
<br>
#17 0xc020d9bc at nfsrpc_getattr+0xb0
<br>
Fatal kernel mode data abort: 'Alignment Fault' on read
<br>
trapframe: 0xdf1f1c90
<br>
FSR=3D00000001, FAR=3Dd7840264, spsr=3D40000013
<br>
r0 =3D6a228eda, r1 =3Ddac0d785, r2 =3Dd7840264, r3 =3Ddb5527c0
<br>
r4 =3Ddf1f1e00, r5 =3Ddac0d75f, r6 =3D00000018, r7 =3Dd9422c00
<br>
r8 =3Dc093e5e4, r9 =3D00000001, r10=3Ddf1f1f5c, r11=3Ddf1f1d38
<br>
r12=3De3098dd0, ssp=3Ddf1f1d20, slr=3De3083bdc, pc =3De3083c10</p>
<br></blockquote></div>
<div class=3D"markdown" style=3D"white-space: normal;">
<p dir=3D"auto">The commit you point at is entirely unrelated to the code=
where the panic occurred, so I=E2=80=99m pretty sure something went wron=
g in your bisect.</p>
<p dir=3D"auto">The backtrace would suggest the issue occurs in the pf_s=
yncookie_validate() function, and likely in the line <code>if (atomic_loa=
d_64(&V_pf_status.syncookies_inflight[cookie.flags.oddeven]) =3D=3D 0=
)</code></p>
<p dir=3D"auto">The obvious way for that to panic would be to call it wit=
hout the curvnet context set, but pf_test() uses it earlier, so that=E2=80=
=99s going to be fine.</p>
<p dir=3D"auto">Given that this is unique to armv7 I=E2=80=99d recommend =
talking to the armv7 maintainer about 64 bit atomic operations.</p>
<p dir=3D"auto">You can probably avoid the atomic load with this patch (a=
nd not enabling syncookie support):</p>
<pre style=3D"margin-left: 15px; margin-right: 15px; padding: 5px; border=
: thin solid gray; overflow-x: auto; max-width: 90vw; background-color: #=
E4E4E4;"><code>diff --git a/sys/netpfil/pf/pf_syncookies.c b/sys/netpfil/=
pf/pf_syncookies.c
index 5230502be30c..c86d469d3cef 100644
--- a/sys/netpfil/pf/pf_syncookies.c
+++ b/sys/netpfil/pf/pf_syncookies.c
@@ -313,6 +313,9 @@ pf_syncookie_validate(struct pf_pdesc *pd)
ack =3D ntohl(pd->hdr.tcp.th_ack) - 1;
cookie.cookie =3D (ack & 0xff) ^ (ack >> 24);
+ if (V_pf_status.syncookies_mode =3D=3D PF_SYNCOOKIES_NEVER)
+ return (0);
+
/* we don't know oddeven before setting the cookie (union) */
if (atomic_load_64(&V_pf_status.syncookies_inflight[cookie.f=
lags.oddeven])
=3D=3D 0)
</code></pre>
<p dir=3D"auto">That shouldn=E2=80=99t be required though.</p>
<p dir=3D"auto">Br,<br>
Kristof</p>
</div></div></body>
</html>
--=_MailMate_43ACE16C-8FBB-4703-A9C0-5E8ED6BC4AE9_=--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?D190268C-3BA3-455D-B1D8-A3876A054434>