Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 31 May 2021 15:52:38 +0000
From:      Rick Macklem <rmacklem@uoguelph.ca>
To:        Mateusz Guzik <mjguzik@gmail.com>, Dimitry Andric <dim@freebsd.org>
Cc:        FreeBSD Current <freebsd-current@freebsd.org>, Rick Macklem <rmacklem@freebsd.org>
Subject:   Re: Panics in recent NFS server
Message-ID:  <YQXPR0101MB096886D867E278B38D207FC6DD3F9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>
In-Reply-To: <CAGudoHGsnMtwDRKoCYLio_SCm3HZdYU7qF=uCHEb0y_HS5m-Ng@mail.gmail.com>
References:  <EA017AF6-25C8-48FC-B52F-D83D9DD314AA@FreeBSD.org> <CAGudoHHd3QVT6mefMcELRKa7nPptjVuc=D-agoHUNY4pfOz8tg@mail.gmail.com> <CAGudoHGk0mrmiLHqhfvCA596kxZa8vD1ex%2BQOgLBE0vt%2B7OJrA@mail.gmail.com>,<CAGudoHGsnMtwDRKoCYLio_SCm3HZdYU7qF=uCHEb0y_HS5m-Ng@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
Mateusz Guzik wrote:=0A=
>I reproduced the panic, things work for me with the patch below.=0A=
>However, there may be more to it so I'm going to ask Rick to weigh in.=0A=
>but short version is that length returned by nfsrv_parsename is off by=0A=
>one compared to copyinstr.=0A=
Yes, it appears that, now, ni_pathlen includes the nul termination characte=
r.=0A=
I don't think that was always the case, but I can't be bothered to search=
=0A=
back through the commits to try and find when it changed.=0A=
=0A=
As such, this patch looks fine and you can consider it reviewed by me.=0A=
=0A=
rick=0A=
=0A=
diff --git a/sys/fs/nfsserver/nfs_nfsdsubs.c b/sys/fs/nfsserver/nfs_nfsdsub=
s.c=0A=
index 2b6e17752544..8c7db36bbd05 100644=0A=
--- a/sys/fs/nfsserver/nfs_nfsdsubs.c=0A=
+++ b/sys/fs/nfsserver/nfs_nfsdsubs.c=0A=
@@ -2065,7 +2065,7 @@ nfsrv_parsename(struct nfsrv_descript *nd, char=0A=
*bufp, u_long *hashp,=0A=
            }=0A=
        }=0A=
        *tocp =3D '\0';=0A=
-       *outlenp =3D (size_t)outlen;=0A=
+       *outlenp =3D (size_t)outlen + 1;=0A=
        if (hashp !=3D NULL)=0A=
                *hashp =3D hash;=0A=
 nfsmout:=0A=
=0A=
=0A=
On 5/31/21, Mateusz Guzik <mjguzik@gmail.com> wrote:=0A=
> On 5/31/21, Mateusz Guzik <mjguzik@gmail.com> wrote:=0A=
>> It's probably my commit d81aefa8b7dd8cbeffeda541fca9962802404983 ,=0A=
>> I'll look at this later.=0A=
>=0A=
> Well let me rephrase. While the panic was added in said commit, I=0A=
> suspect the bug is on nfs side -- it has its own namei variant which I=0A=
> suspect is managing ni_pathlen in a manner different than the=0A=
> original, it just happens to not panic on kernels prior to the above=0A=
> change.=0A=
>=0A=
>>=0A=
>> On 5/31/21, Dimitry Andric <dim@freebsd.org> wrote:=0A=
>>> Hi,=0A=
>>>=0A=
>>> I recently upgraded a -CURRENT NFS server from 2021-05-12 to today=0A=
>>> (2021-05-31), and when the first NFS client attempted to connect, I got=
=0A=
>>> this=0A=
>>> panic:=0A=
>>>=0A=
>>> panic: lookup: expected nul at 0xfffff800104b3002; string [dim]=0A=
>>>=0A=
>>> cpuid =3D 0=0A=
>>> time =3D 1622463863=0A=
>>> KDB: stack backtrace:=0A=
>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame=0A=
>>> 0xfffffe00747e89b0=0A=
>>> vpanic() at vpanic+0x187/frame 0xfffffe00747e8a10=0A=
>>> panic() at panic+0x43/frame 0xfffffe00747e8a70=0A=
>>> lookup() at lookup+0xad2/frame 0xfffffe00747e8b10=0A=
>>> nfsvno_namei() at nfsvno_namei+0x1a4/frame 0xfffffe00747e8bc0=0A=
>>> nfsrvd_lookup() at nfsrvd_lookup+0x191/frame 0xfffffe00747e8eb0=0A=
>>> nfsrvd_dorpc() at nfsrvd_dorpc+0xfab/frame 0xfffffe00747e90c0=0A=
>>> nfssvc_program() at nfssvc_program+0x604/frame 0xfffffe00747e92a0=0A=
>>> svc_run_internal() at svc_run_internal+0xa72/frame 0xfffffe00747e93d0=
=0A=
>>> svc_run() at svc_run+0x250/frame 0xfffffe00747e9430=0A=
>>> nfsrvd_nfsd() at nfsrvd_nfsd+0x33c/frame 0xfffffe00747e9590=0A=
>>> nfssvc_nfsd() at nfssvc_nfsd+0x473/frame 0xfffffe00747e9aa0=0A=
>>> sys_nfssvc() at sys_nfssvc+0xc7/frame 0xfffffe00747e9ac0=0A=
>>> amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe00747e9bf0=0A=
>>> fast_syscall_common() at fast_syscall_common+0xf8/frame=0A=
>>> 0xfffffe00747e9bf0=0A=
>>> --- syscall (155, FreeBSD ELF64, sys_nfssvc), rip =3D 0x8011aa59a, rsp =
=3D=0A=
>>> 0x7fffffffe4e8, rbp =3D 0x7fffffffe780 ---=0A=
>>> KDB: enter: panic=0A=
>>>=0A=
>>> __curthread ()=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/amd64/include/pcpu_aux.h:55=
=0A=
>>> 55          __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(stru=
ct pcpu,=0A=
>>> (kgdb) #0  __curthread ()=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/amd64/include/pcpu_aux.h:55=
=0A=
>>> #1  doadump (textdump=3Dtextdump@entry=3D0)=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/kern/kern_shutdown.c:399=0A=
>>> #2  0xffffffff804cca5a in db_dump (dummy=3D<optimized out>,=0A=
>>>     dummy2=3D<unavailable>, dummy3=3D<unavailable>, dummy4=3D<unavailab=
le>)=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/ddb/db_command.c:575=0A=
>>> #3  0xffffffff804cc912 in db_command (last_cmdp=3D<optimized out>,=0A=
>>>     cmd_table=3D<optimized out>, dopager=3Ddopager@entry=3D1)=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/ddb/db_command.c:482=0A=
>>> #4  0xffffffff804cc58d in db_command_loop ()=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/ddb/db_command.c:535=0A=
>>> #5  0xffffffff804cfd06 in db_trap (type=3D<optimized out>, code=3D<opti=
mized=0A=
>>> out>)=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/ddb/db_main.c:270=0A=
>>> #6  0xffffffff80c69f17 in kdb_trap (type=3Dtype@entry=3D3,=0A=
>>> code=3Dcode@entry=3D0,=0A=
>>>     tf=3Dtf@entry=3D0xfffffe00747e88e0)=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/kern/subr_kdb.c:727=0A=
>>> #7  0xffffffff810d7aee in trap (frame=3D0xfffffe00747e88e0)=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/amd64/amd64/trap.c:576=0A=
>>> #8  <signal handler called>=0A=
>>> #9  kdb_enter (why=3D0xffffffff812d3d27 "panic", msg=3D<optimized out>)=
=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/kern/subr_kdb.c:506=0A=
>>> #10 0xffffffff80c1d248 in vpanic (=0A=
>>>     fmt=3D0xffffffff8129dfef "%s: expected nul at %p; string [%s]\n",=
=0A=
>>>     ap=3D<optimized out>, ap@entry=3D0xfffffe00747e8a50)=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/kern/kern_shutdown.c:907=0A=
>>> #11 0xffffffff80c1cfd3 in panic (=0A=
>>>     fmt=3D0xffffffff81e9b9c8 <cnputs_mtx> "=3D\t)\201\377\377\377\377")=
=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/kern/kern_shutdown.c:843=0A=
>>> #12 0xffffffff80cfa992 in lookup (ndp=3Dndp@entry=3D0xfffffe00747e8d90)=
=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/kern/vfs_lookup.c:919=0A=
>>> #13 0xffffffff80b33f84 in nfsvno_namei (nd=3Dnd@entry=3D0xfffffe00747e9=
100,=0A=
>>>     ndp=3Dndp@entry=3D0xfffffe00747e8d90, dp=3D<optimized out>,=0A=
>>>     dp@entry=3D0xfffff80010544380, islocked=3D<optimized out>,=0A=
>>> islocked@entry=3D0,=0A=
>>>     exp=3Dexp@entry=3D0xfffffe00747e8fd8, p=3Dp@entry=3D0xfffffe00bbfa3=
000,=0A=
>>>     retdirp=3D0xfffffe00747e8e78)=0A=
>>>     at=0A=
>>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdport.c:597=0A=
>>> #14 0xffffffff80b266a1 in nfsrvd_lookup (nd=3D0xfffffe00747e9100,=0A=
>>>     isdgram=3D<optimized out>, dp=3D0xfffff80010544380,=0A=
>>> vpp=3D0xfffffe00747e9010,=0A=
>>>     fhp=3D0xfffffe00747e9074, exp=3D0xfffffe00747e8fd8)=0A=
>>>     at=0A=
>>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdserv.c:607=0A=
>>> #15 0xffffffff80b1073b in nfsrvd_compound (nd=3D0xfffffe00747e9100,=0A=
>>> isdgram=3D0,=0A=
>>>     tag=3D0xf <error: Cannot access memory at address 0xf>, taglen=3D6,=
=0A=
>>>     minorvers=3D4294967294)=0A=
>>>     at=0A=
>>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdsocket.c:1098=
=0A=
>>> #16 nfsrvd_dorpc (nd=3Dnd@entry=3D0xfffffe00747e9100,=0A=
>>> isdgram=3Disdgram@entry=3D0,=0A=
>>>     tag=3D0xf <error: Cannot access memory at address 0xf>, taglen=3D6,=
=0A=
>>>     minorvers=3D4294967294)=0A=
>>>     at=0A=
>>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdsocket.c:626=0A=
>>> #17 0xffffffff80b24c44 in nfs_proc (nd=3D0xfffffe00747e9100,=0A=
>>>     xid=3D<optimized out>, xprt=3D0xfffff80003a14e00, rpp=3D<optimized =
out>)=0A=
>>>     at=0A=
>>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdkrpc.c:402=0A=
>>> #18 nfssvc_program (rqst=3D0xfffff80010455800, xprt=3D0xfffff80003a14e0=
0)=0A=
>>>     at=0A=
>>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdkrpc.c:288=0A=
>>> #19 0xffffffff80edead2 in svc_executereq (rqstp=3D0xfffff80010455800)=
=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/rpc/svc.c:1037=0A=
>>> #20 svc_run_internal (grp=3D<optimized out>, grp@entry=3D0xfffff800100e=
6100,=0A=
>>>     ismaster=3Dismaster@entry=3D1)=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/rpc/svc.c:1313=0A=
>>> #21 0xffffffff80eddf80 in svc_run (pool=3D<optimized out>)=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/rpc/svc.c:1392=0A=
>>> #22 0xffffffff80b251ec in nfsrvd_nfsd (td=3D<optimized out>,=0A=
>>>     td@entry=3D0xfffffe00bbfa3000, args=3Dargs@entry=3D0xfffffe00747e96=
60)=0A=
>>>     at=0A=
>>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdkrpc.c:561=0A=
>>> #23 0xffffffff80b3ec93 in nfssvc_nfsd (td=3D0xfffffe00bbfa3000,=0A=
>>>     uap=3D<optimized out>)=0A=
>>>     at=0A=
>>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdport.c:3714=0A=
>>> #24 0xffffffff80e6f647 in sys_nfssvc (td=3D0xfffffe00bbfa3000,=0A=
>>>     uap=3D0xfffffe00bbfa33e8)=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/nfs/nfs_nfssvc.c:111=0A=
>>> #25 0xffffffff810d891e in syscallenter (td=3D<optimized out>)=0A=
>>>     at=0A=
>>> /share/dim/src/freebsd/src-dim/sys/amd64/amd64/../../kern/subr_syscall.=
c:189=0A=
>>> #26 amd64_syscall (td=3D0xfffffe00bbfa3000, traced=3D0)=0A=
>>>     at /share/dim/src/freebsd/src-dim/sys/amd64/amd64/trap.c:1156=0A=
>>> #27 <signal handler called>=0A=
>>> #28 0x00000008011aa59a in ?? ()=0A=
>>>=0A=
>>> Is anybody seeing this too? :)=0A=
>>>=0A=
>>> I can probably bisect, but it'll take quite a while.=0A=
>>>=0A=
>>> -Dimitry=0A=
>>>=0A=
>>>=0A=
>>=0A=
>>=0A=
>> --=0A=
>> Mateusz Guzik <mjguzik gmail.com>=0A=
>>=0A=
>=0A=
>=0A=
> --=0A=
> Mateusz Guzik <mjguzik gmail.com>=0A=
>=0A=
=0A=
=0A=
--=0A=
Mateusz Guzik <mjguzik gmail.com>=0A=



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQXPR0101MB096886D867E278B38D207FC6DD3F9>