From nobody Mon May 31 15:03:57 2021 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 82211D7D018 for ; Mon, 31 May 2021 15:04:02 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-lf1-x12a.google.com (mail-lf1-x12a.google.com [IPv6:2a00:1450:4864:20::12a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ftz8K5Rdjz3MMH; Mon, 31 May 2021 15:04:01 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-lf1-x12a.google.com with SMTP id v8so17265870lft.8; Mon, 31 May 2021 08:04:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=9PkwFDHZgJIrvPQxk61XtnspkP7e3abDlwrNNdagjUw=; b=rBjVwZvc5tVs7b45RoqPd8VI2TG6t0+GGdwH+tC8IxPEhepNpO+OVaNIWD0CirQAS3 DOy1guf+DfeOZH6uMlcsCWydhKwJdUrgmzAF1j996eeiuY1SSwfoEPMCKB32YsgvnFoG gJhl6cdRGnYZQH686T9cssNhkJMmY0lPN0Cqrr1evZIHsZzagOU2wAPHIsOuJvrA/jNq msxBc6sD/FlxiRTmxkS75Jj28WfwZ6svnKNUP8yfzjo7ohxlo/+zyTfuZV9dwfFwlmhx yEW4J5w549PfrogJk/Y9NSlYC9paKdptdbjdlDmGW29g69+SsrywTkLCTjI+n8/HGg5+ FKYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=9PkwFDHZgJIrvPQxk61XtnspkP7e3abDlwrNNdagjUw=; b=UK4jeMX3Iv5/9EsNEHgjXV2GFTpfs74WKX7Au8m1FciWKv1jJEc/baCLXHUviix8Da 0NWsnTmm/5onzLjRHmsREgi1HN+a5/SOe6fDUeGx+wg28pd/X5LxVz/JxBIFrR/ArMyZ SsX77JyL4ZJVw3jFu/umKDmtcXAwBkNWN7IGfRIUjX/q55xER+m0jZcuwARBZM0vrFQp GbGBJw1KekSbAX/SbVcerMLNgBeAghWTEmKA7QaHkqSM8Fqa7Ts+USIutoTFZG1ta2oj 1pBREF9jo+z5gKlgwtyvun0pthbHpCgKWsoC0H8uhTfDPq3KUvEG4Q7fFw7kxACaa46l ayfA== X-Gm-Message-State: AOAM530taRKHqqqRxTuDYdvaybSkFqvWBoOoVNXIbJg3Mx3S3rXq0dDT CWr6pz08ez9gGKWhtwBNN4bVimopXIL1Vb3LepMEc+s6 X-Google-Smtp-Source: ABdhPJxU+aYuEqs2EvQ1cwonD5XDu3P9T2faXNIQX7L/PZGOO8/QSt0P4s0oAMesfIZZIwFHZkDqFDD9/ts5mRVpKzU= X-Received: by 2002:a19:da12:: with SMTP id r18mr12224050lfg.549.1622473438100; Mon, 31 May 2021 08:03:58 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Received: by 2002:a2e:98c4:0:0:0:0:0 with HTTP; Mon, 31 May 2021 08:03:57 -0700 (PDT) In-Reply-To: References: From: Mateusz Guzik Date: Mon, 31 May 2021 17:03:57 +0200 Message-ID: Subject: Re: Panics in recent NFS server To: Dimitry Andric Cc: FreeBSD Current , Rick Macklem Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4Ftz8K5Rdjz3MMH X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=rBjVwZvc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2a00:1450:4864:20::12a as permitted sender) smtp.mailfrom=mjguzik@gmail.com X-Spamd-Result: default: False [-2.12 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::12a:from]; NEURAL_SPAM_SHORT(0.88)[0.885]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::12a:from:127.0.2.255]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::12a:from]; NEURAL_HAM_LONG(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-current]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N I reproduced the panic, things work for me with the patch below. However, there may be more to it so I'm going to ask Rick to weigh in. but short version is that length returned by nfsrv_parsename is off by one compared to copyinstr. diff --git a/sys/fs/nfsserver/nfs_nfsdsubs.c b/sys/fs/nfsserver/nfs_nfsdsubs.c index 2b6e17752544..8c7db36bbd05 100644 --- a/sys/fs/nfsserver/nfs_nfsdsubs.c +++ b/sys/fs/nfsserver/nfs_nfsdsubs.c @@ -2065,7 +2065,7 @@ nfsrv_parsename(struct nfsrv_descript *nd, char *bufp, u_long *hashp, } } *tocp = '\0'; - *outlenp = (size_t)outlen; + *outlenp = (size_t)outlen + 1; if (hashp != NULL) *hashp = hash; nfsmout: On 5/31/21, Mateusz Guzik wrote: > On 5/31/21, Mateusz Guzik wrote: >> It's probably my commit d81aefa8b7dd8cbeffeda541fca9962802404983 , >> I'll look at this later. > > Well let me rephrase. While the panic was added in said commit, I > suspect the bug is on nfs side -- it has its own namei variant which I > suspect is managing ni_pathlen in a manner different than the > original, it just happens to not panic on kernels prior to the above > change. > >> >> On 5/31/21, Dimitry Andric wrote: >>> Hi, >>> >>> I recently upgraded a -CURRENT NFS server from 2021-05-12 to today >>> (2021-05-31), and when the first NFS client attempted to connect, I got >>> this >>> panic: >>> >>> panic: lookup: expected nul at 0xfffff800104b3002; string [dim] >>> >>> cpuid = 0 >>> time = 1622463863 >>> KDB: stack backtrace: >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >>> 0xfffffe00747e89b0 >>> vpanic() at vpanic+0x187/frame 0xfffffe00747e8a10 >>> panic() at panic+0x43/frame 0xfffffe00747e8a70 >>> lookup() at lookup+0xad2/frame 0xfffffe00747e8b10 >>> nfsvno_namei() at nfsvno_namei+0x1a4/frame 0xfffffe00747e8bc0 >>> nfsrvd_lookup() at nfsrvd_lookup+0x191/frame 0xfffffe00747e8eb0 >>> nfsrvd_dorpc() at nfsrvd_dorpc+0xfab/frame 0xfffffe00747e90c0 >>> nfssvc_program() at nfssvc_program+0x604/frame 0xfffffe00747e92a0 >>> svc_run_internal() at svc_run_internal+0xa72/frame 0xfffffe00747e93d0 >>> svc_run() at svc_run+0x250/frame 0xfffffe00747e9430 >>> nfsrvd_nfsd() at nfsrvd_nfsd+0x33c/frame 0xfffffe00747e9590 >>> nfssvc_nfsd() at nfssvc_nfsd+0x473/frame 0xfffffe00747e9aa0 >>> sys_nfssvc() at sys_nfssvc+0xc7/frame 0xfffffe00747e9ac0 >>> amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe00747e9bf0 >>> fast_syscall_common() at fast_syscall_common+0xf8/frame >>> 0xfffffe00747e9bf0 >>> --- syscall (155, FreeBSD ELF64, sys_nfssvc), rip = 0x8011aa59a, rsp = >>> 0x7fffffffe4e8, rbp = 0x7fffffffe780 --- >>> KDB: enter: panic >>> >>> __curthread () >>> at /share/dim/src/freebsd/src-dim/sys/amd64/include/pcpu_aux.h:55 >>> 55 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu, >>> (kgdb) #0 __curthread () >>> at /share/dim/src/freebsd/src-dim/sys/amd64/include/pcpu_aux.h:55 >>> #1 doadump (textdump=textdump@entry=0) >>> at /share/dim/src/freebsd/src-dim/sys/kern/kern_shutdown.c:399 >>> #2 0xffffffff804cca5a in db_dump (dummy=, >>> dummy2=, dummy3=, dummy4=) >>> at /share/dim/src/freebsd/src-dim/sys/ddb/db_command.c:575 >>> #3 0xffffffff804cc912 in db_command (last_cmdp=, >>> cmd_table=, dopager=dopager@entry=1) >>> at /share/dim/src/freebsd/src-dim/sys/ddb/db_command.c:482 >>> #4 0xffffffff804cc58d in db_command_loop () >>> at /share/dim/src/freebsd/src-dim/sys/ddb/db_command.c:535 >>> #5 0xffffffff804cfd06 in db_trap (type=, code=>> out>) >>> at /share/dim/src/freebsd/src-dim/sys/ddb/db_main.c:270 >>> #6 0xffffffff80c69f17 in kdb_trap (type=type@entry=3, >>> code=code@entry=0, >>> tf=tf@entry=0xfffffe00747e88e0) >>> at /share/dim/src/freebsd/src-dim/sys/kern/subr_kdb.c:727 >>> #7 0xffffffff810d7aee in trap (frame=0xfffffe00747e88e0) >>> at /share/dim/src/freebsd/src-dim/sys/amd64/amd64/trap.c:576 >>> #8 >>> #9 kdb_enter (why=0xffffffff812d3d27 "panic", msg=) >>> at /share/dim/src/freebsd/src-dim/sys/kern/subr_kdb.c:506 >>> #10 0xffffffff80c1d248 in vpanic ( >>> fmt=0xffffffff8129dfef "%s: expected nul at %p; string [%s]\n", >>> ap=, ap@entry=0xfffffe00747e8a50) >>> at /share/dim/src/freebsd/src-dim/sys/kern/kern_shutdown.c:907 >>> #11 0xffffffff80c1cfd3 in panic ( >>> fmt=0xffffffff81e9b9c8 "=\t)\201\377\377\377\377") >>> at /share/dim/src/freebsd/src-dim/sys/kern/kern_shutdown.c:843 >>> #12 0xffffffff80cfa992 in lookup (ndp=ndp@entry=0xfffffe00747e8d90) >>> at /share/dim/src/freebsd/src-dim/sys/kern/vfs_lookup.c:919 >>> #13 0xffffffff80b33f84 in nfsvno_namei (nd=nd@entry=0xfffffe00747e9100, >>> ndp=ndp@entry=0xfffffe00747e8d90, dp=, >>> dp@entry=0xfffff80010544380, islocked=, >>> islocked@entry=0, >>> exp=exp@entry=0xfffffe00747e8fd8, p=p@entry=0xfffffe00bbfa3000, >>> retdirp=0xfffffe00747e8e78) >>> at >>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdport.c:597 >>> #14 0xffffffff80b266a1 in nfsrvd_lookup (nd=0xfffffe00747e9100, >>> isdgram=, dp=0xfffff80010544380, >>> vpp=0xfffffe00747e9010, >>> fhp=0xfffffe00747e9074, exp=0xfffffe00747e8fd8) >>> at >>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdserv.c:607 >>> #15 0xffffffff80b1073b in nfsrvd_compound (nd=0xfffffe00747e9100, >>> isdgram=0, >>> tag=0xf , taglen=6, >>> minorvers=4294967294) >>> at >>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdsocket.c:1098 >>> #16 nfsrvd_dorpc (nd=nd@entry=0xfffffe00747e9100, >>> isdgram=isdgram@entry=0, >>> tag=0xf , taglen=6, >>> minorvers=4294967294) >>> at >>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdsocket.c:626 >>> #17 0xffffffff80b24c44 in nfs_proc (nd=0xfffffe00747e9100, >>> xid=, xprt=0xfffff80003a14e00, rpp=) >>> at >>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdkrpc.c:402 >>> #18 nfssvc_program (rqst=0xfffff80010455800, xprt=0xfffff80003a14e00) >>> at >>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdkrpc.c:288 >>> #19 0xffffffff80edead2 in svc_executereq (rqstp=0xfffff80010455800) >>> at /share/dim/src/freebsd/src-dim/sys/rpc/svc.c:1037 >>> #20 svc_run_internal (grp=, grp@entry=0xfffff800100e6100, >>> ismaster=ismaster@entry=1) >>> at /share/dim/src/freebsd/src-dim/sys/rpc/svc.c:1313 >>> #21 0xffffffff80eddf80 in svc_run (pool=) >>> at /share/dim/src/freebsd/src-dim/sys/rpc/svc.c:1392 >>> #22 0xffffffff80b251ec in nfsrvd_nfsd (td=, >>> td@entry=0xfffffe00bbfa3000, args=args@entry=0xfffffe00747e9660) >>> at >>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdkrpc.c:561 >>> #23 0xffffffff80b3ec93 in nfssvc_nfsd (td=0xfffffe00bbfa3000, >>> uap=) >>> at >>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdport.c:3714 >>> #24 0xffffffff80e6f647 in sys_nfssvc (td=0xfffffe00bbfa3000, >>> uap=0xfffffe00bbfa33e8) >>> at /share/dim/src/freebsd/src-dim/sys/nfs/nfs_nfssvc.c:111 >>> #25 0xffffffff810d891e in syscallenter (td=) >>> at >>> /share/dim/src/freebsd/src-dim/sys/amd64/amd64/../../kern/subr_syscall.c:189 >>> #26 amd64_syscall (td=0xfffffe00bbfa3000, traced=0) >>> at /share/dim/src/freebsd/src-dim/sys/amd64/amd64/trap.c:1156 >>> #27 >>> #28 0x00000008011aa59a in ?? () >>> >>> Is anybody seeing this too? :) >>> >>> I can probably bisect, but it'll take quite a while. >>> >>> -Dimitry >>> >>> >> >> >> -- >> Mateusz Guzik >> > > > -- > Mateusz Guzik > -- Mateusz Guzik