From nobody Mon May 31 14:42:17 2021 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D287CD798F2 for ; Mon, 31 May 2021 14:42:19 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-lf1-x12c.google.com (mail-lf1-x12c.google.com [IPv6:2a00:1450:4864:20::12c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FtygH5T3Vz3HSS; Mon, 31 May 2021 14:42:19 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-lf1-x12c.google.com with SMTP id r5so17186300lfr.5; Mon, 31 May 2021 07:42:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=sUo3KFWtLSB6XAkh0/g1GwCuUrSHNTTCVXUewOXyzhA=; b=jIbWwmLm9JMrdxCgNZXPduiPS/X/iu5Aem+24rLfXQgjioJ/lRGlbSPnXG2o3CQ/8B Lw6HkJfkcL87AVi0D6Saw4aZkQe8K6D+iTDNTXn6DJpxNTUrJzgxNWvpFsxvvhgWufvr 7mmj1eJ86uc0Sfjw+hpZmu+Xg+lJ2dDzC5xK5VToJUy3Bv5mcBs6a/VAAxJLdrpi4G5l JRlDNy81rFrxDWbVUs+oRDynNjvBAPMIQH0UofutE2+mZq5RRVa+Cr07xG1/MgXF0Dm0 cnz/W8IFE9YOeSHdvWip61vwqL1MxoIIvyd2LUytfllF63UWI592AJD6aJ1Bf5cRWxXe 3ihw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=sUo3KFWtLSB6XAkh0/g1GwCuUrSHNTTCVXUewOXyzhA=; b=otpa33+lxcxTWVXFatqOaj70yNh844zZkNpG+4luFWDIpTW8FpGQ0vI5Qh+VIvuvLO Eo1svGhSqLIuImo/CpoPNN6L5MPEw93pOfgdNvo4ARp3GX6faZBKrUZXnw2bPMLDWd4Q oszKt8tryW9RvrZo39dNZP19yxtSrbltVPVcePMjJMSQdHN7JozsOpDysLKD99Dm/mQU eiyok89VHfU3rOWqD77AegZj7a4n0r8mhTxA3kIxXcW2qEXOLpC9vnRYqQ9VVfVZQLXZ dKtWBPaQ6oUnCkVjeea3a3/BBQGHCp6SAbPEl0/Gd1P+EcGBGrsqGFeGNt0DMmQOyMBU Ei3A== X-Gm-Message-State: AOAM532isieXqyGJ+u7A5f+PSToSlSUKGMvDX5vvO7n+YjNiVvWHaSBl cuah1n2XUZU428PWrYZABV2M8fviBvxVbdEHisCrZyVi X-Google-Smtp-Source: ABdhPJzK04J6wRYRLFB4hRPUeNbuen3EbNfIWUbygAOP1FU/YomL3N7i1VQtuPmIzCM6nndd9WZimzCb96loO2QQbNY= X-Received: by 2002:ac2:53ae:: with SMTP id j14mr4011305lfh.296.1622472137757; Mon, 31 May 2021 07:42:17 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Received: by 2002:a2e:98c4:0:0:0:0:0 with HTTP; Mon, 31 May 2021 07:42:17 -0700 (PDT) In-Reply-To: References: From: Mateusz Guzik Date: Mon, 31 May 2021 16:42:17 +0200 Message-ID: Subject: Re: Panics in recent NFS server To: Dimitry Andric Cc: FreeBSD Current , Rick Macklem Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4FtygH5T3Vz3HSS X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N It's probably my commit d81aefa8b7dd8cbeffeda541fca9962802404983 , I'll look at this later. On 5/31/21, Dimitry Andric wrote: > Hi, > > I recently upgraded a -CURRENT NFS server from 2021-05-12 to today > (2021-05-31), and when the first NFS client attempted to connect, I got this > panic: > > panic: lookup: expected nul at 0xfffff800104b3002; string [dim] > > cpuid = 0 > time = 1622463863 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfffffe00747e89b0 > vpanic() at vpanic+0x187/frame 0xfffffe00747e8a10 > panic() at panic+0x43/frame 0xfffffe00747e8a70 > lookup() at lookup+0xad2/frame 0xfffffe00747e8b10 > nfsvno_namei() at nfsvno_namei+0x1a4/frame 0xfffffe00747e8bc0 > nfsrvd_lookup() at nfsrvd_lookup+0x191/frame 0xfffffe00747e8eb0 > nfsrvd_dorpc() at nfsrvd_dorpc+0xfab/frame 0xfffffe00747e90c0 > nfssvc_program() at nfssvc_program+0x604/frame 0xfffffe00747e92a0 > svc_run_internal() at svc_run_internal+0xa72/frame 0xfffffe00747e93d0 > svc_run() at svc_run+0x250/frame 0xfffffe00747e9430 > nfsrvd_nfsd() at nfsrvd_nfsd+0x33c/frame 0xfffffe00747e9590 > nfssvc_nfsd() at nfssvc_nfsd+0x473/frame 0xfffffe00747e9aa0 > sys_nfssvc() at sys_nfssvc+0xc7/frame 0xfffffe00747e9ac0 > amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe00747e9bf0 > fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00747e9bf0 > --- syscall (155, FreeBSD ELF64, sys_nfssvc), rip = 0x8011aa59a, rsp = > 0x7fffffffe4e8, rbp = 0x7fffffffe780 --- > KDB: enter: panic > > __curthread () > at /share/dim/src/freebsd/src-dim/sys/amd64/include/pcpu_aux.h:55 > 55 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu, > (kgdb) #0 __curthread () > at /share/dim/src/freebsd/src-dim/sys/amd64/include/pcpu_aux.h:55 > #1 doadump (textdump=textdump@entry=0) > at /share/dim/src/freebsd/src-dim/sys/kern/kern_shutdown.c:399 > #2 0xffffffff804cca5a in db_dump (dummy=, > dummy2=, dummy3=, dummy4=) > at /share/dim/src/freebsd/src-dim/sys/ddb/db_command.c:575 > #3 0xffffffff804cc912 in db_command (last_cmdp=, > cmd_table=, dopager=dopager@entry=1) > at /share/dim/src/freebsd/src-dim/sys/ddb/db_command.c:482 > #4 0xffffffff804cc58d in db_command_loop () > at /share/dim/src/freebsd/src-dim/sys/ddb/db_command.c:535 > #5 0xffffffff804cfd06 in db_trap (type=, code= out>) > at /share/dim/src/freebsd/src-dim/sys/ddb/db_main.c:270 > #6 0xffffffff80c69f17 in kdb_trap (type=type@entry=3, code=code@entry=0, > tf=tf@entry=0xfffffe00747e88e0) > at /share/dim/src/freebsd/src-dim/sys/kern/subr_kdb.c:727 > #7 0xffffffff810d7aee in trap (frame=0xfffffe00747e88e0) > at /share/dim/src/freebsd/src-dim/sys/amd64/amd64/trap.c:576 > #8 > #9 kdb_enter (why=0xffffffff812d3d27 "panic", msg=) > at /share/dim/src/freebsd/src-dim/sys/kern/subr_kdb.c:506 > #10 0xffffffff80c1d248 in vpanic ( > fmt=0xffffffff8129dfef "%s: expected nul at %p; string [%s]\n", > ap=, ap@entry=0xfffffe00747e8a50) > at /share/dim/src/freebsd/src-dim/sys/kern/kern_shutdown.c:907 > #11 0xffffffff80c1cfd3 in panic ( > fmt=0xffffffff81e9b9c8 "=\t)\201\377\377\377\377") > at /share/dim/src/freebsd/src-dim/sys/kern/kern_shutdown.c:843 > #12 0xffffffff80cfa992 in lookup (ndp=ndp@entry=0xfffffe00747e8d90) > at /share/dim/src/freebsd/src-dim/sys/kern/vfs_lookup.c:919 > #13 0xffffffff80b33f84 in nfsvno_namei (nd=nd@entry=0xfffffe00747e9100, > ndp=ndp@entry=0xfffffe00747e8d90, dp=, > dp@entry=0xfffff80010544380, islocked=, > islocked@entry=0, > exp=exp@entry=0xfffffe00747e8fd8, p=p@entry=0xfffffe00bbfa3000, > retdirp=0xfffffe00747e8e78) > at /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdport.c:597 > #14 0xffffffff80b266a1 in nfsrvd_lookup (nd=0xfffffe00747e9100, > isdgram=, dp=0xfffff80010544380, vpp=0xfffffe00747e9010, > fhp=0xfffffe00747e9074, exp=0xfffffe00747e8fd8) > at /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdserv.c:607 > #15 0xffffffff80b1073b in nfsrvd_compound (nd=0xfffffe00747e9100, > isdgram=0, > tag=0xf , taglen=6, > minorvers=4294967294) > at > /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdsocket.c:1098 > #16 nfsrvd_dorpc (nd=nd@entry=0xfffffe00747e9100, isdgram=isdgram@entry=0, > tag=0xf , taglen=6, > minorvers=4294967294) > at /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdsocket.c:626 > #17 0xffffffff80b24c44 in nfs_proc (nd=0xfffffe00747e9100, > xid=, xprt=0xfffff80003a14e00, rpp=) > at /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdkrpc.c:402 > #18 nfssvc_program (rqst=0xfffff80010455800, xprt=0xfffff80003a14e00) > at /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdkrpc.c:288 > #19 0xffffffff80edead2 in svc_executereq (rqstp=0xfffff80010455800) > at /share/dim/src/freebsd/src-dim/sys/rpc/svc.c:1037 > #20 svc_run_internal (grp=, grp@entry=0xfffff800100e6100, > ismaster=ismaster@entry=1) > at /share/dim/src/freebsd/src-dim/sys/rpc/svc.c:1313 > #21 0xffffffff80eddf80 in svc_run (pool=) > at /share/dim/src/freebsd/src-dim/sys/rpc/svc.c:1392 > #22 0xffffffff80b251ec in nfsrvd_nfsd (td=, > td@entry=0xfffffe00bbfa3000, args=args@entry=0xfffffe00747e9660) > at /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdkrpc.c:561 > #23 0xffffffff80b3ec93 in nfssvc_nfsd (td=0xfffffe00bbfa3000, > uap=) > at /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdport.c:3714 > #24 0xffffffff80e6f647 in sys_nfssvc (td=0xfffffe00bbfa3000, > uap=0xfffffe00bbfa33e8) > at /share/dim/src/freebsd/src-dim/sys/nfs/nfs_nfssvc.c:111 > #25 0xffffffff810d891e in syscallenter (td=) > at > /share/dim/src/freebsd/src-dim/sys/amd64/amd64/../../kern/subr_syscall.c:189 > #26 amd64_syscall (td=0xfffffe00bbfa3000, traced=0) > at /share/dim/src/freebsd/src-dim/sys/amd64/amd64/trap.c:1156 > #27 > #28 0x00000008011aa59a in ?? () > > Is anybody seeing this too? :) > > I can probably bisect, but it'll take quite a while. > > -Dimitry > > -- Mateusz Guzik