From owner-freebsd-fs@FreeBSD.ORG Thu Dec 25 23:47:11 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E247DD0A for ; Thu, 25 Dec 2014 23:47:10 +0000 (UTC) Received: from vps.rulingia.com (vps.rulingia.com [103.243.244.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps.rulingia.com", Issuer "CAcert Class 3 Root" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 914092876 for ; Thu, 25 Dec 2014 23:47:09 +0000 (UTC) Received: from server.rulingia.com (c220-239-242-83.belrs5.nsw.optusnet.com.au [220.239.242.83]) by vps.rulingia.com (8.14.9/8.14.9) with ESMTP id sBPNbmWl036116 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Fri, 26 Dec 2014 10:37:54 +1100 (AEDT) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.14.9/8.14.9) with ESMTP id sBPNbgx6004733 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 26 Dec 2014 10:37:43 +1100 (AEDT) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.14.9/8.14.9/Submit) id sBPNbglm004732 for freebsd-fs@freebsd.org; Fri, 26 Dec 2014 10:37:42 +1100 (AEDT) (envelope-from peter) Date: Fri, 26 Dec 2014 10:37:42 +1100 From: Peter Jeremy To: freebsd-fs@freebsd.org Subject: "panic: len 0" on NFS read Message-ID: <20141225233742.GA3385@server.rulingia.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="x+6KMIRAuhnl3hBn" Content-Disposition: inline X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Dec 2014 23:47:11 -0000 --x+6KMIRAuhnl3hBn Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Whilst trying to debug a RPC issue with a NFS tunneling tool, I mounted a NFS filesystem onto the same host and got a panic when I tried to access it. I'm running FreeBSD/amd64 10-stable r276177. I mounted the filesystem with: # mount -o udp,nfsv3 $(hostname):/tank/src92 /dist (/tank/src92 and / are ZFS) And then ran: $ grep zzzz /dist/* And got: panic: len 0 cpuid =3D 3 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0861448= f30 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0861448fe0 vpanic() at vpanic+0x126/frame 0xfffffe0861449020 kassert_panic() at kassert_panic+0x139/frame 0xfffffe0861449090 nfsm_mbufuio() at nfsm_mbufuio+0x9c/frame 0xfffffe08614490f0 nfsrpc_read() at nfsrpc_read+0x584/frame 0xfffffe08614492d0 ncl_readrpc() at ncl_readrpc+0xa5/frame 0xfffffe08614493e0 ncl_doio() at ncl_doio+0x228/frame 0xfffffe0861449480 ncl_bioread() at ncl_bioread+0xb44/frame 0xfffffe08614495f0 VOP_READ_APV() at VOP_READ_APV+0xf1/frame 0xfffffe0861449620 vn_read() at vn_read+0x211/frame 0xfffffe0861449690 vn_io_fault_doio() at vn_io_fault_doio+0x22/frame 0xfffffe08614496d0 vn_io_fault1() at vn_io_fault1+0x7c/frame 0xfffffe0861449830 vn_io_fault() at vn_io_fault+0x18b/frame 0xfffffe08614498b0 dofileread() at dofileread+0x95/frame 0xfffffe0861449900 kern_readv() at kern_readv+0x68/frame 0xfffffe0861449950 sys_read() at sys_read+0x63/frame 0xfffffe08614499a0 amd64_syscall() at amd64_syscall+0x22e/frame 0xfffffe0861449ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0861449ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip =3D 0x800fd3cba, rsp =3D 0x7f= ffffffe048, rbp =3D 0x7fffffffe090 --- I have a crashdump that looks sane and relevant bits around nfsm_mbufuio() = are: #4 0xffffffff8041e63c in nfsm_mbufuio (nd=3D0xfffffe08614491b0, uiop=3D0xf= ffffe0861449420, siz=3D0x4000) at /usr/src/sys/fs/nfs/nfs_commonsubs.c:222 (kgdb) p mp $1 =3D 0xfffff80053bab500 (kgdb) p *mp $2 =3D { m_hdr =3D { mh_next =3D 0xfffff8023433dc00,=20 mh_nextpkt =3D 0x0,=20 mh_data =3D 0xfffff80053bab57c "=EF=BF=BD=EF=BF=BD=EF=BF=BD"..., mh_len =3D 0x0,=20 mh_type =3D 0x1,=20 mh_flags =3D 0x2 },=20 =2E.. (kgdb) p *nd $4 =3D { nd_md =3D 0xfffff8005366c500,=20 nd_dpos =3D 0xfffff80562d92068 "=EF=BF=BD=EF=BF=BD=EF=BF=BD"..., =2E.. (kgdb) p *nd->nd_md $5 =3D { m_hdr =3D { mh_next =3D 0xfffff80486b05b00,=20 mh_nextpkt =3D 0x0,=20 mh_data =3D 0xfffff80562d92000 "",=20 mh_len =3D 0x68,=20 mh_type =3D 0x1,=20 mh_flags =3D 0x1 },=20 =2E.. (kgdb) p *$5.m_hdr.mh_next $11 =3D { m_hdr =3D { mh_next =3D 0xfffff8005325e400,=20 mh_nextpkt =3D 0x0,=20 mh_data =3D 0xfffff80234291800 "=EF=BF=BD",=20 mh_len =3D 0x800,=20 mh_type =3D 0x1,=20 mh_flags =3D 0x1 },=20 =2E.. (kgdb) p *$11.m_hdr.mh_next $12 =3D { m_hdr =3D { mh_next =3D 0xfffff80486b02400,=20 mh_nextpkt =3D 0x0,=20 mh_data =3D 0xfffff8023453c000 "\t",=20 mh_len =3D 0x800,=20 mh_type =3D 0x1,=20 mh_flags =3D 0x1 },=20 =2E.. (kgdb) p *$12.m_hdr.mh_next $13 =3D { m_hdr =3D { mh_next =3D 0xfffff8023433f800,=20 mh_nextpkt =3D 0x0,=20 mh_data =3D 0xfffff80562d92800 "its",=20 mh_len =3D 0x800,=20 mh_type =3D 0x1,=20 mh_flags =3D 0x1 },=20 =2E.. (kgdb) p *$13.m_hdr.mh_next $14 =3D { m_hdr =3D { mh_next =3D 0xfffff80020f36500,=20 mh_nextpkt =3D 0x0,=20 mh_data =3D 0xfffff8058cb1b000 "sbconfig",=20 mh_len =3D 0x800,=20 mh_type =3D 0x1,=20 mh_flags =3D 0x1 },=20 =2E.. (kgdb) p *$14.m_hdr.mh_next $15 =3D { m_hdr =3D { mh_next =3D 0xfffff800533d5e00,=20 mh_nextpkt =3D 0x0,=20 mh_data =3D 0xfffff8041b423800 "",=20 mh_len =3D 0x800,=20 mh_type =3D 0x1,=20 mh_flags =3D 0x1 },=20 =2E.. (kgdb) p *$15.m_hdr.mh_next $16 =3D { m_hdr =3D { mh_next =3D 0xfffff80053182600,=20 mh_nextpkt =3D 0x0,=20 mh_data =3D 0xfffff8023429a800 "ilters",=20 mh_len =3D 0x800,=20 mh_type =3D 0x1,=20 mh_flags =3D 0x1 },=20 =2E.. (kgdb) p *$16.m_hdr.mh_next $17 =3D { m_hdr =3D { mh_next =3D 0xfffff8005379b200,=20 mh_nextpkt =3D 0x0,=20 mh_data =3D 0xfffff8058cb1e000 "",=20 mh_len =3D 0x800,=20 mh_type =3D 0x1,=20 mh_flags =3D 0x1 },=20 =2E.. (kgdb) p *$17.m_hdr.mh_next $18 =3D { m_hdr =3D { mh_next =3D 0xfffff80053bab500,=20 mh_nextpkt =3D 0x0,=20 mh_data =3D 0xfffff8058cb1c800 "\002",=20 mh_len =3D 0x760,=20 mh_type =3D 0x1,=20 mh_flags =3D 0x1 },=20 =2E.. Which is points to mp. I gather the first mbuf is NFS RPC metadata (since it's skipped). The remaining mbufs are the start of a 3.9MB binary file (an identifier database). Any suggestions as to what has gone wrong? --=20 Peter Jeremy --x+6KMIRAuhnl3hBn Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJUnJ/GXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFRUIyOTg2QzMwNjcxRTc0RTY1QzIyN0Ux NkE1OTdBMEU0QTIwQjM0AAoJEBall6Dkogs0qaUP/RjjDxHDYT2REtWcB79gnEt5 ixElylaIhpbND/uH3rqoJrKRon1WkaSX9TL1hLLT2zaktyWH2UPg3JNTSP5c3LGa iipm71i0QUeaM60y6IwnkgRNUVLRDBO8t67DN7eLQudhZkdq7ew8VoaPCw23cN94 9QSOBTPC2rnVSE0/dHagw2i1+BOmR4XkynQYp/GtWxlvaRP/7lseDE73Jk5D9f5v L1QTJabmfoA1LfgHa783T2Dvo+bxnRkeQFjUVwxMahXBbZYjXmRiWMLgC9jTZg1V +HlbEvLpE9rJDGLPBxnMSf7/SYePg3B+iV81nmxT8/6n2fOsf1SmjWiaL7riQ9OO gW1V5tDhBzeGqmwvRCjOYoZpn2yPdiVhFbPK4j+a3Bkkh0KbmgNJf0cuoNuQkGQF qfY+a4wauLSWmWseadnkNXwk2gQjoL6HXiLsN+Z8sxo+lwJM6xn8TYkkt1pBovFp fC3/CFGzuNTf5HSvVQ9I+yu+MwAQskR83jW5AL9pKs4Sphs6RsDQuZq5Knt1aLVx eZQjNOTN0G5/HYX/R4eq5E1IISk/C/F5sI5kAGr6X90jnmVuHCDSRh+pvNHQqFqD 5aUfovtZFDGPXbv3y1RMBB+EgQw5ki9xSttHQ33yPHgCpPALZs9H/D0Y6SnLST+w VIK21pknaW3pWFeT7g3k =dyix -----END PGP SIGNATURE----- --x+6KMIRAuhnl3hBn--