Date: Fri, 26 Dec 2014 10:37:42 +1100 From: Peter Jeremy <peter@rulingia.com> To: freebsd-fs@freebsd.org Subject: "panic: len 0" on NFS read Message-ID: <20141225233742.GA3385@server.rulingia.com>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
Whilst trying to debug a RPC issue with a NFS tunneling tool, I mounted a
NFS filesystem onto the same host and got a panic when I tried to access it.
I'm running FreeBSD/amd64 10-stable r276177.
I mounted the filesystem with:
# mount -o udp,nfsv3 $(hostname):/tank/src92 /dist
(/tank/src92 and / are ZFS)
And then ran:
$ grep zzzz /dist/*
And got:
panic: len 0
cpuid = 3
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0861448f30
kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0861448fe0
vpanic() at vpanic+0x126/frame 0xfffffe0861449020
kassert_panic() at kassert_panic+0x139/frame 0xfffffe0861449090
nfsm_mbufuio() at nfsm_mbufuio+0x9c/frame 0xfffffe08614490f0
nfsrpc_read() at nfsrpc_read+0x584/frame 0xfffffe08614492d0
ncl_readrpc() at ncl_readrpc+0xa5/frame 0xfffffe08614493e0
ncl_doio() at ncl_doio+0x228/frame 0xfffffe0861449480
ncl_bioread() at ncl_bioread+0xb44/frame 0xfffffe08614495f0
VOP_READ_APV() at VOP_READ_APV+0xf1/frame 0xfffffe0861449620
vn_read() at vn_read+0x211/frame 0xfffffe0861449690
vn_io_fault_doio() at vn_io_fault_doio+0x22/frame 0xfffffe08614496d0
vn_io_fault1() at vn_io_fault1+0x7c/frame 0xfffffe0861449830
vn_io_fault() at vn_io_fault+0x18b/frame 0xfffffe08614498b0
dofileread() at dofileread+0x95/frame 0xfffffe0861449900
kern_readv() at kern_readv+0x68/frame 0xfffffe0861449950
sys_read() at sys_read+0x63/frame 0xfffffe08614499a0
amd64_syscall() at amd64_syscall+0x22e/frame 0xfffffe0861449ab0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0861449ab0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800fd3cba, rsp = 0x7fffffffe048, rbp = 0x7fffffffe090 ---
I have a crashdump that looks sane and relevant bits around nfsm_mbufuio() are:
#4 0xffffffff8041e63c in nfsm_mbufuio (nd=0xfffffe08614491b0, uiop=0xfffffe0861449420, siz=0x4000)
at /usr/src/sys/fs/nfs/nfs_commonsubs.c:222
(kgdb) p mp
$1 = 0xfffff80053bab500
(kgdb) p *mp
$2 = {
m_hdr = {
mh_next = 0xfffff8023433dc00,
mh_nextpkt = 0x0,
mh_data = 0xfffff80053bab57c "���"...,
mh_len = 0x0,
mh_type = 0x1,
mh_flags = 0x2
},
...
(kgdb) p *nd
$4 = {
nd_md = 0xfffff8005366c500,
nd_dpos = 0xfffff80562d92068 "���"...,
...
(kgdb) p *nd->nd_md
$5 = {
m_hdr = {
mh_next = 0xfffff80486b05b00,
mh_nextpkt = 0x0,
mh_data = 0xfffff80562d92000 "",
mh_len = 0x68,
mh_type = 0x1,
mh_flags = 0x1
},
...
(kgdb) p *$5.m_hdr.mh_next
$11 = {
m_hdr = {
mh_next = 0xfffff8005325e400,
mh_nextpkt = 0x0,
mh_data = 0xfffff80234291800 "�",
mh_len = 0x800,
mh_type = 0x1,
mh_flags = 0x1
},
...
(kgdb) p *$11.m_hdr.mh_next
$12 = {
m_hdr = {
mh_next = 0xfffff80486b02400,
mh_nextpkt = 0x0,
mh_data = 0xfffff8023453c000 "\t",
mh_len = 0x800,
mh_type = 0x1,
mh_flags = 0x1
},
...
(kgdb) p *$12.m_hdr.mh_next
$13 = {
m_hdr = {
mh_next = 0xfffff8023433f800,
mh_nextpkt = 0x0,
mh_data = 0xfffff80562d92800 "its",
mh_len = 0x800,
mh_type = 0x1,
mh_flags = 0x1
},
...
(kgdb) p *$13.m_hdr.mh_next
$14 = {
m_hdr = {
mh_next = 0xfffff80020f36500,
mh_nextpkt = 0x0,
mh_data = 0xfffff8058cb1b000 "sbconfig",
mh_len = 0x800,
mh_type = 0x1,
mh_flags = 0x1
},
...
(kgdb) p *$14.m_hdr.mh_next
$15 = {
m_hdr = {
mh_next = 0xfffff800533d5e00,
mh_nextpkt = 0x0,
mh_data = 0xfffff8041b423800 "",
mh_len = 0x800,
mh_type = 0x1,
mh_flags = 0x1
},
...
(kgdb) p *$15.m_hdr.mh_next
$16 = {
m_hdr = {
mh_next = 0xfffff80053182600,
mh_nextpkt = 0x0,
mh_data = 0xfffff8023429a800 "ilters",
mh_len = 0x800,
mh_type = 0x1,
mh_flags = 0x1
},
...
(kgdb) p *$16.m_hdr.mh_next
$17 = {
m_hdr = {
mh_next = 0xfffff8005379b200,
mh_nextpkt = 0x0,
mh_data = 0xfffff8058cb1e000 "",
mh_len = 0x800,
mh_type = 0x1,
mh_flags = 0x1
},
...
(kgdb) p *$17.m_hdr.mh_next
$18 = {
m_hdr = {
mh_next = 0xfffff80053bab500,
mh_nextpkt = 0x0,
mh_data = 0xfffff8058cb1c800 "\002",
mh_len = 0x760,
mh_type = 0x1,
mh_flags = 0x1
},
...
Which is points to mp.
I gather the first mbuf is NFS RPC metadata (since it's skipped). The
remaining mbufs are the start of a 3.9MB binary file (an identifier
database).
Any suggestions as to what has gone wrong?
--
Peter Jeremy
[-- Attachment #2 --]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQJ8BAEBCgBmBQJUnJ/GXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFRUIyOTg2QzMwNjcxRTc0RTY1QzIyN0Ux
NkE1OTdBMEU0QTIwQjM0AAoJEBall6Dkogs0qaUP/RjjDxHDYT2REtWcB79gnEt5
ixElylaIhpbND/uH3rqoJrKRon1WkaSX9TL1hLLT2zaktyWH2UPg3JNTSP5c3LGa
iipm71i0QUeaM60y6IwnkgRNUVLRDBO8t67DN7eLQudhZkdq7ew8VoaPCw23cN94
9QSOBTPC2rnVSE0/dHagw2i1+BOmR4XkynQYp/GtWxlvaRP/7lseDE73Jk5D9f5v
L1QTJabmfoA1LfgHa783T2Dvo+bxnRkeQFjUVwxMahXBbZYjXmRiWMLgC9jTZg1V
+HlbEvLpE9rJDGLPBxnMSf7/SYePg3B+iV81nmxT8/6n2fOsf1SmjWiaL7riQ9OO
gW1V5tDhBzeGqmwvRCjOYoZpn2yPdiVhFbPK4j+a3Bkkh0KbmgNJf0cuoNuQkGQF
qfY+a4wauLSWmWseadnkNXwk2gQjoL6HXiLsN+Z8sxo+lwJM6xn8TYkkt1pBovFp
fC3/CFGzuNTf5HSvVQ9I+yu+MwAQskR83jW5AL9pKs4Sphs6RsDQuZq5Knt1aLVx
eZQjNOTN0G5/HYX/R4eq5E1IISk/C/F5sI5kAGr6X90jnmVuHCDSRh+pvNHQqFqD
5aUfovtZFDGPXbv3y1RMBB+EgQw5ki9xSttHQ33yPHgCpPALZs9H/D0Y6SnLST+w
VIK21pknaW3pWFeT7g3k
=dyix
-----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20141225233742.GA3385>