From owner-freebsd-net@FreeBSD.ORG Thu Jan 24 21:22:20 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 6F3C72D6; Thu, 24 Jan 2013 21:22:20 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id BF89B7DA; Thu, 24 Jan 2013 21:22:19 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0OLMCUx094635; Thu, 24 Jan 2013 23:22:12 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0OLMCUx094635 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0OLMCx5094634; Thu, 24 Jan 2013 23:22:12 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 24 Jan 2013 23:22:12 +0200 From: Konstantin Belousov To: Christian Gusenbauer Subject: Re: 9.1-stable crashes while copying data from a NFS mounted directory Message-ID: <20130124212212.GM2522@kib.kiev.ua> References: <201301241805.57623.c47g@gmx.at> <201301241950.49455.c47g@gmx.at> <20130124193709.GL2522@kib.kiev.ua> <201301242150.52238.c47g@gmx.at> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="RS1722//baS0C3Tp" Content-Disposition: inline In-Reply-To: <201301242150.52238.c47g@gmx.at> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-fs@freebsd.org, net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jan 2013 21:22:20 -0000 --RS1722//baS0C3Tp Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jan 24, 2013 at 09:50:52PM +0100, Christian Gusenbauer wrote: > On Thursday 24 January 2013 20:37:09 Konstantin Belousov wrote: > > On Thu, Jan 24, 2013 at 07:50:49PM +0100, Christian Gusenbauer wrote: > > > On Thursday 24 January 2013 19:07:23 Konstantin Belousov wrote: > > > > On Thu, Jan 24, 2013 at 08:03:59PM +0200, Konstantin Belousov wrote: > > > > > On Thu, Jan 24, 2013 at 06:05:57PM +0100, Christian Gusenbauer wr= ote: > > > > > > Hi! > > > > > >=20 > > > > > > I'm using 9.1 stable svn revision 245605 and I get the panic be= low > > > > > > if I execute the following commands (as single user): > > > > > >=20 > > > > > > # swapon -a > > > > > > # dumpon /dev/ada0s3b > > > > > > # mount -u / > > > > > > # ifconfig age0 inet 192.168.2.2 mtu 6144 up > > > > > > # mount -t nfs -o rsize=3D32768 data:/multimedia /mnt > > > > > > # cp /mnt/Movies/test/a.m2ts /tmp > > > > > >=20 > > > > > > then the system panics almost immediately. I'll attach the stack > > > > > > trace. > > > > > >=20 > > > > > > Note, that I'm using jumbo frames (6144 byte) on a 1Gbit networ= k, > > > > > > maybe that's the cause for the panic, because the bcopy (see st= ack > > > > > > frame #15) fails. > > > > > >=20 > > > > > > Any clues? > > > > >=20 > > > > > I tried a similar operation with the nfs mount of rsize=3D32768 a= nd mtu > > > > > 6144, but the machine runs HEAD and em instead of age. I was unab= le > > > > > to reproduce the panic on the copy of the 5GB file from nfs mount. > > >=20 > > > Hmmm, I did a quick test. If I do not change the MTU, so just configu= ring > > > age0 with > > >=20 > > > # ifconfig age0 inet 192.168.2.2 up > > >=20 > > > then I can copy all files from the mounted directory without any > > > problems, too. So it's probably age0 related? > >=20 > > From your backtrace and the buffer printout, I see somewhat strange thi= ng. > > The buffer data address is 0xffffff8171418000, while kernel faulted > > at the attempt to write at 0xffffff8171413000, which is is lower then > > the buffer data pointer, at the attempt to bcopy to the buffer. > >=20 > > The other data suggests that there were no overflow of the data from the > > server response. So it might be that mbuf_len(mp) returned negative num= ber > > ? I am not sure is it possible at all. > >=20 > > Try this debugging patch, please. You need to add INVARIANTS etc to the > > kernel config. > >=20 > > diff --git a/sys/fs/nfs/nfs_commonsubs.c b/sys/fs/nfs/nfs_commonsubs.c > > index efc0786..9a6bda5 100644 > > --- a/sys/fs/nfs/nfs_commonsubs.c > > +++ b/sys/fs/nfs/nfs_commonsubs.c > > @@ -218,6 +218,7 @@ nfsm_mbufuio(struct nfsrv_descript *nd, struct uio > > *uiop, int siz) } > > mbufcp =3D NFSMTOD(mp, caddr_t); > > len =3D mbuf_len(mp); > > + KASSERT(len > 0, ("len %d", len)); > > } > > xfer =3D (left > len) ? len : left; > > #ifdef notdef > > @@ -239,6 +240,8 @@ nfsm_mbufuio(struct nfsrv_descript *nd, struct uio > > *uiop, int siz) uiop->uio_resid -=3D xfer; > > } > > if (uiop->uio_iov->iov_len <=3D siz) { > > + KASSERT(uiop->uio_iovcnt > 1, ("uio_iovcnt %d", > > + uiop->uio_iovcnt)); > > uiop->uio_iovcnt--; > > uiop->uio_iov++; > > } else { > >=20 > > I thought that server have returned too long response, but it seems to > > be not the case from your data. Still, I think the patch below might be > > due. > >=20 > > diff --git a/sys/fs/nfsclient/nfs_clrpcops.c > > b/sys/fs/nfsclient/nfs_clrpcops.c index be0476a..a89b907 100644 > > --- a/sys/fs/nfsclient/nfs_clrpcops.c > > +++ b/sys/fs/nfsclient/nfs_clrpcops.c > > @@ -1444,7 +1444,7 @@ nfsrpc_readrpc(vnode_t vp, struct uio *uiop, stru= ct > > ucred *cred, NFSM_DISSECT(tl, u_int32_t *, NFSX_UNSIGNED); > > eof =3D fxdr_unsigned(int, *tl); > > } > > - NFSM_STRSIZ(retlen, rsize); > > + NFSM_STRSIZ(retlen, len); > > error =3D nfsm_mbufuio(nd, uiop, retlen); > > if (error) > > goto nfsmout; >=20 > I applied your patches and now I get a >=20 > panic: len -4 > cpuid =3D 1 > KDB: enter: panic > Dumping 377 out of 6116 MB:..5%..13%..22%..34%..43%..51%..64%..73%..81%..= 94% >=20 This means that the age driver either produced corrupted mbuf chain, or filled wrong negative value into the mbuf len field. I am quite certain that the issue is in the driver. I added the net@ to Cc:, hopefully you could get help there. >=20 > #0 doadump (textdump=3D0) > at /spare/tmp/src-stable9/sys/kern/kern_shutdown.c:265 > 265 if (textdump && textdump_pending) { > (kgdb) #0 doadump (textdump=3D0) > at /spare/tmp/src-stable9/sys/kern/kern_shutdown.c:265 > #1 0xffffffff802a7490 in db_dump (dummy=3D, > dummy2=3D, dummy3=3D, > dummy4=3D) > at /spare/tmp/src-stable9/sys/ddb/db_command.c:538 > #2 0xffffffff802a6a7e in db_command (last_cmdp=3D0xffffffff808ca140, > cmd_table=3D, dopager=3D1) > at /spare/tmp/src-stable9/sys/ddb/db_command.c:449 > #3 0xffffffff802a6cd0 in db_command_loop () > at /spare/tmp/src-stable9/sys/ddb/db_command.c:502 > #4 0xffffffff802a8e29 in db_trap (type=3D, > code=3D) > at /spare/tmp/src-stable9/sys/ddb/db_main.c:231 > #5 0xffffffff803bf548 in kdb_trap (type=3D3, code=3D0, tf=3D0xffffff81b2= ba1080) > at /spare/tmp/src-stable9/sys/kern/subr_kdb.c:649 > #6 0xffffffff80594c28 in trap (frame=3D0xffffff81b2ba1080) > at /spare/tmp/src-stable9/sys/amd64/amd64/trap.c:579 > #7 0xffffffff8057e06f in calltrap () > at /spare/tmp/src-stable9/sys/amd64/amd64/exception.S:228 > #8 0xffffffff803beffb in kdb_enter (why=3D0xffffffff8060ebcf "panic", > msg=3D0x80
) at cpufunc.h:63 > #9 0xffffffff80389391 in panic (fmt=3D) > at /spare/tmp/src-stable9/sys/kern/kern_shutdown.c:627 > #10 0xffffffff81e5bab2 in nfsm_mbufuio (nd=3D0xffffff81b2ba1340, uiop=3D0= x7cf, > siz=3D18) > at /spare/tmp/src-stable9/sys/modules/nfscommon/../../fs/nfs/nfs_comm= onsubs.c:202 > #11 0xffffffff81e195c1 in nfsrpc_read (vp=3D0xfffffe0006c94dc8, > uiop=3D0xffffff81b2ba15c0, cred=3D, > p=3D0xfffffe0006aa6490, nap=3D0xffffff81b2ba14a0, > attrflagp=3D0xffffff81b2ba156c, stuff=3D0x0) > at /spare/tmp/src-stable9/sys/modules/nfscl/../../fs/nfsclient/nfs_cl= rpcops.c:1343 > #12 0xffffffff81e3bd80 in ncl_readrpc (vp=3D0xfffffe0006c94dc8, > uiop=3D0xffffff81b2ba15c0, cred=3D) > at /spare/tmp/src-stable9/sys/modules/nfscl/../../fs/nfsclient/nfs_cl= vnops.c:1366 > #13 0xffffffff81e3086b in ncl_doio (vp=3D0xfffffe0006c94dc8, > bp=3D0xffffff816f8f4120, cr=3D0xfffffe0002d58e00, td=3D0xfffffe0006aa= 6490, > called_from_strategy=3D0) > at /spare/tmp/src-stable9/sys/modules/nfscl/../../fs/nfsclient/nfs_cl= bio.c:1605 > #14 0xffffffff81e3254f in ncl_bioread (vp=3D0xfffffe0006c94dc8, > uio=3D0xffffff81b2ba1ad0, ioflag=3D, > cred=3D0xfffffe0002d58e00) > at /spare/tmp/src-stable9/sys/modules/nfscl/../../fs/nfsclient/nfs_cl= bio.c:541 > #15 0xffffffff80434ae8 in vn_read (fp=3D0xfffffe0006abda50, > uio=3D0xffffff81b2ba1ad0, active_cred=3D, > flags=3D, td=3D) at vnode_i= f.h:384 > #16 0xffffffff8043206e in vn_io_fault (fp=3D0xfffffe0006abda50, > uio=3D0xffffff81b2ba1ad0, active_cred=3D0xfffffe0002d58e00, flags=3D0, > td=3D0xfffffe0006aa6490) at /spare/tmp/src-stable9/sys/kern/vfs_vnops= =2Ec:903 > #17 0xffffffff803d7ac1 in dofileread (td=3D0xfffffe0006aa6490, fd=3D3, > fp=3D0xfffffe0006abda50, auio=3D0xffffff81b2ba1ad0, > offset=3D, flags=3D0) at file.h:287 > #18 0xffffffff803d7e1c in kern_readv (td=3D0xfffffe0006aa6490, fd=3D3, > auio=3D0xffffff81b2ba1ad0) > at /spare/tmp/src-stable9/sys/kern/sys_generic.c:250 > #19 0xffffffff803d7f34 in sys_read (td=3D, > uap=3D) > at /spare/tmp/src-stable9/sys/kern/sys_generic.c:166 > #20 0xffffffff80593cb3 in amd64_syscall (td=3D0xfffffe0006aa6490, traced= =3D0) > at subr_syscall.c:135 > #21 0xffffffff8057e357 in Xfast_syscall () > at /spare/tmp/src-stable9/sys/amd64/amd64/exception.S:387 > #22 0x00000008009245fc in ?? () > Previous frame inner to this frame (corrupt stack?) > (kgdb) --RS1722//baS0C3Tp Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJRAaYEAAoJEJDCuSvBvK1B9dwP/109z+fMvI1HtV7JEV64d/TO DXy6Kicq6vVIhWmIpmWnWITbrt2IOIhTpqaw3m5F9jDMP9p1MvTnOuyuJ8bB77Kx ZOMg59gQUaJmfWpR2yPOMOSAw1OAbgPkke3gX+Tn4NEyUrS3p32AR9+tN79QE/Tj ZrPJ8weZ/8A5f76ZfaQMbbrOohvnYc5kHMs9avwc2/x2hiLMNHBC0k/c5ZPAZ9mx o8FH/4HeAcPdUjyobKKVohoJyv1DeQqwOvw086fz2gNhz7uIdVRQE2kI9QytEyz4 T+lHaTMG0QYkWZ2bYK9kkpwpBBGp7pBcP4flRnHExr5WNQk7BHKVYtqhgy3M4r7J heH4lsUnujDKQtIjxALfYH37BUE9oKuca5g3sYDHwL0tTQg0Hh/Bh1ZkD/dLTqO+ Ju5+NmRl18sdQ/jRZlbD/ljTor86YugGpWt6EZnRjMnEgsstKqwhJW+zukzIhgYU NcGSAZL7dhaZXkrzQ2SXPb/n74llmu3Nhhpgavvyfz72FtB2A8g9b8bPTCUeDVfn pzrN4RFAaoXAf9F9Jd0VQhlPIunWNJ7LPYKfu7MjZ2F+Gc5q38qwIrCWr8l9j2R1 oP2Sw3P2rxfsdhXpyqiTUkigXtlrC3SmeV/KzCstSaJghHXTURzUxdSiU5M2GV/v IzHs0gCb02hj3bXW9Lz1 =NsNv -----END PGP SIGNATURE----- --RS1722//baS0C3Tp--