From owner-freebsd-stable@FreeBSD.ORG Thu Jan 24 19:37:21 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 667BC7E4; Thu, 24 Jan 2013 19:37:21 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id CA6466C; Thu, 24 Jan 2013 19:37:20 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0OJb9uU071565; Thu, 24 Jan 2013 21:37:09 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0OJb9uU071565 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0OJb9HT071564; Thu, 24 Jan 2013 21:37:09 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 24 Jan 2013 21:37:09 +0200 From: Konstantin Belousov To: Christian Gusenbauer Subject: Re: 9.1-stable crashes while copying data from a NFS mounted directory Message-ID: <20130124193709.GL2522@kib.kiev.ua> References: <201301241805.57623.c47g@gmx.at> <20130124180359.GH2522@kib.kiev.ua> <20130124180723.GI2522@kib.kiev.ua> <201301241950.49455.c47g@gmx.at> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="08mGw3CZk5wypUJm" Content-Disposition: inline In-Reply-To: <201301241950.49455.c47g@gmx.at> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jan 2013 19:37:21 -0000 --08mGw3CZk5wypUJm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jan 24, 2013 at 07:50:49PM +0100, Christian Gusenbauer wrote: > On Thursday 24 January 2013 19:07:23 Konstantin Belousov wrote: > > On Thu, Jan 24, 2013 at 08:03:59PM +0200, Konstantin Belousov wrote: > > > On Thu, Jan 24, 2013 at 06:05:57PM +0100, Christian Gusenbauer wrote: > > > > Hi! > > > >=20 > > > > I'm using 9.1 stable svn revision 245605 and I get the panic below = if I > > > > execute the following commands (as single user): > > > >=20 > > > > # swapon -a > > > > # dumpon /dev/ada0s3b > > > > # mount -u / > > > > # ifconfig age0 inet 192.168.2.2 mtu 6144 up > > > > # mount -t nfs -o rsize=3D32768 data:/multimedia /mnt > > > > # cp /mnt/Movies/test/a.m2ts /tmp > > > >=20 > > > > then the system panics almost immediately. I'll attach the stack tr= ace. > > > >=20 > > > > Note, that I'm using jumbo frames (6144 byte) on a 1Gbit network, m= aybe > > > > that's the cause for the panic, because the bcopy (see stack frame > > > > #15) fails. > > > >=20 > > > > Any clues? > > >=20 > > > I tried a similar operation with the nfs mount of rsize=3D32768 and m= tu > > > 6144, but the machine runs HEAD and em instead of age. I was unable to > > > reproduce the panic on the copy of the 5GB file from nfs mount. >=20 > Hmmm, I did a quick test. If I do not change the MTU, so just configuring= age0=20 > with >=20 > # ifconfig age0 inet 192.168.2.2 up >=20 > then I can copy all files from the mounted directory without any problems= ,=20 > too. So it's probably age0 related? =46rom your backtrace and the buffer printout, I see somewhat strange thing. The buffer data address is 0xffffff8171418000, while kernel faulted at the attempt to write at 0xffffff8171413000, which is is lower then the buffer data pointer, at the attempt to bcopy to the buffer. The other data suggests that there were no overflow of the data from the server response. So it might be that mbuf_len(mp) returned negative number ? I am not sure is it possible at all. Try this debugging patch, please. You need to add INVARIANTS etc to the kernel config. diff --git a/sys/fs/nfs/nfs_commonsubs.c b/sys/fs/nfs/nfs_commonsubs.c index efc0786..9a6bda5 100644 --- a/sys/fs/nfs/nfs_commonsubs.c +++ b/sys/fs/nfs/nfs_commonsubs.c @@ -218,6 +218,7 @@ nfsm_mbufuio(struct nfsrv_descript *nd, struct uio *uio= p, int siz) } mbufcp =3D NFSMTOD(mp, caddr_t); len =3D mbuf_len(mp); + KASSERT(len > 0, ("len %d", len)); } xfer =3D (left > len) ? len : left; #ifdef notdef @@ -239,6 +240,8 @@ nfsm_mbufuio(struct nfsrv_descript *nd, struct uio *uio= p, int siz) uiop->uio_resid -=3D xfer; } if (uiop->uio_iov->iov_len <=3D siz) { + KASSERT(uiop->uio_iovcnt > 1, ("uio_iovcnt %d", + uiop->uio_iovcnt)); uiop->uio_iovcnt--; uiop->uio_iov++; } else { I thought that server have returned too long response, but it seems to be not the case from your data. Still, I think the patch below might be due. diff --git a/sys/fs/nfsclient/nfs_clrpcops.c b/sys/fs/nfsclient/nfs_clrpcop= s.c index be0476a..a89b907 100644 --- a/sys/fs/nfsclient/nfs_clrpcops.c +++ b/sys/fs/nfsclient/nfs_clrpcops.c @@ -1444,7 +1444,7 @@ nfsrpc_readrpc(vnode_t vp, struct uio *uiop, struct u= cred *cred, NFSM_DISSECT(tl, u_int32_t *, NFSX_UNSIGNED); eof =3D fxdr_unsigned(int, *tl); } - NFSM_STRSIZ(retlen, rsize); + NFSM_STRSIZ(retlen, len); error =3D nfsm_mbufuio(nd, uiop, retlen); if (error) goto nfsmout; --08mGw3CZk5wypUJm Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJRAY1kAAoJEJDCuSvBvK1B3GIP/R35aJSwVPR4UYtMoamu5VC/ eYrSSZ8iREbFXRIWCmBBY3dZGgGB83/h0jo7p9Q9C1eQTuo+3oPc9L3HFp6m9d4c +Lcf/S+GMi9Ncmm6QbNv8IM3WOhBhPnimQlqoJkP0qTYNUoHGCJQSEJuXXs8ca/8 Nl84xHMiuYoc2WU+sQ8gn32+EFDp4DeEzlbXPTT0hLTxgBrMZWsjjGDCJ61POx+u Fy2EcoAGmFQWxM33RZbvp2LPnrO6fczF6Q38kvGsVWsljz0N0vj9ZumV/9H+hElO gkkaAQyy77Kh1AhaAEFLvIkpQQu63FJsUJiNYSauIbO3SM23/7pB75+WPUEHUe0l 6d7NlkYhcxkltVms+AGDNYcOzXmaLr3kvd8iCOeC8EssXGM0Gtsd1IOUBicmZIXK FkI4qCbY7FvrBkiEV59xISIAkYnXH4ov/GZ5ks+VSy0mLr9f2wiwMLsrWdQYNbpm 2m4sKy+ubfHzb0ZOV4R49qR/IJhDbQSOW6J0Ro0GF/Y4zOMl3xOxcOC9zPMixa7E reHs/jERnmH9Rwb7J8U180VNujy8Oab2JXgB5/KrVqO2UVgRMidMirqqMsa+EYiU hu1SdaM5LA+qcmi4FbXhEXLzvGk16o6JnavbIY8IT9hDDtljBDJYP0jnHGjCBWLw naLEkaMdchHSCKIIrRpI =MFh3 -----END PGP SIGNATURE----- --08mGw3CZk5wypUJm--