From owner-freebsd-current@FreeBSD.ORG Sun May 4 07:03:05 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1604E84E for ; Sun, 4 May 2014 07:03:05 +0000 (UTC) Received: from smtprelay06.ispgateway.de (smtprelay06.ispgateway.de [80.67.31.101]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9C0041664 for ; Sun, 4 May 2014 07:03:03 +0000 (UTC) Received: from [84.44.153.26] (helo=fabiankeil.de) by smtprelay06.ispgateway.de with esmtpsa (SSLv3:AES128-SHA:128) (Exim 4.68) (envelope-from ) id 1WgqM4-0000UC-HT for freebsd-current@freebsd.org; Sun, 04 May 2014 08:56:56 +0200 Date: Sun, 4 May 2014 08:57:00 +0200 From: Fabian Keil To: freebsd-current@freebsd.org Subject: Re: Fatal double fault in ZFS with yesterday's CURRENT Message-ID: <20140504085700.797a642c@fabiankeil.de> In-Reply-To: <229058B87F604A469C70F634AA1C793D@multiplay.co.uk> References: <20140503102923.6fadd904@fabiankeil.de> <20140503191424.16f9744b@fabiankeil.de> <229058B87F604A469C70F634AA1C793D@multiplay.co.uk> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/2UVsS_uQvL.K9K/1fV3_msB"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 May 2014 07:03:05 -0000 --Sig_/2UVsS_uQvL.K9K/1fV3_msB Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable "Steven Hartland" wrote: > > "Steven Hartland" wrote: > >=20 > > > From: "Fabian Keil" > > >=20 > > > > After updating my laptop to yesterday's CURRENT (r265216), > > > > I got the following fatal double fault on boot: > > > > http://www.fabiankeil.de/bilder/freebsd/kernel-panic-r265216/ > > > >=20 > > > > My previous kernel was based on r264721. > > > > > > > > I'm using a couple of custom patches, some of them are ZFS-related > > > > and thus may be part of the problem (but worked fine for months). > > > > I'll try to reproduce the panic without the patches tomorrow. > > > > > > >=20 > > > Your seeing a stack overflow in the new ZFS queuing code, which I > > > believe is being triggered by lack of support for TRIM in one of > > > your devices, something Xin reported to me yesterday. > > >=20 > > > I commited a fix for failing TRIM requests processing slowly last > > > night so you could try updating to after r265253 and see if that > > > helps. > >=20 > > Thanks. The hard disk is indeed unlikely to support TRIM requests, > > but I can still reproduce the problem with a kernel based on r265255. >=20 > Thanks for testing, I suspect its still a numbers game with how many items > are outstanding in the queue and now that free / TRIM requests are also > now queued its triggering the failure. >=20 > If your just on a HDD try setting the following in /boot/loader.conf as > a temporary workaround: > vfs.zfs.trim.enabled=3D0 That worked, thanks. > > > I still need to investigate the stack overflow more directly which > > > appears to be caused by the new zfs queuing code when things are > > > running slowly and there's a large backlog of IO's. > > > > > > I would be interested to know you config there so zpool layout and > > > hardware in the mean time. > >=20 > > The system is a Lenovo ThinkPad R500: > > http://www.nycbug.org/index.cgi?action=3Ddmesgd&do=3Dview&dmesgid=3D2449 > >=20 > > I'm booting from UFS, the panic occurs while the pool is being imported. > >=20 > > The pool is located on a single geli-encrypted slice: > >=20 > > fk@r500 ~ $zpool status tank > > pool: tank > > state: ONLINE > > scan: scrub repaired 0 in 4h11m with 0 errors on Sat Mar 22 18:25:01 = 2014 > > config: > >=20 > > NAME STATE READ WRITE CKSUM > > tank ONLINE 0 0 0 > > ada0s1d.eli ONLINE 0 0 0 > >=20 > > errors: No known data errors > >=20 > > Maybe geli fails TRIM requests differently. >=20 > That helps, Xin also reported the issue with geli and thats what I'm test= ing > with, I believe this is a factor because is significantly slows things do= wn > again meaning more items in the queues, but I've only managed to trigger = it > once here as the machine I'm using is pretty quick. It probably doesn't make a difference, but my system is rather old and thus I'm still using geli version 3 for ada0s1d.eli while geli init nowadays defaults to geli version 7. The system certainly is also slow, though. Fabian --Sig_/2UVsS_uQvL.K9K/1fV3_msB Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iEYEARECAAYFAlNl5MIACgkQBYqIVf93VJ3NDwCeKacnkLudZNH1Rem1ryNK/lDs mUMAn1jrUXf89exocIHHtvAM4TOnTG9C =b4CZ -----END PGP SIGNATURE----- --Sig_/2UVsS_uQvL.K9K/1fV3_msB--