From owner-freebsd-fs@FreeBSD.ORG Mon Nov 28 14:19:28 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E2C52106566C; Mon, 28 Nov 2011 14:19:28 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id 9553F8FC0C; Mon, 28 Nov 2011 14:19:28 +0000 (UTC) Received: from localhost (89-73-195-149.dynamic.chello.pl [89.73.195.149]) by mail.dawidek.net (Postfix) with ESMTPSA id D80C13AF; Sat, 26 Nov 2011 11:49:47 +0100 (CET) Date: Sat, 26 Nov 2011 11:48:41 +0100 From: Pawel Jakub Dawidek To: Mark Felder Message-ID: <20111126104840.GA8794@garage.freebsd.pl> References: <95d00c1b714837aa32e7da72bc4afd03@feld.me> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ikeVEW9yuYc//A+q" Content-Disposition: inline In-Reply-To: <95d00c1b714837aa32e7da72bc4afd03@feld.me> X-OS: FreeBSD 9.0-CURRENT amd64 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: zfs i/o hangs on 9-PRERELEASE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Nov 2011 14:19:29 -0000 --ikeVEW9yuYc//A+q Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Nov 25, 2011 at 01:20:01PM -0600, Mark Felder wrote: > 13:14:32 nas:~ > uname -a > FreeBSD nas.feld.me 9.0-PRERELEASE FreeBSD 9.0-PRERELEASE #3 r227971M:=20 > Fri Nov 25 10:07:48 CST 2011 =20 > root@nas.feld.me:/usr/obj/tank/svn/sys/GENERIC amd64 >=20 > This seemed to start happening sometime after RC1. I tried 8-STABLE and= =20 > it's happening there too right now. I think whatever caused this was=20 > MFC'd. I've also reproduced this on completely different hardware=20 > running a single disk ZFS pool. >=20 >=20 > I'm getting this output in dmesg after these hangs I keep seeing. Mark, those backtrace are not related to ZFS, but to PF. Not sure if they are at all related to your hangs. Most cases where ZFS I/O seems to hang are hardware problems, where I/O requests are not completed. 'procstat -kk -a' output might be useful once the hang happens. > uma_zalloc_arg: zone "pfrktable" with the following non-sleepable locks= =20 > held: > exclusive sleep mutex pf task mtx (pf task mtx) r =3D 0=20 > (0xffffffff8199af20) locked @=20 > /tank/svn/sys/modules/pf/../../contrib/pf/net/pf_ioctl.c:1589 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > kdb_backtrace() at kdb_backtrace+0x37 > _witness_debugger() at _witness_debugger+0x2e > witness_warn() at witness_warn+0x2c4 > uma_zalloc_arg() at uma_zalloc_arg+0x335 > pfr_create_ktable() at pfr_create_ktable+0xd8 > pfr_ina_define() at pfr_ina_define+0x12b > pfioctl() at pfioctl+0x1c5a > devfs_ioctl_f() at devfs_ioctl_f+0x7a > kern_ioctl() at kern_ioctl+0xcd > sys_ioctl() at sys_ioctl+0xfd > amd64_syscall() at amd64_syscall+0x3ac > Xfast_syscall() at Xfast_syscall+0xf7 > --- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x800da711c, rsp =3D= =20 > 0x7fffffff9d28, rbp =3D 0x7fffffffa1f0 --- --=20 Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://yomoli.com --ikeVEW9yuYc//A+q Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk7QxAgACgkQForvXbEpPzR0VgCfR/mF7sxZOaNYoHcsvOIDTljh Re0AnR9RoDZr4yLmuwSqGrEaaLDu4B1E =pCIh -----END PGP SIGNATURE----- --ikeVEW9yuYc//A+q--