From owner-freebsd-current@FreeBSD.ORG Mon Nov 28 14:39:27 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1B6851065673; Mon, 28 Nov 2011 14:39:27 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 8FBAC8FC16; Mon, 28 Nov 2011 14:39:26 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap4EAPOb006DaFvO/2dsb2JhbAA7CIUDpwCBcgEBBSMyJBsYAgINGQJZBhOtGpFXgTCCL4NSghuBFgSIIYwpkik X-IronPort-AV: E=Sophos;i="4.69,584,1315195200"; d="scan'208";a="147318916" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 28 Nov 2011 09:39:25 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 66ED0B406C; Mon, 28 Nov 2011 09:39:25 -0500 (EST) Date: Mon, 28 Nov 2011 09:39:25 -0500 (EST) From: Rick Macklem To: Pawel Jakub Dawidek Message-ID: <2111049507.479178.1322491165407.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <20111126104840.GA8794@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org, Mark Felder , freebsd-current@freebsd.org Subject: Re: zfs i/o hangs on 9-PRERELEASE X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Nov 2011 14:39:27 -0000 Pawel Jakub Dawidek wrote: > On Fri, Nov 25, 2011 at 01:20:01PM -0600, Mark Felder wrote: > > 13:14:32 nas:~ > uname -a > > FreeBSD nas.feld.me 9.0-PRERELEASE FreeBSD 9.0-PRERELEASE #3 > > r227971M: > > Fri Nov 25 10:07:48 CST 2011 > > root@nas.feld.me:/usr/obj/tank/svn/sys/GENERIC amd64 > > > > This seemed to start happening sometime after RC1. I tried 8-STABLE > > and > > it's happening there too right now. I think whatever caused this was > > MFC'd. I've also reproduced this on completely different hardware > > running a single disk ZFS pool. > > > > > > I'm getting this output in dmesg after these hangs I keep seeing. > > Mark, those backtrace are not related to ZFS, but to PF. Not sure if > they are at all related to your hangs. Most cases where ZFS I/O seems > to > hang are hardware problems, where I/O requests are not completed. > He recently posted that his hangs went away when he stopped using NFS. NFS does use uma_zalloc() and there are several places in pfioctl() where uma_zalloc(...M_WAITOK...) is called (hidden under pool_get()) when a mutex (the PF_LOCK() one) is held. I've emailed bz@ related to this. I'm also not sure if they could be related to his hangs, but it seems that if uma_zalloc() decides to sleep with the mutex held, something may break and a broken uma_zalloc() would impact NFS. rick > 'procstat -kk -a' output might be useful once the hang happens. > > > uma_zalloc_arg: zone "pfrktable" with the following non-sleepable > > locks > > held: > > exclusive sleep mutex pf task mtx (pf task mtx) r = 0 > > (0xffffffff8199af20) locked @ > > /tank/svn/sys/modules/pf/../../contrib/pf/net/pf_ioctl.c:1589 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > > kdb_backtrace() at kdb_backtrace+0x37 > > _witness_debugger() at _witness_debugger+0x2e > > witness_warn() at witness_warn+0x2c4 > > uma_zalloc_arg() at uma_zalloc_arg+0x335 > > pfr_create_ktable() at pfr_create_ktable+0xd8 > > pfr_ina_define() at pfr_ina_define+0x12b > > pfioctl() at pfioctl+0x1c5a > > devfs_ioctl_f() at devfs_ioctl_f+0x7a > > kern_ioctl() at kern_ioctl+0xcd > > sys_ioctl() at sys_ioctl+0xfd > > amd64_syscall() at amd64_syscall+0x3ac > > Xfast_syscall() at Xfast_syscall+0xf7 > > --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800da711c, rsp = > > 0x7fffffff9d28, rbp = 0x7fffffffa1f0 --- > > -- > Pawel Jakub Dawidek http://www.wheelsystems.com > FreeBSD committer http://www.FreeBSD.org > Am I Evil? Yes, I Am! http://yomoli.com