Date: Mon, 28 Nov 2011 09:39:25 -0500 (EST) From: Rick Macklem <rmacklem@uoguelph.ca> To: Pawel Jakub Dawidek <pjd@FreeBSD.org> Cc: freebsd-fs@freebsd.org, Mark Felder <feld@feld.me>, freebsd-current@freebsd.org Subject: Re: zfs i/o hangs on 9-PRERELEASE Message-ID: <2111049507.479178.1322491165407.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <20111126104840.GA8794@garage.freebsd.pl>
next in thread | previous in thread | raw e-mail | index | archive | help
Pawel Jakub Dawidek wrote: > On Fri, Nov 25, 2011 at 01:20:01PM -0600, Mark Felder wrote: > > 13:14:32 nas:~ > uname -a > > FreeBSD nas.feld.me 9.0-PRERELEASE FreeBSD 9.0-PRERELEASE #3 > > r227971M: > > Fri Nov 25 10:07:48 CST 2011 > > root@nas.feld.me:/usr/obj/tank/svn/sys/GENERIC amd64 > > > > This seemed to start happening sometime after RC1. I tried 8-STABLE > > and > > it's happening there too right now. I think whatever caused this was > > MFC'd. I've also reproduced this on completely different hardware > > running a single disk ZFS pool. > > > > > > I'm getting this output in dmesg after these hangs I keep seeing. > > Mark, those backtrace are not related to ZFS, but to PF. Not sure if > they are at all related to your hangs. Most cases where ZFS I/O seems > to > hang are hardware problems, where I/O requests are not completed. > He recently posted that his hangs went away when he stopped using NFS. NFS does use uma_zalloc() and there are several places in pfioctl() where uma_zalloc(...M_WAITOK...) is called (hidden under pool_get()) when a mutex (the PF_LOCK() one) is held. I've emailed bz@ related to this. I'm also not sure if they could be related to his hangs, but it seems that if uma_zalloc() decides to sleep with the mutex held, something may break and a broken uma_zalloc() would impact NFS. rick > 'procstat -kk -a' output might be useful once the hang happens. > > > uma_zalloc_arg: zone "pfrktable" with the following non-sleepable > > locks > > held: > > exclusive sleep mutex pf task mtx (pf task mtx) r = 0 > > (0xffffffff8199af20) locked @ > > /tank/svn/sys/modules/pf/../../contrib/pf/net/pf_ioctl.c:1589 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > > kdb_backtrace() at kdb_backtrace+0x37 > > _witness_debugger() at _witness_debugger+0x2e > > witness_warn() at witness_warn+0x2c4 > > uma_zalloc_arg() at uma_zalloc_arg+0x335 > > pfr_create_ktable() at pfr_create_ktable+0xd8 > > pfr_ina_define() at pfr_ina_define+0x12b > > pfioctl() at pfioctl+0x1c5a > > devfs_ioctl_f() at devfs_ioctl_f+0x7a > > kern_ioctl() at kern_ioctl+0xcd > > sys_ioctl() at sys_ioctl+0xfd > > amd64_syscall() at amd64_syscall+0x3ac > > Xfast_syscall() at Xfast_syscall+0xf7 > > --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800da711c, rsp = > > 0x7fffffff9d28, rbp = 0x7fffffffa1f0 --- > > -- > Pawel Jakub Dawidek http://www.wheelsystems.com > FreeBSD committer http://www.FreeBSD.org > Am I Evil? Yes, I Am! http://yomoli.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2111049507.479178.1322491165407.JavaMail.root>