From owner-freebsd-current@FreeBSD.ORG Fri Nov 25 19:20:10 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EF5711065670; Fri, 25 Nov 2011 19:20:10 +0000 (UTC) (envelope-from feld@feld.me) Received: from mwi1.coffeenet.org (unknown [IPv6:2607:f4e0:100:300::2]) by mx1.freebsd.org (Postfix) with ESMTP id 9DFD78FC15; Fri, 25 Nov 2011 19:20:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=feld.me; s=blargle; h=Message-Id:Subject:To:From:Date:Content-Transfer-Encoding:Content-Type:Mime-Version; bh=WfSOLegWPyQqhcph8/kt81XRSe83humkyFQZGtDdkO4=; b=dgMst82goRDtRClIqYnfi+qEpHlWoxIQNQZg4quligq9oFi6Zq/R1RdDUtR94CK9e7bcb1Kpo7pAfVt7TdJEpv5cxBhApUdI0sltfXzzKG9EUaUQHbz/C0XwXYUb6t45; Received: from localhost ([127.0.0.1] helo=mwi1.coffeenet.org) by mwi1.coffeenet.org with esmtp (Exim 4.77 (FreeBSD)) (envelope-from ) id 1RU1Jg-00052X-Fl; Fri, 25 Nov 2011 13:20:09 -0600 Received: from feld@feld.me by mwi1.coffeenet.org (Archiveopteryx 3.1.4) with esmtpsa id 1322248801-1863-1862/5/1; Fri, 25 Nov 2011 19:20:01 +0000 Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Date: Fri, 25 Nov 2011 13:20:01 -0600 From: Mark Felder To: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Message-Id: <95d00c1b714837aa32e7da72bc4afd03@feld.me> X-Sender: feld@feld.me User-Agent: Roundcube Webmail/0.6 X-SA-Score: -1.0 Cc: Subject: zfs i/o hangs on 9-PRERELEASE X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Nov 2011 19:20:11 -0000 13:14:32 nas:~ > uname -a FreeBSD nas.feld.me 9.0-PRERELEASE FreeBSD 9.0-PRERELEASE #3 r227971M:=20 Fri Nov 25 10:07:48 CST 2011 =20 root@nas.feld.me:/usr/obj/tank/svn/sys/GENERIC amd64 This seemed to start happening sometime after RC1. I tried 8-STABLE and=20 it's happening there too right now. I think whatever caused this was=20 MFC'd. I've also reproduced this on completely different hardware=20 running a single disk ZFS pool. I'm getting this output in dmesg after these hangs I keep seeing. uma_zalloc_arg: zone "pfrktable" with the following non-sleepable locks=20 held: exclusive sleep mutex pf task mtx (pf task mtx) r =3D 0=20 (0xffffffff8199af20) locked @=20 /tank/svn/sys/modules/pf/../../contrib/pf/net/pf_ioctl.c:1589 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2e witness_warn() at witness_warn+0x2c4 uma_zalloc_arg() at uma_zalloc_arg+0x335 pfr_create_ktable() at pfr_create_ktable+0xd8 pfr_ina_define() at pfr_ina_define+0x12b pfioctl() at pfioctl+0x1c5a devfs_ioctl_f() at devfs_ioctl_f+0x7a kern_ioctl() at kern_ioctl+0xcd sys_ioctl() at sys_ioctl+0xfd amd64_syscall() at amd64_syscall+0x3ac Xfast_syscall() at Xfast_syscall+0xf7 =2D-- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x800da711c, rsp = =3D=20 0x7fffffff9d28, rbp =3D 0x7fffffffa1f0 --- uma_zalloc_arg: zone "pfrktable" with the following non-sleepable locks=20 held: exclusive sleep mutex pf task mtx (pf task mtx) r =3D 0=20 (0xffffffff8199af20) locked @=20 /tank/svn/sys/modules/pf/../../contrib/pf/net/pf_ioctl.c:1589 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2e witness_warn() at witness_warn+0x2c4 uma_zalloc_arg() at uma_zalloc_arg+0x335 pfr_create_ktable() at pfr_create_ktable+0xd8 pfr_ina_define() at pfr_ina_define+0x179 pfioctl() at pfioctl+0x1c5a devfs_ioctl_f() at devfs_ioctl_f+0x7a kern_ioctl() at kern_ioctl+0xcd sys_ioctl() at sys_ioctl+0xfd amd64_syscall() at amd64_syscall+0x3ac Xfast_syscall() at Xfast_syscall+0xf7 =2D-- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x800da711c, rsp = =3D=20 0x7fffffff9d28, rbp =3D 0x7fffffffa1f0 --- uma_zalloc_arg: zone "pfrkentry" with the following non-sleepable locks=20 held: exclusive sleep mutex pf task mtx (pf task mtx) r =3D 0=20 (0xffffffff8199af20) locked @=20 /tank/svn/sys/modules/pf/../../contrib/pf/net/pf_table.c:75 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2e witness_warn() at witness_warn+0x2c4 uma_zalloc_arg() at uma_zalloc_arg+0x335 pfr_create_kentry() at pfr_create_kentry+0x73 pfr_ina_define() at pfr_ina_define+0x2ef pfioctl() at pfioctl+0x1c5a devfs_ioctl_f() at devfs_ioctl_f+0x7a kern_ioctl() at kern_ioctl+0xcd sys_ioctl() at sys_ioctl+0xfd amd64_syscall() at amd64_syscall+0x3ac Xfast_syscall() at Xfast_syscall+0xf7 =2D-- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x800da711c, rsp = =3D=20 0x7fffffff9d28, rbp =3D 0x7fffffffa1f0 --- uma_zalloc_arg: zone "pfrkentry" with the following non-sleepable locks=20 held: exclusive sleep mutex pf task mtx (pf task mtx) r =3D 0=20 (0xffffffff8199af20) locked @=20 /tank/svn/sys/modules/pf/../../contrib/pf/net/pf_table.c:75 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2e witness_warn() at witness_warn+0x2c4 uma_zalloc_arg() at uma_zalloc_arg+0x335 pfr_create_kentry() at pfr_create_kentry+0x73 pfr_ina_define() at pfr_ina_define+0x2ef pfioctl() at pfioctl+0x1c5a devfs_ioctl_f() at devfs_ioctl_f+0x7a kern_ioctl() at kern_ioctl+0xcd sys_ioctl() at sys_ioctl+0xfd amd64_syscall() at amd64_syscall+0x3ac Xfast_syscall() at Xfast_syscall+0xf7 =2D-- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x800da711c, rsp = =3D=20 0x7fffffff9d28, rbp =3D 0x7fffffffa1f0 --- uma_zalloc_arg: zone "pfrkentry" with the following non-sleepable locks=20 held: exclusive sleep mutex pf task mtx (pf task mtx) r =3D 0=20 (0xffffffff8199af20) locked @=20 /tank/svn/sys/modules/pf/../../contrib/pf/net/pf_table.c:75 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2e witness_warn() at witness_warn+0x2c4 uma_zalloc_arg() at uma_zalloc_arg+0x335 pfr_create_kentry() at pfr_create_kentry+0x73 pfr_ina_define() at pfr_ina_define+0x2ef pfioctl() at pfioctl+0x1c5a devfs_ioctl_f() at devfs_ioctl_f+0x7a kern_ioctl() at kern_ioctl+0xcd sys_ioctl() at sys_ioctl+0xfd amd64_syscall() at amd64_syscall+0x3ac Xfast_syscall() at Xfast_syscall+0xf7 =2D-- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x800da711c, rsp = =3D=20 0x7fffffff9d28, rbp =3D 0x7fffffffa1f0 --- uma_zalloc_arg: zone "pfrkentry" with the following non-sleepable locks=20 held: exclusive sleep mutex pf task mtx (pf task mtx) r =3D 0=20 (0xffffffff8199af20) locked @=20 /tank/svn/sys/modules/pf/../../contrib/pf/net/pf_table.c:75 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2e witness_warn() at witness_warn+0x2c4 uma_zalloc_arg() at uma_zalloc_arg+0x335 pfr_create_kentry() at pfr_create_kentry+0x73 pfr_ina_define() at pfr_ina_define+0x2ef pfioctl() at pfioctl+0x1c5a devfs_ioctl_f() at devfs_ioctl_f+0x7a kern_ioctl() at kern_ioctl+0xcd sys_ioctl() at sys_ioctl+0xfd amd64_syscall() at amd64_syscall+0x3ac Xfast_syscall() at Xfast_syscall+0xf7 =2D-- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x800da711c, rsp = =3D=20 0x7fffffff9d28, rbp =3D 0x7fffffffa1f0 --- uma_zalloc_arg: zone "pfrkentry" with the following non-sleepable locks=20 held: exclusive sleep mutex pf task mtx (pf task mtx) r =3D 0=20 (0xffffffff8199af20) locked @=20 /tank/svn/sys/modules/pf/../../contrib/pf/net/pf_table.c:75 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2e witness_warn() at witness_warn+0x2c4 uma_zalloc_arg() at uma_zalloc_arg+0x335 pfr_create_kentry() at pfr_create_kentry+0x73 pfr_ina_define() at pfr_ina_define+0x2ef pfioctl() at pfioctl+0x1c5a devfs_ioctl_f() at devfs_ioctl_f+0x7a kern_ioctl() at kern_ioctl+0xcd sys_ioctl() at sys_ioctl+0xfd amd64_syscall() at amd64_syscall+0x3ac Xfast_syscall() at Xfast_syscall+0xf7 =2D-- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x800da711c, rsp = =3D=20 0x7fffffff9d28, rbp =3D 0x7fffffffa1f0 --- uma_zalloc_arg: zone "pfrktable" with the following non-sleepable locks=20 held: exclusive sleep mutex pf task mtx (pf task mtx) r =3D 0=20 (0xffffffff8199af20) locked @=20 /tank/svn/sys/modules/pf/../../contrib/pf/net/pf_ioctl.c:1589 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2e witness_warn() at witness_warn+0x2c4 uma_zalloc_arg() at uma_zalloc_arg+0x335 pfr_create_ktable() at pfr_create_ktable+0xd8 pfr_ina_define() at pfr_ina_define+0x12b pfioctl() at pfioctl+0x1c5a devfs_ioctl_f() at devfs_ioctl_f+0x7a kern_ioctl() at kern_ioctl+0xcd sys_ioctl() at sys_ioctl+0xfd amd64_syscall() at amd64_syscall+0x3ac Xfast_syscall() at Xfast_syscall+0xf7 =2D-- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x800da711c, rsp = =3D=20 0x7fffffff9d28, rbp =3D 0x7fffffffa1f0 --- uma_zalloc_arg: zone "pfrktable" with the following non-sleepable locks=20 held: exclusive sleep mutex pf task mtx (pf task mtx) r =3D 0=20 (0xffffffff8199af20) locked @=20 /tank/svn/sys/modules/pf/../../contrib/pf/net/pf_ioctl.c:1589 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2e witness_warn() at witness_warn+0x2c4 uma_zalloc_arg() at uma_zalloc_arg+0x335 pfr_create_ktable() at pfr_create_ktable+0xd8 pfr_ina_define() at pfr_ina_define+0x179 pfioctl() at pfioctl+0x1c5a devfs_ioctl_f() at devfs_ioctl_f+0x7a kern_ioctl() at kern_ioctl+0xcd sys_ioctl() at sys_ioctl+0xfd amd64_syscall() at amd64_syscall+0x3ac Xfast_syscall() at Xfast_syscall+0xf7 =2D-- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x800da711c, rsp = =3D=20 0x7fffffff9d28, rbp =3D 0x7fffffffa1f0 ---