From owner-freebsd-fs@FreeBSD.ORG Tue Jan 29 03:21:20 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 28860CFF; Tue, 29 Jan 2013 03:21:20 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) by mx1.freebsd.org (Postfix) with ESMTP id 4E4701B7; Tue, 29 Jan 2013 03:21:19 +0000 (UTC) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.5/8.14.5) with ESMTP id r0T3LHOh080812; Mon, 28 Jan 2013 22:21:17 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.5/8.14.4/Submit) id r0T3LHvB080809; Mon, 28 Jan 2013 22:21:17 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <20743.16429.97668.569869@hergotha.csail.mit.edu> Date: Mon, 28 Jan 2013 22:21:17 -0500 From: Garrett Wollman To: freebsd-stable@freebsd.org, freebsd-fs@freebsd.org Subject: ZFS deadlock on rrl->rr_ -- look familiar to anyone? X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (hergotha.csail.mit.edu [127.0.0.1]); Mon, 28 Jan 2013 22:21:17 -0500 (EST) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hergotha.csail.mit.edu X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jan 2013 03:21:20 -0000 I just had a big fileserver deadlock in an odd way. I was investigating a user's problem, and decided for various reasons to restart mountd. It had been complaining like this: Jan 28 21:06:43 nfs-prod-1 mountd[1108]: can't delete exports for /usr/local/.zfs/snapshot/monthly-2013-01: Invalid argument for a while, which is odd because /usr/local was never exported. When I restarted mountd, it hung waiting on rrl->rr_, but the system may already have been deadlocked at that point. procstat reported: 87678 104365 mountd - mi_switch sleepq_wait _cv_wait rrw_enter zfs_root lookup namei vfs_donmount sys_nmount amd64_syscall Xfast_syscall I was able to run shutdown, and the rc scripts eventually hung in sync(1) and timed out. The kernel then hung trying to do the same thing, but I was able to break into the debugger. The debugger interrupted an idle thread, which was not particularly helpful, but I was able to quickly gather the following information before I had to reset the machine to restore normal service. Locked vnodes 0xfffffe00536383c0: 0xfffffe00536383c0: tag syncer, type VNON tag syncer, type VNON usecount 1, writecount 0, refcount 2 mountedhere 0 usecount 1, writecount 0, refcount 2 mountedhere 0 flags (VI(0x200)) flags (VI(0x200)) lock type syncer: EXCL by thread 0xfffffe00348cc470 (pid 22) lock type syncer: EXCL by thread 0xfffffe00348cc470 (pid 22) db> ps pid ppid pgrp uid state wmesg wchan cmd 87996 1 87994 65534 D rrl->rr_ 0xfffffe0048ff8108 df 87976 1 87726 0 D+ rrl->rr_ 0xfffffe0048ff8108 sync 87707 1 87705 65534 D rrl->rr_ 0xfffffe0048ff8108 df 87700 1 87698 65534 D rrl->rr_ 0xfffffe0048ff8108 df 87678 1 87657 0 D+ rrl->rr_ 0xfffffe0048ff8108 mountd 87531 1 87529 65534 D rrl->rr_ 0xfffffe0048ff8108 df 87387 1 87385 65534 D rrl->rr_ 0xfffffe0048ff8108 df 87380 1 87378 65534 D rrl->rr_ 0xfffffe0048ff8108 df 87103 1 87101 65534 D rrl->rr_ 0xfffffe0048ff8108 df 87096 1 87094 65534 D rrl->rr_ 0xfffffe0048ff8108 df 85193 1 85192 0 D zio->io_ 0xfffffe10d3e75320 zfs 24 0 0 0 DL sdflush 0xffffffff80e50878 [softdepflush] 23 0 0 0 DL vlruwt 0xfffffe0048c0a940 [vnlru] 22 0 0 0 DL rrl->rr_ 0xfffffe0048ff8108 [syncer] 21 0 0 0 DL psleep 0xffffffff80e3c048 [bufdaemon] 20 0 0 0 DL pgzero 0xffffffff80e5a81c [pagezero] 19 0 0 0 DL psleep 0xffffffff80e599e8 [vmdaemon] 18 0 0 0 DL psleep 0xffffffff80e599ac [pagedaemon] 17 0 0 0 DL gkt:wait 0xffffffff80de6c0c [g_mp_kt] 16 0 0 0 DL ipmireq 0xfffffe00347400b8 [ipmi0: kcs] 9 0 0 0 DL ccb_scan 0xffffffff80dc1360 [xpt_thrd] 8 0 0 0 DL waiting_ 0xffffffff80e41e80 [sctp_iterator] 7 0 0 0 DL (threaded) [zfskern] 101355 D tx->tx_s 0xfffffe0050342e10 [txg_thread_enter] 101354 D tx->tx_q 0xfffffe0050342e30 [txg_thread_enter] 100989 D tx->tx_s 0xfffffe004fd27a10 [txg_thread_enter] 100988 D tx->tx_q 0xfffffe004fd27a30 [txg_thread_enter] 100593 D tx->tx_s 0xfffffe004a8c0a10 [txg_thread_enter] 100592 D tx->tx_q 0xfffffe004a8c0a30 [txg_thread_enter] 100216 D l2arc_fe 0xffffffff81228bc0 [l2arc_feed_thread] 100215 D arc_recl 0xffffffff81218d20 [arc_reclaim_thread] 15 0 0 0 DL (threaded) [usb] [32 uninteresting and identical threads deleted] 6 0 0 0 DL mps_scan 0xfffffe00276816a8 [mps_scan2] 5 0 0 0 DL mps_scan 0xfffffe0027612ca8 [mps_scan1] 4 0 0 0 DL mps_scan 0xfffffe00274ef4a8 [mps_scan0] 14 0 0 0 DL - 0xffffffff80ded764 [yarrow] 3 0 0 0 DL crypto_r 0xffffffff80e4e0a0 [crypto returns] 2 0 0 0 DL crypto_w 0xffffffff80e4e060 [crypto] 13 0 0 0 DL (threaded) [geom] 100055 D - 0xffffffff80de6b90 [g_down] 100054 D - 0xffffffff80de6b88 [g_up] 100053 D - 0xffffffff80de6b78 [g_event] 12 0 0 0 WL (threaded) [intr] 100189 I [irq1: atkbd0] 100188 I [swi0: uart uart] 100187 I [irq19: atapci1] 100186 I [irq18: atapci0+] 100169 I [irq294: igb1:link] 100167 I [irq293: igb1:que 7] 100165 I [irq292: igb1:que 6] 100163 I [irq291: igb1:que 5] 100161 I [irq290: igb1:que 4] 100159 I [irq289: igb1:que 3] 100157 I [irq288: igb1:que 2] 100155 I [irq287: igb1:que 1] 100153 I [irq286: igb1:que 0] 100152 I [irq285: igb0:link] 100150 I [irq284: igb0:que 7] 100148 I [irq283: igb0:que 6] 100146 I [irq282: igb0:que 5] 100144 I [irq281: igb0:que 4] 100142 I [irq280: igb0:que 3] 100140 I [irq279: igb0:que 2] 100138 I [irq278: igb0:que 1] 100136 I [irq277: igb0:que 0] 100131 I [irq20: hpet0 ehci0] 100126 I [irq21: uhci2 uhci5] 100121 I [irq22: uhci1 uhci4] 100116 I [irq23: uhci0 uhci3+] 100115 I [irq276: mps2] 100112 I [irq275: mps1] 100108 I [irq274: ix1:link] 100106 I [irq273: ix1:que 7] 100104 I [irq272: ix1:que 6] 100102 I [irq271: ix1:que 5] 100100 I [irq270: ix1:que 4] 100098 I [irq269: ix1:que 3] 100096 I [irq268: ix1:que 2] 100094 I [irq267: ix1:que 1] 100092 I [irq266: ix1:que 0] 100090 I [irq265: ix0:link] 100088 I [irq264: ix0:que 7] 100086 I [irq263: ix0:que 6] 100084 I [irq262: ix0:que 5] 100082 I [irq261: ix0:que 4] 100080 I [irq260: ix0:que 3] 100078 I [irq259: ix0:que 2] 100076 I [irq258: ix0:que 1] 100074 I [irq257: ix0:que 0] 100073 I [irq256: mps0] 100065 I [swi2: cambio] 100064 I [swi6: task queue] 100063 I [swi6: Giant taskq] 100060 I [swi5: +] [24 identical [swi4: clock] threads deleted] 100028 I [swi1: netisr 0] 100027 I [swi3: vm] 11 0 0 0 RL (threaded) [idle] [24 identical idle threads deleted] 1 0 1 0 DLs rrl->rr_ 0xfffffe0048ff8108 [init] 10 0 0 0 DL audit_wo 0xffffffff80e4f7f0 [audit] 0 0 0 0 DLs (threaded) [kernel] 420220 D - 0xfffffe07bf578380 [zil_clean] [66 similar zil_clean threads deleted] 101353 D - 0xfffffe004a481c80 [zfs_vn_rele_taskq] 101352 D - 0xfffffe005324fa80 [zio_ioctl_intr] 101351 D - 0xfffffe005324fb00 [zio_ioctl_issue] 101350 D - 0xfffffe005324fb80 [zio_claim_intr] 101349 D - 0xfffffe005324fc00 [zio_claim_issue] 101348 D - 0xfffffe005324fc80 [zio_free_intr] 101347 D - 0xfffffe005324fd00 [zio_free_issue_99] [99 similar zio_free_issue_* threads deleted] 101247 D - 0xfffffe005324fd80 [zio_write_intr_high] 101246 D - 0xfffffe005324fd80 [zio_write_intr_high] 101245 D - 0xfffffe005324fd80 [zio_write_intr_high] 101244 D - 0xfffffe005324fd80 [zio_write_intr_high] 101243 D - 0xfffffe005324fd80 [zio_write_intr_high] 101242 D - 0xfffffe005324fe00 [zio_write_intr_7] 101241 D - 0xfffffe005324fe00 [zio_write_intr_6] 101240 D - 0xfffffe005324fe00 [zio_write_intr_5] 101239 D - 0xfffffe005324fe00 [zio_write_intr_4] 101238 D - 0xfffffe005324fe00 [zio_write_intr_3] 101237 D - 0xfffffe005324fe00 [zio_write_intr_2] 101236 D - 0xfffffe005324fe00 [zio_write_intr_1] 101235 D - 0xfffffe005324fe00 [zio_write_intr_0] 101234 D - 0xfffffe0053250000 [zio_write_issue_hig] 101233 D - 0xfffffe0053250000 [zio_write_issue_hig] 101232 D - 0xfffffe0053250000 [zio_write_issue_hig] 101231 D - 0xfffffe0053250000 [zio_write_issue_hig] 101230 D - 0xfffffe0053250000 [zio_write_issue_hig] 101229 D - 0xfffffe0053250080 [zio_write_issue_23] 101228 D - 0xfffffe0053250080 [zio_write_issue_22] 101227 D - 0xfffffe0053250080 [zio_write_issue_21] 101226 D - 0xfffffe0053250080 [zio_write_issue_20] 101225 D - 0xfffffe0053250080 [zio_write_issue_19] 101224 D - 0xfffffe0053250080 [zio_write_issue_18] 101223 D - 0xfffffe0053250080 [zio_write_issue_17] 101222 D - 0xfffffe0053250080 [zio_write_issue_16] 101221 D - 0xfffffe0053250080 [zio_write_issue_15] 101220 D - 0xfffffe0053250080 [zio_write_issue_14] 101219 D - 0xfffffe0053250080 [zio_write_issue_13] 101218 D - 0xfffffe0053250080 [zio_write_issue_12] 101217 D - 0xfffffe0053250080 [zio_write_issue_11] 101216 D - 0xfffffe0053250080 [zio_write_issue_10] 101215 D - 0xfffffe0053250080 [zio_write_issue_9] 101214 D - 0xfffffe0053250080 [zio_write_issue_8] 101213 D - 0xfffffe0053250080 [zio_write_issue_7] 101212 D - 0xfffffe0053250080 [zio_write_issue_6] 101211 D - 0xfffffe0053250080 [zio_write_issue_5] 101210 D - 0xfffffe0053250080 [zio_write_issue_4] 101209 D - 0xfffffe0053250080 [zio_write_issue_3] 101208 D - 0xfffffe0053250080 [zio_write_issue_2] 101207 D - 0xfffffe0053250080 [zio_write_issue_1] 101206 D - 0xfffffe0053250080 [zio_write_issue_0] 101205 D - 0xfffffe0053250100 [zio_read_intr_23] 101204 D - 0xfffffe0053250100 [zio_read_intr_22] 101203 D - 0xfffffe0053250100 [zio_read_intr_21] 101202 D - 0xfffffe0053250100 [zio_read_intr_20] 101201 D - 0xfffffe0053250100 [zio_read_intr_19] 101200 D - 0xfffffe0053250100 [zio_read_intr_18] 101199 D - 0xfffffe0053250100 [zio_read_intr_17] 101198 D - 0xfffffe0053250100 [zio_read_intr_16] 101197 D - 0xfffffe0053250100 [zio_read_intr_15] 101196 D - 0xfffffe0053250100 [zio_read_intr_14] 101195 D - 0xfffffe0053250100 [zio_read_intr_13] 101194 D - 0xfffffe0053250100 [zio_read_intr_12] 101193 D - 0xfffffe0053250100 [zio_read_intr_11] 101192 D - 0xfffffe0053250100 [zio_read_intr_10] 101191 D - 0xfffffe0053250100 [zio_read_intr_9] 101190 D - 0xfffffe0053250100 [zio_read_intr_8] 101189 D - 0xfffffe0053250100 [zio_read_intr_7] 101188 D - 0xfffffe0053250100 [zio_read_intr_6] 101187 D - 0xfffffe0053250100 [zio_read_intr_5] 101186 D - 0xfffffe0053250100 [zio_read_intr_4] 101185 D - 0xfffffe0053250100 [zio_read_intr_3] 101184 D - 0xfffffe0053250100 [zio_read_intr_2] 101183 D - 0xfffffe0053250100 [zio_read_intr_1] 101182 D - 0xfffffe0053250100 [zio_read_intr_0] 101181 D - 0xfffffe0053250180 [zio_read_issue_7] 101180 D - 0xfffffe0053250180 [zio_read_issue_6] 101179 D - 0xfffffe0053250180 [zio_read_issue_5] 101178 D - 0xfffffe0053250180 [zio_read_issue_4] 101177 D - 0xfffffe0053250180 [zio_read_issue_3] 101176 D - 0xfffffe0053250180 [zio_read_issue_2] 101175 D - 0xfffffe0053250180 [zio_read_issue_1] 101174 D - 0xfffffe0053250180 [zio_read_issue_0] 101173 D - 0xfffffe0053250200 [zio_null_intr] 101172 D - 0xfffffe0053250280 [zio_null_issue] 100987 D - 0xfffffe0048cc9500 [zfs_vn_rele_taskq] 100986 D - 0xfffffe0048c72280 [zio_ioctl_intr] 100985 D - 0xfffffe0048c71a00 [zio_ioctl_issue] 100984 D - 0xfffffe0048dd0d00 [zio_claim_intr] 100983 D - 0xfffffe0048dd0680 [zio_claim_issue] 100982 D - 0xfffffe004a949080 [zio_free_intr] 100981 D - 0xfffffe0048b77d80 [zio_free_issue_99] [99 more zio_free_issue_* threads deleted] 100881 D - 0xfffffe004a94a480 [zio_write_intr_high] 100880 D - 0xfffffe004a94a480 [zio_write_intr_high] 100879 D - 0xfffffe004a94a480 [zio_write_intr_high] 100878 D - 0xfffffe004a94a480 [zio_write_intr_high] 100877 D - 0xfffffe004a94a480 [zio_write_intr_high] 100876 D - 0xfffffe0048dd1180 [zio_write_intr_7] 100875 D - 0xfffffe0048dd1180 [zio_write_intr_6] 100874 D - 0xfffffe0048dd1180 [zio_write_intr_5] 100873 D - 0xfffffe0048dd1180 [zio_write_intr_4] 100872 D - 0xfffffe0048dd1180 [zio_write_intr_3] 100871 D - 0xfffffe0048dd1180 [zio_write_intr_2] 100870 D - 0xfffffe0048dd1180 [zio_write_intr_1] 100869 D - 0xfffffe0048dd1180 [zio_write_intr_0] 100868 D - 0xfffffe0048dd1100 [zio_write_issue_hig] 100867 D - 0xfffffe0048dd1100 [zio_write_issue_hig] 100866 D - 0xfffffe0048dd1100 [zio_write_issue_hig] 100865 D - 0xfffffe0048dd1100 [zio_write_issue_hig] 100864 D - 0xfffffe0048dd1100 [zio_write_issue_hig] 100863 D - 0xfffffe0048dd1080 [zio_write_issue_23] 100862 D - 0xfffffe0048dd1080 [zio_write_issue_22] 100861 D - 0xfffffe0048dd1080 [zio_write_issue_21] 100860 D - 0xfffffe0048dd1080 [zio_write_issue_20] 100859 D - 0xfffffe0048dd1080 [zio_write_issue_19] 100858 D - 0xfffffe0048dd1080 [zio_write_issue_18] 100857 D - 0xfffffe0048dd1080 [zio_write_issue_17] 100856 D - 0xfffffe0048dd1080 [zio_write_issue_16] 100855 D - 0xfffffe0048dd1080 [zio_write_issue_15] 100854 D - 0xfffffe0048dd1080 [zio_write_issue_14] 100853 D - 0xfffffe0048dd1080 [zio_write_issue_13] 100852 D - 0xfffffe0048dd1080 [zio_write_issue_12] 100851 D - 0xfffffe0048dd1080 [zio_write_issue_11] 100850 D - 0xfffffe0048dd1080 [zio_write_issue_10] 100849 D - 0xfffffe0048dd1080 [zio_write_issue_9] 100848 D - 0xfffffe0048dd1080 [zio_write_issue_8] 100847 D - 0xfffffe0048dd1080 [zio_write_issue_7] 100846 D - 0xfffffe0048dd1080 [zio_write_issue_6] 100845 D - 0xfffffe0048dd1080 [zio_write_issue_5] 100844 D - 0xfffffe0048dd1080 [zio_write_issue_4] 100843 D - 0xfffffe0048dd1080 [zio_write_issue_3] 100842 D - 0xfffffe0048dd1080 [zio_write_issue_2] 100841 D - 0xfffffe0048dd1080 [zio_write_issue_1] 100840 D - 0xfffffe0048dd1080 [zio_write_issue_0] 100839 D - 0xfffffe0048dd1000 [zio_read_intr_23] 100838 D - 0xfffffe0048dd1000 [zio_read_intr_22] 100837 D - 0xfffffe0048dd1000 [zio_read_intr_21] 100836 D - 0xfffffe0048dd1000 [zio_read_intr_20] 100835 D - 0xfffffe0048dd1000 [zio_read_intr_19] 100834 D - 0xfffffe0048dd1000 [zio_read_intr_18] 100833 D - 0xfffffe0048dd1000 [zio_read_intr_17] 100832 D - 0xfffffe0048dd1000 [zio_read_intr_16] 100831 D - 0xfffffe0048dd1000 [zio_read_intr_15] 100830 D - 0xfffffe0048dd1000 [zio_read_intr_14] 100829 D - 0xfffffe0048dd1000 [zio_read_intr_13] 100828 D - 0xfffffe0048dd1000 [zio_read_intr_12] 100827 D - 0xfffffe0048dd1000 [zio_read_intr_11] 100826 D - 0xfffffe0048dd1000 [zio_read_intr_10] 100825 D - 0xfffffe0048dd1000 [zio_read_intr_9] 100824 D - 0xfffffe0048dd1000 [zio_read_intr_8] 100823 D - 0xfffffe0048dd1000 [zio_read_intr_7] 100822 D - 0xfffffe0048dd1000 [zio_read_intr_6] 100821 D - 0xfffffe0048dd1000 [zio_read_intr_5] 100820 D - 0xfffffe0048dd1000 [zio_read_intr_4] 100819 D - 0xfffffe0048dd1000 [zio_read_intr_3] 100818 D - 0xfffffe0048dd1000 [zio_read_intr_2] 100817 D - 0xfffffe0048dd1000 [zio_read_intr_1] 100816 D - 0xfffffe0048dd1000 [zio_read_intr_0] 100815 D - 0xfffffe0048dd0e00 [zio_read_issue_7] 100814 D - 0xfffffe0048dd0e00 [zio_read_issue_6] 100813 D - 0xfffffe0048dd0e00 [zio_read_issue_5] 100812 D - 0xfffffe0048dd0e00 [zio_read_issue_4] 100811 D - 0xfffffe0048dd0e00 [zio_read_issue_3] 100810 D - 0xfffffe0048dd0e00 [zio_read_issue_2] 100809 D - 0xfffffe0048dd0e00 [zio_read_issue_1] 100808 D - 0xfffffe0048dd0e00 [zio_read_issue_0] 100807 D - 0xfffffe0048dd0600 [zio_null_intr] 100806 D - 0xfffffe0048dd0180 [zio_null_issue] 100594 D - 0xfffffe004a3bcc80 [zil_clean] 100591 D - 0xfffffe0048c65100 [zfs_vn_rele_taskq] 100590 D - 0xfffffe0048d5c280 [zio_ioctl_intr] 100589 D - 0xfffffe0048d5c300 [zio_ioctl_issue] 100588 D - 0xfffffe0048d5c380 [zio_claim_intr] 100587 D - 0xfffffe0048d5c400 [zio_claim_issue] 100586 D - 0xfffffe0048d5c480 [zio_free_intr] 100585 D - 0xfffffe0048d5c500 [zio_free_issue_99] [99 more zio_free_issue_* threads deleted] 100485 D - 0xfffffe0048d5c580 [zio_write_intr_high] 100484 D - 0xfffffe0048d5c580 [zio_write_intr_high] 100483 D - 0xfffffe0048d5c580 [zio_write_intr_high] 100482 D - 0xfffffe0048d5c580 [zio_write_intr_high] 100481 D - 0xfffffe0048d5c580 [zio_write_intr_high] 100480 D - 0xfffffe0048d5c600 [zio_write_intr_7] 100479 D - 0xfffffe0048d5c600 [zio_write_intr_6] 100478 D - 0xfffffe0048d5c600 [zio_write_intr_5] 100477 D - 0xfffffe0048d5c600 [zio_write_intr_4] 100476 D - 0xfffffe0048d5c600 [zio_write_intr_3] 100475 D - 0xfffffe0048d5c600 [zio_write_intr_2] 100474 D - 0xfffffe0048d5c600 [zio_write_intr_1] 100473 D - 0xfffffe0048d5c600 [zio_write_intr_0] 100472 D - 0xfffffe0048d5c680 [zio_write_issue_hig] 100471 D - 0xfffffe0048d5c680 [zio_write_issue_hig] 100470 D - 0xfffffe0048d5c680 [zio_write_issue_hig] 100469 D - 0xfffffe0048d5c680 [zio_write_issue_hig] 100468 D - 0xfffffe0048d5c680 [zio_write_issue_hig] 100467 D - 0xfffffe0048d5c700 [zio_write_issue_23] 100466 D - 0xfffffe0048d5c700 [zio_write_issue_22] 100465 D - 0xfffffe0048d5c700 [zio_write_issue_21] 100464 D - 0xfffffe0048d5c700 [zio_write_issue_20] 100463 D - 0xfffffe0048d5c700 [zio_write_issue_19] 100462 D - 0xfffffe0048d5c700 [zio_write_issue_18] 100461 D - 0xfffffe0048d5c700 [zio_write_issue_17] 100460 D - 0xfffffe0048d5c700 [zio_write_issue_16] 100459 D - 0xfffffe0048d5c700 [zio_write_issue_15] 100458 D - 0xfffffe0048d5c700 [zio_write_issue_14] 100457 D - 0xfffffe0048d5c700 [zio_write_issue_13] 100456 D - 0xfffffe0048d5c700 [zio_write_issue_12] 100455 D - 0xfffffe0048d5c700 [zio_write_issue_11] 100454 D - 0xfffffe0048d5c700 [zio_write_issue_10] 100453 D - 0xfffffe0048d5c700 [zio_write_issue_9] 100452 D - 0xfffffe0048d5c700 [zio_write_issue_8] 100451 D - 0xfffffe0048d5c700 [zio_write_issue_7] 100450 D - 0xfffffe0048d5c700 [zio_write_issue_6] 100449 D - 0xfffffe0048d5c700 [zio_write_issue_5] 100448 D - 0xfffffe0048d5c700 [zio_write_issue_4] 100447 D - 0xfffffe0048d5c700 [zio_write_issue_3] 100446 D - 0xfffffe0048d5c700 [zio_write_issue_2] 100445 D - 0xfffffe0048d5c700 [zio_write_issue_1] 100444 D - 0xfffffe0048d5c700 [zio_write_issue_0] 100443 D - 0xfffffe0048d5c780 [zio_read_intr_23] 100442 D - 0xfffffe0048d5c780 [zio_read_intr_22] 100441 D - 0xfffffe0048d5c780 [zio_read_intr_21] 100440 D - 0xfffffe0048d5c780 [zio_read_intr_20] 100439 D - 0xfffffe0048d5c780 [zio_read_intr_19] 100438 D - 0xfffffe0048d5c780 [zio_read_intr_18] 100437 D - 0xfffffe0048d5c780 [zio_read_intr_17] 100436 D - 0xfffffe0048d5c780 [zio_read_intr_16] 100435 D - 0xfffffe0048d5c780 [zio_read_intr_15] 100434 D - 0xfffffe0048d5c780 [zio_read_intr_14] 100433 D - 0xfffffe0048d5c780 [zio_read_intr_13] 100432 D - 0xfffffe0048d5c780 [zio_read_intr_12] 100431 D - 0xfffffe0048d5c780 [zio_read_intr_11] 100430 D - 0xfffffe0048d5c780 [zio_read_intr_10] 100429 D - 0xfffffe0048d5c780 [zio_read_intr_9] 100428 D - 0xfffffe0048d5c780 [zio_read_intr_8] 100427 D - 0xfffffe0048d5c780 [zio_read_intr_7] 100426 D - 0xfffffe0048d5c780 [zio_read_intr_6] 100425 D - 0xfffffe0048d5c780 [zio_read_intr_5] 100424 D - 0xfffffe0048d5c780 [zio_read_intr_4] 100423 D - 0xfffffe0048d5c780 [zio_read_intr_3] 100422 D - 0xfffffe0048d5c780 [zio_read_intr_2] 100421 D - 0xfffffe0048d5c780 [zio_read_intr_1] 100420 D - 0xfffffe0048d5c780 [zio_read_intr_0] 100419 D - 0xfffffe0048d5c800 [zio_read_issue_7] 100418 D - 0xfffffe0048d5c800 [zio_read_issue_6] 100417 D - 0xfffffe0048d5c800 [zio_read_issue_5] 100416 D - 0xfffffe0048d5c800 [zio_read_issue_4] 100415 D - 0xfffffe0048d5c800 [zio_read_issue_3] 100414 D - 0xfffffe0048d5c800 [zio_read_issue_2] 100413 D - 0xfffffe0048d5c800 [zio_read_issue_1] 100412 D - 0xfffffe0048d5c800 [zio_read_issue_0] 100411 D - 0xfffffe0048d5c880 [zio_null_intr] 100410 D - 0xfffffe0048d5c900 [zio_null_issue] 100214 D - 0xfffffe00348bbc00 [system_taskq_23] [23 more system_taskq_* threads deleted] 100190 D - 0xfffffe00348bbc80 [mca taskq] 100168 D - 0xfffffe0034092b00 [igb1 que] 100166 D - 0xfffffe0034092c80 [igb1 que] 100164 D - 0xfffffe0034092e00 [igb1 que] 100162 D - 0xfffffe0034092180 [igb1 que] 100160 D - 0xfffffe0034092300 [igb1 que] 100158 D - 0xfffffe0034092480 [igb1 que] 100156 D - 0xfffffe003408b300 [igb1 que] 100154 D - 0xfffffe003408b480 [igb1 que] 100151 D - 0xfffffe003405b500 [igb0 que] 100149 D - 0xfffffe003405b680 [igb0 que] 100147 D - 0xfffffe0034054580 [igb0 que] 100145 D - 0xfffffe003404d400 [igb0 que] 100143 D - 0xfffffe003404d580 [igb0 que] 100141 D - 0xfffffe003404d700 [igb0 que] 100139 D - 0xfffffe003404d880 [igb0 que] 100137 D - 0xfffffe003404da00 [igb0 que] 100113 D - 0xfffffe0027807300 [mps2 taskq] 100110 D - 0xfffffe0027697a80 [mps1 taskq] 100109 D - 0xfffffe002768a700 [ix1 linkq] 100107 D - 0xfffffe002768a800 [ix1 que] 100105 D - 0xfffffe002768a980 [ix1 que] 100103 D - 0xfffffe002768ab00 [ix1 que] 100101 D - 0xfffffe002768ac80 [ix1 que] 100099 D - 0xfffffe0027680b80 [ix1 que] 100097 D - 0xfffffe0027680d00 [ix1 que] 100095 D - 0xfffffe0027623680 [ix1 que] 100093 D - 0xfffffe0027623380 [ix1 que] 100091 D - 0xfffffe0027600480 [ix0 linkq] 100089 D - 0xfffffe0027600580 [ix0 que] 100087 D - 0xfffffe0027600700 [ix0 que] 100085 D - 0xfffffe0027527300 [ix0 que] 100083 D - 0xfffffe0027527480 [ix0 que] 100081 D - 0xfffffe0027527600 [ix0 que] 100079 D - 0xfffffe0027527780 [ix0 que] 100077 D - 0xfffffe0027527900 [ix0 que] 100075 D - 0xfffffe0027527a80 [ix0 que] 100071 D - 0xfffffe00274ffe00 [mps0 taskq] 100070 D - 0xfffffe00273fdb00 [kqueue taskq] 100069 D - 0xfffffe00273fdb80 [ffs_trim taskq] 100068 D - 0xfffffe00273fdc00 [acpi_task_2] 100067 D - 0xfffffe00273fdc00 [acpi_task_1] 100066 D - 0xfffffe00273fdc00 [acpi_task_0] 100062 D - 0xfffffe002743b280 [aiod_bio taskq] 100061 D zfsvfs-> 0xfffffe0048ff8138 [thread taskq] 100056 D - 0xfffffe002732d600 [firmware taskq] 100000 D sched 0xffffffff80de6d80 [swapper] The stuck df(1) processes running as nobody were undoubtedly started by munin-node, and seem to be related to my user's symptom (munin graphs show no response for about half an hour after the user's problem starts). It may not be a *true* deadlock, because over the past few days, munin has been showing this problem at about the same time of day, but the system always comes back (without a reboot) in a little over half an hour. Does anyone recognize this? If it happens again, which threads' stack traces would be useful in diagnosing this? -GAWollman