From owner-freebsd-bugs@freebsd.org Tue Oct 20 23:15:55 2015 Return-Path: Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E79FCA15340 for ; Tue, 20 Oct 2015 23:15:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CC40D36D for ; Tue, 20 Oct 2015 23:15:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9KNFtiF082687 for ; Tue, 20 Oct 2015 23:15:55 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 203906] ZFS lockup, spa_namespace_lock Date: Tue, 20 Oct 2015 23:15:55 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: uwyo.apalmer@gmail.com X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2015 23:15:56 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203906 Bug ID: 203906 Summary: ZFS lockup, spa_namespace_lock Product: Base System Version: 10.1-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: freebsd-bugs@FreeBSD.org Reporter: uwyo.apalmer@gmail.com zpool/zfs utilities hang when referencing a now, non-existant volume that has not cleaned itself up in the kernel. Makes zfs administration useless. Cannot recover failed volume till reboot. $ uname -a FreeBSD Vortex 10.1-RELEASE FreeBSD 10.1-RELEASE #0 r274401: Tue Nov 11 21:02:49 UTC 2014 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 $ bzcat messages.*.bz2 | grep -rn ATA ... //trimmed 1065:Sep 25 21:12:09 Vortex kernel: ada4: ATA-8 SATA 2.x device 1067:Sep 25 21:12:09 Vortex kernel: ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) 1073:Sep 25 21:12:30 Vortex kernel: (aprobe0:siisch0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00 1086:Sep 25 21:13:00 Vortex kernel: (aprobe0:siisch0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00 1091:Sep 25 21:13:30 Vortex kernel: (aprobe0:siisch0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00 1095:Sep 25 21:13:45 Vortex kernel: (aprobe0:siisch0:0:0:0): CAM status: ATA Status Error 1096:Sep 25 21:13:45 Vortex kernel: (aprobe0:siisch0:0:0:0): ATA status: 00 () 1100:Sep 25 21:14:01 Vortex kernel: (aprobe0:siisch0:0:0:0): CAM status: ATA Status Error 1101:Sep 25 21:14:01 Vortex kernel: (aprobe0:siisch0:0:0:0): ATA status: 00 () ... $ ps -ax|grep D+ 38986 8- D+ 0:00.00 zpool iostat 1 10 42143 0- D+ 0:00.01 zpool list 42210 1- D+ 0:00.01 zpool iostat 1 10 42242 2- D+ 0:00.01 zfs list 51117 4- D+ 0:00.01 ls 65547 5- D+ 0:00.01 zfs unmount -f zStorage1 45501 11 S+ 0:00.01 grep D+ 65624 9- D+ 0:00.01 zfs list 65496 6- D+ 0:00.01 ls zStorage1 $ procstat -kk -a PID TID COMM TDNAME KSTACK 1252 100078 csh - mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_sigsuspend+0xf4 sys_sigsuspend+0x31 amd64_syscall+0x351 Xfast_syscall+0xfb 38986 100168 zpool - mi_switch+0xe1 sleepq_wait+0x3a _sx_xlock_hard+0x48a _sx_xlock+0x5d spa_all_configs+0x6e zfs_ioc_pool_configs+0x19 zfsdev_ioctl+0x6f0 devfs_ioctl_f+0x114 kern_ioctl+0x255 sys_ioctl+0x13c amd64_syscall+0x351 Xfast_syscall+0xfb 41360 101772 csh - mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_sigsuspend+0xf4 sys_sigsuspend+0x31 amd64_syscall+0x351 Xfast_syscall+0xfb 41661 100159 gdb - mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_wait6+0x5f4 sys_wait4+0x72 amd64_syscall+0x351 Xfast_syscall+0xfb 41662 101773 zpool - mi_switch+0xe1 sleepq_wait+0x3a _sx_xlock_hard+0x48a _sx_xlock+0x5d spa_all_configs+0x6e zfs_ioc_pool_configs+0x19 zfsdev_ioctl+0x6f0 devfs_ioctl_f+0x114 kern_ioctl+0x255 sys_ioctl+0x13c amd64_syscall+0x351 Xfast_syscall+0xfb 42143 100174 zpool - mi_switch+0xe1 sleepq_wait+0x3a _sx_xlock_hard+0x48a _sx_xlock+0x5d spa_all_configs+0x6e zfs_ioc_pool_configs+0x19 zfsdev_ioctl+0x6f0 devfs_ioctl_f+0x114 kern_ioctl+0x255 sys_ioctl+0x13c amd64_syscall+0x351 Xfast_syscall+0xfb 42242 101757 zfs - mi_switch+0xe1 sleepq_wait+0x3a _sx_xlock_hard+0x48a _sx_xlock+0x5d spa_all_configs+0x6e zfs_ioc_pool_configs+0x19 zfsdev_ioctl+0x6f0 devfs_ioctl_f+0x114 kern_ioctl+0x255 sys_ioctl+0x13c amd64_syscall+0x351 Xfast_syscall+0xfb 45485 101761 csh - mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_sigsuspend+0xf4 sys_sigsuspend+0x31 amd64_syscall+0x351 Xfast_syscall+0xfb 45553 100079 procstat - 51117 101760 ls - mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x16d dbuf_read+0x45b dmu_buf_hold+0x3d zap_lockdir+0x46 zap_cursor_retrieve+0x131 zfs_freebsd_readdir+0x3e1 VOP_READDIR_APV+0xa7 kern_getdirentries+0x21c sys_getdirentries+0x28 amd64_syscall+0x351 Xfast_syscall+0xfb 65149 101762 csh - mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a tty_wait+0x1c ttydisc_read+0x2d4 ttydev_read+0x86 devfs_read_f+0xeb dofileread+0x95 kern_readv+0x68 sys_read+0x63 amd64_syscall+0x351 Xfast_syscall+0xfb 65496 101767 ls - mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x16d dbuf_read+0x45b dmu_buf_hold+0x3d zap_lockdir+0x46 zap_cursor_retrieve+0x131 zfs_freebsd_readdir+0x3e1 VOP_READDIR_APV+0xa7 kern_getdirentries+0x21c sys_getdirentries+0x28 amd64_syscall+0x351 Xfast_syscall+0xfb 65624 101755 zfs - mi_switch+0xe1 sleepq_wait+0x3a _sx_xlock_hard+0x48a _sx_xlock+0x5d spa_all_configs+0x6e zfs_ioc_pool_configs+0x19 zfsdev_ioctl+0x6f0 devfs_ioctl_f+0x114 kern_ioctl+0x255 sys_ioctl+0x13c amd64_syscall+0x351 Xfast_syscall+0xfb $ gdb zpool GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)... (gdb) r list Starting program: /sbin/zpool list (no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...[New LWP 101773] (no debugging symbols found)...(no debugging symbols found)... load: 0.14 cmd: zpool 41662 [spa_namespace_lock] 9.17r 0.05u 0.03s 0% 3220k load: 0.11 cmd: zpool 41662 [spa_namespace_lock] 18011.15r 0.05u 0.03s 0% 3220k load: 0.64 cmd: zpool 41662 [spa_namespace_lock] 82405.76r 0.05u 0.03s 0% 3220k Any and all commands that reference zpool volume zStorage1 (on ada4, mounted on /vol/zStorage1) enter D+ state, and ctrl-T (before I exit the session) indicates the process is stuck in spa_namespace_lock. The umount [brutally] succeeded (via normal umount, camcontrol reset and hot plug on the SATA drive), but ls still hangs on that directory. Drive responds normally via camcontrol. Other zpools/filesystems are operational -- I've resisted rebooting because I only have one affected volume (which I would like back). This is reoccurring on reboots: my system lasts for about 2 weeks before I start seeing errors related to ZFS. Read https://wiki.freebsd.org/AvgZfsDeadlockDebug with little help. Point me in a direction, and I can certainly help research the codebase. -- You are receiving this mail because: You are the assignee for the bug.