From owner-freebsd-current@FreeBSD.ORG Tue Nov 25 19:23:22 2008 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C61E106564A for ; Tue, 25 Nov 2008 19:23:22 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.159]) by mx1.freebsd.org (Postfix) with ESMTP id CAD8B8FC1A for ; Tue, 25 Nov 2008 19:23:20 +0000 (UTC) (envelope-from artemb@gmail.com) Received: by fg-out-1718.google.com with SMTP id l26so56667fgb.35 for ; Tue, 25 Nov 2008 11:23:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender :to:subject:mime-version:content-type:content-transfer-encoding :content-disposition:x-google-sender-auth; bh=GI730UXaLJ6Ti8FdHXI7bJydUJJTpcWWQ5Df8mpktt8=; b=Y+O8Qhztn6Csfjjzq9rMM1egCtbgb7xHl0oDn7ID3Ha51T6GU/1Uz/8Wufu4Z3qu59 YZ9RsftyGPBM+lXVBPcxuavkWE+vNLguLyxhfw6sMECMPbgHi22MKsXE7dnYvmZAyKjY 2mJt/tuoQNO25sq1evWwGsAKuoPgKg2+0WxGQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition:x-google-sender-auth; b=M2hZra9Tbiij2osgPxsOE9roL3HPT6ZiDHjEE6ryzMoqsD2rzYBmr6QIhEK7cqNiJf kwIvW+zt6+wgpcF1PKl2I+8FL0RSqT+vr/b5NCkqCX0qvT5A7ZWb93DFJhMQzE/ukEq7 FnlghOUdkZG7783BWldQjUpxgTq0TBkm3ZZwU= Received: by 10.181.197.1 with SMTP id z1mr1638709bkp.118.1227640995970; Tue, 25 Nov 2008 11:23:15 -0800 (PST) Received: by 10.180.217.13 with HTTP; Tue, 25 Nov 2008 11:23:15 -0800 (PST) Message-ID: Date: Tue, 25 Nov 2008 11:23:15 -0800 From: "Artem Belevich" Sender: artemb@gmail.com To: freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Google-Sender-Auth: 3291416e6b036f25 Subject: -current panic (ZFS-related?): Sleeping thread ... owns a non-sleepable lock X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Nov 2008 19:23:22 -0000 Hi, After updating sources to yesterday -current and upgrading ZFS pool and filesystems I'm getting reproducible kernel panic during a build on ZFS. It looks like zfs_freebsd_getattr went to sleep while process was holding kqueue lock (acquired in knote()). If someone needs additional info, just email me ddb commands and I'd be happy to help. Sleeping thread (tid 100247, pid 5591) owns a non-sleepable lock sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0xfa sleepq_wait() at sleepq_wait+0x46 _sx_xlock_hard() at _sx_xlock_hard+0x1a2 _sx_xlock() at _sx_xlock+0xa3 rrw_enter() at rrw_enter+0x98 zfs_freebsd_getattr() at zfs_freebsd_getattr+0xa5 filt_vfsread() at filt_vfsread+0x51 knote() at knote+0xfa VOP_WRITE_APV() at VOP_WRITE_APV+0x153 vn_write() at vn_write+0x236 dofilewrite() at dofilewrite+0x7a kern_writev() at kern_writev+0x52 write() at write+0x4d ia32_syscall() at ia32_syscall+0x17c Xint0x80_syscall() at Xint0x80_syscall+0x60 panic: sleeping thread cpuid = 1 KDB: enter: panic [thread pid 8466 tid 100199 ] Stopped at kdb_enter+0x3d: movq $0,0x47dfd4(%rip) db> trace 5591 Tracing pid 5591 tid 100247 td 0xffffff0039491720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0xfa sleepq_wait() at sleepq_wait+0x46 _sx_xlock_hard() at _sx_xlock_hard+0x1a2 _sx_xlock() at _sx_xlock+0xa3 rrw_enter() at rrw_enter+0x98 zfs_freebsd_getattr() at zfs_freebsd_getattr+0xa5 filt_vfsread() at filt_vfsread+0x51 knote() at knote+0xfa VOP_WRITE_APV() at VOP_WRITE_APV+0x153 vn_write() at vn_write+0x236 dofilewrite() at dofilewrite+0x7a kern_writev() at kern_writev+0x52 write() at write+0x4d ia32_syscall() at ia32_syscall+0x17c Xint0x80_syscall() at Xint0x80_syscall+0x60 db> trace 8466 Tracing pid 8466 tid 100199 td 0xffffff000c0b9ab0 kdb_enter() at kdb_enter+0x3d panic() at panic+0x176 propagate_priority() at propagate_priority+0x221 turnstile_wait() at turnstile_wait+0x212 _mtx_lock_sleep() at _mtx_lock_sleep+0xe0 _mtx_lock_flags() at _mtx_lock_flags+0xc3 kqueue_acquire() at kqueue_acquire+0x31 kern_kevent() at kern_kevent+0x6a freebsd32_kevent() at freebsd32_kevent+0x69 ia32_syscall() at ia32_syscall+0x17c Xint0x80_syscall() at Xint0x80_syscall+0x60 db> show alllocks Process 31111 (bmake) thread 0xffffff013cd96000 (100385) shared lockmgr zfs (zfs) r = 0 (0xffffff001e069098) locked @ /usr/src-8-mq/src/sys/kern/vfs_lookup.c:442 Process 31108 (as) thread 0xffffff000c261ab0 (100165) exclusive sx user map (user map) r = 0 (0xffffff013cdb6360) locked @ /usr/src-8-mq/src/sys/vm/vm_map.c:3115 Process 31100 (cc1) thread 0xffffff003948f000 (100225) exclusive sx user map (user map) r = 0 (0xffffff013ca5dc30) locked @ /usr/src-8-mq/src/sys/vm/vm_map.c:3115 Process 31063 (cpp) thread 0xffffff000c062390 (100106) shared lockmgr nfs (nfs) r = 0 (0xffffff015a3527f8) locked @ /usr/src-8-mq/src/sys/kern/vfs_subr.c:2053 shared lockmgr nfs (nfs) r = 0 (0xffffff000cea3448) locked @ /usr/src-8-mq/src/sys/kern/vfs_subr.c:2053 Process 31017 (cc1) thread 0xffffff0043d31720 (100299) exclusive sx user map (user map) r = 0 (0xffffff0039beb070) locked @ /usr/src-8-mq/src/sys/vm/vm_map.c:3115 Process 1138 (bmake) thread 0xffffff000c082390 (100162) shared lockmgr zfs (zfs) r = 0 (0xffffff015a8c7270) locked @ /usr/src-8-mq/src/sys/kern/vfs_vnops.c:676 Process 55197 (sshd) thread 0xffffff0113652390 (100352) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff00993825e8) locked @ /usr/src-8-mq/src/sys/kern/uipc_sockbuf.c:148 Process 1066 (bmake) thread 0xffffff0039d40000 (100285) exclusive sleep mutex pmap (pmap) r = 0 (0xffffffff80993d80) locked @ /usr/src-8-mq/src/sys/amd64/amd64/pmap.c:2471 exclusive sleep mutex vm page queue mutex (vm page queue mutex) r = 0 (0xffffffff8097c580) locked @ /usr/src-8-mq/src/sys/amd64/amd64/pmap.c:2470 exclusive sx user map (user map) r = 0 (0xffffff00042a5070) locked @ /usr/src-8-mq/src/sys/vm/vm_map.c:2425 Process 11678 (sshd) thread 0xffffff000c087ab0 (100146) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff001e02a5e8) locked @ /usr/src-8-mq/src/sys/kern/uipc_sockbuf.c:148 Process 5591 (bmake) thread 0xffffff0039491720 (100247) exclusive sleep mutex kqueue (kqueue) r = 0 (0xffffff0039154500) locked @ /usr/src-8-mq/src/sys/kern/kern_event.c:1635 exclusive lockmgr zfs (zfs) r = 0 (0xffffff001ec5f7f8) locked @ /usr/src-8-mq/src/sys/kern/vfs_vnops.c:594 Process 1495 (sshd) thread 0xffffff000c0cc720 (100114) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff000cfe9ae8) locked @ /usr/src-8-mq/src/sys/kern/uipc_sockbuf.c:148 Process 1479 (sshd) thread 0xffffff000c13a000 (100103) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff001e2b8368) locked @ /usr/src-8-mq/src/sys/kern/uipc_sockbuf.c:148 --Artem