From owner-freebsd-bugs@FreeBSD.ORG Fri Oct 22 23:30:12 2010 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5BED31065673 for ; Fri, 22 Oct 2010 23:30:12 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1EB768FC08 for ; Fri, 22 Oct 2010 23:30:12 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o9MNUBJI074740 for ; Fri, 22 Oct 2010 23:30:11 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o9MNUBtn074737; Fri, 22 Oct 2010 23:30:11 GMT (envelope-from gnats) Resent-Date: Fri, 22 Oct 2010 23:30:11 GMT Resent-Message-Id: <201010222330.o9MNUBtn074737@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Bartosz Nowicki Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4143E1065670 for ; Fri, 22 Oct 2010 23:27:54 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 16F128FC1D for ; Fri, 22 Oct 2010 23:27:54 +0000 (UTC) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o9MNRrPt089082 for ; Fri, 22 Oct 2010 23:27:53 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id o9MNRrL9089080; Fri, 22 Oct 2010 23:27:53 GMT (envelope-from nobody) Message-Id: <201010222327.o9MNRrL9089080@www.freebsd.org> Date: Fri, 22 Oct 2010 23:27:53 GMT From: Bartosz Nowicki To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/151648: [zfs] disk wait bug X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Oct 2010 23:30:12 -0000 >Number: 151648 >Category: kern >Synopsis: [zfs] disk wait bug >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri Oct 22 23:30:11 UTC 2010 >Closed-Date: >Last-Modified: >Originator: Bartosz Nowicki >Release: 8.1-STABLE >Organization: admin.net.pl >Environment: FreeBSD mydevil.mydevil.net 8.1-STABLE FreeBSD 8.1-STABLE #4: Fri Oct 1 00:04:06 UTC 2010 root@:/usr/obj/usr/src/sys/mydevil amd64 >Description: During normal usage, nginx www server stopped responding. Problem looks like ZFS issue: # ps lU www UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 80 3645 67746 0 54 10 46012 33764 kqread SN ?? 0:00,06 nginx: worker process (nginx) 80 8410 1 0 54 10 46012 34120 zfs DN ?? 0:01,14 nginx: worker process (nginx) 80 15225 67746 0 54 10 46012 33388 zfs DN ?? 0:02,50 nginx: worker process (nginx) 80 15226 67746 0 54 10 46012 33372 zfs DN ?? 0:00,47 nginx: worker process (nginx) 80 18427 1 0 54 10 46012 34160 zfs DN ?? 6:51,11 nginx: worker process (nginx) 80 22955 67746 0 54 10 46012 33416 zfs DN ?? 0:01,07 nginx: worker process (nginx) 80 22956 67746 0 54 10 46012 33400 zfs DN ?? 0:00,68 nginx: worker process (nginx) 80 26904 67746 0 54 10 46012 33776 zfs DN ?? 0:00,53 nginx: worker process (nginx) 80 26905 67746 0 54 10 46012 33424 zfs DN ?? 0:00,96 nginx: worker process (nginx) 80 29355 1 0 54 10 46012 34136 zfs DN ?? 0:38,33 nginx: worker process (nginx) 80 29356 1 0 54 10 46012 34144 zfs DN ?? 0:40,52 nginx: worker process (nginx) 80 43125 67746 0 54 10 46012 33700 zfs DN ?? 0:00,15 nginx: worker process (nginx) 80 94412 67746 0 54 10 46012 33724 zfs DN ?? 0:00,01 nginx: worker process (nginx) # procstat -k 26904 PID TID COMM TDNAME KSTACK 26904 100390 nginx - mi_switch sleepq_wait __lockmgr_args vop_stdlock VOP_LOCK1_APV _vn_lock vget cache_lookup vfs_cache_lookup VOP_LOOKUP_APV lookup namei vn_open_cred kern_openat syscall Xfast_syscall # procstat -k 29355 PID TID COMM TDNAME KSTACK 29355 102233 nginx - mi_switch sleepq_wait __lockmgr_args vop_stdlock VOP_LOCK1_APV _vn_lock vget cache_lookup vfs_cache_lookup VOP_LOOKUP_APV lookup namei vn_open_cred kern_openat syscall Xfast_syscall Processes are also immportal. Kill gives no result. # kill -9 29355 # ps lp 29355 UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 80 29355 1 0 54 10 46012 34136 zfs DN ?? 0:38,33 nginx: worker process (nginx) >How-To-Repeat: Start nginx server, wait several hours. >Fix: >Release-Note: >Audit-Trail: >Unformatted: