From owner-freebsd-fs@FreeBSD.ORG Tue Jan 26 12:06:12 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0040B106566C for ; Tue, 26 Jan 2010 12:06:11 +0000 (UTC) (envelope-from gallasch@free.de) Received: from smtp.free.de (smtp.free.de [91.204.6.103]) by mx1.freebsd.org (Postfix) with ESMTP id 500308FC08 for ; Tue, 26 Jan 2010 12:06:10 +0000 (UTC) Received: (qmail 883 invoked from network); 26 Jan 2010 13:06:09 +0100 Received: from smtp.free.de (HELO orwell.free.de) (gallasch@free.de@[91.204.4.103]) (envelope-sender ) by smtp.free.de (qmail-ldap-1.03) with AES128-SHA encrypted SMTP for ; 26 Jan 2010 13:06:09 +0100 Date: Tue, 26 Jan 2010 13:06:07 +0100 From: Kai Gallasch To: freebsd-fs@freebsd.org Message-ID: <20100126130607.3204351d@orwell.free.de> X-Mailer: Claws Mail 3.7.3 (GTK+ 2.18.2; powerpc-apple-darwin9.8.0) X-Face: 7"x0zA5=*cXGZw-xjU<">'+!3(KXTUXZVLD42KVN{'go[UQr"Mc.e(XW92N8plZ(9x.{x; I<|95e+b&GH-36\15F~L$YD*Y +u}o&KV?6.%"mJIkaY3G>BKNt`1|Y+%K1P4t; 47D65&(Y7h5Ll-[ltkhamx.-; ,jggK'}oMpUgEHFG YQ"9oXKAl>!d,J}T{)@uxvfu?YFWC*\~h+,^f Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: zfs problem with processes stuck in state zfs and zilog- X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jan 2010 12:06:12 -0000 Hi. Some mail services (qmail, dovecot) on one of my freebsd jails became unavailable. The jail is using zfs as a filesystem. After logging in I found that I could not kill processes related to the unavailable services running inside the jail and that the jail could not be restartet. top revealed the following: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 8320 root 1 44 0 4772K 1088K zilog- 1 0:58 0.00% multilog 8336 qmaill 1 44 0 4772K 1088K zilog- 6 0:21 0.00% multilog 8464 mysql 2 44 0 108M 70356K STOP 7 0:21 0.00% mysqld 8292 spamd 1 44 0 4772K 1088K zilog- 7 0:18 0.00% multilog 8295 nobody 1 44 0 4772K 1088K zilog- 2 0:10 0.00% multilog 8298 qmaill 1 44 0 4772K 1088K zilog- 3 0:01 0.00% multilog 8294 root 1 44 0 4772K 1088K zilog- 4 0:01 0.00% multilog 8304 qmaill 1 44 0 4772K 1088K zilog- 1 0:01 0.00% multilog 34278 www 1 63 0 29852K 14828K zfs 4 0:00 0.00% perl 34368 www 1 76 0 29852K 14828K zfs 4 0:00 0.00% perl 8319 root 1 44 0 4772K 1088K zilog- 2 0:00 0.00% multilog 4311 qmaild 1 59 0 2792K 1040K zilog- 1 0:00 0.00% qmail-queue 32203 qmaild 1 60 0 2792K 1040K zilog- 0 0:00 0.00% qmail-queue 16225 qmaild 1 59 0 2792K 1040K zilog- 4 0:00 0.00% qmail-queue 32254 qmaild 1 60 0 2792K 1040K zilog- 0 0:00 0.00% qmail-queue 16545 qmaild 1 57 0 2792K 1040K zilog- 4 0:00 0.00% qmail-queue 4521 qmaild 1 56 0 2792K 1040K zilog- 5 0:00 0.00% qmail-queue 8327 root 1 44 0 4772K 1088K zilog- 6 0:00 0.00% multilog 36733 qmaild 1 52 0 2792K 1040K zilog- 4 0:00 0.00% qmail-queue 47903 root 1 44 0 10284K 3412K pause 0 0:00 0.00% tcsh 35517 qmaild 1 63 0 2792K 1040K zilog- 4 0:00 0.00% qmail-queue 35492 qmaild 1 63 0 2792K 1040K zilog- 0 0:00 0.00% qmail-queue 62826 qmaild 1 53 0 2792K 1040K zilog- 4 0:00 0.00% qmail-queue 69033 qmaild 1 53 0 2792K 1040K zilog- 0 0:00 0.00% qmail-queue 66010 qmaild 1 50 0 2792K 1040K zilog- 4 0:00 0.00% qmail-queue 40947 qmaild 1 50 0 2792K 1040K zilog- 0 0:00 0.00% qmail-queue 27726 qmaild 1 51 0 2792K 1040K zilog- 0 0:00 0.00% qmail-queue 35520 qmaild 1 53 0 2792K 1040K zilog- 6 0:00 0.00% qmail-queue 30639 root 1 44 0 12164K 3420K zfs 1 0:00 0.00% perl 23538 root 1 44 0 12164K 3420K zfs 1 0:00 0.00% perl 4834 root 1 76 0 12164K 3420K zfs 1 0:00 0.00% perl 26410 root 1 47 0 12164K 3420K zfs 0 0:00 0.00% perl 97136 root 1 76 0 12164K 3420K zfs 4 0:00 0.00% perl 9290 root 1 44 0 12164K 3420K zfs 1 0:00 0.00% perl 18127 root 1 44 0 12164K 3420K zfs 5 0:00 0.00% perl 12553 root 1 44 0 12164K 3420K zfs 4 0:00 0.00% perl 21574 root 1 44 0 12164K 3420K zfs 4 0:00 0.00% perl 2323 root 1 44 0 12164K 3420K zfs 0 0:00 0.00% perl 68707 root 1 45 0 12164K 3420K zfs 2 0:00 0.00% perl 9294 root 1 44 0 12164K 3420K zfs 1 0:00 0.00% perl 40393 root 1 44 0 12164K 3420K zfs 1 0:00 0.00% perl 41985 root 1 44 0 12164K 3420K zfs 0 0:00 0.00% perl 21672 root 1 44 0 12164K 3420K zfs 0 0:00 0.00% perl 10420 root 1 44 0 12164K 3420K zfs 4 0:00 0.00% perl 6965 root 1 44 0 12164K 3420K zfs 6 0:00 0.00% perl 14132 root 1 44 0 12164K 3420K zfs 1 0:00 0.00% perl 14867 root 1 44 0 12164K 3420K zfs 4 0:00 0.00% perl 77145 root 1 44 0 12164K 3420K zfs 4 0:00 0.00% perl 16534 root 1 44 0 12164K 3420K zfs 0 0:00 0.00% perl 25130 root 1 44 0 12164K 3420K zfs 0 0:00 0.00% perl 63026 root 1 44 0 12164K 3420K zfs 4 0:00 0.00% perl 44663 root 1 44 0 12164K 3420K zfs 4 0:00 0.00% perl 28139 root 1 44 0 12164K 3420K zfs 0 0:00 0.00% perl 34440 root 1 44 0 12164K 3420K zfs 4 0:00 0.00% perl 35619 qmaild 1 46 0 2792K 1040K zilog- 0 0:00 0.00% qmail-queue 26146 root 1 44 0 12164K 3420K zfs 0 0:00 0.00% perl Because the server is in active use, there was no time for me to look into this further - and no trace of the problem in syslog. Any suggestions how to debug this problem and find out what's wrong here? Does the state zilog- give any hint? BTW: FreeBSD 8.0-STABLE-amd64 / zpool version 13 --Kai -- You had mail, but the super-user read it, laughed and deleted it!