From owner-freebsd-bugs@FreeBSD.ORG Wed Jun 27 09:20:08 2012 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 10F4D1065673 for ; Wed, 27 Jun 2012 09:20:08 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id BF5348FC16 for ; Wed, 27 Jun 2012 09:20:07 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q5R9K7ZO088706 for ; Wed, 27 Jun 2012 09:20:07 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q5R9K7DN088705; Wed, 27 Jun 2012 09:20:07 GMT (envelope-from gnats) Resent-Date: Wed, 27 Jun 2012 09:20:07 GMT Resent-Message-Id: <201206270920.q5R9K7DN088705@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Levent Serinol Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9481E106566B for ; Wed, 27 Jun 2012 09:14:53 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id 800EE8FC12 for ; Wed, 27 Jun 2012 09:14:53 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q5R9Eqh7010510 for ; Wed, 27 Jun 2012 09:14:52 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id q5R9EqsC010501; Wed, 27 Jun 2012 09:14:52 GMT (envelope-from nobody) Message-Id: <201206270914.q5R9EqsC010501@red.freebsd.org> Date: Wed, 27 Jun 2012 09:14:52 GMT From: Levent Serinol To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/169480: ZFS stalls on heavy I/O X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jun 2012 09:20:08 -0000 >Number: 169480 >Category: kern >Synopsis: ZFS stalls on heavy I/O >Confidential: no >Severity: serious >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Wed Jun 27 09:20:07 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Levent Serinol >Release: 9.0 and 10.0 >Organization: >Environment: FreeBSD localhost 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Wed Mar 21 11:15:41 EET 2012 root@webmail122.mynet.com:/usr/obj/usr/src/sys/GENERIC amd64 >Description: Under heavy I/O load we face freeze problems on ZFS volumes on both Freebsd 9 Release and 10 Current versions. Machines are HP servers (64bit) with HP Smart array 6400 raid controllers (with battery units). Every da device is a hardware raid5 where each one includes 9x300GB 10K SCSI hard drivers. Main of I/O pattern happens on local system except some small NFS I/O from some other servers (NFS lookup/getattr/ etc.). These servers are mail servers (qmail) with small I/O patterns (64K Read/Write). Below you can find procstat output on freeze time. write_limit is set to 200MB because of the huge amount of txg_wait_opens observed before. Every process stops on D state I think due to txg queue and other 2 queues are full. Is there any suggestion to fix the problem ? btw inject_compress is the main process injecting emails to user inboxes (databases). Also, those machines were running without problems on Linux/XFS filesystem. For a while ago, we started migration from Linux to Freebsd http://pastebin.com/raw.php?i=ic3YepWQ >How-To-Repeat: >Fix: >Release-Note: >Audit-Trail: >Unformatted: