From owner-freebsd-fs@FreeBSD.ORG Wed Jun 27 15:58:06 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 299CA1065670 for ; Wed, 27 Jun 2012 15:58:06 +0000 (UTC) (envelope-from dean.jones@oregonstate.edu) Received: from smtp2.oregonstate.edu (smtp2.oregonstate.edu [128.193.15.36]) by mx1.freebsd.org (Postfix) with ESMTP id F2FC08FC14 for ; Wed, 27 Jun 2012 15:58:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.oregonstate.edu (Postfix) with ESMTP id B6D1B3C2BA for ; Wed, 27 Jun 2012 08:50:45 -0700 (PDT) X-Virus-Scanned: amavisd-new at oregonstate.edu Received: from smtp2.oregonstate.edu ([127.0.0.1]) by localhost (smtp.oregonstate.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BkeCes5TvBLR for ; Wed, 27 Jun 2012 08:50:45 -0700 (PDT) Received: from mail-bk0-f54.google.com (mail-bk0-f54.google.com [209.85.214.54]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by smtp2.oregonstate.edu (Postfix) with ESMTPSA id 2775F3C2B5 for ; Wed, 27 Jun 2012 08:50:44 -0700 (PDT) Received: by bkvi18 with SMTP id i18so1305344bkv.13 for ; Wed, 27 Jun 2012 08:50:43 -0700 (PDT) Received: by 10.152.147.33 with SMTP id th1mr21486783lab.9.1340812243056; Wed, 27 Jun 2012 08:50:43 -0700 (PDT) MIME-Version: 1.0 Received: by 10.112.20.41 with HTTP; Wed, 27 Jun 2012 08:50:22 -0700 (PDT) In-Reply-To: References: From: Dean Jones Date: Wed, 27 Jun 2012 08:50:22 -0700 Message-ID: To: Levent Serinol Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-fs@freebsd.org" Subject: Re: ZFS stalls on Heavy I/O X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jun 2012 15:58:06 -0000 On Wed, Jun 27, 2012 at 2:15 AM, Levent Serinol wrote: > Hi, > > =A0Under heavy I/O load we face freeze problems on ZFS volumes on both > Freebsd 9 Release and 10 Current versions. Machines are HP servers (64bit= ) > with HP Smart array 6400 raid controllers (with battery units). Every da > device is a hardware raid5 where each one includes 9x300GB 10K SCSI hard > drivers. Main of I/O pattern happens on local system except some small NF= S > I/O from some other servers (NFS lookup/getattr/ etc.). These servers are > mail servers (qmail) with small I/O patterns (64K Read/Write). =A0Below y= ou > can =A0find procstat output on freeze time. write_limit is set to 200MB > because of the huge amount of txg_wait_opens observed before. Every proce= ss > stops on D state I think due to txg queue and other 2 queues are full. Is > there any suggestion to fix the problem ? > > btw inject_compress is the main process injecting emails to user inboxes > (databases). Also, those machines were running without problems on > Linux/XFS filesystem. For a while ago, we started =A0migration from Linux= to > Freebsd > > > http://pastebin.com/raw.php?i=3Dic3YepWQ > _______________________________________________ Looks like you are running dedup with only 12 gigs of ram? Dedup is very ram hungry and the dedup tables are probably no longer fitting entirely in memory and therefore the system is swapping and thrashing about during writes. Also ZFS really prefers to directly address drives instead of RAID controllers. It can not guarantee or know what the controller is doing behind the scenes.