From owner-freebsd-fs@FreeBSD.ORG Wed Jun 27 16:34:14 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E49C6106564A for ; Wed, 27 Jun 2012 16:34:14 +0000 (UTC) (envelope-from andrnils@gmail.com) Received: from mail-gg0-f182.google.com (mail-gg0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 974AC8FC12 for ; Wed, 27 Jun 2012 16:34:14 +0000 (UTC) Received: by ggnm2 with SMTP id m2so1297199ggn.13 for ; Wed, 27 Jun 2012 09:34:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=skPxLc4KPgItDfYr1lRxw9O0WIYwpqcdvUBY/TcB78c=; b=b2vlT4RZBfgppRMR98gux2SDEbzsj8OsVYlCnCoTR5SCfpEEOdPQSa/20hC6rR+j1e z9FyINHK/FJ/fJnzLgk18hiJIApuo4Y2GJoFROKqa9PWMhvap51rVHk8rx7cXkm016+r x3zRa8LeGUoCNnPBPxX06d3YWAmfY5LMBxTvKgiH/RqeemBWLevLzBNfQjqhlOz6e1Tl ObcLLfv4H2xSYJ19LPenO9lTN0wiRuCcNZYG0Bi+i9apRSYDmgicFrMCAYEUArNt+Oyh vi7zLZt9rQqNuf/KDKGuOoDQvbClbQtChDo/3PJjM8z96mIXreUb01fCqFszUaKwG9kW CDEA== MIME-Version: 1.0 Received: by 10.60.29.137 with SMTP id k9mr21501003oeh.23.1340814853778; Wed, 27 Jun 2012 09:34:13 -0700 (PDT) Received: by 10.60.80.36 with HTTP; Wed, 27 Jun 2012 09:34:13 -0700 (PDT) In-Reply-To: References: Date: Wed, 27 Jun 2012 18:34:13 +0200 Message-ID: From: Andreas Nilsson To: Dean Jones Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "freebsd-fs@freebsd.org" Subject: Re: ZFS stalls on Heavy I/O X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jun 2012 16:34:15 -0000 On Wed, Jun 27, 2012 at 5:50 PM, Dean Jones wrote: > On Wed, Jun 27, 2012 at 2:15 AM, Levent Serinol > wrote: > > Hi, > > > > Under heavy I/O load we face freeze problems on ZFS volumes on both > > Freebsd 9 Release and 10 Current versions. Machines are HP servers > (64bit) > > with HP Smart array 6400 raid controllers (with battery units). Every da > > device is a hardware raid5 where each one includes 9x300GB 10K SCSI hard > > drivers. Main of I/O pattern happens on local system except some small > NFS > > I/O from some other servers (NFS lookup/getattr/ etc.). These servers are > > mail servers (qmail) with small I/O patterns (64K Read/Write). Below you > > can find procstat output on freeze time. write_limit is set to 200MB > > because of the huge amount of txg_wait_opens observed before. Every > process > > stops on D state I think due to txg queue and other 2 queues are full. Is > > there any suggestion to fix the problem ? > > > > btw inject_compress is the main process injecting emails to user inboxes > > (databases). Also, those machines were running without problems on > > Linux/XFS filesystem. For a while ago, we started migration from Linux > to > > Freebsd > > > > > > http://pastebin.com/raw.php?i=ic3YepWQ > > _______________________________________________ > > Looks like you are running dedup with only 12 gigs of ram? > > Dedup is very ram hungry and the dedup tables are probably no longer > fitting entirely in memory and therefore the system is swapping and > thrashing about during writes. > > Also ZFS really prefers to directly address drives instead of RAID > controllers. It can not guarantee or know what the controller is > doing behind the scenes. > You might want to read http://constantin.glez.de/blog/2011/07/zfs-dedupe-or-not-dedupe and see if you need more ram. And yes, having raid below zfs somewhat defeats the point of zfs. Regards Andreas