From owner-freebsd-fs@FreeBSD.ORG Fri Oct 2 07:59:10 2009 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0D3AF10656B3 for ; Fri, 2 Oct 2009 07:59:10 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id D7D0D8FC1E for ; Fri, 2 Oct 2009 07:59:08 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id E47AF13564E; Fri, 2 Oct 2009 09:59:06 +0200 (CEST) X-CRM114-Version: 20090423-BlameSteveJobs ( TRE 0.7.6 (BSD) ) MF-ACE0E1EA [pR: 22.1521] X-CRM114-CacheID: sfid-20091002_09590_F93EDD13 X-CRM114-Status: Good ( pR: 22.1521 ) Message-ID: <4AC5B2C7.2000200@fsn.hu> Date: Fri, 02 Oct 2009 09:59:03 +0200 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090817 Thunderbird/2.0.0.23 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org, Kip Macy References: <4AC1E540.9070001@fsn.hu> In-Reply-To: <4AC1E540.9070001@fsn.hu> X-Stationery: 0.4.10 Content-Type: text/plain; charset=ISO-8859-2; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.3 (people.fsn.hu); Fri, 02 Oct 2009 09:59:04 +0200 (CEST) Cc: Subject: Re: ARC size constantly shrinks, then ZFS slows down extremely X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2009 07:59:10 -0000 On 09/29/09 12:45, Attila Nagy wrote: > I'm using FreeBSD 8 (previously 7) on a machine with a lot of disks > and 32 GB RAM. With 7.x it ran very well for about 50 days, but > suddenly every operation have slowed down. > gstat showed that the disks are working a lot more than usual the > zpool/zfs was pretty unusable. > > I've rebooted the machine then with FreeBSD 8 in the hope the new ZFS > fixes will correct this issue (no 50 days have passed since then, so I > don't know yet) and started to monitor ZFS's statistics. > > It seems that after a reboot, the ARC size starts to grow, then > something flips the switch and it changes to shrinking, instead of > maintaining the size. > > Please see the pictures here: > http://people.fsn.hu/~bra/freebsd/20090929-zfs-arcsize/ > > Before the 27th, the machine ran FreeBSD 7, after that date it runs 8. > > As you can see, no user process tooks the memory, so I don't know why > the ARC size grows first and then start to decrease. > > Could it be that the ARC size decreases such a big amount that it > effectively disappears and this causes the IO activity go up and kill > the machine? I've upgraded another machine from an older 8-CURRENT to 8-STABLE. It has low memory (1GB) and it's i386. The above symptoms can be triggered very easily: if I do an IMAP search on a lot of mailboxes (which I do regularly), about 10 minutes needed for the IMAP server to become completely inaccessible. The machine runs fine, but every operation of the ZFS pool take ages. According to gstat there is only a very minimal disk activity. The machine can't even be rebooted, at least not in ten minutes (reboot, wait 10 minutes, nearly nothing happens, reboot -qn makes the machine disappear from the net, but it doesn't restart). Backing out this change from the 8-STABLE kernel: http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c?r1=191901&r2=191902 makes it survive about half and hour of IMAP searching. Of course only time will tell whether this helps in the long run, but so far 10/10 tries succeeded to kill the machine with this method... According to this, I would say that this change makes things worse even on low memory, i386 (1G RAM) and "there's a plenty of RAM" (32 G) amd64 servers.