From owner-freebsd-fs@FreeBSD.ORG Fri Apr 5 15:46:09 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 872DBA8F for ; Fri, 5 Apr 2013 15:46:09 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from mail.egr.msu.edu (hill.egr.msu.edu [35.9.37.162]) by mx1.freebsd.org (Postfix) with ESMTP id 428EAF34 for ; Fri, 5 Apr 2013 15:46:08 +0000 (UTC) Received: from hill (localhost [127.0.0.1]) by mail.egr.msu.edu (Postfix) with ESMTP id 1B69942444 for ; Fri, 5 Apr 2013 11:46:02 -0400 (EDT) X-Virus-Scanned: amavisd-new at egr.msu.edu Received: from mail.egr.msu.edu ([127.0.0.1]) by hill (hill.egr.msu.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gD-h4wyl0t1R for ; Fri, 5 Apr 2013 11:46:01 -0400 (EDT) Received: from EGR authenticated sender Message-ID: <515EF1B9.1010505@egr.msu.edu> Date: Fri, 05 Apr 2013 11:46:01 -0400 From: Adam McDougall User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130309 Thunderbird/17.0.4 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: Regarding regular zfs References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2013 15:46:09 -0000 On 04/05/13 06:17, Joar Jegleim wrote: > Hi FreeBSD ! > > I've already sent this one to questions@freebsd.org, but realised this list > would be a better option. > > So I've got this setup where we have a storage server delivering about > 2 million jpeg's as a backend for a website ( it's ~1TB of data) > The storage server is running zfs and every 15 minutes it does a zfs > send to a 'slave', and our proxy will fail over to the slave if the > main storage server goes down . > I've got this script that initially zfs send's a whole zfs volume, and > for every send after that only sends the diff . So after the initial zfs > send, the diff's usually take less than a minute to send over. > > I've had increasing problems on the 'slave', it seem to grind to a > halt for anything between 5-20 seconds after every zfs receive . Everything > on the server halts / hangs completely. > > I've had a couple go's on trying to solve / figure out what's > happening without luck, and this 3rd time I've invested even more time > on the problem . > > To sum it up: > -Server was initially on 8.2-RELEASE > -I've set some sysctl variables such as: > > # 16GB arc_max ( server got 30GB of ram, but had a couple 'freeze' > situations, suspect zfs.arc ate too much memory) > vfs.zfs.arc_max=17179869184 What is vm.kmem_size? I suggest setting it to 2x physical ram (60G) but higher won't hurt because recent-ish kernels will properly cap it at 2x. This will allow more space in kmem to reduce ARC fragmentation in memory which can cause stalls or panics. Also, I would set vfs.zfs.arc_max almost as high as your physical memory (or not at all) unless you need to reserve some for non-zfs usage. With ZFS from the latest year or two, I generally don't adjust anything other than making sure vm.kmem_size is sufficiently high on all systems, and some systems I use arc_max. Rarely arc_min but in an attempt to encourage ARC use. By avoiding ARC fragmentation, I generally feel more ARC is better than less. Also keep an eye on kstat.zfs.misc.arcstats.size when you run zfs recv to see how it behaves. > > # 8.2 default to 30 here, setting it to 5 which is default from 8.3 and > onwards > vfs.zfs.txg.timeout="5" > > # Set TXG write limit to a lower threshold. This helps "level out" > # the throughput rate (see "zpool iostat"). A value of 256MB works well > # for systems with 4 GB of RAM, while 1 GB works well for us w/ 8 GB on > # disks which have 64 MB cache. <
> > # NOTE: in #vfs.zfs.txg.write_limit_override=1073741824 # for 8.2 > vfs.zfs.write_limit_override=1073741824 # for 8.3 and above > > -I've implemented mbuffer for the zfs send / receive operations. With > mbuffer the sync went a lot faster, but still got the same symptoms > when the zfs receive is done, the hang / unresponsiveness returns for > 5-20 seconds > -I've upgraded to 8.3-RELEASE ( + zpool upgrade and zfs upgrade to > V28), same symptoms > -I've upgraded to 9.1-RELEASE, still same symptoms > I think it is good to be on 9.x to get ZFS fixes before 8.