From owner-freebsd-questions@freebsd.org Mon Sep 28 02:51:20 2015 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C5746A0B6C3 for ; Mon, 28 Sep 2015 02:51:20 +0000 (UTC) (envelope-from healer@rpi.edu) Received: from smtp9.server.rpi.edu (smtp9.server.rpi.edu [128.113.2.229]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "canit.localdomain", Issuer "canit.localdomain" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 80E137A7 for ; Mon, 28 Sep 2015 02:51:20 +0000 (UTC) (envelope-from healer@rpi.edu) Received: from smtp-auth1.server.rpi.edu (smtp-auth1.server.rpi.edu [128.113.2.231]) by smtp9.server.rpi.edu (8.14.3/8.14.3/Debian-9.4) with ESMTP id t8S2p42F009374 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Sun, 27 Sep 2015 22:51:05 -0400 Received: from smtp-auth1.server.rpi.edu (localhost [127.0.0.1]) by smtp-auth1.server.rpi.edu (Postfix) with ESMTP id DCAF6580C9 for ; Sun, 27 Sep 2015 22:51:04 -0400 (EDT) Received: from [128.113.208.45] (vpn-208-45.net.rpi.edu [128.113.208.45]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: healer) by smtp-auth1.server.rpi.edu (Postfix) with ESMTPSA id B739E5800F for ; Sun, 27 Sep 2015 22:51:04 -0400 (EDT) Subject: Re: Problems with ZFS file servers To: freebsd-questions@freebsd.org References: <55F5CF06.5080602@rpi.edu> <1442191342.426007007.amnhjwng@frv35.fwdcdn.com> <55F62523.7010402@rpi.edu> <1442267713.360040098.um8wwy4s@frv35.fwdcdn.com> <5605A3A5.7080905@rpi.edu> <5605AD8D.4080706@FreeBSD.org> <20150925232658.GB42532@neutralgood.org> <4abb7c52.5adf5378@fabiankeil.de> From: Bob Healey Message-ID: <5608AB17.1010300@rpi.edu> Date: Sun, 27 Sep 2015 22:51:03 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <4abb7c52.5adf5378@fabiankeil.de> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP X-Bayes-Prob: 0.0001 (Score 0, tokens from: outgoing, @@RPTN) X-Spam-Score: 0.00 () [Hold at 7.10] X-CanIt-Incident-Id: 02PmqP5HB X-CanIt-Geo: ip=128.113.208.45; country=US; region=New York; city=Troy; latitude=42.7495; longitude=-73.5951; http://maps.google.com/maps?q=42.7495,-73.5951&z=6 X-CanItPRO-Stream: outgoing X-Canit-Stats-ID: Bayes signature not available X-Scanned-By: CanIt (www . roaringpenguin . com) on 128.113.2.229 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Sep 2015 02:51:21 -0000 My plan, which won't be implemented before 6/9/16 is to set "-tso4 -tso -txcsum -rxcsum -vlanhwtso -lro" on all the nics, and in loader.conf, set hw.igb.rxd=4096 hw.igb.txd=4096 vfs.zfs.arc_max="75% of physical mem" Any other options I may have overlooked? I've got 9 months before I get an outage window. Academia demands no downtime not caused by acts of god or facilities shutdowns. These machines have no purpose in life other than nfs, occasional rsync, and sometimes samba 3. Bob Healey Systems Administrator Biocomputation and Bioinformatics Constellation and Molecularium healer@rpi.edu (518) 276-4407 On 9/26/2015 7:04 AM, Fabian Keil wrote: > kpneal@pobox.com wrote: > >> On Fri, Sep 25, 2015 at 09:24:45PM +0100, Matthew Seaman wrote: >>> On 25/09/2015 20:42, Bob Healey wrote: >>>> I've got another machine acting up, this machine is a Sun X2250, >>>> originally was running Solaris until this summer when the owner dropped >>>> the support contract. A zpool export, reinstall to FreeBSD 10.1, and >>>> zpool import and it was back in business. Most of the requested info >>>> can be found at http://origami.phys.rpi.edu/~healer/lepton. I am >>>> working on getting cacti installed. I am currently trying to rsync a >>>> workstation to this pool so I can reload the OS. If I use >>>> --bwlimit=10240 or lower, I have no issues, but if I don't rsync freezes >>>> on me on the client side. >>> I've found, through bitter experience, that you need to apply some >>> tunings to ZFS machines, and quite possibly some kernel patches too. >>> When you're pumping wads of data into a ZFS machine at high speed, it is >>> all too easy to get it to lock up. >>> >>> First up, the default setting where ZFS grabs all but 1GB of available >>> RAM for use by the ARC is nuts. You need to chop that down and give the >>> rest of the OS a fair share of RAM to play with by setting >>> vfs.zfs.arc_max in /boot/loader.conf. What you set it to depends on the >>> application mix on your server, but somewhere around 50% of available >>> RAM seems reasonable to me. Reboot to enable that, obviously. >> My personal experience: >> >> I have swap space configured. My box has 8GB of memory, and when I save >> large mailboxes with mutt I see up to 4GB of swap space used. >> >> It may be the case that adding swap space will eliminate the need to >> limit the ARC manually in /boot/loader.conf. > Without the ARC patches or manual tuning the swap use could be the result > of the ARC failing to adapt to the memory pressure in which case even > active processes may start paging that otherwise wouldn't have to. > > In that situation, adding more swap space may degrade performance even > further as it allows the ARC to hold on to the memory even longer. > Thus I wouldn't recommend it without analysing the cause of the swap > use first. > > Fabian