From owner-freebsd-questions Mon Nov 4 21:31: 4 2002 Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 84FBB37B401 for ; Mon, 4 Nov 2002 21:31:02 -0800 (PST) Received: from mail.au.darkbluesea.com (mail.au.darkbluesea.com [203.185.208.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 14E6A43E4A for ; Mon, 4 Nov 2002 21:31:01 -0800 (PST) (envelope-from d.anker@au.darkbluesea.com) Received: (qmail 49353 invoked by uid 82); 5 Nov 2002 05:29:44 -0000 Received: from unknown (HELO ?10.0.0.188?) (10.0.0.188) by mail.au.darkbluesea.com with SMTP; 5 Nov 2002 05:29:44 -0000 Subject: NFS Performance woes From: Duncan Anker To: freebsd-questions@freebsd.org Content-Type: text/plain Content-Transfer-Encoding: 7bit X-Mailer: Ximian Evolution 1.0.8 Date: 05 Nov 2002 15:31:00 +1000 Message-Id: <1036474260.6619.48.camel@duncan> Mime-Version: 1.0 Sender: owner-freebsd-questions@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG I recently did some research into NFS performance tuning and came across the suggestion in an article on onlamp.com by Michael Lucas, that 32768 is a good value for the read and write buffers. His suggestion is these flags: tcp,intr,nfsv3,-r=32768,-w=32768 I used these options (I found tcp was mandatory, as we have multiple IPs and UDP was refusing to play nice), also adding dumbtimer to avoid the log messages about server not responding. Shortly thereafter, our project manager complained about a lot of timeouts on his web servers. More research suggested that bigger is not always better for NFS buffers (some articles indicate that r/w buffers are only for UDP, which is clearly rubbish in this case). I have tried fiddling with varying parameters to find optimal values, but I am having difficulty working out what they are. Our web servers all mount via NFS - my expectation therefore is that I should see approximately equal amounts of traffic in and out (working on the principle that the web request and the file request cancel out and the file read from the NFS server is going out unmodified). My observations have borne out this expectation. However, tuning various values from 1k up to 8k has resulted in slight increases in traffic - I don't know whether this is more throughput or transmission retries. Once I get up to 16k and 32k the amount of traffic into the boxes exceeds the amount out quite substantially, which seems to indicate problems (and indeed is causing timeouts). On the NFS server side of things, we saw a lot of watchdog timeouts, with the network choking at around 20Mbs. The card in the server is 1Gb so it really should be able to push through more than that without a sweat (and it does - it hits about 60Mbs during backups). There's something pretty screwy going on with all of this. As mentioned, the server is 1Gb, all the clients are running at 100Mbs. All machines are on the same switch, so there is nothing to degrade the performance. The NFS section of the FreeBSD handbook talked about slow FreeBSD clients with highspeed servers, and I am wondering if this is that type of scenario. I'd appreciate hearing the experience of others in terms of tuning a setup like this, or even if someone could point me in the direction of some more useful analysis tools. Thanks -- Duncan Anker Senior Systems Administrato Dark Blue Sea To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message