From owner-freebsd-net@FreeBSD.ORG Wed May 19 17:55:47 2004 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7780916A4CE; Wed, 19 May 2004 17:55:47 -0700 (PDT) Received: from otter3.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1B8AE43D2F; Wed, 19 May 2004 17:55:47 -0700 (PDT) (envelope-from anderson@centtech.com) Received: from centtech.com ([192.168.42.25]) by otter3.centtech.com (8.12.3/8.12.3) with ESMTP id i4K0tfE8075672; Wed, 19 May 2004 19:55:42 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <40AC01F0.3030409@centtech.com> Date: Wed, 19 May 2004 19:55:12 -0500 From: Eric Anderson User-Agent: Mozilla Thunderbird 0.5 (X11/20040406) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Dan Nelson References: <40ABD7C8.7050405@centtech.com> <20040519221048.GA86452@dan.emsphone.com> In-Reply-To: <20040519221048.GA86452@dan.emsphone.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-net@freebsd.org cc: freebsd-questions@freebsd.org Subject: Re: Max NFSD processes X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 May 2004 00:55:47 -0000 Dan Nelson wrote: >In the last episode (May 19), Eric Anderson said: > > >>I have several heavily used NFS servers, currently running FreeBSD >>4.9-RELEASE. I'm getting jammed up with all my nfsd processes being >>busy, so clients see slow connections to the server. I have the nfsd >>starting with a count of 20, which is the max set in the nfsd.c file. >> >>Are there any risks I should be aware of before bumping up the max to >>say 40, or even 50? >> >>What would it take to make this a sysctl adjustable value? >> >>Should the max be bumped higher by default nowdays? >> >> > >What's the output of "ps ax | grep nfsd"? How much CPU does the last >nfsd process have? > >If your backend storage is a RAID with lots of disks, and your last >nfsd is actually getting some use, then bumping up the nfsds will >probably help. Although if you're hitting a kernel bottleneck (locking >for example), more nfsds won't do any good. > > Here's the output: 97 ?? Is 0:00.01 nfsd: master (nfsd) 99 ?? S 4:52.61 nfsd: server (nfsd) 100 ?? S 1:15.74 nfsd: server (nfsd) 101 ?? S 0:44.05 nfsd: server (nfsd) 102 ?? S 0:31.79 nfsd: server (nfsd) 103 ?? S 0:26.15 nfsd: server (nfsd) 104 ?? S 0:20.36 nfsd: server (nfsd) 105 ?? S 0:18.47 nfsd: server (nfsd) 106 ?? S 0:16.86 nfsd: server (nfsd) 107 ?? S 0:19.11 nfsd: server (nfsd) 108 ?? S 0:16.68 nfsd: server (nfsd) 109 ?? S 0:13.59 nfsd: server (nfsd) 110 ?? S 0:13.60 nfsd: server (nfsd) 111 ?? S 0:12.30 nfsd: server (nfsd) 112 ?? S 0:12.44 nfsd: server (nfsd) 113 ?? S 0:13.84 nfsd: server (nfsd) 114 ?? S 0:12.65 nfsd: server (nfsd) 115 ?? S 0:13.57 nfsd: server (nfsd) 116 ?? S 0:11.31 nfsd: server (nfsd) 117 ?? S 0:11.21 nfsd: server (nfsd) 118 ?? I 0:11.99 nfsd: server (nfsd) The machine has been up now less than 5 hours, and this is a 'quiet' time. During the 'slow' time, top showed the nfsd processes in "biorw" and "inode" states. All were consumed in those states. The machine has two raid 5 arrays, with a hardware raid controller. iostat showed xfer speeds to the first array about 2MB/s, and nothing really abnormal about it. Clients had difficulty with simple things like 'ls' on the partition. mountd was responding quickly with mount requests (I believe), but once the mount was made, accessing the nfs disk was horribly slow. Any more ideas? I'm not subscribed on -questions or -net, so please keep me on the cc's. Eric -- ------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Today is the tomorrow you worried about yesterday. ------------------------------------------------------------------