From owner-freebsd-performance@FreeBSD.ORG Fri Sep 23 12:23:20 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2312316A41F for ; Fri, 23 Sep 2005 12:23:20 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh2.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id A9D3E43D45 for ; Fri, 23 Sep 2005 12:23:19 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh2.centtech.com (8.13.1/8.13.1) with ESMTP id j8NCNIxH071297; Fri, 23 Sep 2005 07:23:18 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <4333F3B4.600@centtech.com> Date: Fri, 23 Sep 2005 07:23:16 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.11) Gecko/20050914 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Francisco Reyes References: <15412.1126634818@www56.gmx.net> <20050922214142.N50836@zoraida.natserv.net> <43336294.2020403@centtech.com> <20050923080227.T58927@zoraida.natserv.net> In-Reply-To: <20050923080227.T58927@zoraida.natserv.net> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/1098/Thu Sep 22 15:57:50 2005 on mh2.centtech.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org, mariano benedettini Subject: Re: High load average mail server 5.3-RELEASE X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Sep 2005 12:23:20 -0000 Francisco Reyes wrote: > On Thu, 22 Sep 2005, Eric Anderson wrote: > >> Also, if it is an NFS server, one should check the cpu times on the >> nfsd processes. I've found that many times there aren't enough nfsd >> processes to take the load from many clients. Increasing the number >> (double it) often helps this. The max in 5.3 is 20, but you can >> easily change it and get around it. > > > What is the parameter to change the number of nfsd processes? Use the -n flag to nfsd, so in /etc/rc.conf: nfs_server_flags="-u -t -n 1024" Yep, that's right, I have mine set to 1024. How I got to that number is simply by watching the cpu time on my nfsd processes. I do this: ps -auxw | grep nfsd | head -n 20 Which would reveal something like: root 410 4.2 0.0 1236 732 ?? S 13Sep05 1400:55.70 nfsd: server (nfsd) root 411 0.0 0.0 1236 732 ?? S 13Sep05 320:48.27 nfsd: server (nfsd) root 409 0.0 0.0 1352 924 ?? Is 13Sep05 0:00.47 nfsd: master (nfsd) root 412 0.0 0.0 1236 732 ?? S 13Sep05 83:29.31 nfsd: server (nfsd) root 413 0.0 0.0 1236 732 ?? S 13Sep05 39:18.94 nfsd: server (nfsd) root 414 0.0 0.0 1236 732 ?? S 13Sep05 25:16.79 nfsd: server (nfsd) root 416 0.0 0.0 1236 732 ?? S 13Sep05 19:39.53 nfsd: server (nfsd) root 417 0.0 0.0 1236 732 ?? S 13Sep05 15:23.14 nfsd: server (nfsd) root 418 0.0 0.0 1236 732 ?? S 13Sep05 13:05.99 nfsd: server (nfsd) root 419 0.0 0.0 1236 732 ?? S 13Sep05 11:05.15 nfsd: server (nfsd) root 420 0.0 0.0 1236 732 ?? S 13Sep05 9:59.39 nfsd: server (nfsd) root 421 0.0 0.0 1236 732 ?? S 13Sep05 8:39.97 nfsd: server (nfsd) root 422 0.0 0.0 1236 732 ?? S 13Sep05 7:34.38 nfsd: server (nfsd) root 423 0.0 0.0 1236 732 ?? S 13Sep05 6:56.37 nfsd: server (nfsd) root 424 0.0 0.0 1236 732 ?? S 13Sep05 6:40.99 nfsd: server (nfsd) root 425 0.0 0.0 1236 732 ?? S 13Sep05 6:08.44 nfsd: server (nfsd) root 426 0.0 0.0 1236 732 ?? S 13Sep05 5:40.81 nfsd: server (nfsd) root 427 0.0 0.0 1236 732 ?? S 13Sep05 6:34.98 nfsd: server (nfsd) root 428 0.0 0.0 1236 732 ?? S 13Sep05 4:34.36 nfsd: server (nfsd) root 429 0.0 0.0 1236 732 ?? S 13Sep05 3:17.89 nfsd: server (nfsd) That's on a very fast box, with a fiber channel connected 16 disk RAID0+1, with the fastest disks I can buy. Now, with slower disk back end, you'll probably see something more like this: root 438 0.0 0.1 1372 964 ?? Is 17Jun05 0:03.14 nfsd: master (nfsd) root 439 0.0 0.1 1224 744 ?? S 17Jun05 1010:27.86 nfsd: server (nfsd) root 440 0.0 0.1 1224 744 ?? I 17Jun05 231:17.30 nfsd: server (nfsd) root 441 0.0 0.1 1224 744 ?? I 17Jun05 125:17.14 nfsd: server (nfsd) root 442 0.0 0.1 1224 744 ?? I 17Jun05 76:30.62 nfsd: server (nfsd) root 444 0.0 0.1 1224 744 ?? I 17Jun05 94:18.99 nfsd: server (nfsd) root 445 0.0 0.1 1224 744 ?? I 17Jun05 54:48.31 nfsd: server (nfsd) root 446 0.0 0.1 1224 744 ?? I 17Jun05 63:40.74 nfsd: server (nfsd) root 447 0.0 0.1 1224 744 ?? I 17Jun05 38:29.15 nfsd: server (nfsd) root 448 0.0 0.1 1224 744 ?? I 17Jun05 46:12.29 nfsd: server (nfsd) root 449 0.0 0.1 1224 744 ?? I 17Jun05 15:24.99 nfsd: server (nfsd) root 450 0.0 0.1 1224 744 ?? I 17Jun05 13:20.40 nfsd: server (nfsd) root 451 0.0 0.1 1224 744 ?? I 17Jun05 28:09.07 nfsd: server (nfsd) root 452 0.0 0.1 1224 744 ?? I 17Jun05 11:19.46 nfsd: server (nfsd) root 453 0.0 0.1 1224 744 ?? I 17Jun05 19:34.98 nfsd: server (nfsd) root 454 0.0 0.1 1224 744 ?? I 17Jun05 9:59.46 nfsd: server (nfsd) root 455 0.0 0.1 1224 744 ?? I 17Jun05 8:36.71 nfsd: server (nfsd) root 456 0.0 0.1 1224 744 ?? I 17Jun05 7:28.79 nfsd: server (nfsd) root 457 0.0 0.1 1224 744 ?? I 17Jun05 6:18.29 nfsd: server (nfsd) root 458 0.0 0.1 1224 744 ?? I 17Jun05 5:43.77 nfsd: server (nfsd) See how much run time that 20th process has? I like to always have a couple with zero run time. So I double the number until I get there, then tune back a little. I've found that a very rough rule of thumb is: number of nfs clients * 1.8 = nfsd processes as long as you have the memory for it. You should consider about: nfsd processes * 2.5MB = memory desired for all the nfsd's Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Anything that works is better than anything that doesn't. ------------------------------------------------------------------------