From owner-freebsd-isp@FreeBSD.ORG Thu Dec 29 10:19:16 2005 Return-Path: X-Original-To: freebsd-isp@freebsd.org Delivered-To: freebsd-isp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 75FB216A41F for ; Thu, 29 Dec 2005 10:19:16 +0000 (GMT) (envelope-from boisan@gmail.com) Received: from nproxy.gmail.com (nproxy.gmail.com [64.233.182.193]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7741843D5A for ; Thu, 29 Dec 2005 10:19:15 +0000 (GMT) (envelope-from boisan@gmail.com) Received: by nproxy.gmail.com with SMTP id o60so93447nfa for ; Thu, 29 Dec 2005 02:19:15 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=X3wZw5pHIem36R9+EACNdgMuQcDFRGj+ktcXZ+75Ioxn9t8RGNq70qWR4UKaBBCepts9dnIHqhQkl0t92FPhfUmqk8YxCqFzLcPGy7ZEYQSjLe5g/W/ixY+g7pG5tO3+rO7Zy9g69yLzAdnEQvWnnDe5BH04XPrqgWY+cqsi6z0= Received: by 10.48.142.8 with SMTP id p8mr351312nfd; Thu, 29 Dec 2005 02:19:14 -0800 (PST) Received: by 10.49.41.9 with HTTP; Thu, 29 Dec 2005 02:19:14 -0800 (PST) Message-ID: Date: Thu, 29 Dec 2005 11:19:14 +0100 From: Angel Blazquez To: freebsd-isp@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Subject: nfs server overload (nfsd) X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Dec 2005 10:19:16 -0000 Hello, We are expecting incredible overload in a NFS server. A top shows nfsd consuming most of the CPU: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU COMMA= ND 6000 root -8 0 1204K 660K biord 1 124:15 27.88% 27.88% nfsd 6002 root 4 0 1204K 660K *Giant 0 124:18 17.58% 17.58% nfsd 6006 root 4 0 1204K 660K *Giant 0 123:38 10.21% 10.21% nfsd 6005 root 4 0 1204K 660K *Giant 0 123:36 7.47% 7.47% nfsd 6003 root 4 0 1204K 660K *Giant 0 123:08 4.15% 4.15% nfsd 6001 root 4 0 1204K 660K *Giant 0 123:16 2.83% 2.83% nfsd Memory looks fine: Mem: 27M Active, 910M Inact, 136M Wired, 51M Cache, 112M Buf, 1828K Free Swap: 2048M Total, 72K Used, 2048M Free Typing in the nfs server (console/ssh) becomes terrible, the server does not reply well. We are running this nfs server in FreeBSD 5.3-RELEASE-p23 on a Compaq Proliant server with a Compaq Smart Array 5300 that comunicates with a array of disks: /dev/da0s1d 164G 124G 27G 82% /data0 /dev/da1s1d 131G 80G 41G 66% /data1 We have /data0 and /data1 exported: /data0 -maproot=3Droot -alldirs -network 192.168.62.0 -mask 255.255.255.0 /data1 -maproot=3Droot -alldirs -network 192.168.62.0 -mask 255.255.255.0 so a couple of incoming SMTP servers we have can deliver e-mail to those filesystems. We are running exim 4.60.0 in those other servers, 4.10-RELEASE-p5 in one of them, and FreeBSD 6.0-RELEASE #0 in the other one. If we stop exim delivering e-mail, nfs server does well, the cpu gets free, and the nfs server works fine (replies to user interaction, etc). FreeBSD 6.0 sysctl output (nfs related): vfs.nfs4.access_cache_timeout: 60 vfs.nfs4.nfsv3_commit_on_close: 0 vfs.nfs.downdelayinitial: 12 vfs.nfs.downdelayinterval: 30 vfs.nfs.realign_test: 1294030 vfs.nfs.realign_count: 0 vfs.nfs.bufpackets: 4 vfs.nfs.reconnects: 2 vfs.nfs.iodmaxidle: 120 vfs.nfs.iodmin: 4 vfs.nfs.iodmax: 20 vfs.nfs.defect: 0 vfs.nfs.nfs_ip_paranoia: 1 vfs.nfs.diskless_valid: 0 vfs.nfs.diskless_rootpath: vfs.nfs.access_cache_timeout: 2 vfs.nfs.nfsv3_commit_on_close: 0 vfs.nfs.clean_pages_on_close: 1 vfs.nfs.nfs_directio_enable: 0 vfs.nfs.nfs_directio_allow_mmap: 1 vfs.nfsrv.nfs_privport: 0 vfs.nfsrv.async: 0 vfs.nfsrv.commit_blks: 0 vfs.nfsrv.commit_miss: 0 vfs.nfsrv.realign_test: 0 vfs.nfsrv.realign_count: 0 vfs.nfsrv.gatherdelay: 10000 vfs.nfsrv.gatherdelay_v3: 0 FreeBSD 4.10 sysctl output (nfs related): vfs.nfs.nfs_privport: 0 vfs.nfs.async: 0 vfs.nfs.commit_blks: 0 vfs.nfs.commit_miss: 0 vfs.nfs.realign_test: 84602323 vfs.nfs.realign_count: 99713 vfs.nfs.bufpackets: 4 vfs.nfs.gatherdelay: 10000 vfs.nfs.gatherdelay_v3: 0 vfs.nfs.defect: 0 vfs.nfs.nfs_ip_paranoia: 1 vfs.nfs.diskless_valid: 0 vfs.nfs.diskless_rootpath: vfs.nfs.diskless_swappath: vfs.nfs.access_cache_timeout: 2 vfs.nfs.nfsv3_commit_on_close: 0 This couple of servers mounts the filesystems with this options: 192.168.62.54:/data1 /mail nfs =20 rw,nfsv3,intr,dumbtimer,rdirplus,nosuid,nodev 0 0 192.168.62.54:/data0 /data0 nfs =20 rw,nfsv3,intr,dumbtimer,rdirplus,nosuid,nodev 0 0 On the server, sysctl nfs related output looks like this: vfs.nfs.downdelayinitial: 12 vfs.nfs.downdelayinterval: 30 vfs.nfs.realign_test: 2694 vfs.nfs.realign_count: 0 vfs.nfs.bufpackets: 4 vfs.nfs.reconnects: 2 vfs.nfs.iodmaxidle: 120 vfs.nfs.iodmin: 4 vfs.nfs.iodmax: 20 vfs.nfs.defect: 0 vfs.nfs.nfs_ip_paranoia: 1 vfs.nfs.diskless_valid: 0 vfs.nfs.diskless_rootpath: vfs.nfs.access_cache_timeout: 2 vfs.nfs.nfsv3_commit_on_close: 0 vfs.nfs4.access_cache_timeout: 60 vfs.nfs4.nfsv3_commit_on_close: 0 vfs.nfsrv.nfs_privport: 0 vfs.nfsrv.async: 1 vfs.nfsrv.commit_blks: 579238 vfs.nfsrv.commit_miss: 413059 vfs.nfsrv.realign_test: 88269083 vfs.nfsrv.realign_count: 11961 vfs.nfsrv.gatherdelay: 10000 vfs.nfsrv.gatherdelay_v3: 0 debug.hashstat.nfsnode: 65536 5 1 0 Thanks in advance, Best regards, Angel Blazquez