Date: Mon, 17 Mar 2003 21:02:01 -0500 From: John <jwd@bsdwins.com> To: Hackers List <freebsd-hackers@freebsd.org> Subject: NFS performance tuning Message-ID: <20030318020201.GA2329@BSDWins.Com>
next in thread | raw e-mail | index | archive | help
Hi Folks, This is an open ended email with a question about how to increase performance of a 4-stable system running in a high-load environment. The src is current as of: FreeBSD 4.8-RC #1: Sun Mar 16 15:44:01 Running top on the system: PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 93 root 2 0 360K 148K RUN 197:27 48.49% 48.49% nfsd 94 root 2 0 360K 148K RUN 28:36 5.71% 5.71% nfsd 96 root 2 0 360K 148K RUN 10:37 3.66% 3.66% nfsd 95 root 2 0 360K 148K RUN 18:34 3.27% 3.27% nfsd 98 root 2 0 360K 148K RUN 6:25 1.03% 1.03% nfsd 97 root 2 0 360K 148K RUN 8:17 0.83% 0.83% nfsd 3201 admin 30 0 1908K 1056K RUN 0:00 1.19% 0.73% top 99 root 2 0 360K 148K RUN 5:21 0.63% 0.63% nfsd 101 root 2 0 360K 148K RUN 4:18 0.20% 0.20% nfsd and a ps a minute or so later: PID PPID UID %CPU %MEM STAT TIME COMMAND 92 1 0 0.0 0.0 Is 0:00.00 nfsd: master (nfsd) 93 92 0 45.3 0.0 R 199:46.61 nfsd: server (nfsd) 94 92 0 5.0 0.0 R 28:57.60 nfsd: server (nfsd) 95 92 0 2.1 0.0 R 18:48.99 nfsd: server (nfsd) 96 92 0 1.1 0.0 R 10:45.36 nfsd: server (nfsd) 97 92 0 0.3 0.0 R 8:23.25 nfsd: server (nfsd) 98 92 0 0.4 0.0 R 6:30.47 nfsd: server (nfsd) 99 92 0 0.2 0.0 R 5:25.05 nfsd: server (nfsd) 101 92 0 0.2 0.0 R 4:21.38 nfsd: server (nfsd) The nfsd processes are almost always Runnable. The box is an athlon MP 2200+ based system on a tyan S2466 motherboard (http://www.tyan.com/products/html/tigermpx.html) with only one processor currently installed, 2Gig of Ram. The filesystem being served out lives on an Adaptec 5400S Raid controller: aac0: <Adaptec SCSI RAID 5400S> port 0x1000-0x10ff mem 0xe8000000-0xe8001fff irq 9 at device 9.0 on pci0 aac0: StrongARM SA110 233MHz, 128MB cache memory, required battery present aac0: Kernel 1.0-0, Build 5262, S/N 6b1830 aacd1: <RAID 5> on aac0 aacd1: 769984MB (1576929024 sectors) The system is located on 5 different networks: (netstat -i) Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll gx0 1500 <Link#1> 00:90:27:a1:5d:0b 12056041 0 14715548 0 0 gx0 1500 192.168.30 192.168.30.250 12055944 - 14715450 - - gx1 1500 <Link#2> 00:90:27:a1:44:5a 2269975 0 2753705 0 0 gx1 1500 192.168.31 192.168.31.250 2269878 - 2753607 - - fxp0 1500 <Link#3> 00:02:b3:60:b5:3a 241440300 0 246915204 946 0 fxp0 1500 10.14.2/24 bb03na1a.hsr.sa 241447878 - 246924642 - - fxp1 1500 <Link#4> 00:02:b3:4a:76:e6 117933878 706 122043638 921 0 fxp1 1500 10.14.3/24 bb03na1b.hsr.sa 117939587 - 122051034 - - fxp2 1500 <Link#5> 00:02:b3:60:a8:9d 303742731 0 310860180 901 0 fxp2 1500 10.14.4/24 bb03na1c.hsr.sa 303760088 - 310879510 - - (The gx cards are 1000baseSX <full-duplex>). (The fxp cards are 100baseTX <full-duplex>). and from netstat -m: %netstat -m 916/2512/34816 mbufs in use (current/peak/max): 894 mbufs allocated to data 22 mbufs allocated to packet headers 729/1160/8704 mbuf clusters in use (current/peak/max) 2948 Kbytes allocated to network (11% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines and from nfsstat: Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 31122 62636 600335411 186827 5952269 815008 90198 59781 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 22236 0 15 8067 15989 198862 48 78463316 Mknod Fsstat Fsinfo PathConf Commit GLease Vacate Evict 0 242190 8 0 149734 0 0 0 Server Ret-Failed 591695228 Server Faults 0 Server Cache Stats: Inprog Idem Non-idem Misses 228 11658 52 686794492 Server Lease Stats: Leases PeakL GLeases 0 0 0 Server Write Gathering: WriteOps WriteRPC Opsaved 815008 815008 0 The system holds alot of c and h files along with other pretty static data. The performance of the system is not bad, but I'm curious about what folks might do to tune it up some. Anyways, comments welcome! -john To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030318020201.GA2329>