Date: Tue, 10 Sep 2013 11:44:11 +0200 From: Attila Nagy <bra@fsn.hu> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: High CPU usage with newnfs(d) - seems to be a cache issue Message-ID: <522EE9EB.4010706@fsn.hu> In-Reply-To: <1721695444.20803113.1378767479011.JavaMail.root@uoguelph.ca> References: <1721695444.20803113.1378767479011.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, On 09/10/13 00:57, Rick Macklem wrote: > Attila Nagy wrote: >> Hi, >> >> I've observed some insane CPU usage on stable/9@r255367. >> About the machine: >> CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2400.14-MHz >> K8-class CPU) >> real memory = 34359738368 (32768 MB) >> FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs >> FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads >> >> It does some NFS serving like this (now running oldnfs) -not quite >> peak >> times actually: >> # nfsstat -w 1 -os >> GtAttr Lookup Rdlink Read Write Rename Access Rddir >> 763 7206 1 175 92 0 915 3589 >> 748 7665 10 131 60 0 905 2923 >> 787 9657 23 204 50 0 974 2387 >> 517 9881 9 150 41 0 572 2321 >> 709 8708 71 235 70 0 1220 3271 >> 621 9157 9 254 208 0 928 2563 >> 699 5336 29 271 103 0 1242 3448 >> 656 4291 11 201 209 0 1119 3908 >> 506 3722 0 215 183 0 970 2516 >> 698 1476 1 151 66 0 903 2094 >> 501 2865 11 268 117 0 995 1392 >> 638 6284 46 233 47 0 1096 4847 >> 893 7909 47 175 73 0 870 4070 >> 651 3936 48 255 51 0 955 2514 >> 424 4211 17 223 29 0 745 1458 >> 589 8197 26 199 39 0 918 2983 >> >> It's being hammered by about 40 machines on multiple connections (it >> has >> 35 UFS file systems exported). >> >> When running newnfs (admittedly in some stupid way, with -n 32, the >> profiling was made with this, maybe this causes some lock >> contention), >> it occasionally eats 1600% CPU (means: 0 idle). >> Lowering the thread number doesn't really solves the problem, I've >> seen >> -n X*100 CPU usage peaks lately on machines with lower (4-8) -n >> counts... >> >> Doing a profiling with pmc shows that most of the time is spent in >> nfsrvd_updatecache and nfsrvd_getcache: >> http://pastebin.com/knyppv4d >> >> Switching back to oldnfsd (even with -n 32) gives a stable 50-60% CPU >> usage (out of the "possible" 1600%) when loaded. >> >> I know that there are some changes regarding this cache in the >> CURRENT >> code (along with the possibility to set some values with sysctls), >> but I >> can't run CURRENT. >> >> Any ideas on how to improve newnfsd, so we can continue serving NFS >> in >> the future days, where I won't be able to switch back to the old one? >> :) >> > Well, I put a 1 month MFC on r254337 (which I believe fixes this), so > it should be in stable/9 in about a week. Alternately, an uglier (but > semantically equivalent) patch can be found at: > http://people.freebsd.org/~rmacklem/drc4-stable9.patch > Great, I'm eagerly waiting for this to happen then. :)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?522EE9EB.4010706>