Date: Tue, 10 Sep 2013 00:25:09 +0200 From: Attila Nagy <bra@fsn.hu> To: freebsd-fs <freebsd-fs@freebsd.org> Subject: High CPU usage with newnfs(d) - seems to be a cache issue Message-ID: <522E4AC5.4040606@fsn.hu>
next in thread | raw e-mail | index | archive | help
Hi,
I've observed some insane CPU usage on stable/9@r255367.
About the machine:
CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2400.14-MHz
K8-class CPU)
real memory = 34359738368 (32768 MB)
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads
It does some NFS serving like this (now running oldnfs) -not quite peak
times actually:
# nfsstat -w 1 -os
GtAttr Lookup Rdlink Read Write Rename Access Rddir
763 7206 1 175 92 0 915 3589
748 7665 10 131 60 0 905 2923
787 9657 23 204 50 0 974 2387
517 9881 9 150 41 0 572 2321
709 8708 71 235 70 0 1220 3271
621 9157 9 254 208 0 928 2563
699 5336 29 271 103 0 1242 3448
656 4291 11 201 209 0 1119 3908
506 3722 0 215 183 0 970 2516
698 1476 1 151 66 0 903 2094
501 2865 11 268 117 0 995 1392
638 6284 46 233 47 0 1096 4847
893 7909 47 175 73 0 870 4070
651 3936 48 255 51 0 955 2514
424 4211 17 223 29 0 745 1458
589 8197 26 199 39 0 918 2983
It's being hammered by about 40 machines on multiple connections (it has
35 UFS file systems exported).
When running newnfs (admittedly in some stupid way, with -n 32, the
profiling was made with this, maybe this causes some lock contention),
it occasionally eats 1600% CPU (means: 0 idle).
Lowering the thread number doesn't really solves the problem, I've seen
-n X*100 CPU usage peaks lately on machines with lower (4-8) -n counts...
Doing a profiling with pmc shows that most of the time is spent in
nfsrvd_updatecache and nfsrvd_getcache:
http://pastebin.com/knyppv4d
Switching back to oldnfsd (even with -n 32) gives a stable 50-60% CPU
usage (out of the "possible" 1600%) when loaded.
I know that there are some changes regarding this cache in the CURRENT
code (along with the possibility to set some values with sysctls), but I
can't run CURRENT.
Any ideas on how to improve newnfsd, so we can continue serving NFS in
the future days, where I won't be able to switch back to the old one? :)
Thanks,
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?522E4AC5.4040606>