Date: Fri, 6 Oct 2006 06:25:27 +1000 (EST) From: Bruce Evans <bde@zeta.org.au> To: fs@freebsd.org Subject: lost dotdot caching pessimizes nfs especially Message-ID: <20061006050913.Y5250@epsplex.bde.org>
next in thread | raw e-mail | index | archive | help
This change:
% Index: vfs_cache.c
% ===================================================================
% RCS file: /home/ncvs/src/sys/kern/vfs_cache.c,v
% retrieving revision 1.102
% retrieving revision 1.103
% diff -u -2 -r1.102 -r1.103
% --- vfs_cache.c 13 Jun 2005 05:59:59 -0000 1.102
% +++ vfs_cache.c 17 Jun 2005 01:05:13 -0000 1.103
% @@ -494,6 +494,16 @@
% return;
% }
% + /*
% + * For dotdot lookups only cache the v_dd pointer if the
% + * directory has a link back to its parent via v_cache_dst.
% + * Without this an unlinked directory would keep a soft
% + * reference to its parent which could not be NULLd at
% + * cache_purge() time.
% + */
% if (cnp->cn_namelen == 2 && cnp->cn_nameptr[1] == '.') {
% - dvp->v_dd = vp;
% + CACHE_LOCK();
% + if (!TAILQ_EMPTY(&dvp->v_cache_dst))
% + dvp->v_dd = vp;
% + CACHE_UNLOCK();
% return;
% }
is responsible for about half of the performance loss since RELENG_4
for building kernels over nfs (/usr and sys trees on nfs). The kernel
build uses "../../" a lot, and the above change apparently results in
lots of network activity for things that should be cached locally.
Some times for building a RELENG_4 kernel under conditions invariant
except for the host kernel (after "make clean; sleep 2; make depend;
make; make clean; sleep 2; make depend" to warm up caches):
kernel:
RELENG_4 77.51 real 60.62 user 4.36 sys
current.2004.07.01 ~78.5 (lost details)
current.2005.01.01 ~79 (lost details)
current.2005.06.17 82.42 real 62.50 user 4.71 sys
current.2005.06.19 89.53 real 62.18 user 5.44 sys
current.2005.06.17+ ~89.5 (lost details)
.17+ = .17 plus above change
current.2005.06.17+* 86.08 real 62.43 user 5.13 sys
.17+* = .17+ with ../.. in Makefile avoided using a symlink
@ -> <path to sys not using ..>
RELENG_6 91.14 real 62.04 user 5.71 sys
current similar to RELENG_6 (lost details)
The total performance loss is about 18%.
The total performance loss for a local sys tree (/usr still on nfs) is much
smaller (about 4%):
RELENG_4 65.19 real 60.50 user 3.95 sys
current.2005.06.17 67.49 real 62.13 user 4.27 sys
RELENG_6 67.83 real 61.84 user 4.71 sys
current similar to RELENG_6 (lost details)
The nfs performance for building of things that should be entirely
cached locally is very dependent on network latency. Not caching
things very well causes lots of unnecessary network traffic for Getattr
and Lookup. The packets are small, so throughput is unimportant and
latency dominates. For building over nfs without -j, the dead time
(real - user - sys) is almost directly proportional to the latency.
My usual local network has fairly low latency (~100uS unloaded) and
the ~14 seconds dead time in the above is for it. Switching to a 1
Gbps network with lower quality NICs gives an unloaded latency of ~160uS
and a dead time of ~21 seconds. Building with -j helps even for UP,
at the cost of extra CPU, by letting some processes advance using cached
stuff while others are waiting for the network. Building with -j helps
even more on FreeBSD cluster machines, more because they have a much
higher network latency than because they are SMP.
Bruce
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20061006050913.Y5250>