Date: Wed, 3 Oct 2012 16:59:16 -0400 From: Garrett Wollman <wollman@freebsd.org> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: freebsd-fs@freebsd.org, rmacklem@freebsd.org, hackers@freebsd.org Subject: Re: NFS server bottlenecks Message-ID: <20588.42788.103863.179701@hergotha.csail.mit.edu> In-Reply-To: <1571646304.1630985.1349270466529.JavaMail.root@erie.cs.uoguelph.ca> References: <20587.47363.504969.926603@hergotha.csail.mit.edu> <1571646304.1630985.1349270466529.JavaMail.root@erie.cs.uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
<<On Wed, 3 Oct 2012 09:21:06 -0400 (EDT), Rick Macklem <rmacklem@uoguelph.ca> said: >> Simple: just use a sepatate mutex for each list that a cache entry >> is on, rather than a global lock for everything. This would reduce >> the mutex contention, but I'm not sure how significantly since I >> don't have the means to measure it yet. >> > Well, since the cache trimming is removing entries from the lists, I don't > see how that can be done with a global lock for list updates? Well, the global lock is what we have now, but the cache trimming process only looks at one list at a time, so not locking the list that isn't being iterated over probably wouldn't hurt, unless there's some mechanism (that I didn't see) for entries to move from one list to another. Note that I'm considering each hash bucket a separate "list". (One issue to worry about in that case would be cache-line contention in the array of hash buckets; perhaps NFSRVCACHE_HASHSIZE ought to be increased to reduce that.) > Only doing it once/sec would result in a very large cache when bursts of > traffic arrives. My servers have 96 GB of memory so that's not a big deal for me. > I'm not sure I see why doing it as a separate thread will improve things. > There are N nfsd threads already (N can be bumped up to 256 if you wish) > and having a bunch more "cache trimming threads" would just increase > contention, wouldn't it? Only one cache-trimming thread. The cache trim holds the (global) mutex for much longer than any individual nfsd service thread has any need to, and having N threads doing that in parallel is why it's so heavily contended. If there's only one thread doing the trim, then the nfsd service threads aren't spending time either contending on the mutex (it will be held less frequently and for shorter periods). > The only negative effect I can think of w.r.t. having the nfsd > threads doing it would be a (I believe negligible) increase in RPC > response times (the time the nfsd thread spends trimming the cache). > As noted, I think this time would be negligible compared to disk I/O > and network transit times in the total RPC response time? With adaptive mutexes, many CPUs, lots of in-memory cache, and 10G network connectivity, spinning on a contended mutex takes a significant amount of CPU time. (For the current design of the NFS server, it may actually be a win to turn off adaptive mutexes -- I should give that a try once I'm able to do more testing.) -GAWollman
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20588.42788.103863.179701>