Date: Sat, 13 Oct 2012 09:03:22 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Garrett Wollman <wollman@freebsd.org> Cc: Nikolay Denev <ndenev@gmail.com>, FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: NFS server bottlenecks Message-ID: <611092759.2189637.1350133402953.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <20600.62541.243673.307571@hergotha.csail.mit.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Garrett Wollman wrote:
> <<On Fri, 12 Oct 2012 22:05:54 -0400 (EDT), Rick Macklem
> <rmacklem@uoguelph.ca> said:
>
> > I've attached the patch drc3.patch (it assumes drc2.patch has
> > already been
> > applied) that replaces the single mutex with one for each hash list
> > for tcp. It also increases the size of NFSRVCACHE_HASHSIZE to 200.
>
> I haven't tested this at all, but I think putting all of the mutexes
> in an array like that is likely to cause cache-line ping-ponging. It
> may be better to use a pool mutex, or to put the mutexes adjacent in
> memory to the list heads that they protect.
Well, I'll admit I don't know how to do this.
What the code does need is a "set of mutexes", where any of the mutexes
can be referred to by an "index". I could easily define a structure that
has:
struct nfsrc_hashhead {
struct nfsrvcachehead head;
struct mtx mutex;
} nfsrc_hashhead[NFSRVCACHE_HASHSIZE];
- but all that does is leave a small structure between each "struct mtx" and I
wouldn't have thought that would make much difference. (How big is a typical
hardware cache line these days? I have no idea.)
- I suppose I could "waste space" and define a glob of unused space
between them, like:
struct nfsrc_hashhead {
struct nfsrvcachehead head;
char garbage[N];
struct mtx mutex;
} nfsrc_hashhead[NFSRVCACHE_HASHSIZE];
- If this makes sense, how big should N be? (Somewhat less that the length
of a cache line, I'd guess. It seems that the structure should be at least
a cache line length in size.)
All this seems "kinda hokey" to me and beyond what code at this level should
be worrying about, but I'm game to make changes, if others think it's appropriate.
I've never use mtx_pool(9) mutexes, but it doesn't sound like they would
be the right fit, from reading the man page. (Assuming the mtx_pool_find()
is guaranteed to return the same mutex for the same address passed in as
an argument, it would seem that they would work, since I can pass
&nfsrvcachehead[i] in as the pointer arg to "index" a mutex.)
Hopefully jhb@ can say if using mtx_pool(9) for this would be better than
an array:
struct mtx nfsrc_tcpmtx[NFSRVCACHE_HASHSIZE];
Does anyone conversant with mutexes know what the best coding approach is?
>(But I probably won't be
> able to do the performance testing on any of these for a while. I
> have a server running the "drc2" code but haven't gotten my users to
> put a load on it yet.)
>
No rush. At this point, the earliest I could commit something like this to
head would be December.
rick
ps: I hope John doesn't mind being added to the cc list yet again. It's
just that I suspect he knows a fair bit about mutex implementation
and possible hardware cache line effects.
> -GAWollman
> _______________________________________________
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to
> "freebsd-hackers-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?611092759.2189637.1350133402953.JavaMail.root>