Date: Fri, 27 Dec 2013 15:51:47 -0800 From: Alfred Perlstein <alfred@freebsd.org> To: Adrian Chadd <adrian@freebsd.org> Cc: "freebsd-arch@freebsd.org" <freebsd-arch@freebsd.org> Subject: Re: Default knote hash table size is too .. small Message-ID: <52BE1293.3050305@freebsd.org> In-Reply-To: <CAJ-Vmon5xBnJ%2BJHAJ%2B7LNrrKk0Rzy3SynpXc-5FLYe6Y7W9zxQ@mail.gmail.com> References: <CAJ-VmomteHhbNv2pLX1cma74hExbtQKhxiRd3D653abneuJwhg@mail.gmail.com> <52BE0D40.5020304@freebsd.org> <CAJ-Vmon5xBnJ%2BJHAJ%2B7LNrrKk0Rzy3SynpXc-5FLYe6Y7W9zxQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 12/27/13, 3:39 PM, Adrian Chadd wrote: > On 27 December 2013 15:29, Alfred Perlstein <alfred@freebsd.org> wrote: > >> Cool! What about auto-tune to maxfiles? > Well, ideally it'd be tunable per actual kqueue rather than a global. > That way some crazy ioctl could be done to set the knote hash size per > queue. > > I'm just worried that for some people doing say, high throughput UDP > with one connected socket, it's just wasted RAM. But for mad parallel > TCP the hash has to be bigger. But then if yo'ure doing say, 30 worker > processes versus 1000 sockets each versus say, 8 worker processes > doing 4000 sockets each, you need a different hash size. > > See? It's not as easy as tune to maxfiles. the only thing that is that > simple is "can we massively overspecify it." :-) > > I'll do a followup at some point to allow an ioctl to set/get it on a > per-kqfd basis. Then we won't need the build limit. That's only partially true, on a machine with let's say 20GB ram, then it's probably perfectly fine to autotune the kqueue hashes to be 128, 256 or even 512 so that out of the box performance on these machines is better. There's memory so "waste it" a person who truly knows what they are doing can save a few hundred bytes by setting the tunable lower. In all honesty, aren't kqueues used somewhat sparingly? Truly very little harm can come from raising it on large memory machines. In your specific examples, 30 or 8 worker processes, how many kqueues will be active on the box (relative to other kernel data structures such as sockets?). I could be wrong, there may be some use cases of 1000s of kqueues per thread, but I'm having a hard time wrapping my head around it. Another question, does the kqueue locking model allow for rehashing once there is something like 4*KN_HASHSIZE entries? Can it be dynamically set? Anyhow, I'm not suggesting that you shouldn't just add the tunable and be on your way, but more asking if maybe *I* should look at making it dynamically scale. -Alfred
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?52BE1293.3050305>