From owner-freebsd-arch@FreeBSD.ORG Fri Dec 27 23:51:49 2013 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AA545F15; Fri, 27 Dec 2013 23:51:49 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 951A81B33; Fri, 27 Dec 2013 23:51:49 +0000 (UTC) Received: from Alfreds-MacBook-Air.local (50-204-88-5-static.hfc.comcastbusiness.net [50.204.88.5]) by elvis.mu.org (Postfix) with ESMTPSA id ED5E61A3C1F; Fri, 27 Dec 2013 15:51:48 -0800 (PST) Message-ID: <52BE1293.3050305@freebsd.org> Date: Fri, 27 Dec 2013 15:51:47 -0800 From: Alfred Perlstein Organization: FreeBSD User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: Default knote hash table size is too .. small References: <52BE0D40.5020304@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Dec 2013 23:51:49 -0000 On 12/27/13, 3:39 PM, Adrian Chadd wrote: > On 27 December 2013 15:29, Alfred Perlstein wrote: > >> Cool! What about auto-tune to maxfiles? > Well, ideally it'd be tunable per actual kqueue rather than a global. > That way some crazy ioctl could be done to set the knote hash size per > queue. > > I'm just worried that for some people doing say, high throughput UDP > with one connected socket, it's just wasted RAM. But for mad parallel > TCP the hash has to be bigger. But then if yo'ure doing say, 30 worker > processes versus 1000 sockets each versus say, 8 worker processes > doing 4000 sockets each, you need a different hash size. > > See? It's not as easy as tune to maxfiles. the only thing that is that > simple is "can we massively overspecify it." :-) > > I'll do a followup at some point to allow an ioctl to set/get it on a > per-kqfd basis. Then we won't need the build limit. That's only partially true, on a machine with let's say 20GB ram, then it's probably perfectly fine to autotune the kqueue hashes to be 128, 256 or even 512 so that out of the box performance on these machines is better. There's memory so "waste it" a person who truly knows what they are doing can save a few hundred bytes by setting the tunable lower. In all honesty, aren't kqueues used somewhat sparingly? Truly very little harm can come from raising it on large memory machines. In your specific examples, 30 or 8 worker processes, how many kqueues will be active on the box (relative to other kernel data structures such as sockets?). I could be wrong, there may be some use cases of 1000s of kqueues per thread, but I'm having a hard time wrapping my head around it. Another question, does the kqueue locking model allow for rehashing once there is something like 4*KN_HASHSIZE entries? Can it be dynamically set? Anyhow, I'm not suggesting that you shouldn't just add the tunable and be on your way, but more asking if maybe *I* should look at making it dynamically scale. -Alfred