From owner-freebsd-arch@FreeBSD.ORG Sat Dec 28 02:59:50 2013 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EBB537CB; Sat, 28 Dec 2013 02:59:50 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A741E16FC; Sat, 28 Dec 2013 02:59:50 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id rBS2xmAM008362 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 27 Dec 2013 18:59:49 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id rBS2xmMs008361; Fri, 27 Dec 2013 18:59:48 -0800 (PST) (envelope-from jmg) Date: Fri, 27 Dec 2013 18:59:48 -0800 From: John-Mark Gurney To: Adrian Chadd Subject: Re: Default knote hash table size is too .. small Message-ID: <20131228025948.GP99167@funkthat.com> Mail-Followup-To: Adrian Chadd , "freebsd-arch@freebsd.org" References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Fri, 27 Dec 2013 18:59:49 -0800 (PST) Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Dec 2013 02:59:51 -0000 Adrian Chadd wrote this message on Fri, Dec 27, 2013 at 14:41 -0800: > The default knote hash table size (KN_HASHSIZE) is 64. When doing > dirty things with lots of sockets (say, 64k and more) that involve > plenty of knote insert/remove (ie, oneshot events - think posix AIO > and my upcoming kqueue sendfile notification stuff) it consumes stupid > amounts of CPU. Just for the record, KN_HASHSIZE is not used by sockets, since they have an fd to do the look up... It's AIO that's probably the one using the CPU time, but definately not sockets.. The notes put on the hash are _AIO/_LIO, _PROC, _TIMER, _USER, _FS, and _SIGNAL... So, should we look at another data structure like RB trees for this then? At least RB trees would scale better, or we should look at making the hash table growable? It looks like there is support to grow it, since kq_knhas and kq_knhashmask are both local to kq... >From a quick look, we can resize the hash only by holding the KQ lock... > I'd like to make this a tunable so people like Adrian at Netflix can > bump this to higher values (say 32k) but people like Adrian at > Embedded can bump this to lower values (say 64.) Yeh, a smaller default, say of 8 or 16 would make sense for embedded since there aren't many non-fd consumers of kq out there... This would mean each kq only consumes 32 or 64 bytes instead of 2k... Heck, even for normal systems, a default of 8 could/would make sense... Hmm... maybe we should get some stats on how many notes use the hash? -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."