From owner-freebsd-arch@FreeBSD.ORG Sat Dec 28 06:02:52 2013 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D30D7D47; Sat, 28 Dec 2013 06:02:52 +0000 (UTC) Received: from mail-qa0-x232.google.com (mail-qa0-x232.google.com [IPv6:2607:f8b0:400d:c00::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 810951140; Sat, 28 Dec 2013 06:02:52 +0000 (UTC) Received: by mail-qa0-f50.google.com with SMTP id i13so9196666qae.2 for ; Fri, 27 Dec 2013 22:02:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=jVzBtw8xQr0UvwuMHcMWKtao3G8Bl8LWXg272IEiMws=; b=dy4IDCgJ4S8p1E126oVLpCXCtJ3ludivy3BorxCuhTS70p27k4VXS6B9f2bHqitdz8 KNGUg/8xz8Rrk9fbEOb87ec3vz3kLca5I3h5WEF6KmUHgWH89C5ibOfJjS8dElUcupcZ ojcFsMJMhKHu50qA29QwhgpII2di65+VbtV7E+Z5MDbzQuypeC3hubk1xPSV3dJcbkkl RsJORnKg9s9+TR52aFfqfVg2rO6aAZ1EPaa5cyxa50KCKwfG98sXXsedHyzxYLw9kTgJ YQyEq3GNc9Tcn4FV7x4pDuHPlhAO/t6ta36tBXqPGKmrbrg8ghUnKsdmMSUy+iPbyGdj xc8Q== MIME-Version: 1.0 X-Received: by 10.224.13.141 with SMTP id c13mr79106339qaa.76.1388210570677; Fri, 27 Dec 2013 22:02:50 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.224.52.8 with HTTP; Fri, 27 Dec 2013 22:02:50 -0800 (PST) In-Reply-To: <20131228025948.GP99167@funkthat.com> References: <20131228025948.GP99167@funkthat.com> Date: Fri, 27 Dec 2013 22:02:50 -0800 X-Google-Sender-Auth: C4x7mvGJskb1tMVum5DvK6w6K5s Message-ID: Subject: Re: Default knote hash table size is too .. small From: Adrian Chadd To: Adrian Chadd , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Dec 2013 06:02:52 -0000 On 27 December 2013 18:59, John-Mark Gurney wrote: > Adrian Chadd wrote this message on Fri, Dec 27, 2013 at 14:41 -0800: >> The default knote hash table size (KN_HASHSIZE) is 64. When doing >> dirty things with lots of sockets (say, 64k and more) that involve >> plenty of knote insert/remove (ie, oneshot events - think posix AIO >> and my upcoming kqueue sendfile notification stuff) it consumes stupid >> amounts of CPU. > > Just for the record, KN_HASHSIZE is not used by sockets, since they > have an fd to do the look up... It's AIO that's probably the one > using the CPU time, but definately not sockets.. > > The notes put on the hash are _AIO/_LIO, _PROC, _TIMER, _USER, _FS, > and _SIGNAL... > > So, should we look at another data structure like RB trees for this > then? At least RB trees would scale better, or we should look at > making the hash table growable? It looks like there is support to > grow it, since kq_knhas and kq_knhashmask are both local to kq... > > From a quick look, we can resize the hash only by holding the KQ > lock... > >> I'd like to make this a tunable so people like Adrian at Netflix can >> bump this to higher values (say 32k) but people like Adrian at >> Embedded can bump this to lower values (say 64.) > > Yeh, a smaller default, say of 8 or 16 would make sense for embedded > since there aren't many non-fd consumers of kq out there... This would > mean each kq only consumes 32 or 64 bytes instead of 2k... Heck, > even for normal systems, a default of 8 could/would make sense... > > Hmm... maybe we should get some stats on how many notes use the hash? It again is highly application dependent. If you're just doing socket IO then you'r elikely fine. If you're also doing other things in high volumes and lots of pending operations (eg posix aio, or as you said timers, etc) then it could get highly crappy very quickly. -a