From owner-freebsd-questions@FreeBSD.ORG Thu Nov 13 01:23:16 2014 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A5333532; Thu, 13 Nov 2014 01:23:16 +0000 (UTC) Received: from mail-wi0-x22b.google.com (mail-wi0-x22b.google.com [IPv6:2a00:1450:400c:c05::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 30EB87C5; Thu, 13 Nov 2014 01:23:16 +0000 (UTC) Received: by mail-wi0-f171.google.com with SMTP id r20so6756505wiv.4 for ; Wed, 12 Nov 2014 17:23:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=2NOtbKUkjDHilfwFIRlIRUCK5U5uvCcsQG4pmoTdcJE=; b=uAO5WecCncZQWT/oz2bZ+sZr3zvk7Z9WHu0xrbdJGPRzdoxKca4dAIcffioo02AtEt W/fhSqkNKjhw1sChov8YhI2xOTuOdRohrDm32I3IkmeNtilJI1LbEZ7EV5K6/dP/kTSF t/aq7jLx0Q2lmcGhQ6S+eLJ7+llFJm4U9ruYXtgLG1Qf7RVcLUhLCsrNAk3uX600BJlH OESjQh3oUiDNXZIbhyESG5CLN/JWTh6vEMN9gTwCxaY5j36xVcw0KImhzK6bEs88fiQw uogawVEUeI0Zs46qqMFoK/N6TJiNg4Fzf+2lI10fKwW6T6EvCHK2JLiBuNbAg/oEt9Md hvFQ== MIME-Version: 1.0 X-Received: by 10.180.92.169 with SMTP id cn9mr35049083wib.26.1415841794557; Wed, 12 Nov 2014 17:23:14 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.216.106.136 with HTTP; Wed, 12 Nov 2014 17:23:14 -0800 (PST) In-Reply-To: References: <20141110071353.GO24601@funkthat.com> <20141112084909.GV24601@funkthat.com> Date: Wed, 12 Nov 2014 17:23:14 -0800 X-Google-Sender-Auth: 1goJdnRGgOKKim-A4NWyM3nvJQA Message-ID: Subject: Re: How thread-friendly is kevent? From: Adrian Chadd To: J David Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" , "K. Macy" , "freebsd-questions@freebsd.org" X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Nov 2014 01:23:16 -0000 On 12 November 2014 15:54, J David wrote: > On Wed, Nov 12, 2014 at 5:31 PM, K. Macy wrote: >> Well, this was 8 cores with 45 or so processes. > > Here's the bench test on a 16 (physical) core machine with 24 threads: > > Server thread ends after 853093 events. > Server thread ends after 834169 events. > Server thread ends after 826117 events. > Server thread ends after 824130 events. > Server thread ends after 826998 events. > Server thread ends after 838043 events. > Server thread ends after 859163 events. > Server thread ends after 844742 events. > Server thread ends after 821916 events. > Server thread ends after 807776 events. > Server thread ends after 805819 events. > Server thread ends after 840685 events. > Server thread ends after 847173 events. > Server thread ends after 834560 events. > Server thread ends after 833862 events. > Server thread ends after 842143 events. > Server thread ends after 866425 events. > Server thread ends after 813094 events. > Server thread ends after 833364 events. > Server thread ends after 805277 events. > Server thread ends after 816654 events. > Server thread ends after 834174 events. > Server thread ends after 851322 events. > Server thread ends after 839701 events. > > This particular test is CPU bound, so more threads don't make much > sense, but they also don't make much difference as far as balance. > The results are equally even with 128 threads, though the time wasted > in kqflxw and context switching is *enormously* larger, aptly > demonstrating what John-Mark was talking about. > > So it seems fairly scalable, at least as of 9.3-STABLE. Good work, > whoever it was! Have you done any profiling of the system? At that many CPUs I'd expect you'd see the shared kqueue fileops lock causing some contending. Hopefully the RSS stuff is mature enough in -11 for you to consider using. I'm trying to address both work load balancing / steering as well as CPU affinity scaling issues. I'd love to see this stuff presented at a BSD conference. Are you at all interested in talking some more about it? -adrian