From owner-freebsd-arch@freebsd.org Fri May 19 15:41:16 2017 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 09A46D748D0 for ; Fri, 19 May 2017 15:41:16 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wr0-x234.google.com (mail-wr0-x234.google.com [IPv6:2a00:1450:400c:c0c::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 946A512DB for ; Fri, 19 May 2017 15:41:15 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-wr0-x234.google.com with SMTP id l50so19466799wrc.3 for ; Fri, 19 May 2017 08:41:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=2J1B2j75/riAsiT3zP6FsMCXii2xkDSnlPeiSdco7RA=; b=eT9HyhJ8rcL/ZppW8I4XEuFraxcdpPd8k74ZFrRQMKjBtYrTaUA2sRFdkYzOXEnWu1 MCUh7+alaZAKOe1bcqVVdAPmWN981zZoYl64i1BOaOF50EnEAlmGnIbzSDhuTg76jDs8 ifZ7+Z+uQHUaM5RHiQPWuvG3qdDUbV/f/PYCnh72DmECRbHWmRk0iL29wbUeIWrUT5h9 Q3/v+wW4XiNkB+9rxK5Q/DygDdXUDkFvqWEg9KZunSrzJXVkZQOo0q2O9SmB/7l5WNew cUNF59IQuxYDaZp0gCwJT66Io6vqntIB/jzVEGAavULp/alBUZ3vCB7CftaLb7VWK2s7 Dh0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=2J1B2j75/riAsiT3zP6FsMCXii2xkDSnlPeiSdco7RA=; b=oGMKdbzsrdJ54JWiV/Mr2j1uwSrb6pPhcUC7yPvZ8MGE9Rc55T5dePdXQRaAbcGLfg WxUFqQ9P58UJi7SRUvHbg9CxVvMo+bC8TaycsIbQzxeRQ1nxdO77SF+HMVxeapZBKm7j djCM4AvufVy2XtA9FBUNydOzuaOsYzmyeRvXR7f2+BdnSB/lsVjVhIOOB/+WNA+ScTeY oP2jeL1oV4T+w24ICFlOkbiJuX8UlIoJ4gbCdVuDhsnj6dekVUkvlzFEOZ040Zob6NOy zaC9NJPakg4lD0eIEVsdpVMSaCeXzxWbVGqTycfr4h0blFUPiZ59VAbmhFndMVtGz2d/ N+iQ== X-Gm-Message-State: AODbwcCz0PDT0306ygabXTIxb9eRDb6u34PvB069v7MfnT3gXq4jRL1+ x1NL2a2XO3P7M0cbXrC5zIEd5WlAH2dc X-Received: by 10.223.134.80 with SMTP id 16mr4215670wrw.62.1495208473666; Fri, 19 May 2017 08:41:13 -0700 (PDT) MIME-Version: 1.0 Received: by 10.28.193.134 with HTTP; Fri, 19 May 2017 08:41:13 -0700 (PDT) In-Reply-To: <1914359731.54283525.1495178031163.JavaMail.zimbra@stormshield.eu> References: <1914359731.54283525.1495178031163.JavaMail.zimbra@stormshield.eu> From: Adrian Chadd Date: Fri, 19 May 2017 08:41:13 -0700 Message-ID: Subject: Re: numa and taskqueues To: Emeric POUPON Cc: freebsd-arch Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 May 2017 15:41:16 -0000 Hi, I've been worried about the trend to create ncpu*taskqueue or ndomain*taskqueue for things unless we really need the priority / preemption behaviour. Otherwise we will just end up with a lot of pcpu/pdomain taskqueues that sit idly and/or compete infficiently. Anyway - I think it'd be nice to have domain aware and pcpu aware taskqueues so we can eventually migrate to a taskqueue group model of "one top level things for net processing" for devices to share, etc, etc. But for the short term just prototype it with some thin API in crypto that wraps the taskqueue / kproc work so it gets done, then push that work out for review/evaluation. if it does indeed work the way you intend, we can try to use it as a template for a higher level, shared taskqueue thing. Thanks, -adrian On 19 May 2017 at 00:13, Emeric POUPON wrote: > Hello, > > I have made a review to boost ipsec performance when very few flows are involved: https://reviews.freebsd.org/D10680 (reviews would be appreciated btw!) > The idea is to dispatch the crypto jobs using a taskqueue (with nb threads = nbcpus), details are in the review. > > However, this does not scale well on multi socket architectures (ex: 2*6 cores), a lot of time is wasted in the locks. > > For testing purposes, I created as many taskqueues as domains and I modified the taskqueue_start_threads function to specify a cpuset_t mask. > The idea here is to stay on the same domain to dispatch the crypto jobs and to notify back the crypto users. > This gives quite good performance so it seems to be an promising way. > > Now the question is: how can I make the taskqueues "domain aware"? > Do I have to add some logic in crypto(9) or could this be abstracted in some other part of the kernel? > Another annoying part is the kprocs used by the return queues. We would also have to bind them to a single domain. How? > > What do you think? > > Emeric > _______________________________________________ > freebsd-arch@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org"