From owner-freebsd-arch@freebsd.org  Fri May 19 07:21:14 2017
Return-Path: <owner-freebsd-arch@freebsd.org>
Delivered-To: freebsd-arch@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9F8AD73658
 for <freebsd-arch@mailman.ysv.freebsd.org>;
 Fri, 19 May 2017 07:21:14 +0000 (UTC)
 (envelope-from emeric.poupon@stormshield.eu)
Received: from work.stormshield.eu (gwlille.netasq.com [91.212.116.1])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 9433D1C02
 for <freebsd-arch@freebsd.org>; Fri, 19 May 2017 07:21:13 +0000 (UTC)
 (envelope-from emeric.poupon@stormshield.eu)
Received: from work.stormshield.eu (localhost [127.0.0.1])
 by work.stormshield.eu (Postfix) with ESMTPS id 9BACA37610B3
 for <freebsd-arch@freebsd.org>; Fri, 19 May 2017 09:13:51 +0200 (CEST)
Received: from localhost (localhost [127.0.0.1])
 by work.stormshield.eu (Postfix) with ESMTP id 8E43C3761387
 for <freebsd-arch@freebsd.org>; Fri, 19 May 2017 09:13:51 +0200 (CEST)
Received: from work.stormshield.eu ([127.0.0.1])
 by localhost (work.stormshield.eu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id yZihUApLJy7e for <freebsd-arch@freebsd.org>;
 Fri, 19 May 2017 09:13:51 +0200 (CEST)
Received: from work.stormshield.eu (localhost [127.0.0.1])
 by work.stormshield.eu (Postfix) with ESMTP id 7B91A37610B3
 for <freebsd-arch@freebsd.org>; Fri, 19 May 2017 09:13:51 +0200 (CEST)
Date: Fri, 19 May 2017 09:13:51 +0200 (CEST)
From: Emeric POUPON <emeric.poupon@stormshield.eu>
To: freebsd-arch <freebsd-arch@freebsd.org>
Message-ID: <1914359731.54283525.1495178031163.JavaMail.zimbra@stormshield.eu>
Subject: numa and taskqueues
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Thread-Topic: numa and taskqueues
Thread-Index: OmwNiBQRFi7BTXWKFdrri3vfOJZN5w==
X-BeenThere: freebsd-arch@freebsd.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: Discussion related to FreeBSD architecture <freebsd-arch.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-arch>,
 <mailto:freebsd-arch-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-arch/>
List-Post: <mailto:freebsd-arch@freebsd.org>
List-Help: <mailto:freebsd-arch-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-arch>,
 <mailto:freebsd-arch-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 19 May 2017 07:21:14 -0000

Hello,

I have made a review to boost ipsec performance when very few flows are involved: https://reviews.freebsd.org/D10680 (reviews would be appreciated btw!)
The idea is to dispatch the crypto jobs using a taskqueue (with nb threads = nbcpus), details are in the review.

However, this does not scale well on multi socket architectures (ex: 2*6 cores), a lot of time is wasted in the locks.

For testing purposes, I created as many taskqueues as domains and I modified the taskqueue_start_threads function to specify a cpuset_t mask.
The idea here is to stay on the same domain to dispatch the crypto jobs and to notify back the crypto users.
This gives quite good performance so it seems to be an promising way.

Now the question is: how can I make the taskqueues "domain aware"?
Do I have to add some logic in crypto(9) or could this be abstracted in some other part of the kernel?
Another annoying part is the kprocs used by the return queues. We would also have to bind them to a single domain. How?

What do you think?

Emeric