From owner-freebsd-arch@freebsd.org  Tue May 30 14:54:10 2017
Return-Path: <owner-freebsd-arch@freebsd.org>
Delivered-To: freebsd-arch@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 296FAB7ECCB
 for <freebsd-arch@mailman.ysv.freebsd.org>;
 Tue, 30 May 2017 14:54:10 +0000 (UTC)
 (envelope-from emeric.poupon@stormshield.eu)
Received: from work.stormshield.eu (gwlille.netasq.com [91.212.116.1])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id E359F6528A;
 Tue, 30 May 2017 14:54:09 +0000 (UTC)
 (envelope-from emeric.poupon@stormshield.eu)
Received: from work.stormshield.eu (localhost [127.0.0.1])
 by work.stormshield.eu (Postfix) with ESMTPS id B1AB9376179E;
 Tue, 30 May 2017 16:46:01 +0200 (CEST)
Received: from localhost (localhost [127.0.0.1])
 by work.stormshield.eu (Postfix) with ESMTP id 927043761794;
 Tue, 30 May 2017 16:46:01 +0200 (CEST)
Received: from work.stormshield.eu ([127.0.0.1])
 by localhost (work.stormshield.eu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id WWe7UUK49nrL; Tue, 30 May 2017 16:46:01 +0200 (CEST)
Received: from work.stormshield.eu (localhost [127.0.0.1])
 by work.stormshield.eu (Postfix) with ESMTP id 7A2CC376176C;
 Tue, 30 May 2017 16:46:01 +0200 (CEST)
Date: Tue, 30 May 2017 16:46:01 +0200 (CEST)
From: Emeric POUPON <emeric.poupon@stormshield.eu>
To: Adrian Chadd <adrian@freebsd.org>
Cc: freebsd-arch <freebsd-arch@freebsd.org>
Message-ID: <608664209.55736023.1496155561181.JavaMail.zimbra@stormshield.eu>
In-Reply-To: <CAJ-Vmo=h3ASXiFWYT1E=xHQ+zZ_5+027dXibbPezj2CcHvGxVg@mail.gmail.com>
References: <1914359731.54283525.1495178031163.JavaMail.zimbra@stormshield.eu>
 <CAJ-Vmo=6bpo1Yu6XosN3BiYOakjeXS8J7wenfubzkWz2SxXR1g@mail.gmail.com>
 <816581118.55670987.1496141816904.JavaMail.zimbra@stormshield.eu>
 <CAJ-Vmo=h3ASXiFWYT1E=xHQ+zZ_5+027dXibbPezj2CcHvGxVg@mail.gmail.com>
Subject: Re: numa and taskqueues
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Thread-Topic: numa and taskqueues
Thread-Index: aT1hZYkNVS61k+uN8YKtzmebjUGFoQ==
X-BeenThere: freebsd-arch@freebsd.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: Discussion related to FreeBSD architecture <freebsd-arch.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-arch>,
 <mailto:freebsd-arch-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-arch/>
List-Post: <mailto:freebsd-arch@freebsd.org>
List-Help: <mailto:freebsd-arch-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-arch>,
 <mailto:freebsd-arch-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 30 May 2017 14:54:10 -0000

Hi,

> 
>> 2/ about https://reviews.freebsd.org/D10680, I think it would be great to have
>> this commited as a first step.
>> Since it seems to be stuck, maybe I can add more people on this. Any suggestion?
> 
> Well, what's with the ~ 8% performance decrease? Do you know what's
> going on? For a "we're parallelising IPSEC operations", seeing it get
> slower with more flows is a bit concerning.
> 
> Thanks,
> 

Actually, there is a performance boost only when few flows are involved.
That's why this is not activated by default and a sysctl is here to enable the feature.

To sum up, the more different flows you process (both ciphered and unciphered), the more network queues are hit and the more CPU units are triggered from ipsec.
In this case, we indeed notice a loss, certainly due to the extra queing/reordering performed.