Date: Thu, 17 Sep 2020 13:21:39 -0700 From: John Baldwin <jhb@FreeBSD.org> To: =?UTF-8?Q?Olivier_Cochard-Labb=c3=a9?= <olivier@freebsd.org>, "Andrey V. Elsukov" <ae@freebsd.org> Cc: src-committers <src-committers@freebsd.org>, svn-src-all <svn-src-all@freebsd.org>, svn-src-head <svn-src-head@freebsd.org> Subject: Re: svn commit: r365449 - head/sbin/rcorder Message-ID: <1ccf2cf7-eb01-a200-4335-4674a010f8a1@FreeBSD.org> In-Reply-To: <CA%2Bq%2BTcr%2B3Gp4EcmJoLYC0JEOmuVzYdujMQdC4P=n9Zi4jYMmkQ@mail.gmail.com> References: <202009081036.088AaCk8085096@repo.freebsd.org> <CA%2Bq%2BTcr%2B3Gp4EcmJoLYC0JEOmuVzYdujMQdC4P=n9Zi4jYMmkQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 9/17/20 10:49 AM, Olivier Cochard-Labbé wrote: > On Tue, Sep 8, 2020 at 12:36 PM Andrey V. Elsukov <ae@freebsd.org> wrote: > >> Author: ae >> Date: Tue Sep 8 10:36:11 2020 >> New Revision: 365449 >> URL: https://svnweb.freebsd.org/changeset/base/365449 >> >> Log: >> Add a few features to rcorder: >> >> >> > Hi Andrey, > > I've spent some time bisecting an IPSec gateway performance regression on > -head that points to this commit. > > So my bench uses a simple static IPSec aes-gcm-16 setup, and their results > vary a lot between those 2 revisions: > > - r365448: Estimated Equilibrium Ethernet throughput= 1413 Mb/s (maximum > value seen: 1419 Mb/s) > - r365449: Estimated Equilibrium Ethernet throughput= 469 Mb/s (maximum > value seen: 469 Mb/s) > > How could a modification to the rcoder be the source cause of that ? > My rc.conf contains this: > kld_list="aesni" > ipsec_enable="YES" > > So My first thought was that the aesni module wasn't loaded anymore: > Before upgrade: > [root@hp]~# uname -a > FreeBSD hp 13.0-CURRENT FreeBSD 13.0-CURRENT #15 r365448M: Thu Sep 17 > 18:17:58 CEST 2020 olivier@lame4.bsdrp.net:/usr/src/amd64.amd64/sys/amd64 > amd64 > [root@hp]~# kldstat > Id Refs Address Size Name > 1 11 0xffffffff80200000 1ee58b0 kernel > 2 1 0xffffffff82319000 34c8 fdescfs.ko > 3 1 0xffffffff8231d000 a240 aesni.ko > 4 1 0xffffffff82328000 8c98 ioat.ko > 5 1 0xffffffff82331000 e350 ipsec.ko > > Then after upgrade: > > [root@hp]~# uname -a > FreeBSD hp 13.0-CURRENT FreeBSD 13.0-CURRENT #14 r365449M: Thu Sep 17 > 17:01:35 CEST 2020 olivier@lame4.bsdrp.net:/usr/src/amd64.amd64/sys/amd64 > amd64 > [root@hp]~# kldstat > Id Refs Address Size Name > 1 11 0xffffffff80200000 1ee58b0 kernel > 2 1 0xffffffff82319000 34c8 fdescfs.ko > 3 1 0xffffffff8231d000 e350 ipsec.ko > 4 1 0xffffffff8232c000 a240 aesni.ko > 5 1 0xffffffff82337000 8c98 ioat.ko > > => aesni.ko is correctly loaded, so it is not the problem, but notice the > order of the kernel modules that have changed. > Could be this the source of the problem ? Let's try: > > [root@hp]~# service ipsec stop > Clearing ipsec manual keys/policies. > [root@hp]~# kld > kldconfig kldload kldstat kldunload kldxref > [root@hp]~# kldunload ioat > [root@hp]~# kldunload aesni > [root@hp]~# kldunload ipsec > [root@hp]~# kldload aesni > [root@hp]~# kldload ipsec > [root@hp]~# service ipsec start > Installing ipsec manual keys/policies. > [root@hp]~# kldstat > Id Refs Address Size Name > 1 11 0xffffffff80200000 1ee58b0 kernel > 2 1 0xffffffff82319000 34c8 fdescfs.ko > 5 1 0xffffffff82337000 8c98 ioat.ko > 6 1 0xffffffff8231d000 a240 aesni.ko > 7 1 0xffffffff82328000 e350 ipsec.ko > > And after that the IPSec bench results are back to their previous value :-) > So rcorder needs to load aesni before ipsec. I don't think the issue is with rcorder though. I think the reason the ordering matters warrants further investigation. Is aesni not getting used when ipsec is loaded first? You can use dtrace with the script at https://github.com/bsdjhb/kdbg/blob/master/dtrace/crypto_drivers.d to see which driver is being used. Hmm, the crypto driver gets selected when keys for SAs are set, so if /etc/rc.d/ipsec is configuring SAs, then having it run before aesni is loaded would indeed cause this. However, that still isn't an issue with this commit, per se, it just exposed the lack of an explicit ordering requirement of /etc/rc.d/ipsec after loading aesni. The problem though is that kld_list is pretty generic, so it's hard to know when /etc/rc.d/kld should run. Possibly it should even run multiple times where subsequent runs try to load any modules not loaded by the previous runs? I suspect btw that you could just do 'sh /etc/rc.d/ipsec restart' post-boot without unloading any modules which would also fix your benchmark. I think long term we want OCF's notions of sessions to be a bit more fluid such that "client" sessions for things like GELI and IPSec can be backed by one or more "driver" sessions (including "driver" sessions coming and going as devices come and go). That's a fair bit more work however. -- John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1ccf2cf7-eb01-a200-4335-4674a010f8a1>