Date: Tue, 2 Jan 2018 11:37:32 -0800 From: John-Mark Gurney <jmg@funkthat.com> To: Pete French <petefrench@ingresso.co.uk> Cc: freebsd-geom@freebsd.org Subject: Re: GELI strangeness with gstat Message-ID: <20180102193732.GE75576@funkthat.com> In-Reply-To: <11df15ff-7a30-3698-ff3b-ffced80a78c8@ingresso.co.uk> References: <E1eJJ9M-000FSw-GZ@dilbert.ingresso.co.uk> <20171128070811.GZ42467@funkthat.com> <11df15ff-7a30-3698-ff3b-ffced80a78c8@ingresso.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
Pete French wrote this message on Tue, Nov 28, 2017 at 09:37 +0000:
> Tnaks for the reply....
>
> > If you just did a kldload aesni, but did not reattach the geli device,
> > then you are still using software encryption... You should see something
> > like this:
> > GEOM_ELI: Device gpt/werner.eli created.
> > GEOM_ELI: Encryption: AES-XTS 256
> > GEOM_ELI: Crypto: hardware
> >
> > if you are using AES-NI...
>
> Yes, thats exactly what I see - I rebooted the system (had to as I was
> enabling AES_NI in the BIOS) and am loading the crypto and aesni modules
> at boot time form loader.conf. I see this in dmesg:
>
> Enter passphrase for ada0p4:
> GEOM_ELI: Device ada0p4.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI: Crypto: hardware
> GEOM_ELI: Device ada1p4.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI: Crypto: hardware
>
> and 'geli list' says 'Crypto: hardware' in its output
>
> > Also, what version of FreeBSD are you using? If you're using pre-10.0-R,
> > the performance increase from using AES-NI is only marginal...
>
> Am using 11.1-STABLE from mid June - is it worth getting the last few
> weeks of updates ? I wouldnt have thought so as I havent see any crypto
> chnages go past, but I can give it a go...
Nope. Not much has changed in a while..
> > The above does make it look like you're disks are CPU bound by the
> > encryption...
>
> Indeed. But the odd this is it only started happening *after* I upgraded
> the machine to much faster CPU's. I am suspecting (hoping!) its actually
> a statistical anomaly in gstat and not a real effect.
>
> > To get a better idea of what is happening, you can run top -S to see
> > how much CPU the geli threads are using.
>
> Ah, good idea.
>
> Hmmmm... that shows up 12 threads, each using about 0.2% CPU.
That doesn't make sense then.. That is about what I would expect...
Also, have you tried to benchmark w/o geli? do you get similar
performance?
> > Also, how many eli volumes do you have? In my case, I have 13... In
> > order to reduce the load on the scheduler, I have:
> > kern.geom.eli.threads="1"
>
> This is interesting - my understanding was that with hardware encryption
> that setting did nothing. From the manpage:
>
> kern.geom.eli.threads: 0
> Specifies how many kernel threads should be used for doing software
> cryptography. Its purpose is to increase performance on SMP systems. If
> hardware acceleration is available, only one thread will be started. If
> set to 0, CPU-bound thread will be started for every active CPU.
This is a slight issue w/ the crypto framework.. When AES-NI was imported,
it was declared a "hardware" device even though it's really a software
device.. I believe to allow it priority over the software devices which
are often used as a last resort... Even though it's listed as software
only, it does effect things for AES-NI...
> So my expectation would be that I only get 1 thread - but actuyally I
> see 12 in 'top -S'. Again, is that a bug or out of date documentation ?
I don't know, as I don't see any output... It should look something like:
99 root 20 - 0K 16K geli:w 0 15:59 0.02% g_eli[0] gpt/stanle
and if you see numbers other than 0 in the brackets, then that setting
isn't working..
The setting needs to be set before the first attach.. As my root volume
is geli, I have to set it in loader.conf, otherwise sysctl.conf would
set it too late...
> I only have two volumes, and am runnign ZFS over the top with
> compresison enabled, which I think only uses a single write thread, so
> maybe reducing that would help. But theres still a very large
> discrepancy between what all of my other jetricsare showing me and gstat.
>
> Te underlying discs are SSD's by the way, so I am fairly sure thats not
> a bottleneck (and gstats dosnt show the undrelying drive busy).
Yeah, it looks like you have geli setup resonably, so I'm not sure what
to say at this point..
--
John-Mark Gurney Voice: +1 415 225 5579
"All that I will do, has been done, All that I have, has not."
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20180102193732.GE75576>