From owner-freebsd-geom@freebsd.org Tue Jan 2 19:46:10 2018 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ECDFFEB579A for ; Tue, 2 Jan 2018 19:46:10 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C6BD96F669 for ; Tue, 2 Jan 2018 19:46:10 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id w02JbXqi003075 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 2 Jan 2018 11:37:33 -0800 (PST) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id w02JbWkF003074; Tue, 2 Jan 2018 11:37:32 -0800 (PST) (envelope-from jmg) Date: Tue, 2 Jan 2018 11:37:32 -0800 From: John-Mark Gurney To: Pete French Cc: freebsd-geom@freebsd.org Subject: Re: GELI strangeness with gstat Message-ID: <20180102193732.GE75576@funkthat.com> Mail-Followup-To: Pete French , freebsd-geom@freebsd.org References: <20171128070811.GZ42467@funkthat.com> <11df15ff-7a30-3698-ff3b-ffced80a78c8@ingresso.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <11df15ff-7a30-3698-ff3b-ffced80a78c8@ingresso.co.uk> X-Operating-System: FreeBSD 11.0-RELEASE-p7 amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Tue, 02 Jan 2018 11:37:33 -0800 (PST) X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jan 2018 19:46:11 -0000 Pete French wrote this message on Tue, Nov 28, 2017 at 09:37 +0000: > Tnaks for the reply.... > > > If you just did a kldload aesni, but did not reattach the geli device, > > then you are still using software encryption... You should see something > > like this: > > GEOM_ELI: Device gpt/werner.eli created. > > GEOM_ELI: Encryption: AES-XTS 256 > > GEOM_ELI: Crypto: hardware > > > > if you are using AES-NI... > > Yes, thats exactly what I see - I rebooted the system (had to as I was > enabling AES_NI in the BIOS) and am loading the crypto and aesni modules > at boot time form loader.conf. I see this in dmesg: > > Enter passphrase for ada0p4: > GEOM_ELI: Device ada0p4.eli created. > GEOM_ELI: Encryption: AES-XTS 128 > GEOM_ELI: Crypto: hardware > GEOM_ELI: Device ada1p4.eli created. > GEOM_ELI: Encryption: AES-XTS 128 > GEOM_ELI: Crypto: hardware > > and 'geli list' says 'Crypto: hardware' in its output > > > Also, what version of FreeBSD are you using? If you're using pre-10.0-R, > > the performance increase from using AES-NI is only marginal... > > Am using 11.1-STABLE from mid June - is it worth getting the last few > weeks of updates ? I wouldnt have thought so as I havent see any crypto > chnages go past, but I can give it a go... Nope. Not much has changed in a while.. > > The above does make it look like you're disks are CPU bound by the > > encryption... > > Indeed. But the odd this is it only started happening *after* I upgraded > the machine to much faster CPU's. I am suspecting (hoping!) its actually > a statistical anomaly in gstat and not a real effect. > > > To get a better idea of what is happening, you can run top -S to see > > how much CPU the geli threads are using. > > Ah, good idea. > > Hmmmm... that shows up 12 threads, each using about 0.2% CPU. That doesn't make sense then.. That is about what I would expect... Also, have you tried to benchmark w/o geli? do you get similar performance? > > Also, how many eli volumes do you have? In my case, I have 13... In > > order to reduce the load on the scheduler, I have: > > kern.geom.eli.threads="1" > > This is interesting - my understanding was that with hardware encryption > that setting did nothing. From the manpage: > > kern.geom.eli.threads: 0 > Specifies how many kernel threads should be used for doing software > cryptography. Its purpose is to increase performance on SMP systems. If > hardware acceleration is available, only one thread will be started. If > set to 0, CPU-bound thread will be started for every active CPU. This is a slight issue w/ the crypto framework.. When AES-NI was imported, it was declared a "hardware" device even though it's really a software device.. I believe to allow it priority over the software devices which are often used as a last resort... Even though it's listed as software only, it does effect things for AES-NI... > So my expectation would be that I only get 1 thread - but actuyally I > see 12 in 'top -S'. Again, is that a bug or out of date documentation ? I don't know, as I don't see any output... It should look something like: 99 root 20 - 0K 16K geli:w 0 15:59 0.02% g_eli[0] gpt/stanle and if you see numbers other than 0 in the brackets, then that setting isn't working.. The setting needs to be set before the first attach.. As my root volume is geli, I have to set it in loader.conf, otherwise sysctl.conf would set it too late... > I only have two volumes, and am runnign ZFS over the top with > compresison enabled, which I think only uses a single write thread, so > maybe reducing that would help. But theres still a very large > discrepancy between what all of my other jetricsare showing me and gstat. > > Te underlying discs are SSD's by the way, so I am fairly sure thats not > a bottleneck (and gstats dosnt show the undrelying drive busy). Yeah, it looks like you have geli setup resonably, so I'm not sure what to say at this point.. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."