From owner-freebsd-geom@freebsd.org Tue Nov 28 09:37:01 2017 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 14D19DFF4CD for ; Tue, 28 Nov 2017 09:37:01 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from constantine.ingresso.co.uk (unknown [IPv6:2a02:b90:3002:411::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D6E4D7D917 for ; Tue, 28 Nov 2017 09:37:00 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from cpc73666-dals20-2-0-cust303.20-2.cable.virginm.net ([82.47.237.48] helo=foula.drayhouse.twisted.org.uk) by constantine.ingresso.co.uk with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.89 (FreeBSD)) (envelope-from ) id 1eJcK2-000FDZ-3c for freebsd-geom@freebsd.org; Tue, 28 Nov 2017 09:36:58 +0000 Subject: Re: GELI strangeness with gstat To: freebsd-geom@freebsd.org References: <20171128070811.GZ42467@funkthat.com> From: Pete French Message-ID: <11df15ff-7a30-3698-ff3b-ffced80a78c8@ingresso.co.uk> Date: Tue, 28 Nov 2017 09:37:03 +0000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <20171128070811.GZ42467@funkthat.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Nov 2017 09:37:01 -0000 Tnaks for the reply.... > If you just did a kldload aesni, but did not reattach the geli device, > then you are still using software encryption... You should see something > like this: > GEOM_ELI: Device gpt/werner.eli created. > GEOM_ELI: Encryption: AES-XTS 256 > GEOM_ELI: Crypto: hardware > > if you are using AES-NI... Yes, thats exactly what I see - I rebooted the system (had to as I was enabling AES_NI in the BIOS) and am loading the crypto and aesni modules at boot time form loader.conf. I see this in dmesg: Enter passphrase for ada0p4: GEOM_ELI: Device ada0p4.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: hardware GEOM_ELI: Device ada1p4.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: hardware and 'geli list' says 'Crypto: hardware' in its output > Also, what version of FreeBSD are you using? If you're using pre-10.0-R, > the performance increase from using AES-NI is only marginal... Am using 11.1-STABLE from mid June - is it worth getting the last few weeks of updates ? I wouldnt have thought so as I havent see any crypto chnages go past, but I can give it a go... > The above does make it look like you're disks are CPU bound by the > encryption... Indeed. But the odd this is it only started happening *after* I upgraded the machine to much faster CPU's. I am suspecting (hoping!) its actually a statistical anomaly in gstat and not a real effect. > To get a better idea of what is happening, you can run top -S to see > how much CPU the geli threads are using. Ah, good idea. Hmmmm... that shows up 12 threads, each using about 0.2% CPU. > Also, how many eli volumes do you have? In my case, I have 13... In > order to reduce the load on the scheduler, I have: > kern.geom.eli.threads="1" This is interesting - my understanding was that with hardware encryption that setting did nothing. From the manpage: kern.geom.eli.threads: 0 Specifies how many kernel threads should be used for doing software cryptography. Its purpose is to increase performance on SMP systems. If hardware acceleration is available, only one thread will be started. If set to 0, CPU-bound thread will be started for every active CPU. So my expectation would be that I only get 1 thread - but actuyally I see 12 in 'top -S'. Again, is that a bug or out of date documentation ? I only have two volumes, and am runnign ZFS over the top with compresison enabled, which I think only uses a single write thread, so maybe reducing that would help. But theres still a very large discrepancy between what all of my other jetricsare showing me and gstat. Te underlying discs are SSD's by the way, so I am fairly sure thats not a bottleneck (and gstats dosnt show the undrelying drive busy). -pete.