From owner-freebsd-geom@freebsd.org Sat Jul 4 23:16:55 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5E1B535AE1E for ; Sat, 4 Jul 2020 23:16:55 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49znlp6wCHz4FVn; Sat, 4 Jul 2020 23:16:54 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id 064NGgMI090018 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 4 Jul 2020 16:16:42 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id 064NGgoi090017; Sat, 4 Jul 2020 16:16:42 -0700 (PDT) (envelope-from jmg) Date: Sat, 4 Jul 2020 16:16:42 -0700 From: John-Mark Gurney To: Alan Somers Cc: Pawe?? Jakub Dawidek , freebsd-geom@freebsd.org Subject: Re: Single-threaded bottleneck in geli Message-ID: <20200704231642.GU4213@funkthat.com> Mail-Followup-To: Alan Somers , Pawe?? Jakub Dawidek , freebsd-geom@freebsd.org References: <80B62FE6-FCFB-42B8-A34C-B28E7DDBF45D@dawidek.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 11.3-STABLE amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Sat, 04 Jul 2020 16:16:42 -0700 (PDT) X-Rspamd-Queue-Id: 49znlp6wCHz4FVn X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:32354, ipnet:208.87.216.0/21, country:US] X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Jul 2020 23:16:55 -0000 Alan Somers wrote this message on Sat, Jul 04, 2020 at 15:59 -0600: > I might give this a shot. What is the best way tell if geli ought to use > direct dispatch? Is there a generic "are crypto instructions available" > macro that would cover aesni as well as other platform-specific > instructions? Direct dispatch has the advantage of saving scheduling context switches... Geli has two modes, one is hardware acceleration mode, where it does a bit of work to put together the request, and then hands it off to the crypto framework (say an accelerator card), and then there is the mode where it has to be done in software, where it dispatches to a set of worker threads... In both modes, it would make sense for GELI to be able to do the work to construct those requests via direct dispatch... This would eliminate a context switch, which is always a good thing... I haven't looked at the OpenCrypto code in a while, so I don't know what the locking requirements are... The key cache is already locked by an mtx, but I believe it's a leaf lock, and so shouldn't be an issue... I'll add this to my list of things to look at... Also, if you have that many geli devices, you might also want to set: kern.geom.eli.threads=1 As it stands, geli fires up ncpu threads for EACH geli device, so likely have thousands of geli threads... > On Sat, Jul 4, 2020, 2:55 PM Pawe?? Jakub Dawidek wrote: > > > Direct dispatch would be great for geli, especially that geli can use own > > (multiple) threads when necessary (eg. using crypto cards). With AES-NI you > > could go straight to the disk. > > > > > On Jul 3, 2020, at 13:22, Alan Somers wrote: > > > > > > ???I don't. What I meant was that a single thread (geom) is limiting the > > > performance of the system overall. I'm certain, based on top, gstat, and > > > zpool iostat, that geom is the limiting factor on this system. > > > -Alan > > > > > >> On Fri, Jul 3, 2020 at 2:18 PM Pawe?? Jakub Dawidek > > >> wrote: > > >> > > >> Hi Alan, > > >> > > >> why do you think it will hurt single-threaded performance? > > >> > > >>>> On Jul 3, 2020, at 12:30, Alan Somers wrote: > > >>> > > >>> ???I'm using geli, gmultipath, and ZFS on a large system, with hundreds > > of > > >>> drives. What I'm seeing is that under at least some workloads, the > > >> overall > > >>> performance is limited by the single geom kernel process. procstat and > > >>> kgdb aren't much help in telling exactly why this process is using so > > >> much > > >>> CPU, but it certainly must be related to the fact that over 15,000 IOPs > > >> are > > >>> going through that thread. What can I do to improve this situation? > > >> Would > > >>> it make sense to enable direct dispatch for geli? That would hurt > > >>> single-threaded performance, but probably improve performance for > > highly > > >>> multithreaded workloads like mine. > > >>> > > >>> Example top output: > > >>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU > > COMMAND > > >>> 13 root -8 - 0B 96K CPU46 46 82.7H 70.54% > > >>> geom{g_down} > > >>> 13 root -8 - 0B 96K - 9 35.5H 25.32% > > >>> geom{g_up} -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."