From owner-freebsd-geom@freebsd.org Sat Jul 4 20:55:44 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F1DF6358212 for ; Sat, 4 Jul 2020 20:55:44 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (garage.dawidek.net [91.121.88.72]) by mx1.freebsd.org (Postfix) with ESMTP id 49zkcv45Pvz47X0; Sat, 4 Jul 2020 20:55:43 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from [IPv6:2607:fb90:9c84:f6f6:382a:457e:10d0:c0ea] (unknown [172.58.92.5]) by mail.dawidek.net (Postfix) with ESMTPSA id 84F24675; Sat, 4 Jul 2020 22:55:35 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) Subject: Re: Single-threaded bottleneck in geli From: =?utf-8?Q?Pawe=C5=82_Jakub_Dawidek?= In-Reply-To: Date: Sat, 4 Jul 2020 13:55:30 -0700 Cc: freebsd-geom@freebsd.org Message-Id: <80B62FE6-FCFB-42B8-A34C-B28E7DDBF45D@dawidek.net> References: To: Alan Somers X-Mailer: iPhone Mail (17F80) X-Rspamd-Queue-Id: 49zkcv45Pvz47X0 X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of pawel@dawidek.net has no SPF policy when checking 91.121.88.72) smtp.mailfrom=pawel@dawidek.net X-Spamd-Result: default: False [3.25 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[dawidek.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.37)[0.368]; NEURAL_SPAM_SHORT(0.22)[0.218]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.61)[0.611]; R_SPF_NA(0.00)[no SPF record]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16276, ipnet:91.121.0.0/16, country:FR]; RCVD_COUNT_TWO(0.00)[2]; R_MIXED_CHARSET(0.56)[subject]; RECEIVED_SPAMHAUS_PBL(0.00)[172.58.92.5:received] X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Jul 2020 20:55:45 -0000 Direct dispatch would be great for geli, especially that geli can use own (m= ultiple) threads when necessary (eg. using crypto cards). With AES-NI you co= uld go straight to the disk. --=20 Pawe=C5=82 Jakub Dawidek > On Jul 3, 2020, at 13:22, Alan Somers wrote: >=20 > =EF=BB=BFI don't. What I meant was that a single thread (geom) is limitin= g the > performance of the system overall. I'm certain, based on top, gstat, and > zpool iostat, that geom is the limiting factor on this system. > -Alan >=20 >> On Fri, Jul 3, 2020 at 2:18 PM Pawe=C5=82 Jakub Dawidek >> wrote: >>=20 >> Hi Alan, >>=20 >> why do you think it will hurt single-threaded performance? >>=20 >> -- >> Pawe=C5=82 Jakub Dawidek >>=20 >>=20 >>=20 >>>> On Jul 3, 2020, at 12:30, Alan Somers wrote: >>>=20 >>> =EF=BB=BFI'm using geli, gmultipath, and ZFS on a large system, with hun= dreds of >>> drives. What I'm seeing is that under at least some workloads, the >> overall >>> performance is limited by the single geom kernel process. procstat and >>> kgdb aren't much help in telling exactly why this process is using so >> much >>> CPU, but it certainly must be related to the fact that over 15,000 IOPs >> are >>> going through that thread. What can I do to improve this situation? >> Would >>> it make sense to enable direct dispatch for geli? That would hurt >>> single-threaded performance, but probably improve performance for highly= >>> multithreaded workloads like mine. >>>=20 >>> Example top output: >>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND= >>> 13 root -8 - 0B 96K CPU46 46 82.7H 70.54% >>> geom{g_down} >>> 13 root -8 - 0B 96K - 9 35.5H 25.32% >>> geom{g_up} >>>=20 >>> -Alan >>> _______________________________________________ >>> freebsd-geom@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-geom >>> To unsubscribe, send any mail to "freebsd-geom-unsubscribe@freebsd.org" >>=20 >>=20 > _______________________________________________ > freebsd-geom@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-geom > To unsubscribe, send any mail to "freebsd-geom-unsubscribe@freebsd.org"