From owner-freebsd-geom@freebsd.org Sat Jul 4 21:59:29 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4FD55359818 for ; Sat, 4 Jul 2020 21:59:29 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oo1-f44.google.com (mail-oo1-f44.google.com [209.85.161.44]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49zm2S2CR6z4BVc for ; Sat, 4 Jul 2020 21:59:28 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oo1-f44.google.com with SMTP id k47so3282850ool.2 for ; Sat, 04 Jul 2020 14:59:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=VYUFAf2y0gdOeoWRtLZch+yOXxisX//+9M4MMV6edDE=; b=hn221SQYy96DL5I7cCpsORiPQIbNhJowUkHTjHQjIkbfzcvvJLShKv+d5M9HEhxs3p fWCYCnbXtt6hqzchGqfysZN+y0Svlg1k/VWLqKIeH271mjWZKlgmTUny/9/ZvBO8368X JfngJWNukZb4QQTQa6jX0K56//TYeJFFLNudx5H1do3fHwX29r2HYp+b6iUvqR18GA9v o0+Iq8joUJ99f9XE7I/rJ12kOXTDi7G1TLfZ8dSUAusjn7UjLyAkJjCLmmc0w/I9cZ0R EOC29EIkJ5N9zDzCQ6BSQ2E0SEyGlKf14gnhHD2x/0YxPKrUld5/SJHuLbYT19ucBoGg E/Ew== X-Gm-Message-State: AOAM533k82AAhWRLcD+9nYX6ziH8LvcdB78KHn+C27OHHgjIyLaB2kfB 3J3rFJgVha9f1TbayJpjbICL3NK+34uRFv9q6VJkL1vH X-Google-Smtp-Source: ABdhPJyq/Hl0dpkGKr0nDvkybUV2LW54h21zaKA/AlHIvrgwTy1GwoHIxQb1EwuHbMdRR7U8DtqEbNfOCaK7zt+HAsg= X-Received: by 2002:a4a:a603:: with SMTP id e3mr20331195oom.61.1593899966940; Sat, 04 Jul 2020 14:59:26 -0700 (PDT) MIME-Version: 1.0 References: <80B62FE6-FCFB-42B8-A34C-B28E7DDBF45D@dawidek.net> In-Reply-To: <80B62FE6-FCFB-42B8-A34C-B28E7DDBF45D@dawidek.net> From: Alan Somers Date: Sat, 4 Jul 2020 15:59:16 -0600 Message-ID: Subject: Re: Single-threaded bottleneck in geli To: =?UTF-8?Q?Pawe=C5=82_Jakub_Dawidek?= Cc: freebsd-geom@freebsd.org X-Rspamd-Queue-Id: 49zm2S2CR6z4BVc X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.161.44 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-1.97 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.93)[-0.927]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; NEURAL_HAM_LONG(-0.96)[-0.965]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-geom@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RWL_MAILSPIKE_GOOD(0.00)[209.85.161.44:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.08)[-0.076]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[209.85.161.44:from]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Jul 2020 21:59:29 -0000 I might give this a shot. What is the best way tell if geli ought to use direct dispatch? Is there a generic "are crypto instructions available" macro that would cover aesni as well as other platform-specific instructions? -Alan On Sat, Jul 4, 2020, 2:55 PM Pawe=C5=82 Jakub Dawidek w= rote: > Direct dispatch would be great for geli, especially that geli can use own > (multiple) threads when necessary (eg. using crypto cards). With AES-NI y= ou > could go straight to the disk. > > -- > Pawe=C5=82 Jakub Dawidek > > > > > On Jul 3, 2020, at 13:22, Alan Somers wrote: > > > > =EF=BB=BFI don't. What I meant was that a single thread (geom) is limi= ting the > > performance of the system overall. I'm certain, based on top, gstat, a= nd > > zpool iostat, that geom is the limiting factor on this system. > > -Alan > > > >> On Fri, Jul 3, 2020 at 2:18 PM Pawe=C5=82 Jakub Dawidek > >> wrote: > >> > >> Hi Alan, > >> > >> why do you think it will hurt single-threaded performance? > >> > >> -- > >> Pawe=C5=82 Jakub Dawidek > >> > >> > >> > >>>> On Jul 3, 2020, at 12:30, Alan Somers wrote: > >>> > >>> =EF=BB=BFI'm using geli, gmultipath, and ZFS on a large system, with = hundreds > of > >>> drives. What I'm seeing is that under at least some workloads, the > >> overall > >>> performance is limited by the single geom kernel process. procstat a= nd > >>> kgdb aren't much help in telling exactly why this process is using so > >> much > >>> CPU, but it certainly must be related to the fact that over 15,000 IO= Ps > >> are > >>> going through that thread. What can I do to improve this situation? > >> Would > >>> it make sense to enable direct dispatch for geli? That would hurt > >>> single-threaded performance, but probably improve performance for > highly > >>> multithreaded workloads like mine. > >>> > >>> Example top output: > >>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > >>> 13 root -8 - 0B 96K CPU46 46 82.7H 70.54% > >>> geom{g_down} > >>> 13 root -8 - 0B 96K - 9 35.5H 25.32% > >>> geom{g_up} > >>> > >>> -Alan > >>> _______________________________________________ > >>> freebsd-geom@freebsd.org mailing list > >>> https://lists.freebsd.org/mailman/listinfo/freebsd-geom > >>> To unsubscribe, send any mail to "freebsd-geom-unsubscribe@freebsd.or= g > " > >> > >> > > _______________________________________________ > > freebsd-geom@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-geom > > To unsubscribe, send any mail to "freebsd-geom-unsubscribe@freebsd.org" > >