From owner-freebsd-geom@freebsd.org Tue Jul 7 22:03:09 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C249234E73E for ; Tue, 7 Jul 2020 22:03:09 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f48.google.com (mail-ot1-f48.google.com [209.85.210.48]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B1bzJ57Ndz4BDB for ; Tue, 7 Jul 2020 22:03:08 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-ot1-f48.google.com with SMTP id 5so33475101oty.11 for ; Tue, 07 Jul 2020 15:03:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=6RGcQknGCz6XUT7Eu/BTch7O6xq+oDLGz1QoO7fbom8=; b=uoIBkZRzAQn6LNszq9AQGh/i4g/ejJzHbm/5hISzU9IUD+368im5feHRJWDtnTqDPb wlCPqE86blh3ABbBhSDjBBTbJHEQeJ+BP/7mJ2/VmiRmiMNUl5GiwuZQj1VpY47Os7wR hB+EB2txaPC+dZZQjT3Q2Av+qiKHRZpUHpyCC7mWYGY35fNf3ga3Qj57mUnUJWgugFmF tkmpKB2iu54EzY+ijgyRoyB3HS8JX5pnNXtstw5oM/YVNeGhuKbMoG8eoN8U8aNym3gF yEbqndbg4o4Tg5ulvqao7cV8aJFGYWhI8EZmzwM52Gf2B6YjHltpVFrCeBsJeEH3eFvH tcwg== X-Gm-Message-State: AOAM532TZiLiiDhh4FOgmHxKVBvT3+S9Rfeyb2c1Y8PLWRAO+Fd/kG1w +vXZQpHbzofp8ErliE3xWhgdLo3GVJkGqKYnzQSfp+yr9U8= X-Google-Smtp-Source: ABdhPJw0SZ5UZNtO8hLNmt/5dkeVV4c1MThM3dynATTmZxeHVNH2Bi/gu/ycLdclLFf2o4RWr2sydGqkanXYgghUkWA= X-Received: by 2002:a9d:429:: with SMTP id 38mr26767053otc.291.1594159387504; Tue, 07 Jul 2020 15:03:07 -0700 (PDT) MIME-Version: 1.0 References: <80B62FE6-FCFB-42B8-A34C-B28E7DDBF45D@dawidek.net> <20200704231642.GU4213@funkthat.com> In-Reply-To: <20200704231642.GU4213@funkthat.com> From: Alan Somers Date: Tue, 7 Jul 2020 16:02:56 -0600 Message-ID: Subject: Re: Single-threaded bottleneck in geli To: "Pawe?? Jakub Dawidek" , freebsd-geom@freebsd.org X-Rspamd-Queue-Id: 4B1bzJ57Ndz4BDB X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.210.48 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-2.86 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.03)[-1.034]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-0.98)[-0.984]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-geom@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RWL_MAILSPIKE_GOOD(0.00)[209.85.210.48:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.84)[-0.840]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[209.85.210.48:from]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jul 2020 22:03:09 -0000 Ok, this turned out to be embarrassingly easy. I just turned it on, and everything worked. On a dual-socket Intel CPU E5-2680 (16 cores, hyperthreading disabled), this change roughly doubles geli's IOPs when reading or writing to 32 malloc-backed md devices. But I don't have any hardware crypto accelerator to test it on. Could anybody help me out with that? https://reviews.freebsd.org/D25587 On Sat, Jul 4, 2020 at 5:16 PM John-Mark Gurney wrote: > Alan Somers wrote this message on Sat, Jul 04, 2020 at 15:59 -0600: > > I might give this a shot. What is the best way tell if geli ought to use > > direct dispatch? Is there a generic "are crypto instructions available" > > macro that would cover aesni as well as other platform-specific > > instructions? > > Direct dispatch has the advantage of saving scheduling context > switches... > > Geli has two modes, one is hardware acceleration mode, where it does > a bit of work to put together the request, and then hands it off to > the crypto framework (say an accelerator card), and then there is the > mode where it has to be done in software, where it dispatches to a > set of worker threads... > > In both modes, it would make sense for GELI to be able to do the work > to construct those requests via direct dispatch... This would eliminate > a context switch, which is always a good thing... I haven't looked at > the OpenCrypto code in a while, so I don't know what the locking > requirements are... > > The key cache is already locked by an mtx, but I believe it's a leaf > lock, and so shouldn't be an issue... > > I'll add this to my list of things to look at... > > Also, if you have that many geli devices, you might also want to set: > kern.geom.eli.threads=1 > > As it stands, geli fires up ncpu threads for EACH geli device, so likely > have thousands of geli threads... > > > On Sat, Jul 4, 2020, 2:55 PM Pawe?? Jakub Dawidek > wrote: > > > > > Direct dispatch would be great for geli, especially that geli can use > own > > > (multiple) threads when necessary (eg. using crypto cards). With > AES-NI you > > > could go straight to the disk. > > > > > > > On Jul 3, 2020, at 13:22, Alan Somers wrote: > > > > > > > > ???I don't. What I meant was that a single thread (geom) is > limiting the > > > > performance of the system overall. I'm certain, based on top, > gstat, and > > > > zpool iostat, that geom is the limiting factor on this system. > > > > -Alan > > > > > > > >> On Fri, Jul 3, 2020 at 2:18 PM Pawe?? Jakub Dawidek < > pawel@dawidek.net> > > > >> wrote: > > > >> > > > >> Hi Alan, > > > >> > > > >> why do you think it will hurt single-threaded performance? > > > >> > > > >>>> On Jul 3, 2020, at 12:30, Alan Somers > wrote: > > > >>> > > > >>> ???I'm using geli, gmultipath, and ZFS on a large system, with > hundreds > > > of > > > >>> drives. What I'm seeing is that under at least some workloads, the > > > >> overall > > > >>> performance is limited by the single geom kernel process. > procstat and > > > >>> kgdb aren't much help in telling exactly why this process is using > so > > > >> much > > > >>> CPU, but it certainly must be related to the fact that over 15,000 > IOPs > > > >> are > > > >>> going through that thread. What can I do to improve this > situation? > > > >> Would > > > >>> it make sense to enable direct dispatch for geli? That would hurt > > > >>> single-threaded performance, but probably improve performance for > > > highly > > > >>> multithreaded workloads like mine. > > > >>> > > > >>> Example top output: > > > >>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU > > > COMMAND > > > >>> 13 root -8 - 0B 96K CPU46 46 82.7H 70.54% > > > >>> geom{g_down} > > > >>> 13 root -8 - 0B 96K - 9 35.5H 25.32% > > > >>> geom{g_up} > > -- > John-Mark Gurney Voice: +1 415 225 5579 > > "All that I will do, has been done, All that I have, has not." >