From owner-freebsd-geom@freebsd.org Sun Jul 5 21:00:10 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BA01A34CEE6 for ; Sun, 5 Jul 2020 21:00:10 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4B0LgZ4X7tz4N2r for ; Sun, 5 Jul 2020 21:00:10 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id 9B49834CEE4; Sun, 5 Jul 2020 21:00:10 +0000 (UTC) Delivered-To: geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9B05034D206 for ; Sun, 5 Jul 2020 21:00:10 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B0LgZ3jKWz4NMG for ; Sun, 5 Jul 2020 21:00:10 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 6151D135B9 for ; Sun, 5 Jul 2020 21:00:10 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 065L0Arh084644 for ; Sun, 5 Jul 2020 21:00:10 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 065L0AkC084642 for geom@FreeBSD.org; Sun, 5 Jul 2020 21:00:10 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202007052100.065L0AkC084642@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: geom@FreeBSD.org Subject: Problem reports for geom@FreeBSD.org that need special attention Date: Sun, 5 Jul 2020 21:00:10 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jul 2020 21:00:10 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- In Progress | 218679 | [geli] add a verify command Open | 237269 | panic in glabel (g_label_destroy) stop after resi Open | 238814 | geom: topology lock being dropped in dumpconf of Open | 242747 | geli: AMD Epyc+GELI not using Hardware AES 4 problems total for which you should take action. From owner-freebsd-geom@freebsd.org Mon Jul 6 18:21:46 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 78B7836DA31 for ; Mon, 6 Jul 2020 18:21:46 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from mail.rlwinm.de (mail.rlwinm.de [138.201.35.217]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B0v6K3skXz4NNj for ; Mon, 6 Jul 2020 18:21:45 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from crest-mbp.fritz.box (200116b864060700ed126ddbf990e0d6.dip.versatel-1u1.de [IPv6:2001:16b8:6406:700:ed12:6ddb:f990:e0d6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.rlwinm.de (Postfix) with ESMTPSA id 9BFE41FD1C for ; Mon, 6 Jul 2020 18:21:37 +0000 (UTC) Subject: Re: Single-threaded bottleneck in geli To: freebsd-geom@freebsd.org References: From: Jan Bramkamp Message-ID: <550d61f9-506e-710c-8800-4f13143cf976@rlwinm.de> Date: Mon, 6 Jul 2020 20:21:36 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 4B0v6K3skXz4NNj X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of crest@rlwinm.de designates 138.201.35.217 as permitted sender) smtp.mailfrom=crest@rlwinm.de X-Spamd-Result: default: False [-2.56 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-geom@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.98)[-0.975]; DMARC_NA(0.00)[rlwinm.de]; NEURAL_HAM_SHORT(-0.33)[-0.330]; NEURAL_HAM_MEDIUM(-0.96)[-0.957]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:138.201.0.0/16, country:DE]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[2001:16b8:6406:700:ed12:6ddb:f990:e0d6:received] X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jul 2020 18:21:46 -0000 On 03.07.20 21:30, Alan Somers wrote: > I'm using geli, gmultipath, and ZFS on a large system, with hundreds of > drives. What I'm seeing is that under at least some workloads, the overall > performance is limited by the single geom kernel process. procstat and > kgdb aren't much help in telling exactly why this process is using so much > CPU, but it certainly must be related to the fact that over 15,000 IOPs are > going through that thread. What can I do to improve this situation? Would > it make sense to enable direct dispatch for geli? That would hurt > single-threaded performance, but probably improve performance for highly > multithreaded workloads like mine. > > Example top output: > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 13 root -8 - 0B 96K CPU46 46 82.7H 70.54% > geom{g_down} > 13 root -8 - 0B 96K - 9 35.5H 25.32% > geom{g_up} > > -Alan The problem isn't GELI. It's the problem is that gmultipath lacks direct dispatch support. Last one and a half years ago I ran into the same problem. Because I needed the performance I looked at what gmultipath did and found now reason why it has run in the GEOM up and down threads. So i patched in the flags claiming direct dispatch support. It improved my read performance from 2.2GB/s to 3.4GB/s and write performance from 750MB/s to 1.5GB/s the system worked for a few days under high load (saturated a 2 x 10Gb/s lagg(4) as read only WebDAV server and while receiving uploads via SFTP). It worked until I attempted to shutdown the system. It hung on shutdown an never powered off. I had to power cycle the box via IPMI to recover. I never found the time to debug this problem. From owner-freebsd-geom@freebsd.org Mon Jul 6 18:27:01 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 66A6536DB43 for ; Mon, 6 Jul 2020 18:27:01 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from mail.rlwinm.de (mail.rlwinm.de [IPv6:2a01:4f8:171:f902::5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B0vDL0xkNz4NL2 for ; Mon, 6 Jul 2020 18:26:57 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from crest-mbp.fritz.box (200116b864060700ed126ddbf990e0d6.dip.versatel-1u1.de [IPv6:2001:16b8:6406:700:ed12:6ddb:f990:e0d6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits)) (No client certificate requested) by mail.rlwinm.de (Postfix) with ESMTPSA id C64BD1FD71 for ; Mon, 6 Jul 2020 18:26:49 +0000 (UTC) Subject: Re: Single-threaded bottleneck in geli To: freebsd-geom@freebsd.org References: <550d61f9-506e-710c-8800-4f13143cf976@rlwinm.de> From: Jan Bramkamp Message-ID: <8ca74f4b-8935-b089-675d-df9b7fc2ae50@rlwinm.de> Date: Mon, 6 Jul 2020 20:26:49 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <550d61f9-506e-710c-8800-4f13143cf976@rlwinm.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4B0vDL0xkNz4NL2 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of crest@rlwinm.de designates 2a01:4f8:171:f902::5 as permitted sender) smtp.mailfrom=crest@rlwinm.de X-Spamd-Result: default: False [-2.54 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx:c]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-geom@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.97)[-0.969]; DMARC_NA(0.00)[rlwinm.de]; NEURAL_HAM_SHORT(-0.32)[-0.324]; NEURAL_HAM_MEDIUM(-0.95)[-0.950]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[2001:16b8:6406:700:ed12:6ddb:f990:e0d6:received] X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jul 2020 18:27:01 -0000 On 06.07.20 20:21, Jan Bramkamp wrote: > On 03.07.20 21:30, Alan Somers wrote: >> I'm using geli, gmultipath, and ZFS on a large system, with hundreds of >> drives.  What I'm seeing is that under at least some workloads, the >> overall >> performance is limited by the single geom kernel process. procstat and >> kgdb aren't much help in telling exactly why this process is using so >> much >> CPU, but it certainly must be related to the fact that over 15,000 >> IOPs are >> going through that thread.  What can I do to improve this situation?  >> Would >> it make sense to enable direct dispatch for geli?  That would hurt >> single-threaded performance, but probably improve performance for highly >> multithreaded workloads like mine. >> >> Example top output: >>    PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME WCPU COMMAND >>     13 root         -8    -     0B    96K CPU46   46  82.7H 70.54% >> geom{g_down} >>     13 root         -8    -     0B    96K -        9  35.5H 25.32% >> geom{g_up} >> >> -Alan > > The problem isn't GELI. It's the problem is that gmultipath lacks > direct dispatch support. Last one and a half years ago I ran into the > same problem. Because I needed the performance I looked at what > gmultipath did and found now reason why it has run in the GEOM up and > down threads. So i patched in the flags claiming direct dispatch > support. It improved my read performance from 2.2GB/s to 3.4GB/s and > write performance from 750MB/s to 1.5GB/s the system worked for a few > days under high load (saturated a 2 x 10Gb/s lagg(4) as read only > WebDAV server and while receiving uploads via SFTP). It worked until I > attempted to shutdown the system. It hung on shutdown an never powered > off. I had to power cycle the box via IPMI to recover. I never found > the time to debug this problem. > The server in question was an old Westmere dual hexacore with 96GB RAM and had 42 (+ 3 spares) 7200rpm SAS2 disks in a dual ported JBOD connected via 4 x 4 lanes to two LSI2?08 HBAs with IT firmware configured as 3way mirrored ZFS pool with GELI underneath. From owner-freebsd-geom@freebsd.org Tue Jul 7 22:03:09 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C249234E73E for ; Tue, 7 Jul 2020 22:03:09 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f48.google.com (mail-ot1-f48.google.com [209.85.210.48]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B1bzJ57Ndz4BDB for ; Tue, 7 Jul 2020 22:03:08 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-ot1-f48.google.com with SMTP id 5so33475101oty.11 for ; Tue, 07 Jul 2020 15:03:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=6RGcQknGCz6XUT7Eu/BTch7O6xq+oDLGz1QoO7fbom8=; b=uoIBkZRzAQn6LNszq9AQGh/i4g/ejJzHbm/5hISzU9IUD+368im5feHRJWDtnTqDPb wlCPqE86blh3ABbBhSDjBBTbJHEQeJ+BP/7mJ2/VmiRmiMNUl5GiwuZQj1VpY47Os7wR hB+EB2txaPC+dZZQjT3Q2Av+qiKHRZpUHpyCC7mWYGY35fNf3ga3Qj57mUnUJWgugFmF tkmpKB2iu54EzY+ijgyRoyB3HS8JX5pnNXtstw5oM/YVNeGhuKbMoG8eoN8U8aNym3gF yEbqndbg4o4Tg5ulvqao7cV8aJFGYWhI8EZmzwM52Gf2B6YjHltpVFrCeBsJeEH3eFvH tcwg== X-Gm-Message-State: AOAM532TZiLiiDhh4FOgmHxKVBvT3+S9Rfeyb2c1Y8PLWRAO+Fd/kG1w +vXZQpHbzofp8ErliE3xWhgdLo3GVJkGqKYnzQSfp+yr9U8= X-Google-Smtp-Source: ABdhPJw0SZ5UZNtO8hLNmt/5dkeVV4c1MThM3dynATTmZxeHVNH2Bi/gu/ycLdclLFf2o4RWr2sydGqkanXYgghUkWA= X-Received: by 2002:a9d:429:: with SMTP id 38mr26767053otc.291.1594159387504; Tue, 07 Jul 2020 15:03:07 -0700 (PDT) MIME-Version: 1.0 References: <80B62FE6-FCFB-42B8-A34C-B28E7DDBF45D@dawidek.net> <20200704231642.GU4213@funkthat.com> In-Reply-To: <20200704231642.GU4213@funkthat.com> From: Alan Somers Date: Tue, 7 Jul 2020 16:02:56 -0600 Message-ID: Subject: Re: Single-threaded bottleneck in geli To: "Pawe?? Jakub Dawidek" , freebsd-geom@freebsd.org X-Rspamd-Queue-Id: 4B1bzJ57Ndz4BDB X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.210.48 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-2.86 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.03)[-1.034]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-0.98)[-0.984]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-geom@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RWL_MAILSPIKE_GOOD(0.00)[209.85.210.48:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.84)[-0.840]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[209.85.210.48:from]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jul 2020 22:03:09 -0000 Ok, this turned out to be embarrassingly easy. I just turned it on, and everything worked. On a dual-socket Intel CPU E5-2680 (16 cores, hyperthreading disabled), this change roughly doubles geli's IOPs when reading or writing to 32 malloc-backed md devices. But I don't have any hardware crypto accelerator to test it on. Could anybody help me out with that? https://reviews.freebsd.org/D25587 On Sat, Jul 4, 2020 at 5:16 PM John-Mark Gurney wrote: > Alan Somers wrote this message on Sat, Jul 04, 2020 at 15:59 -0600: > > I might give this a shot. What is the best way tell if geli ought to use > > direct dispatch? Is there a generic "are crypto instructions available" > > macro that would cover aesni as well as other platform-specific > > instructions? > > Direct dispatch has the advantage of saving scheduling context > switches... > > Geli has two modes, one is hardware acceleration mode, where it does > a bit of work to put together the request, and then hands it off to > the crypto framework (say an accelerator card), and then there is the > mode where it has to be done in software, where it dispatches to a > set of worker threads... > > In both modes, it would make sense for GELI to be able to do the work > to construct those requests via direct dispatch... This would eliminate > a context switch, which is always a good thing... I haven't looked at > the OpenCrypto code in a while, so I don't know what the locking > requirements are... > > The key cache is already locked by an mtx, but I believe it's a leaf > lock, and so shouldn't be an issue... > > I'll add this to my list of things to look at... > > Also, if you have that many geli devices, you might also want to set: > kern.geom.eli.threads=1 > > As it stands, geli fires up ncpu threads for EACH geli device, so likely > have thousands of geli threads... > > > On Sat, Jul 4, 2020, 2:55 PM Pawe?? Jakub Dawidek > wrote: > > > > > Direct dispatch would be great for geli, especially that geli can use > own > > > (multiple) threads when necessary (eg. using crypto cards). With > AES-NI you > > > could go straight to the disk. > > > > > > > On Jul 3, 2020, at 13:22, Alan Somers wrote: > > > > > > > > ???I don't. What I meant was that a single thread (geom) is > limiting the > > > > performance of the system overall. I'm certain, based on top, > gstat, and > > > > zpool iostat, that geom is the limiting factor on this system. > > > > -Alan > > > > > > > >> On Fri, Jul 3, 2020 at 2:18 PM Pawe?? Jakub Dawidek < > pawel@dawidek.net> > > > >> wrote: > > > >> > > > >> Hi Alan, > > > >> > > > >> why do you think it will hurt single-threaded performance? > > > >> > > > >>>> On Jul 3, 2020, at 12:30, Alan Somers > wrote: > > > >>> > > > >>> ???I'm using geli, gmultipath, and ZFS on a large system, with > hundreds > > > of > > > >>> drives. What I'm seeing is that under at least some workloads, the > > > >> overall > > > >>> performance is limited by the single geom kernel process. > procstat and > > > >>> kgdb aren't much help in telling exactly why this process is using > so > > > >> much > > > >>> CPU, but it certainly must be related to the fact that over 15,000 > IOPs > > > >> are > > > >>> going through that thread. What can I do to improve this > situation? > > > >> Would > > > >>> it make sense to enable direct dispatch for geli? That would hurt > > > >>> single-threaded performance, but probably improve performance for > > > highly > > > >>> multithreaded workloads like mine. > > > >>> > > > >>> Example top output: > > > >>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU > > > COMMAND > > > >>> 13 root -8 - 0B 96K CPU46 46 82.7H 70.54% > > > >>> geom{g_down} > > > >>> 13 root -8 - 0B 96K - 9 35.5H 25.32% > > > >>> geom{g_up} > > -- > John-Mark Gurney Voice: +1 415 225 5579 > > "All that I will do, has been done, All that I have, has not." > From owner-freebsd-geom@freebsd.org Thu Jul 9 02:06:23 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 266B035C2EE for ; Thu, 9 Jul 2020 02:06:23 +0000 (UTC) (envelope-from diabetesfree@centuryrescue.digital) Received: from centuryrescue.digital (centuryrescue.digital [84.54.12.25]) by mx1.freebsd.org (Postfix) with ESMTP id 4B2KKV4SQgz4MGs for ; Thu, 9 Jul 2020 02:06:22 +0000 (UTC) (envelope-from diabetesfree@centuryrescue.digital) Date: Wed, 08 Jul 2020 21:02:22 -0500 From: "Diabetes Free" To: Subject: Americans eating =?ISO-8859-1?Q?=E2=80=9Cstainless=20?= =?ISO-8859-1?Q?steel=E2=80=9D=20?=to reverse diabetes Message-ID: X-Rspamd-Queue-Id: 4B2KKV4SQgz4MGs X-Spamd-Bar: +++++ X-Spamd-Result: default: False [5.81 / 15.00]; ARC_NA(0.00)[]; R_DKIM_ALLOW(0.00)[centuryrescue.digital:s=mail]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(0.00)[+ip4:84.54.12.25:c]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; NEURAL_SPAM_MEDIUM(0.99)[0.986]; RCPT_COUNT_ONE(0.00)[1]; BAD_REP_POLICIES(0.10)[]; MANY_INVISIBLE_PARTS(0.05)[1]; RBL_VIRUSFREE_BOTNET(2.00)[84.54.12.25:from]; NEURAL_SPAM_SHORT(0.77)[0.768]; DKIM_TRACE(0.00)[centuryrescue.digital:+]; DMARC_POLICY_ALLOW(0.00)[centuryrescue.digital,quarantine]; NEURAL_SPAM_LONG(0.99)[0.993]; MIME_HTML_ONLY(0.20)[]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:~]; R_MIXED_CHARSET(0.71)[subject]; ASN(0.00)[asn:202505, ipnet:84.54.12.0/24, country:TR]; MID_RHS_MATCH_FROM(0.00)[] MIME-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jul 2020 02:06:23 -0000