From owner-freebsd-geom@freebsd.org Fri Jul 3 20:22:25 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3B5653588CA for ; Fri, 3 Jul 2020 20:22:25 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f65.google.com (mail-ot1-f65.google.com [209.85.210.65]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49z5ww1rd2z3SYk for ; Fri, 3 Jul 2020 20:22:24 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-ot1-f65.google.com with SMTP id t18so14658156otq.5 for ; Fri, 03 Jul 2020 13:22:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=rT88MxRqkdjmgZJB7dh9Ah3ZsgLrpKvmP2znfbPRyVg=; b=Dv4ou92t+O2s2MM3C+RXzugUxnLKe5ChnR6WsvTcVBjKmioRTp+0I2ItlhaT0X//qb LeMRGivZctFEkbxTOv0C8WkSzRjWScKrQby7J7b0HlyG02BZe0Pfu+INpBmeSkza8UjN eC9Odx7GGXDeaTEjopU2sPefC6FLSXX3uzVZNEOlWCc8lVyNh2SZEwMy4wtfIy41I5T1 /ebaV7vZSG9KsjgBz+0Ygi9oGYaz8fnY2Vt9QzIcj3gtHG14vF4EqpTaLIEEg1Srxv7w JlKr9vF/Uh7adQpP/gW1LhCTwg5oR58DtWmi2Iwx48/ZE0ja2g/itUMQ2FswIZddgiir GR4A== X-Gm-Message-State: AOAM531SctH82FVH7ajQFCVBssCvcI9ILmpp3HiwiRWERWDN50SCJ1UZ OqsTg6YPkHOfc0xmYW4TBwQleMOqQIRxyf8TtySyeZcW X-Google-Smtp-Source: ABdhPJx4n2F4LancOF9xXoahvM2oV4ATuV7640QiOhxdOoiqm6zLObyqAobqYT8O7CcQ64Zfhq0ybfhNX1a12JOE3vk= X-Received: by 2002:a05:6830:14c6:: with SMTP id t6mr19862133otq.18.1593807743032; Fri, 03 Jul 2020 13:22:23 -0700 (PDT) MIME-Version: 1.0 References: <48AB9191-5984-4BE0-BF98-7548424D06BF@dawidek.net> In-Reply-To: <48AB9191-5984-4BE0-BF98-7548424D06BF@dawidek.net> From: Alan Somers Date: Fri, 3 Jul 2020 14:22:11 -0600 Message-ID: Subject: Re: Single-threaded bottleneck in geli To: =?UTF-8?Q?Pawe=C5=82_Jakub_Dawidek?= Cc: freebsd-geom@freebsd.org X-Rspamd-Queue-Id: 49z5ww1rd2z3SYk X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.210.65 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-2.06 / 15.00]; RCVD_TLS_ALL(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-0.91)[-0.911]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-geom@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RWL_MAILSPIKE_GOOD(0.00)[209.85.210.65:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.20)[-0.197]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[209.85.210.65:from]; NEURAL_HAM_MEDIUM(-0.95)[-0.950]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jul 2020 20:22:25 -0000 I don't. What I meant was that a single thread (geom) is limiting the performance of the system overall. I'm certain, based on top, gstat, and zpool iostat, that geom is the limiting factor on this system. -Alan On Fri, Jul 3, 2020 at 2:18 PM Pawe=C5=82 Jakub Dawidek wrote: > Hi Alan, > > why do you think it will hurt single-threaded performance? > > -- > Pawe=C5=82 Jakub Dawidek > > > > > On Jul 3, 2020, at 12:30, Alan Somers wrote: > > > > =EF=BB=BFI'm using geli, gmultipath, and ZFS on a large system, with hu= ndreds of > > drives. What I'm seeing is that under at least some workloads, the > overall > > performance is limited by the single geom kernel process. procstat and > > kgdb aren't much help in telling exactly why this process is using so > much > > CPU, but it certainly must be related to the fact that over 15,000 IOPs > are > > going through that thread. What can I do to improve this situation? > Would > > it make sense to enable direct dispatch for geli? That would hurt > > single-threaded performance, but probably improve performance for highl= y > > multithreaded workloads like mine. > > > > Example top output: > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMA= ND > > 13 root -8 - 0B 96K CPU46 46 82.7H 70.54% > > geom{g_down} > > 13 root -8 - 0B 96K - 9 35.5H 25.32% > > geom{g_up} > > > > -Alan > > _______________________________________________ > > freebsd-geom@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-geom > > To unsubscribe, send any mail to "freebsd-geom-unsubscribe@freebsd.org" > >