Date: Sat, 4 Jul 2020 13:55:30 -0700 From: =?utf-8?Q?Pawe=C5=82_Jakub_Dawidek?= <pawel@dawidek.net> To: Alan Somers <asomers@freebsd.org> Cc: freebsd-geom@freebsd.org Subject: Re: Single-threaded bottleneck in geli Message-ID: <80B62FE6-FCFB-42B8-A34C-B28E7DDBF45D@dawidek.net> In-Reply-To: <CAOtMX2hHaEzOT0jmc_QcukVZjRKUtCm55bTT9Q5=BNCcL9rf%2Bg@mail.gmail.com> References: <CAOtMX2hHaEzOT0jmc_QcukVZjRKUtCm55bTT9Q5=BNCcL9rf%2Bg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Direct dispatch would be great for geli, especially that geli can use own (m=
ultiple) threads when necessary (eg. using crypto cards). With AES-NI you co=
uld go straight to the disk.
--=20
Pawe=C5=82 Jakub Dawidek
> On Jul 3, 2020, at 13:22, Alan Somers <asomers@freebsd.org> wrote:
>=20
> =EF=BB=BFI don't. What I meant was that a single thread (geom) is limitin=
g the
> performance of the system overall. I'm certain, based on top, gstat, and
> zpool iostat, that geom is the limiting factor on this system.
> -Alan
>=20
>> On Fri, Jul 3, 2020 at 2:18 PM Pawe=C5=82 Jakub Dawidek <pawel@dawidek.ne=
t>
>> wrote:
>>=20
>> Hi Alan,
>>=20
>> why do you think it will hurt single-threaded performance?
>>=20
>> --
>> Pawe=C5=82 Jakub Dawidek
>>=20
>>=20
>>=20
>>>> On Jul 3, 2020, at 12:30, Alan Somers <asomers@freebsd.org> wrote:
>>>=20
>>> =EF=BB=BFI'm using geli, gmultipath, and ZFS on a large system, with hun=
dreds of
>>> drives. What I'm seeing is that under at least some workloads, the
>> overall
>>> performance is limited by the single geom kernel process. procstat and
>>> kgdb aren't much help in telling exactly why this process is using so
>> much
>>> CPU, but it certainly must be related to the fact that over 15,000 IOPs
>> are
>>> going through that thread. What can I do to improve this situation?
>> Would
>>> it make sense to enable direct dispatch for geli? That would hurt
>>> single-threaded performance, but probably improve performance for highly=
>>> multithreaded workloads like mine.
>>>=20
>>> Example top output:
>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND=
>>> 13 root -8 - 0B 96K CPU46 46 82.7H 70.54%
>>> geom{g_down}
>>> 13 root -8 - 0B 96K - 9 35.5H 25.32%
>>> geom{g_up}
>>>=20
>>> -Alan
>>> _______________________________________________
>>> freebsd-geom@freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-geom
>>> To unsubscribe, send any mail to "freebsd-geom-unsubscribe@freebsd.org"
>>=20
>>=20
> _______________________________________________
> freebsd-geom@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-geom
> To unsubscribe, send any mail to "freebsd-geom-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?80B62FE6-FCFB-42B8-A34C-B28E7DDBF45D>