Date: Tue, 3 Dec 2019 23:28:48 +0100 From: Peter Eriksson <pen@lysator.liu.se> To: FreeBSD FS <freebsd-fs@freebsd.org> Subject: Re: Slow reboots due to ZFS cleanup in kern_shutdown() .. zio_fini() Message-ID: <4D44B25E-2F15-4536-A653-DA242051C8A9@lysator.liu.se> In-Reply-To: <247C6990-BC6E-4E3D-8CEF-5A861D8A25EC@lysator.liu.se> References: <AD17E454-6A51-436D-A853-07F04A406EC9@lysator.liu.se> <D2A11CE9-9B24-4E40-A51A-8D318E0288C9@lysator.liu.se> <20191202225424.GG43802@raichu> <3b71fe37-c29f-e3e5-ff96-5dce15cc7553@FreeBSD.org> <20191203162219.GI43802@raichu> <247C6990-BC6E-4E3D-8CEF-5A861D8A25EC@lysator.liu.se>
next in thread | previous in thread | raw e-mail | index | archive | help
=E2=80=A6 and X levels deeper it gives this (not all filesystems mounted =
- I rebooted fairly quickly):
keg_drain: while (SLIST_FIRST/SLIST_REMOVE)-loop took 14 seconds [20021 =
loops]
zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 14 seconds
zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 14 seconds
zone_free_item(zone=3DUMA Zones): zone->uz_dtor() took 14 seconds
uma_zdestroy(zio_buf_12288) took 14 seconds
kmem_cache_destroy: uma_zdestroy(0xfffff803467c8ac0) [zio_buf_12288] =
took 14 seconds
kmem_cache_destroy(zio_buf_cache[20]) took 14 seconds
Hmm=E2=80=A6 I wonder if it isn=E2=80=99t this code I =
/usr/src/sys/sys/queue.h:
> #define SLIST_REMOVE(head, elm, type, field) do { =
\
> QMD_SAVELINK(oldnext, (elm)->field.sle_next); =
\
> if (SLIST_FIRST((head)) =3D=3D (elm)) { =
\
> SLIST_REMOVE_HEAD((head), field); =
\
> } =
\
> else { =
\
> QUEUE_TYPEOF(type) *curelm =3D SLIST_FIRST(head); =
\
> while (SLIST_NEXT(curelm, field) !=3D (elm)) =
\
> curelm =3D SLIST_NEXT(curelm, field); =
\
> SLIST_REMOVE_AFTER(curelm, field); =
\
> } =
\
> TRASHIT(*oldnext); =
\
> } while (0)
Combined with this in /usr/src/sys/vm/uma_core.c: keg_drain():
> finished:
> KEG_UNLOCK(keg);
>
> while ((slab =3D SLIST_FIRST(&freeslabs)) !=3D NULL) {
> SLIST_REMOVE(&freeslabs, slab, uma_slab, us_hlink);
> keg_free_slab(keg, slab, keg->uk_ipers);
> }
(The keg_drain print above is the time for the while() loop).
If I=E2=80=99m reading this right it looks like it=E2=80=99s potentially =
doing a linear search through a (long) linked list, for every entry in =
the free slabs list. 20000 entries in 14 seconds is 1400 entries per =
second.
Hmmm=E2=80=A6 It feels like this code could be optimised at bit.
- Peter
> On 3 Dec 2019, at 18:31, Peter Eriksson <pen@lysator.liu.se> wrote:
>=20
> zio_fini: kmem_cache_destroy(zio_buf_cache & zio_data_buf_cache):
>=20
> zone_free_item(zone=3DUMA Zones): zone->uz_dtor() took 16 seconds
> uma_zdestroy(zio_buf_12288) took 16 seconds
> kmem_cache_destroy: uma_zdestroy(0xfffff8034678e980) [zio_buf_12288] =
took 16 seconds
> kmem_cache_destroy(zio_buf_cache[20]) took 16 seconds
>=20
> zone_free_item(zone=3DUMA Zones): zone->uz_dtor() took 60 seconds
> uma_zdestroy(zio_buf_16384) took 60 seconds
> kmem_cache_destroy: uma_zdestroy(0xfffff8034678e880) [zio_buf_16384] =
took 60 seconds
> kmem_cache_destroy(zio_buf_cache[28]) took 60 seconds
>=20
>=20
> Looks like it is the call to zone->uz_dtor that is taking time (added =
time check around the various parts of zone_free_item().
>=20
> - Peter
>=20
>> Indeed, looking at the vmstat -z output this doesn't really make =
sense.
>> It would be useful to confirm that we are in fact spending most of =
the
>> time reclaiming items.
>=20
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4D44B25E-2F15-4536-A653-DA242051C8A9>