Date: Thu, 6 Mar 2025 06:56:27 -0800 From: Warner Losh <imp@bsdimp.com> To: John Baldwin <jhb@freebsd.org> Cc: Mateusz Guzik <mjguzik@gmail.com>, Zhenlei Huang <zlei@freebsd.org>, Mateusz Guzik <mjg@freebsd.org>, src-committers <src-committers@freebsd.org>, "<dev-commits-src-all@freebsd.org>" <dev-commits-src-all@freebsd.org>, Warner Losh <imp@freebsd.org>, "<dev-commits-src-main@freebsd.org>" <dev-commits-src-main@freebsd.org> Subject: Re: git: 234683726708 - main - devclass: make devclass_alloc_unit use M_NOWAIT Message-ID: <CANCZdfr-1WTp0XoWYieHfo9GZ3Du6QKtyVzh_6aX%2BQnPHyN8ig@mail.gmail.com> In-Reply-To: <d09b8cd1-ac71-43a4-9625-e796fd142f16@FreeBSD.org> References: <202503061103.526B32Id022652@gitrepo.freebsd.org> <F1B3652E-0D0C-402A-8509-D510992DAC15@FreeBSD.org> <CAGudoHF=eRaHCcjRrvd4sG4-OBu0GrmVRpiHEeU1ayG=M9oXrg@mail.gmail.com> <d09b8cd1-ac71-43a4-9625-e796fd142f16@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--0000000000008ba1a2062fadb614 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Mar 6, 2025, 5:33=E2=80=AFAM John Baldwin <jhb@freebsd.org> wrote: > On 3/6/25 06:35, Mateusz Guzik wrote: > > On Thu, Mar 6, 2025 at 12:32=E2=80=AFPM Zhenlei Huang <zlei@freebsd.org= > wrote: > >> > >> > >> > >> On Mar 6, 2025, at 7:03 PM, Mateusz Guzik <mjg@FreeBSD.org> wrote: > >> > >> The branch main has been updated by mjg: > >> > >> URL: > https://cgit.FreeBSD.org/src/commit/?id=3D234683726708cf5212d672d676d3005= 6d4133859 > >> > >> commit 234683726708cf5212d672d676d30056d4133859 > >> Author: Mateusz Guzik <mjg@FreeBSD.org> > >> AuthorDate: 2025-03-06 11:01:49 +0000 > >> Commit: Mateusz Guzik <mjg@FreeBSD.org> > >> CommitDate: 2025-03-06 11:01:49 +0000 > >> > >> devclass: make devclass_alloc_unit use M_NOWAIT > >> > >> The only caller already does this. > >> > >> The routine can be called with a mutex held making M_WAITOK illega= l. > >> > >> Sponsored by: Rubicon Communications, LLC ("Netgate") > >> --- > >> sys/kern/subr_bus.c | 8 ++++++-- > >> 1 file changed, 6 insertions(+), 2 deletions(-) > >> > >> diff --git a/sys/kern/subr_bus.c b/sys/kern/subr_bus.c > >> index 9506e471705c..0422352bba51 100644 > >> --- a/sys/kern/subr_bus.c > >> +++ b/sys/kern/subr_bus.c > >> @@ -1208,6 +1208,7 @@ devclass_get_sysctl_tree(devclass_t dc) > >> static int > >> devclass_alloc_unit(devclass_t dc, device_t dev, int *unitp) > >> { > >> + device_t *devices; > >> const char *s; > >> int unit =3D *unitp; > >> > >> @@ -1264,8 +1265,11 @@ devclass_alloc_unit(devclass_t dc, device_t dev= , > int *unitp) > >> int newsize; > >> > >> newsize =3D unit + 1; > >> - dc->devices =3D reallocf(dc->devices, > >> - newsize * sizeof(*dc->devices), M_BUS, M_WAITOK); > >> + devices =3D reallocf(dc->devices, > >> + newsize * sizeof(*dc->devices), M_BUS, M_NOWAIT); > >> > >> > >> I'd recommend against this. From the commit message of f3d3c63442ff, > Warner said, > >>> In addition, transition to M_WAITOK since this is a sleepable context > >> So, the M_WAITOK is intentional. > >> > >> Rather than reverting this, the caller devclass_add_device() should us= e > M_WAITOK. > >> > > > > Per my commit message this is callable from a *NOT* sleepable context. > > > > Here is a splat we got at Netgate: > > > > uma_zalloc_debug: zone "malloc-16" with the following non-sleepable > locks held: > > exclusive sleep mutex SD slot mtx (sdhci) r =3D 0 (0xd8dec028) locked @ > > > /var/jenkins/workspace/pfSense-Plus-snapshots-25_03-main/sources/FreeBSD-= src-plus-RELENG_25_03/sys/dev/sdhci/sdhci.c:688 > > stack backtrace: > > #0 0xc0330ebc at witness_debugger+0x78 > > #1 0xc033217c at witness_warn+0x428 > > #2 0xc05b0a58 at uma_zalloc_debug+0x34 > > #3 0xc05b067c at uma_zalloc_arg+0x30 > > #4 0xc0291760 at malloc+0x8c > > #5 0xc02920ec at reallocf+0x14 > > #6 0xc02f8894 at devclass_add_device+0x1e8 > > #7 0xc02f6c78 at make_device+0xe0 > > #8 0xc02f6abc at device_add_child_ordered+0x30 > > #9 0xc0156e0c at sdhci_card_task+0x238 > > #10 0xc0324090 at taskqueue_run_locked+0x1b4 > > #11 0xc0323ea0 at taskqueue_run+0x50 > > #12 0xc0275f88 at ithread_loop+0x264 > > Just use a regular taskqueue like taskqueue_thread instead of > taskqueue_swi? > PCI hotplug defines its own thread taskqueue for adding and removing > devices. > > The bug is here, IMO. Eventually new-bus will need some sort of topology > lock and that will have to be an sx lock, so this code needs to change > anyway. The sound code that tries to frob devices with a regular mutex > also needs to change. > I should dust off the branch that i have this one. There's about a dozen places I had to change at the time... In terms of taskqueue_swi, it's probably something that needs to go away. > Generally speaking, code uses tasks for functions that need to sleep, > and taskqueue_swi breaks that. > Its a holdover from spl days for sure. I will fix sdhci to use a proper taskqueue and then revert this commit. > Thanks. You can add me to the review Warner > -- > John Baldwin > > --0000000000008ba1a2062fadb614 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"auto"><div><br><br><div class=3D"gmail_quote gmail_quote_contai= ner"><div dir=3D"ltr" class=3D"gmail_attr">On Thu, Mar 6, 2025, 5:33=E2=80= =AFAM John Baldwin <<a href=3D"mailto:jhb@freebsd.org">jhb@freebsd.org</= a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 = 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 3/6/25 06:35, Mate= usz Guzik wrote:<br> > On Thu, Mar 6, 2025 at 12:32=E2=80=AFPM Zhenlei Huang <<a href=3D"m= ailto:zlei@freebsd.org" target=3D"_blank" rel=3D"noreferrer">zlei@freebsd.o= rg</a>> wrote:<br> >><br> >><br> >><br> >> On Mar 6, 2025, at 7:03 PM, Mateusz Guzik <mjg@FreeBSD.org> = wrote:<br> >><br> >> The branch main has been updated by mjg:<br> >><br> >> URL: <a href=3D"https://cgit.FreeBSD.org/src/commit/?id=3D23468372= 6708cf5212d672d676d30056d4133859" rel=3D"noreferrer noreferrer" target=3D"_= blank">https://cgit.FreeBSD.org/src/commit/?id=3D234683726708cf5212d672d676= d30056d4133859</a><br> >><br> >> commit 234683726708cf5212d672d676d30056d4133859<br> >> Author:=C2=A0 =C2=A0 =C2=A0Mateusz Guzik <mjg@FreeBSD.org><b= r> >> AuthorDate: 2025-03-06 11:01:49 +0000<br> >> Commit:=C2=A0 =C2=A0 =C2=A0Mateusz Guzik <mjg@FreeBSD.org><b= r> >> CommitDate: 2025-03-06 11:01:49 +0000<br> >><br> >>=C2=A0 =C2=A0 =C2=A0devclass: make devclass_alloc_unit use M_NOWAIT= <br> >><br> >>=C2=A0 =C2=A0 =C2=A0The only caller already does this.<br> >><br> >>=C2=A0 =C2=A0 =C2=A0The routine can be called with a mutex held mak= ing M_WAITOK illegal.<br> >><br> >>=C2=A0 =C2=A0 =C2=A0Sponsored by:=C2=A0 =C2=A0Rubicon Communication= s, LLC ("Netgate")<br> >> ---<br> >> sys/kern/subr_bus.c | 8 ++++++--<br> >> 1 file changed, 6 insertions(+), 2 deletions(-)<br> >><br> >> diff --git a/sys/kern/subr_bus.c b/sys/kern/subr_bus.c<br> >> index 9506e471705c..0422352bba51 100644<br> >> --- a/sys/kern/subr_bus.c<br> >> +++ b/sys/kern/subr_bus.c<br> >> @@ -1208,6 +1208,7 @@ devclass_get_sysctl_tree(devclass_t dc)<br> >> static int<br> >> devclass_alloc_unit(devclass_t dc, device_t dev, int *unitp)<br> >> {<br> >> + device_t *devices;<br> >> const char *s;<br> >> int unit =3D *unitp;<br> >><br> >> @@ -1264,8 +1265,11 @@ devclass_alloc_unit(devclass_t dc, device_t= dev, int *unitp)<br> >> int newsize;<br> >><br> >> newsize =3D unit + 1;<br> >> - dc->devices =3D reallocf(dc->devices,<br> >> -=C2=A0 =C2=A0 newsize * sizeof(*dc->devices), M_BUS, M_WAITOK)= ;<br> >> + devices =3D reallocf(dc->devices,<br> >> +=C2=A0 =C2=A0 newsize * sizeof(*dc->devices), M_BUS, M_NOWAIT)= ;<br> >><br> >><br> >> I'd recommend against this. From the commit message of f3d3c63= 442ff, Warner said,<br> >>> In addition, transition to M_WAITOK since this is a sleepable = context<br> >> So, the M_WAITOK is intentional.<br> >><br> >> Rather than reverting this, the caller devclass_add_device() shoul= d use M_WAITOK.<br> >><br> > <br> > Per my commit message this is callable from a *NOT* sleepable context.= <br> > <br> > Here is a splat we got at Netgate:<br> > <br> > uma_zalloc_debug: zone "malloc-16" with the following non-sl= eepable locks held:<br> > exclusive sleep mutex SD slot mtx (sdhci) r =3D 0 (0xd8dec028) locked = @<br> > /var/jenkins/workspace/pfSense-Plus-snapshots-25_03-main/sources/FreeB= SD-src-plus-RELENG_25_03/sys/dev/sdhci/sdhci.c:688<br> > stack backtrace:<br> > #0 0xc0330ebc at witness_debugger+0x78<br> > #1 0xc033217c at witness_warn+0x428<br> > #2 0xc05b0a58 at uma_zalloc_debug+0x34<br> > #3 0xc05b067c at uma_zalloc_arg+0x30<br> > #4 0xc0291760 at malloc+0x8c<br> > #5 0xc02920ec at reallocf+0x14<br> > #6 0xc02f8894 at devclass_add_device+0x1e8<br> > #7 0xc02f6c78 at make_device+0xe0<br> > #8 0xc02f6abc at device_add_child_ordered+0x30<br> > #9 0xc0156e0c at sdhci_card_task+0x238<br> > #10 0xc0324090 at taskqueue_run_locked+0x1b4<br> > #11 0xc0323ea0 at taskqueue_run+0x50<br> > #12 0xc0275f88 at ithread_loop+0x264<br> <br> Just use a regular taskqueue like taskqueue_thread instead of taskqueue_swi= ?<br> PCI hotplug defines its own thread taskqueue for adding and removing device= s.<br> <br> The bug is here, IMO.=C2=A0 Eventually new-bus will need some sort of topol= ogy<br> lock and that will have to be an sx lock, so this code needs to change<br> anyway.=C2=A0 The sound code that tries to frob devices with a regular mute= x<br> also needs to change.<br></blockquote></div></div><div dir=3D"auto"><br></d= iv><div dir=3D"auto">I should dust off the branch that i have this one. The= re's about a dozen places I had to change at the time...</div><div dir= =3D"auto"><br></div><div dir=3D"auto"><div class=3D"gmail_quote gmail_quote= _container"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo= rder-left:1px #ccc solid;padding-left:1ex"> In terms of taskqueue_swi, it's probably something that needs to go awa= y.<br> Generally speaking, code uses tasks for functions that need to sleep,<br> and taskqueue_swi breaks that.<br></blockquote></div></div><div dir=3D"auto= "><br></div><div dir=3D"auto">Its a holdover from spl days for sure.</div><= div dir=3D"auto"><br></div><div dir=3D"auto"><div class=3D"gmail_quote gmai= l_quote_container"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"> I will fix sdhci to use a proper taskqueue and then revert this commit.<br>= </blockquote></div></div><div dir=3D"auto"><br></div><div dir=3D"auto">Than= ks. You can add me to the review=C2=A0</div><div dir=3D"auto"><br></div><di= v dir=3D"auto">Warner</div><div dir=3D"auto"><div class=3D"gmail_quote gmai= l_quote_container"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"> -- <br> John Baldwin<br> <br> </blockquote></div></div></div> --0000000000008ba1a2062fadb614--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfr-1WTp0XoWYieHfo9GZ3Du6QKtyVzh_6aX%2BQnPHyN8ig>