Date: Thu, 6 Mar 2025 06:56:27 -0800 From: Warner Losh <imp@bsdimp.com> To: John Baldwin <jhb@freebsd.org> Cc: Mateusz Guzik <mjguzik@gmail.com>, Zhenlei Huang <zlei@freebsd.org>, Mateusz Guzik <mjg@freebsd.org>, src-committers <src-committers@freebsd.org>, "<dev-commits-src-all@freebsd.org>" <dev-commits-src-all@freebsd.org>, Warner Losh <imp@freebsd.org>, "<dev-commits-src-main@freebsd.org>" <dev-commits-src-main@freebsd.org> Subject: Re: git: 234683726708 - main - devclass: make devclass_alloc_unit use M_NOWAIT Message-ID: <CANCZdfr-1WTp0XoWYieHfo9GZ3Du6QKtyVzh_6aX%2BQnPHyN8ig@mail.gmail.com> In-Reply-To: <d09b8cd1-ac71-43a4-9625-e796fd142f16@FreeBSD.org> References: <202503061103.526B32Id022652@gitrepo.freebsd.org> <F1B3652E-0D0C-402A-8509-D510992DAC15@FreeBSD.org> <CAGudoHF=eRaHCcjRrvd4sG4-OBu0GrmVRpiHEeU1ayG=M9oXrg@mail.gmail.com> <d09b8cd1-ac71-43a4-9625-e796fd142f16@FreeBSD.org>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] On Thu, Mar 6, 2025, 5:33 AM John Baldwin <jhb@freebsd.org> wrote: > On 3/6/25 06:35, Mateusz Guzik wrote: > > On Thu, Mar 6, 2025 at 12:32 PM Zhenlei Huang <zlei@freebsd.org> wrote: > >> > >> > >> > >> On Mar 6, 2025, at 7:03 PM, Mateusz Guzik <mjg@FreeBSD.org> wrote: > >> > >> The branch main has been updated by mjg: > >> > >> URL: > https://cgit.FreeBSD.org/src/commit/?id=234683726708cf5212d672d676d30056d4133859 > >> > >> commit 234683726708cf5212d672d676d30056d4133859 > >> Author: Mateusz Guzik <mjg@FreeBSD.org> > >> AuthorDate: 2025-03-06 11:01:49 +0000 > >> Commit: Mateusz Guzik <mjg@FreeBSD.org> > >> CommitDate: 2025-03-06 11:01:49 +0000 > >> > >> devclass: make devclass_alloc_unit use M_NOWAIT > >> > >> The only caller already does this. > >> > >> The routine can be called with a mutex held making M_WAITOK illegal. > >> > >> Sponsored by: Rubicon Communications, LLC ("Netgate") > >> --- > >> sys/kern/subr_bus.c | 8 ++++++-- > >> 1 file changed, 6 insertions(+), 2 deletions(-) > >> > >> diff --git a/sys/kern/subr_bus.c b/sys/kern/subr_bus.c > >> index 9506e471705c..0422352bba51 100644 > >> --- a/sys/kern/subr_bus.c > >> +++ b/sys/kern/subr_bus.c > >> @@ -1208,6 +1208,7 @@ devclass_get_sysctl_tree(devclass_t dc) > >> static int > >> devclass_alloc_unit(devclass_t dc, device_t dev, int *unitp) > >> { > >> + device_t *devices; > >> const char *s; > >> int unit = *unitp; > >> > >> @@ -1264,8 +1265,11 @@ devclass_alloc_unit(devclass_t dc, device_t dev, > int *unitp) > >> int newsize; > >> > >> newsize = unit + 1; > >> - dc->devices = reallocf(dc->devices, > >> - newsize * sizeof(*dc->devices), M_BUS, M_WAITOK); > >> + devices = reallocf(dc->devices, > >> + newsize * sizeof(*dc->devices), M_BUS, M_NOWAIT); > >> > >> > >> I'd recommend against this. From the commit message of f3d3c63442ff, > Warner said, > >>> In addition, transition to M_WAITOK since this is a sleepable context > >> So, the M_WAITOK is intentional. > >> > >> Rather than reverting this, the caller devclass_add_device() should use > M_WAITOK. > >> > > > > Per my commit message this is callable from a *NOT* sleepable context. > > > > Here is a splat we got at Netgate: > > > > uma_zalloc_debug: zone "malloc-16" with the following non-sleepable > locks held: > > exclusive sleep mutex SD slot mtx (sdhci) r = 0 (0xd8dec028) locked @ > > > /var/jenkins/workspace/pfSense-Plus-snapshots-25_03-main/sources/FreeBSD-src-plus-RELENG_25_03/sys/dev/sdhci/sdhci.c:688 > > stack backtrace: > > #0 0xc0330ebc at witness_debugger+0x78 > > #1 0xc033217c at witness_warn+0x428 > > #2 0xc05b0a58 at uma_zalloc_debug+0x34 > > #3 0xc05b067c at uma_zalloc_arg+0x30 > > #4 0xc0291760 at malloc+0x8c > > #5 0xc02920ec at reallocf+0x14 > > #6 0xc02f8894 at devclass_add_device+0x1e8 > > #7 0xc02f6c78 at make_device+0xe0 > > #8 0xc02f6abc at device_add_child_ordered+0x30 > > #9 0xc0156e0c at sdhci_card_task+0x238 > > #10 0xc0324090 at taskqueue_run_locked+0x1b4 > > #11 0xc0323ea0 at taskqueue_run+0x50 > > #12 0xc0275f88 at ithread_loop+0x264 > > Just use a regular taskqueue like taskqueue_thread instead of > taskqueue_swi? > PCI hotplug defines its own thread taskqueue for adding and removing > devices. > > The bug is here, IMO. Eventually new-bus will need some sort of topology > lock and that will have to be an sx lock, so this code needs to change > anyway. The sound code that tries to frob devices with a regular mutex > also needs to change. > I should dust off the branch that i have this one. There's about a dozen places I had to change at the time... In terms of taskqueue_swi, it's probably something that needs to go away. > Generally speaking, code uses tasks for functions that need to sleep, > and taskqueue_swi breaks that. > Its a holdover from spl days for sure. I will fix sdhci to use a proper taskqueue and then revert this commit. > Thanks. You can add me to the review Warner > -- > John Baldwin > > [-- Attachment #2 --] <div dir="auto"><div><br><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Thu, Mar 6, 2025, 5:33 AM John Baldwin <<a href="mailto:jhb@freebsd.org">jhb@freebsd.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 3/6/25 06:35, Mateusz Guzik wrote:<br> > On Thu, Mar 6, 2025 at 12:32 PM Zhenlei Huang <<a href="mailto:zlei@freebsd.org" target="_blank" rel="noreferrer">zlei@freebsd.org</a>> wrote:<br> >><br> >><br> >><br> >> On Mar 6, 2025, at 7:03 PM, Mateusz Guzik <mjg@FreeBSD.org> wrote:<br> >><br> >> The branch main has been updated by mjg:<br> >><br> >> URL: <a href="https://cgit.FreeBSD.org/src/commit/?id=234683726708cf5212d672d676d30056d4133859" rel="noreferrer noreferrer" target="_blank">https://cgit.FreeBSD.org/src/commit/?id=234683726708cf5212d672d676d30056d4133859</a><br>; >><br> >> commit 234683726708cf5212d672d676d30056d4133859<br> >> Author: Mateusz Guzik <mjg@FreeBSD.org><br> >> AuthorDate: 2025-03-06 11:01:49 +0000<br> >> Commit: Mateusz Guzik <mjg@FreeBSD.org><br> >> CommitDate: 2025-03-06 11:01:49 +0000<br> >><br> >> devclass: make devclass_alloc_unit use M_NOWAIT<br> >><br> >> The only caller already does this.<br> >><br> >> The routine can be called with a mutex held making M_WAITOK illegal.<br> >><br> >> Sponsored by: Rubicon Communications, LLC ("Netgate")<br> >> ---<br> >> sys/kern/subr_bus.c | 8 ++++++--<br> >> 1 file changed, 6 insertions(+), 2 deletions(-)<br> >><br> >> diff --git a/sys/kern/subr_bus.c b/sys/kern/subr_bus.c<br> >> index 9506e471705c..0422352bba51 100644<br> >> --- a/sys/kern/subr_bus.c<br> >> +++ b/sys/kern/subr_bus.c<br> >> @@ -1208,6 +1208,7 @@ devclass_get_sysctl_tree(devclass_t dc)<br> >> static int<br> >> devclass_alloc_unit(devclass_t dc, device_t dev, int *unitp)<br> >> {<br> >> + device_t *devices;<br> >> const char *s;<br> >> int unit = *unitp;<br> >><br> >> @@ -1264,8 +1265,11 @@ devclass_alloc_unit(devclass_t dc, device_t dev, int *unitp)<br> >> int newsize;<br> >><br> >> newsize = unit + 1;<br> >> - dc->devices = reallocf(dc->devices,<br> >> - newsize * sizeof(*dc->devices), M_BUS, M_WAITOK);<br> >> + devices = reallocf(dc->devices,<br> >> + newsize * sizeof(*dc->devices), M_BUS, M_NOWAIT);<br> >><br> >><br> >> I'd recommend against this. From the commit message of f3d3c63442ff, Warner said,<br> >>> In addition, transition to M_WAITOK since this is a sleepable context<br> >> So, the M_WAITOK is intentional.<br> >><br> >> Rather than reverting this, the caller devclass_add_device() should use M_WAITOK.<br> >><br> > <br> > Per my commit message this is callable from a *NOT* sleepable context.<br> > <br> > Here is a splat we got at Netgate:<br> > <br> > uma_zalloc_debug: zone "malloc-16" with the following non-sleepable locks held:<br> > exclusive sleep mutex SD slot mtx (sdhci) r = 0 (0xd8dec028) locked @<br> > /var/jenkins/workspace/pfSense-Plus-snapshots-25_03-main/sources/FreeBSD-src-plus-RELENG_25_03/sys/dev/sdhci/sdhci.c:688<br> > stack backtrace:<br> > #0 0xc0330ebc at witness_debugger+0x78<br> > #1 0xc033217c at witness_warn+0x428<br> > #2 0xc05b0a58 at uma_zalloc_debug+0x34<br> > #3 0xc05b067c at uma_zalloc_arg+0x30<br> > #4 0xc0291760 at malloc+0x8c<br> > #5 0xc02920ec at reallocf+0x14<br> > #6 0xc02f8894 at devclass_add_device+0x1e8<br> > #7 0xc02f6c78 at make_device+0xe0<br> > #8 0xc02f6abc at device_add_child_ordered+0x30<br> > #9 0xc0156e0c at sdhci_card_task+0x238<br> > #10 0xc0324090 at taskqueue_run_locked+0x1b4<br> > #11 0xc0323ea0 at taskqueue_run+0x50<br> > #12 0xc0275f88 at ithread_loop+0x264<br> <br> Just use a regular taskqueue like taskqueue_thread instead of taskqueue_swi?<br> PCI hotplug defines its own thread taskqueue for adding and removing devices.<br> <br> The bug is here, IMO. Eventually new-bus will need some sort of topology<br> lock and that will have to be an sx lock, so this code needs to change<br> anyway. The sound code that tries to frob devices with a regular mutex<br> also needs to change.<br></blockquote></div></div><div dir="auto"><br></div><div dir="auto">I should dust off the branch that i have this one. There's about a dozen places I had to change at the time...</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote gmail_quote_container"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> In terms of taskqueue_swi, it's probably something that needs to go away.<br> Generally speaking, code uses tasks for functions that need to sleep,<br> and taskqueue_swi breaks that.<br></blockquote></div></div><div dir="auto"><br></div><div dir="auto">Its a holdover from spl days for sure.</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote gmail_quote_container"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> I will fix sdhci to use a proper taskqueue and then revert this commit.<br></blockquote></div></div><div dir="auto"><br></div><div dir="auto">Thanks. You can add me to the review </div><div dir="auto"><br></div><div dir="auto">Warner</div><div dir="auto"><div class="gmail_quote gmail_quote_container"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> -- <br> John Baldwin<br> <br> </blockquote></div></div></div>help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfr-1WTp0XoWYieHfo9GZ3Du6QKtyVzh_6aX%2BQnPHyN8ig>