Date: Mon, 21 Mar 2016 15:34:40 -0700 From: John Baldwin <jhb@freebsd.org> To: "K. Macy" <kmacy@freebsd.org>, "arch@freebsd.org" <arch@freebsd.org> Subject: Re: Starting APs earlier during boot Message-ID: <4566552.97FFaNSfpg@ralph.baldwin.cx> In-Reply-To: <CAHM0Q_P_DZg9XpuP0ZRj0nBFhPxUK1VZ8SuEuaB4r9wmEnqJ2Q@mail.gmail.com> References: <1730061.8Ii36ORVKt@ralph.baldwin.cx> <2980696.6AEyEjetGn@ralph.baldwin.cx> <CAHM0Q_P_DZg9XpuP0ZRj0nBFhPxUK1VZ8SuEuaB4r9wmEnqJ2Q@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Friday, March 18, 2016 12:37:24 PM K. Macy wrote: > So none of these changes have been committed yet? > > I'm hitting hangs in USB on boot with recent HEAD and without having > investigating had thought this might be what exposed the problem. > > Thanks. I've committed some comestic ones (e.g., moving some SYSINITs from SI_SUB_SMP to SI_SUB_LAST), but nothing that should change actual behavior yet. > -M > > On Friday, March 18, 2016, John Baldwin <jhb@freebsd.org> wrote: > > > On Tuesday, February 16, 2016 12:50:22 PM John Baldwin wrote: > > > Currently the kernel bootstraps the non-boot processors fairly early in > > the > > > SI_SUB_CPU SYSINIT. The APs then spin waiting to be "released". We > > currently > > > release the APs as one of the last steps at SI_SUB_SMP. On the one hand > > this > > > removes much of the need for synchronization while SYSINITs are running > > since > > > SYSINITs basically assume they are single-threaded. However, it also > > enforces > > > some odd quirks. Several places that deal with per-CPU resources have to > > > split initialization up so that the BSP init happens in one SYSINIT and > > the > > > initialization of the APs happens in a second SYSINIT at SI_SUB_SMP. > > > > > > Another issue that is becoming more prominent on x86 (and probably will > > also > > > affect other platforms if it isn't already) is that to support working > > > interrupts for interrupt config hooks we bind all interrupts to the BSP > > during > > > boot and only distribute them among other CPUs near the end at > > SI_SUB_SMP. > > > This is especially problematic with drivers for modern hardware > > allocating > > > num(CPUs) interrupts (hoping to use one per CPU). On x86 we have aboug > > 190 > > > IDT vectors available for device interrupts, so in theory we should be > > able to > > > tolerate a lot of drivers doing this (e.g. 60 drivers could allocate 3 > > > interrupts for every CPU and we should still be fine). However, if you > > have, > > > say, 32 cores in a system, then you can only handle about 5 drivers doing > > > this before you run out of vectors on CPU 0. > > > > > > Longer term we would also like to eventually have most drivers attach in > > the > > > same environment during boot as during post-boot. Right now post-boot is > > > quite different as all CPUs are running, interrupts work, etc. One of > > the > > > goals of multipass support for new-bus is to help us get there by probing > > > enough hardware to get timers working and starting the scheduler before > > > probing the rest of the devices. That goal isn't quite realized yet. > > > > > > However, we can run a slightly simpler version of our scheduler before > > > timers are working. In fact, sleep/wakeup work just fine fairly early > > (we > > > allocate the necessary structures at SI_SUB_KMEM which is before the APs > > > are even started). Once idle threads are created and ready we could in > > > theory let the APs startup and run other threads. You just don't have > > working > > > timeouts. OTOH, you can sort of simulate timeouts if you modify the > > scheduler > > > to yield the CPU instead of blocking the thread for a sleep with a > > timeout. > > > The effect would be for threads that do sleeps with a timeout to fall > > back to > > > polling before timers are working. In practice, all of the early kernel > > > threads use sleeps without timeouts when idle so this doesn't really > > matter. > > > > After some more testing, I've simplified the early scheduler a bit. It no > > longer tries to simulate timeouts by just keeping the thread runnable. > > Instead, > > a sleep with a timeout just panics. However, it does still permit sleeps > > with > > infinite sleeps. Some code that uses a timeout really wants a timeout > > (note > > that pause() has a hack to fallback to DELAY() internally if cold is true > > for > > this reason). Instead, my feeling is that any kthreads that need timeouts > > to > > work need to defer their startup until SI_SUB_KICK_SCHEDULER. > > > > > However, I'd like feedback on the general idea and if it is acceptable > > I'd > > > like to coordinate testing with other platforms so this can go into the > > > tree. > > > > I don't think I've seen any objections? This does need more testing. I > > will > > update the patch to add a new EARLY_AP_STARTUP kernel option so this can be > > committed (but not yet enabled) allowing for easier testing (and allowing > > other platforms to catch up to x86). > > > > > The current changes are in the 'ap_startup' branch at > > github/bsdjhb/freebsd. > > > You can view them here: > > > > > > https://github.com/bsdjhb/freebsd/compare/master...bsdjhb:ap_startup > > > > -- > > John Baldwin > > _______________________________________________ > > freebsd-arch@freebsd.org <javascript:;> mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-arch > > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org > > <javascript:;>" > > -- John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4566552.97FFaNSfpg>