Date: Mon, 5 Jan 2015 08:27:21 -0800 From: Adrian Chadd <adrian@freebsd.org> To: Hans Petter Selasky <hps@selasky.org> Cc: Konstantin Belousov <kostikbel@gmail.com>, markb@mellanox.com, FreeBSD Current <freebsd-current@freebsd.org> Subject: Re: [RFC] Start SMP subsystem earlier Message-ID: <CAJ-VmomKZVSzisQhPN077j8bV-N3_oZxmLoB4H0Pi-X6hHO4Yw@mail.gmail.com> In-Reply-To: <54AA9AF1.5020807@selasky.org> References: <54AA8F19.9030300@selasky.org> <20150105134316.GE42409@kib.kiev.ua> <54AA9AF1.5020807@selasky.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 5 January 2015 at 06:08, Hans Petter Selasky <hps@selasky.org> wrote: > On 01/05/15 14:43, Konstantin Belousov wrote: >> >> On Mon, Jan 05, 2015 at 02:18:17PM +0100, Hans Petter Selasky wrote: >>> >>> Hi, >>> >>> There is a limitiation on the number of interrupt vectors available when >>> only a single processor is running. To have more interrupts available we >>> need to start SMP earlier when building a monotolith kernel and not >>> loading drivers as modules. The driver in question is a network driver >>> and because it cannot be started after SI_SUB_ROOT_CONF due to PXE >>> support I see no other option than to move SI_SUB_SMP earlier. >>> >>> Suggested patch: >>> >>>> Index: sys/kernel.h >>>> =================================================================== >>>> --- sys/kernel.h (revision 276691) >>>> +++ sys/kernel.h (working copy) >>>> @@ -152,6 +152,7 @@ >>>> SI_SUB_KPROF = 0x9000000, /* kernel profiling*/ >>>> SI_SUB_KICK_SCHEDULER = 0xa000000, /* start the timeout >>>> events*/ >>>> SI_SUB_INT_CONFIG_HOOKS = 0xa800000, /* Interrupts enabled >>>> config */ >>>> + SI_SUB_SMP = 0xa850000, /* start the APs*/ >>>> SI_SUB_ROOT_CONF = 0xb000000, /* Find root devices */ >>>> SI_SUB_DUMP_CONF = 0xb200000, /* Find dump devices */ >>>> SI_SUB_RAID = 0xb380000, /* Configure GEOM >>>> classes */ >>>> @@ -165,7 +166,6 @@ >>>> SI_SUB_KTHREAD_BUF = 0xea00000, /* buffer daemon*/ >>>> SI_SUB_KTHREAD_UPDATE = 0xec00000, /* update daemon*/ >>>> SI_SUB_KTHREAD_IDLE = 0xee00000, /* idle procs*/ >>>> - SI_SUB_SMP = 0xf000000, /* start the APs*/ >>>> SI_SUB_RACCTD = 0xf100000, /* start racctd*/ >>>> SI_SUB_LAST = 0xfffffff /* final initialization >>>> */ >>>> }; >> >> Did you inspected all reordered sysinit routines and ensured that the >> reordering is safe ? At very least, SUB_SMP starts event timers, >> while KTHREAD_IDLE is about configuring some hardware which might >> be required/not ready for that. > > > Hi, > > I did not inspect everything myself yet regarding this change. That's why > I'm sending this e-mail out. The problem is simply that the total number of > interrupts appears to be limited by "APIC_NUM_IOINTS" and "NUM_IO_INTS" > which is per CPU from what I understand. Until SMP is activated the newbus > code is simply distributing the IRQ vectors on the available IRQs, then when > SMP is up it is re-shuffling them all. > > I was initially thinking that a hack might be possible, like using RF_SHARED > for the IRQ resource, but then noticed that we were using MSI interrupts, > which are not allocated in the same manner. > > The other issue is that the IRQs should be functional too, so that PXE boot > can work. > I'm also starting to see increasing amounts of wifi hardware that expects interrupts to be up and working during probe/attach. (I think i915kms has the same issue too, no?) -adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-VmomKZVSzisQhPN077j8bV-N3_oZxmLoB4H0Pi-X6hHO4Yw>