From owner-freebsd-current@FreeBSD.ORG Tue Jan 6 14:37:33 2015 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0FC4876D for ; Tue, 6 Jan 2015 14:37:33 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DD2AD6676F for ; Tue, 6 Jan 2015 14:37:32 +0000 (UTC) Received: from new-host.home (pool-173-70-85-31.nwrknj.fios.verizon.net [173.70.85.31]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 131FBB946; Tue, 6 Jan 2015 09:37:31 -0500 (EST) Message-ID: <54ABF32A.6010409@FreeBSD.org> Date: Tue, 06 Jan 2015 09:37:30 -0500 From: John Baldwin User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: Hans Petter Selasky , FreeBSD Current , markb@mellanox.com Subject: Re: [RFC] Start SMP subsystem earlier References: <54AA8F19.9030300@selasky.org> In-Reply-To: <54AA8F19.9030300@selasky.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 06 Jan 2015 09:37:31 -0500 (EST) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jan 2015 14:37:33 -0000 On 1/5/15 8:18 AM, Hans Petter Selasky wrote: > Hi, > > There is a limitiation on the number of interrupt vectors available when > only a single processor is running. To have more interrupts available we > need to start SMP earlier when building a monotolith kernel and not > loading drivers as modules. The driver in question is a network driver > and because it cannot be started after SI_SUB_ROOT_CONF due to PXE > support I see no other option than to move SI_SUB_SMP earlier. > > Suggested patch: > >> Index: sys/kernel.h >> =================================================================== >> --- sys/kernel.h (revision 276691) >> +++ sys/kernel.h (working copy) >> @@ -152,6 +152,7 @@ >> SI_SUB_KPROF = 0x9000000, /* kernel profiling*/ >> SI_SUB_KICK_SCHEDULER = 0xa000000, /* start the timeout >> events*/ >> SI_SUB_INT_CONFIG_HOOKS = 0xa800000, /* Interrupts enabled >> config */ >> + SI_SUB_SMP = 0xa850000, /* start the APs*/ >> SI_SUB_ROOT_CONF = 0xb000000, /* Find root devices */ >> SI_SUB_DUMP_CONF = 0xb200000, /* Find dump devices */ >> SI_SUB_RAID = 0xb380000, /* Configure GEOM classes */ >> @@ -165,7 +166,6 @@ >> SI_SUB_KTHREAD_BUF = 0xea00000, /* buffer daemon*/ >> SI_SUB_KTHREAD_UPDATE = 0xec00000, /* update daemon*/ >> SI_SUB_KTHREAD_IDLE = 0xee00000, /* idle procs*/ >> - SI_SUB_SMP = 0xf000000, /* start the APs*/ >> SI_SUB_RACCTD = 0xf100000, /* start racctd*/ >> SI_SUB_LAST = 0xfffffff /* final initialization */ >> }; > > This fixes a problem for Mellanox drivers in the OFED layer. Possibly we > need to move the SMP even earlier to not miss the generic FreeBSD PCI > device enumeration or maybe this is not possible. Does anyone know how > early we can start SMP? We need a lot more work before this is ready. This is one of the goals of the multipass new-bus stuff. In particular, we have to enumerate enough devices to bring event timer hardware up so that timer interrupts work so that tsleep() will actually sleep. In addition, we also need idle threads created and working before APs are started as otherwise they will have no thread to run initially. This is certainly a desired feature, but it is not as simple as moving the sysinit up I'm afraid. -- John Baldwin