From owner-freebsd-current@FreeBSD.ORG Tue Jan 6 15:55:20 2015 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1CCC3ECD; Tue, 6 Jan 2015 15:55:20 +0000 (UTC) Received: from mho-02-ewr.mailhop.org (mho-02-ewr.mailhop.org [204.13.248.72]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E678127A2; Tue, 6 Jan 2015 15:55:19 +0000 (UTC) Received: from [73.34.117.227] (helo=ilsoft.org) by mho-02-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1Y8WTV-000KPb-KP; Tue, 06 Jan 2015 15:55:18 +0000 Received: from revolution.hippie.lan (revolution.hippie.lan [172.22.42.240]) by ilsoft.org (8.14.9/8.14.9) with ESMTP id t06FtFAG040316; Tue, 6 Jan 2015 08:55:15 -0700 (MST) (envelope-from ian@freebsd.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 73.34.117.227 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX19VA/Av/rKVML1X7clwPwxr Message-ID: <1420559715.14601.25.camel@freebsd.org> Subject: Re: [RFC] Start SMP subsystem earlier From: Ian Lepore To: John Baldwin Date: Tue, 06 Jan 2015 08:55:15 -0700 In-Reply-To: <54ABF32A.6010409@FreeBSD.org> References: <54AA8F19.9030300@selasky.org> <54ABF32A.6010409@FreeBSD.org> Content-Type: text/plain; charset="us-ascii" X-Mailer: Evolution 3.12.8 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Cc: Hans Petter Selasky , markb@mellanox.com, FreeBSD Current X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jan 2015 15:55:20 -0000 On Tue, 2015-01-06 at 09:37 -0500, John Baldwin wrote: > On 1/5/15 8:18 AM, Hans Petter Selasky wrote: > > Hi, > > > > There is a limitiation on the number of interrupt vectors available when > > only a single processor is running. To have more interrupts available we > > need to start SMP earlier when building a monotolith kernel and not > > loading drivers as modules. The driver in question is a network driver > > and because it cannot be started after SI_SUB_ROOT_CONF due to PXE > > support I see no other option than to move SI_SUB_SMP earlier. > > > > Suggested patch: > > > >>[...] > > > > This fixes a problem for Mellanox drivers in the OFED layer. Possibly we > > need to move the SMP even earlier to not miss the generic FreeBSD PCI > > device enumeration or maybe this is not possible. Does anyone know how > > early we can start SMP? > > We need a lot more work before this is ready. This is one of the goals > of the multipass new-bus stuff. In particular, we have to enumerate > enough devices to bring event timer hardware up so that timer interrupts > work so that tsleep() will actually sleep. In addition, we also need > idle threads created and working before APs are started as otherwise > they will have no thread to run initially. This is certainly a desired > feature, but it is not as simple as moving the sysinit up I'm afraid. > Just an FYI, the ARM world is now using the multipass newbus stuff. It works well, with some quirks... The predefined pass names don't always makes sense for the arm world. There aren't enough predefined pass names and even though the number space for them is 4 billion wide all the predefined names are in the range < 100 and separated by only 10 so it's tricky to wedge things between the existing names. The strangest bit is when you have interdependent drivers at different early pass numbers. Sometimes it's necessary to do almost nothing in the attach() routine and do all the real attach-time type stuff in a bus_new_pass() routine after the pass number becomes high enough that your co-dependent driver peers are available. -- Ian