From owner-freebsd-current@FreeBSD.ORG Mon Jan 5 13:43:23 2015 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 39BFE40A for ; Mon, 5 Jan 2015 13:43:23 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A060E64456 for ; Mon, 5 Jan 2015 13:43:22 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t05DhGdn083338 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 5 Jan 2015 15:43:16 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t05DhGdn083338 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id t05DhGj3083337; Mon, 5 Jan 2015 15:43:16 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 5 Jan 2015 15:43:16 +0200 From: Konstantin Belousov To: Hans Petter Selasky Subject: Re: [RFC] Start SMP subsystem earlier Message-ID: <20150105134316.GE42409@kib.kiev.ua> References: <54AA8F19.9030300@selasky.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <54AA8F19.9030300@selasky.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: markb@mellanox.com, FreeBSD Current X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jan 2015 13:43:23 -0000 On Mon, Jan 05, 2015 at 02:18:17PM +0100, Hans Petter Selasky wrote: > Hi, > > There is a limitiation on the number of interrupt vectors available when > only a single processor is running. To have more interrupts available we > need to start SMP earlier when building a monotolith kernel and not > loading drivers as modules. The driver in question is a network driver > and because it cannot be started after SI_SUB_ROOT_CONF due to PXE > support I see no other option than to move SI_SUB_SMP earlier. > > Suggested patch: > > > Index: sys/kernel.h > > =================================================================== > > --- sys/kernel.h (revision 276691) > > +++ sys/kernel.h (working copy) > > @@ -152,6 +152,7 @@ > > SI_SUB_KPROF = 0x9000000, /* kernel profiling*/ > > SI_SUB_KICK_SCHEDULER = 0xa000000, /* start the timeout events*/ > > SI_SUB_INT_CONFIG_HOOKS = 0xa800000, /* Interrupts enabled config */ > > + SI_SUB_SMP = 0xa850000, /* start the APs*/ > > SI_SUB_ROOT_CONF = 0xb000000, /* Find root devices */ > > SI_SUB_DUMP_CONF = 0xb200000, /* Find dump devices */ > > SI_SUB_RAID = 0xb380000, /* Configure GEOM classes */ > > @@ -165,7 +166,6 @@ > > SI_SUB_KTHREAD_BUF = 0xea00000, /* buffer daemon*/ > > SI_SUB_KTHREAD_UPDATE = 0xec00000, /* update daemon*/ > > SI_SUB_KTHREAD_IDLE = 0xee00000, /* idle procs*/ > > - SI_SUB_SMP = 0xf000000, /* start the APs*/ > > SI_SUB_RACCTD = 0xf100000, /* start racctd*/ > > SI_SUB_LAST = 0xfffffff /* final initialization */ > > }; Did you inspected all reordered sysinit routines and ensured that the reordering is safe ? At very least, SUB_SMP starts event timers, while KTHREAD_IDLE is about configuring some hardware which might be required/not ready for that. > > This fixes a problem for Mellanox drivers in the OFED layer. Possibly we > need to move the SMP even earlier to not miss the generic FreeBSD PCI > device enumeration or maybe this is not possible. Does anyone know how > early we can start SMP?