From owner-freebsd-current@FreeBSD.ORG Tue Jan 6 15:12:17 2015 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D7A897E7; Tue, 6 Jan 2015 15:12:17 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7B88F1E59; Tue, 6 Jan 2015 15:12:17 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t06FCBOD072920 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 6 Jan 2015 17:12:12 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t06FCBOD072920 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id t06FCBKd072919; Tue, 6 Jan 2015 17:12:11 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 6 Jan 2015 17:12:11 +0200 From: Konstantin Belousov To: John Baldwin Subject: Re: [RFC] Start SMP subsystem earlier Message-ID: <20150106151211.GH42409@kib.kiev.ua> References: <54AA8F19.9030300@selasky.org> <54ABF32A.6010409@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <54ABF32A.6010409@FreeBSD.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: Hans Petter Selasky , markb@mellanox.com, FreeBSD Current X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jan 2015 15:12:18 -0000 On Tue, Jan 06, 2015 at 09:37:30AM -0500, John Baldwin wrote: > On 1/5/15 8:18 AM, Hans Petter Selasky wrote: > > Hi, > > > > There is a limitiation on the number of interrupt vectors available when > > only a single processor is running. To have more interrupts available we > > need to start SMP earlier when building a monotolith kernel and not > > loading drivers as modules. The driver in question is a network driver > > and because it cannot be started after SI_SUB_ROOT_CONF due to PXE > > support I see no other option than to move SI_SUB_SMP earlier. > > > > Suggested patch: > > > >> Index: sys/kernel.h > >> =================================================================== > >> --- sys/kernel.h (revision 276691) > >> +++ sys/kernel.h (working copy) > >> @@ -152,6 +152,7 @@ > >> SI_SUB_KPROF = 0x9000000, /* kernel profiling*/ > >> SI_SUB_KICK_SCHEDULER = 0xa000000, /* start the timeout > >> events*/ > >> SI_SUB_INT_CONFIG_HOOKS = 0xa800000, /* Interrupts enabled > >> config */ > >> + SI_SUB_SMP = 0xa850000, /* start the APs*/ > >> SI_SUB_ROOT_CONF = 0xb000000, /* Find root devices */ > >> SI_SUB_DUMP_CONF = 0xb200000, /* Find dump devices */ > >> SI_SUB_RAID = 0xb380000, /* Configure GEOM classes */ > >> @@ -165,7 +166,6 @@ > >> SI_SUB_KTHREAD_BUF = 0xea00000, /* buffer daemon*/ > >> SI_SUB_KTHREAD_UPDATE = 0xec00000, /* update daemon*/ > >> SI_SUB_KTHREAD_IDLE = 0xee00000, /* idle procs*/ > >> - SI_SUB_SMP = 0xf000000, /* start the APs*/ > >> SI_SUB_RACCTD = 0xf100000, /* start racctd*/ > >> SI_SUB_LAST = 0xfffffff /* final initialization */ > >> }; > > > > This fixes a problem for Mellanox drivers in the OFED layer. Possibly we > > need to move the SMP even earlier to not miss the generic FreeBSD PCI > > device enumeration or maybe this is not possible. Does anyone know how > > early we can start SMP? > > We need a lot more work before this is ready. This is one of the goals > of the multipass new-bus stuff. In particular, we have to enumerate > enough devices to bring event timer hardware up so that timer interrupts > work so that tsleep() will actually sleep. In addition, we also need > idle threads created and working before APs are started as otherwise > they will have no thread to run initially. This is certainly a desired > feature, but it is not as simple as moving the sysinit up I'm afraid. > I believe that idle threads are still created before the APs start with the patch posted, this was the thing I checked first. It is SUB_SCHED_IDLE, which is done long before even drivers are configured.