From owner-freebsd-current@FreeBSD.ORG Mon Jan 5 16:27:23 2015 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 480216F0 for ; Mon, 5 Jan 2015 16:27:23 +0000 (UTC) Received: from mail-wi0-x22f.google.com (mail-wi0-x22f.google.com [IPv6:2a00:1450:400c:c05::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CE71E2AA3 for ; Mon, 5 Jan 2015 16:27:22 +0000 (UTC) Received: by mail-wi0-f175.google.com with SMTP id l15so3640095wiw.8 for ; Mon, 05 Jan 2015 08:27:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=DXyLmzY/q5+PK4SX+AH3ynn8pe66dKfpfCH6pzGIl28=; b=hrU96sVuVV7CKFJvNrdysrDeB3cv2sAlU1dXS9pWFjw4HKppBfbPkzxp0Pw//W+/Dd 5apAdCXOv8eMLME5jIfuhvOLOQ80hTlXIjeWObY5LWAGHeTyG2vaXa6sk/RwGqdLCN3w Oz4iH/vE2T3TalqeGttwQ8cNWfnuDceMs4Ai7e9VjFykRx0MAcFlhTdBA1GWgTeZEjl1 DiC7U6JgHaJU7U2DfaqL08Byj23xx2OIBMb2DIF4/WTbaVOG+h5gD8f+1+S8rT9UyyOc uvJR9wE1A8psGbCPzdmQhNhWkid+ZhfDBRq4HQt+WdI7E1ZRpFYNNWqCylEapczb1HyU XGGA== MIME-Version: 1.0 X-Received: by 10.194.108.9 with SMTP id hg9mr177944715wjb.68.1420475241277; Mon, 05 Jan 2015 08:27:21 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.216.41.136 with HTTP; Mon, 5 Jan 2015 08:27:21 -0800 (PST) In-Reply-To: <54AA9AF1.5020807@selasky.org> References: <54AA8F19.9030300@selasky.org> <20150105134316.GE42409@kib.kiev.ua> <54AA9AF1.5020807@selasky.org> Date: Mon, 5 Jan 2015 08:27:21 -0800 X-Google-Sender-Auth: mVFdtvEqLcftTyAuZmfnfh8QmLE Message-ID: Subject: Re: [RFC] Start SMP subsystem earlier From: Adrian Chadd To: Hans Petter Selasky Content-Type: text/plain; charset=UTF-8 Cc: Konstantin Belousov , markb@mellanox.com, FreeBSD Current X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jan 2015 16:27:23 -0000 On 5 January 2015 at 06:08, Hans Petter Selasky wrote: > On 01/05/15 14:43, Konstantin Belousov wrote: >> >> On Mon, Jan 05, 2015 at 02:18:17PM +0100, Hans Petter Selasky wrote: >>> >>> Hi, >>> >>> There is a limitiation on the number of interrupt vectors available when >>> only a single processor is running. To have more interrupts available we >>> need to start SMP earlier when building a monotolith kernel and not >>> loading drivers as modules. The driver in question is a network driver >>> and because it cannot be started after SI_SUB_ROOT_CONF due to PXE >>> support I see no other option than to move SI_SUB_SMP earlier. >>> >>> Suggested patch: >>> >>>> Index: sys/kernel.h >>>> =================================================================== >>>> --- sys/kernel.h (revision 276691) >>>> +++ sys/kernel.h (working copy) >>>> @@ -152,6 +152,7 @@ >>>> SI_SUB_KPROF = 0x9000000, /* kernel profiling*/ >>>> SI_SUB_KICK_SCHEDULER = 0xa000000, /* start the timeout >>>> events*/ >>>> SI_SUB_INT_CONFIG_HOOKS = 0xa800000, /* Interrupts enabled >>>> config */ >>>> + SI_SUB_SMP = 0xa850000, /* start the APs*/ >>>> SI_SUB_ROOT_CONF = 0xb000000, /* Find root devices */ >>>> SI_SUB_DUMP_CONF = 0xb200000, /* Find dump devices */ >>>> SI_SUB_RAID = 0xb380000, /* Configure GEOM >>>> classes */ >>>> @@ -165,7 +166,6 @@ >>>> SI_SUB_KTHREAD_BUF = 0xea00000, /* buffer daemon*/ >>>> SI_SUB_KTHREAD_UPDATE = 0xec00000, /* update daemon*/ >>>> SI_SUB_KTHREAD_IDLE = 0xee00000, /* idle procs*/ >>>> - SI_SUB_SMP = 0xf000000, /* start the APs*/ >>>> SI_SUB_RACCTD = 0xf100000, /* start racctd*/ >>>> SI_SUB_LAST = 0xfffffff /* final initialization >>>> */ >>>> }; >> >> Did you inspected all reordered sysinit routines and ensured that the >> reordering is safe ? At very least, SUB_SMP starts event timers, >> while KTHREAD_IDLE is about configuring some hardware which might >> be required/not ready for that. > > > Hi, > > I did not inspect everything myself yet regarding this change. That's why > I'm sending this e-mail out. The problem is simply that the total number of > interrupts appears to be limited by "APIC_NUM_IOINTS" and "NUM_IO_INTS" > which is per CPU from what I understand. Until SMP is activated the newbus > code is simply distributing the IRQ vectors on the available IRQs, then when > SMP is up it is re-shuffling them all. > > I was initially thinking that a hack might be possible, like using RF_SHARED > for the IRQ resource, but then noticed that we were using MSI interrupts, > which are not allocated in the same manner. > > The other issue is that the IRQs should be functional too, so that PXE boot > can work. > I'm also starting to see increasing amounts of wifi hardware that expects interrupts to be up and working during probe/attach. (I think i915kms has the same issue too, no?) -adrian