From owner-freebsd-arch@freebsd.org Tue Mar 15 21:46:58 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BA97AAD20E7; Tue, 15 Mar 2016 21:46:58 +0000 (UTC) (envelope-from mokhi64@gmail.com) Received: from mail-lb0-x234.google.com (mail-lb0-x234.google.com [IPv6:2a00:1450:4010:c04::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 475CE645; Tue, 15 Mar 2016 21:46:58 +0000 (UTC) (envelope-from mokhi64@gmail.com) Received: by mail-lb0-x234.google.com with SMTP id x1so35920592lbj.3; Tue, 15 Mar 2016 14:46:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:cc; bh=/HP/5rDn3uvkcsajgozOrVapxOJcoUss+QlM4nxBR+M=; b=LR2qmWp3VGq0PRcirCaVO/apMo31NT2EW6rKC8m3k0tuDbZ5m6iNguCeZORUe0hkzC NyLP7DaAE5Uof36cZmMy4Xa7eAFBEQzpG6VoSuBiYL+XRSuHqbXEQrYaE5hQexpRO8H9 W6tE7h/8TAdB8w0U6f/3SEsSpp4PXe0rJ9jPGw7mAouue3wRRe57u3bVxo6y/0KZ7fUp 0Y3KNBJUrqZNgn9iiAN4tSn7BDPBSdQ9NF2NoX0WaaYRUjAPnccyyw6wR0DHtV03EcSc 9TdLNVEhomcgLYyxA+ZMi1n0vsBLL39txI2NspWVizBy2I/vTx1RhdR3VxGHwCguMuSQ CXQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to:cc; bh=/HP/5rDn3uvkcsajgozOrVapxOJcoUss+QlM4nxBR+M=; b=W7T21eUDzeuejBlq1yInNzUCwBiAN7/V1n/C9BbckEb+wv4i0BOCpbgawv/cXmcVOM E26/6xCig9xFinQx4xnXC6fKr4ZkyvJMMxNCK+ZQpLiRRLOa1C+ZUX/4lF4ehGb91G3G c0YHaYjMX1CdZH/nPsDEFHr7xSnCOpiVHXpiGAVAF4EzNQ6jyro6MYdbS3tRbXYnkiL1 tRxee03swR24MTK11dRNJGcIaHJofMBXr+d6n1tJQ+LovGaO8wNg7csZkaPz210U5KmS Sfj6/Snho+UHPwsikE5GEvLwXHo9uQ1H3I5yNb61NLZ65npBknsfonaqt2taKGY5tyzG 59XA== X-Gm-Message-State: AD7BkJKVx95xC2QCjlJxUpu1y/8LWHgzm0NTAXMzJcCOjkOYnkEvq5OhLDlX06pjnDkvUExa66eneMriKn7OVQ== MIME-Version: 1.0 X-Received: by 10.112.77.138 with SMTP id s10mr92223lbw.125.1458078416306; Tue, 15 Mar 2016 14:46:56 -0700 (PDT) Received: by 10.25.139.68 with HTTP; Tue, 15 Mar 2016 14:46:56 -0700 (PDT) Date: Wed, 16 Mar 2016 01:16:56 +0330 Message-ID: Subject: Hi, a question about "struct sysentvc" From: mokhi To: freebsd-arch@freebsd.org Cc: freebsd-emulation@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Mar 2016 21:46:58 -0000 Hi guys. I was reading/studying code of Elf image activator to see how binary stars on FreeBSD. On included file (sysent.h), I saw 3 declarations: extern struct sysentvec aout_sysvec; extern struct sysentvec elf_freebsd_sysvec; extern struct sysentvec null_sysvec; I looked at their definitions too (though i couldn't find elf_freebsd_sysvec, where's it ?). I have some (maybe simple, so sorry for it :D) questions. 1) what's sysentvec for, basically ? and in which concept it connects to image-activators (such as ELF image activator [or a.out's]) 2) is there any reference/guide to see how should/can we define a "struct sysentvec" for new format ? FWIW, I'm also a little familiar with NetBSD way of binary running/compat layer devel. Thanks and thousands of regards, Mokhi. From owner-freebsd-arch@freebsd.org Wed Mar 16 00:25:30 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D5620AD15A4; Wed, 16 Mar 2016 00:25:30 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5846416F; Wed, 16 Mar 2016 00:25:30 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u2G0POi3051278 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 16 Mar 2016 02:25:24 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u2G0POi3051278 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u2G0POq4051274; Wed, 16 Mar 2016 02:25:24 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 16 Mar 2016 02:25:24 +0200 From: Konstantin Belousov To: mokhi Cc: freebsd-arch@freebsd.org, freebsd-emulation@freebsd.org Subject: Re: Hi, a question about "struct sysentvc" Message-ID: <20160316002524.GA1741@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Mar 2016 00:25:30 -0000 On Wed, Mar 16, 2016 at 01:16:56AM +0330, mokhi wrote: > Hi guys. > I was reading/studying code of Elf image activator to see how binary > stars on FreeBSD. > On included file (sysent.h), I saw 3 declarations: > extern struct sysentvec aout_sysvec; > extern struct sysentvec elf_freebsd_sysvec; > extern struct sysentvec null_sysvec; > > I looked at their definitions too (though i couldn't find > elf_freebsd_sysvec, where's it ?). It is in sys///elf_machdep.c. Grep would find it to you. > I have some (maybe simple, so sorry for it :D) questions. > > 1) what's sysentvec for, basically ? and in which concept it connects > to image-activators (such as ELF image activator [or a.out's]) The structure describes an ABI to kernel. Consider it the object in C++ sense which provides implementations of the ABI-specific operations from the kernel side. Image activator sets curproc->p_sysent for the process at exec(2) time according to the information specified in the binary, which effectively determines the kernel-side of the ABI for the process. > > 2) is there any reference/guide to see how should/can we define a > "struct sysentvec" for new format ? There is no guide, read the code. From owner-freebsd-arch@freebsd.org Wed Mar 16 12:18:16 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BF5AEAD084D; Wed, 16 Mar 2016 12:18:16 +0000 (UTC) (envelope-from mokhi64@gmail.com) Received: from mail-lb0-x22f.google.com (mail-lb0-x22f.google.com [IPv6:2a00:1450:4010:c04::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 497D0650; Wed, 16 Mar 2016 12:18:16 +0000 (UTC) (envelope-from mokhi64@gmail.com) Received: by mail-lb0-x22f.google.com with SMTP id x1so46496153lbj.3; Wed, 16 Mar 2016 05:18:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=aetL0lzuA4rxsOA78TtMCAKpLnnOgadEUvGrFhgbP0w=; b=cbTLbg4Gys68smLxPnFTcqrC0MiWnQfP2GumSaopZbqUaeVOBxiOqDLJBAB3bAw6NH 8x9dKND4s1oOyI2jn0yv8nX3PM3Vd2ThLabPMJ1QBtCiCLsORyOjdBdlp2F8eLCDHrfT K0WMTbQ+eYMk++nweyEFseb1wElOwg4oQwvH5IXXkmRK/v1y1oBJbw2uyqBGMB4+tZXu ScAymOGq3XDiG0bHB5yk4xDqYtDk7OJQGrfW4zBwBxDRWkHro06W4wTmdUFY+u+uxpus wox3XVlTHBRS2W9Wc2LIk04vApSfA2Lr3R6ZCn04J6ivH/K0LW5xWSeHMDOiznZm7ljI Yhiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=aetL0lzuA4rxsOA78TtMCAKpLnnOgadEUvGrFhgbP0w=; b=InqW/Xrl1u1nYFR6W1SSk8iNfML9qiJRA52xu7q6MQ9cAlcl12H7IHh1eiKg9rl+xz vKARZBEo005/qUgLWa90zZ/ELADNk9walPwQQ5ueh+qwDBwKP+4obgOqOcZvA700B4wM pHr4VUK61/A0q8AH16XpMo1uS8hnpWb/Z67rKdY3iazIL1ftm9TKbB7DifrFwWrs02R+ uEpj+57+itocR0UTm+pzGvBo2TawytaaOMBYmotDTnJSeh25QKrYu8BIndkZkbZ4L8YS ReANR+X2OblKHsPts4o3htwgtKEL+IxZfCvbj3KNzo0OeLLdChEdgFkDz2NKWKbyUZTu WOGQ== X-Gm-Message-State: AD7BkJJKb1Fnt1Lpqv+gBMzHHoj91dMLwzNZWVk2vJ4V9x4yKMr5ZJSKEtgkwPldxXzjJt0CyKI2pwIuIE3QwQ== MIME-Version: 1.0 X-Received: by 10.112.62.165 with SMTP id z5mr1046567lbr.89.1458130693945; Wed, 16 Mar 2016 05:18:13 -0700 (PDT) Received: by 10.25.139.68 with HTTP; Wed, 16 Mar 2016 05:18:13 -0700 (PDT) In-Reply-To: <20160316002524.GA1741@kib.kiev.ua> References: <20160316002524.GA1741@kib.kiev.ua> Date: Wed, 16 Mar 2016 15:48:13 +0330 Message-ID: Subject: Re: Hi, a question about "struct sysentvc" From: mokhi To: Konstantin Belousov , kib@freebsd.org Cc: freebsd-arch@freebsd.org, freebsd-emulation@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Mar 2016 12:18:16 -0000 Thanks for your answer :D On 3/16/16, Konstantin Belousov wrote: > There is no guide, read the code. Elf code is better in help ? or aout ? Regards, Mokhi. From owner-freebsd-arch@freebsd.org Wed Mar 16 15:40:34 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 21040AD3367; Wed, 16 Mar 2016 15:40:34 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AFA32834; Wed, 16 Mar 2016 15:40:33 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u2GFeNhs074825 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 16 Mar 2016 17:40:24 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u2GFeNhs074825 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u2GFeNkn074821; Wed, 16 Mar 2016 17:40:23 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 16 Mar 2016 17:40:23 +0200 From: Konstantin Belousov To: mokhi Cc: freebsd-arch@freebsd.org, freebsd-emulation@freebsd.org Subject: Re: Hi, a question about "struct sysentvc" Message-ID: <20160316154023.GL1741@kib.kiev.ua> References: <20160316002524.GA1741@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Mar 2016 15:40:34 -0000 On Wed, Mar 16, 2016 at 03:48:13PM +0330, mokhi wrote: > Thanks for your answer :D > > On 3/16/16, Konstantin Belousov wrote: > > There is no guide, read the code. > > Elf code is better in help ? or aout ? You need to read all the involved code to understand the interface. You could start with a.out part since it is easier, but the point is that you must read the callers code to see what for are the sysent methods. From owner-freebsd-arch@freebsd.org Fri Mar 18 19:02:45 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA0C7AD44E6; Fri, 18 Mar 2016 19:02:45 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 862BE1BE8; Fri, 18 Mar 2016 19:02:45 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id C5F39B94B; Fri, 18 Mar 2016 15:02:43 -0400 (EDT) From: John Baldwin To: freebsd-arch@freebsd.org Cc: arch@freebsd.org Subject: Re: Starting APs earlier during boot Date: Fri, 18 Mar 2016 12:02:30 -0700 Message-ID: <2980696.6AEyEjetGn@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; ) In-Reply-To: <1730061.8Ii36ORVKt@ralph.baldwin.cx> References: <1730061.8Ii36ORVKt@ralph.baldwin.cx> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 18 Mar 2016 15:02:43 -0400 (EDT) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Mar 2016 19:02:45 -0000 On Tuesday, February 16, 2016 12:50:22 PM John Baldwin wrote: > Currently the kernel bootstraps the non-boot processors fairly early in the > SI_SUB_CPU SYSINIT. The APs then spin waiting to be "released". We currently > release the APs as one of the last steps at SI_SUB_SMP. On the one hand this > removes much of the need for synchronization while SYSINITs are running since > SYSINITs basically assume they are single-threaded. However, it also enforces > some odd quirks. Several places that deal with per-CPU resources have to > split initialization up so that the BSP init happens in one SYSINIT and the > initialization of the APs happens in a second SYSINIT at SI_SUB_SMP. > > Another issue that is becoming more prominent on x86 (and probably will also > affect other platforms if it isn't already) is that to support working > interrupts for interrupt config hooks we bind all interrupts to the BSP during > boot and only distribute them among other CPUs near the end at SI_SUB_SMP. > This is especially problematic with drivers for modern hardware allocating > num(CPUs) interrupts (hoping to use one per CPU). On x86 we have aboug 190 > IDT vectors available for device interrupts, so in theory we should be able to > tolerate a lot of drivers doing this (e.g. 60 drivers could allocate 3 > interrupts for every CPU and we should still be fine). However, if you have, > say, 32 cores in a system, then you can only handle about 5 drivers doing > this before you run out of vectors on CPU 0. > > Longer term we would also like to eventually have most drivers attach in the > same environment during boot as during post-boot. Right now post-boot is > quite different as all CPUs are running, interrupts work, etc. One of the > goals of multipass support for new-bus is to help us get there by probing > enough hardware to get timers working and starting the scheduler before > probing the rest of the devices. That goal isn't quite realized yet. > > However, we can run a slightly simpler version of our scheduler before > timers are working. In fact, sleep/wakeup work just fine fairly early (we > allocate the necessary structures at SI_SUB_KMEM which is before the APs > are even started). Once idle threads are created and ready we could in > theory let the APs startup and run other threads. You just don't have working > timeouts. OTOH, you can sort of simulate timeouts if you modify the scheduler > to yield the CPU instead of blocking the thread for a sleep with a timeout. > The effect would be for threads that do sleeps with a timeout to fall back to > polling before timers are working. In practice, all of the early kernel > threads use sleeps without timeouts when idle so this doesn't really matter. After some more testing, I've simplified the early scheduler a bit. It no longer tries to simulate timeouts by just keeping the thread runnable. Instead, a sleep with a timeout just panics. However, it does still permit sleeps with infinite sleeps. Some code that uses a timeout really wants a timeout (note that pause() has a hack to fallback to DELAY() internally if cold is true for this reason). Instead, my feeling is that any kthreads that need timeouts to work need to defer their startup until SI_SUB_KICK_SCHEDULER. > However, I'd like feedback on the general idea and if it is acceptable I'd > like to coordinate testing with other platforms so this can go into the > tree. I don't think I've seen any objections? This does need more testing. I will update the patch to add a new EARLY_AP_STARTUP kernel option so this can be committed (but not yet enabled) allowing for easier testing (and allowing other platforms to catch up to x86). > The current changes are in the 'ap_startup' branch at github/bsdjhb/freebsd. > You can view them here: > > https://github.com/bsdjhb/freebsd/compare/master...bsdjhb:ap_startup -- John Baldwin From owner-freebsd-arch@freebsd.org Fri Mar 18 19:37:25 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 405C8AD519E; Fri, 18 Mar 2016 19:37:25 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-ig0-x22d.google.com (mail-ig0-x22d.google.com [IPv6:2607:f8b0:4001:c05::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0A7ABF2B; Fri, 18 Mar 2016 19:37:25 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: by mail-ig0-x22d.google.com with SMTP id vf5so30370779igb.0; Fri, 18 Mar 2016 12:37:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc; bh=ew4APn9OQCidAShcqjMUBVyd+F9SCtyssVE7uz4EhY4=; b=VxS4C0Zx6EFZAZX8KffsozACj80btLH1shDY31/TOjCbcGvxsfKZ5gpqym/H44FC4J /15lWvwpkTyPLe8DhxwhYyIGJCRhXQIuGmxBCTNtXngW1hTGh+qAU3U+SLFo5fGcZQVz AhvwWb1vnT+XctpHEXh+Q8L4glnGDE909cf/9NrUD8Rz+KfGGoXaC4x0rTQtuZ+6BhJ+ 81hVRDS5N+OG8WzEJdX3JMPRxDATJUjLyeu3EPV6xi4/m26b1e2SgZy0FiGRG5+MpcF5 v6/HuIooEBu9DJcizCGS2bNKSelrCQ8z1W4nSJs6XntmTodNd2TChMkgdKpTNW2Z1sIe YoVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc; bh=ew4APn9OQCidAShcqjMUBVyd+F9SCtyssVE7uz4EhY4=; b=e/ls+16LH/ncTDRRJZ7NZmLMQWkouJhDH0UnXCXRGvmFemST/Kx2i2oEJn7Kcxl8+R rHkF1d8FITI720/I97ERCmJUc4jV4rrcg+NKQxGe00xpJ9xeARjqj/V7XELFlplaSnN1 g8Inw6NJ46vbjAs1Qslc70aenvf4WsU6U3haXTCz+5upkc2Yf4zEZcjJJpIRi62BhwX5 hzcMt2+8G6n4aOBRaAxF9TDbZPKvXWS11q+Hk3caYbin+DRyoF7nyfJdWyOb80rPj+uN +mM8kqjyV3A6aZmmyejW9lIRd40CYI045cG2boD6cJmtdtyPWVU5MuB+yNzpV6OU+rhZ aypg== X-Gm-Message-State: AD7BkJIbKSXeJPN2p3gvvwt8grC/bvNHVqygIjY902zzlvO4YuIJYBmqfeEkWnuhRyKHzNuFN2YTNj6VWdJudw== MIME-Version: 1.0 X-Received: by 10.50.17.67 with SMTP id m3mr1148325igd.95.1458329844365; Fri, 18 Mar 2016 12:37:24 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.107.132.81 with HTTP; Fri, 18 Mar 2016 12:37:24 -0700 (PDT) In-Reply-To: <2980696.6AEyEjetGn@ralph.baldwin.cx> References: <1730061.8Ii36ORVKt@ralph.baldwin.cx> <2980696.6AEyEjetGn@ralph.baldwin.cx> Date: Fri, 18 Mar 2016 12:37:24 -0700 X-Google-Sender-Auth: 7a_NvKuKDSlwF9CjH5srBrUQZoA Message-ID: Subject: Re: Starting APs earlier during boot From: "K. Macy" To: John Baldwin Cc: "freebsd-arch@freebsd.org" , "arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Mar 2016 19:37:25 -0000 So none of these changes have been committed yet? I'm hitting hangs in USB on boot with recent HEAD and without having investigating had thought this might be what exposed the problem. Thanks. -M On Friday, March 18, 2016, John Baldwin wrote: > On Tuesday, February 16, 2016 12:50:22 PM John Baldwin wrote: > > Currently the kernel bootstraps the non-boot processors fairly early in > the > > SI_SUB_CPU SYSINIT. The APs then spin waiting to be "released". We > currently > > release the APs as one of the last steps at SI_SUB_SMP. On the one hand > this > > removes much of the need for synchronization while SYSINITs are running > since > > SYSINITs basically assume they are single-threaded. However, it also > enforces > > some odd quirks. Several places that deal with per-CPU resources have to > > split initialization up so that the BSP init happens in one SYSINIT and > the > > initialization of the APs happens in a second SYSINIT at SI_SUB_SMP. > > > > Another issue that is becoming more prominent on x86 (and probably will > also > > affect other platforms if it isn't already) is that to support working > > interrupts for interrupt config hooks we bind all interrupts to the BSP > during > > boot and only distribute them among other CPUs near the end at > SI_SUB_SMP. > > This is especially problematic with drivers for modern hardware > allocating > > num(CPUs) interrupts (hoping to use one per CPU). On x86 we have aboug > 190 > > IDT vectors available for device interrupts, so in theory we should be > able to > > tolerate a lot of drivers doing this (e.g. 60 drivers could allocate 3 > > interrupts for every CPU and we should still be fine). However, if you > have, > > say, 32 cores in a system, then you can only handle about 5 drivers doing > > this before you run out of vectors on CPU 0. > > > > Longer term we would also like to eventually have most drivers attach in > the > > same environment during boot as during post-boot. Right now post-boot is > > quite different as all CPUs are running, interrupts work, etc. One of > the > > goals of multipass support for new-bus is to help us get there by probing > > enough hardware to get timers working and starting the scheduler before > > probing the rest of the devices. That goal isn't quite realized yet. > > > > However, we can run a slightly simpler version of our scheduler before > > timers are working. In fact, sleep/wakeup work just fine fairly early > (we > > allocate the necessary structures at SI_SUB_KMEM which is before the APs > > are even started). Once idle threads are created and ready we could in > > theory let the APs startup and run other threads. You just don't have > working > > timeouts. OTOH, you can sort of simulate timeouts if you modify the > scheduler > > to yield the CPU instead of blocking the thread for a sleep with a > timeout. > > The effect would be for threads that do sleeps with a timeout to fall > back to > > polling before timers are working. In practice, all of the early kernel > > threads use sleeps without timeouts when idle so this doesn't really > matter. > > After some more testing, I've simplified the early scheduler a bit. It no > longer tries to simulate timeouts by just keeping the thread runnable. > Instead, > a sleep with a timeout just panics. However, it does still permit sleeps > with > infinite sleeps. Some code that uses a timeout really wants a timeout > (note > that pause() has a hack to fallback to DELAY() internally if cold is true > for > this reason). Instead, my feeling is that any kthreads that need timeouts > to > work need to defer their startup until SI_SUB_KICK_SCHEDULER. > > > However, I'd like feedback on the general idea and if it is acceptable > I'd > > like to coordinate testing with other platforms so this can go into the > > tree. > > I don't think I've seen any objections? This does need more testing. I > will > update the patch to add a new EARLY_AP_STARTUP kernel option so this can be > committed (but not yet enabled) allowing for easier testing (and allowing > other platforms to catch up to x86). > > > The current changes are in the 'ap_startup' branch at > github/bsdjhb/freebsd. > > You can view them here: > > > > https://github.com/bsdjhb/freebsd/compare/master...bsdjhb:ap_startup > > -- > John Baldwin > _______________________________________________ > freebsd-arch@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org > " > From owner-freebsd-arch@freebsd.org Sat Mar 19 02:02:43 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 56E64AD5B31; Sat, 19 Mar 2016 02:02:43 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-yw0-x231.google.com (mail-yw0-x231.google.com [IPv6:2607:f8b0:4002:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 17719C47; Sat, 19 Mar 2016 02:02:43 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: by mail-yw0-x231.google.com with SMTP id g127so160115115ywf.2; Fri, 18 Mar 2016 19:02:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc; bh=ajh4HnKlTC2JR5l9bBDXYd0daMDHKmlaz9eOuenIv3c=; b=GvJ3cx6iVmlU6D7YyKnvqVlVGcR5HVj0bWT5+kw/75duuOlRnPStmionK/gTzKBCAc 1smMz7NkSNMaTQ3qk9//gpACEkj0B+xfAbl5u15sKqRT06jcSu/PzSB3LtPSE2/UTz7F Q27NwtcJkQljgbtgccxz+qZIfbe0PSt02cpE5/axb9YmwJ8CHV2Bmvb3ES6te+SLNSyl zbVxfqYpBFWcZnCMGhmP3yfG3ZB2qXtPqfNxrpvWMtpt8Tb62+cqeCSaIpEXcsOi661o km/ujGUx5nrvvVNkYjX8Z6CnHMOvPXCM6YM7e9FLaaubfNN7VQ5BCOK5yyoQwwkL3Da9 muGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc; bh=ajh4HnKlTC2JR5l9bBDXYd0daMDHKmlaz9eOuenIv3c=; b=WhCyzYZoLy7g/NdxMOmSZCr+zSymNP043C06gSph1AZPVEoEMa9qIIdHIaleqeGANb cvDx6o02TcbhaFuhitWEepTl5NGz0MeG98FBzoaFTQPCq7TWy/iJsLW6PjqqOP5JCst3 vHpIPq9fUXOdsa5pLgGLRHnmRW/3/w/NwhPITfTap0gik9T7hfrMDLDzf45bvVvh/TAz ACqi0z5M8njIDnHfP+Rt93HmlNBmqc+HRMe231czeJvs11pXZ7Im0TiNwXvk0Prqwjpj /VPg+WYIUvggH16vXHSmwbR1YHoMb93SbP7be+9KuAt7G08PsQQkvPMjsbP0k6WAaNZi 7mvA== X-Gm-Message-State: AD7BkJL2hQ0PpaKnY78cKJdk+TQmSZPSOdda6snpkxnRFY7Ii1ITNMtiqHypDyAyzX68QB9+51dzt953qIBhrQ== MIME-Version: 1.0 X-Received: by 10.129.145.215 with SMTP id i206mr6976539ywg.161.1458352962394; Fri, 18 Mar 2016 19:02:42 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.13.235.16 with HTTP; Fri, 18 Mar 2016 19:02:42 -0700 (PDT) In-Reply-To: References: <1730061.8Ii36ORVKt@ralph.baldwin.cx> <2980696.6AEyEjetGn@ralph.baldwin.cx> Date: Fri, 18 Mar 2016 19:02:42 -0700 X-Google-Sender-Auth: 828sVAraj_94hcKUzVoaaDfCXyI Message-ID: Subject: Re: Starting APs earlier during boot From: "K. Macy" To: John Baldwin Cc: "freebsd-arch@freebsd.org" , "arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Mar 2016 02:02:43 -0000 On Fri, Mar 18, 2016 at 12:37 PM, K. Macy wrote: > So none of these changes have been committed yet? > > I'm hitting hangs in USB on boot with recent HEAD and without having > investigating had thought this might be what exposed the problem. Never mind. It's yet another ZFS namespace deadlock. -M > > > On Friday, March 18, 2016, John Baldwin wrote: >> >> On Tuesday, February 16, 2016 12:50:22 PM John Baldwin wrote: >> > Currently the kernel bootstraps the non-boot processors fairly early in >> > the >> > SI_SUB_CPU SYSINIT. The APs then spin waiting to be "released". We >> > currently >> > release the APs as one of the last steps at SI_SUB_SMP. On the one hand >> > this >> > removes much of the need for synchronization while SYSINITs are running >> > since >> > SYSINITs basically assume they are single-threaded. However, it also >> > enforces >> > some odd quirks. Several places that deal with per-CPU resources have >> > to >> > split initialization up so that the BSP init happens in one SYSINIT and >> > the >> > initialization of the APs happens in a second SYSINIT at SI_SUB_SMP. >> > >> > Another issue that is becoming more prominent on x86 (and probably will >> > also >> > affect other platforms if it isn't already) is that to support working >> > interrupts for interrupt config hooks we bind all interrupts to the BSP >> > during >> > boot and only distribute them among other CPUs near the end at >> > SI_SUB_SMP. >> > This is especially problematic with drivers for modern hardware >> > allocating >> > num(CPUs) interrupts (hoping to use one per CPU). On x86 we have aboug >> > 190 >> > IDT vectors available for device interrupts, so in theory we should be >> > able to >> > tolerate a lot of drivers doing this (e.g. 60 drivers could allocate 3 >> > interrupts for every CPU and we should still be fine). However, if you >> > have, >> > say, 32 cores in a system, then you can only handle about 5 drivers >> > doing >> > this before you run out of vectors on CPU 0. >> > >> > Longer term we would also like to eventually have most drivers attach in >> > the >> > same environment during boot as during post-boot. Right now post-boot >> > is >> > quite different as all CPUs are running, interrupts work, etc. One of >> > the >> > goals of multipass support for new-bus is to help us get there by >> > probing >> > enough hardware to get timers working and starting the scheduler before >> > probing the rest of the devices. That goal isn't quite realized yet. >> > >> > However, we can run a slightly simpler version of our scheduler before >> > timers are working. In fact, sleep/wakeup work just fine fairly early >> > (we >> > allocate the necessary structures at SI_SUB_KMEM which is before the APs >> > are even started). Once idle threads are created and ready we could in >> > theory let the APs startup and run other threads. You just don't have >> > working >> > timeouts. OTOH, you can sort of simulate timeouts if you modify the >> > scheduler >> > to yield the CPU instead of blocking the thread for a sleep with a >> > timeout. >> > The effect would be for threads that do sleeps with a timeout to fall >> > back to >> > polling before timers are working. In practice, all of the early kernel >> > threads use sleeps without timeouts when idle so this doesn't really >> > matter. >> >> After some more testing, I've simplified the early scheduler a bit. It no >> longer tries to simulate timeouts by just keeping the thread runnable. >> Instead, >> a sleep with a timeout just panics. However, it does still permit sleeps >> with >> infinite sleeps. Some code that uses a timeout really wants a timeout >> (note >> that pause() has a hack to fallback to DELAY() internally if cold is true >> for >> this reason). Instead, my feeling is that any kthreads that need timeouts >> to >> work need to defer their startup until SI_SUB_KICK_SCHEDULER. >> >> > However, I'd like feedback on the general idea and if it is acceptable >> > I'd >> > like to coordinate testing with other platforms so this can go into the >> > tree. >> >> I don't think I've seen any objections? This does need more testing. I >> will >> update the patch to add a new EARLY_AP_STARTUP kernel option so this can >> be >> committed (but not yet enabled) allowing for easier testing (and allowing >> other platforms to catch up to x86). >> >> > The current changes are in the 'ap_startup' branch at >> > github/bsdjhb/freebsd. >> > You can view them here: >> > >> > https://github.com/bsdjhb/freebsd/compare/master...bsdjhb:ap_startup >> >> -- >> John Baldwin >> _______________________________________________ >> freebsd-arch@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-arch >> To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org"