From owner-freebsd-arch@freebsd.org Sat Mar 19 02:02:43 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 56E64AD5B31; Sat, 19 Mar 2016 02:02:43 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-yw0-x231.google.com (mail-yw0-x231.google.com [IPv6:2607:f8b0:4002:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 17719C47; Sat, 19 Mar 2016 02:02:43 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: by mail-yw0-x231.google.com with SMTP id g127so160115115ywf.2; Fri, 18 Mar 2016 19:02:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc; bh=ajh4HnKlTC2JR5l9bBDXYd0daMDHKmlaz9eOuenIv3c=; b=GvJ3cx6iVmlU6D7YyKnvqVlVGcR5HVj0bWT5+kw/75duuOlRnPStmionK/gTzKBCAc 1smMz7NkSNMaTQ3qk9//gpACEkj0B+xfAbl5u15sKqRT06jcSu/PzSB3LtPSE2/UTz7F Q27NwtcJkQljgbtgccxz+qZIfbe0PSt02cpE5/axb9YmwJ8CHV2Bmvb3ES6te+SLNSyl zbVxfqYpBFWcZnCMGhmP3yfG3ZB2qXtPqfNxrpvWMtpt8Tb62+cqeCSaIpEXcsOi661o km/ujGUx5nrvvVNkYjX8Z6CnHMOvPXCM6YM7e9FLaaubfNN7VQ5BCOK5yyoQwwkL3Da9 muGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc; bh=ajh4HnKlTC2JR5l9bBDXYd0daMDHKmlaz9eOuenIv3c=; b=WhCyzYZoLy7g/NdxMOmSZCr+zSymNP043C06gSph1AZPVEoEMa9qIIdHIaleqeGANb cvDx6o02TcbhaFuhitWEepTl5NGz0MeG98FBzoaFTQPCq7TWy/iJsLW6PjqqOP5JCst3 vHpIPq9fUXOdsa5pLgGLRHnmRW/3/w/NwhPITfTap0gik9T7hfrMDLDzf45bvVvh/TAz ACqi0z5M8njIDnHfP+Rt93HmlNBmqc+HRMe231czeJvs11pXZ7Im0TiNwXvk0Prqwjpj /VPg+WYIUvggH16vXHSmwbR1YHoMb93SbP7be+9KuAt7G08PsQQkvPMjsbP0k6WAaNZi 7mvA== X-Gm-Message-State: AD7BkJL2hQ0PpaKnY78cKJdk+TQmSZPSOdda6snpkxnRFY7Ii1ITNMtiqHypDyAyzX68QB9+51dzt953qIBhrQ== MIME-Version: 1.0 X-Received: by 10.129.145.215 with SMTP id i206mr6976539ywg.161.1458352962394; Fri, 18 Mar 2016 19:02:42 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.13.235.16 with HTTP; Fri, 18 Mar 2016 19:02:42 -0700 (PDT) In-Reply-To: References: <1730061.8Ii36ORVKt@ralph.baldwin.cx> <2980696.6AEyEjetGn@ralph.baldwin.cx> Date: Fri, 18 Mar 2016 19:02:42 -0700 X-Google-Sender-Auth: 828sVAraj_94hcKUzVoaaDfCXyI Message-ID: Subject: Re: Starting APs earlier during boot From: "K. Macy" To: John Baldwin Cc: "freebsd-arch@freebsd.org" , "arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Mar 2016 02:02:43 -0000 On Fri, Mar 18, 2016 at 12:37 PM, K. Macy wrote: > So none of these changes have been committed yet? > > I'm hitting hangs in USB on boot with recent HEAD and without having > investigating had thought this might be what exposed the problem. Never mind. It's yet another ZFS namespace deadlock. -M > > > On Friday, March 18, 2016, John Baldwin wrote: >> >> On Tuesday, February 16, 2016 12:50:22 PM John Baldwin wrote: >> > Currently the kernel bootstraps the non-boot processors fairly early in >> > the >> > SI_SUB_CPU SYSINIT. The APs then spin waiting to be "released". We >> > currently >> > release the APs as one of the last steps at SI_SUB_SMP. On the one hand >> > this >> > removes much of the need for synchronization while SYSINITs are running >> > since >> > SYSINITs basically assume they are single-threaded. However, it also >> > enforces >> > some odd quirks. Several places that deal with per-CPU resources have >> > to >> > split initialization up so that the BSP init happens in one SYSINIT and >> > the >> > initialization of the APs happens in a second SYSINIT at SI_SUB_SMP. >> > >> > Another issue that is becoming more prominent on x86 (and probably will >> > also >> > affect other platforms if it isn't already) is that to support working >> > interrupts for interrupt config hooks we bind all interrupts to the BSP >> > during >> > boot and only distribute them among other CPUs near the end at >> > SI_SUB_SMP. >> > This is especially problematic with drivers for modern hardware >> > allocating >> > num(CPUs) interrupts (hoping to use one per CPU). On x86 we have aboug >> > 190 >> > IDT vectors available for device interrupts, so in theory we should be >> > able to >> > tolerate a lot of drivers doing this (e.g. 60 drivers could allocate 3 >> > interrupts for every CPU and we should still be fine). However, if you >> > have, >> > say, 32 cores in a system, then you can only handle about 5 drivers >> > doing >> > this before you run out of vectors on CPU 0. >> > >> > Longer term we would also like to eventually have most drivers attach in >> > the >> > same environment during boot as during post-boot. Right now post-boot >> > is >> > quite different as all CPUs are running, interrupts work, etc. One of >> > the >> > goals of multipass support for new-bus is to help us get there by >> > probing >> > enough hardware to get timers working and starting the scheduler before >> > probing the rest of the devices. That goal isn't quite realized yet. >> > >> > However, we can run a slightly simpler version of our scheduler before >> > timers are working. In fact, sleep/wakeup work just fine fairly early >> > (we >> > allocate the necessary structures at SI_SUB_KMEM which is before the APs >> > are even started). Once idle threads are created and ready we could in >> > theory let the APs startup and run other threads. You just don't have >> > working >> > timeouts. OTOH, you can sort of simulate timeouts if you modify the >> > scheduler >> > to yield the CPU instead of blocking the thread for a sleep with a >> > timeout. >> > The effect would be for threads that do sleeps with a timeout to fall >> > back to >> > polling before timers are working. In practice, all of the early kernel >> > threads use sleeps without timeouts when idle so this doesn't really >> > matter. >> >> After some more testing, I've simplified the early scheduler a bit. It no >> longer tries to simulate timeouts by just keeping the thread runnable. >> Instead, >> a sleep with a timeout just panics. However, it does still permit sleeps >> with >> infinite sleeps. Some code that uses a timeout really wants a timeout >> (note >> that pause() has a hack to fallback to DELAY() internally if cold is true >> for >> this reason). Instead, my feeling is that any kthreads that need timeouts >> to >> work need to defer their startup until SI_SUB_KICK_SCHEDULER. >> >> > However, I'd like feedback on the general idea and if it is acceptable >> > I'd >> > like to coordinate testing with other platforms so this can go into the >> > tree. >> >> I don't think I've seen any objections? This does need more testing. I >> will >> update the patch to add a new EARLY_AP_STARTUP kernel option so this can >> be >> committed (but not yet enabled) allowing for easier testing (and allowing >> other platforms to catch up to x86). >> >> > The current changes are in the 'ap_startup' branch at >> > github/bsdjhb/freebsd. >> > You can view them here: >> > >> > https://github.com/bsdjhb/freebsd/compare/master...bsdjhb:ap_startup >> >> -- >> John Baldwin >> _______________________________________________ >> freebsd-arch@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-arch >> To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org"