From owner-freebsd-current@freebsd.org Tue Jun 7 10:07:01 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AD950B6DDF1 for ; Tue, 7 Jun 2016 10:07:01 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 8CF7A1384 for ; Tue, 7 Jun 2016 10:07:01 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 71ABCB6DDEF; Tue, 7 Jun 2016 10:07:01 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 71564B6DDEE for ; Tue, 7 Jun 2016 10:07:01 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: from mail-wm0-x22f.google.com (mail-wm0-x22f.google.com [IPv6:2a00:1450:400c:c09::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1ABB71382; Tue, 7 Jun 2016 10:07:01 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: by mail-wm0-x22f.google.com with SMTP id v199so11711246wmv.0; Tue, 07 Jun 2016 03:07:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=vFIQR+YfbGjwPghuWgGN5yPMRAMn4AUWVphUg1oT788=; b=wx+Ai4iO8nHuIV9nca3q+ISJD++vIgBbZM439IMa1X8DG4BxZDtOAjq6sOe6UVcGoh AybhS6Orrhf19k6o0lRn6TA6Zmw3NJoTHgJAYIOGYeuBMmtx6X8lKveIYue+b+cgLIuS gAXKvJfO1h+y2E9JHew9stZIgp2JpmguKzEjfeFVRA48zHt9CH6KHgDMbIDTBPuODAGd vzUzh7r5UgbFIgo99OQk7kyFB7QYdQE8mDlYJgTffdoRS0VPFtoQ7MBRY/3DyMNkqAHS 935rHYU0lWLnBuFwlQbkRDRSHptcvTAoNElEP8vD0533TKx/sYsOa2k3PUbtE60Dzh5V RU+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=vFIQR+YfbGjwPghuWgGN5yPMRAMn4AUWVphUg1oT788=; b=AUgy/ojiH3qWu7bjgyp8rPDqre5/vm1BjD2BtShQszQDSo1xRGn2Zmu8+N94MkwSEV o8xNpV0vabgfP7cSZmJyDJvdcyfg/vDffBKUxJwvw4E4TbWT+inquu4SJ+j4NK/DxMSA jbyTRlG0xFbMtwaWP1pvxeTW2NOy6OvOOJOjjJJrx9fI3+KeKRTa1CC0YqwxuSzhmbgS 7/CirRAYC9yygTOIvD+/OOF48KIB9twUw7Mm9mnFSq2R1qJmZoyCnXMcwuCn40rePTTa wwnfJry5PRz6pzNKF2j1gCitWHzwNUjDqODR64iFe4x/nB6Q0TFf21wwb9zSctzJGxGp ABXg== X-Gm-Message-State: ALyK8tIzJo0RrPxIuB3yfL4uqHVutjLjsCLtZ8NB7GzckNbvKsiopvyvSbrULon3qZIKrg== X-Received: by 10.194.248.97 with SMTP id yl1mr22926759wjc.130.1465294019208; Tue, 07 Jun 2016 03:06:59 -0700 (PDT) Received: from ernst.home (p578E1CA6.dip0.t-ipconnect.de. [87.142.28.166]) by smtp.gmail.com with ESMTPSA id o76sm18757704wme.0.2016.06.07.03.06.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 07 Jun 2016 03:06:58 -0700 (PDT) Date: Tue, 7 Jun 2016 12:06:54 +0200 From: Gary Jennejohn To: John Baldwin Cc: current@freebsd.org Subject: Re: EARLY_AP_STARTUP hangs during boot Message-ID: <20160607120654.42d375c7@ernst.home> In-Reply-To: <8812233.S6jxPboLEa@ralph.baldwin.cx> References: <20160516122242.39249a54@ernst.home> <20160527095005.0e0dc1be@ernst.home> <20160528141141.232185a9@ernst.home> <8812233.S6jxPboLEa@ralph.baldwin.cx> Reply-To: gljennjohn@gmail.com X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.29; amd64-portbld-freebsd11.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jun 2016 10:07:01 -0000 On Tue, 31 May 2016 13:10:06 -0700 John Baldwin wrote: > On Saturday, May 28, 2016 02:11:41 PM Gary Jennejohn wrote: > > On Fri, 27 May 2016 09:50:05 +0200 > > Gary Jennejohn wrote: > > > > > On Thu, 26 May 2016 16:54:35 -0700 > > > John Baldwin wrote: > > > > > > > On Tuesday, May 17, 2016 06:47:41 PM Gary Jennejohn wrote: > > > > > On Mon, 16 May 2016 10:54:19 -0700 > > > > > John Baldwin wrote: > > > > > > > > > > > On Monday, May 16, 2016 12:22:42 PM Gary Jennejohn wrote: > > > > > > > I tried out EARLY_AP_STARTUP, but the kernel hangs and I can't > > > > > > > break into DDB. > > > > > > > > > > > > > > I did a verbose boot and the last lines I see are related to routing > > > > > > > MSI-X to various local APIC vectors. I copied the last few lines and > > > > > > > they look like this: > > > > > > > > > > > > > > msi: routing MSI-X IRQ 256 to local APIC 2 vector 48 > > > > > > > msi: routing MSI-X IRQ 257 to local APIC 3 vector 48 > > > > > > > msi: routing MSI-X IRQ 258 to local APIC 4 vector 48 > > > > > > > msi: routing MSI-X IRQ 256 to local APIC 0 vector 49 > > > > > ^^^^^^^ Assigning > > > > > > > > > > > > > > I tried disabling msi and msix in /boot/loader.conf, but the settings > > > > > > > were ignored (probabaly too early). > > > > > > > > > > > > No, those settings are not too early. However, the routing to different > > > > > > CPUs now happens earlier than it used to. What is the line before the > > > > > > MSI lines? You can take a picture with your phone/camera if that's simplest. > > > > > > > > > > > > > > > > Here a few lines before the MSI routing happens: > > > > > > > > > > hpet0: iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0 > > > > > hpet0: vendor 0x4353, rev 0x1, 14318180 Hz, 3 timers, legacy route > > > > > hpet0: t0 : irqs 0x00c0ff (0), MSI, periodic > > > > > hpet0: t1 : irqs 0x00c0ff (0), MSI, periodic > > > > > hpet0: t2 : irqs 0x00c0ff (0), MSI, periodic > > > > > Timecounter "HPET" frequency 14318180 Hz quality 950 > > > > > > > > The assigning message means it is in the loop using > > > > bus_bind_intr() to setup per-CPU timers. Can you please try > > > > setting 'hint.hpet.0.per_cpu=0' at the loader prompt to see if > > > > disabling the use of per-CPU timers allows you to boot? > > > > > > > > > > Something has changed since the last time I generated a kernel with > > > this option. > > > > > > Now I get a NULL-pointer dereference in the kernel, doesn't matter > > > whether I set the hint or not. > > > > > > > OK, now that the startup has been fixed, I tried setting the hint at > > the loader prompt, but the kenel hangs in exactly the same place as > > before. I actually booted twice to make certain I hadn't made a > > typo when setting the hint. > > Humm, it shouldn't be calling bus_bind_intr() if the hint is set. Actually, > I guess it just binds them all to first CPU if per-CPU timers aren't set. > Can you add debug printfs to hpet_attach() in sys/dev/acpica/acpi_hpet.c to > narrow down which line in that function it hangs after? > > Another option to try is to add the following to your kernel config: > > options KTR > options KTR_COMPILE=KTR_PROC > options KTR_MASK=KTR_PROC > options KTR_VERBOSE=1 > > this will spew a lot of crap to the screen, but if it stops spewing when it > hangs then it might be tell us where the system is hung. If you have any way > to configure a serial console then this would also be useful even if it spews > constantly when it is hung (assuming you could log the output of the serial > console). > I used the KTR options. After the Timecounter "HPET" frequency 14318180 Hz quality 950 I see cpu0 mi_switch: old thread 10000 (swapper) cpu0 mi_switch: new thread 10022 (if_config_tqg_0) cpu0 sleep_broadcast(0x80002f9a600, 0) cpu0 msleep_spin: old thread 100022 cpu0 mi_switch: old thread 10022 cpu0 mi_switch: new thread 10016 (if_io_tqg_0) cpu0 sleep_broadcast(0x80002f9a780, 0) cpu0 msleep_spin: old thread 10016 cpu0 mi_switch: old thread 10016 cpu0 fork_exit: new thread 0x80004239510 (td_sched 0x8000042399d8, pid 10, idle: cpu0) And that's all that came out, really not very much at all. -- Gary Jennejohn