From owner-freebsd-stable@freebsd.org Fri Feb 5 00:59:25 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4EDFA53754C for ; Fri, 5 Feb 2021 00:59:25 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4DWxrs0qDzz3lRk for ; Fri, 5 Feb 2021 00:59:25 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 1C1B753763C; Fri, 5 Feb 2021 00:59:25 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1BE7A5375D6 for ; Fri, 5 Feb 2021 00:59:25 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DWxrr50MCz3lr4; Fri, 5 Feb 2021 00:59:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 1150xGWq059341 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 5 Feb 2021 02:59:19 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 1150xGWq059341 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 1150xGfm059340; Fri, 5 Feb 2021 02:59:16 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 5 Feb 2021 02:59:16 +0200 From: Konstantin Belousov To: Alan Somers Cc: Mark Johnston , Matthew Macy , FreeBSD Stable ML Subject: Re: Page fault in _mca_init during startup Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4DWxrr50MCz3lr4 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2021 00:59:25 -0000 On Thu, Feb 04, 2021 at 05:19:43PM -0700, Alan Somers wrote: > On Thu, Feb 4, 2021 at 4:27 PM Mark Johnston wrote: > > > On Fri, Feb 05, 2021 at 12:58:34AM +0200, Konstantin Belousov wrote: > > > On Thu, Feb 04, 2021 at 01:34:13PM -0800, Matthew Macy wrote: > > > > On Thu, Feb 4, 2021 at 1:31 PM Alan Somers > > wrote: > > > > > > > > > > After upgrading a machine to FreeBSD, 12.2, it hit the following > > panic on > > > > > its first reboot. I suspect that a few other servers have hit this > > too, > > > > > but since it happens before swap is mounted there are no core dumps, > > and > > > > > they usually reboot immediately. The code in question hasn't > > changed since > > > > > 2018. The panic happened in cmci_monitor at line 930. Does anybody > > have > > > > > any suggestions for how I could debug further? I can't readily > > reproduce > > > > > it, and I can't dump core, but I'd like to investigate it any way I > > can. > > > > > The server in question has dual Xeon Gold 6142 CPUs. > > > > > > > > Try this. > > > > > > I think that there is no other dependencies in the startup order, but > > > cannot know it for sure. > > > > > > commit 19584e3d3e9606d591fa30999b370ed758960e8c > > > Author: Konstantin Belousov > > > Date: Fri Feb 5 00:56:09 2021 +0200 > > > > > > x86: init mca before APs are started > > > > APs only call mca_init() after they have been released by the BSP > > though, and that happens later in SI_SUB_SMP. > > > > > diff --git a/sys/x86/x86/mca.c b/sys/x86/x86/mca.c > > > index 03100e77d455..e2bf2673cf69 100644 > > > --- a/sys/x86/x86/mca.c > > > +++ b/sys/x86/x86/mca.c > > > @@ -1371,7 +1371,7 @@ mca_init_bsp(void *arg __unused) > > > > > > mca_init(); > > > } > > > -SYSINIT(mca_init_bsp, SI_SUB_CPU, SI_ORDER_ANY, mca_init_bsp, NULL); > > > +SYSINIT(mca_init_bsp, SI_SUB_CPU, SI_ORDER_SECOND, mca_init_bsp, NULL); > > > > > > /* Called when a machine check exception fires. */ > > > void > > > > kib's patch causes a different problem, and this one is reproducible: > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0x18 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff8125762c > stack pointer = 0x28:0xffffffff828dad90 > frame pointer = 0x28:0xffffffff828dad90 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = resume, IOPL = 0 > current process = 0 () > trap number = 12 > panic: page fault > cpuid = 0 > time = 1 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xffffffff828daa50 > vpanic() at vpanic+0x17b/frame 0xffffffff828daaa0 > panic() at panic+0x43/frame 0xffffffff828dab00 > trap_fatal() at trap_fatal+0x391/frame 0xffffffff828dab60 > trap_pfault() at trap_pfault+0x4f/frame 0xffffffff828dabb0 > trap() at trap+0x286/frame 0xffffffff828dacc0 > calltrap() at calltrap+0x8/frame 0xffffffff828dacc0 > --- trap 0xc, rip = 0xffffffff8125762c, rsp = 0xffffffff828dad90, rbp = > 0xffffffff828dad90 --- > native_lapic_enable_cmc() at native_lapic_enable_cmc+0x1c/frame > 0xffffffff828dad90 > _mca_init() at _mca_init+0x94c/frame 0xffffffff828dadd0 > mi_startup() at mi_startup+0xdf/frame 0xffffffff828dadf0 > btext() at btext+0x2c > KDB: enter: panic > [ thread pid 0 tid 0 ] > Stopped at kdb_enter+0x37: movq $0,0x12bc396(%rip) > > If you're wondering, the panic happens at this point in > native_lapic_enable_cmc: > > apic_id = PCPU_GET(apic_id); > KASSERT(lapics[apic_id].la_present, > ("%s: missing APIC %u", __func__, apic_id)); > lapics[apic_id].la_lvts[APIC_LVT_CMCI].lvt_masked = 0; <- panic here > lapics[apic_id].la_lvts[APIC_LVT_CMCI].lvt_active = 1; > if (bootverbose) > printf("lapic%u: CMCI unmasked\n", apic_id); > } Scratch this patch. Do you have INVARIANTS enabled? If not, I am curious if enabling them would convert that rare page fault into rare "CPU %d has more MC banks" assert. Also might be the output of the # for x in $(jot $(sysctl -n hw.ncpu) 0) ; do cpucontrol -m 0x179 /dev/cpuctl$x; done command will show the issue (0x179 is the MCG_CAP MSR). You need to load cpuctl(4) if it is not loaded yet.