From owner-freebsd-stable@freebsd.org Thu Feb 4 21:31:51 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EE2DD52FAC5 for ; Thu, 4 Feb 2021 21:31:51 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4DWsFM5Ptlz3CGy for ; Thu, 4 Feb 2021 21:31:51 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id B99F952FAC4; Thu, 4 Feb 2021 21:31:51 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B967852F9DB for ; Thu, 4 Feb 2021 21:31:51 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi1-f180.google.com (mail-oi1-f180.google.com [209.85.167.180]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DWsFM0Rhxz3CMj for ; Thu, 4 Feb 2021 21:31:50 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oi1-f180.google.com with SMTP id k25so5237608oik.13 for ; Thu, 04 Feb 2021 13:31:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=Wlex4Bc8wKvZSIo5XK+1ry7pCnUYiMAvVeyM4Qhm8jw=; b=emaEbxYODuDfMRisPn5H6tzR8MsEKw/kf2eR7+CPrrlsymD2OYc6Dix5w0OBMbtsBr iBwiwjezw/MoWxRm+4ObBBcbikv0mXn2pfnMMfGbAd+5xIXzML4acfkq6rmLJOK1Rakd BZfFAk2ugX+qm/6LZDOdWZPPSrXDgUAtMHrHk49K/8A6ux0JwnhnD0r90ByM156kVKVq wcumjzmeewrrsDLuuKiKjAEuCKckyd/gIhysLyCJUTespoUpFJSjXZ2JusgincanRati QbEZk/K5u91enXyOMyyThkdMyWLimQb3HXTHCBorVZbpxBBniQbSVqADDUdX3eOIxNzb Xzzw== X-Gm-Message-State: AOAM531ExU6I95h3VN21ka8Bu/Ld15DZj/xJrgCiOEzQ+0fbQtCZ3BR+ Uo3DMWVE+s76xV616rQ9SbZhf65r3mkbJizjoeNdRUT0n147Pg== X-Google-Smtp-Source: ABdhPJxGsGzT0rDT8NuBNni2oJn+L/1zxQGB/FpJJiF6DeOdBflEunQOlMMHTFpBIqsh5TV+kJhsjwY9J9QR+OxZMGc= X-Received: by 2002:a54:4813:: with SMTP id j19mr1009832oij.73.1612474309613; Thu, 04 Feb 2021 13:31:49 -0800 (PST) MIME-Version: 1.0 From: Alan Somers Date: Thu, 4 Feb 2021 14:31:38 -0700 Message-ID: Subject: Page fault in _mca_init during startup To: FreeBSD Stable ML X-Rspamd-Queue-Id: 4DWsFM0Rhxz3CMj X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.167.180 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-2.99 / 15.00]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.989]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[209.85.167.180:from]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_DOM_EQ_FROM_DOM(0.00)[]; R_DKIM_NA(0.00)[]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[asomers]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[209.85.167.180:from:127.0.2.255]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RCVD_IN_DNSWL_NONE(0.00)[209.85.167.180:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.167.180:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[stable] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Feb 2021 21:31:52 -0000 After upgrading a machine to FreeBSD, 12.2, it hit the following panic on its first reboot. I suspect that a few other servers have hit this too, but since it happens before swap is mounted there are no core dumps, and they usually reboot immediately. The code in question hasn't changed since 2018. The panic happened in cmci_monitor at line 930. Does anybody have any suggestions for how I could debug further? I can't readily reproduce it, and I can't dump core, but I'd like to investigate it any way I can. The server in question has dual Xeon Gold 6142 CPUs. if (!(ctl & MC_CTL2_CMCI_EN)) /* This bank does not support CMCI. */ return; cc = &cmc_state[PCPU_GET(cpuid)][i]; // <- panic here /* Determine maximum threshold. */ Fatal trap 12: page fault while in kernel mode cpuid = 26; apic id = 34 fault virtual address = 0xd0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff8125a009 stack pointer = 0x28:0xfffffe0000b65f20 frame pointer = 0x28:0xfffffe0000b65f50 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 11 (idle: cpu26) trap number = 12 panic: page fault cpuid = 26 time = 1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0000b65be0 vpanic() at vpanic+0x17b/frame 0xfffffe0000b65c30 panic() at panic+0x43/frame 0xfffffe0000b65c90 trap_fatal() at trap_fatal+0x391/frame 0xfffffe0000b65cf0 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe0000b65d40 trap() at trap+0x286/frame 0xfffffe0000b65e50 calltrap() at calltrap+0x8/frame 0xfffffe0000b65e50 --- trap 0xc, rip = 0xffffffff8125a009, rsp = 0xfffffe0000b65f20, rbp = 0xfffffe0000b65f50 --- _mca_init() at _mca_init+0x5d9/frame 0xfffffe0000b65f50 init_secondary_tail() at init_secondary_tail+0xfd/frame 0xfffffe0000b65f80 init_secondary() at init_secondary+0x2d1/frame 0xfffffe0000b65ff0 KDB: enter: panic [ thread pid 11 tid 100029 ] Stopped at kdb_enter+0x37: movq $0,0x12bc1f6(%rip)