From owner-freebsd-hackers@freebsd.org Thu Jun 6 14:43:39 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A086D15B580A for ; Thu, 6 Jun 2019 14:43:39 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-ed1-x541.google.com (mail-ed1-x541.google.com [IPv6:2a00:1450:4864:20::541]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CEA906FA8B for ; Thu, 6 Jun 2019 14:43:38 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: by mail-ed1-x541.google.com with SMTP id p15so3687441eds.8 for ; Thu, 06 Jun 2019 07:43:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=longcount-org.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=yQ2FqJEfVP+Fva98AB7fGgAaczDZfRYUe1f9dzHq41U=; b=ZDGM0OKNymRGPlN6Q+Sivh6+8qFGK049dfrXZJChNh5MotNEuaEo8P24OJEROBAH8Z kUv2EyuCHX6D0szAACdc1CIqA2JsDBkEBhIIcDEpoDhFtYNucjoHiWQxRTjatE/4SNIP RZHE/mZ06nL+11qcj3y++XIUYi1j2lCyGHxUsymO6Ch8J8xt06NHPfMh9gEXI0wYETvt SkOsL3iMSC4IUJHf1jsLifcazlmgb8NJdSXWz7MniLbzHkkxM/NpgIsr5/auM1Z7r06w 2WgmqUUDmmP/zQZxv9hKD5/NSSRtaA98pGYWU4sP3O8x5JnCQczT/+l/Kvl1lBKAheUn ltlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=yQ2FqJEfVP+Fva98AB7fGgAaczDZfRYUe1f9dzHq41U=; b=TwmUP+YaAd5GrqD+H2+s20YxpkjKnIc95u8QFKZvfNrUYiFCDX7xJMR363mNwSKzPR RhqS5hJ+jDoi2u7XlhBMxFtpuKEhVaYu/4C61EsZXxhF5GxNbHZsMaTXdCRVGEyg9cFY 4Rty7SKtHImqn+9HJnGiXuQS4+5vTe1s6msj4eEQG4l0l96ZtQ3Xt2cDBfpk8xuVe/hm sQxjj/YhhQMdeXv/J2N6Aj0kTWOlLWNCah3vgRfg3ZOrfHm6CbBolIe/tmGQ6Y+ozfog PLCRzBCpesvX2OEUhDCvEf20gJrjWdW33eoOQuj7rbHEBbwKoUYsG1bFUpmBZwvY/4kr V47g== X-Gm-Message-State: APjAAAU/V0YcSrmyNdniBtQec9dxDtdE96NdxFmRzf/jwIqqWsfNrf2h 9p6jbCFoNSNyUZw20WoZdzjz84PpmWpDRshzJgDAKuwyV04= X-Google-Smtp-Source: APXvYqxF8SQpLvxl3d18PFkExkjqBD/dluCN4QwYKLwW6LVu2wFXUhHIgIMPo4WM8IoBIR+nPnLCbQFoOy8MtRsbuTs= X-Received: by 2002:a50:90af:: with SMTP id c44mr24556147eda.126.1559832217452; Thu, 06 Jun 2019 07:43:37 -0700 (PDT) MIME-Version: 1.0 From: Mark Saad Date: Thu, 6 Jun 2019 10:43:26 -0400 Message-ID: Subject: Kernel panic on 12-STABLE-r348203 amd64 E5-2690v4 with Cluster on die mode enabled To: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: CEA906FA8B X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=longcount-org.20150623.gappssmtp.com header.s=20150623 header.b=ZDGM0OKN X-Spamd-Result: default: False [-3.84 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[longcount-org.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[longcount.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-1.00)[ip: (-0.30), ipnet: 2a00:1450::/32(-2.32), asn: 15169(-2.30), country: US(-0.06)]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[longcount-org.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[cached: alt1.aspmx.l.google.com]; RCVD_IN_DNSWL_NONE(0.00)[1.4.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.53)[-0.533,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jun 2019 14:43:39 -0000 All I posed this yesterday but; I am not sure what happened. Here is the short version. I received two new Dell r630's each with a E5-2690 v4 . The E5-2690 v4 has 14 Cores in two packages on the one chip. I don't remember the exact topology however as a result the BIOS supports the NUMA / Memory mode know as Cluster on Die were each package on the one chip shows up as its own NUMA domain. The issue is this when enabled the box boots 12-RELEASE a-ok. When I rebuilt 12.0-STABLE-r348203 it would panic early in the boot process. Here is a dump of the console =============================== Loading kernel... /boot/kernel/kernel text=0x168d811 data=0x1cf968+0x768c80 syms=[0x8+0x1778e8+0x8 / +0x194f1d] Loading configured modules... /boot/kernel/ipmi.ko size 0x11e10 at 0x2645000 loading required module 'smbus' /boot/kernel/smbus.ko size 0x2ef0 at 0x2657000 /boot/entropy size=0x1000 /boot/kernel/cc_httcp.ko size 0x2330 at 0x265b000 ---<>---c_hmodule 'smbus' Copyright (c) 1992-2019 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-STABLE r348693 GENERIC amd64 FreeBSD clang version 8.0.0 (tags/RELEASE_800/final 356365) (based on LLVM 8.0.0) panic: UMA zone "UMA Zones": Increase vm.boot_pages cpuid = 0 time = 1 KDB: stack backtrace: #0 0xffffffff80c16df7 at ??+0 #1 0xffffffff80bcaccd at ??+0 #2 0xffffffff80bcab23 at ??+0 #3 0xffffffff80f0b03c at ??+0 #4 0xffffffff80f08d8d at ??+0 #5 0xffffffff80f0bb3d at ??+0 #6 0xffffffff80f0b301 at ??+0 #7 0xffffffff80f0b3d1 at ??+0 #8 0xffffffff80f066c4 at ??+0 #9 0xffffffff80f0543f at ??+0 #10 0xffffffff80f23aef at ??+0 #11 0xffffffff80f1133b at ??+0 #12 0xffffffff80b619c8 at ??+0 #13 0xffffffff8036a02c at ??+0 Uptime: 1s =============================== The only solution was to mess with vm.boot_pages . I got it booted with 128 as the value. Also to be clear if I switched back to Home Snoop, Early Snoop the box is fine. Its only unhappy whit Cluster on Die and 12.0-STABLE . Anyone know whats going on ? -- mark saad | nonesuch@longcount.org