From owner-freebsd-hackers@freebsd.org Thu Jun 6 21:37:37 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 514F515C0F96 for ; Thu, 6 Jun 2019 21:37:37 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-ed1-x544.google.com (mail-ed1-x544.google.com [IPv6:2a00:1450:4864:20::544]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B02D78921D for ; Thu, 6 Jun 2019 21:37:34 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: by mail-ed1-x544.google.com with SMTP id p26so5412273edr.2 for ; Thu, 06 Jun 2019 14:37:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=longcount-org.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=8ZsJ8Ed2PrEtoVKYMVSO8CevjX9EPLC3SZHCLD59Fi0=; b=roAF3F9sjJn0iyT8ImJcsyT4XtEbWJXqysnUnR49cQ2l5Ls44c3eWE0Gg1j0lodeuS id/l3xU5OYLwQzNi+4Kg0WCbOSo+rKJToE9XeuMS0ses2YDdpQDfNdXHYFOONcN8ryGj ZFgyiaPVkk6cmzkSiBAb3D4FZlg7gpYoF3kYylRfTB2+6P6DjbNJPckdNP9wlk9AJUwv h2yzlH/Qg4125+uaJXV3XqTeQxm9J4hGsR+BoR3nj7Yqms6J0FkBDhr2m2c8qyhLkdsx vO3PBx3ILNXzQJkxACbvtUWmZik1MzV6zJoV/tZEssLSlHsf4lqCIJor7gWKRqcfzUm8 RTVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=8ZsJ8Ed2PrEtoVKYMVSO8CevjX9EPLC3SZHCLD59Fi0=; b=gjjyBj0ROBbVt3yXb6llGzbxqkxySuFkEUJ7CYo7ftUNVkUOqjOqVs4uloBMep6Bki DuwdLtvPxjMy67fX4063uKBbBpdjdqxy0jRiimIxu/wljsugoOQ35pzdEAp+Rpkfhn4p ytUM7q17kc+vnfIcKXzmFALJ1McUt5FyO8KkysOKH5yRsE0yrApMxQZHIQeipk+b8D/q SINWDanEIfqKpjIWAQCNPz81h26Q6r3MDVZghNZk5y3FBAjuqfzSKe2lRssctMfgtSen +8XFPXu0zegJmE/hRXGmzjhw/YEztr8lFNvQGGFV2zyGXyJhysJn3AIRC2p6Y0by4q/J xYTA== X-Gm-Message-State: APjAAAWpzFkP+J7iG/Jbn2As8eSUhJiJQG89RAFGSiW2oQbv8GZWPBg9 HWi3DcQF+4AvTFaXYvxWhOYwFX6/ZKFyrLbtlKHGjQ== X-Google-Smtp-Source: APXvYqzUl4Jo9uPE47aC1hOfHh2tOK7LYiYYdekQ9E9wiOVgGhi/L7wG/eLJDf3/ldLSwTx370VfNLQqr7EDFtr7pOo= X-Received: by 2002:a50:b665:: with SMTP id c34mr55274743ede.148.1559857053380; Thu, 06 Jun 2019 14:37:33 -0700 (PDT) MIME-Version: 1.0 References: <20190606145016.GA4116@raichu> In-Reply-To: <20190606145016.GA4116@raichu> From: Mark Saad Date: Thu, 6 Jun 2019 17:37:22 -0400 Message-ID: Subject: Re: Kernel panic on 12-STABLE-r348203 amd64 E5-2690v4 with Cluster on die mode enabled To: Mark Johnston Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: B02D78921D X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=longcount-org.20150623.gappssmtp.com header.s=20150623 header.b=roAF3F9s X-Spamd-Result: default: False [-4.28 / 15.00]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[longcount-org.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; MX_GOOD(-0.01)[alt1.aspmx.l.google.com,aspmx.l.google.com,aspmx5.googlemail.com,aspmx4.googlemail.com,aspmx3.googlemail.com,alt2.aspmx.l.google.com,aspmx2.googlemail.com]; NEURAL_HAM_SHORT(-0.96)[-0.963,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; IP_SCORE(-1.01)[ip: (-0.36), ipnet: 2a00:1450::/32(-2.32), asn: 15169(-2.29), country: US(-0.06)]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.997,0]; R_DKIM_ALLOW(-0.20)[longcount-org.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[longcount.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[4.4.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jun 2019 21:37:37 -0000 Mark I am stumped , I realized I had the stock bios settings, and I just applied my standard config and it just worked. I am not 100% sure what issue was. Went back and turned on COD mode after and it booted fine. I even turned on x2apic for fun; it booted but the mfid driver went nuts and would not load correctly. So when I have more time I am going to find a un-touched r630 dump the bios options and reload it and see what happens. On Thu, Jun 6, 2019 at 10:50 AM Mark Johnston wrote: > > On Thu, Jun 06, 2019 at 10:43:26AM -0400, Mark Saad wrote: > > All > > I posed this yesterday but; I am not sure what happened. Here is the > > short version. > > I received two new Dell r630's each with a E5-2690 v4 . The E5-2690 v4 > > has 14 Cores in two packages on the one chip. I don't remember the > > exact topology however as a result the BIOS supports the NUMA / Memory > > mode know as Cluster on Die were each package on the one chip shows up > > as its own NUMA domain. The issue is this when enabled the box boots > > 12-RELEASE a-ok. When I rebuilt 12.0-STABLE-r348203 it would panic > > early in the boot process. > > Here is a dump of the console > > =============================== > > Loading kernel... > > /boot/kernel/kernel text=0x168d811 data=0x1cf968+0x768c80 > > syms=[0x8+0x1778e8+0x8 / > > +0x194f1d] > > Loading configured modules... > > /boot/kernel/ipmi.ko size 0x11e10 at 0x2645000 > > loading required module 'smbus' > > /boot/kernel/smbus.ko size 0x2ef0 at 0x2657000 > > /boot/entropy size=0x1000 > > /boot/kernel/cc_httcp.ko size 0x2330 at 0x265b000 > > ---<>---c_hmodule 'smbus' > > Copyright (c) 1992-2019 The FreeBSD Project. > > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > > The Regents of the University of California. All rights reserved. > > FreeBSD is a registered trademark of The FreeBSD Foundation. > > FreeBSD 12.0-STABLE r348693 GENERIC amd64 > > FreeBSD clang version 8.0.0 (tags/RELEASE_800/final 356365) (based on > > LLVM 8.0.0) > > panic: UMA zone "UMA Zones": Increase vm.boot_pages > > cpuid = 0 > > time = 1 > > KDB: stack backtrace: > > #0 0xffffffff80c16df7 at ??+0 > > #1 0xffffffff80bcaccd at ??+0 > > #2 0xffffffff80bcab23 at ??+0 > > #3 0xffffffff80f0b03c at ??+0 > > #4 0xffffffff80f08d8d at ??+0 > > #5 0xffffffff80f0bb3d at ??+0 > > #6 0xffffffff80f0b301 at ??+0 > > #7 0xffffffff80f0b3d1 at ??+0 > > #8 0xffffffff80f066c4 at ??+0 > > #9 0xffffffff80f0543f at ??+0 > > #10 0xffffffff80f23aef at ??+0 > > #11 0xffffffff80f1133b at ??+0 > > #12 0xffffffff80b619c8 at ??+0 > > #13 0xffffffff8036a02c at ??+0 > > Uptime: 1s > > > > =============================== > > > > The only solution was to mess with vm.boot_pages . I got it booted > > with 128 as the value. > > Also to be clear if I switched back to Home Snoop, Early Snoop the box > > is fine. Its only > > unhappy whit Cluster on Die and 12.0-STABLE . > > > > Anyone know whats going on ? > > Could you build a kernel with "options DIAGNOSTIC" configured and boot > in verbose mode? The kernel should print its boot page allocations to > the console. Then, compare the output with a boot with cluster on die > disabled. -- mark saad | nonesuch@longcount.org