From owner-freebsd-virtualization@freebsd.org Tue Feb 27 17:55:22 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 067C6F3B137 for ; Tue, 27 Feb 2018 17:55:22 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost2.sentex.ca (smarthost2.sentex.ca [IPv6:2607:f3e0:80:80::2]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "smarthost.sentex.ca", Issuer "smarthost.sentex.ca" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A7BE486042; Tue, 27 Feb 2018 17:55:21 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (lava.sentex.ca [IPv6:2607:f3e0:0:5::11]) by smarthost2.sentex.ca (8.15.2/8.15.2) with ESMTPS id w1RHtKcH071210 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 27 Feb 2018 12:55:21 -0500 (EST) (envelope-from mike@sentex.net) Received: from [192.168.43.26] (saphire3.sentex.net [192.168.43.26]) by lava.sentex.ca (8.15.2/8.15.2) with ESMTP id w1RHtJCH040626; Tue, 27 Feb 2018 12:55:19 -0500 (EST) (envelope-from mike@sentex.net) Subject: Re: Ryzen lockup on bhyve was (Re: new Ryzen lockup issue ?) To: Peter Grehan Cc: "freebsd-virtualization@freebsd.org" References: <92a60e14-f532-2647-d45d-b500fc59ba88@sentex.net> <425be16f-9fdc-9ed6-72b1-02e28bfd130f@sentex.net> <2f2b6f96-d6ef-8e02-966d-dcbbfb0c92fb@freebsd.org> From: Mike Tancsa Organization: Sentex Communications Message-ID: <48768738-e699-fd06-9154-9b5be1bbabcc@sentex.net> Date: Tue, 27 Feb 2018 12:55:20 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <2f2b6f96-d6ef-8e02-966d-dcbbfb0c92fb@freebsd.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.78 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Feb 2018 17:55:22 -0000 On 2/25/2018 12:51 PM, Peter Grehan wrote: > Hi Mike, > >  (moving this to -virtualization) > >> Actually I can confirm the same sort of hard lockup happens on my Epyc >> board with RELENG11.  It also happens in current. I will file a PR and >> post on freebsd-current in case someone has any suggestions on how to >> try and figure out whats going on. > >  I could reproduce this on a Ryzen 1700 with HT enabled - it took ~700 > seconds, and the entire machine locked up. > >  An interesting point is a single VM was able to run to completion. > >  I'll keep testing with various combinations of numbers of VMs, > different vCPUs per VM, and pinning guest vCPUs. Hi Peter, I upgraded both boxes (Ryzen and Epyc) to HEAD. For the record, the Epyc CPU is CPU1 Information AMD EPYC 7281 16-Core Family: 17h Model: 00h-0Fh Microcode Patch Level: 8001207 Bios Release Date: 01/08/2018 SuperMicro: H11SSL-i Not sure if its related to core count or not, but it takes several instances on the Epyc to lock up and takes a lot longer. I enabled IOMMU on it as well, and it's dmesg shows below. But the lockups are still there with or without it enabled. ivhd0: on acpi0 ivhd0: Flag:b0 ivhd0: Features(type:0x11) MsiNumPPR = 0 PNBanks= 2 PNCounters= 0 ivhd0: Extended features[31:0]:22294ada HATS = 0x2 GATS = 0x0 GLXSup = 0x1 SmiFSup = 0x1 SmiFRC = 0x2 GAMSup = 0x1 DualPortLogSup = 0x2 DualEventLogSup = 0x2 ivhd0: Extended features[62:32]:f77ef Max PASID: 0x2f DevTblSegSup = 0x3 MarcSup = 0x1 ivhd0: supported paging level:7, will use only: 4 ivhd0: device range: 0x0 - 0xffff ivhd0: PCI cap 0x190b640f@0x40 feature:19 ivhd1: on acpi0 ivhd1: Flag:b0 ivhd1: Features(type:0x11) MsiNumPPR = 0 PNBanks= 2 PNCounters= 0 ivhd1: Extended features[31:0]:22294ada HATS = 0x2 GATS = 0x0 GLXSup = 0x1 SmiFSup = 0x1 SmiFRC = 0x2 GAMSup = 0x1 DualPortLogSup = 0x2 DualEventLogSup = 0x2 ivhd1: Extended features[62:32]:f77ef Max PASID: 0x2f DevTblSegSup = 0x3 MarcSup = 0x1 ivhd1: supported paging level:7, will use only: 4 ivhd1: device range: 0x0 - 0xffff ivhd1: PCI cap 0x190b640f@0x40 feature:19 ivhd2: on acpi0 ivhd2: Flag:b0 ivhd2: Features(type:0x11) MsiNumPPR = 0 PNBanks= 2 PNCounters= 0 ivhd2: Extended features[31:0]:22294ada HATS = 0x2 GATS = 0x0 GLXSup = 0x1 SmiFSup = 0x1 SmiFRC = 0x2 GAMSup = 0x1 DualPortLogSup = 0x2 DualEventLogSup = 0x2 ivhd2: Extended features[62:32]:f77ef Max PASID: 0x2f DevTblSegSup = 0x3 MarcSup = 0x1 ivhd2: supported paging level:7, will use only: 4 ivhd2: device range: 0x0 - 0x3ffe ivhd2: PCI cap 0x190b640f@0x40 feature:19 ivhd3: on acpi0 ivhd3: Flag:b0 ivhd3: Features(type:0x11) MsiNumPPR = 0 PNBanks= 2 PNCounters= 0 ivhd3: Extended features[31:0]:22294ada HATS = 0x2 GATS = 0x0 GLXSup = 0x1 SmiFSup = 0x1 SmiFRC = 0x2 GAMSup = 0x1 DualPortLogSup = 0x2 DualEventLogSup = 0x2 ivhd3: Extended features[62:32]:f77ef Max PASID: 0x2f DevTblSegSup = 0x3 MarcSup = 0x1 ivhd3: supported paging level:7, will use only: 4 ivhd3: device range: 0x0 - 0x3ffe ivhd3: PCI cap 0x190b640f@0x40 feature:19 ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 x203 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada