From owner-freebsd-virtualization@freebsd.org Sat Jul 27 17:22:11 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 46126BE466 for ; Sat, 27 Jul 2019 17:22:11 +0000 (UTC) (envelope-from aoife@finch.ink) Received: from tegan.alphasoft.com (tegan.alphasoft.com [198.187.188.5]) by mx1.freebsd.org (Postfix) with ESMTP id 1FA778DDA2 for ; Sat, 27 Jul 2019 17:22:09 +0000 (UTC) (envelope-from aoife@finch.ink) Received: from peri.alphasoft.com (peri.alphasoft.com [198.187.188.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tegan.alphasoft.com (Postfix) with ESMTPSA id 796CE22D02E for ; Sat, 27 Jul 2019 10:22:01 -0700 (PDT) From: Aoife Finch Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Linux guest kernel GPF in bhyve host on AMD Ryzen Message-Id: <556F1962-3440-4314-B162-10A7599BDEC9@finch.ink> Date: Sat, 27 Jul 2019 10:22:01 -0700 To: freebsd-virtualization@freebsd.org X-Mailer: Apple Mail (2.3445.104.11) X-Rspamd-Queue-Id: 1FA778DDA2 X-Spamd-Bar: +++++ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [5.19 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MV_CASE(0.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_NONE(0.00)[]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_SPAM_MEDIUM(0.87)[0.868,0]; RBL_VIRUSFREE_BOTNET(2.00)[5.188.187.198.bip.virusfree.cz : 127.0.0.2]; MX_GOOD(-0.01)[tegan.alphasoft.com]; NEURAL_SPAM_LONG(1.00)[0.995,0]; DMARC_NA(0.00)[finch.ink]; NEURAL_HAM_SHORT(-0.15)[-0.149,0]; R_SPF_NA(0.00)[]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:23175, ipnet:198.187.188.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-0.01)[country: US(-0.05)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jul 2019 17:22:11 -0000 Linux guests with recent kernels panic with a GPF during boot on my = system: Ryzen 7 1700 or Ryzen 7 2700 Asus PRIME B350 PLUS - bios 5007 64GB DDR4 RAM FreeBSD 12.0-p7 Here are boot messages from the Ubuntu live server iso = (ubuntu-18.04.2-live-server-amd64.iso): [ 0.024000] Spectre V2 : Spectre mitigation: LFENCE not serializing, = switching to generic retpoline [ 0.948696] ACPI Error: Could not enable RealTimeClock event = (20170831/evxfevnt-218) [ TIME ] Timed out waiting for device = dev-di=E2=80=A6fd\x2db21e\x2dc3186f34105d.device. [DEPEND] Dependency failed for /subiquity_config. [ OK ] Started Uncomplicated firewall. [ OK ] Started Create list of required sta=E2=80=A6vice nodes for the = current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Remount Root and Kernel File Systems. Starting Load/Save Random Seed... [ OK ] Started Load/Save Random Seed. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Load Kernel Modules. Mounting Kernel Configuration File System... Starting Apply Kernel Variables... Mounting FUSE Control File System... [ OK ] Mounted FUSE Control File System. [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Kernel Device Manager. [ OK ] Started Apply Kernel Variables. [ OK ] Started LVM2 metadata daemon. [ OK ] Started Monitoring of LVM2 mirrors,=E2=80=A6sing dmeventd or = progress polling. [ OK ] Started Set the console keyboard layout. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems (Pre). Mounting /media/region.lower... Mounting /tmp... Mounting /media/rack.lower... Mounting /media/filesystem... [ OK ] Mounted /tmp. [ OK ] Reached target Local File Systems. Starting ebtables ruleset management... Starting Set console font and keymap... Starting Create Volatile Files and Directories... Starting Tell Plymouth To Write Out Runtime Data... [ OK ] Mounted /media/filesystem. [ OK ] Mounted /media/region.lower. [ OK ] Mounted /media/rack.lower. Mounting /media/region... Mounting /media/rack... [ OK ] Mounted /media/region. [ OK ] Mounted /media/rack. [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Set console font and keymap. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting Network Time Synchronization... [ 7.012798] general protection fault: 0000 [#1] SMP NOPTI [ 7.013343] Modules linked in: sch_fq_codel ib_iser rdma_cm iw_cm = ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi = ip_tables x_tables autofs4 overlay nls_utf8 isofs raid10 raid456 = async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq = libcrc32c raid1 raid0 multipath linear aesni_intel aes_x86_64 = crypto_simd cryptd glue_helper psmouse virtio_net virtio_blk ahci = libahci [ 7.016815] CPU: 0 PID: 31 Comm: kworker/0:1 Not tainted = 4.15.0-45-generic #48-Ubuntu [ 7.017567] Hardware name: BHYVE, BIOS 1.00 03/14/2014 [ 7.018076] RIP: 0010:switch_mm_irqs_off+0x327/0x400 [ 7.018561] RSP: 0018:ffffa957801e7e20 EFLAGS: 00010046 [ 7.019070] RAX: 0000000000000001 RBX: ffff971ac91a4a40 RCX: = 0000000000000049 [ 7.019765] RDX: 0000000000000000 RSI: ffff971ac91a4a40 RDI: = ffff971ac91a18c0 [ 7.020457] RBP: ffffa957801e7e48 R08: 0000000000000383 R09: = 0000000000000000 [ 7.021141] R10: 0000000000000000 R11: 0000000000000383 R12: = ffff971ac91a18c0 [ 7.021828] R13: 0000000000000000 R14: ffff971ac91a4a40 R15: = ffff971ac91a18c0 [ 7.022520] FS: 0000000000000000(0000) GS:ffff971adfc00000(0000) = knlGS:0000000000000000 [ 7.023295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7.023854] CR2: 00000000000000b0 CR3: 000000001a14c000 CR4: = 00000000000406f0 [ 7.024547] Call Trace: [ 7.024798] __schedule+0x252/0x8a0 [ 7.025141] ? pwq_dec_nr_in_flight+0x49/0xa0 [ 7.025568] schedule+0x2c/0x80 [ 7.025880] worker_thread+0xba/0x410 [ 7.026232] kthread+0x121/0x140 [ 7.026544] ? process_one_work+0x410/0x410 [ 7.026951] ? kthread_create_worker_on_cpu+0x70/0x70 [ 7.027438] ret_from_fork+0x35/0x40 [ 7.027794] Code: c1 49 8b 06 49 8b 7e 08 49 83 c6 18 31 d2 31 f6 e8 = df 73 b8 00 49 8b 06 48 85 c0 75 e7 eb a3 b9 49 00 00 00 b8 01 00 00 00 = 31 d2 <0f> 30 e9 3d fd ff ff 9c 58 0f 1f 44 00 00 49 89 c5 fa 66 0f 1f=20= [ 7.029568] RIP: switch_mm_irqs_off+0x327/0x400 RSP: ffffa957801e7e20 [ 7.030171] ---[ end trace 1379b5776b2d7813 ]--- Similar crashes occur with other Linux distros running recent kernels. = Older kernels (like 4.4.0-135-generic in Ubuntu 16.04) boot and run = fine. Current FreeBSD guests run fine. I swapped the Ryzen 1700 for a 2700 with no change. Unfortunately, I = don't have a spare motherboard or ram to swap. Is this a Ryzen issue or is there something I can tweak to fix this? = Thanks! -- Aoife