From owner-freebsd-virtualization@freebsd.org Thu Mar 26 12:38:08 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EA76927E5C7 for ; Thu, 26 Mar 2020 12:38:08 +0000 (UTC) (envelope-from elenamihailescu22@gmail.com) Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48p4Jr5qsNz4GcH; Thu, 26 Mar 2020 12:38:03 +0000 (UTC) (envelope-from elenamihailescu22@gmail.com) Received: by mail-oi1-f176.google.com with SMTP id w2so5317078oic.5; Thu, 26 Mar 2020 05:38:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc; bh=Z4lkAjJiz/hfwCiNW3GebUe1dDffWEAaPMyHUUKoM48=; b=ANTa8zwG/x64yUHeH3fKw13nkzvI/HA7TqKC1zbLK1FW0rNTHpsW9s+ZmquXMhsjrG rRoDwIbqX1Hi3z3qc6We+olS8UkPxy787r92v0BFD6BtEODajPjXCfxwtlg5j+0cHMip Um+BwWzbHDnxgn2iTsZnDXgpiRgr4DPalSxJJ4hqKKSieEFwV7A/ROvXJifenMY94MRH pWPg8ALtelpoYEFXYB615Cn2PYNB0821u0LP5C8QvRGZgndLGB8dPaxx4hqhjmDq4u3n 9Y9nP/9xo9lnnRdxrbnEhNgKM2lbYkCY/gEZm5Qn56EPUtT4761cBhoC6IL4bGA1vHmt 9GCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=Z4lkAjJiz/hfwCiNW3GebUe1dDffWEAaPMyHUUKoM48=; b=S4yZe3AY+vCIhHGERBZGruGQlPU/x+6t1yNXXpSKoFX7BHuqEl79n/qyLB/SVi1aA9 DAJMklxYu0XtI8Nmzr0zDzdRB+Kr0s+k225U974yWvfRkgtd+duILUExUC/ZqbuExCVy 7Mvnwy3VPmaQW2gdJCyAmw+X6wOfOlQt9ShoBdKSHawjXzyMGIO6+1OR7fZUaQtLYJ3L NAORcGtz0SW6qXJGDZ5Xlio9HwURIb24PwHsWbSSFLWi2NhGXNbIDi+S2rk4zMvV6kW+ MmLlm4XVH1WDFrNPWB5S3bq3WlIi/aS8AYc0ieHxnodaGtH9HSnPdjBDrz2EeljvKcvg oFow== X-Gm-Message-State: ANhLgQ1/LhWqkPyzaze62QrHH23MWFM51UVc9dCUf0rPDP6Rtyukf0JB cEXZQlBKUqC2puXiawEHfEzLSNZ7xPvNPFl7ouqhHHXG X-Google-Smtp-Source: ADFU+vv9qnC+d6oPEQJqTOsgtQgR/m1NLrC77x+anJMYvirfEYjZ0hL3jg0jgY0IXCkcOwoyEfrM+lLGmYUG3tjz42c= X-Received: by 2002:a54:4197:: with SMTP id 23mr1072211oiy.87.1585226275140; Thu, 26 Mar 2020 05:37:55 -0700 (PDT) MIME-Version: 1.0 From: Elena Mihailescu Date: Thu, 26 Mar 2020 14:37:29 +0200 Message-ID: Subject: Run bhyve in KVM on Intel E5-2660 v4 issues To: freebsd-virtualization@freebsd.org Cc: Darius Mihai , Mihai Carabas , John Baldwin Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 48p4Jr5qsNz4GcH X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=ANTa8zwG; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of elenamihailescu22@gmail.com designates 209.85.167.176 as permitted sender) smtp.mailfrom=elenamihailescu22@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[176.167.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(0.00)[ip: (-1.20), ipnet: 209.85.128.0/17(-1.40), asn: 15169(-0.48), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FREEMAIL_CC(0.00)[gmail.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2020 12:38:09 -0000 Hello, We have some issues on using bhyve from inside a FreeBSD KVM virtual machine on Intel E5-2660 v4 servers. The host operating systems we tried and the errors are described below. If the host is one of the following operating systems: Ubuntu 18.04, Ubuntu 18.04 for servers or CentOS8 we get the following behavior: Tested on FreeBSD-Current (FreeBSD-13.0-CURRENT-amd64-20200213-r357847) and FreeBSD12.1: - starting bhyve using bhyveload as a loader ends with a kernel panic in the bhyve VM due to APIC error: # sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 1G -d guest.img -i -I FreeBSD-12.1-RELEASE-amd64-disc1.iso machine ==== Event timer "LAPIC" quality 600 ACPI APIC Table: panic: APIC: CPU with APIC ID 255 is not enabled cpuid = 0 time = 1 KDB: stack backtrace: #0 0xffffffff80c1d297 at kdb_backtrace+0x67 #1 0xffffffff80bd05cd at vpanic+0x19d #2 0xffffffff80bd0423 at panic+0x43 #3 0xffffffff811f7569 at madt_set_ids+0x99 #4 0xffffffff80b66918 at mi_startup+0x118 #5 0xffffffff8036b02c at btext+0x2c Uptime: 1s ==== - starting bhyve using uefi as a loader seems to stuck or run in an infinite loop - nothing is displayed on screen after starting the VM, we cannot interact with it. bhyve -c 1 -m 1G -H -P -S -s 0:0,hostbridge -s 1:0,lpc -s 4:0,ahci-hd,guest.img -s 3:0,ahci-cd, -l com1,stdio -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd *stuck - does not receive input, does not display anything. We also tried disabling x2apic from the host kernel options and also from the KVM guest options. As for libvirt options, we tried both cpu options: host-model and host-passthrough. If the host is a CentOS7, we didn't manage to get the FreeBSD KVM virtual machine to see the hardware virtualization features and we cannot run `kldload vmm` successfully (error: vmx_init: processor does not support desired basic capabilities). We have tested with a CentOS7 guest and nested virtualization with kvm does not have any issues. In the document at [1], we centralized the tests we run and their output, including virsh xml files, bhyve commands, and the errors we have encounters. As a parallel, we installed FreeBSD directly on the server and ran the same bhyve commands that worked fine. Does anyone know any way to solve this issue? [1] https://docs.google.com/document/d/1DIzdJKHNZNLZXk9dJquUDwwYJpTH2zDkT9eF2sVIb-A/edit?usp=sharing Thank you, Elena