From owner-freebsd-virtualization@freebsd.org Fri Mar 12 19:05:30 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B8A675AA974 for ; Fri, 12 Mar 2021 19:05:30 +0000 (UTC) (envelope-from khng300@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4DxwHt3cQ9z4l9q for ; Fri, 12 Mar 2021 19:05:30 +0000 (UTC) (envelope-from khng300@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 79D265AA9A1; Fri, 12 Mar 2021 19:05:30 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 799425AA673 for ; Fri, 12 Mar 2021 19:05:30 +0000 (UTC) (envelope-from khng300@gmail.com) Received: from mail-vs1-xe2d.google.com (mail-vs1-xe2d.google.com [IPv6:2607:f8b0:4864:20::e2d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DxwHs5Mxrz4lGj for ; Fri, 12 Mar 2021 19:05:29 +0000 (UTC) (envelope-from khng300@gmail.com) Received: by mail-vs1-xe2d.google.com with SMTP id m18so13069199vsa.1 for ; Fri, 12 Mar 2021 11:05:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=e71Mb8hGfkfN8R0MdI5Grr89NjHjV1BzyWzN1/IxEiM=; b=GZWiV3tgUhYQ/VmErB6O7+wUPDpWMuD9XViDXROydKB3xrnZJlsDHD8f3yj1u2N3q0 yvvKyIsWJ7GZiQZbYLtzfc1p6gKivdB7P/nUVuQSxA89i4wzl4sLXM3wDmXv8mi+lOt8 mRABpVUNEYiwoFhaY44ThCHi66Gd70v0bJRS9cI+kvORIZfMjNP2CwH5oh9ZlBvP7eB1 cIiSfmv67KvIHZAvIEGAStWj/We1qDpVWnfM0LER+sxLcqSaENLFMRFcclReHMEOO6cR 5AxQT8itwosfGytbuL/z6X8zKy0ygvAGKGxeXdR4TqNKw7TqaBDstUL+NPd2WIj9gpYO EMIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=e71Mb8hGfkfN8R0MdI5Grr89NjHjV1BzyWzN1/IxEiM=; b=uaR5Y40aYYL43VgCcTTA4tvWRv4/oiAoNZcv2U/2iS6UYXHVoUwyeYXNY6pXHt86ol 9Ky3AblKyNL8BqHN35h1Wc6gpFdIRsUP46fKU3M2mVZ5jnm3PTUcdWobICty9Rfz2I9T DEHCgFZ7NosVHLyfpMZ5T2B97YWHJIBNUf0TdqVTrdt73vTBTmZ4ig5+JgMmvNtcOOgm 55FqAHmu29tI9usZgACi15WyAzbyymkmMmrtNQ9rw/yfNShzTKOF1WgapAzosNNwcyNo Rzn6AXIsj+rzRh81PXC8RfoYXdad33wa4d0Z7/w7m5jIkkHGwXhR6Tt4GkJtThkEYB1y qjhQ== X-Gm-Message-State: AOAM533UQ5qdONgfLXYu9cGEGbrQdUvk/o4BbSwUk630RFi+6RxDokVN HR8nDOs3DU16ELBQbwgKpe1/XokLe73T5jmX X-Google-Smtp-Source: ABdhPJyJcGOwHDLaWhWgLNPuPw7oKJanuT4DcGZs6V8UoKkpXSyAODwcqG/Lzf+/4G1PfIP6Y99aEg== X-Received: by 2002:a17:902:a513:b029:e5:d91c:2ede with SMTP id s19-20020a170902a513b02900e5d91c2edemr432312plq.65.1615575457175; Fri, 12 Mar 2021 10:57:37 -0800 (PST) Received: from Kas-MacBook-Pro.lan ([2001:470:f816:0:38a3:20aa:ec2a:644b]) by smtp.gmail.com with ESMTPSA id e11sm6244373pfm.24.2021.03.12.10.57.35 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 12 Mar 2021 10:57:36 -0800 (PST) Subject: Re: bhyve PCIe passthrough on AMD Epyc To: toasty@dragondata.com References: <201902280534.x1S5YMQS054621@pdx.rh.CN85.dnsmgr.net> <98792A76-1723-458D-A7B4-BB3F74D9F26C@dragondata.com> Cc: virtualization@freebsd.org From: Ka Ho Ng Message-ID: <3fb42042-7f76-5369-cac0-1ad975fa48d7@gmail.com> Date: Sat, 13 Mar 2021 02:57:35 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Thunderbird/78.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-GB Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4DxwHs5Mxrz4lGj X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=GZWiV3tg; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of khng300@gmail.com designates 2607:f8b0:4864:20::e2d as permitted sender) smtp.mailfrom=khng300@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::e2d:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::e2d:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::e2d:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Mar 2021 19:05:30 -0000 On 3/1/19 2:30 am, toasty at dragondata.com (Kevin Day) wrote: >> On Feb 28, 2019, at 12:28 PM, Anish wrote: >> >> Hi Kevin, >>> I'm just pointing out that it's an AMD Epyc because I know the AMD IOMMU/AMD-Vi code was added somewhat more recently than Intel's and is maybe less tested. >> I tested on Ryzen-I, not sure if anyone tried it on newer Ryzen/Epyc systems. >> >>> CPU0: local APIC error 0x40 >> This is concerning, as per AMD docs, bit6 is for illegal vector which indicate CPU is getting invalid interrupts. >> >> Can you share hw.vmm sysctl output? >> >> -Anish > > Sure: > > hw.vmm.amdvi.domain_id: 4 > hw.vmm.amdvi.disable_io_fault: 0 > hw.vmm.amdvi.ptp_level: 4 > hw.vmm.amdvi.host_ptp: 1 > hw.vmm.amdvi.enable: 1 Could you try https://reviews.freebsd.org/D28945 and https://reviews.freebsd.org/D28984 ? Ka Ho