From owner-svn-src-all@freebsd.org Mon May 25 06:25:32 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4479B2F4FCA; Mon, 25 May 2020 06:25:32 +0000 (UTC) (envelope-from grehan@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49VnCJ14DJz4Pn1; Mon, 25 May 2020 06:25:32 +0000 (UTC) (envelope-from grehan@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1BBA4DCA1; Mon, 25 May 2020 06:25:32 +0000 (UTC) (envelope-from grehan@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 04P6PVs3006579; Mon, 25 May 2020 06:25:31 GMT (envelope-from grehan@FreeBSD.org) Received: (from grehan@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 04P6PVQb006577; Mon, 25 May 2020 06:25:31 GMT (envelope-from grehan@FreeBSD.org) Message-Id: <202005250625.04P6PVQb006577@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: grehan set sender to grehan@FreeBSD.org using -f From: Peter Grehan Date: Mon, 25 May 2020 06:25:31 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r361442 - head/usr.sbin/bhyve X-SVN-Group: head X-SVN-Commit-Author: grehan X-SVN-Commit-Paths: head/usr.sbin/bhyve X-SVN-Commit-Revision: 361442 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 May 2020 06:25:32 -0000 Author: grehan Date: Mon May 25 06:25:31 2020 New Revision: 361442 URL: https://svnweb.freebsd.org/changeset/base/361442 Log: Fix pci-passthru MSI issues with OpenBSD guests - Return 2 x 16-bit registers in the correct byte order for a 4-byte read that spans the CMD/STATUS register. This reversal was hiding the capabilities-list, which prevented the MSI capability from being found for XHCI passthru. - Reorganize MSI/MSI-x config writes so that a 4-byte write at the capability offset would have the read-only portion skipped. This prevented MSI interrupts from being enabled. Reported and extensively tested by Anatoli (me at anatoli dot ws) PR: 245392 Reported by: Anatoli (me at anatoli dot ws) Reviewed by: jhb (bhyve) Approved by: jhb, bz (mentor) MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D24951 Modified: head/usr.sbin/bhyve/pci_emul.c head/usr.sbin/bhyve/pci_emul.h head/usr.sbin/bhyve/pci_passthru.c Modified: head/usr.sbin/bhyve/pci_emul.c ============================================================================== --- head/usr.sbin/bhyve/pci_emul.c Mon May 25 04:57:57 2020 (r361441) +++ head/usr.sbin/bhyve/pci_emul.c Mon May 25 06:25:31 2020 (r361442) @@ -875,7 +875,7 @@ pci_emul_add_msixcap(struct pci_devinst *pi, int msgnu sizeof(msixcap))); } -void +static void msixcap_cfgwrite(struct pci_devinst *pi, int capoff, int offset, int bytes, uint32_t val) { @@ -899,7 +899,7 @@ msixcap_cfgwrite(struct pci_devinst *pi, int capoff, i CFGWRITE(pi, offset, val, bytes); } -void +static void msicap_cfgwrite(struct pci_devinst *pi, int capoff, int offset, int bytes, uint32_t val) { @@ -978,30 +978,34 @@ pci_emul_add_pciecap(struct pci_devinst *pi, int type) /* * This function assumes that 'coff' is in the capabilities region of the - * config space. + * config space. A capoff parameter of zero will force a search for the + * offset and type. */ -static void -pci_emul_capwrite(struct pci_devinst *pi, int offset, int bytes, uint32_t val) +void +pci_emul_capwrite(struct pci_devinst *pi, int offset, int bytes, uint32_t val, + uint8_t capoff, int capid) { - int capid; - uint8_t capoff, nextoff; + uint8_t nextoff; /* Do not allow un-aligned writes */ if ((offset & (bytes - 1)) != 0) return; - /* Find the capability that we want to update */ - capoff = CAP_START_OFFSET; - while (1) { - nextoff = pci_get_cfgdata8(pi, capoff + 1); - if (nextoff == 0) - break; - if (offset >= capoff && offset < nextoff) - break; + if (capoff == 0) { + /* Find the capability that we want to update */ + capoff = CAP_START_OFFSET; + while (1) { + nextoff = pci_get_cfgdata8(pi, capoff + 1); + if (nextoff == 0) + break; + if (offset >= capoff && offset < nextoff) + break; - capoff = nextoff; + capoff = nextoff; + } + assert(offset >= capoff); + capid = pci_get_cfgdata8(pi, capoff); } - assert(offset >= capoff); /* * Capability ID and Next Capability Pointer are readonly. @@ -1018,7 +1022,6 @@ pci_emul_capwrite(struct pci_devinst *pi, int offset, return; } - capid = pci_get_cfgdata8(pi, capoff); switch (capid) { case PCIY_MSI: msicap_cfgwrite(pi, capoff, offset, bytes, val); @@ -1897,7 +1900,7 @@ pci_cfgrw(struct vmctx *ctx, int vcpu, int in, int bus pci_set_cfgdata32(pi, coff, bar); } else if (pci_emul_iscap(pi, coff)) { - pci_emul_capwrite(pi, coff, bytes, *eax); + pci_emul_capwrite(pi, coff, bytes, *eax, 0, 0); } else if (coff >= PCIR_COMMAND && coff < PCIR_REVID) { pci_emul_cmdsts_write(pi, coff, *eax, bytes); } else { Modified: head/usr.sbin/bhyve/pci_emul.h ============================================================================== --- head/usr.sbin/bhyve/pci_emul.h Mon May 25 04:57:57 2020 (r361441) +++ head/usr.sbin/bhyve/pci_emul.h Mon May 25 06:25:31 2020 (r361442) @@ -218,10 +218,6 @@ typedef void (*pci_lintr_cb)(int b, int s, int pin, in int ioapic_irq, void *arg); int init_pci(struct vmctx *ctx); -void msicap_cfgwrite(struct pci_devinst *pi, int capoff, int offset, - int bytes, uint32_t val); -void msixcap_cfgwrite(struct pci_devinst *pi, int capoff, int offset, - int bytes, uint32_t val); void pci_callback(void); int pci_emul_alloc_bar(struct pci_devinst *pdi, int idx, enum pcibar_type type, uint64_t size); @@ -229,6 +225,8 @@ int pci_emul_alloc_pbar(struct pci_devinst *pdi, int i uint64_t hostbase, enum pcibar_type type, uint64_t size); int pci_emul_add_msicap(struct pci_devinst *pi, int msgnum); int pci_emul_add_pciecap(struct pci_devinst *pi, int pcie_device_type); +void pci_emul_capwrite(struct pci_devinst *pi, int offset, int bytes, + uint32_t val, uint8_t capoff, int capid); void pci_emul_cmd_changed(struct pci_devinst *pi, uint16_t old); void pci_generate_msi(struct pci_devinst *pi, int msgnum); void pci_generate_msix(struct pci_devinst *pi, int msgnum); Modified: head/usr.sbin/bhyve/pci_passthru.c ============================================================================== --- head/usr.sbin/bhyve/pci_passthru.c Mon May 25 04:57:57 2020 (r361441) +++ head/usr.sbin/bhyve/pci_passthru.c Mon May 25 06:25:31 2020 (r361442) @@ -811,8 +811,8 @@ passthru_cfgread(struct vmctx *ctx, int vcpu, struct p if (coff == PCIR_COMMAND) { if (bytes <= 2) return (-1); - *rv = pci_get_cfgdata16(pi, PCIR_COMMAND) << 16 | - read_config(&sc->psc_sel, PCIR_STATUS, 2); + *rv = read_config(&sc->psc_sel, PCIR_STATUS, 2) << 16 | + pci_get_cfgdata16(pi, PCIR_COMMAND); return (0); } @@ -842,8 +842,8 @@ passthru_cfgwrite(struct vmctx *ctx, int vcpu, struct * MSI capability is emulated */ if (msicap_access(sc, coff)) { - msicap_cfgwrite(pi, sc->psc_msi.capoff, coff, bytes, val); - + pci_emul_capwrite(pi, coff, bytes, val, sc->psc_msi.capoff, + PCIY_MSI); error = vm_setup_pptdev_msi(ctx, vcpu, sc->psc_sel.pc_bus, sc->psc_sel.pc_dev, sc->psc_sel.pc_func, pi->pi_msi.addr, pi->pi_msi.msg_data, @@ -854,7 +854,8 @@ passthru_cfgwrite(struct vmctx *ctx, int vcpu, struct } if (msixcap_access(sc, coff)) { - msixcap_cfgwrite(pi, sc->psc_msix.capoff, coff, bytes, val); + pci_emul_capwrite(pi, coff, bytes, val, sc->psc_msix.capoff, + PCIY_MSIX); if (pi->pi_msix.enabled) { msix_table_entries = pi->pi_msix.table_count; for (i = 0; i < msix_table_entries; i++) {