From owner-svn-src-all@freebsd.org Wed May 8 23:24:48 2019 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 90D7B159676A; Wed, 8 May 2019 23:24:48 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 35CAC6C965; Wed, 8 May 2019 23:24:48 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E9C90CCFA; Wed, 8 May 2019 23:24:47 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x48NOlHG062686; Wed, 8 May 2019 23:24:47 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x48NOlcJ062685; Wed, 8 May 2019 23:24:47 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201905082324.x48NOlcJ062685@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Wed, 8 May 2019 23:24:47 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org Subject: svn commit: r347374 - stable/11/sys/amd64/vmm/io X-SVN-Group: stable-11 X-SVN-Commit-Author: jhb X-SVN-Commit-Paths: stable/11/sys/amd64/vmm/io X-SVN-Commit-Revision: 347374 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 35CAC6C965 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.93 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.93)[-0.932,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 May 2019 23:24:48 -0000 Author: jhb Date: Wed May 8 23:24:47 2019 New Revision: 347374 URL: https://svnweb.freebsd.org/changeset/base/347374 Log: MFC 333174: Use PCI power-mgmt to reset a device if FLR fails. A large number of devices don't support PCIe FLR, in particular graphics adapters. Use PCI power management to perform the reset if FLR fails or isn't available, by cycling the device through the D3 state. This has been tested by a number of users with Nvidia and AMD GPUs. Modified: stable/11/sys/amd64/vmm/io/ppt.c Directory Properties: stable/11/ (props changed) Modified: stable/11/sys/amd64/vmm/io/ppt.c ============================================================================== --- stable/11/sys/amd64/vmm/io/ppt.c Wed May 8 23:13:27 2019 (r347373) +++ stable/11/sys/amd64/vmm/io/ppt.c Wed May 8 23:24:47 2019 (r347374) @@ -351,6 +351,30 @@ ppt_is_mmio(struct vm *vm, vm_paddr_t gpa) return (FALSE); } +static void +ppt_pci_reset(device_t dev) +{ + int ps; + + if (pcie_flr(dev, + max(pcie_get_max_completion_timeout(dev) / 1000, 10), + true)) + return; + + /* + * If FLR fails, attempt a power-management reset by cycling + * the device in/out of D3 state. + * PCI spec says we can only go into D3 state from D0 state. + * Transition from D[12] into D0 before going to D3 state. + */ + ps = pci_get_powerstate(dev); + if (ps != PCI_POWERSTATE_D0 && ps != PCI_POWERSTATE_D3) + pci_set_powerstate(dev, PCI_POWERSTATE_D0); + if (pci_get_powerstate(dev) != PCI_POWERSTATE_D3) + pci_set_powerstate(dev, PCI_POWERSTATE_D3); + pci_set_powerstate(dev, ps); +} + int ppt_assign_device(struct vm *vm, int bus, int slot, int func) { @@ -366,9 +390,7 @@ ppt_assign_device(struct vm *vm, int bus, int slot, in return (EBUSY); pci_save_state(ppt->dev); - pcie_flr(ppt->dev, - max(pcie_get_max_completion_timeout(ppt->dev) / 1000, 10), - true); + ppt_pci_reset(ppt->dev); pci_restore_state(ppt->dev); ppt->vm = vm; iommu_add_device(vm_iommu_domain(vm), pci_get_rid(ppt->dev)); @@ -391,9 +413,7 @@ ppt_unassign_device(struct vm *vm, int bus, int slot, return (EBUSY); pci_save_state(ppt->dev); - pcie_flr(ppt->dev, - max(pcie_get_max_completion_timeout(ppt->dev) / 1000, 10), - true); + ppt_pci_reset(ppt->dev); pci_restore_state(ppt->dev); ppt_unmap_mmio(vm, ppt); ppt_teardown_msi(ppt);