From owner-svn-src-head@freebsd.org Wed May 2 17:41:01 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE63DFAFCF7; Wed, 2 May 2018 17:41:00 +0000 (UTC) (envelope-from grehan@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6FB6973916; Wed, 2 May 2018 17:41:00 +0000 (UTC) (envelope-from grehan@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 6AA9F14272; Wed, 2 May 2018 17:41:00 +0000 (UTC) (envelope-from grehan@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w42Hf0pB078874; Wed, 2 May 2018 17:41:00 GMT (envelope-from grehan@FreeBSD.org) Received: (from grehan@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w42Hf06u078873; Wed, 2 May 2018 17:41:00 GMT (envelope-from grehan@FreeBSD.org) Message-Id: <201805021741.w42Hf06u078873@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: grehan set sender to grehan@FreeBSD.org using -f From: Peter Grehan Date: Wed, 2 May 2018 17:41:00 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r333174 - head/sys/amd64/vmm/io X-SVN-Group: head X-SVN-Commit-Author: grehan X-SVN-Commit-Paths: head/sys/amd64/vmm/io X-SVN-Commit-Revision: 333174 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 May 2018 17:41:01 -0000 Author: grehan Date: Wed May 2 17:41:00 2018 New Revision: 333174 URL: https://svnweb.freebsd.org/changeset/base/333174 Log: Use PCI power-mgmt to reset a device if FLR fails. A large number of devices don't support PCIe FLR, in particular graphics adapters. Use PCI power management to perform the reset if FLR fails or isn't available, by cycling the device through the D3 state. This has been tested by a number of users with Nvidia and AMD GPUs. Submitted and tested by: Matt Macy Reviewed by: jhb, imp, rgrimes MFC after: 3 weeks Differential Revision: https://reviews.freebsd.org/D15268 Modified: head/sys/amd64/vmm/io/ppt.c Modified: head/sys/amd64/vmm/io/ppt.c ============================================================================== --- head/sys/amd64/vmm/io/ppt.c Wed May 2 15:59:15 2018 (r333173) +++ head/sys/amd64/vmm/io/ppt.c Wed May 2 17:41:00 2018 (r333174) @@ -353,6 +353,30 @@ ppt_is_mmio(struct vm *vm, vm_paddr_t gpa) return (FALSE); } +static void +ppt_pci_reset(device_t dev) +{ + int ps; + + if (pcie_flr(dev, + max(pcie_get_max_completion_timeout(dev) / 1000, 10), + true)) + return; + + /* + * If FLR fails, attempt a power-management reset by cycling + * the device in/out of D3 state. + * PCI spec says we can only go into D3 state from D0 state. + * Transition from D[12] into D0 before going to D3 state. + */ + ps = pci_get_powerstate(dev); + if (ps != PCI_POWERSTATE_D0 && ps != PCI_POWERSTATE_D3) + pci_set_powerstate(dev, PCI_POWERSTATE_D0); + if (pci_get_powerstate(dev) != PCI_POWERSTATE_D3) + pci_set_powerstate(dev, PCI_POWERSTATE_D3); + pci_set_powerstate(dev, ps); +} + int ppt_assign_device(struct vm *vm, int bus, int slot, int func) { @@ -368,9 +392,7 @@ ppt_assign_device(struct vm *vm, int bus, int slot, in return (EBUSY); pci_save_state(ppt->dev); - pcie_flr(ppt->dev, - max(pcie_get_max_completion_timeout(ppt->dev) / 1000, 10), - true); + ppt_pci_reset(ppt->dev); pci_restore_state(ppt->dev); ppt->vm = vm; iommu_add_device(vm_iommu_domain(vm), pci_get_rid(ppt->dev)); @@ -393,9 +415,7 @@ ppt_unassign_device(struct vm *vm, int bus, int slot, return (EBUSY); pci_save_state(ppt->dev); - pcie_flr(ppt->dev, - max(pcie_get_max_completion_timeout(ppt->dev) / 1000, 10), - true); + ppt_pci_reset(ppt->dev); pci_restore_state(ppt->dev); ppt_unmap_mmio(vm, ppt); ppt_teardown_msi(ppt);