From owner-svn-src-head@freebsd.org Fri Dec 30 08:37:32 2016 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 15CE6C97828; Fri, 30 Dec 2016 08:37:32 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E4913122B; Fri, 30 Dec 2016 08:37:31 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id uBU8bVec082815; Fri, 30 Dec 2016 08:37:31 GMT (envelope-from hselasky@FreeBSD.org) Received: (from hselasky@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id uBU8bV6t082814; Fri, 30 Dec 2016 08:37:31 GMT (envelope-from hselasky@FreeBSD.org) Message-Id: <201612300837.uBU8bV6t082814@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: hselasky set sender to hselasky@FreeBSD.org using -f From: Hans Petter Selasky Date: Fri, 30 Dec 2016 08:37:31 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r310806 - head/sys/dev/pci X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Dec 2016 08:37:32 -0000 Author: hselasky Date: Fri Dec 30 08:37:30 2016 New Revision: 310806 URL: https://svnweb.freebsd.org/changeset/base/310806 Log: Add MSIX rewrite table quirk for use with VMs. This patch solves IRQ generation problems using the mlx5en(4) driver with xenserver v6.5.0 in SRIOV and PCI-passthrough modes. Until further the hw.pci.msix_rewrite_table quirk must be set manually in /boot/loader.conf . Reviewed by: jhb @ Sponsored by: Mellanox Technologies MFC after: 2 weeks Modified: head/sys/dev/pci/pci.c Modified: head/sys/dev/pci/pci.c ============================================================================== --- head/sys/dev/pci/pci.c Fri Dec 30 06:26:17 2016 (r310805) +++ head/sys/dev/pci/pci.c Fri Dec 30 08:37:30 2016 (r310806) @@ -356,6 +356,11 @@ static int pci_do_msix = 1; SYSCTL_INT(_hw_pci, OID_AUTO, enable_msix, CTLFLAG_RWTUN, &pci_do_msix, 1, "Enable support for MSI-X interrupts"); +static int pci_msix_rewrite_table = 0; +SYSCTL_INT(_hw_pci, OID_AUTO, msix_rewrite_table, CTLFLAG_RWTUN, + &pci_msix_rewrite_table, 0, + "Rewrite entire MSI-X table when updating MSI-X entries"); + static int pci_honor_msi_blacklist = 1; SYSCTL_INT(_hw_pci, OID_AUTO, honor_msi_blacklist, CTLFLAG_RDTUN, &pci_honor_msi_blacklist, 1, "Honor chipset blacklist for MSI/MSI-X"); @@ -1482,11 +1487,10 @@ pci_find_extcap_method(device_t dev, dev /* * Support for MSI-X message interrupts. */ -void -pci_enable_msix_method(device_t dev, device_t child, u_int index, - uint64_t address, uint32_t data) +static void +pci_write_msix_entry(device_t dev, u_int index, uint64_t address, uint32_t data) { - struct pci_devinfo *dinfo = device_get_ivars(child); + struct pci_devinfo *dinfo = device_get_ivars(dev); struct pcicfg_msix *msix = &dinfo->cfg.msix; uint32_t offset; @@ -1495,6 +1499,31 @@ pci_enable_msix_method(device_t dev, dev bus_write_4(msix->msix_table_res, offset, address & 0xffffffff); bus_write_4(msix->msix_table_res, offset + 4, address >> 32); bus_write_4(msix->msix_table_res, offset + 8, data); +} + +void +pci_enable_msix_method(device_t dev, device_t child, u_int index, + uint64_t address, uint32_t data) +{ + + if (pci_msix_rewrite_table) { + struct pci_devinfo *dinfo = device_get_ivars(child); + struct pcicfg_msix *msix = &dinfo->cfg.msix; + + /* + * Some VM hosts require MSIX to be disabled in the + * control register before updating the MSIX table + * entries are allowed. It is not enough to only + * disable MSIX while updating a single entry. MSIX + * must be disabled while updating all entries in the + * table. + */ + pci_write_config(child, + msix->msix_location + PCIR_MSIX_CTRL, + msix->msix_ctrl & ~PCIM_MSIXCTRL_MSIX_ENABLE, 2); + pci_resume_msix(child); + } else + pci_write_msix_entry(child, index, address, data); /* Enable MSI -> HT mapping. */ pci_ht_map_msi(child, address); @@ -1570,7 +1599,8 @@ pci_resume_msix(device_t dev) if (mte->mte_vector == 0 || mte->mte_handlers == 0) continue; mv = &msix->msix_vectors[mte->mte_vector - 1]; - pci_enable_msix(dev, i, mv->mv_address, mv->mv_data); + pci_write_msix_entry(dev, i, mv->mv_address, + mv->mv_data); pci_unmask_msix(dev, i); } } @@ -4401,12 +4431,20 @@ pci_setup_intr(device_t dev, device_t ch mv->mv_address = addr; mv->mv_data = data; } - if (mte->mte_handlers == 0) { + + /* + * The MSIX table entry must be made valid by + * incrementing the mte_handlers before + * calling pci_enable_msix() and + * pci_resume_msix(). Else the MSIX rewrite + * table quirk will not work as expected. + */ + mte->mte_handlers++; + if (mte->mte_handlers == 1) { pci_enable_msix(child, rid - 1, mv->mv_address, mv->mv_data); pci_unmask_msix(child, rid - 1); } - mte->mte_handlers++; } /*