Date: Thu, 25 Oct 2018 08:58:40 -0700 From: John Baldwin <jhb@FreeBSD.org> To: Peter Jeremy <peter@rulingia.com> Cc: Greg V <greg@unrelenting.technology>, freebsd-arm@freebsd.org Subject: Re: Poor virtio performance on Scaleway ARM systems Message-ID: <f232c2cd-eecc-91e9-eb44-dbeb5ccefd0a@FreeBSD.org> In-Reply-To: <20181023184949.GB20902@server.rulingia.com> References: <20180916184657.GB24416@server.rulingia.com> <1540309565.1771.1@smtp.migadu.com> <966ed99b-4709-5693-dca2-77e3e6d7d830@FreeBSD.org> <20181023184949.GB20902@server.rulingia.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 10/23/18 11:49 AM, Peter Jeremy wrote: > On 2018-Oct-23 10:04:04 -0700, John Baldwin <jhb@FreeBSD.org> wrote: >> On 10/23/18 8:46 AM, Greg V wrote: >>> On Sun, Sep 16, 2018 at 9:46 PM, Peter Jeremy <peter@rulingia.com> >>> wrote: >>>> I have been playing with the 4-core ARM64 VPSs on >>>> https://www.scaleway.com >>>> and notice that the disk I/O performance (using virtio_blk) is >>>> abyssmal. >>>> >>>> The only oddity I've found is that FreeBSD is reporting very high >>>> interrupt >>>> rates on gic0,p11, gic0,s4 and gic0,s5 whilst disk I/O is occurring. >>>> Unfortunately, I can't tell what is attached to those interrupts (it's >>>> not obvious from the dmesg and reported as "+"). >>> gic0,s4 correlates to network activity. >>> gic0,s5 correlates to disk activity. >>> >>> echo 'hw.pci.honor_msi_blacklist="0"' >> /boot/loader.conf > >> Can you make 'pciconf -lc' output available? There is a "generic" >> blacklist that assumes we will see a host bridge device with PCI-e >> or PCI-X and if neither of those is present we also blacklist MSI. >> However, we can certainly figure out how to make this work out of >> the box. > > $ sudo pciconf -lc > hostb0@pci0:0:0:0: class=0x060000 card=0x11001af4 chip=0x00081b36 rev=0x00 hdr=0x00 > virtio_pci0@pci0:0:1:0: class=0x020000 card=0x00011af4 chip=0x10001af4 rev=0x00 hdr=0x00 > cap 11[98] = MSI-X supports 3 messages > Table in map 0x14[0x0], PBA in map 0x14[0x800] > cap 09[84] = vendor (length 20) > cap 09[70] = vendor (length 20) > cap 09[60] = vendor (length 16) > cap 09[50] = vendor (length 16) > cap 09[40] = vendor (length 16) > virtio_pci1@pci0:0:2:0: class=0x010000 card=0x00021af4 chip=0x10011af4 rev=0x00 hdr=0x00 > cap 11[98] = MSI-X supports 2 messages > Table in map 0x14[0x0], PBA in map 0x14[0x800] > cap 09[84] = vendor (length 20) > cap 09[70] = vendor (length 20) > cap 09[60] = vendor (length 16) > cap 09[50] = vendor (length 16) > cap 09[40] = vendor (length 16) The relevant code is here in sys/dev/pci/pci.c: /* * Determine if MSI is blacklisted globally on this system. Currently, * we just check for blacklisted chipsets as represented by the * host-PCI bridge at device 0:0:0. In the future, it may become * necessary to check other system attributes, such as the kenv values * that give the motherboard manufacturer and model number. */ static int pci_msi_blacklisted(void) { device_t dev; if (!pci_honor_msi_blacklist) return (0); /* Blacklist all non-PCI-express and non-PCI-X chipsets. */ if (!(pcie_chipset || pcix_chipset)) { if (vm_guest != VM_GUEST_NO) { /* * Whitelist older chipsets in virtual * machines known to support MSI. */ dev = pci_find_bsf(0, 0, 0); if (dev != NULL) return (!pci_has_quirk(pci_get_devid(dev), PCI_QUIRK_ENABLE_MSI_VM)); } return (1); } dev = pci_find_bsf(0, 0, 0); if (dev != NULL) return (pci_msi_device_blacklisted(dev)); return (0); } In this case no devices have PCI-express or PCI-X capabilities. We could perhaps whitelist your hostb device, but I think a more general solution might be to whitelist MSI if there are any virtio devices as virtio devices can always do MSI. I can work on a patch unless someone else beats me to it, but for the virtio approach what I would do is have a global 'virtio_seen' similar to 'pcie_chipset' that gets set to true if we find any virtio devices during the initial bus scans and then add '|| virtio_seen' to the second 'if' there. -- John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?f232c2cd-eecc-91e9-eb44-dbeb5ccefd0a>