From owner-freebsd-arm@freebsd.org Fri Oct 26 04:34:50 2018 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3D5B510D5CF7 for ; Fri, 26 Oct 2018 04:34:50 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vtr.rulingia.com (vtr.rulingia.com [IPv6:2001:19f0:5801:ebe:5400:1ff:fe53:30fd]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vtr.rulingia.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8190283481 for ; Fri, 26 Oct 2018 04:34:49 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (ppp59-167-167-3.static.internode.on.net [59.167.167.3]) by vtr.rulingia.com (8.15.2/8.15.2) with ESMTPS id w9Q4YabT069156 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Fri, 26 Oct 2018 15:34:42 +1100 (AEDT) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.15.2/8.15.2) with ESMTPS id w9Q4YTfi065779 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 26 Oct 2018 15:34:29 +1100 (AEDT) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.15.2/8.15.2/Submit) id w9Q4YTrg065778 for freebsd-arm@freebsd.org; Fri, 26 Oct 2018 15:34:29 +1100 (AEDT) (envelope-from peter) Date: Fri, 26 Oct 2018 15:34:29 +1100 From: Peter Jeremy To: freebsd-arm@freebsd.org Subject: Re: Poor virtio performance on Scaleway ARM systems Message-ID: <20181026043429.GA64669@server.rulingia.com> References: <20180916184657.GB24416@server.rulingia.com> <1540309565.1771.1@smtp.migadu.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="BOKacYhQ+x31HxR3" Content-Disposition: inline In-Reply-To: <1540309565.1771.1@smtp.migadu.com> X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.10.0 (2018-05-17) X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Oct 2018 04:34:50 -0000 --BOKacYhQ+x31HxR3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2018-Oct-23 18:46:05 +0300, Greg V wrote: >echo 'hw.pci.honor_msi_blacklist=3D"0"' >> /boot/loader.conf > >With the fix, we're getting ~1000 interrupts on 'its0,4: virtio_pci1'=20 >(instead of ~30000 on gic), latency is no longer awful, and bandwidth=20 >is about 220 MB/s. YAY!! Unfortunately, this isn't working for me on 13-CURRENT. I am still seeing ~4.5MBps disk I/O with ~30,000 interrupts/sec on gic0,s5. I don't even see an its0,4. =20 Looking at a verbose boot, the relevant errors are: its0: mem 0x8080000-0x809ffff on gi= c0 its0: GITS_BASER0: unable to be updated: a907000040a40607 !=3D a907000040ee= 0607 device_attach: its0 attach returned 6 =2E.. virtio_pci0: port 0xf040-0xf05f mem 0x3ee09000= -0x3ee09fff,0x3ee00000-0x3ee03fff at device 1.0 on pci0 vtnet0: on virtio_pci0 virtio_pci0: host features: 0x79bfffe7 virtio_pci0: negotiated features: 0x308fbbe3 vtnet0: bpf attached vtnet0: Ethernet address: de:2b:88:06:a0:4c virtio_pci0: attempting to allocate 3 MSI-X vectors (3 supported) virtio_pci0: attempting to allocate 2 MSI-X vectors (3 supported) virtio_pci0: using legacy interrupt virtio_pci1: port 0xf000-0xf03f mem 0x3ee08000-0= x3ee08fff,0x3ee04000-0x3ee07fff at device 2.0 on pci0 vtblk0: on virtio_pci1 virtio_pci1: host features: 0x79000e54 virtio_pci1: negotiated features: 0x10000e54 virtio_pci1: attempting to allocate 2 MSI-X vectors (2 supported) virtio_pci1: attempting to allocate 2 MSI-X vectors (2 supported) virtio_pci1: using legacy interrupt vtblk0: 47683MB (97656250 512 byte sectors) I haven't had a chance to dig into either of these any further but hope to = do so over the weekend. -- Peter Jeremy --BOKacYhQ+x31HxR3 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE7rKYbDBnHnTmXCJ+FqWXoOSiCzQFAlvSmVRfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEVF QjI5ODZDMzA2NzFFNzRFNjVDMjI3RTE2QTU5N0EwRTRBMjBCMzQACgkQFqWXoOSi CzQ4Yg/+Nap/BL1gukJpj5bO5V7EBC0aijNpgq72zLXqFzmtjvfUjIoAZmdIqtq4 F/U1hRz1oMA932x83fvMlChO45bjBq/BymZG508cGuL6NvlSuy4lFlQxBzR1XFsC sW/AXYBCdavZFdN6qMY3WGKOrfvqzvqCVYGAkcVwmvukBUbvoDy5amy3TXcRPKyL Tr/Bp4woIIZ7LZfbpDdur3u0PAVAfuCNYMQ8esUWGkxGZ43VqNbnmrWGQWPHqCcY 5Ozr8Y53KwdQqsruBAmCWKhcsvBg7YYAkTEfnzt9a8FTw/cc+i5NPIeS0nvaMroT NAJKoh+kt4d0xCHk2MkqIWMnk634XFczzDaOoNU94FbgezAByaou7u7padkFht+B 4tRSsf3KQtJFL6DMvNnxFcfoYdBZ00WGWBgUrFI+CtEyfsxZt990m6GySNsLJwRm h21UnS1Ro6Kne2rrvBeFnCap14qvtePdQlvOMyipcgXX8q5pLxFQ+HKSx7KsJ29v 7q/F0z8NqB0ziAO2w2axPahj9vh7nuER+9LxYq3j2sF5QY9vP6Uj9dRg8EiIEq0Q /kYwCy1I4fS+5YFyvrL1vATP/3E1RuUXn5DcdpiN2g9xhP1RaGTT089rLkyE+jr/ vdzFNKLTDLEUFPIGvc2gyT43qpi3Em/hx11Y6ul6pp37JBrcEBM= =2p1g -----END PGP SIGNATURE----- --BOKacYhQ+x31HxR3--