From owner-freebsd-hackers@FreeBSD.ORG Wed Mar 19 14:02:47 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 21A2E85C for ; Wed, 19 Mar 2014 14:02:47 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 55C25AA7 for ; Wed, 19 Mar 2014 14:02:46 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.8/8.14.8) with ESMTP id s2JE2aP5023708; Wed, 19 Mar 2014 16:02:36 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua s2JE2aP5023708 Received: (from kostik@localhost) by tom.home (8.14.8/8.14.8/Submit) id s2JE2adp023707; Wed, 19 Mar 2014 16:02:36 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 19 Mar 2014 16:02:36 +0200 From: Konstantin Belousov To: Ryan Stone Subject: Re: [PATCH] Support PCIe Alternative RID Interpretation (ARI) Message-ID: <20140319140236.GM21331@kib.kiev.ua> References: <20140316141216.GA21331@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="kunpHVz1op/+13PW" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: "freebsd-hackers@freebsd.org" , Neel Natu X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Mar 2014 14:02:47 -0000 --kunpHVz1op/+13PW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Mar 18, 2014 at 01:51:47PM -0400, Ryan Stone wrote: > On Mon, Mar 17, 2014 at 5:20 PM, Neel Natu wrote: > > Hi Ryan, > > In any case it seems that the VT-d implementation in bhyve will work > > fine with ARI enabled devices. >=20 > There was an assert that would trip for the function number being > greater than 8. >=20 >=20 > I've put together the following patch series: >=20 > http://people.freebsd.org/~rstone/patches/ari/0001-Add-a-method-to-get-th= e-PCI-RID-for-a-device.patch >=20 I do not like this, in fact. Not the general idea, but the implementation. I think that what is needed is method which would return combined slot/function number (or just function number for ARI-enabled device). Then, existing pci_get_bus() and this new method are enough for proper construction of the translating structures. > This adds a method to get the RID that will be consumed by the VT-d > drivers. This patch is non-ARI only. >=20 >=20 > http://people.freebsd.org/~rstone/patches/ari/0002-Re-implement-the-DMAR-= I-O-MMU-code-in-terms-of-PCI-R.patch >=20 > This reworks the busdma DMAR code to work in terms of RIDs where > necessary. This should be a no-op. I tested this with > hw.dmar.enable=3D1 on a Nehalem with the em driver and a Sandy Bridge > with the igb and ixgbe driver and was able to pass traffic. Again, I do not like this. IMO the patch is too conservative. Almost all occurences of the s/f in the IOMMU code must be removed, and replaced by the half-rid returned by the new method from above. In particular, context must be stripped of s/f at all. As before, if you want, omit this part altogether, and I will try to do the pass later. >=20 >=20 > http://people.freebsd.org/~rstone/patches/ari/0003-Re-write-bhyve-s-I-O-M= MU-handling-in-terms-of-PCI-RI.patch >=20 > Same thing, but for bhyve. I'm not sure that passing the rid down to > the CPU-dependent layer is the right thing to do, because I'm not sure > what the AMD VT-d equivalent requires. Should I just pass down the > entire device_t and let the CPU layer deal with it? I tested loading > vmm.ko with a device assigned to the ppt driver but I didn't go as far > as starting a VM and using PCI passthrough. >=20 > (Also, as you'd probably expected doing this with hw.dmar.enable=3D1 > causes all hell to break loose). >=20 >=20 > http://people.freebsd.org/~rstone/patches/ari/0004-Add-support-for-PCIe-A= RI.patch >=20 > This is a slightly reworked version of the previous patch. The main > difference are that there is a new implementation of pcib_get_rid that > understands ARI RIDs. I also fixed a bug where the default > implementation of pcib_numslots was not actually being used because I > misspelled DEFAULT as default in pcib_if.m. >=20 >=20 > http://people.freebsd.org/~rstone/patches/ari/0005-Print-status-of-ARI-ca= pability-in-pciconf-c.patch >=20 > This makes pciconf -c dump the status of ARI on PCIe downstream ports. --kunpHVz1op/+13PW Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJTKaN7AAoJEJDCuSvBvK1BRVsP/AsDqJ1VK7EfC21CI21l+6ym Xrp55HNV7CUDA6joK1pctpKKz33Ve1RoX4wINpt3cHXYygCDBfSH4VdTKGGrWywD Rxi73aZQH+0KCszm/Cm0TH0ldFD8EvAOPfQWba2tyGn3eDXg5pq8gXMhRLkv4t/8 JtDLhkqbwWQXTSvKHI6m9cqSLaxZPqpSH0sqv0C2L8py2mYQW3/5RcygS45hU61T VmLxaUC/bPjf3GWm7xkf8q9fqhZzplRtX8hAadqpXExzUhbJVyaUcymkwlPGZvOj nGIfZh6Jn+BQ5zLYAgfSPxpDMcprpSRJ4eThOzyH2s3sgxHmxpI4SSmesx9/7JC8 9KQnvUpt0YZdZhJE4RMLlx7/9d5sZ7WtsDZ5NvDJa4gOQl8axrT1alkWA0kcHU0U 5fm88bAZyTiT9H8Nad0a9rdSDbZF/4u/lEZ5kqt9YM0iISZsI6fkgpWkV/uEUtkg h8/Nph8ALfH688o3CjQgk/QiD/C61c2+iuw8BWEMWg31rXKsU0kXtiHeU6LSzVNF 3b6gStixo2e5dgJeXbstO/sCSOS6VtQiF4q4hbVnxR3JT9uaVJEcigL5hseIEZyD GodshFKhKUVWPa+Pa06zJcGkPyqCQrKcZwGBRd6Ymi+1ooF9E2GvtvpM46oIjP6I i0nZoR1KUEG/wnaFl39A =1lCL -----END PGP SIGNATURE----- --kunpHVz1op/+13PW--