From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 16 21:03:58 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2235F33A for ; Sun, 16 Mar 2014 21:03:58 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9B0FD31D for ; Sun, 16 Mar 2014 21:03:57 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.8/8.14.8) with ESMTP id s2GL3q3k056304; Sun, 16 Mar 2014 23:03:52 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua s2GL3q3k056304 Received: (from kostik@localhost) by tom.home (8.14.8/8.14.8/Submit) id s2GL3qcj056303; Sun, 16 Mar 2014 23:03:52 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 16 Mar 2014 23:03:52 +0200 From: Konstantin Belousov To: Ryan Stone Subject: Re: [PATCH] Support PCIe Alternative RID Interpretation (ARI) Message-ID: <20140316210352.GB21331@kib.kiev.ua> References: <20140316141216.GA21331@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="oLBj+sq0vYjzfsbl" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Mar 2014 21:03:58 -0000 --oLBj+sq0vYjzfsbl Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Mar 16, 2014 at 11:05:16AM -0400, Ryan Stone wrote: > On Sun, Mar 16, 2014 at 10:12 AM, Konstantin Belousov > wrote: > > This is expected, but it would break VT-d busdma, I think. > > > > This is not said in the VT-d spec about ARI, but I believe that > > DMAR would split the function number by 7-3/2-0 bits, same as for > > the non-ARI devices. Then the transactions will be translated by > > the wrong context. >=20 > Ah, good catch. I took a quick look at the spec and the code, and I > believe that I see the problem. I think that the proper solution is > to add a new method, pcib_get_rid(), that returns (bus << 8) | (slot > << 5) | func for non-ARI devices and (bus << 8) | func for ARI > devices. Then we could add a pci_get_rid() that just calls the pcib > method, and the DMAR code could be changed to work in terms of the RID > instead of bus/slot/function. My reading of the spec is that VT-d is > really implemented in terms of the RID anyway, but the spec authors > took pains to give examples in terms of bus/slot/function because > that's how the software developers understand things. Well, I agree, but the current DMAR driver is also written in term of bsf. So using the pci_get_rid() is possible, but adding a methods like pci_get_fake_slot/func, which would return ARI rid converted into fake slot and function at the 3/2 boundary is less work. Anyway, please add whatever you find suitable as the accessor method, and I hope to be able to produce a patch that would convert Intel IOMMU to use it instead of direct manipulations of bsf. >=20 > Do you know if bhyve's pci passthrough code uses the same code to add > DMAR entries as the busdma code? If not I'll have to track down how > bhyve does it because it would likely have the same problem. I am not aware of bhyve code. They definitely use their own code for the passthrough right now. >=20 > > From other minor notes, having additional line for "ARI enabled" > > message under bootverbose would make already excessive PCI config > > dump even more problematic. >=20 > I can remove it. At the time I wanted some kind of indication that > ARI was being used, but pciconf can tell you that now so it's not > really necessary. Or stuff it into the already printed line. --oLBj+sq0vYjzfsbl Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJTJhG3AAoJEJDCuSvBvK1BMNkQAIreZSfturnisdhh1GOnGYyv Uu8JQBUOzCivPm98akiZa4p2b3TfyOtC1piXeEhKkP0eXj1yxcWusnVjX4kqj2Y9 4alTbA3gATLRUhywK74hHa6glsmKIYy5/sMkO1MtvkOPux9simo/Uv6WZjjyKmNV cKeUlA392xCFD+HJnvHvmxYh7VoCfZvVanMK5H3ZRP0GZHpZV8DsrLl2Ww8WJm+m hyQIN3+Pql4Fgsx8iz6tUfUizgij1u03ux181ShFCKXpAQqMGDp7wexyRsfVloNM 8EFRoJeGjM7mPNr2Fi9YsRXQpMaIzt2MTe4PtUtYqXSLKsK8UZSdxN6A7PW7dpK0 V8DIXJZItMtjh4iocGn4pHNyjXSjS68pzb3lYqfvWhW+XlOTu3KgaCZys2fvb0Iv dNZ2j0WIoKbP6HvXsaPIWYZwcTceM42nleZFmEhkUUbpYAv+YwmRzW9DujW767iK dXXLG/SLj9MBkWJDkDTnyNA1f42BGX1PfJgddqxIrsiigbe8KXCO5vHisrLQgTzg xB6Smv4ilvBkCG0Ed8ouuMsFzranTswlvRRKQa6RbE+iechueg97+7C4OoEBflXW dP0lBnIz+lkSyExOm58fk0iyGHqLFyFjGbeCzmFyC8cn9LfUVve9O0bZmDSjNYKF Xgdp6RohFvNV/3QYjWlf =8XT7 -----END PGP SIGNATURE----- --oLBj+sq0vYjzfsbl--