From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 16 15:05:18 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1A6571B6 for ; Sun, 16 Mar 2014 15:05:18 +0000 (UTC) Received: from mail-oa0-x232.google.com (mail-oa0-x232.google.com [IPv6:2607:f8b0:4003:c02::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D8ACF1AC for ; Sun, 16 Mar 2014 15:05:17 +0000 (UTC) Received: by mail-oa0-f50.google.com with SMTP id i7so4644677oag.9 for ; Sun, 16 Mar 2014 08:05:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=G9u4lacyQkYvsEBRZk0UuxUd0QbKnFzouKv2jjAX8wc=; b=iO3BLGbSw1gu8Qy2Tqoj9Vu70SAYl3WOLhPxUrZ9IVBt6lj0xb5ygwluojAP69gF78 aZ0mVc3BrjEuG/+OK3M3QgiEVFsnxIVskw0aIgbrqQLgyX0QT1DWRxxkYpimY/bqAJ4g WsLUXzzIjjRzjCTYwyrlvreonFDhav4dUpOGBp/aQ7nGgHDnLXWwknAv/GYLnQekGOgG WwLg+iqW3fnO8J5PK1Ksxio6NwOV2rQp0gvCbR8ZYC5ODIad4uKHmOI7phFNF+LgDWFO 7DqItNBUPXKdmH4VpW5cYx2cgHzPq3ujAoHUevG5Q0ZTmDtukd0z7j4rISWH/uAJ7C5o 8ONw== MIME-Version: 1.0 X-Received: by 10.182.117.195 with SMTP id kg3mr16226756obb.17.1394982316633; Sun, 16 Mar 2014 08:05:16 -0700 (PDT) Received: by 10.76.115.129 with HTTP; Sun, 16 Mar 2014 08:05:16 -0700 (PDT) In-Reply-To: <20140316141216.GA21331@kib.kiev.ua> References: <20140316141216.GA21331@kib.kiev.ua> Date: Sun, 16 Mar 2014 11:05:16 -0400 Message-ID: Subject: Re: [PATCH] Support PCIe Alternative RID Interpretation (ARI) From: Ryan Stone To: Konstantin Belousov Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Mar 2014 15:05:18 -0000 On Sun, Mar 16, 2014 at 10:12 AM, Konstantin Belousov wrote: > This is expected, but it would break VT-d busdma, I think. > > This is not said in the VT-d spec about ARI, but I believe that > DMAR would split the function number by 7-3/2-0 bits, same as for > the non-ARI devices. Then the transactions will be translated by > the wrong context. Ah, good catch. I took a quick look at the spec and the code, and I believe that I see the problem. I think that the proper solution is to add a new method, pcib_get_rid(), that returns (bus << 8) | (slot << 5) | func for non-ARI devices and (bus << 8) | func for ARI devices. Then we could add a pci_get_rid() that just calls the pcib method, and the DMAR code could be changed to work in terms of the RID instead of bus/slot/function. My reading of the spec is that VT-d is really implemented in terms of the RID anyway, but the spec authors took pains to give examples in terms of bus/slot/function because that's how the software developers understand things. Do you know if bhyve's pci passthrough code uses the same code to add DMAR entries as the busdma code? If not I'll have to track down how bhyve does it because it would likely have the same problem. > From other minor notes, having additional line for "ARI enabled" > message under bootverbose would make already excessive PCI config > dump even more problematic. I can remove it. At the time I wanted some kind of indication that ARI was being used, but pciconf can tell you that now so it's not really necessary.