From owner-freebsd-virtualization@freebsd.org Fri Nov 17 03:33:46 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 791C8DB902E for ; Fri, 17 Nov 2017 03:33:46 +0000 (UTC) (envelope-from akgupt3@gmail.com) Received: from mail-qt0-x22e.google.com (mail-qt0-x22e.google.com [IPv6:2607:f8b0:400d:c0d::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2A84F781EB; Fri, 17 Nov 2017 03:33:46 +0000 (UTC) (envelope-from akgupt3@gmail.com) Received: by mail-qt0-x22e.google.com with SMTP id n61so2912625qte.10; Thu, 16 Nov 2017 19:33:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=Dmc74Oo0JADAGm+CWfRfCL10Au0ENF7Xq9CgPZOoUsU=; b=AIzxspLvGum0Ynvq5N/KhYXBJuWz77DjRwp0Zp4DinDcueOMpA14Wj/Jt02N4bFRq+ 3dPUat7qy+GMdc9g8oGRaCRA0OQKr+VZ4Y0UOgW2kjlzKk3mmC73NRJ3mCqup8XF5CB9 S4XNYCsXLaA8fshgGp+mOVnaBtNeAEbTEzoGIafsZVLgVR2iindFPwc1H6wpk0GYs6Dy TQ8/VSIyiohUCm1lFrITFlBa2szi1Lnuyd3GxjUyAde6HJKn2ScbqJYxV5nQ043p+NMk 4b3PrHtvBz+WOXxIXZ5TrFhGi/c651l1p6lvg+Sw21+af/HTqzr7dGHCNAdbKiytihtM WIRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=Dmc74Oo0JADAGm+CWfRfCL10Au0ENF7Xq9CgPZOoUsU=; b=XygzUhNGxYCldkIBv5NilOI3d2fxnSc3TLPYTTjVPfVZ1nqqM0XSi3libyDIhRuXC4 o9YRAnt1K4jrtC5I8dtqkaC45YgUh9UFK5FZTT1s6TTKrSUvm9I2MiRNVWUPTit9Ayoe 71JcwZwgXsbB5V0s0b61EBLWaCFlrE/cpY1mSQrsRw/mhxN6wn1bJDsuhbKMy35QStf/ aVF8HvVVvjRQnEolG7K5us7eEz4HfIpsCB6+Ym8OA6eBBfD4kQdjmUteHPkxaAHtoOI9 rq/ddk1NNCMm+Xyt4BK/BvI3S+lmPwEQBOWcPeicNqwsv7j+TgyAMDhqYy3yMFOPoPyA GNhw== X-Gm-Message-State: AJaThX68F5RmJANI4xT8WX1kxnmgaLo5LY+iSZ2Wl7s3SQ5CPbPsHcGH 2t2QNrm3Ps0oJOU16nbvQkN2hR/mw01Nz7g1EbY1qg== X-Google-Smtp-Source: AGs4zMYiN0MTC7CnTR1ymz/xl+sLnB+pjZzn3F+bVEo2jDw27oUf6KIGNkg++nAKeMqrCV75bVPX886yFHIUUmn4xrQ= X-Received: by 10.55.126.198 with SMTP id z189mr6422962qkc.100.1510889625011; Thu, 16 Nov 2017 19:33:45 -0800 (PST) MIME-Version: 1.0 Received: by 10.12.172.247 with HTTP; Thu, 16 Nov 2017 19:33:44 -0800 (PST) In-Reply-To: References: From: Anish Date: Thu, 16 Nov 2017 19:33:44 -0800 Message-ID: Subject: Re: problem with pass-through on amd To: Andriy Gapon Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Nov 2017 03:33:46 -0000 Hi Andriy, >After I fixed the MSI setup problem that I described in the previous email I was finally able to see the real problem. Awesome! fault interrupt handler has been very helpful in debugging AMD-Vi. >I've passed both of the PCI device to bhyve and everything started working. Thanks! Bhyve uses single domain for all the passthrough devices of a guest, that's why even with aliasing it worked. >But it appears to me that FreeBSD currently ignores those aliases. Also, it looks like we are not making any use of IVHD entries beyond printing them. We make use of of very few IVRS entries, see ivhd_dev_parse() but not much for aliasing entries. Found out AMD IOMMU spec has some details on it: "An Alias device type entry is used for each peripheral that does not use its own DeviceID information in bus transactions. For example, peripherals downstream of a bridge device that use the DeviceID of the bridge must have a corresponding Alias Select or Alias Start of Range entry to inform system software which IOMMU Device Table entry will be used for translation information." Thanks a lot for trying out AMD-Vi, let me know if you have any other feedback. Thanks and regards, Anish On Thu, Nov 16, 2017 at 12:51 PM, Andriy Gapon wrote: > On 14/11/2017 06:22, Anish wrote: > > [root@ryzen /home/anish/FreeBSD/head]# vmstat -ia |grep ivh > > irq256: ivhd0:fault 0 0 > > irq257: ivhd1:fault 0 0 > > After I fixed the MSI setup problem that I described in the previous email > I was > finally able to see the real problem. > Now I have: > > irq263: ivhd0:fault 9 0 > > dev.ivhd.0.event_tail: 240 > dev.ivhd.0.event_head: 240 > dev.ivhd.0.event_intr_count: 9 > > > I've got a bunch of messages like these in the log: > [IO_PAGE_FAULT EVT: devId:0xa01 DomId:0x2 Addr:0x70f70400x30] > [IO_PAGE_FAULT EVT: devId:0xa01 DomId:0x2 Addr:0x2fdf0400x30] > (BTW, there seems to be a missing space before 0x30) > > Now it's obvious what the problem is. > The controller has a RID of 0xa00 (its PCI locator is 10:0:0) but the > requests > are coming from 0xa01. The card actually has another PCI device at that > location (10:0:1), it's a vestigial IDE controller (in a sense that it is > not > connected to any ports, so it cannot really provide any functionality). > > I've passed both of the PCI device to bhyve and everything started working. > Thanks! > > I've googled a little bit and it seems that this is not an uncommon > problem. > Linux has a bunch of quirks for this and similar kinds of problems (they > call it > PCI or DMA aliasing): > http://elixir.free-electrons.com/linux/v4.8.16/source/ > drivers/pci/quirks.c#L3701 > The comments say that sometimes requests come from devices that do not > exist at > all (or hidden beyond non-transparent bridges). > > An interesting new world for me :-) > > By the way, my understanding is that the alias IVHD entries in IVRS are > designed > to report such aliasing issues when they are known to the platform. > But it appears to me that FreeBSD currently ignores those aliases. > Also, it looks like we are not making any use of IVHD entries beyond > printing > them. Based on the specification I think that we should use IVHD flags to > set > up appropriate bits in the corresponding device table entries. But we are > not > doing that. That's probably okay since we do not support the pass-though > for > devices with complex properties anyway. > > -- > Andriy Gapon >