Date: Thu, 16 Nov 2017 22:51:55 +0200 From: Andriy Gapon <avg@FreeBSD.org> To: Anish <akgupt3@gmail.com> Cc: "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: Re: problem with pass-through on amd Message-ID: <c73ab383-4dd4-5aaf-1682-830fc625a80b@FreeBSD.org> In-Reply-To: <CALnRwMTq7pD0SBj3D0MRp7mxQ9nK5d1vk_XLA8uCnaYchCf18A@mail.gmail.com> References: <cee09b94-3a1d-ac36-c99a-99196faa75a9@FreeBSD.org> <CALnRwMShTha5aKfY6EbB=n2JiiK1z_Y--39WoWGgbUUufafnmw@mail.gmail.com> <e26820b8-048b-9684-f810-2cba364d0474@FreeBSD.org> <CALnRwMTq7pD0SBj3D0MRp7mxQ9nK5d1vk_XLA8uCnaYchCf18A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 14/11/2017 06:22, Anish wrote: > [root@ryzen /home/anish/FreeBSD/head]# vmstat -ia |grep ivh > irq256: ivhd0:fault 0 0 > irq257: ivhd1:fault 0 0 After I fixed the MSI setup problem that I described in the previous email I was finally able to see the real problem. Now I have: irq263: ivhd0:fault 9 0 dev.ivhd.0.event_tail: 240 dev.ivhd.0.event_head: 240 dev.ivhd.0.event_intr_count: 9 I've got a bunch of messages like these in the log: [IO_PAGE_FAULT EVT: devId:0xa01 DomId:0x2 Addr:0x70f70400x30<PR,RW>] [IO_PAGE_FAULT EVT: devId:0xa01 DomId:0x2 Addr:0x2fdf0400x30<PR,RW>] (BTW, there seems to be a missing space before 0x30) Now it's obvious what the problem is. The controller has a RID of 0xa00 (its PCI locator is 10:0:0) but the requests are coming from 0xa01. The card actually has another PCI device at that location (10:0:1), it's a vestigial IDE controller (in a sense that it is not connected to any ports, so it cannot really provide any functionality). I've passed both of the PCI device to bhyve and everything started working. Thanks! I've googled a little bit and it seems that this is not an uncommon problem. Linux has a bunch of quirks for this and similar kinds of problems (they call it PCI or DMA aliasing): http://elixir.free-electrons.com/linux/v4.8.16/source/drivers/pci/quirks.c#L3701 The comments say that sometimes requests come from devices that do not exist at all (or hidden beyond non-transparent bridges). An interesting new world for me :-) By the way, my understanding is that the alias IVHD entries in IVRS are designed to report such aliasing issues when they are known to the platform. But it appears to me that FreeBSD currently ignores those aliases. Also, it looks like we are not making any use of IVHD entries beyond printing them. Based on the specification I think that we should use IVHD flags to set up appropriate bits in the corresponding device table entries. But we are not doing that. That's probably okay since we do not support the pass-though for devices with complex properties anyway. -- Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?c73ab383-4dd4-5aaf-1682-830fc625a80b>