Date: Mon, 1 May 2023 09:14:11 -0700 From: John Baldwin <jhb@FreeBSD.org> To: "Jason A. Harmening" <jah@freebsd.org>, Konstantin Belousov <kostikbel@gmail.com> Cc: Dimitry Andric <dim@freebsd.org>, src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-branches@freebsd.org Subject: Re: git: 060699e91369 - stable/13 - Merge llvm-project release/15.x llvmorg-15.0.7-0-g8dfdcc7b7bf6 Message-ID: <df22667e-831f-deaa-3cb0-7b9797caa833@FreeBSD.org> In-Reply-To: <ZE8yqfOkrrMddua1@corona> References: <202304092135.339LZMeJ081640@gitrepo.freebsd.org> <ZE1jEcA7iL3QrbOP@corona> <76DD2CB9-986B-4349-8F46-3B7BF63EB315@FreeBSD.org> <ZE1vtvghEV_RsD2b@corona> <ZE33_AWw0DO8-EjM@kib.kiev.ua> <ZE7-AcTXtCkSzsHb@corona> <ZE8JJQizER2WrTr2@corona> <ZE8yqfOkrrMddua1@corona>
next in thread | previous in thread | raw e-mail | index | archive | help
On 4/30/23 8:31 PM, Jason A. Harmening wrote: > On Sun, Apr 30, 2023 at 07:34:45PM -0500, Jason A. Harmening wrote: >> On Sun, Apr 30, 2023 at 06:47:13PM -0500, Jason A. Harmening wrote: >>> On Sun, Apr 30, 2023 at 08:09:16AM +0300, Konstantin Belousov wrote: >>>> On Sat, Apr 29, 2023 at 02:27:50PM -0500, Jason A. Harmening wrote: >>>>> On Sat, Apr 29, 2023 at 08:49:28PM +0200, Dimitry Andric wrote: >>>>>> On 29 Apr 2023, at 20:33, Jason A. Harmening <jah@FreeBSD.org> wrote: >>>>>>> >>>>>>> On Sun, Apr 09, 2023 at 09:35:22PM +0000, Dimitry Andric wrote: >>>>>>>> The branch stable/13 has been updated by dim: >>>>>>>> >>>>>>>> URL: https://cgit.FreeBSD.org/src/commit/?id=060699e9136975d51d3f726b9785bdbac9a62ba6 >>>>>>>> >>>>>>>> commit 060699e9136975d51d3f726b9785bdbac9a62ba6 >>>>>>>> Author: Dimitry Andric <dim@FreeBSD.org> >>>>>>>> AuthorDate: 2023-01-14 16:33:24 +0000 >>>>>>>> Commit: Dimitry Andric <dim@FreeBSD.org> >>>>>>>> CommitDate: 2023-04-09 14:54:52 +0000 >>>>>>>> >>>>>>>> Merge llvm-project release/15.x llvmorg-15.0.7-0-g8dfdcc7b7bf6 >>>>>>>> >>>>>>>> This updates llvm, clang, compiler-rt, libc++, libunwind, lld, lldb and >>>>>>>> openmp to llvmorg-15.0.7-0-g8dfdcc7b7bf6. >>>>>>>> >>>>>>>> PR: 265425 >>>>>>>> MFC after: 2 weeks >>>>>>> >>>>>>> This MFC of llvm15 appears to have completely broken the Intel IOMMU >>>>>>> driver on my stable/13 machine. After this series of commits, any >>>>>>> downstream DMA seems to produce an IOMMU translation fault, which >>>>>>> renders the machine completely unusable: no nvme boot disk, no usb >>>>>>> keyboard, etc. >>>>>>> >>>>>>> The faults I see look something like this: >>>>>>> >>>>>>> DMAR4: ahci0: pci0:17:5 sid 8d fault acc 0 adt 0x0 reason 0x3 addr 26000 >>>>>>> >>>>>>> It's a bit surprising to see a toolchain upgrade produce breakage like >>>>>>> this, but that's what git bisect clearly tells me. I wonder if some of >>>>>>> the IOMMU control structures might be defined as C bitfields and the new >>>>>>> compiler is emitting them differently? Also, was any breakage like this >>>>>>> observed when -current was upgraded to llvm15 several months ago? >>>>>> >>>>>> I haven't heard anything about such breakage, no. >>>>>> >>>>>> >>>>>>> More generally, this is the second time in as many months I've had to >>>>>>> deal with IOMMU breakage on -stable. I can't imagine I'm the only >>>>>>> person who sees value in running with DMA remapping enabled; do we need >>>>>>> a dedicated DMAR-enabled machine in the cluster to smoke-test changes >>>>>>> like this? More generally, should we avoid MFCing high-risk changes >>>>>>> like this? >>>>>> >>>>>> Since there were very few bug reports, it was not deemed high risk. >>>>>> >>>>>> In any case, it would be good to get the bottom of what is causing the >>>>>> problem, so is there any way you can isolate which code seems to be >>>>>> going "bad"? >>>>>> >>>>>> For example, if this problem affects code in sys/dev/iommu, is there >>>>>> some way you can compile that part with -O1, or with an older version >>>>>> of clang (from ports), to see if the problem goes away? >>>>> >>>>> I did try removing all custom make.conf settings (previously I just had >>>>> CPUTYPE?=icelake-server), but that didn't change the behavior. >>>>> >>>>> Before I try further build tweaks, I'd like to ask if the IOMMU fault >>>>> report can provide guidance here? AFAICT all the faults I'm getting >>>>> show "reason 0x3". If I'm reading the VT-d spec correctly, FR=0x3 >>>>> indicates an invalid context entry, in other words there's something the >>>>> hardware doesn't like in the way the address width or pagetable base is >>>>> configured for the PCIe requestor. >>>> >>>> I would start looking at the other direction: might be, there are still some >>>> left shifts for int32 values with the shift count > 30, or uint32 with the >>>> count > 31. >>>> >>>> Also might be useful to dump each context entry on creation, it is kept >>>> constant after. >>> >>> I did look over the constants in intel_reg.h, and didn't see anything >>> that looked as though it would be susceptible to sign-extension or >>> truncation bugs. In the failing case it's much easier for me to catch >>> the fault messages than any initialization message, so I instrumented >>> the fault handler to get the context entry from the dmar_ctx object >>> using the same logic as dmar_map_ctx_entry(), and then dump out the ctx1 >>> and ctx2 fields. What I see are messages like: >>> >>> ... ctx1 0x10013b001 ctx2 0x103 >>> >>> At first glance these "look right": the P bit is set in ctx1, and the >>> rest of the field looks like a valid physical address. ctx2 also >>> doesn't have any of the reserved bits set, but in all cases it does have >>> AW=3, which would indicate 57-bit AGAW. But when I boot the last >>> working kernel, from the revision prior to the llvm15 MFC, I see this in >>> dmesg: >>> >>> ahci0: dmar4 pci0:0:17:5 rid 8d domain 1 mgaw 48 agaw 48 re-mapped >>> >>> ...all reported devices show 48-bit MGAW/AGAW, so I would expect ctx2 to >>> have AW=2. I suspect this may be the source of the fault, but I'm not >>> sure how it's getting configured that way, whether it's an issue with >>> reading the capability register or something else. >>> >> >> I can confirm that hacking domain_set_agaw() to always use the settings >> from sagaw_bits[2] eliminates the faults and at least allows the machine >> to boot to single-user mode. > > I see what's happening now. When I added the hack to always set > sagaw_bits[2], I noted that the passed-in MGAW was still 57, while > unit->hw_cap had the correct value of 0x4 (=> 4-level paging, 48-bit AW) > in bits 12:8. The problem is that sagaw_bits has agaw=64 in its last > entry. This results in dmar_maxaddr2mgaw() attempting a comparison > against 1ULL << 64 in the final iteration of its first loop. I suspect > the new compiler probably determines that last iteration is meaningless > and simply omits it from the (probably unrolled) loop. Since the "loop" > terminates with i < nitems(sagaw_bits), the subsequent "allow_less ..." > case doesn't execute and we end up erroneously selecting a 57-bit > address width. Just commenting out that last entry in sagaw_bits fixes > the problem. > > So, two questions: > 1) Does any VT-d hardware actually support 6-level paging? The ca. 2021 > VT-d spec I'm looking at indicates 5-level is the greatest depth > supported, with everything above that being reserved. > > 2) I'd expect clang to try very hard to error out in a situation like > this, but I see that sys/conf/kern.mk sets -Wno-shift-count-overflow > among other things, and more of them were added for clang 15. This > seems like a really bad idea, regardless of how much of a PITA it may be > to fix these warnings. FWIW, I've been working on trying to re-enable some of the warnings that were disabled for clang 15 in main. I'll move that one higher up on my todo list. -- John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?df22667e-831f-deaa-3cb0-7b9797caa833>