Date: Sat, 30 Dec 2017 10:11:32 +0200 From: Andriy Gapon <avg@FreeBSD.org> To: Peter Grehan <grehan@freebsd.org> Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve/amd: interrupt delivered when it shouldn't be? Message-ID: <b7bc8760-9cb2-0a0e-bfd0-d8baf909fe82@FreeBSD.org> In-Reply-To: <450137ba-52dd-8b4c-63d2-3c3ce1909d69@freebsd.org> References: <42c22179-ae42-e4bb-e77d-a1d49fe634ed@FreeBSD.org> <450137ba-52dd-8b4c-63d2-3c3ce1909d69@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 30/12/2017 01:45, Peter Grehan wrote: > Hi Andriy, > >> The hardware is AMD. > > Ryzen ? No, a pretty old family 10h. >> But what I see suggests that at this point a Local APIC timer interrupt gets >> delivered to the thread. And that causes all the mess as the thread holding the >> spinlock gets preempted. >> >> Does this ring a bell to anyone? > > I have seen something similar to this after about ~20 mins when doing a current > -j 16 buildworld in a guest, with the symptom being a spinlock timeout, with one > vCPU spinning in > > smp_targeted_tlb_shootdown() at smp_targeted_tlb_shootdown+0x352/frame > 0xfffffe02c80098d0 > smp_masked_invlpg() at smp_masked_invlpg+0x4c/frame 0xfffffe02c8009900 > pmap_invalidate_page() at pmap_invalidate_page+0x191/frame 0xfffffe02c8009950 > pmap_ts_referenced() at pmap_ts_referenced+0x7b3/frame 0xfffffe02c8009a00 > vm_pageout() at vm_pageout+0xe04/frame 0xfffffe02c8009a70 > > ... and all the others eventually spinning on that held lock. This looks very similar to what I see as well. > However, NMIs are still able to get through (the post-panic ddb NMI IPI) so the > VM isn't completely locked up - either an interrupt is missed, or a write isn't > seen by the vCPU issuing the tlb shootdown. Yes, I can also enter ddb and remote kgdb. >> Is there any suspect code? > > Not sure yet, but the interrupt-injection path could do with a close inspection. > >> It seems that we set v_intr_masking bit, so the rFLAGS / eFLAGS should be >> completely virtualized. So, maybe a hardware issue? > > Hard to say. Running with all vCPUs pinned makes the problem go away, but that > could just mean the issue is isolated to when vCPUs migrate. I was going to try pinned CPUs and wired memory next, so your observation narrows the problem scope. Thank you! -- Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?b7bc8760-9cb2-0a0e-bfd0-d8baf909fe82>