From owner-freebsd-virtualization@freebsd.org Sat Dec 30 08:11:38 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 722CAEB0A30 for ; Sat, 30 Dec 2017 08:11:38 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f47.google.com (mail-lf0-f47.google.com [209.85.215.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1919573D10; Sat, 30 Dec 2017 08:11:37 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f47.google.com with SMTP id h137so5731472lfe.8; Sat, 30 Dec 2017 00:11:37 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=xPHP5YtbnOs7rK8vHdYcCxg6EV9SKd30vMoOfRdl3FI=; b=b2aAaVLAwzRmjpV4Zvxaa9bPvPuf0bCX7PihA8tu8UnW4ALFndocVar7c2xyoKjIo8 tchZ8g2JcpuPUUSy94auKtkJQ/5iKjAGuwnMuEceqKZ6ckjrbo6yWPfebUEDFQ3GTBQl 3j+6dww66AjSRBCfAwwGHCThWT3nz8H29bl882Rq7GQn0zXgN85jrNgOFi992gxgKfzf Cf1RIMNzY+j02ktt6ZSxPaVC80o7yB1KZIaCcj0S1A2yOVyy6D48LGIR7+IreOlPJBbI 1fXuXEmvBiKYTjxZoXhiqM7TtrTBrK50v8L2FS/OLXXHeUrmPXlIbthWbbs9OidrlPDD j/Rg== X-Gm-Message-State: AKGB3mKTFE3tvnJc3Lccjb/bme0dnwTyM/CRzTEgP1Qrr7curkc9e1Hz UbdsP5TPLACNKbR70obZNlICD+/S X-Google-Smtp-Source: ACJfBouL+svxyhV/sxG37fak1jdcnrSjF9wg+6jxvuJwzBX8Xbut21rbmJNXGvomk4/B8Z5Fk7O5NQ== X-Received: by 10.46.88.4 with SMTP id m4mr21328935ljb.15.1514621495108; Sat, 30 Dec 2017 00:11:35 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id 1sm7532098ljt.35.2017.12.30.00.11.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 30 Dec 2017 00:11:33 -0800 (PST) Subject: Re: bhyve/amd: interrupt delivered when it shouldn't be? To: Peter Grehan Cc: freebsd-virtualization@freebsd.org References: <42c22179-ae42-e4bb-e77d-a1d49fe634ed@FreeBSD.org> <450137ba-52dd-8b4c-63d2-3c3ce1909d69@freebsd.org> From: Andriy Gapon Message-ID: Date: Sat, 30 Dec 2017 10:11:32 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: <450137ba-52dd-8b4c-63d2-3c3ce1909d69@freebsd.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Dec 2017 08:11:38 -0000 On 30/12/2017 01:45, Peter Grehan wrote: > Hi Andriy, > >> The hardware is AMD. > >  Ryzen ? No, a pretty old family 10h. >> But what I see suggests that at this point a Local APIC timer interrupt gets >> delivered to the thread.  And that causes all the mess as the thread holding the >> spinlock gets preempted. >> >> Does this ring a bell to anyone? > >  I have seen something similar to this after about ~20 mins when doing a current > -j 16 buildworld in a guest, with the symptom being a spinlock timeout, with one > vCPU spinning in > > smp_targeted_tlb_shootdown() at smp_targeted_tlb_shootdown+0x352/frame > 0xfffffe02c80098d0 > smp_masked_invlpg() at smp_masked_invlpg+0x4c/frame 0xfffffe02c8009900 > pmap_invalidate_page() at pmap_invalidate_page+0x191/frame 0xfffffe02c8009950 > pmap_ts_referenced() at pmap_ts_referenced+0x7b3/frame 0xfffffe02c8009a00 > vm_pageout() at vm_pageout+0xe04/frame 0xfffffe02c8009a70 > > ... and all the others eventually spinning on that held lock. This looks very similar to what I see as well. > However, NMIs are still able to get through (the post-panic ddb NMI IPI) so the > VM isn't completely locked up - either an interrupt is missed, or a write isn't > seen by the vCPU issuing the tlb shootdown. Yes, I can also enter ddb and remote kgdb. >> Is there any suspect code? > >  Not sure yet, but the interrupt-injection path could do with a close inspection. > >> It seems that we set v_intr_masking bit, so the rFLAGS / eFLAGS should be >> completely virtualized.  So, maybe a hardware issue? > >  Hard to say. Running with all vCPUs pinned makes the problem go away, but that > could just mean the issue is isolated to when vCPUs migrate. I was going to try pinned CPUs and wired memory next, so your observation narrows the problem scope. Thank you! -- Andriy Gapon