From owner-freebsd-virtualization@freebsd.org Fri Dec 29 22:28:03 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3407DEB3A62 for ; Fri, 29 Dec 2017 22:28:03 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f47.google.com (mail-lf0-f47.google.com [209.85.215.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BAD567E8BC for ; Fri, 29 Dec 2017 22:28:02 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f47.google.com with SMTP id c19so29359907lfg.3 for ; Fri, 29 Dec 2017 14:28:02 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-language:content-transfer-encoding; bh=Cq0ziDJdYuLxyDrmhy0ZUxsSlT772RQ9CvtekIrJdlU=; b=nQh+gMuf7yXUerDYoaxZQ08iQZPgGl6gH3Ch298eC+WKie/dPE0S9pwQ8/zuqCoz9Q u+w6XHH6w2mkojaHQCCSBTrbRiRNXlzJ7MRXBeRHZVuw5uK3PJ4fQq1sZbYYrNcEqycZ AZNs7SQl67t2l6Ffe4itKXDNPk7IpuqubsZh6lxLV0q+tvnhkyTuncxTZaq1HFHK+gI5 klf6AJiTBtMEgyx966oTWEcuIMhdALKo/KwYEzA5YNpF/i/gMvMGoTLOnPHUBK5imA+O BZHhYIRPUBSNvykbs9LEK9YKUKod2q82oFcaeEH+FJuVEzO1N6s/1OlCQJnUoTG2rRnN ucrQ== X-Gm-Message-State: AKGB3mLtO7ALvKpicJj8g3hfEO1Z/wnqNYhCPqZI7z+BUNL+4uVN8G4A NP5o30Vw7qqgD3wr5nPihVs= X-Google-Smtp-Source: ACJfBosnTy0MmUUr0sx8ND4IHMGSro6zrFMiCI5vXwM8VTQJSXWEQQu6L8RBpHGlyxbCz4+4dkzU/Q== X-Received: by 10.46.16.138 with SMTP id 10mr24320983ljq.115.1514586144280; Fri, 29 Dec 2017 14:22:24 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id b75sm8053110lfe.47.2017.12.29.14.22.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 29 Dec 2017 14:22:23 -0800 (PST) To: freebsd-virtualization@freebsd.org From: Andriy Gapon Subject: bhyve/amd: interrupt delivered when it shouldn't be? Message-ID: <42c22179-ae42-e4bb-e77d-a1d49fe634ed@FreeBSD.org> Date: Sat, 30 Dec 2017 00:22:21 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Dec 2017 22:28:03 -0000 First, about the setup. It's a FreeBSD/amd64 head guest on a FreeBSD/amd64 head host. The hardware is AMD. The hypervisor is bhyve. Under a certain specific load, that involves a lot of page faults and IPI-s, I see the guest system getting stuck. This is pretty consistent. Typically I find a thread spinning on smp_ipi_mtx. And an owner of the mutex appears to be in mi_switch() -> sched_switch(). The debugging data that I have is somewhat flaky, but it seems that the owner is typically in this code path: smp_targeted_tlb_shootdown -> ipi_send_cpu -> native_lapic_ipi_raw smp_targeted_tlb_shootdown holds smp_ipi_mtx. native_lapic_ipi_raw, in this setup, performs the following manipulations: saveintr = intr_disable(); ... intr_restore(saveintr); The interrupts are already disabled when this function is entered, because smp_ipi_mtx is a spinlock and our spinlock implementation disables interrupts. So, intr_restore() in this case should be a NOP (BTW, it's implemented via popf). But what I see suggests that at this point a Local APIC timer interrupt gets delivered to the thread. And that causes all the mess as the thread holding the spinlock gets preempted. Does this ring a bell to anyone? Is there any suspect code? It seems that we set v_intr_masking bit, so the rFLAGS / eFLAGS should be completely virtualized. So, maybe a hardware issue? Thank you! -- Andriy Gapon