From owner-freebsd-virtualization@freebsd.org Thu Nov 15 18:32:10 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 906F61105C77 for ; Thu, 15 Nov 2018 18:32:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 049C777B30 for ; Thu, 15 Nov 2018 18:32:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id BD7AF1105C76; Thu, 15 Nov 2018 18:32:09 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9B99F1105C74 for ; Thu, 15 Nov 2018 18:32:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 225C377B2A for ; Thu, 15 Nov 2018 18:32:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 595EC1E786 for ; Thu, 15 Nov 2018 18:32:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wAFIW8Cv037884 for ; Thu, 15 Nov 2018 18:32:08 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wAFIW8xp037883 for virtualization@FreeBSD.org; Thu, 15 Nov 2018 18:32:08 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 203874] [patch] MSI/MSI-X interrupts don't work in VMware virtual machines Date: Thu, 15 Nov 2018 18:32:08 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: jhb@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-Rspamd-Queue-Id: 049C777B30 X-Spamd-Result: default: False [-105.92 / 200.00]; FORGED_RECIPIENTS_FORWARDING(0.00)[]; ALLOW_DOMAIN_WHITELIST(-100.00)[freebsd.org]; FORWARDED(0.00)[virtualization@mailman.ysv.freebsd.org]; SPF_FAIL_FORWARDING(0.00)[]; TO_DN_NONE(0.00)[]; HAS_XAW(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; XAW_SERVICE_ACCT(1.00)[]; RCVD_IN_DNSWL_MED(-0.20)[5.0.0.0.0.5.0.0.0.0.0.0.0.0.0.0.a.6.0.2.4.5.2.2.0.0.9.1.1.0.0.2.list.dnswl.org : 127.0.9.2]; MX_GOOD(-0.01)[cached: mx66.freebsd.org]; NEURAL_HAM_SHORT(-1.00)[-1.000,0]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; IP_SCORE(-3.71)[ip: (-9.87), ipnet: 2001:1900:2254::/48(-4.81), asn: 10310(-3.75), country: US(-0.10)]; ASN(0.00)[asn:10310, ipnet:2001:1900:2254::/48, country:US]; FORGED_RECIPIENTS(0.00)[virtualization@FreeBSD.org,freebsd-virtualization@freebsd.org]; TO_DOM_EQ_FROM_DOM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; FROM_NO_DN(0.00)[]; RCVD_COUNT_SEVEN(0.00)[7] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Nov 2018 18:32:10 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203874 John Baldwin changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jhb@FreeBSD.org --- Comment #12 from John Baldwin --- Hmm, I don't recall which issue we ran into with VMWare. We have had some issues with Xen hypervisors that might be related. They had to do with how= the hypervisor cached MSI-X table entries and that it didn't notice updates to = the table that FreeBSD's kernel performed because it assumed the table was not modified once MSI-X was enabled in the capability register. Commits related to those issues are r302181 + r302635 and r310806. >From the thread about igb(4) that was quoted, it sounds like it might be a similar issue where VMWare might be assuming that MSI-X tables are only upd= ated in a particular order. FreeBSD follows a scheme where it marks all the tab= le entries as disabled and then enables MSI-X in the capability register and t= hen writes individual table entries as the driver calls bus_setup_intr(). This means that the table can change while MSI-X is enabled. A table entry might also be written multiple times if a device driver moves an interrupt to a different CPU than the one chosen by the system. In real hardware this wor= ks fine as the hardware checks the table to compose the MSI message before sen= ding each message. Hypervisors might choose to cache the table since reading the actual table might be too expensive. However, in that case hypervisors hav= e to trap all writes to the table and update their cached copy for each write. = The Xen issues had to do with Xen assuming it could ignore writes made while MS= I-X was active since Linux writes the whole table and then enables MSI-X. I wo= uld start by seeing if you have a shadow copy of the MSI-X table and if you make assumptions about which writes to that table have to be trapped to update t= he shadow table. --=20 You are receiving this mail because: You are the assignee for the bug.=