From owner-freebsd-virtualization@freebsd.org Tue Jun 13 16:30:38 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB358BF8FF2 for ; Tue, 13 Jun 2017 16:30:38 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7A63F823C5 for ; Tue, 13 Jun 2017 16:30:38 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (mh0.gentlemail.de [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id v5DGUaHZ043080; Tue, 13 Jun 2017 18:30:36 +0200 (CEST) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id CC04BB3A; Tue, 13 Jun 2017 18:30:35 +0200 (CEST) Message-ID: <5940131E.8020604@omnilan.de> Date: Tue, 13 Jun 2017 18:30:22 +0200 From: Harry Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Anish CC: "freebsd-virtualization@freebsd.org" Subject: Re: PCIe passthrough really that expensive? References: <59383F5C.8020801@omnilan.de> <593A5AB1.7090301@omnilan.de> <593D1299.3030402@omnilan.de> In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Tue, 13 Jun 2017 18:30:36 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jun 2017 16:30:38 -0000 Bezüglich Anish's Nachricht vom 13.06.2017 06:13 (localtime): > Hi Harry, >>Any hints highly appreciated! … > > Now use cpuset to route IRQ#265 to say core 0 > > $cpuset -l 0 -x 265 > > Again use cpuset to force VM[PID 1222] to run on all core except #0 > > root@svmhost:~ # ps > > PID TT STAT TIME COMMAND > > .... > > 1222 1 I+ 5:59.45 bhyve: vm1 (bhyve) > > > VM can run on all cores except #0. > > $ cpuset -l 1-3 -p 1222 > > > You can monitor guest due to interrupts using > > $root@svmhost:~ # bhyvectl --get-stats --vm= --cpu= | > grep external > > vm exits due to external interrupt 27273 > > root@svmhost:~ # Thank you very much for that detailed explanation. I didn't thought that cpuset(1) could also pin IRQ (handlers?) to specific cpus. In my case, I couldn't get a noticable difference. Since I have hyperthreading enabled on a single-socket quad core, I pinned cpu2+3 to the bhyve pid (2vCPUs) and for testing copied two times the same 8GB file ofer NFS, while having ppt's IRQ handler pinned to CPU1, CPU4, CPU3 and CPU2. So two times different hostCPUs than guest uses and two times the same. I couldn't see any load nor performance difference and similar 'vm exits due to external interrupt' count groth. To name numbers: 1st CPU had about 40k "vm exits due to external interrupt" per 8GP transfer, the other vCPU ~160k "vm exits due to external interrupt". Like mentioned, different host-CPU-pinning didn't influence that noticable. Thanks four this lesson! -harry