Date: Tue, 13 Jun 2017 18:30:22 +0200 From: Harry Schmalzbauer <freebsd@omnilan.de> To: Anish <akgupt3@gmail.com> Cc: "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: Re: PCIe passthrough really that expensive? Message-ID: <5940131E.8020604@omnilan.de> In-Reply-To: <CALnRwMSFQO6LMJ6ueqLrjx=a7r3UbZhsFPhvRnymh8vRn3dZGw@mail.gmail.com> References: <59383F5C.8020801@omnilan.de> <CALnRwMRst1d_O_ix-_JaS=tH8=dPtNNkDo9WyzRH1_nBi1N6zA@mail.gmail.com> <593A5AB1.7090301@omnilan.de> <593D1299.3030402@omnilan.de> <CALnRwMSFQO6LMJ6ueqLrjx=a7r3UbZhsFPhvRnymh8vRn3dZGw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Bezüglich Anish's Nachricht vom 13.06.2017 06:13 (localtime): > Hi Harry, >>Any hints highly appreciated! … > > Now use cpuset to route IRQ#265 to say core 0 > > $cpuset -l 0 -x 265 > > Again use cpuset to force VM[PID 1222] to run on all core except #0 > > root@svmhost:~ # ps > > PID TT STAT TIME COMMAND > > .... > > 1222 1 I+ 5:59.45 bhyve: vm1 (bhyve) > > > VM can run on all cores except #0. > > $ cpuset -l 1-3 -p 1222 > > > You can monitor guest due to interrupts using > > $root@svmhost:~ # bhyvectl --get-stats --vm=<vm name> --cpu=<vcpu> | > grep external > > vm exits due to external interrupt 27273 > > root@svmhost:~ # Thank you very much for that detailed explanation. I didn't thought that cpuset(1) could also pin IRQ (handlers?) to specific cpus. In my case, I couldn't get a noticable difference. Since I have hyperthreading enabled on a single-socket quad core, I pinned cpu2+3 to the bhyve pid (2vCPUs) and for testing copied two times the same 8GB file ofer NFS, while having ppt's IRQ handler pinned to CPU1, CPU4, CPU3 and CPU2. So two times different hostCPUs than guest uses and two times the same. I couldn't see any load nor performance difference and similar 'vm exits due to external interrupt' count groth. To name numbers: 1st CPU had about 40k "vm exits due to external interrupt" per 8GP transfer, the other vCPU ~160k "vm exits due to external interrupt". Like mentioned, different host-CPU-pinning didn't influence that noticable. Thanks four this lesson! -harry
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5940131E.8020604>