Date: Mon, 12 Jun 2017 21:13:16 -0700 From: Anish <akgupt3@gmail.com> To: Harry Schmalzbauer <freebsd@omnilan.de> Cc: "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: Re: PCIe passthrough really that expensive? Message-ID: <CALnRwMSFQO6LMJ6ueqLrjx=a7r3UbZhsFPhvRnymh8vRn3dZGw@mail.gmail.com> In-Reply-To: <593D1299.3030402@omnilan.de> References: <59383F5C.8020801@omnilan.de> <CALnRwMRst1d_O_ix-_JaS=tH8=dPtNNkDo9WyzRH1_nBi1N6zA@mail.gmail.com> <593A5AB1.7090301@omnilan.de> <593D1299.3030402@omnilan.de>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Harry, >Any hints highly appreciated! In my setup, I have dual port Intel GigE, one assigned to host and other one is used by guest root@svmhost:~ # pciconf -l |grep ppt ... ppt2@pci0:2:0:0: class=3D0x020000 card=3D0x125e8086 chip=3D0x105e8086 rev= =3D0x06 hdr=3D0x00 root@svmhost:~ # This show up as 'em0' in BSD guest with 2 vcpus. root@bsdguest:~ # pciconf -l |grep em0 em0@pci0:0:21:0: class=3D0x020000 card=3D0x125e8086 chip=3D0x105e808= 6 rev=3D0x06 hdr=3D0x00 root@bsdguest:~ # Once guest is booted, ppt2 will claim interrupt resources, in this case just 1 interrupt line #265 root@svmhost:~ # vmstat -i interrupt total rate ... irq263: em0:irq0 1028705 634 .. irq265: ppt2 1041187 641 Total 2835121 1747 Now use cpuset to route IRQ#265 to say core 0 $cpuset -l 0 -x 265 Again use cpuset to force VM[PID 1222] to run on all core except #0 root@svmhost:~ # ps PID TT STAT TIME COMMAND .... 1222 1 I+ 5:59.45 bhyve: vm1 (bhyve) VM can run on all cores except #0. $ cpuset -l 1-3 -p 1222 You can monitor guest due to interrupts using $root@svmhost:~ # bhyvectl --get-stats --vm=3D<vm name> --cpu=3D<vcpu> | gr= ep external vm exits due to external interrupt 27273 root@svmhost:~ # Regards, Anish On Sun, Jun 11, 2017 at 2:51 AM, Harry Schmalzbauer <freebsd@omnilan.de> wrote: > Bez=C3=BCglich Harry Schmalzbauer's Nachricht vom 09.06.2017 10:22 (loca= ltime): > > Bez=C3=BCglich Anish's Nachricht vom 08.06.2017 14:35 (localtime): > >> Hi Harry, > >>> I thought I'd save these expensive VM_Exits by using the passthru pat= h. > >> Completely wrong, is it? > >> > >> It depends on which processor you are using. For example APICv was > >> introduced in IvyBridge which enabled h/w assisted localAPIC rather th= an > >> using s/w emulated, bhyve supports it on Intel processors. > =E2=80=A6 > > I'm still usign IvyBridge (E3v2) with this "new" machine, but haven't > > ever heard/thought about APCIv! > > It seems APICv is available on IvyBridge-EP (Xeon E5/E7v2) only, not for > E3v2 :-( > Furthermore, if I didn't miss anything in the datasheets, no currently > available E3 Xeon offers local APIC virtualization. Can somebody of the > xperts confirm that? > > > =E2=80=A6 > >> Can you run a simple experiment, assign pptdev interrupts to core that= 's > >> not running guest/vcpu? This will reduce #VMEXIT on vcpu which we know > >> is expensive. > > Interesting approach. But I have no idea how I should assign a PCIe > > specific core to a PCIe dev. Is it pptdev specific? The tunables in > > device.hints(5) can't be used for that, can they? > > I wasn't able to find out how to do that. > Any hints highly appreciated! > > -harry > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALnRwMSFQO6LMJ6ueqLrjx=a7r3UbZhsFPhvRnymh8vRn3dZGw>