From owner-freebsd-virtualization@freebsd.org Tue Jun 13 04:13:18 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B63C5D8D01A for ; Tue, 13 Jun 2017 04:13:18 +0000 (UTC) (envelope-from akgupt3@gmail.com) Received: from mail-qt0-x22c.google.com (mail-qt0-x22c.google.com [IPv6:2607:f8b0:400d:c0d::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7E023684B8 for ; Tue, 13 Jun 2017 04:13:18 +0000 (UTC) (envelope-from akgupt3@gmail.com) Received: by mail-qt0-x22c.google.com with SMTP id u19so154605125qta.3 for ; Mon, 12 Jun 2017 21:13:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=/AUGoSxDfv7u4+EoYLuGdNnjus2Cwg88M+/g7yBlhvE=; b=JWImGfApiIEP5r/Zmt9EzfcUXsx1SjaJtoSR8YfcmtqL85QfipYCwpIhDo/ttu3+a4 q87y0ROldKQEBDy/vqO3VVD+kGSythAY4e9ywYmi6GpsX05k3+Qjo5/qGtAdzml7XtFM tc3p3QQoTfONO6NqyWyLLypUmUyhFiXo+N6ZAuhaqFXy4Y5zcdX0l1eTTnqH7xDnn32/ fRuoHeRSKW7iRN6mzH00oMkrodQt4y8ND6J3VebHYjZXI4IU8ROVwB2c+QlEWGhZ2pN1 cm+zbNEq8tspyIPsd1qKHHBW1xCbKiVVDjH4Abg2Dix2mbazKiNTEsmDEOBGY/0LXXF4 s+MA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=/AUGoSxDfv7u4+EoYLuGdNnjus2Cwg88M+/g7yBlhvE=; b=dWYyMPe9lmkGWPyJZIBJAVB4prHjMZ7QjJOz9bT1dVK4yabVVlA2C/2pPJt/2lO3eR VPEBRvvzG+MQ7OzTjhELiS6FUgMRkLKsM4znGhV9RAtMDnEH6yb9gmE4/2ZO8txauGHm EZw2IOCo9VcevkVMmnTCgpZaavIlvUJOEjvwn6vIbCMM/SNrk8u3+64WQr8H914Faphq jSGjZen+jwcUPK2YopR715e58n/XcQtc2SZnmXvvqSpq2gwBvis9KR//epx3I/80PMJW VF8bsPyDNMOcjMBYnGJAsO7neFVsdbvkdQoZ1MPFz7oJf4/TmIyGHp2oPr4o9W61K8Z3 Rozw== X-Gm-Message-State: AODbwcAgLhqQ8uI3X/2gkVOkyoTf1d4IbRzMVzDXSVXXmz3dv3k0Fu6v Axqs/gfTJ3XSCupTti4hwqshqcY+qBWK X-Received: by 10.237.43.227 with SMTP id e90mr55732359qtd.103.1497327197432; Mon, 12 Jun 2017 21:13:17 -0700 (PDT) MIME-Version: 1.0 Received: by 10.12.181.203 with HTTP; Mon, 12 Jun 2017 21:13:16 -0700 (PDT) In-Reply-To: <593D1299.3030402@omnilan.de> References: <59383F5C.8020801@omnilan.de> <593A5AB1.7090301@omnilan.de> <593D1299.3030402@omnilan.de> From: Anish Date: Mon, 12 Jun 2017 21:13:16 -0700 Message-ID: Subject: Re: PCIe passthrough really that expensive? To: Harry Schmalzbauer Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jun 2017 04:13:18 -0000 Hi Harry, >Any hints highly appreciated! In my setup, I have dual port Intel GigE, one assigned to host and other one is used by guest root@svmhost:~ # pciconf -l |grep ppt ... ppt2@pci0:2:0:0: class=3D0x020000 card=3D0x125e8086 chip=3D0x105e8086 rev= =3D0x06 hdr=3D0x00 root@svmhost:~ # This show up as 'em0' in BSD guest with 2 vcpus. root@bsdguest:~ # pciconf -l |grep em0 em0@pci0:0:21:0: class=3D0x020000 card=3D0x125e8086 chip=3D0x105e808= 6 rev=3D0x06 hdr=3D0x00 root@bsdguest:~ # Once guest is booted, ppt2 will claim interrupt resources, in this case just 1 interrupt line #265 root@svmhost:~ # vmstat -i interrupt total rate ... irq263: em0:irq0 1028705 634 .. irq265: ppt2 1041187 641 Total 2835121 1747 Now use cpuset to route IRQ#265 to say core 0 $cpuset -l 0 -x 265 Again use cpuset to force VM[PID 1222] to run on all core except #0 root@svmhost:~ # ps PID TT STAT TIME COMMAND .... 1222 1 I+ 5:59.45 bhyve: vm1 (bhyve) VM can run on all cores except #0. $ cpuset -l 1-3 -p 1222 You can monitor guest due to interrupts using $root@svmhost:~ # bhyvectl --get-stats --vm=3D --cpu=3D | gr= ep external vm exits due to external interrupt 27273 root@svmhost:~ # Regards, Anish On Sun, Jun 11, 2017 at 2:51 AM, Harry Schmalzbauer wrote: > Bez=C3=BCglich Harry Schmalzbauer's Nachricht vom 09.06.2017 10:22 (loca= ltime): > > Bez=C3=BCglich Anish's Nachricht vom 08.06.2017 14:35 (localtime): > >> Hi Harry, > >>> I thought I'd save these expensive VM_Exits by using the passthru pat= h. > >> Completely wrong, is it? > >> > >> It depends on which processor you are using. For example APICv was > >> introduced in IvyBridge which enabled h/w assisted localAPIC rather th= an > >> using s/w emulated, bhyve supports it on Intel processors. > =E2=80=A6 > > I'm still usign IvyBridge (E3v2) with this "new" machine, but haven't > > ever heard/thought about APCIv! > > It seems APICv is available on IvyBridge-EP (Xeon E5/E7v2) only, not for > E3v2 :-( > Furthermore, if I didn't miss anything in the datasheets, no currently > available E3 Xeon offers local APIC virtualization. Can somebody of the > xperts confirm that? > > > =E2=80=A6 > >> Can you run a simple experiment, assign pptdev interrupts to core that= 's > >> not running guest/vcpu? This will reduce #VMEXIT on vcpu which we know > >> is expensive. > > Interesting approach. But I have no idea how I should assign a PCIe > > specific core to a PCIe dev. Is it pptdev specific? The tunables in > > device.hints(5) can't be used for that, can they? > > I wasn't able to find out how to do that. > Any hints highly appreciated! > > -harry > >