From owner-freebsd-current@FreeBSD.ORG Thu Dec 27 14:00:03 2012 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9FF37B0B; Thu, 27 Dec 2012 14:00:03 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 23BA48FC17; Thu, 27 Dec 2012 14:00:01 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 7B2477300A; Thu, 27 Dec 2012 14:58:56 +0100 (CET) Date: Thu, 27 Dec 2012 14:58:56 +0100 From: Luigi Rizzo To: Garrett Cooper Subject: Re: [RFC] proposed 'lem' patch to improve behaviour under emulation Message-ID: <20121227135856.GA51105@onelab2.iet.unipi.it> References: <20121227094649.GA48891@onelab2.iet.unipi.it> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: emulation@freebsd.org, current@freebsd.org, jfvogel@gmail.com X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2012 14:00:03 -0000 On Thu, Dec 27, 2012 at 02:26:44AM -0800, Garrett Cooper wrote: > On Thu, Dec 27, 2012 at 1:46 AM, Luigi Rizzo wrote: > > This patch implements two features for the 'lem' driver that > > greatly improve the throughput under proper hypervisor. > > This is joint work with Vincenzo Maffione and Giuseppe Lettieri, > > I am posting it here for review, will then commit it > > if there are no objections. > > > > The first change is to implement a sysctl to access the 'itr' > > interrupt moderation register for the devices supported by this > > driver. It is little more than adding a struct into the device > > descriptor, and one line to create the dynamic sysctl entry, same > > as it is done for the other mitigation registers. > > > > The second change is more interesting and has huge benefits on througput. > > > > Under virtualization, "VM exits" (which happen every time there is > > an access to a register of the emulated peripheral) are extremely > > expensive. In the tx path of the 'lem' driver, there is a write > > to the TDT register on every packet sent. > > > > The patch we propose, if enabled through a sysctl (defaults off, > > so no change from current behaviour) defers writes to the TDT > > register when there is a pending transmit interrupt. > > This means that, together with proper emulation of interrupt > > mitigation on the hypervisor side, the number of VM exits > > is dramatically reduced. To give you an idea, on a modern > > system with qemu-kvm and companion patches, UDP throughput is > > > > KVM QEMU > > standard KVM, standard driver 20 Kpps 6.3 Kpps > > modified KVM, standard driver 37 Kpps 28 Kpps > > modified KVM, modified driver 200 Kpps 34 Kpps > > > > As you can see, on kvm this change gives a 5x speedup to the tx path, > > which combines nicely with the 2x speedup that comes from supporting > > interrupt mitigation alone in the hypervisor. Without kvm (or kqemu ?) > > the benefits are much lower, as the guest becomes too slow. > > > > Patch follows. It would be good if people with real hardware > > using the 'lem' driver could test it to make sure it does no > > harm on their devices (in any case the sysctl variable > > dev.em.0.mit_enable must be set to explicitly enable it > > at runtime). > > > > (for those curious to test it under kvm, i am also attaching a > > patch that you need to apply to qemu in order to exploit the > > effect of interrupt mitigation; it is a followup of a similar > > patch i posted in july to the qemu mailing list, and i will > > post it the update there as well, shortly. Unfortunately > > we do not have kvm on freebsd..) > > A few comments. > Thanks! > -Garrett > > Index: if_lem.c > =================================================================== > --- if_lem.c (revision 244673) > +++ if_lem.c (working copy) > @@ -32,6 +32,8 @@ > ******************************************************************************/ > /*$FreeBSD$*/ > > +#define MITIGATION > + > > gcooper> Could you please make MITIGATION into a proper compile time > flag via sys/conf/options with a more descript name? this is actually going away, with the code compiled in by default ... > @@ -281,6 +283,9 @@ > #define EM_TICKS_TO_USECS(ticks) ((1024 * (ticks) + 500) / 1000) > #define EM_USECS_TO_TICKS(usecs) ((1000 * (usecs) + 512) / 1024) > > +#define MAX_INTS_PER_SEC 8000 > +#define DEFAULT_ITR 1000000000/(MAX_INTS_PER_SEC * 256) > + > > gcooper> Add parentheses around DEFAULT_ITR (I know the code was just > shuffled around, but thought I could ask :)..)? will do thanks luigi