From owner-freebsd-emulation@FreeBSD.ORG Thu Dec 27 10:26:46 2012 Return-Path: Delivered-To: emulation@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0215099F; Thu, 27 Dec 2012 10:26:46 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-ob0-f179.google.com (mail-ob0-f179.google.com [209.85.214.179]) by mx1.freebsd.org (Postfix) with ESMTP id A57F28FC13; Thu, 27 Dec 2012 10:26:45 +0000 (UTC) Received: by mail-ob0-f179.google.com with SMTP id x4so8597351obh.38 for ; Thu, 27 Dec 2012 02:26:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=LX/kYef3j8qMNQQ7r8FmHnV2+R35YoW3YPMJ7HkM6IA=; b=n/+oHrO13zKKMxIR+AUrkPoA8oUJMTIY3bRiQ9GjZ5q01+dcl89c0Hei0yidqvrUhR 4h53/ql86VBedlzTJgoKqPl+IS+brf0s7UTOOe6+B8ljCB6oiWB0X3XHsk5pgPxW3tZM l5ek5DEnoXvfEbZfh/veXw52MUxiNj4WNO6fCcXARHeIhb7DHMgkORxIhRS3yayvBxMs 2lc9miQJMeNcqZTxxV4pTE3sU5Obcr5VjtOnCWLX604Qf1CYU45nP2vUqABm1NKUqWqN GnCi1ujt58z5Jv+VcmPO1mYbj5Qog9yM0C9FXKGgklTFtDN4eGPQ11nRKcPaQhGJh+NX lDLw== MIME-Version: 1.0 Received: by 10.60.6.226 with SMTP id e2mr13287361oea.56.1356604004810; Thu, 27 Dec 2012 02:26:44 -0800 (PST) Received: by 10.76.143.33 with HTTP; Thu, 27 Dec 2012 02:26:44 -0800 (PST) In-Reply-To: <20121227094649.GA48891@onelab2.iet.unipi.it> References: <20121227094649.GA48891@onelab2.iet.unipi.it> Date: Thu, 27 Dec 2012 02:26:44 -0800 Message-ID: Subject: Re: [RFC] proposed 'lem' patch to improve behaviour under emulation From: Garrett Cooper To: Luigi Rizzo Content-Type: text/plain; charset=ISO-8859-1 Cc: emulation@freebsd.org, current@freebsd.org, jfvogel@gmail.com X-BeenThere: freebsd-emulation@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Development of Emulators of other operating systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2012 10:26:46 -0000 On Thu, Dec 27, 2012 at 1:46 AM, Luigi Rizzo wrote: > This patch implements two features for the 'lem' driver that > greatly improve the throughput under proper hypervisor. > This is joint work with Vincenzo Maffione and Giuseppe Lettieri, > I am posting it here for review, will then commit it > if there are no objections. > > The first change is to implement a sysctl to access the 'itr' > interrupt moderation register for the devices supported by this > driver. It is little more than adding a struct into the device > descriptor, and one line to create the dynamic sysctl entry, same > as it is done for the other mitigation registers. > > The second change is more interesting and has huge benefits on througput. > > Under virtualization, "VM exits" (which happen every time there is > an access to a register of the emulated peripheral) are extremely > expensive. In the tx path of the 'lem' driver, there is a write > to the TDT register on every packet sent. > > The patch we propose, if enabled through a sysctl (defaults off, > so no change from current behaviour) defers writes to the TDT > register when there is a pending transmit interrupt. > This means that, together with proper emulation of interrupt > mitigation on the hypervisor side, the number of VM exits > is dramatically reduced. To give you an idea, on a modern > system with qemu-kvm and companion patches, UDP throughput is > > KVM QEMU > standard KVM, standard driver 20 Kpps 6.3 Kpps > modified KVM, standard driver 37 Kpps 28 Kpps > modified KVM, modified driver 200 Kpps 34 Kpps > > As you can see, on kvm this change gives a 5x speedup to the tx path, > which combines nicely with the 2x speedup that comes from supporting > interrupt mitigation alone in the hypervisor. Without kvm (or kqemu ?) > the benefits are much lower, as the guest becomes too slow. > > Patch follows. It would be good if people with real hardware > using the 'lem' driver could test it to make sure it does no > harm on their devices (in any case the sysctl variable > dev.em.0.mit_enable must be set to explicitly enable it > at runtime). > > (for those curious to test it under kvm, i am also attaching a > patch that you need to apply to qemu in order to exploit the > effect of interrupt mitigation; it is a followup of a similar > patch i posted in july to the qemu mailing list, and i will > post it the update there as well, shortly. Unfortunately > we do not have kvm on freebsd..) A few comments. Thanks! -Garrett Index: if_lem.c =================================================================== --- if_lem.c (revision 244673) +++ if_lem.c (working copy) @@ -32,6 +32,8 @@ ******************************************************************************/ /*$FreeBSD$*/ +#define MITIGATION + gcooper> Could you please make MITIGATION into a proper compile time flag via sys/conf/options with a more descript name? #ifdef HAVE_KERNEL_OPTION_HEADERS #include "opt_device_polling.h" #include "opt_inet.h" @@ -281,6 +283,9 @@ #define EM_TICKS_TO_USECS(ticks) ((1024 * (ticks) + 500) / 1000) #define EM_USECS_TO_TICKS(usecs) ((1000 * (usecs) + 512) / 1024) +#define MAX_INTS_PER_SEC 8000 +#define DEFAULT_ITR 1000000000/(MAX_INTS_PER_SEC * 256) + gcooper> Add parentheses around DEFAULT_ITR (I know the code was just shuffled around, but thought I could ask :)..)?