From owner-freebsd-emulation@FreeBSD.ORG Thu Dec 27 18:46:58 2012 Return-Path: Delivered-To: emulation@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2DE73D7D; Thu, 27 Dec 2012 18:46:58 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vb0-f44.google.com (mail-vb0-f44.google.com [209.85.212.44]) by mx1.freebsd.org (Postfix) with ESMTP id B683A8FC12; Thu, 27 Dec 2012 18:46:57 +0000 (UTC) Received: by mail-vb0-f44.google.com with SMTP id fc26so10295852vbb.3 for ; Thu, 27 Dec 2012 10:46:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=TgKDM8NpNo0ZrfGlbYkVA97bdedyThROEb2EHJANCVk=; b=kUlApyeGB44dRvdckb9Rt9+wWTIRDdvhcWtThvsxYIp9BFoGvy67zxIYNp7L+qz7/j W6dz0aGgaXTl1lUC/5oKhpnw2DZ/W5SLeSU9jkCb73ybS0xpKtJ6vSHm1aWavLRCnnOE pYHyosUeVkE6KUv29DUUQwJk6cNGByW0NsBC17wQfbGik/0E9pB/XC/XbWLBvzAe7RPi ODFnr7YYnsHasASoyNn0B7W+CTun6iwXE0Hb9xJLkQ72En/vOo9yHmRA3hjekxKam+o6 hY6wiohf6N8AqBH4UvXh1gzQgda2N0fwiHiTcetz78d3fcRIxVhJaqxxQiwBr4LLU8CV eenA== MIME-Version: 1.0 Received: by 10.58.116.212 with SMTP id jy20mr48701085veb.5.1356634011440; Thu, 27 Dec 2012 10:46:51 -0800 (PST) Received: by 10.220.50.6 with HTTP; Thu, 27 Dec 2012 10:46:51 -0800 (PST) In-Reply-To: <20121227094649.GA48891@onelab2.iet.unipi.it> References: <20121227094649.GA48891@onelab2.iet.unipi.it> Date: Thu, 27 Dec 2012 10:46:51 -0800 Message-ID: Subject: Re: [RFC] proposed 'lem' patch to improve behaviour under emulation From: Jack Vogel To: Luigi Rizzo Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: emulation@freebsd.org, current@freebsd.org X-BeenThere: freebsd-emulation@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Development of Emulators of other operating systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2012 18:46:58 -0000 LOL, it's ironic, my intention in creating lem was to isolate the old pre-PCIE driver from active changes so as to assure it's stability... but virtualization comes around to bit me in the butt :) I guess I'm agreeable in principle with what you're doing Luigi, but can you do me a favor and hold off until I'm technically back from vacation (after the new year) and let me review the code then? Thanks, Jack On Thu, Dec 27, 2012 at 1:46 AM, Luigi Rizzo wrote: > This patch implements two features for the 'lem' driver that > greatly improve the throughput under proper hypervisor. > This is joint work with Vincenzo Maffione and Giuseppe Lettieri, > I am posting it here for review, will then commit it > if there are no objections. > > The first change is to implement a sysctl to access the 'itr' > interrupt moderation register for the devices supported by this > driver. It is little more than adding a struct into the device > descriptor, and one line to create the dynamic sysctl entry, same > as it is done for the other mitigation registers. > > The second change is more interesting and has huge benefits on througput. > > Under virtualization, "VM exits" (which happen every time there is > an access to a register of the emulated peripheral) are extremely > expensive. In the tx path of the 'lem' driver, there is a write > to the TDT register on every packet sent. > > The patch we propose, if enabled through a sysctl (defaults off, > so no change from current behaviour) defers writes to the TDT > register when there is a pending transmit interrupt. > This means that, together with proper emulation of interrupt > mitigation on the hypervisor side, the number of VM exits > is dramatically reduced. To give you an idea, on a modern > system with qemu-kvm and companion patches, UDP throughput is > > KVM QEMU > standard KVM, standard driver 20 Kpps 6.3 Kpps > modified KVM, standard driver 37 Kpps 28 Kpps > modified KVM, modified driver 200 Kpps 34 Kpps > > As you can see, on kvm this change gives a 5x speedup to the tx path, > which combines nicely with the 2x speedup that comes from supporting > interrupt mitigation alone in the hypervisor. Without kvm (or kqemu ?) > the benefits are much lower, as the guest becomes too slow. > > Patch follows. It would be good if people with real hardware > using the 'lem' driver could test it to make sure it does no > harm on their devices (in any case the sysctl variable > dev.em.0.mit_enable must be set to explicitly enable it > at runtime). > > (for those curious to test it under kvm, i am also attaching a > patch that you need to apply to qemu in order to exploit the > effect of interrupt mitigation; it is a followup of a similar > patch i posted in july to the qemu mailing list, and i will > post it the update there as well, shortly. Unfortunately > we do not have kvm on freebsd..) > > cheers > luigi >