From owner-freebsd-net@FreeBSD.ORG Mon Sep 22 22:15:04 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BAB60C9D for ; Mon, 22 Sep 2014 22:15:04 +0000 (UTC) Received: from mail-wg0-x229.google.com (mail-wg0-x229.google.com [IPv6:2a00:1450:400c:c00::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 52C63F7A for ; Mon, 22 Sep 2014 22:15:04 +0000 (UTC) Received: by mail-wg0-f41.google.com with SMTP id k14so3459900wgh.12 for ; Mon, 22 Sep 2014 15:15:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=M09xIKxQypfdoazG+OYbPv2CvjKk6L500Whetc2qudE=; b=vWcKO1ZhXPmTwZjoBGJz3f8fHEwT/BaGb044Aqvbox2oP4D6t4J4A8gmEHOAIbNE2E pGoWfqKboeAm8/2gjG2BEbQ5P8DCnikxjZT0A8sS4apwaPPwnkllxEc76dCX2JBJ64dA v2GefnQwD7YbS6e40bmTgVA/taCHphG1LqoxOX+5pGidx5vJTY+FtWySySMl9oygXTXg qaHKylhg4EJQ7KDXYxsIPoimDdgGleULhC141TJ2cX5OoEYYAytPrFhjYp8MQsriX3L7 4f4nn8AEx8OlObpnak6R5ATw6HXxha76YMc8N6qwfH5yaL0ospmIfcQGWuR3K2rG2kCw ePuA== MIME-Version: 1.0 X-Received: by 10.180.107.100 with SMTP id hb4mr17874642wib.59.1411424102622; Mon, 22 Sep 2014 15:15:02 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.216.106.199 with HTTP; Mon, 22 Sep 2014 15:15:02 -0700 (PDT) In-Reply-To: References: Date: Mon, 22 Sep 2014 15:15:02 -0700 X-Google-Sender-Auth: bUjsUqzQcOm3aVEcxett_pWDrAk Message-ID: Subject: Re: How do I balance bandwidth over several virtual NICs? From: Adrian Chadd To: Elof Ofel Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Sep 2014 22:15:04 -0000 On 22 September 2014 13:39, Elof Ofel wrote: > Hi Adrian! > > Now this sounds promising! All my sensors use the ixgbe driver. > However, my skills in programming/compiling isn't vast. I know how to patch > and use poudriere. That's about it. > > I must admit I don't really understand what you mean with "patch it to use a > symmetric RSS key", but it sounds like the functionality I'm looking for is > not yet there in the driver. A few years ago a couple of researchers figured out you could abuse the toeplitz hash to do symmetric RSS hashing: http://www.ndsl.kaist.edu/~shinae/papers/TR-symRSS.pdf This means that the same RSS hash value is chosen no matter which direction the traffic is going on. This is what you need to ensure that the packets going both directions in a connection end up in the same NIC hardware receive ring. So, all you have to do (!) is: * grab freebsd-head, because the ixgbe driver there has some recent bug fixes that you need for this to completely work; * look at ixgbe_initialise_rss_mapping() - that's where the RSS key, mapping and RSS hash types are configured; * patch it to use the example symmetric RSS key that was provided in the paper; * patch it to only hash on IPv4 / IPv6 2-tuple, that way you don't end up with IPv4/IPv6 fragments in the wrong queue; * configure up say, 4 or 8 rings in /boot/loader.conf: hw.ix.num_queues=8 (I think it's hw.ix, it used to be hw.ixgbe..) * then, when you use netmap on ixgbe, you just bind to each TX and RX ring with a separate process or thread. That thread will get packets in both directions for a given flow. > If we assume that someone in the future write and submit the above into the > ixgbe driver, could I be so bold as to ask you for a > commandline/configuration example (a brief guide) of how one would setup > netmap and how to configure it to use the RX-queues? I don't know of any examples of using netmap in this way from the command line. I've normally written C code (And when I do, i start with the bridge example in src/tools/tools/netmap/bridge.c) . > That way I can start playing around with netmap and learning it while I wait > for the ixgbe driver to be updated... I've got two professional programmer > colleagues who've dealt extensively with e.g. the libnids and pfring source > code, so if I get a grasp of how to setup netmap, and I find it interesting, > it is likely that they can dive into and fix the ixgbe driver and improve it > as per above. So please, can you help me with a "netmap guide"? > > When I try to find documentation or examples of how to setup netmap I find > none. Not even the netmap-enabled pcaplib contain any information as how to > use it. I'm no programmer, so showing me different C structs for delivering > data is of no use. :-/ You mean: https://code.google.com/p/netmap-libpcap/ ? I've not used it before, sorry :( -a