From owner-freebsd-current@FreeBSD.ORG Fri Jun 3 13:42:28 2011 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 79B28106566B for ; Fri, 3 Jun 2011 13:42:28 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 40A9F8FC13 for ; Fri, 3 Jun 2011 13:42:27 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id D0C787300B; Fri, 3 Jun 2011 15:58:28 +0200 (CEST) Date: Fri, 3 Jun 2011 15:58:28 +0200 From: Luigi Rizzo To: Patrick Tracanelli Message-ID: <20110603135828.GA42615@onelab2.iet.unipi.it> References: <20110602223140.GA31285@onelab2.iet.unipi.it> <85C4C9B4-9413-4695-B625-BDAA00B97662@freebsdbrasil.com.br> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <85C4C9B4-9413-4695-B625-BDAA00B97662@freebsdbrasil.com.br> User-Agent: Mutt/1.4.2.3i Cc: current@freebsd.org Subject: Re: Introducing netmap: line-rate packet send/receive at 10Gbit/s X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2011 13:42:28 -0000 On Fri, Jun 03, 2011 at 10:20:50AM -0300, Patrick Tracanelli wrote: > > Em 02/06/2011, ?s 19:31, Luigi Rizzo escreveu: > > > Hi, > > we have recently worked on a project, called netmap, which lets > > FreeBSD send/receive packets at line rate even at 10 Gbit/s with > > very low CPU overhead: one core at 1.33 GHz does 14.88 Mpps with a > > modified ixgbe driver, which gives plenty of CPU cycles to handle > > multiple interface and/or do useful work (packet forwarding, analysis, etc.) > > > > You can find full documentation and source code and even a picobsd image at > > > > http://info.iet.unipi.it/~luigi/netmap/ > > > > The system uses memory mapped packet buffers to reduce the cost of > > data movements, but this would not be enough to make it useful or > > novel. Netmap uses many other small but important tricks to make > > the system fast, safe and easy to use, and support transmission, > > reception, and communication with the host stack. > > > > You can see full details in documentation at the above link. > > > > Feedback welcome. > > Dear Rizzo, > > Which packet len you transmitted at 14.8MPPS? According to figure 5 or the description I could not find it. Did you test TCP? The paper gives all details in sec.6, please read it carefully. The test is done with individual streams (either tx or rx) of packets, the protocol is irrelevant. In fig.5 of the paper, packet size is on the X axis, pps is on the Y axis. You get the maximum PPS rate with min-sized packets (60 bytes + 4 byte crc). If you want to do both send and receive and perhaps on multiple interfaces you should make sure there are enough resources (CPU cycles, bus bandwidth and transactions etc.) for the task. In my tests, CPU does not seem to be a problem (i can send about 27Mpps with just one core and two interfaces), but bus cycles perhaps are (e.g. receiving with some of the "bad" packet sizes also slows down the sender on the same bus, no matter how many cores i put in). cheers luigi > How did you perform this test? Multihomed with forwarding between NICs or you generated the data from userland to the wire and let it flow? If not tell me how you believe netmap may impact in our current forwarding rate (specially the pps limit) and FreeBSD should be changed to take advantage of netmap for pkt forwarding. > > Thank you for your time, code and all the stuff in between :) > > -- > Patrick Tracanelli >