From owner-freebsd-net@freebsd.org Sat Oct 10 18:04:30 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DAC71A10D16 for ; Sat, 10 Oct 2015 18:04:30 +0000 (UTC) (envelope-from sbruno@freebsd.org) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C02801163 for ; Sat, 10 Oct 2015 18:04:30 +0000 (UTC) (envelope-from sbruno@freebsd.org) Received: from [192.168.200.208] (unknown [50.136.155.142]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id 94E3A193E80 for ; Sat, 10 Oct 2015 18:04:28 +0000 (UTC) Subject: Re: netmap: recommended NIC for 40GbE capture on Linux? To: freebsd-net@freebsd.org References: From: Sean Bruno Message-ID: <5619532B.4020304@freebsd.org> Date: Sat, 10 Oct 2015 11:04:27 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Oct 2015 18:04:30 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 10/10/15 07:03, Bruce Merry wrote: > Hi > > We're aim to do some high-speed data capture and processing on > data being sent from an FPGA over Ethernet. The data is about > 3.3Mpps with 512 bytes of application-level data per packet (plus > somewhere around 64 bytes of app-level headers, plus > UDP+IP+Ethernet headers). I'm looking for some advice on what > hardware to use. > > We currently have Mellanox ConnectX-3 NICs with QSFP+. My > experiments with pkt-gen on a single flow (generated by pkt-gen on > another machine) are topping out at receiving 2.4 Mpps, with > ksortirqd pushing one core at 100%. This is with netmap trunk, > which does not build a custom mlx4 driver. I've also tried > https://github.com/caldweba/netmap, which does, but which is > broken (receives only 1023 pps, confirmed broken by the repo > owner). > > Before we spend money, I'd love to hear someone report success > with capturing a single flow at >4Mpps, >20Gbps using netmap on > Linux and what NIC they use. Ideally this would include transparent > mode working so that ARP, ICMP etc can be handled by the host > stack. > > Alternatively, if anyone is managing this on ConnectX-3, I'd love > to know how (I have already disabled flow control and adaptive > interrupt mitigation on the receiver). > > Thanks Bruce > I'm unsure where to ask netmap+linux question to be honest. So, my responses won't be very useful to you. If you feel like doing even *more* science, FreeBSD does have the same capabilities and supports the card that you're using. It might be more work for you to do, but you might find FreeBSD-current (11.0 test) refreshing enough to warrant more investigation. ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/amd64/amd64/ISO-IMAGES/11.0/ sean -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAEBCgBmBQJWGVMoXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5kyooH/1bIoaytmkUR73ysuRoBTKcY UHSc7bvngiMLK15cAKRn9UzXoQHjZRQBMXb9mr1omPwdkzsmfx3JAdOIuNhKi0li nYYth8/Cs/GUyKxjaYBs/oLydKoZahgiaeNaEqftQmxkOINOXMI6xLxr4p1kMy6k AbF/LkaSgoyBH/Yu96EK7dQBNCroJuI6BKI7EZdLL93seecsTLx78+qEfXJJI7FC w5vSD5amFMDKMiAewMzhiHzY+uO5rcYzxGbCineF+6aaKT4jbxJ/X6OoEvw8Zzto tkSmDiwxFG14YAsn0RQ4BXF3NDFQOLcH0YzTe+mSNwkrv5rSmAdUaTADuiBd+2o= =Sz+N -----END PGP SIGNATURE-----