From owner-freebsd-net@freebsd.org Sat Oct 10 14:33:01 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8207AA105F0 for ; Sat, 10 Oct 2015 14:33:01 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lb0-x22a.google.com (mail-lb0-x22a.google.com [IPv6:2a00:1450:4010:c04::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0BB2B1B83 for ; Sat, 10 Oct 2015 14:33:01 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by lbwr8 with SMTP id r8so106113353lbw.2 for ; Sat, 10 Oct 2015 07:32:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=rj9iSTSXURl3676IJIVrJIw702qtn4Hi8neUfVinRmY=; b=QFGnEsP0XZO53PGVLejrSYb4Gj7xbIeN+WT41FL9MTlFw7Dmn8+5s0fbwy5PCzUI9m tXecmACWw8+o+/WzPKOHFxiK/62jsBP0LYHqmoE5b7Xl3BXtcTFWHAIOLTTum7gD3Px9 6Ono5gL9KovY/m48y+QYKjMPHbTB6UhzkIqELFAte1U4lXfon4PTPaj89GLozRyM5/61 Y88Ic06aK7cWBXU4sMI5RC1uuTgvSKfULPRHePsz9k2O35wl0yCpTg5PEB+9nxdDgsMD jzv/dPvvEvw7wrBGMc2nL4gZt1rBrGL5Azi78D0AWNjzamnTnz56SHmNSR3hKJVWQae2 4uRA== MIME-Version: 1.0 X-Received: by 10.25.211.85 with SMTP id k82mr5900184lfg.100.1444487578656; Sat, 10 Oct 2015 07:32:58 -0700 (PDT) Sender: rizzo.unipi@gmail.com Received: by 10.114.96.168 with HTTP; Sat, 10 Oct 2015 07:32:58 -0700 (PDT) In-Reply-To: References: Date: Sat, 10 Oct 2015 16:32:58 +0200 X-Google-Sender-Auth: 5-HPohQm82EiFmSuutRqiqqyQSw Message-ID: Subject: Re: netmap: recommended NIC for 40GbE capture on Linux? From: Luigi Rizzo To: Bruce Merry Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Oct 2015 14:33:01 -0000 On Sat, Oct 10, 2015 at 4:03 PM, Bruce Merry wrote: > Hi > > We're aim to do some high-speed data capture and processing on data > being sent from an FPGA over Ethernet. The data is about 3.3Mpps with > 512 bytes of application-level data per packet (plus somewhere around > 64 bytes of app-level headers, plus UDP+IP+Ethernet headers). I'm > looking for some advice on what hardware to use. > > We currently have Mellanox ConnectX-3 NICs with QSFP+. My experiments > with pkt-gen on a single flow (generated by pkt-gen on another > machine) are topping out at receiving 2.4 Mpps, with ksortirqd pushing > one core at 100%. This is with netmap trunk, which does not build a > custom mlx4 driver. I've also tried > https://github.com/caldweba/netmap, which does, but which is broken > (receives only 1023 pps, confirmed broken by the repo owner). > > Before we spend money, I'd love to hear someone report success with > capturing a single flow at >4Mpps, >20Gbps using netmap on Linux and > what NIC they use. Ideally this would include transparent mode working > so that ARP, ICMP etc can be handled by the host stack. > > Alternatively, if anyone is managing this on ConnectX-3, I'd love to > know how (I have already disabled flow control and adaptive interrupt > mitigation on the receiver). > hi Bruce, email me privately for more details. We have native netmap support in FreeBSD for the intel 40G and and almost ready linux version of the same, which we hope to be available by the end of the year. Recently mellanox kindly sent us a couple of their 40g cards (not sure which type), for which i plan to adapt the mlx4_netmap_linux code that you found in the other repo (which i wrote a few years ago). Again we may work on that around the end of the year. cheers luigi -- -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+-------------------------------