Date: Sat, 10 Oct 2015 16:03:48 +0200 From: Bruce Merry <bmerry@ska.ac.za> To: freebsd-net@freebsd.org Subject: netmap: recommended NIC for 40GbE capture on Linux? Message-ID: <CAOm-9aps449E9uZshtMmwNRD=azt1-Ciz0TNiuG4LHaG4UCW2Q@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hi We're aim to do some high-speed data capture and processing on data being sent from an FPGA over Ethernet. The data is about 3.3Mpps with 512 bytes of application-level data per packet (plus somewhere around 64 bytes of app-level headers, plus UDP+IP+Ethernet headers). I'm looking for some advice on what hardware to use. We currently have Mellanox ConnectX-3 NICs with QSFP+. My experiments with pkt-gen on a single flow (generated by pkt-gen on another machine) are topping out at receiving 2.4 Mpps, with ksortirqd pushing one core at 100%. This is with netmap trunk, which does not build a custom mlx4 driver. I've also tried https://github.com/caldweba/netmap, which does, but which is broken (receives only 1023 pps, confirmed broken by the repo owner). Before we spend money, I'd love to hear someone report success with capturing a single flow at >4Mpps, >20Gbps using netmap on Linux and what NIC they use. Ideally this would include transparent mode working so that ARP, ICMP etc can be handled by the host stack. Alternatively, if anyone is managing this on ConnectX-3, I'd love to know how (I have already disabled flow control and adaptive interrupt mitigation on the receiver). Thanks Bruce -- Bruce Merry Senior Science Processing Developer SKA South Africa
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOm-9aps449E9uZshtMmwNRD=azt1-Ciz0TNiuG4LHaG4UCW2Q>