From owner-freebsd-net@FreeBSD.ORG Sat Jul 27 08:42:26 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 665969AA for ; Sat, 27 Jul 2013 08:42:26 +0000 (UTC) (envelope-from melifaro@FreeBSD.org) Received: from mail.ipfw.ru (unknown [IPv6:2a01:4f8:120:6141::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id F05232C9A for ; Sat, 27 Jul 2013 08:42:25 +0000 (UTC) Received: from v6.mpls.in ([2a02:978:2::5] helo=ws.su29.net) by mail.ipfw.ru with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.76 (FreeBSD)) (envelope-from ) id 1V3086-000DH9-4b; Sat, 27 Jul 2013 12:45:34 +0400 Message-ID: <51F387E8.6090704@FreeBSD.org> Date: Sat, 27 Jul 2013 12:42:16 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20120121 Thunderbird/9.0 MIME-Version: 1.0 To: Luigi Rizzo Subject: Re: Recommendations for 10gbps NIC References: <51F0386D.2000709@acm.poly.edu> <51F16A07.9030505@FreeBSD.org> <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> <51F2A313.9070105@FreeBSD.org> <1374876857.42890.YahooMailNeo@web121603.mail.ne1.yahoo.com> <51F37E97.3090203@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Barney Cordoba , Daniel Feenberg , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jul 2013 08:42:26 -0000 On 27.07.2013 12:15, Luigi Rizzo wrote: > On Sat, Jul 27, 2013 at 10:02 AM, Alexander V. Chernikov > wrote: >> On 27.07.2013 02:14, Barney Cordoba wrote: >>> >>> >>> >>> ------------------------------------------------------------------------ >>> *From:* Daniel Feenberg >>> *To:* Alexander V. Chernikov >>> *Cc:* Barney Cordoba; >>> "freebsd-net@freebsd.org" >>> *Sent:* Friday, July 26, 2013 4:59 PM >>> >>> *Subject:* Re: Recommendations for 10gbps NIC >>> >>> >>> On Fri, 26 Jul 2013, Alexander V. Chernikov wrote: >>> >>> > On 26.07.2013 19:30, Barney Cordoba wrote: >>> >> >>> >> >>> >> >>> ------------------------------------------------------------------------ >>> >> *From:* Alexander V. Chernikov>> > >>> >> *To:* Boris Kochergin> >>> >> *Cc:* freebsd-net@freebsd.org >>> >>> >> *Sent:* Thursday, July 25, 2013 2:10 PM >>> >> *Subject:* Re: Recommendations for 10gbps NIC >>> >> >>> >> On 25.07.2013 00:26, Boris Kochergin wrote: >>> >> > Hi. >>> >> Hello. >>> >> > >>> >> > I am looking for recommendations for a 10gbps NIC from someone who >>> has >>> >> > successfully used it on FreeBSD. It will be used on FreeBSD >>> 9.1-R/amd64 >>> >> > to capture packets. Some desired features are: >>> >> > >>> >>> We have experience with HP NC523SFP and Chelsio N320E. The key difference >>> among 10GBE cards for us is how they treat foreign DACs. The HP would PXE >>> boot with several brands and generic DACs, but the Chelsio required a >>> Chelsio brand DAC to PXE boot. There was firmware on the NIC to check the >>> brand of cable. Both worked fine once booted. The Chelsio cables were hard >>> to find, which became a problem. Also, when used with diskless Unix >>> clients the Chelsio cards seemed to hang from time to time. Otherwise >>> packet loss was one in a million for both cards, even with 7 meter cables. >>> >>> We liked the fact that the Chelsio cards were single-port and cheaper. I >>> don't really understand why nearly all 10GBE cards are dual-port. Surely >>> there is a market for NICs between 1 gigabit and 20 gigabit. >>> >>> The NIC heatsinks are too hot to touch during use unless specially cooled. >>> >>> Daniel Feenberg >>> NBER >>> >>> >>> --------------------- >>> The same reason that they don't make single core cpus anymore. It costs >>> about the >>> same to make a 1 port chip as a 2 port chip. >>> >>> I find it interesting how so many talk about "the cards", when most >>> often the >>> differences are with "the drivers". Luigi made the most useful comment; >>> if you ever >>> want to use netmap, you need to buy a card compatible with netmap. >>> Although >>> you don't need netmap just to capture 10Gb/s. Forwarding, Maybe. >>> >>> I also find it interesting that nobody seems to have a handle on the >>> performance >>> differences. Obviously they're all different. Maybe substantially >>> different. >> >> It depends on what kind of performance you are talking about. >> All NICs are capable of doing linerate RX/TX for both small/big packets. > > this is actually not true. I have direct experience with Intel, > Mellanox and Broadcom, > and small packets are a problem across the board even with 1 port. > > From my experience only intel can do line rate (14.88Mpps) with 64-byte frames, > but suffers a bit with sizes that are not multiple of 64. > Mellanox peaks at around 7Mpps. > Broadcom is limited to some 2.5Mpps. Wow. So I'm wrong. However, Chelsio T4 can do at least ~8/port. I'll check for full linerate and report. > This is all with netmap, using the regular stack you are going to see > much much less. > > Large frames (1400+) are probably not a problem for anyone, but since the > original post asked for packet capture, i thought the small-frame case > is a relevant one. > >> The only notable exception I;m aware of are Intel 82598-based NICs which >> advertise PCI-E X8 gen2 with _2.5GT_ link speed, giving you maximum >> ~14Gbit/s bw for 2 ports instead of 20. > > This makes me curious because i believe people have used netmap with > the 82598 and achieved close to line rate even with 64-byte frames/one port, > and i thought (maybe I am wrong ?) the various 2-port NICs use 4 lanes per port. > So the number i remember does not match with your quote of 2.5Gt/s. > Are all 82598 using 2.5GT/s (which is a gen.1 speed) instead of 5 ? Quoting 82598EB datasheet: The PCIe v2.0 (2.5 GT/s) interface is used by the 82598EB as a host interface. It supports x8, x4, x2 and x1 configurations at a speed of 2.5 GHz. The maximum aggregated raw ban.. Actually I discovered this exactly with netmap and 82598*-DA2 NIC :) > > cheers > luigi >