Date: Sat, 27 Jul 2013 12:42:16 +0400 From: "Alexander V. Chernikov" <melifaro@FreeBSD.org> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: Barney Cordoba <barney_cordoba@yahoo.com>, Daniel Feenberg <feenberg@nber.org>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: Recommendations for 10gbps NIC Message-ID: <51F387E8.6090704@FreeBSD.org> In-Reply-To: <CA%2BhQ2%2BgQxdNM1o_rScuyPb_T9sm%2B=gEF1dUa5aknfUYrO4K9HA@mail.gmail.com> References: <51F0386D.2000709@acm.poly.edu> <51F16A07.9030505@FreeBSD.org> <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> <51F2A313.9070105@FreeBSD.org> <alpine.LRH.2.03.1307261646050.25254@nber.org> <1374876857.42890.YahooMailNeo@web121603.mail.ne1.yahoo.com> <51F37E97.3090203@FreeBSD.org> <CA%2BhQ2%2BgQxdNM1o_rScuyPb_T9sm%2B=gEF1dUa5aknfUYrO4K9HA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 27.07.2013 12:15, Luigi Rizzo wrote: > On Sat, Jul 27, 2013 at 10:02 AM, Alexander V. Chernikov > <melifaro@freebsd.org> wrote: >> On 27.07.2013 02:14, Barney Cordoba wrote: >>> >>> >>> >>> ------------------------------------------------------------------------ >>> *From:* Daniel Feenberg<feenberg@nber.org> >>> *To:* Alexander V. Chernikov<melifaro@FreeBSD.org> >>> *Cc:* Barney Cordoba<barney_cordoba@yahoo.com>; >>> "freebsd-net@freebsd.org"<freebsd-net@freebsd.org> >>> *Sent:* Friday, July 26, 2013 4:59 PM >>> >>> *Subject:* Re: Recommendations for 10gbps NIC >>> >>> >>> On Fri, 26 Jul 2013, Alexander V. Chernikov wrote: >>> >>> > On 26.07.2013 19:30, Barney Cordoba wrote: >>> >> >>> >> >>> >> >>> ------------------------------------------------------------------------ >>> >> *From:* Alexander V. Chernikov<melifaro@FreeBSD.org >>> <mailto:melifaro@FreeBSD.org>> >>> >> *To:* Boris Kochergin<spawk@acm.poly.edu<mailto:spawk@acm.poly.edu>> >>> >> *Cc:* freebsd-net@freebsd.org<mailto:freebsd-net@freebsd.org> >>> >>> >> *Sent:* Thursday, July 25, 2013 2:10 PM >>> >> *Subject:* Re: Recommendations for 10gbps NIC >>> >> >>> >> On 25.07.2013 00:26, Boris Kochergin wrote: >>> >> > Hi. >>> >> Hello. >>> >> > >>> >> > I am looking for recommendations for a 10gbps NIC from someone who >>> has >>> >> > successfully used it on FreeBSD. It will be used on FreeBSD >>> 9.1-R/amd64 >>> >> > to capture packets. Some desired features are: >>> >> > >>> >>> We have experience with HP NC523SFP and Chelsio N320E. The key difference >>> among 10GBE cards for us is how they treat foreign DACs. The HP would PXE >>> boot with several brands and generic DACs, but the Chelsio required a >>> Chelsio brand DAC to PXE boot. There was firmware on the NIC to check the >>> brand of cable. Both worked fine once booted. The Chelsio cables were hard >>> to find, which became a problem. Also, when used with diskless Unix >>> clients the Chelsio cards seemed to hang from time to time. Otherwise >>> packet loss was one in a million for both cards, even with 7 meter cables. >>> >>> We liked the fact that the Chelsio cards were single-port and cheaper. I >>> don't really understand why nearly all 10GBE cards are dual-port. Surely >>> there is a market for NICs between 1 gigabit and 20 gigabit. >>> >>> The NIC heatsinks are too hot to touch during use unless specially cooled. >>> >>> Daniel Feenberg >>> NBER >>> >>> >>> --------------------- >>> The same reason that they don't make single core cpus anymore. It costs >>> about the >>> same to make a 1 port chip as a 2 port chip. >>> >>> I find it interesting how so many talk about "the cards", when most >>> often the >>> differences are with "the drivers". Luigi made the most useful comment; >>> if you ever >>> want to use netmap, you need to buy a card compatible with netmap. >>> Although >>> you don't need netmap just to capture 10Gb/s. Forwarding, Maybe. >>> >>> I also find it interesting that nobody seems to have a handle on the >>> performance >>> differences. Obviously they're all different. Maybe substantially >>> different. >> >> It depends on what kind of performance you are talking about. >> All NICs are capable of doing linerate RX/TX for both small/big packets. > > this is actually not true. I have direct experience with Intel, > Mellanox and Broadcom, > and small packets are a problem across the board even with 1 port. > > From my experience only intel can do line rate (14.88Mpps) with 64-byte frames, > but suffers a bit with sizes that are not multiple of 64. > Mellanox peaks at around 7Mpps. > Broadcom is limited to some 2.5Mpps. Wow. So I'm wrong. However, Chelsio T4 can do at least ~8/port. I'll check for full linerate and report. > This is all with netmap, using the regular stack you are going to see > much much less. > > Large frames (1400+) are probably not a problem for anyone, but since the > original post asked for packet capture, i thought the small-frame case > is a relevant one. > >> The only notable exception I;m aware of are Intel 82598-based NICs which >> advertise PCI-E X8 gen2 with _2.5GT_ link speed, giving you maximum >> ~14Gbit/s bw for 2 ports instead of 20. > > This makes me curious because i believe people have used netmap with > the 82598 and achieved close to line rate even with 64-byte frames/one port, > and i thought (maybe I am wrong ?) the various 2-port NICs use 4 lanes per port. > So the number i remember does not match with your quote of 2.5Gt/s. > Are all 82598 using 2.5GT/s (which is a gen.1 speed) instead of 5 ? Quoting 82598EB datasheet: The PCIe v2.0 (2.5 GT/s) interface is used by the 82598EB as a host interface. It supports x8, x4, x2 and x1 configurations at a speed of 2.5 GHz. The maximum aggregated raw ban.. Actually I discovered this exactly with netmap and 82598*-DA2 NIC :) > > cheers > luigi >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?51F387E8.6090704>