From owner-freebsd-net@FreeBSD.ORG Fri Jul 26 22:14:19 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C4927B36 for ; Fri, 26 Jul 2013 22:14:19 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm7-vm0.bullet.mail.ne1.yahoo.com (nm7-vm0.bullet.mail.ne1.yahoo.com [98.138.91.66]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 7AF232380 for ; Fri, 26 Jul 2013 22:14:19 +0000 (UTC) Received: from [98.138.101.129] by nm7.bullet.mail.ne1.yahoo.com with NNFMP; 26 Jul 2013 22:14:18 -0000 Received: from [98.138.89.240] by tm17.bullet.mail.ne1.yahoo.com with NNFMP; 26 Jul 2013 22:14:18 -0000 Received: from [127.0.0.1] by omp1013.mail.ne1.yahoo.com with NNFMP; 26 Jul 2013 22:14:18 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 999721.17298.bm@omp1013.mail.ne1.yahoo.com Received: (qmail 43147 invoked by uid 60001); 26 Jul 2013 22:14:17 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1374876857; bh=Fn7U9xa/nf7rJ1PA7qEZie1WLiWJ4BhOv8CfGoj0l0g=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=ozKZRm+Eg21Z487jUmjIKfkvssT78fl9qA/G3BeHxKHnCCyoAM1EWPcmcrEjYtlXHlLEwSbTLXy1lngN6mHhghRRPFGDmL/ERBN7j9OY+yne1G46lneQxB1sUMKGlztoAz2prdGfD/M0SGc5rIErzue6olmlVhwqJzYXj0bMawc= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=s0R0XSM00cmykEZhDyHpLW34Xt8OCTMKNOG3+RVYViO5EB5S0vOswymzDx03kw2XjssX6GJT4Lf77I02tsBz62kG2D6fCrsj76/U+lv5WVm8/hQXFelkw9EC3Xr3Ft9hAB8r/RUb1lbiyMgs2cr6amF+TX3tkAFgRKS+oBnP3xw=; X-YMail-OSG: QyQqr8oVM1lz8.dAEktyq6OCVDD53D5XT81tDknRaP9eO2s ExzwsDx.6FREC1Hjn7MIZCrXmoSjX.Y6CkdHsLn4tdoMGEuLTIVELpOXBO9I 2WH1YN_6.t3juNosUIhJ8CYCW59YzrnYXhLAXgHyUaxAMDeSlnKI9PhBmdUE m28i3dAvfvAzFBSGYHGwb_EXF6quuVvb4wuuU3UkJM1rwkhOei6DZ3CRExKP otAr.rXhG6lN72N6nhNEYKJnJFnRmk5id2Od.dyCKI916BUlNonCDCrkw8nu 6OZBETMuYD9YNONDR3Lf_u8_h6qTn9KcaHletbZSEcGS5rQDaMQHaxkJQdIB Gyt9HpAcAYsk7ZbyFO0gFlPb2Ak2_prvxTxZTJ7jZjNMOn7ujbOneSplCJug 8V3Wf.XQ2aoLt4NQ4WoL5SEg7SMUSozPrJVuxRqnJgKPENce71jAvA0WMMlR JfQF1Cb394fcx8XKYFZoLVKUsMon7gghkIQEsMMhQUA.Lg2jLD25Zw3LqJNn Zb.g3uB_MADPRRa_XrEPg1yAF0xBa_R87LJ.ZrWcpbAMZQKqLkPlVc42ZKpU _6TkSy1KscdX_g3k- Received: from [98.203.118.124] by web121603.mail.ne1.yahoo.com via HTTP; Fri, 26 Jul 2013 15:14:17 PDT X-Rocket-MIMEInfo: 002.001, CgoKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCiBGcm9tOiBEYW5pZWwgRmVlbmJlcmcgPGZlZW5iZXJnQG5iZXIub3JnPgpUbzogQWxleGFuZGVyIFYuIENoZXJuaWtvdiA8bWVsaWZhcm9ARnJlZUJTRC5vcmc.IApDYzogQmFybmV5IENvcmRvYmEgPGJhcm5leV9jb3Jkb2JhQHlhaG9vLmNvbT47ICJmcmVlYnNkLW5ldEBmcmVlYnNkLm9yZyIgPGZyZWVic2QtbmV0QGZyZWVic2Qub3JnPiAKU2VudDogRnJpZGF5LCBKdWx5IDI2LCAyMDEzIDQ6NTkgUE0KU3ViamVjdDogUmU6IFJlY29tbWVuZGEBMAEBAQE- X-Mailer: YahooMailWebService/0.8.150.561 References: <51F0386D.2000709@acm.poly.edu> <51F16A07.9030505@FreeBSD.org> <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> <51F2A313.9070105@FreeBSD.org> Message-ID: <1374876857.42890.YahooMailNeo@web121603.mail.ne1.yahoo.com> Date: Fri, 26 Jul 2013 15:14:17 -0700 (PDT) From: Barney Cordoba Subject: Re: Recommendations for 10gbps NIC To: Daniel Feenberg , "Alexander V. Chernikov" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Barney Cordoba List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 22:14:19 -0000 =0A=0A=0A=0A________________________________=0A From: Daniel Feenberg =0ATo: Alexander V. Chernikov =0ACc: B= arney Cordoba ; "freebsd-net@freebsd.org" =0ASent: Friday, July 26, 2013 4:59 PM=0ASubject: Re: Re= commendations for 10gbps NIC=0A =0A=0A=0AOn Fri, 26 Jul 2013, Alexander V. = Chernikov wrote:=0A=0A> On 26.07.2013 19:30, Barney Cordoba wrote:=0A>> =0A= >> =0A>> ------------------------------------------------------------------= ------=0A>> *From:* Alexander V. Chernikov =0A>> *To:= * Boris Kochergin =0A>> *Cc:* freebsd-net@freebsd.org= =0A>> *Sent:* Thursday, July 25, 2013 2:10 PM=0A>> *Subject:* Re: Recommend= ations for 10gbps NIC=0A>> =0A>> On 25.07.2013 00:26, Boris Kochergin wrote= :=0A>> > Hi.=0A>> Hello.=0A>> >=0A>> > I am looking for recommendations for= a 10gbps NIC from someone who has=0A>> > successfully used it on FreeBSD. = It will be used on FreeBSD 9.1-R/amd64=0A>> > to capture packets. Some desi= red features are:=0A>> >=0A=0AWe have experience with HP NC523SFP and Chels= io N320E. The key difference =0Aamong 10GBE cards for us is how they treat = foreign DACs. The HP would PXE =0Aboot with several brands and generic DACs= , but the Chelsio required a =0AChelsio brand DAC to PXE boot.=A0 There was= firmware on the NIC to check the =0Abrand of cable. Both worked fine once = booted. The Chelsio cables were hard =0Ato find, which became a problem. Al= so, when used with diskless Unix =0Aclients the Chelsio cards seemed to han= g from time to time. Otherwise =0Apacket loss was one in a million for both= cards, even with 7 meter cables.=0A=0AWe liked the fact that the Chelsio c= ards were single-port and cheaper. I =0Adon't really understand why nearly = all 10GBE cards are dual-port. Surely =0Athere is a market for NICs between= 1 gigabit and 20 gigabit.=0A=0AThe NIC heatsinks are too hot to touch duri= ng use unless specially cooled.=0A=0ADaniel Feenberg=0ANBER=0A=0A=0A=0A----= -----------------=0AThe same reason that they don't make single core cpus a= nymore. It costs about the=0Asame to make a 1 port chip as a 2 port chip.= =0A=0AI find it interesting =A0how so many talk about "the cards", when mos= t often the=0Adifferences are with "the drivers". Luigi made the most usefu= l comment; if you ever=0Awant to use netmap, =A0you need to buy a card comp= atible with netmap. Although=0Ayou don't =A0need netmap just to capture 10G= b/s. Forwarding, Maybe.=A0=0A=0AI also find it interesting that nobody seem= s to have a handle on the performance=0Adifferences. Obviously they're all = different. Maybe substantially different.=0A=0AThe x540 with RJ45 has the o= bvious advantage of being compatible with regular gigabit cards,=A0=0Aand s= ingle port adapters are about $325 in the US.=A0=0A=0AWhen cheap(er) 10g RJ= 45 switches become available, it will start to be used more and more.=0AVer= y soon.=0A=0ABC=0A From owner-freebsd-net@FreeBSD.ORG Sat Jul 27 08:02:40 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id B36AA2FA for ; Sat, 27 Jul 2013 08:02:40 +0000 (UTC) (envelope-from melifaro@FreeBSD.org) Received: from mail.ipfw.ru (unknown [IPv6:2a01:4f8:120:6141::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 742A92BAC for ; Sat, 27 Jul 2013 08:02:40 +0000 (UTC) Received: from v6.mpls.in ([2a02:978:2::5] helo=ws.su29.net) by mail.ipfw.ru with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.76 (FreeBSD)) (envelope-from ) id 1V2zVc-000CyT-8n; Sat, 27 Jul 2013 12:05:48 +0400 Message-ID: <51F37E97.3090203@FreeBSD.org> Date: Sat, 27 Jul 2013 12:02:31 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20120121 Thunderbird/9.0 MIME-Version: 1.0 To: Barney Cordoba Subject: Re: Recommendations for 10gbps NIC References: <51F0386D.2000709@acm.poly.edu> <51F16A07.9030505@FreeBSD.org> <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> <51F2A313.9070105@FreeBSD.org> <1374876857.42890.YahooMailNeo@web121603.mail.ne1.yahoo.com> In-Reply-To: <1374876857.42890.YahooMailNeo@web121603.mail.ne1.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-net@freebsd.org" , Daniel Feenberg X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jul 2013 08:02:40 -0000 On 27.07.2013 02:14, Barney Cordoba wrote: > > > ------------------------------------------------------------------------ > *From:* Daniel Feenberg > *To:* Alexander V. Chernikov > *Cc:* Barney Cordoba ; > "freebsd-net@freebsd.org" > *Sent:* Friday, July 26, 2013 4:59 PM > *Subject:* Re: Recommendations for 10gbps NIC > > > On Fri, 26 Jul 2013, Alexander V. Chernikov wrote: > > > On 26.07.2013 19:30, Barney Cordoba wrote: > >> > >> > >> ------------------------------------------------------------------------ > >> *From:* Alexander V. Chernikov > > >> *To:* Boris Kochergin > > >> *Cc:* freebsd-net@freebsd.org > >> *Sent:* Thursday, July 25, 2013 2:10 PM > >> *Subject:* Re: Recommendations for 10gbps NIC > >> > >> On 25.07.2013 00:26, Boris Kochergin wrote: > >> > Hi. > >> Hello. > >> > > >> > I am looking for recommendations for a 10gbps NIC from someone who has > >> > successfully used it on FreeBSD. It will be used on FreeBSD > 9.1-R/amd64 > >> > to capture packets. Some desired features are: > >> > > > We have experience with HP NC523SFP and Chelsio N320E. The key difference > among 10GBE cards for us is how they treat foreign DACs. The HP would PXE > boot with several brands and generic DACs, but the Chelsio required a > Chelsio brand DAC to PXE boot. There was firmware on the NIC to check the > brand of cable. Both worked fine once booted. The Chelsio cables were hard > to find, which became a problem. Also, when used with diskless Unix > clients the Chelsio cards seemed to hang from time to time. Otherwise > packet loss was one in a million for both cards, even with 7 meter cables. > > We liked the fact that the Chelsio cards were single-port and cheaper. I > don't really understand why nearly all 10GBE cards are dual-port. Surely > there is a market for NICs between 1 gigabit and 20 gigabit. > > The NIC heatsinks are too hot to touch during use unless specially cooled. > > Daniel Feenberg > NBER > > > --------------------- > The same reason that they don't make single core cpus anymore. It costs > about the > same to make a 1 port chip as a 2 port chip. > > I find it interesting how so many talk about "the cards", when most > often the > differences are with "the drivers". Luigi made the most useful comment; > if you ever > want to use netmap, you need to buy a card compatible with netmap. Although > you don't need netmap just to capture 10Gb/s. Forwarding, Maybe. > > I also find it interesting that nobody seems to have a handle on the > performance > differences. Obviously they're all different. Maybe substantially different. It depends on what kind of performance you are talking about. All NICs are capable of doing linerate RX/TX for both small/big packets. The only notable exception I;m aware of are Intel 82598-based NICs which advertise PCI-E X8 gen2 with _2.5GT_ link speed, giving you maximum ~14Gbit/s bw for 2 ports instead of 20. > > The x540 with RJ45 has the obvious advantage of being compatible with > regular gigabit cards, > and single port adapters are about $325 in the US. > > When cheap(er) 10g RJ45 switches become available, it will start to be > used more and more. > Very soon. > > BC >