From owner-freebsd-net@FreeBSD.ORG Fri Jul 26 15:31:05 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0F972228 for ; Fri, 26 Jul 2013 15:31:05 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm2-vm5.bullet.mail.ne1.yahoo.com (nm2-vm5.bullet.mail.ne1.yahoo.com [98.138.91.224]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BD63421F9 for ; Fri, 26 Jul 2013 15:31:04 +0000 (UTC) Received: from [98.138.226.179] by nm2.bullet.mail.ne1.yahoo.com with NNFMP; 26 Jul 2013 15:30:58 -0000 Received: from [98.138.89.170] by tm14.bullet.mail.ne1.yahoo.com with NNFMP; 26 Jul 2013 15:30:58 -0000 Received: from [127.0.0.1] by omp1026.mail.ne1.yahoo.com with NNFMP; 26 Jul 2013 15:30:58 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 455354.40765.bm@omp1026.mail.ne1.yahoo.com Received: (qmail 90245 invoked by uid 60001); 26 Jul 2013 15:30:58 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1374852658; bh=Xr0QMD8agYb8yGuC+h/bsYcrnAGJQdudxcQfZqgbYUY=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=c6ToKezncXSvpc/k2rD6fwvmQnbJcc/yXEAlmwrPBeeRw2Q3heBvyPpI2vX1FhGG9Q9iYw5s5a5oDrJrThBS+XKJq3rWn2lhu0vRADQwhNhbIoQWereMjGTkb+70wQ8Neawj1nb28gkZ1KqD8O0tzpwE/FUp3sI9AJfqc5dvFqM= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=2bNU1flqby28BahFBQL887XYp9YHENcOjI4BcYD6ERKsi6F2pIJhpK6Y46NVN0LvLKdTcKwRjO5Fgqgb6gzE69/23+n64A5C8Ta7stPPFsk6byCKwP+ZMXqvbtfUIie48juL9YQJwQPybInTCq25WDv27BjbKUbOH2zycn9ajNk=; X-YMail-OSG: hQ6kJC4VM1nzhCJAc5683Ns6097dDVDpic1fYkUk8YrgfkM JkLG0PUCV.cPRz0tuQJrnVkjWR1Gl1eCLqpO8KapuPxL2i97QmYRnZuAYS1x DB92xZpdOfesacyNGWZ5MVOwuoe8IZAf19v2W1XV7TKa9Sa65ISnG.j9YP.X ISvfYn6NfWdEeAAGgPzJhVKpVjM.u6Hha8_tMI82TKI2uQPzEUsz73ZF9zSW gDwUnAhkS9cBul0C5xs645XqeLCdT2mUADSTmLPvnLI3yHXPTabuXxOIxNcL h4hUDsUyxcwiidPrBYaDQVlvLBIw8DVpQPFoRzgzt1apQdEBoGh148jx1QfC .29Dle0gGSGGty6RLoMpzO5li1u8Tq1aLuBh1WMQwomRRtvld4av6BWMwhPo BS9hfxHWfSIdHNrqMy3YlIlrzSiOhSm5Oa44DR0PUys1n_h0IJXDUukK58iX v0DD0D3GSaFjMq2b1dr26L.jZovBPH3FNkEiWURaBlYTGMVa4unLo7f4cM23 j6mz12ogIhc6BtYXI3GNgA0sQuwC2toiVeuk9KHLjlgB7GF48AeUlDMDydne QInp8r.j1mu4PqEia34QPcQ-- Received: from [98.203.118.124] by web121601.mail.ne1.yahoo.com via HTTP; Fri, 26 Jul 2013 08:30:58 PDT X-Rocket-MIMEInfo: 002.001, CgoKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCiBGcm9tOiBBbGV4YW5kZXIgVi4gQ2hlcm5pa292IDxtZWxpZmFyb0BGcmVlQlNELm9yZz4KVG86IEJvcmlzIEtvY2hlcmdpbiA8c3Bhd2tAYWNtLnBvbHkuZWR1PiAKQ2M6IGZyZWVic2QtbmV0QGZyZWVic2Qub3JnIApTZW50OiBUaHVyc2RheSwgSnVseSAyNSwgMjAxMyAyOjEwIFBNClN1YmplY3Q6IFJlOiBSZWNvbW1lbmRhdGlvbnMgZm9yIDEwZ2JwcyBOSUMKIAoKT24gMjUuMDcuMjAxMyAwMDoyNiwgQm9yaXMgS29jaGVyZ2luIHdyb3RlOgoBMAEBAQE- X-Mailer: YahooMailWebService/0.8.150.561 References: <51F0386D.2000709@acm.poly.edu> <51F16A07.9030505@FreeBSD.org> Message-ID: <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> Date: Fri, 26 Jul 2013 08:30:58 -0700 (PDT) From: Barney Cordoba Subject: Re: Recommendations for 10gbps NIC To: "Alexander V. Chernikov" In-Reply-To: <51F16A07.9030505@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Barney Cordoba List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 15:31:05 -0000 ________________________________ From: Alexander V. Chernikov To: Boris Kochergin Cc: freebsd-net@freebsd.org Sent: Thursday, July 25, 2013 2:10 PM Subject: Re: Recommendations for 10gbps NIC On 25.07.2013 00:26, Boris Kochergin wrote: > Hi. Hello. > > I am looking for recommendations for a 10gbps NIC from someone who has > successfully used it on FreeBSD. It will be used on FreeBSD 9.1-R/amd64 > to capture packets. Some desired features are: > > - PCIe > - LC connectors > - 10GBASE-SR > - Either single- or dual-port > - Multiqueue Intel 82598/99/X520 Emulex OCe10102-NM Mellanox ConnectX Chelsio T4 Do they all cost the same, have the exact same features and have equally well-written drivers? Which do you recommend and why? BC From owner-freebsd-net@FreeBSD.ORG Fri Jul 26 16:26:54 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id F19DD36D for ; Fri, 26 Jul 2013 16:26:54 +0000 (UTC) (envelope-from melifaro@FreeBSD.org) Received: from mail.ipfw.ru (unknown [IPv6:2a01:4f8:120:6141::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 882DD24F4 for ; Fri, 26 Jul 2013 16:26:54 +0000 (UTC) Received: from dhcp170-36-red.yandex.net ([95.108.170.36]) by mail.ipfw.ru with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.76 (FreeBSD)) (envelope-from ) id 1V2ku4-0005aC-5Z; Fri, 26 Jul 2013 20:30:04 +0400 Message-ID: <51F2A313.9070105@FreeBSD.org> Date: Fri, 26 Jul 2013 20:25:55 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130418 Thunderbird/17.0.5 MIME-Version: 1.0 To: Barney Cordoba Subject: Re: Recommendations for 10gbps NIC References: <51F0386D.2000709@acm.poly.edu> <51F16A07.9030505@FreeBSD.org> <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> In-Reply-To: <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 16:26:55 -0000 On 26.07.2013 19:30, Barney Cordoba wrote: > > > ------------------------------------------------------------------------ > *From:* Alexander V. Chernikov > *To:* Boris Kochergin > *Cc:* freebsd-net@freebsd.org > *Sent:* Thursday, July 25, 2013 2:10 PM > *Subject:* Re: Recommendations for 10gbps NIC > > On 25.07.2013 00:26, Boris Kochergin wrote: > > Hi. > Hello. > > > > I am looking for recommendations for a 10gbps NIC from someone who has > > successfully used it on FreeBSD. It will be used on FreeBSD 9.1-R/amd64 > > to capture packets. Some desired features are: > > > > - PCIe > > - LC connectors > > - 10GBASE-SR > > - Either single- or dual-port > > - Multiqueue > Intel 82598/99/X520 > Emulex OCe10102-NM > Mellanox ConnectX > Chelsio T4 > > Do they all cost the same, have the exact same features and have > equally well-written drivers? Which do you recommend > and why? Well, Intel/Chelsio/Mellanox costs $500-600 / 2 ports, Emulex costs a bit higher (of course, YMMV) Regarding features: it depends on your needs mostly (e.g. forwarding / TCP server / IB ). for forwarding: Each one is capable of doing at least 8 rx/tx queues per port with adjustable rx/tx ring size. One important thing to note is that netmap is currently not available for any non-Intel NIC. General hw offload features: LRO/TSO4/vlans/jumbo are supported by all of them (not sure about TSO6 in Mellanox) however I can't say how good it works. SFP+/Twinax: Intel was very restrictive on supported spf/cables (however, Jack changed this a moth ago). Chelsio is much better here. Advanced features: Intel (82599-based NICs) has hw firewall called FlowDirector which can be programmed to redirect particular IPv4/IPv6 TCP/UDP streams to given queue (or dropped). However it is currently not supported on FreeBSD. (Intel also has IPSEC offload functionality, which, afaik, is currently not supported) On the other side, Chelsio has similar functionality which is supported via cxgbetool binary (available in tools/) Usability: IMHO Chelsio/Intel are the best (a lot of things are runtime-configurable, lots of counters, etc..) > > BC