From owner-freebsd-performance@FreeBSD.ORG Mon Feb 28 04:17:10 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C839F16A505; Mon, 28 Feb 2005 04:17:10 +0000 (GMT) Received: from mail.utcorp.net (mail.utcorp.net [146.145.135.97]) by mx1.FreeBSD.org (Postfix) with ESMTP id 185AC43D41; Mon, 28 Feb 2005 04:17:10 +0000 (GMT) (envelope-from lister@primetime.com) Received: from [10.200.1.90] (helo=[10.200.1.90]) by mail.utcorp.net with esmtp (Exim 4.30; FreeBSD) id 1D5cha-000CI9-LG; Sun, 27 Feb 2005 23:40:14 -0500 Message-ID: <4222C64D.4050007@primetime.com> Date: Sun, 27 Feb 2005 23:20:45 -0800 From: Lister User-Agent: Mozilla Thunderbird 1.0 (X11/20050116) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-net@freebsd.org, freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-yoursite-MailScanner-Information: Please contact the ISP for more information X-yoursite-MailScanner: Found to be clean Subject: ng_fec and cisco 2931 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Feb 2005 04:17:10 -0000 I have setup ng_fec on a machine with a quad ethernet NIC : de0: port 0xd000-0xd07f mem 0xe3000000-0xe300007f irq 9 at device 4.0 on pci2 de0: Cogent EM440TX 21140A [10-100Mb/s] pass 2.2 de0: address 00:00:d1:1c:c3:a1 de1: port 0xd400-0xd47f mem 0xe3001000-0xe300107f irq 5 at device 5.0 on pci2 de1: Cogent EM440TX 21140A [10-100Mb/s] pass 2.2 de1: address 00:00:d1:1c:c3:a2 de2: port 0xd800-0xd87f mem 0xe3002000-0xe300207f irq 10 at device 6.0 on pci2 de2: Cogent EM440TX 21140A [10-100Mb/s] pass 2.2 de2: address 00:00:d1:1c:c3:a3 de3: port 0xdc00-0xdc7f mem 0xe3003000-0xe300307f irq 11 at device 7.0 on pci2 de3: Cogent EM440TX 21140A [10-100Mb/s] pass 2.2 de3: address 00:00:d1:1c:c3:a4 Commands are : #!/bin/sh ngctl mkpeer fec dummy fec ngctl msg fec0: add_iface "de0" ngctl msg fec0: add_iface "de1" ngctl msg fec0: add_iface "de2" ngctl msg fec0: add_iface "de3" ngctl msg fec0: set_mode_inet ifconfig de0 promisc ifconfig de1 promisc ifconfig de2 promisc ifconfig de3 promisc ifconfig fec0 promisc So, I dhclient fec0 and I have : > ifconfig fec0 fec0: flags=28943 mtu 1500 inet6 fe80::200:d1ff:fe1c:c3a1%fec0 prefixlen 64 scopeid 0xd inet 10.200.1.205 netmask 0xffffff00 broadcast 10.200.1.255 ether 00:00:d1:1c:c3:a1 media: Ethernet none status: active I have all 4 ports connected to the catalyst. From what I have read on fast etherchannel in the cisco docs, it is supposed to detect the etherchannel, e.g. no commands at the switch. Lights blink on and off, change color (orange -> green) and it seems to work ... but only on one interface in the bundle. No faster than 80Mb. I have a 1000Mb intel card in the 2nd test machine that does run much faster than 100. So, is there something I have done wrong, or what? What should I expect to get from 4 x 100 Mb ports? From owner-freebsd-performance@FreeBSD.ORG Mon Feb 28 16:11:28 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2834416A4CE; Mon, 28 Feb 2005 16:11:28 +0000 (GMT) Received: from ox.eicat.ca (ox.eicat.ca [66.96.30.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id C3DDD43D48; Mon, 28 Feb 2005 16:11:27 +0000 (GMT) (envelope-from dgilbert@daveg.ca) Received: by ox.eicat.ca (Postfix, from userid 66) id DAB5AF222; Mon, 28 Feb 2005 11:11:26 -0500 (EST) Received: by canoe.dclg.ca (Postfix, from userid 101) id B2FB34662; Mon, 28 Feb 2005 11:11:25 -0500 (EST) From: David Gilbert MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16931.17069.667570.224967@canoe.dclg.ca> Date: Mon, 28 Feb 2005 11:11:25 -0500 To: Lister In-Reply-To: <4222C64D.4050007@primetime.com> References: <4222C64D.4050007@primetime.com> X-Mailer: VM 7.17 under 21.4 (patch 17) "Jumbo Shrimp" XEmacs Lucid cc: freebsd-net@freebsd.org cc: freebsd-performance@freebsd.org Subject: ng_fec and cisco 2931 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Feb 2005 16:11:28 -0000 >>>>> "lister" == lister writes: lister> I have setup ng_fec on a machine with a quad ethernet NIC : lister> de0: port 0xd000-0xd07f mem Our own testing with this card (not using fec ... just traffic on the 4 ports) has determined that it appears to have a 100 megabit limit to the total of 4 ports on the card. Now... this could be a FreeBSD driver issue ... or a PCI bus issue, but in all our tests with several motherboards and many versions of FreeBSD (from 3.2 or so through about 4.5) we were never able to achieve more than 100 megabit on the card in total. Our application was an NFS server that had 100's of diskless nodes running from it. We suspected that this could be some interaction between the speed of the disks (and their pci cost) and the card, so we isolated the card by doing straight packet tests (no meaningful data) and still found the card maxxing out at 100 megabit total over the 4 ports. Dave. -- ============================================================================ |David Gilbert, Independent Contractor. | Two things can only be | |Mail: dave@daveg.ca | equal if and only if they | |http://daveg.ca | are precisely opposite. | =========================================================GLO================ From owner-freebsd-performance@FreeBSD.ORG Mon Feb 28 20:17:05 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4CD8216A4CE for ; Mon, 28 Feb 2005 20:17:05 +0000 (GMT) Received: from mail.utcorp.net (mail.utcorp.net [146.145.135.97]) by mx1.FreeBSD.org (Postfix) with ESMTP id B9BC743D1F for ; Mon, 28 Feb 2005 20:17:04 +0000 (GMT) (envelope-from lister@primetime.com) Received: from [10.200.1.90] (helo=[10.200.1.90]) by mail.utcorp.net with esmtp (Exim 4.30; FreeBSD) id 1D5rVy-000HMB-MY for freebsd-performance@freebsd.org; Mon, 28 Feb 2005 15:29:14 -0500 Message-ID: <4223A4B3.7000706@primetime.com> Date: Mon, 28 Feb 2005 15:09:39 -0800 From: Lister User-Agent: Mozilla Thunderbird 1.0 (X11/20050116) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <4222C64D.4050007@primetime.com> <16931.17069.667570.224967@canoe.dclg.ca> In-Reply-To: <16931.17069.667570.224967@canoe.dclg.ca> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-yoursite-MailScanner-Information: Please contact the ISP for more information X-yoursite-MailScanner: Found to be clean Subject: Re: ng_fec and cisco 2931 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Feb 2005 20:17:05 -0000 David Gilbert wrote: >lister> I have setup ng_fec on a machine with a quad ethernet NIC : > >lister> de0: port 0xd000-0xd07f mem > >Our own testing with this card (not using fec ... just traffic on the >4 ports) has determined that it appears to have a 100 megabit limit to >the total of 4 ports on the card. Now... this could be a FreeBSD >driver issue ... or a PCI bus issue, but in all our tests with several >motherboards and many versions of FreeBSD (from 3.2 or so through >about 4.5) we were never able to achieve more than 100 megabit on the >card in total. > >Our application was an NFS server that had 100's of diskless nodes >running from it. We suspected that this could be some interaction >between the speed of the disks (and their pci cost) and the card, so >we isolated the card by doing straight packet tests (no meaningful >data) and still found the card maxxing out at 100 megabit total over >the 4 ports. > >Dave. > > > This might explain why I got it for $50 :) Did you go with another quad that performed better? Some more observations on ng_fec : it _appears_ that it (along with the cisco you plug it into) 'balances' on a 'host per port' basis. It seems geared for a server more than a client. I am trying one2many next ala : http://bsdvault.net/sections.php?op=viewarticle&artid=98 With similar results. I have a 3com (xl0) and a davicom 9102 (dc0) one2many-ed thus : ifconfig xl0 up ifconfig dc0 up kldload /modules/ng_ether.ko ngctl mkpeer xl0: one2many upper one ngctl connect xl0: xl0:upper lower many0 ngctl connect dc0: xl0:upper lower many1 ngctl msg dc0: setpromisc 1 ngctl msg dc0: setautosrc 0 ngctl msg xl0:upper setconfig "{ xmitAlg=1 failAlg=1 enabledLinks=[ 1 1 ] }" I have an NFS mount to a GB NIC equipt machine with a rocketraid 646 controller and a maxtor ata-133 7200 rpm drive on it. I nfs mount a directory and I still can't crack 100mbs ... :\ The card in the server is an intel pro 1000 : em0: port 0x1800-0x183f mem 0xf0000000-0xf001ffff irq 5 at device 9.0 on pci0 em0: Speed:N/A Duplex:N/A Should I enable polling for the server? I am using 4.11 on all. TIA for any thoughts ... From owner-freebsd-performance@FreeBSD.ORG Thu Mar 3 02:47:01 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4BD8D16A4CE for ; Thu, 3 Mar 2005 02:47:01 +0000 (GMT) Received: from mail.utcorp.net (mail.utcorp.net [146.145.135.97]) by mx1.FreeBSD.org (Postfix) with ESMTP id C053643D41 for ; Thu, 3 Mar 2005 02:47:00 +0000 (GMT) (envelope-from lister@primetime.com) Received: from [10.200.1.90] (helo=[10.200.1.90]) by mail.utcorp.net with esmtp (Exim 4.30; FreeBSD) id 1D6gdF-000HFw-GT; Wed, 02 Mar 2005 22:04:09 -0500 Message-ID: <4226A42B.2010601@primetime.com> Date: Wed, 02 Mar 2005 21:44:11 -0800 From: Lister User-Agent: Mozilla Thunderbird 1.0 (X11/20050116) X-Accept-Language: en-us, en MIME-Version: 1.0 To: "Michael G. Jung" , freebsd-performance@freebsd.org References: <9D7F0DF3FB16D41184010050DA90E00001C874BB@neo.confluentasp.local> In-Reply-To: <9D7F0DF3FB16D41184010050DA90E00001C874BB@neo.confluentasp.local> Content-Type: text/plain; charset=Windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-yoursite-MailScanner-Information: Please contact the ISP for more information X-yoursite-MailScanner: Found to be clean Subject: Re: ng_fec and cisco 2931 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Mar 2005 02:47:01 -0000 Michael G. Jung wrote: >Sorry, not subscribed but here go's.... > >To quote cisco > >http://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a0080094646.shtml > >"EtherChannel load balancing can use either MAC addresses or IP addresses. Also, EtherChannel load balancing can use either source or destination, or both source and destination, addresses. The mode that you select applies to all EtherChannels that you have configured on the switch." > >On older switches that I deployed this on, say the 5000 seriers which was based on MAC address only, on a Nx100 channel between two hosts throughput could never exceed "N". So 4x100Mbit links between two hosts would never exceed 100Mbit but multiple connections across the connection would aggregate. > >In other words etherchannel does not load balance per packet across bonded ethernet connections but per connection. Try running tests to multiple hosts from your FreeBSD box and see if you don't aggregate above 100Mbit total.... > > Well, problem #1, as Dave G noted earlier is a crappy quad port NIC, so I dumped that and started testing with 2 singles. >Hope this is helpful. > > It is, thanks. >I'd be very curious to know if you find this is your issue. > > I am going to get my act together, e.g. good cards, document my techniques, graph results. I am also trying one2many, are there other possibilities (than these 2)? >Kind regards, > >--mikej >Michael Jung > > From owner-freebsd-performance@FreeBSD.ORG Wed Mar 2 17:58:07 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6E18C16A4CF for ; Wed, 2 Mar 2005 17:58:07 +0000 (GMT) Received: from electra.nolink.net (electra.nolink.net [195.139.204.207]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2FD5243D5D for ; Wed, 2 Mar 2005 17:58:06 +0000 (GMT) (envelope-from lerik@nolink.net) Received: (qmail 51057 invoked by uid 1000); 2 Mar 2005 17:58:02 -0000 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 2 Mar 2005 17:58:02 -0000 Date: Wed, 2 Mar 2005 18:58:02 +0100 (CET) From: Lars Erik Gullerud To: Lister In-Reply-To: <4222C64D.4050007@primetime.com> Message-ID: <20050302174915.V38850@electra.nolink.net> References: <4222C64D.4050007@primetime.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Mailman-Approved-At: Thu, 03 Mar 2005 13:21:45 +0000 cc: freebsd-net@freebsd.org cc: freebsd-performance@freebsd.org Subject: Re: ng_fec and cisco 2931 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2005 17:58:07 -0000 On Sun, 27 Feb 2005, Lister wrote: > > I have setup ng_fec on a machine with a quad ethernet NIC : [snip] > I have all 4 ports connected to the catalyst. From what I have read on > fast etherchannel in the cisco docs, it is supposed to detect the > etherchannel, > e.g. no commands at the switch. Lights blink on and off, change color > (orange -> green) and it seems to work ... but only on one interface in the > bundle. No faster than 80Mb. I have a 1000Mb intel card in the 2nd > test machine that does run much faster than 100. > So, is there something I have done wrong, or what? What should I expect > to get from 4 x 100 Mb ports? Not really related to FreeBSD's ng_fec at all I think, this is a common FEC issue. If you are testing this between two hosts as you indicate above, then you are getting exactly the speed you should be getting. The low-end Cisco switches offers two varieties of "load-balancing" over the FEC members, source or destination-based hash (both operate on MAC-address level). So if all your test traffic goes between a set of two mac-addresses, traffic in either direction will only flow over one member of the FEC. The higher-end devices like the 6500 series can also look at layer 3 and layer 4 flow-information to distribute the load, but basically a FEC is mostly useful in scenarios where you have a large number of hosts communicating. /leg From owner-freebsd-performance@FreeBSD.ORG Wed Mar 2 22:40:24 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C297F16A4CF; Wed, 2 Mar 2005 22:40:24 +0000 (GMT) Received: from mx2.confluentasp.com (mx2.confluentasp.com [216.26.153.14]) by mx1.FreeBSD.org (Postfix) with ESMTP id 45FE043D2D; Wed, 2 Mar 2005 22:40:19 +0000 (GMT) (envelope-from mikej@confluenttech.com) Received: from neo.confluentasp.local (35.in-addr.arpa.confluentasp.com [216.26.153.35] (may be forged))j22MeIdC070780; Wed, 2 Mar 2005 17:40:18 -0500 (EST) (envelope-from mikej@confluenttech.com) X-MimeOLE: Produced By Microsoft Exchange V6.0.6487.1 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Date: Wed, 2 Mar 2005 17:40:13 -0500 Message-ID: <9D7F0DF3FB16D41184010050DA90E00001C874BB@neo.confluentasp.local> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: ng_fec and cisco 2931 Thread-Index: AcUfeMvO3epRWRJBRrC3DcsS5J5Ujw== From: "Michael G. Jung" To: , X-Mailman-Approved-At: Thu, 03 Mar 2005 13:21:45 +0000 cc: freebsd-net@freebsd.org cc: freebsd-performance@freebsd.org Subject: ng_fec and cisco 2931 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2005 22:40:25 -0000 Sorry, not subscribed but here go's.... To quote cisco http://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a0= 080094646.shtml "EtherChannel load balancing can use either MAC addresses or IP = addresses. Also, EtherChannel load balancing can use either source or = destination, or both source and destination, addresses. The mode that = you select applies to all EtherChannels that you have configured on the = switch." On older switches that I deployed this on, say the 5000 seriers which = was based on MAC address only, on a Nx100 channel between two hosts = throughput could never exceed "N". So 4x100Mbit links between two hosts = would never exceed 100Mbit but multiple connections across the = connection would aggregate.=20 In other words etherchannel does not load balance per packet across = bonded ethernet connections but per connection. Try running tests to = multiple hosts from your FreeBSD box and see if you don't aggregate = above 100Mbit total.... Hope this is helpful. I'd be very curious to know if you find this is your issue. Kind regards, --mikej Michael Jung