From owner-freebsd-stable@FreeBSD.ORG Fri Apr 8 13:52:49 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EFEF4106564A for ; Fri, 8 Apr 2011 13:52:49 +0000 (UTC) (envelope-from telbizov@gmail.com) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id A7F358FC15 for ; Fri, 8 Apr 2011 13:52:49 +0000 (UTC) Received: by qwc9 with SMTP id 9so2327076qwc.13 for ; Fri, 08 Apr 2011 06:52:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=H0Ml+G0CPPPjtg6LW43y48+03PgCbXytxfpdZGUzMwk=; b=OGpWAzUfUlXBNB10oXdSVSV0pcklv0QxVzpdHeJxrIN4nRUOVRlvcg2JJMqo0aIwU7 mK6vd+xQawNiCL6B7cPJoqhdr6i8sflW4egwsu35LvVaGQSfLGU5ijJcUaDdlzQLzwE8 S0TW2lU1JfcemLmTLH4wV2iDdOEZPITg6ITfs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=CIBePegDPelBzWDJmJZx1QcCKPdfScteFwwNphqNg1maEyB528EGi+5+JtVtGAlE5P EeYGindP2fi/ebgaC2iomuXSCZYtEd10ZecTQzlZN7w54jtIdyb+Xv8f8tyZxwi9DN/y 2kNgUlVaZNJRsxHBhhv/Itt7DVvKGpu/5j3WQ= MIME-Version: 1.0 Received: by 10.229.141.71 with SMTP id l7mr1845107qcu.44.1302270768506; Fri, 08 Apr 2011 06:52:48 -0700 (PDT) Received: by 10.229.36.147 with HTTP; Fri, 8 Apr 2011 06:52:48 -0700 (PDT) In-Reply-To: <1302258590.3109.91.camel@pcdenny> References: <1302258590.3109.91.camel@pcdenny> Date: Fri, 8 Apr 2011 06:52:48 -0700 Message-ID: From: Rumen Telbizov To: Denny Schierz Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-stable Subject: Re: FreeBSD-8.2: Channel Bounding: LACP or Roundrobin? With Cisco Catalyst X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Apr 2011 13:52:50 -0000 Denny, Since LACP uses hashing to determine which channel to send the packet to the traffic between two nodes (ip:mac ip:mac) will always get bound to only one of the two channels. I am using HP procurve's here and they do seem to hash by ip too although I can't see/tweak that as in catalyst. If you can add the IP address in the hashing function of the switch then you have a better chance of achieving what you're trying to by bringing an IP alias (might have to try a few until you find one that maps to the opposite channel) and then having two iperfs between different IP addresses. In the real live scenario it will be equivalent to having multiple iscsi targets between the two hosts on top of different IPs if this works for you. As for the load balancing option - my HP 2910al's seemed to get CPU overloaded when I push a gigabit of traffic with this option. My guess is the crazy mac address change might be exhausting the cpu much faster. I'd be interested to see how this affects catalyst - let me know. Good luck, Rumen Telbizov On Fri, Apr 8, 2011 at 3:29 AM, Denny Schierz wrote: > hi, > > I want to bound two e1000 (1Gb/s) channels and use at the moment LCAP, > but the max throughput is slower, than without channel bounding. I've > got round about 70MB/s instead of > 150MB/s - 200MB/s. > > I used iperf with standard options: > > :~$ iperf -f M -c 1.2.3.4 > ------------------------------------------------------------ > Client connecting to 1.2.3.4, TCP port 5001 > TCP window size: 0.02 MByte (default) > ------------------------------------------------------------ > [ 3] local 1.2.3.5 port 58637 connected with 1.2.3.4 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 705 MBytes 70.5 MBytes/sec > > If a second PC do the same, than my 70MB/s splittet into ~30MB/s and > ~40MB/s > > config: > > root@iscsihead-m:~# ifconfig lagg0 > lagg0: flags=8843 metric 0 mtu > 1500 > > > options=219b > ether 00:15:17:f1:5d:5f > inet6 fe80::215:17ff:fef1:5d5f%lagg0 prefixlen 64 scopeid 0x5 > inet 1.2.3.4 netmask 0xffffffc0 broadcast 1.2.3.255 > nd6 options=3 > media: Ethernet autoselect > status: active > laggproto lacp > laggport: em1 flags=1c > laggport: em0 flags=1c > > > Config from the Cisco: > > > cisco#sh run int po3 > Building configuration... > > Current configuration : 119 bytes > ! > interface Port-channel3 > description iscsi-test > switchport > switchport access vlan 111 > switchport mode access > end > > > > #sh etherchannel summary > Flags: D - down P - bundled in port-channel > I - stand-alone s - suspended > R - Layer3 S - Layer2 > U - in use f - failed to allocate aggregator > > M - not in use, minimum links not met > u - unsuitable for bundling > w - waiting to be aggregated > d - default port > > > Number of channel-groups in use: 3 > Number of aggregators: 3 > > Group Port-channel Protocol Ports > > ------+-------------+-----------+----------------------------------------------- > 1 Po1(SU) - Gi1/1(P) Gi1/2(P) > 2 Po2(SU) - Gi6/17(P) Gi6/18(P) Gi6/19(P) > 3 Po3(SU) LACP Gi5/41(P) Gi5/44(P) > > cisco#sh run int gi5/41 > Building configuration... > > Current configuration : 182 bytes > ! > interface GigabitEthernet5/41 > description iscsi-head1 > switchport access vlan 111 > switchport mode access > no cdp enable > channel-group 3 mode active > spanning-tree portfast > end > > > > #sh run int gi5/44 > Building configuration... > > Current configuration : 183 bytes > ! > interface GigabitEthernet5/44 > description GRAU1_iscsi2 > switchport access vlan 111 > switchport mode access > no cdp enable > channel-group 3 mode active > spanning-tree portfast > end > > > #sh etherchannel load-balance > EtherChannel Load-Balancing Configuration: > src-dst-mac > > EtherChannel Load-Balancing Addresses Used Per-Protocol: > Non-IP: Source XOR Destination MAC address > IPv4: Source XOR Destination MAC address > IPv6: Source XOR Destination MAC address > > > What I saw with tcpdump: it seems, that only one device is used. > > Maybe, Cisco uses mac and FreeBSD IP ? > > Any suggestions? > > -- Rumen Telbizov http://telbizov.com