From owner-freebsd-net@FreeBSD.ORG Fri Apr 25 11:48:59 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E5FFCEB6 for ; Fri, 25 Apr 2014 11:48:58 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 70AA7177B for ; Fri, 25 Apr 2014 11:48:57 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqUEAIVKWlODaFve/2dsb2JhbABZg1VXgmW6AoZnUYEldIIlAQEBAwEBAQEgKyALBRYYAgINGQIpAQkmBggCBQQBHASIGAgNpw2jTReBKYxfAQEbNAeCb4FKBJYrhBKRJINNITGBBDk X-IronPort-AV: E=Sophos;i="4.97,926,1389762000"; d="scan'208";a="117547210" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 25 Apr 2014 07:48:50 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 470CBB40A2; Fri, 25 Apr 2014 07:48:50 -0400 (EDT) Date: Fri, 25 Apr 2014 07:48:50 -0400 (EDT) From: Rick Macklem To: Marek Salwerowicz Message-ID: <562289962.736994.1398426530226.JavaMail.root@uoguelph.ca> In-Reply-To: <535A1354.2040309@wp.pl> Subject: Re: NFS over LAGG / lacp poor performance MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Apr 2014 11:48:59 -0000 Marek Salwerowicz wrote: > Hi list, > > I have two FreeBSD boxes (both based on SuperMicro X9DRD-7LN4F-JBOD > motherboard, with 32GB RAM, 1 CPU :Intel(R) Xeon(R) CPU E5-2640 v2) > > storage1% uname -a > FreeBSD storage1 9.1-RELEASE-p10 FreeBSD 9.1-RELEASE-p10 #0: Sun Jan > 12 > 20:11:23 UTC 2014 > root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC > amd64 > > > storage2% uname > -a > FreeBSD storage2 10.0-RELEASE-p1 FreeBSD 10.0-RELEASE-p1 #0: Tue Apr > 8 > 06:45:06 UTC 2014 > root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC > amd64 > > > > They work as NFS storages for VMware ESXi. > > On both there are installed 4 igb 1Gbit NICs, with LACP and 2 VLANs > (vlan900 is for NFS, vlan14 is for management/general purpose): > > on storage1: > igb0: port > 0xd060-0xd07f mem 0xfbb60000-0xfbb7ffff,0xfbb8c000-0xfbb8ffff irq 42 > at > device 0.0 on pci5 > igb1: port > 0xd040-0xd05f mem 0xfbb40000-0xfbb5ffff,0xfbb88000-0xfbb8bfff irq 45 > at > device 0.1 on pci5 > igb2: port > 0xd020-0xd03f mem 0xfbb20000-0xfbb3ffff,0xfbb84000-0xfbb87fff irq 44 > at > device 0.2 on pci5 > igb3: port > 0xd000-0xd01f mem 0xfbb00000-0xfbb1ffff,0xfbb80000-0xfbb83fff irq 46 > at > device 0.3 on pci5 > > on storage2: > igb0: port > 0xd060-0xd07f mem 0xfbb60000-0xfbb7ffff,0xfbb8c000-0xfbb8ffff irq 42 > at > device 0.0 on pci5 > igb1: port > 0xd040-0xd05f mem 0xfbb40000-0xfbb5ffff,0xfbb88000-0xfbb8bfff irq 45 > at > device 0.1 on pci5 > igb2: port > 0xd020-0xd03f mem 0xfbb20000-0xfbb3ffff,0xfbb84000-0xfbb87fff irq 44 > at > device 0.2 on pci5 > igb3: port > 0xd000-0xd01f mem 0xfbb00000-0xfbb1ffff,0xfbb80000-0xfbb83fff irq 46 > at > device 0.3 on pci5 > > > storage1% ifconfig -a > igb0: flags=8843 metric 0 mtu > 9000 > > options=401bb > ether 00:25:90:c1:1d:18 > inet6 fe80::225:90ff:fec1:1d18%igb0 prefixlen 64 scopeid 0x1 > nd6 options=29 > media: Ethernet autoselect (1000baseT ) > status: active > igb1: flags=8843 metric 0 mtu > 9000 > > options=401bb > ether 00:25:90:c1:1d:18 > inet6 fe80::225:90ff:fec1:1d19%igb1 prefixlen 64 scopeid 0x2 > nd6 options=29 > media: Ethernet autoselect (1000baseT ) > status: active > igb2: flags=8843 metric 0 mtu > 9000 > > options=401bb > ether 00:25:90:c1:1d:18 > inet6 fe80::225:90ff:fec1:1d1a%igb2 prefixlen 64 scopeid 0x3 > nd6 options=29 > media: Ethernet autoselect (1000baseT ) > status: active > igb3: flags=8843 metric 0 mtu > 9000 > > options=401bb > ether 00:25:90:c1:1d:18 > inet6 fe80::225:90ff:fec1:1d1b%igb3 prefixlen 64 scopeid 0x4 > nd6 options=29 > media: Ethernet autoselect (1000baseT ) > status: active > lo0: flags=8049 metric 0 mtu 16384 > options=600003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7 > inet 127.0.0.1 netmask 0xff000000 > nd6 options=21 > lagg0: flags=8843 metric 0 > mtu 9000 > > options=401bb > ether 00:25:90:c1:1d:18 > inet6 fe80::225:90ff:fec1:1d18%lagg0 prefixlen 64 scopeid 0x8 > nd6 options=21 > media: Ethernet autoselect > status: active > laggproto lacp lagghash l2,l3,l4 > laggport: igb3 flags=1c > laggport: igb2 flags=1c > laggport: igb1 flags=1c > laggport: igb0 flags=1c > vlan14: flags=8843 metric 0 > mtu 9000 > options=103 > ether 00:25:90:c1:1d:18 > inet 192.168.1.65 netmask 0xffffff00 broadcast 192.168.1.255 > inet6 fe80::225:90ff:fec1:1d18%vlan14 prefixlen 64 scopeid > 0x9 > nd6 options=29 > media: Ethernet autoselect > status: active > vlan: 14 parent interface: lagg0 > vlan900: flags=8843 metric 0 > mtu > 9000 > options=103 > ether 00:25:90:c1:1d:18 > inet 172.25.25.65 netmask 0xffffff00 broadcast 172.25.25.255 > inet6 fe80::225:90ff:fec1:1d18%vlan900 prefixlen 64 scopeid > 0xa > nd6 options=29 > media: Ethernet autoselect > status: active > vlan: 900 parent interface: lagg0 > > > storage2% ifconfig > -a > igb0: flags=8843 metric 0 mtu > 9000 > > options=403bb > ether 00:25:90:ca:3b:e0 > inet6 fe80::225:90ff:feca:3be0%igb0 prefixlen 64 scopeid 0x1 > nd6 options=29 > media: Ethernet autoselect (1000baseT ) > status: active > igb1: flags=8843 metric 0 mtu > 9000 > > options=403bb > ether 00:25:90:ca:3b:e0 > inet6 fe80::225:90ff:feca:3be1%igb1 prefixlen 64 scopeid 0x2 > nd6 options=29 > media: Ethernet autoselect (1000baseT ) > status: active > igb2: flags=8843 metric 0 mtu > 9000 > > options=403bb > ether 00:25:90:ca:3b:e0 > inet6 fe80::225:90ff:feca:3be2%igb2 prefixlen 64 scopeid 0x3 > nd6 options=29 > media: Ethernet autoselect (1000baseT ) > status: active > igb3: flags=8843 metric 0 mtu > 9000 > > options=403bb > ether 00:25:90:ca:3b:e0 > inet6 fe80::225:90ff:feca:3be3%igb3 prefixlen 64 scopeid 0x4 > nd6 options=29 > media: Ethernet autoselect (1000baseT ) > status: active > lo0: flags=8049 metric 0 mtu 16384 > options=600003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 > inet 127.0.0.1 netmask 0xff000000 > nd6 options=21 > lagg0: flags=8843 metric 0 > mtu 9000 > > options=403bb > ether 00:25:90:ca:3b:e0 > inet6 fe80::225:90ff:feca:3be0%lagg0 prefixlen 64 scopeid 0x6 > nd6 options=21 > media: Ethernet autoselect > status: active > laggproto lacp lagghash l2,l3,l4 > laggport: igb3 flags=1c > laggport: igb2 flags=1c > laggport: igb1 flags=1c > laggport: igb0 flags=1c > vlan14: flags=8843 metric 0 > mtu 9000 > options=303 > ether 00:25:90:ca:3b:e0 > inet 192.168.1.72 netmask 0xffffff00 broadcast 192.168.1.255 > inet6 fe80::225:90ff:feca:3be0%vlan14 prefixlen 64 scopeid > 0x7 > nd6 options=29 > media: Ethernet autoselect > status: active > vlan: 14 parent interface: lagg0 > vlan900: flags=8843 metric 0 > mtu > 9000 > options=303 > ether 00:25:90:ca:3b:e0 > inet 172.25.25.72 netmask 0xffffff00 broadcast 172.25.25.255 > inet6 fe80::225:90ff:feca:3be0%vlan900 prefixlen 64 scopeid > 0x8 > nd6 options=29 > media: Ethernet autoselect > status: active > vlan: 900 parent interface: lagg0 > > > Both boxes are connected to the same switch (HP 1910-48G) > > I need to transfer around 10 TB of data from storage1 to storage2 > I obeserve that during copying, only one NIC (instead of all 4) is > used. > Well, you don't mention what command(s) you are using to transfer the data, but I would guess you have one serial data transfer for each command. (Put another way, if you are only running one command to transfer the data, there will only be one RPC happening at a time and that will only use one network interface.) I don't know anything about lagg, so I can't comment related to it, but if there is only one NFS RPC at a time, you'll only be transferring one message at a time on the wire.) Adding the mount option "readahead=8" to the machine receiving the data might help, if the data transfer command is being done there. (ie. The machine the data is being copied to has the other one NFS mounted and it is where you are running the data transfer command(s).) If you can run commands concurrently to transfer parts of the data, that might help, too. Good luck with it, rick > Boxes are not stressed during copying > > What's more, apart from having 1 NIC saturated (transfer around 120 > MB/s), I observe transfer rate on level of 70-80 MB/s > > I haven't changed any kernel configuration, nor sysctl's - am I > missing > sth ? > > Regards, > Marek > > -- > Marek Salwerowicz > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" >