From owner-freebsd-net@FreeBSD.ORG Fri Apr 25 13:08:52 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 06530B35 for ; Fri, 25 Apr 2014 13:08:52 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id C06F611B0 for ; Fri, 25 Apr 2014 13:08:51 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqIEAKRQWlODaFve/2dsb2JhbABZhCyCZb4rgw+BJXSCJQEBBAEjVgUWGAICDRkCWQYKiEIIpx2jTheBKYx8NAeCb4FKBKthg00hgW4 X-IronPort-AV: E=Sophos;i="4.97,927,1389762000"; d="scan'208";a="117959068" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 25 Apr 2014 09:08:41 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id AA832B3F4A; Fri, 25 Apr 2014 09:08:41 -0400 (EDT) Date: Fri, 25 Apr 2014 09:08:41 -0400 (EDT) From: Rick Macklem To: Marek Salwerowicz Message-ID: <31234037.791330.1398431321686.JavaMail.root@uoguelph.ca> In-Reply-To: <535A4DC2.605@wp.pl> Subject: Re: NFS over LAGG / lacp poor performance MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Apr 2014 13:08:52 -0000 Marek Salwerowicz wrote: > W dniu 2014-04-25 13:48, Rick Macklem pisze: > > Well, you don't mention what command(s) you are using to transfer > > the > > data, but I would guess you have one serial data transfer for each > > command. > > (Put another way, if you are only running one command to transfer > > the data, > > there will only be one RPC happening at a time and that will only > > use one > > network interface.) I don't know anything about lagg, so I can't > > comment > > related to it, but if there is only one NFS RPC at a time, you'll > > only > > be transferring one message at a time on the wire.) > > I need to transfer 15 files, each is about 1TB sized. > > From 9.1-RELEASE[storage1] to 10-RELEASE[storage2] > > I have tried to run concurrent 'cp' and transfer 4 files at the same > time: > > (executed on storage1) > # cp -a file1 /net/storage2/ & > # cp -a file2 /net/storage2/ & > # cp -a file3 /net/storage2/ & > # cp -a file4 /net/storage2/ & > Although I doubt it will make much difference, you might want to try "dd" with a fairly large blocksize (at least 64K). I don't know what blocksize "cp" uses and whether or not it does mmap'd file access. (mmap will only do I/O in page size blocks, so I think it will be slower.) > > But in fact I did not observe bigger throughput > > Both servers have filesystem exported using NFS, so I can execute > copy > on source, or destination. > Would you recommend running this on source-side, or rather > destination-side ? > Usually reads run faster than writes for NFS, so I'd try doing the mounts and running the commands on the destination side. That is also when "readahead=8" might help some. I'd add that option to the NFS mount (you can try any value you'd like, up to 16, but if 8 doesn't run faster than the default of 1, it probably isn't worth trying other values). > > > > Adding the mount option "readahead=8" to the machine receiving the > > data > > might help, if the data transfer command is being done there. (ie. > > The machine > > the data is being copied to has the other one NFS mounted and it is > > where > > you are running the data transfer command(s).) > > > Regarding what I wrote above - how should I mount the NFS volumes? > As above, I'd use nfsv3,readahead=8 options on the destination as a starting point. rick > Cheers, > Marek >