From owner-freebsd-net@freebsd.org Fri Jun 26 00:49:20 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4C3B698D30D for ; Fri, 26 Jun 2015 00:49:20 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 0D7EC1647 for ; Fri, 26 Jun 2015 00:49:19 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2B8AwDioIxV/61jaINbg2VfBoMYuXYJgVwKhS5KAoFxFAEBAQEBAQGBCoQiAQEBAwEBAQEgKyALEAIBCBgCAg0ZAgInAQkmAgQIAgUEARwEiAYIDbghllQBAQEBAQEEAQEBAQEBARuBIYophCQQAQEFFzQHgmiBQwWUBoRYhDGERZY7AiZjgSYDHIFuIjEHfQgXI4ECAQEB X-IronPort-AV: E=Sophos;i="5.13,681,1427774400"; d="scan'208";a="219798762" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 25 Jun 2015 20:49:12 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 466F015F533; Thu, 25 Jun 2015 20:49:12 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id GI6zcEbVc4aN; Thu, 25 Jun 2015 20:49:11 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id A6BC415F54D; Thu, 25 Jun 2015 20:49:11 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id MZSAkatbU_JU; Thu, 25 Jun 2015 20:49:11 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 7B39B15F533; Thu, 25 Jun 2015 20:49:11 -0400 (EDT) Date: Thu, 25 Jun 2015 20:49:11 -0400 (EDT) From: Rick Macklem To: Gerrit =?utf-8?B?S8O8aG4=?= Cc: freebsd-net@freebsd.org, carsten aulbert Message-ID: <623856025.328424.1435279751389.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <20150625145238.12cf9da3b368ef0b9a30f193@aei.mpg.de> References: <20150625145238.12cf9da3b368ef0b9a30f193@aei.mpg.de> Subject: Re: NFS on 10G interface terribly slow MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: NFS on 10G interface terribly slow Thread-Index: bw2RjllCsH33QHtC58SnPo+3M9j5SQ== X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jun 2015 00:49:20 -0000 Gerrit Kuhn wrote: > Hi all, > > We have a recent FreeBSD 10.1 installation here that is supposed to act as > nfs (v3) client to an Oracle x4-2l server running Soalris 11.2. > We have Intel 10-Gigabit X540-AT2 NICs on both ends, iperf is showing > plenty of bandwidth (9.xGB/s) in both directions. > However, nfs appears to be terribly slow, especially for writing: > > root@crest:~ # dd if=/dev/zero of=/net/hellpool/Z bs=1024k count=1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 20.263190 secs (51747824 bytes/sec) > Recent commits to stable/10 (not in 10.1) done by Alexander Motin (mav@) might help w.r.t. write performance (it avoids large writes doing synchronous writes when the wcommitsize is exceeded). If you can try stable/10, that might be worth it. Otherwise, the main mount option you can try is "wcommitsize", which you probably want to make larger. (It sounds like you already tried most of what I could suggest.) > > Reading appears to be faster, but still far away from full bandwidth: > > root@crest:~ # dd of=/dev/null if=/net/hellpool/Z bs=1024k > 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 5.129869 secs (204406000 bytes/sec) > You could try increasing readahead. Look for the mount option and try cranking it up to 8 or 16. Good luck with it, rick > > We have already tried to tune rsize/wsize parameters, but they appear to > have little (if any) impact on these results. Also, neither stripping down > rxsum, txsum, tso etc. from the interface nor increasing MTU to 9000 for > jumbo frames did improve anything. > It is quite embarrassing to achieve way less than 1GBE performance with > 10GBE equipment. Are there any hints what else might be causing this (and > how to fix it)? > > > cu > Gerrit > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >