From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 22 23:37:15 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 70AE985F; Mon, 22 Oct 2012 23:37:15 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 04E1F8FC16; Mon, 22 Oct 2012 23:37:14 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqAEAG3YhVCDaFvO/2dsb2JhbABEFoV+vDOCIAEBAQQBAQEgKyALBRYOCgICDRkCIwYBCSYGCAcEARwEh1EDDwupM4koDYlUgSCJT2kahUOBEgOTRFiBVYEXihKFEIMLgUc1 X-IronPort-AV: E=Sophos;i="4.80,633,1344225600"; d="scan'208";a="184785046" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 22 Oct 2012 19:36:41 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 35B4379462; Mon, 22 Oct 2012 19:36:41 -0400 (EDT) Date: Mon, 22 Oct 2012 19:36:41 -0400 (EDT) From: Rick Macklem To: Ivan Voras Message-ID: <656944923.2668891.1350949001204.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: Subject: Re: NFS server bottlenecks MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: "freebsd-hackers@freebsd.org Hackers" , Nikolay Denev X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2012 23:37:15 -0000 Ivan Voras wrote: > On 20 October 2012 13:42, Nikolay Denev wrote: > > > Here are the results from testing both patches : > > http://home.totalterror.net/freebsd/nfstest/results.html > > Both tests ran for about 14 hours ( a bit too much, but I wanted to > > compare different zfs recordsize settings ), > > and were done first after a fresh reboot. > > The only noticeable difference seems to be much more context > > switches with Ivan's patch. > > Thank you very much for your extensive testing! > > I don't know how to interpret the rise in context switches; as this is > kernel code, I'd expect no context switches. I hope someone else can > explain. > > But, you have also shown that my patch doesn't do any better than > Rick's even on a fairly large configuration, so I don't think there's > value in adding the extra complexity, and Rick knows NFS much better > than I do. > > But there are a few things other than that I'm interested in: like why > does your load average spike almost to 20-ties, and how come that with > 24 drives in RAID-10 you only push through 600 MBit/s through the 10 > GBit/s Ethernet. Have you tested your drive setup locally (AESNI > shouldn't be a bottleneck, you should be able to encrypt well into > Gbyte/s range) and the network? > > If you have the time, could you repeat the tests but with a recent > Samba server and a CIFS mount on the client side? This is probably not > important, but I'm just curious of how would it perform on your > machine. Oh, I realized that, if you are testing 9/stable (and not head), that you won't have r227809. Without that, all reads on a given file will be serialized, because the server will acquire an exclusive lock on the vnode. The patch for r227809 in head is at: http://people.freebsd.org/~rmacklem/lkshared.patch This should apply fine to a 9 system (but not 8.n), I think. Good luck with it and have fun, rick > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org"