From owner-freebsd-fs@FreeBSD.ORG Wed Oct 3 12:36:23 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A346106566C for ; Wed, 3 Oct 2012 12:36:23 +0000 (UTC) (envelope-from peter.maloney@brockmann-consult.de) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.171]) by mx1.freebsd.org (Postfix) with ESMTP id A8AE98FC17 for ; Wed, 3 Oct 2012 12:36:22 +0000 (UTC) Received: from [192.168.179.201] (hmbg-4d06dfd3.pool.mediaWays.net [77.6.223.211]) by mrelayeu.kundenserver.de (node=mreu4) with ESMTP (Nemesis) id 0MEP3U-1TCkMW48kK-00FR5g; Wed, 03 Oct 2012 14:36:21 +0200 Message-ID: <506C3143.6060000@brockmann-consult.de> Date: Wed, 03 Oct 2012 14:36:19 +0200 From: Peter Maloney User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20120825 Thunderbird/15.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <1363900011.1436778.1348962614353.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <1363900011.1436778.1348962614353.JavaMail.root@erie.cs.uoguelph.ca> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Provags-ID: V02:K0:ZfMSL66Z8OmznM7gIU78BJtCXtXyohP4QN8Y4f9datC EDAbazPhknj9032fN/eFyXzrPvIJHJ1uprS4w3njIZXkPJIjga 9sOM8piYx7bALI65M/Q+usjbg5BxhY/GgfYmqkcWy/yld5mHh6 n4FhQE6ovs2kxlehYEi7q2WGzlXUJ3wb63ZP/QiOGaKLhJkEVA C0eM6kjPyRKNQfpqVkU7JnIf1NwxpERa/NMCCnZtBcPgm6lf/B mLtAGBrQ10v1D5IiYjl4DQgyVpcf0CdEgORrJ4Yp47lcTg9Bbq 9Bs+CjFB0SSkVYNb3ht0bqa4pv9u2yQNkHyX8gMtuZK7NTM6jH UBbf7A0jPyAXOWBhEOeASI2tBThJ20AQuF3U++j36P6BPQ5BQ9 6eKUiqI6pu/sg== Subject: Re: NFS Performance Help X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Oct 2012 12:36:23 -0000 On 09/30/2012 01:50 AM, Rick Macklem wrote: > Wayne Hotmail wrote: >> Like others I am having issues getting any decent performance out of >> my NFS clients on FreeBSD.I have tried 8.3 and 9.1 beta on stand alone >> servers or as vmware clients. Used 1 Gig connections or a 10 Gig >> connection.Tried mounting using Version 3 and Version 4.I have tried >> the noatime, sync, and tcp options nothing seems to help.I am >> connecting to a IceWeb NAS. My performance with DD is 60 meg a second >> at best when writing to the server. If I load a Redhat Linux server on >> the same hardware using the same connection my write performance is >> about 340 Meg a second. >> It really falls apart when I run a test script where I create a 100 >> folders then create a 100 files in the folders and append to these >> files 5 times using 5 random files. I am trying to simulate a IMAP >> email server. If I run the script on my local mirror drives it takes >> about a one minute and thirty seconds to complete. If I run the script >> on the NFS mounted drives it takes hours to complete. With my Linux >> install on the same hardware this NFS mounted script takes about 4 >> minutes. >> Google is tired of me asking the same question over and over. So if >> anyone would be so kind as to point out some kernel or system tweaks >> to get me passed my NFS issues that would be greatly appreciated. >> Wayne >> > You could try a smaller rsize,wsize by setting the command line args > for the mount. In general, larger rsize,wsize should perform better, > but if a large write generates a burst of traffic that overloads > some part of the network fabric or server, such that packets get > dropped, performance will be hit big time. > > Other than that, if you capture packets and look at them in > wireshark, you might be able to spot where packets are getting lost > and retransmitted. (If packets are getting dropped, then the fun > part is figuring out why and coming up with a workaround.) > > Hopefully others will have more/better suggestions, rick > My only suggestion is to try (but not necessarily in production) the changes suggested in the thread "NFSv3, ZFS, 10GE performance" started by Sven Brandenburg. It didn't do much for my testing, but he says it does a bunch. However, he is using a Linux client, and a ram based ZIL (using ZFS). Other than that, I can only say that I observed the same thing as you (testing both FreeBSD and Linux clients), but I always tested with ZFS. And I found that with FreeBSD, it was putting high load on the ZIL, meaning FreeBSD was using sync writes, but Linux was not. ESXi did the same thing as a client. So with a cheap SSD as a ZIL, ESXi and FreeBSD were writing at around 40-70 MB/s and Linux was writing at 600. The same test using a virtual machine disk mounted over NFS shows how extreme the problem can be, and was instead 7 MB/s with FreeBSD and ESXi, and around 90-200 with Linux. (And to compare 10Gbps performance with other non-NFS tests, I could get something like 600MB/s with a simple netcat from local RAM to remote /dev/null, and 800-900 with more threads or NICs, I don't remember). I couldn't figure out for sure, but I couldn't cause any corruption in my testing, so I just assume Linux is only running "sync" calls when creating files, write barriers to virtual disks, etc. like it does with local file systems instead of doing every single write synchronously. Peter