From owner-freebsd-fs@FreeBSD.ORG Sun May 12 07:13:57 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3E494CD0 for ; Sun, 12 May 2013 07:13:57 +0000 (UTC) (envelope-from scrappy@hub.org) Received: from hub.org (hub.org [200.46.208.146]) by mx1.freebsd.org (Postfix) with ESMTP id E8128E4A for ; Sun, 12 May 2013 07:13:56 +0000 (UTC) Received: from maia.hub.org (unknown [200.46.151.188]) by hub.org (Postfix) with ESMTP id 2D11517CF18E; Sun, 12 May 2013 04:13:55 -0300 (ADT) Received: from hub.org ([200.46.208.146]) by maia.hub.org (mx1.hub.org [200.46.151.188]) (amavisd-maia, port 10024) with ESMTP id 06408-01; Sun, 12 May 2013 07:13:54 +0000 (UTC) Received: from Marc-Fourniers-Mac-mini.local (S01067cb21b2ff4ca.gv.shawcable.net [24.108.26.71]) by hub.org (Postfix) with ESMTPA id E0DED17CF18B; Sun, 12 May 2013 04:13:53 -0300 (ADT) Message-ID: <518F4130.6080201@hub.org> Date: Sun, 12 May 2013 00:13:52 -0700 From: "Marc G. Fournier" User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: freebsd-fs@freebsd.org, Rick Macklem Subject: Re: NFS Performance issue against NetApp References: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> <518EFE05.8010100@hub.org> In-Reply-To: <518EFE05.8010100@hub.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 May 2013 07:13:57 -0000 'k, here is on Linux ... this is right after rebooting the server, doing a mount and running the startup once: Client rpc stats: calls retrans authrefrsh 40602 0 40609 Client nfs v3: null getattr setattr lookup access readlink 0 0% 13000 32% 5 0% 6140 15% 6741 16% 0 0% read write create mkdir symlink mknod 3556 8% 6711 16% 3743 9% 307 0% 0 0% 0 0% remove rmdir rename link readdir readdirplus 1 0% 0 0% 0 0% 0 0% 16 0% 380 0% fsstat fsinfo pathconf commit 0 0% 2 0% 1 0% 0 0% One thing to note is that both Linux/FreeBSD have "rsize=65536,wsize=65536" ... but there are 63x as many reads / 34x as many writes on FreeBSD as on Linux ... ? Just noticed this on the FreeBSD stats: Rpc Info: TimedOut Invalid X Replies Retries Requests 0 0 0 0 818479 818k Retries? Is that normal ... ? Also, the NetApp volumes being used here are not shared ... there are no other clients mounting these, and the Linux/FreeBSD volumes are seperate ... same size, same jboss install, same configuration, same war file ... I could mount /vol/linux_jboss onto the FreeBSD, or /vol/freebsd_jboss onto the Linux, and they would load the same way ... in fact, the jboss install itself was done onto the FreeBSD and copied over to the Linux ... and both are using OpenJDK7 ... I tried to make it as identical as I could ... On 2013-05-11 7:27 PM, Marc G. Fournier wrote: > > With > > vfs.nfs.noconsist=3 ... 385595ms > > nfsstat -z before startup, nfsstat -c after: > > Client Info: > Rpc Counts: > Getattr Setattr Lookup Readlink Read Write Create > Remove > 332594 5 17238 0 224426 231137 > 3743 1 > Rename Link Symlink Mkdir Rmdir Readdir > RdirPlus Access > 0 0 0 307 0 71 0 8447 > Mknod Fsstat Fsinfo PathConf Commit > 0 509 0 0 0 > Rpc Info: > TimedOut Invalid X Replies Retries Requests > 0 0 0 0 818479 > Cache Info: > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW > Hits Misses > 608296 332596 526200 17245 -95425 224426 13178 > 231137 > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs > Hits Misses > 0 0 1050 55 502 7 > 543340 8448 > > > ============ > > vfs.nfs.noconsist=2 ... 392201ms > > Client Info: > Rpc Counts: > Getattr Setattr Lookup Readlink Read Write Create > Remove > 332557 5 17228 0 224421 231131 > 3743 1 > Rename Link Symlink Mkdir Rmdir Readdir > RdirPlus Access > 0 0 0 307 0 72 0 8430 > Mknod Fsstat Fsinfo PathConf Commit > 0 502 0 0 0 > Rpc Info: > TimedOut Invalid X Replies Retries Requests > 0 0 0 0 818395 > Cache Info: > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW > Hits Misses > 607834 332557 525801 17231 -95401 224421 13178 > 231131 > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs > Hits Misses > 0 0 1028 56 502 0 > 542925 8431 > > > ============ > vfs.nfs.noconsist=0 ... 391622ms > > > Client Info: > Rpc Counts: > Getattr Setattr Lookup Readlink Read Write Create > Remove > 236122 5 17221 0 230575 230823 > 3743 1 > Rename Link Symlink Mkdir Rmdir Readdir > RdirPlus Access > 0 0 0 307 0 71 0 8425 > Mknod Fsstat Fsinfo PathConf Commit > 0 516 0 0 0 > Rpc Info: > TimedOut Invalid X Replies Retries Requests > 0 0 0 0 727799 > Cache Info: > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW > Hits Misses > 711860 236124 526549 17225 -101525 230490 13178 > 230823 > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs > Hits Misses > 0 0 1057 55 516 0 > 543709 8425 > > > I checked a second time with nonconsist=0, and the nfsstat -c values > seem to come out pretty much the same ... > > I'm going to head down to the office and try again with Solaris (I'd > have to re-install, since I used that system for the Solaris), and see > what nfsstat -c results I get out of that ... will post a followup on > this when completed ... > > > > On 2013-05-10 5:32 PM, Rick Macklem wrote: >> Marc G. Fournier wrote: >>> FYI … I just installed Solaris 11 onto the same hardware and ran the >>> same test … so far, I'm seeing: >>> >>> Linux @ ~30s >>> Solaris @ ~44s >>> >>> OpenBSD @ ~200s >>> FreeBSD @ ~240s >>> >>> I've even tried FreeBSD 8.3 just to see if maybe its as 'newish' issue >>> … same as 9.x … I could see Linux 'cutting corners', but >>> Oracle/Solaris too … ? >>> >> The three client implementations (BSD, Linux, Solaris) were developed >> independently and, as such, will all implement somewaht different >> caching algorithms (the RFCs specify what goes on the wire, but say >> little w.r.t. client side caching). >> >> I have a attached a patch that might be useful for determining if >> the client side buffer cache consistency algorithm in FreeBSD is >> causing the slow startup of jboss. Do not run this patch on a >> production system, since it pretty well disables all buffer cache >> coherency (ie. if another client modifies a file, the patched client >> won't notice and will continue to cache stale file data). >> >> If the patch does speed up startup of jboss significantly, you can >> use the sysctl: >> vfs.nfs.noconsist >> to check for which coherency check is involved by decreasing the >> value for the sysctl by 1 and then trying a startup again. (When >> vfs.nfs.noconsist=0, normal cache coherency will be applied.) >> >> I have no idea if buffer cache coherency is a factor, but trying >> the attached patch might determine if it is. >> >> Note that you have never posted updated "nfsstat -c" values. >> (Remember that what you posted indicated 88 RPCs, which seemed >> bogus.) Finding out if FreeBSD does a lot more of certain RPCs >> that Linux/Solaris might help isolate what is going on. >> >> rick >> >>> On 2013-05-03, at 04:50 , Mark Felder wrote: >>> >>>> On Thu, 02 May 2013 18:43:17 -0500, Marc G. Fournier >>>> wrote: >>>> >>>>> Hadn't thought to do so with Linux, but … >>>>> Linux ……. 20732ms, 20117ms, 20935ms, 20130ms, 20560ms >>>>> FreeBSD .. 28996ms, 24794ms, 24702ms, 23311ms, 24153ms >>>> Please make sure both platforms are using similar atime settings. I >>>> think most distros use ext4 with diratime by default. I'd just do >>>> noatime on both platforms to be safe. >>>> _______________________________________________ >>>> freebsd-fs@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> To unsubscribe, send any mail to >>>> "freebsd-fs-unsubscribe@freebsd.org" >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"