Date: Fri, 10 May 2013 20:32:46 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: "Marc G. Fournier" <scrappy@hub.org> Cc: freebsd-fs@freebsd.org Subject: Re: NFS Performance issue against NetApp Message-ID: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <030E4A04-D597-49BD-8979-27C3EFB6D276@hub.org>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Marc G. Fournier wrote: > FYI … I just installed Solaris 11 onto the same hardware and ran the > same test … so far, I'm seeing: > > Linux @ ~30s > Solaris @ ~44s > > OpenBSD @ ~200s > FreeBSD @ ~240s > > I've even tried FreeBSD 8.3 just to see if maybe its as 'newish' issue > … same as 9.x … I could see Linux 'cutting corners', but > Oracle/Solaris too … ? > The three client implementations (BSD, Linux, Solaris) were developed independently and, as such, will all implement somewaht different caching algorithms (the RFCs specify what goes on the wire, but say little w.r.t. client side caching). I have a attached a patch that might be useful for determining if the client side buffer cache consistency algorithm in FreeBSD is causing the slow startup of jboss. Do not run this patch on a production system, since it pretty well disables all buffer cache coherency (ie. if another client modifies a file, the patched client won't notice and will continue to cache stale file data). If the patch does speed up startup of jboss significantly, you can use the sysctl: vfs.nfs.noconsist to check for which coherency check is involved by decreasing the value for the sysctl by 1 and then trying a startup again. (When vfs.nfs.noconsist=0, normal cache coherency will be applied.) I have no idea if buffer cache coherency is a factor, but trying the attached patch might determine if it is. Note that you have never posted updated "nfsstat -c" values. (Remember that what you posted indicated 88 RPCs, which seemed bogus.) Finding out if FreeBSD does a lot more of certain RPCs that Linux/Solaris might help isolate what is going on. rick > On 2013-05-03, at 04:50 , Mark Felder <feld@feld.me> wrote: > > > On Thu, 02 May 2013 18:43:17 -0500, Marc G. Fournier > > <scrappy@hub.org> wrote: > > > >> Hadn't thought to do so with Linux, but … > >> Linux ……. 20732ms, 20117ms, 20935ms, 20130ms, 20560ms > >> FreeBSD .. 28996ms, 24794ms, 24702ms, 23311ms, 24153ms > > > > Please make sure both platforms are using similar atime settings. I > > think most distros use ext4 with diratime by default. I'd just do > > noatime on both platforms to be safe. > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to > > "freebsd-fs-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" [-- Attachment #2 --] --- fs/nfsclient/nfs_clvnops.c.sav 2013-05-10 18:31:01.000000000 -0400 +++ fs/nfsclient/nfs_clvnops.c 2013-05-10 19:38:45.000000000 -0400 @@ -246,6 +246,10 @@ int nfs_keep_dirty_on_error; SYSCTL_INT(_vfs_nfs, OID_AUTO, nfs_keep_dirty_on_error, CTLFLAG_RW, &nfs_keep_dirty_on_error, 0, "Retry pageout if error returned"); +int nfscl_noconsist = 3; +SYSCTL_INT(_vfs_nfs, OID_AUTO, noconsist, CTLFLAG_RW, + &nfscl_noconsist, 0, "Try disabling cache consistency"); + /* * This sysctl allows other processes to mmap a file that has been opened * O_DIRECT by a process. In general, having processes mmap the file while @@ -538,6 +542,7 @@ nfs_open(struct vop_open_args *ap) */ mtx_lock(&np->n_mtx); if (np->n_flag & NMODIFIED) { + if (nfscl_noconsist < 2) { mtx_unlock(&np->n_mtx); error = ncl_vinvalbuf(vp, V_SAVE, ap->a_td, 1); if (error == EINTR || error == EIO) { @@ -561,6 +566,7 @@ nfs_open(struct vop_open_args *ap) np->n_mtime = vattr.va_mtime; if (NFS_ISV4(vp)) np->n_change = vattr.va_filerev; + } } else { mtx_unlock(&np->n_mtx); error = VOP_GETATTR(vp, &vattr, ap->a_cred); @@ -570,8 +576,9 @@ nfs_open(struct vop_open_args *ap) return (error); } mtx_lock(&np->n_mtx); - if ((NFS_ISV4(vp) && np->n_change != vattr.va_filerev) || - NFS_TIMESPEC_COMPARE(&np->n_mtime, &vattr.va_mtime)) { + if (((NFS_ISV4(vp) && np->n_change != vattr.va_filerev) || + NFS_TIMESPEC_COMPARE(&np->n_mtime, &vattr.va_mtime)) && + nfscl_noconsist < 1) { if (vp->v_type == VDIR) np->n_direofoffset = 0; mtx_unlock(&np->n_mtx); --- fs/nfsclient/nfs_clbio.c.sav 2013-05-10 18:34:24.000000000 -0400 +++ fs/nfsclient/nfs_clbio.c 2013-05-10 19:38:57.000000000 -0400 @@ -69,6 +69,7 @@ extern enum nfsiod_state ncl_iodwant[NFS extern struct nfsmount *ncl_iodmount[NFS_MAXASYNCDAEMON]; extern int newnfs_directio_enable; extern int nfs_keep_dirty_on_error; +extern int nfscl_noconsist; int ncl_pbuf_freecnt = -1; /* start out unlimited */ @@ -402,7 +403,8 @@ nfs_bioread_check_cons(struct vnode *vp, return (error); mtx_lock(&np->n_mtx); if ((np->n_flag & NSIZECHANGED) - || (NFS_TIMESPEC_COMPARE(&np->n_mtime, &vattr.va_mtime))) { + || (NFS_TIMESPEC_COMPARE(&np->n_mtime, &vattr.va_mtime) && + nfscl_noconsist < 3)) { mtx_unlock(&np->n_mtx); if (vp->v_type == VDIR) ncl_invaldir(vp);
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?968416157.282645.1368232366317.JavaMail.root>