Date: Sun, 24 Nov 2013 07:59:05 -0700 From: Eric Browning <ericbrowning@skaggscatholiccenter.org> To: aurfalien <aurfalien@gmail.com> Cc: FreeBSD FS <freebsd-fs@freebsd.org> Subject: Re: Performance difference between UFS and ZFS with NFS Message-ID: <CAM=5oeAybYZsn-TQTvfYO3Kx-chZULeYUEhg9fRMi-CFX0aYBA@mail.gmail.com> In-Reply-To: <CAM=5oeAXiRn2aHvNPuZRPFJp6G45OqdQEDsz2_xGobCUHJp_VQ@mail.gmail.com> References: <2103733116.16923158.1384866769683.JavaMail.root@uoguelph.ca> <9F76D61C-EFEB-44B3-9717-D0795789832D@gmail.com> <CAM=5oeAF2gfccrGNdbApUDpqRae4OQjZ7oaZZi4y1j%2BsF6PsTw@mail.gmail.com> <5969250F-0987-4304-BB95-52C7BAE8D84D@gmail.com> <CAM=5oeBmCAq9unFGC2CBoJ3rZMm9MtDw1DWkFpo2ZqQtx3G%2B=Q@mail.gmail.com> <18391B9C-2FC4-427B-A4B6-1739B3C17498@gmail.com> <CAM=5oeAXiRn2aHvNPuZRPFJp6G45OqdQEDsz2_xGobCUHJp_VQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On a side note I forgot that I had used dd to test the disk performance a while ago when I was using ZFS. ZFS performance: 3072000000 bytes transferred in 34.167480 secs (89910055 bytes/sec) 34.17s real 0.61s user 31.89s sys UFS performance: 3072000000 bytes transferred in 11.848883 secs (259264942 bytes/sec) 11.85s real 0.58s user 11.25s sys Again, even with dd performance is about 3x faster with UFS with the same disks. On Thu, Nov 21, 2013 at 4:04 PM, Eric Browning < ericbrowning@skaggscatholiccenter.org> wrote: > Just as a bit of a followup I had 163 kids all logged in at once today and > nfsd usage was only 1-5% > > @Aurf > How are your results with your AE and C4D clients going? > > > On Tue, Nov 19, 2013 at 12:38 PM, aurfalien <aurfalien@gmail.com> wrote: > >> Wow, those are great mount options, I use em too :) >> >> Well, this is very interesting on the +3x access/getattrs with ZFS. >> >> I'll report back my findings as I'm going down a similar road, albeit not >> home dirs but rendering using AE and C4D on many clients. >> >> Until then hoping some one chime in on this with some added nuggets. >> >> - aurf >> >> On Nov 19, 2013, at 11:11 AM, Eric Browning wrote: >> >> Locking is set to locallocks, cache folders and similar folders are >> redirected to the local hard drive. All applications run just fine >> including Adobe CS6 and MS 2011 apps. >> >> This is my client NFS conf: >> nfs.client.mount.options = >> noatime,nobrowse,tcp,vers=3,rsize=32768,wsize=32768,readahead=0,acregmax=3600,acdirmax=3600,locallocks,inet,noquota,nfc >> nfs.client.statfs_rate_limit = 5 >> nfs.client.access_for_getattr = 1 >> nfs.client.is_mobile = 0 >> >> I'm sure this is more complex than it needs to be and I can probably get >> rid of most of this now, forcing nfc did cure some unicode issues between >> mac and freebsd. Packets are not being fragmented and there are only one or >> two errors here and there despite traversing vlans through the core router, >> MSS is set at 1460. >> >> One thing Rick M suggested is actually trying these entire setup on a UFS >> system. I tested by copying my home folder to another server with a UFS >> system and ran it for like 45 minutes and compared it to another 45 minute >> jaunt on the main file server and I had about 3x less Access and Getattrs >> on UFS than I had on ZFS. Seeing this prompted me to move one server over >> to a UFS raid and since doing that it's like day and night >> performance-wise. >> >> Server's NFS is set to 256 threads ARC is currently only at 46G of 56G >> total and NFS is 9.9G on the ZFS server and CPU usage is 878%. On the UFS >> server NFS is the same 256 threads and 9.9G but as I look at it with >> currently 52 users logged in NFS is at CPU 0.00% usage. >> >> This is the server NFS configs from rc.conf >> ## NFS Server >> rpcbind_enable="YES" >> nfs_server_enable="YES" >> mountd_flags="-r -l" >> nfsd_enable="YES" >> mountd_enable="YES" >> rpc_lockd_enable="NO" >> rpc_statd_enable="NO" >> nfs_server_flags="-t -n 256" >> nfsv4_server_enable="NO" >> nfsuserd_enable="YES" >> >> UFS Server mem stats: >> Mem: 49M Active, 56G Inact, 3246M Wired, 1434M Cache, 1654M Buf, 1002M >> Free >> ARC: 1884K Total, 149K MFU, 1563K MRU, 16K Anon, 56K Header, 99K Other >> Swap: 4096M Total, 528K Used, 4095M Free >> >> ZFS mem stats: >> Mem: 3180K Active, 114M Inact, 60G Wired, 1655M Buf, 2412M Free >> ARC: 46G Total, 26G MFU, 13G MRU, 3099K Anon, 4394M Header, 4067M Other >> Swap: 4096M Total, 4096M Free >> >> >> >> On Tue, Nov 19, 2013 at 11:25 AM, aurfalien <aurfalien@gmail.com> wrote: >> >>> Curious. >>> >>> Do you have NFS locking enabled client side? >>> >>> Most likely you do as Mac Mail will not run w/o locks, nor will Adobe >>> prefs like temp cache. etc... >>> >>> So being this is prolly the case, could it be a mem pressure issue and >>> not enough RAM? >>> >>> So NFS locks take up RAM as does ARC. What are your mem stats and swap >>> stats during the 700% (yikes) experience? >>> >>> - aurf >>> >>> On Nov 19, 2013, at 10:19 AM, Eric Browning wrote: >>> >>> Aurf, >>> >>> I ran those two commands and it doesn't seem to have made a difference. >>> Usage is still above 700% and it still takes 30s to list a directory. The >>> time to list is proportional to the number of users logged in. On UFS with >>> all students logged in and hammering away at their files there is no >>> noticeable speed decrease. >>> >>> >>> On Tue, Nov 19, 2013 at 11:12 AM, aurfalien <aurfalien@gmail.com> wrote: >>> >>>> >>>> On Nov 19, 2013, at 5:12 AM, Rick Macklem wrote: >>>> >>>> > Eric Browning wrote: >>>> >> Some background: >>>> >> -Two identical servers, dual AMD Athlon 6220's 16 cores total @ 3Ghz, >>>> >> -64GB ram each server >>>> >> -Four Intel DC S3700 800GB SSDs for primary storage, each server. >>>> >> -FreeBSD 9 stable as of 902503 >>>> >> -ZFS v28 and later updated to feature flags (v29?) >>>> >> -LSI 9200-8i controller >>>> >> -Intel I350T4 nic (only one port being used currently) using all four >>>> >> in >>>> >> LACP overtaxed the server's NFS queue from what we found out making >>>> >> the >>>> >> server basically unusable. >>>> >> >>>> >> There is definitely something going on between NFS and ZFS when used >>>> >> as a >>>> >> file server (random workload) for mac home directories. They do not >>>> >> jive >>>> >> well at all and pretty much drag down these beefy servers and cause >>>> >> 20-30 >>>> >> second delays when just attempting to list a directory on Mac 10.7, >>>> >> 10.8 >>>> >> clients although throughput seems fast when copying files. >>>> >> >>>> >> This server's NFS was sitting north of 700% (7+ cores) all day long >>>> >> when >>>> >> using ZFSv28 raidz1. I have also tried stripe, compression on/off, >>>> >> sync >>>> >> enabled/disabled, and no dedup with 56GB of ram dedicated to ARC. >>>> >> I've >>>> >> tried just 100% stock settings in loader.conf and and some >>>> >> recommended >>>> >> tuning from various sources on the freebsd lists and other sites >>>> >> including >>>> >> the freebsd handbook. >>>> >> >>>> >> This is my mountpoint creation: >>>> >> zfs create -o mountpoint=/users -o sharenfs=on -o >>>> >> casesensitivity=insensitive -o aclmode=passthrough -o compression=lz4 >>>> >> -o >>>> >> atime=off -o aclinherit=passthrough tank/users >>>> >> >>>> >> This last weekend I switched one of these servers over to a UFS raid >>>> >> 0 >>>> >> setup and NFS now only eats about 36% of one core during the initial >>>> >> login >>>> >> phase of 150-ish users over about 10 minutes and sits under 1-3% >>>> >> during >>>> >> normal usage and directories all list instantly even when drilling >>>> >> down 10 >>>> >> or so directories on the client's home files. The same NFS config on >>>> >> server >>>> >> and clients are still active. >>>> >> >>>> >> Right now I'm going to have to abandon ZFS until it works with NFS. >>>> >> I >>>> >> don't want to get into a finger pointing game, I'd just like to help >>>> >> get >>>> >> this fixed, I have one old i386 server I can try things out on if >>>> >> that >>>> >> helps and it's already on 9 stable and ZFS v28. >>>> >> >>>> > Btw, in previous discussions with Eric on this, he provided nfsstat >>>> > output that seemed to indicate most of his RPC load from the Macs >>>> > were Access and Getattr RPCs. >>>> > >>>> > I suspect the way ZFS handles VOP_ACCESSX() and VOP_GETATTR() is a >>>> > significant part of this issue. I know nothing about ZFS, but I >>>> believe >>>> > it does always have ACLs enabled and presumably needs to check the >>>> > ACL for each VOP_ACCESSX(). >>>> > >>>> > Hopefully someone familiar with how ZFS handles VOP_ACCESSX() and >>>> > VOP_GETATTR() can look at these? >>>> >>>> Indeed. However couldn't one simply disable ACL mode via; >>>> >>>> zfs set aclinherit=discard pool/dataset >>>> zfs set aclmode=discard pool/dataset >>>> >>>> Eric, mind setting these and see? >>>> >>>> Mid/late this week I'll be doing a rather large render farm test >>>> amongst our Mac fleet against ZFS. >>>> >>>> Will reply to this thread with outcome when I'm done. Should be >>>> interesting. >>>> >>>> - aurf >>>> >>>> > >>>> > rick >>>> > >>>> >> Thanks, >>>> >> -- >>>> >> Eric Browning >>>> >> Systems Administrator >>>> >> 801-984-7623 >>>> >> >>>> >> Skaggs Catholic Center >>>> >> Juan Diego Catholic High School >>>> >> Saint John the Baptist Middle >>>> >> Saint John the Baptist Elementary >>>> >> _______________________________________________ >>>> >> freebsd-fs@freebsd.org mailing list >>>> >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org >>>> " >>>> >> >>>> > _______________________________________________ >>>> > freebsd-fs@freebsd.org mailing list >>>> > http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>>> >>>> >>> >>> >>> -- >>> Eric Browning >>> Systems Administrator >>> 801-984-7623 >>> >>> Skaggs Catholic Center >>> Juan Diego Catholic High School >>> Saint John the Baptist Middle >>> Saint John the Baptist Elementary >>> >>> >>> >> >> >> -- >> Eric Browning >> Systems Administrator >> 801-984-7623 >> >> Skaggs Catholic Center >> Juan Diego Catholic High School >> Saint John the Baptist Middle >> Saint John the Baptist Elementary >> >> >> > > > -- > Eric Browning > Systems Administrator > 801-984-7623 > > Skaggs Catholic Center > Juan Diego Catholic High School > Saint John the Baptist Middle > Saint John the Baptist Elementary > -- Eric Browning Systems Administrator 801-984-7623 Skaggs Catholic Center Juan Diego Catholic High School Saint John the Baptist Middle Saint John the Baptist Elementary
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAM=5oeAybYZsn-TQTvfYO3Kx-chZULeYUEhg9fRMi-CFX0aYBA>