From owner-freebsd-fs@FreeBSD.ORG Thu Nov 21 23:04:38 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B1643C05 for ; Thu, 21 Nov 2013 23:04:38 +0000 (UTC) Received: from mail-pa0-f53.google.com (mail-pa0-f53.google.com [209.85.220.53]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 868602E64 for ; Thu, 21 Nov 2013 23:04:38 +0000 (UTC) Received: by mail-pa0-f53.google.com with SMTP id hz1so442119pad.12 for ; Thu, 21 Nov 2013 15:04:32 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=uypEHGYAPqqCv+ExWvRJnemGXsVnNaDEMaQVZotxSdk=; b=NgAKlttqMAeysuALjCAAPTD/6thzeCkAslTzV6Nu3J53VE946nkpJsqncZ/lt/sBh3 bLMug7CNee606q10TSksTY4ucpPO2ZizP2BmLWJ6/rN3CAd5zjEGT6w5FhzHhBfjZLIY S7ftUrP9m8KYMMQlVh+6SLCZZFTDoBHFK/yUBe1DqjaHRCSgr1CnMxjjkuveJTkrpCpI QnktE3gDOflQBODiHKW/6/RCzaJShFxiJlisY3gv942Gl3qGSjskE/0knVShQNQviVEn 0ZadBygAVWSkQxj1ieyijMrn5wIiE86/gx9y2FzLBYwqDFDJ/QXzd75kQksQtmkRoTIg WouA== X-Gm-Message-State: ALoCoQlVabF/YgG/l7cI5sP4TvXSnG5VZC2ez/MZFkD9MwpE/F2rOzB/PmAuB90bLif/8TZ2m7ud MIME-Version: 1.0 X-Received: by 10.66.248.202 with SMTP id yo10mr3753366pac.177.1385075072312; Thu, 21 Nov 2013 15:04:32 -0800 (PST) Received: by 10.70.102.133 with HTTP; Thu, 21 Nov 2013 15:04:32 -0800 (PST) In-Reply-To: <18391B9C-2FC4-427B-A4B6-1739B3C17498@gmail.com> References: <2103733116.16923158.1384866769683.JavaMail.root@uoguelph.ca> <9F76D61C-EFEB-44B3-9717-D0795789832D@gmail.com> <5969250F-0987-4304-BB95-52C7BAE8D84D@gmail.com> <18391B9C-2FC4-427B-A4B6-1739B3C17498@gmail.com> Date: Thu, 21 Nov 2013 16:04:32 -0700 Message-ID: Subject: Re: Performance difference between UFS and ZFS with NFS From: Eric Browning To: aurfalien Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.16 Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Nov 2013 23:04:38 -0000 Just as a bit of a followup I had 163 kids all logged in at once today and nfsd usage was only 1-5% @Aurf How are your results with your AE and C4D clients going? On Tue, Nov 19, 2013 at 12:38 PM, aurfalien wrote: > Wow, those are great mount options, I use em too :) > > Well, this is very interesting on the +3x access/getattrs with ZFS. > > I'll report back my findings as I'm going down a similar road, albeit not > home dirs but rendering using AE and C4D on many clients. > > Until then hoping some one chime in on this with some added nuggets. > > - aurf > > On Nov 19, 2013, at 11:11 AM, Eric Browning wrote: > > Locking is set to locallocks, cache folders and similar folders are > redirected to the local hard drive. All applications run just fine > including Adobe CS6 and MS 2011 apps. > > This is my client NFS conf: > nfs.client.mount.options = > noatime,nobrowse,tcp,vers=3,rsize=32768,wsize=32768,readahead=0,acregmax=3600,acdirmax=3600,locallocks,inet,noquota,nfc > nfs.client.statfs_rate_limit = 5 > nfs.client.access_for_getattr = 1 > nfs.client.is_mobile = 0 > > I'm sure this is more complex than it needs to be and I can probably get > rid of most of this now, forcing nfc did cure some unicode issues between > mac and freebsd. Packets are not being fragmented and there are only one or > two errors here and there despite traversing vlans through the core router, > MSS is set at 1460. > > One thing Rick M suggested is actually trying these entire setup on a UFS > system. I tested by copying my home folder to another server with a UFS > system and ran it for like 45 minutes and compared it to another 45 minute > jaunt on the main file server and I had about 3x less Access and Getattrs > on UFS than I had on ZFS. Seeing this prompted me to move one server over > to a UFS raid and since doing that it's like day and night > performance-wise. > > Server's NFS is set to 256 threads ARC is currently only at 46G of 56G > total and NFS is 9.9G on the ZFS server and CPU usage is 878%. On the UFS > server NFS is the same 256 threads and 9.9G but as I look at it with > currently 52 users logged in NFS is at CPU 0.00% usage. > > This is the server NFS configs from rc.conf > ## NFS Server > rpcbind_enable="YES" > nfs_server_enable="YES" > mountd_flags="-r -l" > nfsd_enable="YES" > mountd_enable="YES" > rpc_lockd_enable="NO" > rpc_statd_enable="NO" > nfs_server_flags="-t -n 256" > nfsv4_server_enable="NO" > nfsuserd_enable="YES" > > UFS Server mem stats: > Mem: 49M Active, 56G Inact, 3246M Wired, 1434M Cache, 1654M Buf, 1002M Free > ARC: 1884K Total, 149K MFU, 1563K MRU, 16K Anon, 56K Header, 99K Other > Swap: 4096M Total, 528K Used, 4095M Free > > ZFS mem stats: > Mem: 3180K Active, 114M Inact, 60G Wired, 1655M Buf, 2412M Free > ARC: 46G Total, 26G MFU, 13G MRU, 3099K Anon, 4394M Header, 4067M Other > Swap: 4096M Total, 4096M Free > > > > On Tue, Nov 19, 2013 at 11:25 AM, aurfalien wrote: > >> Curious. >> >> Do you have NFS locking enabled client side? >> >> Most likely you do as Mac Mail will not run w/o locks, nor will Adobe >> prefs like temp cache. etc... >> >> So being this is prolly the case, could it be a mem pressure issue and >> not enough RAM? >> >> So NFS locks take up RAM as does ARC. What are your mem stats and swap >> stats during the 700% (yikes) experience? >> >> - aurf >> >> On Nov 19, 2013, at 10:19 AM, Eric Browning wrote: >> >> Aurf, >> >> I ran those two commands and it doesn't seem to have made a difference. >> Usage is still above 700% and it still takes 30s to list a directory. The >> time to list is proportional to the number of users logged in. On UFS with >> all students logged in and hammering away at their files there is no >> noticeable speed decrease. >> >> >> On Tue, Nov 19, 2013 at 11:12 AM, aurfalien wrote: >> >>> >>> On Nov 19, 2013, at 5:12 AM, Rick Macklem wrote: >>> >>> > Eric Browning wrote: >>> >> Some background: >>> >> -Two identical servers, dual AMD Athlon 6220's 16 cores total @ 3Ghz, >>> >> -64GB ram each server >>> >> -Four Intel DC S3700 800GB SSDs for primary storage, each server. >>> >> -FreeBSD 9 stable as of 902503 >>> >> -ZFS v28 and later updated to feature flags (v29?) >>> >> -LSI 9200-8i controller >>> >> -Intel I350T4 nic (only one port being used currently) using all four >>> >> in >>> >> LACP overtaxed the server's NFS queue from what we found out making >>> >> the >>> >> server basically unusable. >>> >> >>> >> There is definitely something going on between NFS and ZFS when used >>> >> as a >>> >> file server (random workload) for mac home directories. They do not >>> >> jive >>> >> well at all and pretty much drag down these beefy servers and cause >>> >> 20-30 >>> >> second delays when just attempting to list a directory on Mac 10.7, >>> >> 10.8 >>> >> clients although throughput seems fast when copying files. >>> >> >>> >> This server's NFS was sitting north of 700% (7+ cores) all day long >>> >> when >>> >> using ZFSv28 raidz1. I have also tried stripe, compression on/off, >>> >> sync >>> >> enabled/disabled, and no dedup with 56GB of ram dedicated to ARC. >>> >> I've >>> >> tried just 100% stock settings in loader.conf and and some >>> >> recommended >>> >> tuning from various sources on the freebsd lists and other sites >>> >> including >>> >> the freebsd handbook. >>> >> >>> >> This is my mountpoint creation: >>> >> zfs create -o mountpoint=/users -o sharenfs=on -o >>> >> casesensitivity=insensitive -o aclmode=passthrough -o compression=lz4 >>> >> -o >>> >> atime=off -o aclinherit=passthrough tank/users >>> >> >>> >> This last weekend I switched one of these servers over to a UFS raid >>> >> 0 >>> >> setup and NFS now only eats about 36% of one core during the initial >>> >> login >>> >> phase of 150-ish users over about 10 minutes and sits under 1-3% >>> >> during >>> >> normal usage and directories all list instantly even when drilling >>> >> down 10 >>> >> or so directories on the client's home files. The same NFS config on >>> >> server >>> >> and clients are still active. >>> >> >>> >> Right now I'm going to have to abandon ZFS until it works with NFS. >>> >> I >>> >> don't want to get into a finger pointing game, I'd just like to help >>> >> get >>> >> this fixed, I have one old i386 server I can try things out on if >>> >> that >>> >> helps and it's already on 9 stable and ZFS v28. >>> >> >>> > Btw, in previous discussions with Eric on this, he provided nfsstat >>> > output that seemed to indicate most of his RPC load from the Macs >>> > were Access and Getattr RPCs. >>> > >>> > I suspect the way ZFS handles VOP_ACCESSX() and VOP_GETATTR() is a >>> > significant part of this issue. I know nothing about ZFS, but I believe >>> > it does always have ACLs enabled and presumably needs to check the >>> > ACL for each VOP_ACCESSX(). >>> > >>> > Hopefully someone familiar with how ZFS handles VOP_ACCESSX() and >>> > VOP_GETATTR() can look at these? >>> >>> Indeed. However couldn't one simply disable ACL mode via; >>> >>> zfs set aclinherit=discard pool/dataset >>> zfs set aclmode=discard pool/dataset >>> >>> Eric, mind setting these and see? >>> >>> Mid/late this week I'll be doing a rather large render farm test amongst >>> our Mac fleet against ZFS. >>> >>> Will reply to this thread with outcome when I'm done. Should be >>> interesting. >>> >>> - aurf >>> >>> > >>> > rick >>> > >>> >> Thanks, >>> >> -- >>> >> Eric Browning >>> >> Systems Administrator >>> >> 801-984-7623 >>> >> >>> >> Skaggs Catholic Center >>> >> Juan Diego Catholic High School >>> >> Saint John the Baptist Middle >>> >> Saint John the Baptist Elementary >>> >> _______________________________________________ >>> >> freebsd-fs@freebsd.org mailing list >>> >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>> >> >>> > _______________________________________________ >>> > freebsd-fs@freebsd.org mailing list >>> > http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>> >>> >> >> >> -- >> Eric Browning >> Systems Administrator >> 801-984-7623 >> >> Skaggs Catholic Center >> Juan Diego Catholic High School >> Saint John the Baptist Middle >> Saint John the Baptist Elementary >> >> >> > > > -- > Eric Browning > Systems Administrator > 801-984-7623 > > Skaggs Catholic Center > Juan Diego Catholic High School > Saint John the Baptist Middle > Saint John the Baptist Elementary > > > -- Eric Browning Systems Administrator 801-984-7623 Skaggs Catholic Center Juan Diego Catholic High School Saint John the Baptist Middle Saint John the Baptist Elementary