From owner-freebsd-fs@FreeBSD.ORG Tue Nov 19 18:25:02 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 31380310 for ; Tue, 19 Nov 2013 18:25:02 +0000 (UTC) Received: from mail-pb0-x233.google.com (mail-pb0-x233.google.com [IPv6:2607:f8b0:400e:c01::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 083D62257 for ; Tue, 19 Nov 2013 18:25:02 +0000 (UTC) Received: by mail-pb0-f51.google.com with SMTP id up15so4026174pbc.10 for ; Tue, 19 Nov 2013 10:25:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :message-id:references:to; bh=/5tu8Co/8KfQuraDa+xSnrTV8RBaBWzb79kk7EFy3hc=; b=DEKwYbKlB6W4wBUm2jVjlGUJEs7EdqBDW8ysrM4PRRxnMi5zMzntNahQLf9vxwPjyu i2Qs+DppU1hGZM4D+yyOvpOaEcdxkegTeBJ+IxV9rWR0xidlUnBx05JkbGUDROluk6Al GTJvkGi06n0q9YwBABIAmDro57lxvCZiYfRUy9sFrtccePEHxzeDxyKJMEEJuxUp2t2t LZHpLzcWalf11Fv65BaTVr9zqH1Fg2UwoiRJz3xruODYbiLc++72PiuLcOJOoppzffof ogOfH6l3TusvqeIZ01t3tvb673pPi+9QFPGzNysZEjEICjDkCmw9HncJbbzafPe1nrjE 7Swg== X-Received: by 10.68.13.104 with SMTP id g8mr27704739pbc.33.1384885501581; Tue, 19 Nov 2013 10:25:01 -0800 (PST) Received: from briankrusicw.logan.tv ([64.17.255.138]) by mx.google.com with ESMTPSA id kd1sm36317321pab.20.2013.11.19.10.25.00 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 19 Nov 2013 10:25:01 -0800 (PST) Subject: Re: Performance difference between UFS and ZFS with NFS Mime-Version: 1.0 (Apple Message framework v1085) From: aurfalien In-Reply-To: Date: Tue, 19 Nov 2013 10:25:03 -0800 Message-Id: <5969250F-0987-4304-BB95-52C7BAE8D84D@gmail.com> References: <2103733116.16923158.1384866769683.JavaMail.root@uoguelph.ca> <9F76D61C-EFEB-44B3-9717-D0795789832D@gmail.com> To: Eric Browning X-Mailer: Apple Mail (2.1085) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.16 Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Nov 2013 18:25:02 -0000 Curious. Do you have NFS locking enabled client side? Most likely you do as Mac Mail will not run w/o locks, nor will Adobe = prefs like temp cache. etc... So being this is prolly the case, could it be a mem pressure issue and = not enough RAM? So NFS locks take up RAM as does ARC. What are your mem stats and swap = stats during the 700% (yikes) experience? - aurf On Nov 19, 2013, at 10:19 AM, Eric Browning wrote: > Aurf, >=20 > I ran those two commands and it doesn't seem to have made a = difference. Usage is still above 700% and it still takes 30s to list a = directory. The time to list is proportional to the number of users = logged in. On UFS with all students logged in and hammering away at = their files there is no noticeable speed decrease. >=20 >=20 > On Tue, Nov 19, 2013 at 11:12 AM, aurfalien = wrote: >=20 > On Nov 19, 2013, at 5:12 AM, Rick Macklem wrote: >=20 > > Eric Browning wrote: > >> Some background: > >> -Two identical servers, dual AMD Athlon 6220's 16 cores total @ = 3Ghz, > >> -64GB ram each server > >> -Four Intel DC S3700 800GB SSDs for primary storage, each server. > >> -FreeBSD 9 stable as of 902503 > >> -ZFS v28 and later updated to feature flags (v29?) > >> -LSI 9200-8i controller > >> -Intel I350T4 nic (only one port being used currently) using all = four > >> in > >> LACP overtaxed the server's NFS queue from what we found out making > >> the > >> server basically unusable. > >> > >> There is definitely something going on between NFS and ZFS when = used > >> as a > >> file server (random workload) for mac home directories. They do = not > >> jive > >> well at all and pretty much drag down these beefy servers and cause > >> 20-30 > >> second delays when just attempting to list a directory on Mac 10.7, > >> 10.8 > >> clients although throughput seems fast when copying files. > >> > >> This server's NFS was sitting north of 700% (7+ cores) all day long > >> when > >> using ZFSv28 raidz1. I have also tried stripe, compression on/off, > >> sync > >> enabled/disabled, and no dedup with 56GB of ram dedicated to ARC. > >> I've > >> tried just 100% stock settings in loader.conf and and some > >> recommended > >> tuning from various sources on the freebsd lists and other sites > >> including > >> the freebsd handbook. > >> > >> This is my mountpoint creation: > >> zfs create -o mountpoint=3D/users -o sharenfs=3Don -o > >> casesensitivity=3Dinsensitive -o aclmode=3Dpassthrough -o = compression=3Dlz4 > >> -o > >> atime=3Doff -o aclinherit=3Dpassthrough tank/users > >> > >> This last weekend I switched one of these servers over to a UFS = raid > >> 0 > >> setup and NFS now only eats about 36% of one core during the = initial > >> login > >> phase of 150-ish users over about 10 minutes and sits under 1-3% > >> during > >> normal usage and directories all list instantly even when drilling > >> down 10 > >> or so directories on the client's home files. The same NFS config = on > >> server > >> and clients are still active. > >> > >> Right now I'm going to have to abandon ZFS until it works with NFS. > >> I > >> don't want to get into a finger pointing game, I'd just like to = help > >> get > >> this fixed, I have one old i386 server I can try things out on if > >> that > >> helps and it's already on 9 stable and ZFS v28. > >> > > Btw, in previous discussions with Eric on this, he provided nfsstat > > output that seemed to indicate most of his RPC load from the Macs > > were Access and Getattr RPCs. > > > > I suspect the way ZFS handles VOP_ACCESSX() and VOP_GETATTR() is a > > significant part of this issue. I know nothing about ZFS, but I = believe > > it does always have ACLs enabled and presumably needs to check the > > ACL for each VOP_ACCESSX(). > > > > Hopefully someone familiar with how ZFS handles VOP_ACCESSX() and > > VOP_GETATTR() can look at these? >=20 > Indeed. However couldn't one simply disable ACL mode via; >=20 > zfs set aclinherit=3Ddiscard pool/dataset > zfs set aclmode=3Ddiscard pool/dataset >=20 > Eric, mind setting these and see? >=20 > Mid/late this week I'll be doing a rather large render farm test = amongst our Mac fleet against ZFS. >=20 > Will reply to this thread with outcome when I'm done. Should be = interesting. >=20 > - aurf >=20 > > > > rick > > > >> Thanks, > >> -- > >> Eric Browning > >> Systems Administrator > >> 801-984-7623 > >> > >> Skaggs Catholic Center > >> Juan Diego Catholic High School > >> Saint John the Baptist Middle > >> Saint John the Baptist Elementary > >> _______________________________________________ > >> freebsd-fs@freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" > >> > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >=20 >=20 >=20 >=20 > --=20 > Eric Browning > Systems Administrator > 801-984-7623 >=20 > Skaggs Catholic Center > Juan Diego Catholic High School > Saint John the Baptist Middle > Saint John the Baptist Elementary