From owner-freebsd-fs@FreeBSD.ORG Tue Nov 19 18:12:47 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1DB2FC2 for ; Tue, 19 Nov 2013 18:12:47 +0000 (UTC) Received: from mail-pd0-x230.google.com (mail-pd0-x230.google.com [IPv6:2607:f8b0:400e:c02::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id EB55621B4 for ; Tue, 19 Nov 2013 18:12:46 +0000 (UTC) Received: by mail-pd0-f176.google.com with SMTP id w10so6320480pde.35 for ; Tue, 19 Nov 2013 10:12:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=js/8pMt0B9LGsw8Irx5Oxx7pvihDha3kZ+GOx7EbvF8=; b=AOCyPEYWpR4jihu+82TFvExOyLPeRkJkfvA6ALJ1W1wYMViJgsMDquRMWuaqH9eX4z hD3lmnMBsw/CT83f5WU5xrIUo01iITdlfQic26EGsYMz8EGv/fV44LiM+iyRJClaXm3E tj5bzfCDbjPtPrFPibgf79QYQ6jho3aaKeaCLBwSGx4CrPNsHzpNgqWIxb7B5kzzJ/Qy vdcPaivfgL3hvF8uO8GPE92JYLm3dyL2BCb+uDXWb1VqD7B/zztzCfKEPna0j06W7c+s cH7jE5yeR6UxTGZ42QwcQhV3xZZB4LGak1mkBoapjOwY8Z7xjWRlC7+8MEsGd++Wb7Tq ww6Q== X-Received: by 10.68.185.68 with SMTP id fa4mr9393317pbc.136.1384884766596; Tue, 19 Nov 2013 10:12:46 -0800 (PST) Received: from briankrusicw.logan.tv ([64.17.255.138]) by mx.google.com with ESMTPSA id yg3sm36263078pab.16.2013.11.19.10.12.45 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 19 Nov 2013 10:12:46 -0800 (PST) Subject: Re: Performance difference between UFS and ZFS with NFS Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: text/plain; charset=us-ascii From: aurfalien In-Reply-To: <2103733116.16923158.1384866769683.JavaMail.root@uoguelph.ca> Date: Tue, 19 Nov 2013 10:12:46 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <9F76D61C-EFEB-44B3-9717-D0795789832D@gmail.com> References: <2103733116.16923158.1384866769683.JavaMail.root@uoguelph.ca> To: Rick Macklem X-Mailer: Apple Mail (2.1085) Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Nov 2013 18:12:47 -0000 On Nov 19, 2013, at 5:12 AM, Rick Macklem wrote: > Eric Browning wrote: >> Some background: >> -Two identical servers, dual AMD Athlon 6220's 16 cores total @ 3Ghz, >> -64GB ram each server >> -Four Intel DC S3700 800GB SSDs for primary storage, each server. >> -FreeBSD 9 stable as of 902503 >> -ZFS v28 and later updated to feature flags (v29?) >> -LSI 9200-8i controller >> -Intel I350T4 nic (only one port being used currently) using all four >> in >> LACP overtaxed the server's NFS queue from what we found out making >> the >> server basically unusable. >>=20 >> There is definitely something going on between NFS and ZFS when used >> as a >> file server (random workload) for mac home directories. They do not >> jive >> well at all and pretty much drag down these beefy servers and cause >> 20-30 >> second delays when just attempting to list a directory on Mac 10.7, >> 10.8 >> clients although throughput seems fast when copying files. >>=20 >> This server's NFS was sitting north of 700% (7+ cores) all day long >> when >> using ZFSv28 raidz1. I have also tried stripe, compression on/off, >> sync >> enabled/disabled, and no dedup with 56GB of ram dedicated to ARC. >> I've >> tried just 100% stock settings in loader.conf and and some >> recommended >> tuning from various sources on the freebsd lists and other sites >> including >> the freebsd handbook. >>=20 >> This is my mountpoint creation: >> zfs create -o mountpoint=3D/users -o sharenfs=3Don -o >> casesensitivity=3Dinsensitive -o aclmode=3Dpassthrough -o = compression=3Dlz4 >> -o >> atime=3Doff -o aclinherit=3Dpassthrough tank/users >>=20 >> This last weekend I switched one of these servers over to a UFS raid >> 0 >> setup and NFS now only eats about 36% of one core during the initial >> login >> phase of 150-ish users over about 10 minutes and sits under 1-3% >> during >> normal usage and directories all list instantly even when drilling >> down 10 >> or so directories on the client's home files. The same NFS config on >> server >> and clients are still active. >>=20 >> Right now I'm going to have to abandon ZFS until it works with NFS. >> I >> don't want to get into a finger pointing game, I'd just like to help >> get >> this fixed, I have one old i386 server I can try things out on if >> that >> helps and it's already on 9 stable and ZFS v28. >>=20 > Btw, in previous discussions with Eric on this, he provided nfsstat > output that seemed to indicate most of his RPC load from the Macs > were Access and Getattr RPCs. >=20 > I suspect the way ZFS handles VOP_ACCESSX() and VOP_GETATTR() is a > significant part of this issue. I know nothing about ZFS, but I = believe > it does always have ACLs enabled and presumably needs to check the > ACL for each VOP_ACCESSX(). >=20 > Hopefully someone familiar with how ZFS handles VOP_ACCESSX() and > VOP_GETATTR() can look at these? Indeed. However couldn't one simply disable ACL mode via; zfs set aclinherit=3Ddiscard pool/dataset zfs set aclmode=3Ddiscard pool/dataset Eric, mind setting these and see? Mid/late this week I'll be doing a rather large render farm test amongst = our Mac fleet against ZFS. Will reply to this thread with outcome when I'm done. Should be = interesting. - aurf =20 >=20 > rick >=20 >> Thanks, >> -- >> Eric Browning >> Systems Administrator >> 801-984-7623 >>=20 >> Skaggs Catholic Center >> Juan Diego Catholic High School >> Saint John the Baptist Middle >> Saint John the Baptist Elementary >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"