From owner-freebsd-fs@FreeBSD.ORG Tue Nov 19 13:12:57 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3DA6895D for ; Tue, 19 Nov 2013 13:12:57 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 05BF72F51 for ; Tue, 19 Nov 2013 13:12:56 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqQEADVji1KDaFve/2dsb2JhbABWA4M/U4J2u3pOgTV0giUBAQEDAQEBASArIAsFFhgCAg0FARMCKQEJJgYIBwQBHASHWgYNrVeSJheBKYxzAQEGfyQQBxEBAYJYgUcDiUKMAYN+kF6DRh4xfAEHFyI X-IronPort-AV: E=Sophos;i="4.93,729,1378872000"; d="scan'208";a="71186384" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 19 Nov 2013 08:12:50 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 2B593B4093; Tue, 19 Nov 2013 08:12:50 -0500 (EST) Date: Tue, 19 Nov 2013 08:12:50 -0500 (EST) From: Rick Macklem To: Eric Browning Message-ID: <2103733116.16923158.1384866769683.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: Performance difference between UFS and ZFS with NFS MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.209] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Nov 2013 13:12:57 -0000 Eric Browning wrote: > Some background: > -Two identical servers, dual AMD Athlon 6220's 16 cores total @ 3Ghz, > -64GB ram each server > -Four Intel DC S3700 800GB SSDs for primary storage, each server. > -FreeBSD 9 stable as of 902503 > -ZFS v28 and later updated to feature flags (v29?) > -LSI 9200-8i controller > -Intel I350T4 nic (only one port being used currently) using all four > in > LACP overtaxed the server's NFS queue from what we found out making > the > server basically unusable. > > There is definitely something going on between NFS and ZFS when used > as a > file server (random workload) for mac home directories. They do not > jive > well at all and pretty much drag down these beefy servers and cause > 20-30 > second delays when just attempting to list a directory on Mac 10.7, > 10.8 > clients although throughput seems fast when copying files. > > This server's NFS was sitting north of 700% (7+ cores) all day long > when > using ZFSv28 raidz1. I have also tried stripe, compression on/off, > sync > enabled/disabled, and no dedup with 56GB of ram dedicated to ARC. > I've > tried just 100% stock settings in loader.conf and and some > recommended > tuning from various sources on the freebsd lists and other sites > including > the freebsd handbook. > > This is my mountpoint creation: > zfs create -o mountpoint=/users -o sharenfs=on -o > casesensitivity=insensitive -o aclmode=passthrough -o compression=lz4 > -o > atime=off -o aclinherit=passthrough tank/users > > This last weekend I switched one of these servers over to a UFS raid > 0 > setup and NFS now only eats about 36% of one core during the initial > login > phase of 150-ish users over about 10 minutes and sits under 1-3% > during > normal usage and directories all list instantly even when drilling > down 10 > or so directories on the client's home files. The same NFS config on > server > and clients are still active. > > Right now I'm going to have to abandon ZFS until it works with NFS. > I > don't want to get into a finger pointing game, I'd just like to help > get > this fixed, I have one old i386 server I can try things out on if > that > helps and it's already on 9 stable and ZFS v28. > Btw, in previous discussions with Eric on this, he provided nfsstat output that seemed to indicate most of his RPC load from the Macs were Access and Getattr RPCs. I suspect the way ZFS handles VOP_ACCESSX() and VOP_GETATTR() is a significant part of this issue. I know nothing about ZFS, but I believe it does always have ACLs enabled and presumably needs to check the ACL for each VOP_ACCESSX(). Hopefully someone familiar with how ZFS handles VOP_ACCESSX() and VOP_GETATTR() can look at these? rick > Thanks, > -- > Eric Browning > Systems Administrator > 801-984-7623 > > Skaggs Catholic Center > Juan Diego Catholic High School > Saint John the Baptist Middle > Saint John the Baptist Elementary > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >