From owner-freebsd-fs@FreeBSD.ORG Tue Nov 19 01:16:02 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A07582E9 for ; Tue, 19 Nov 2013 01:16:02 +0000 (UTC) Received: from mail-pb0-f45.google.com (mail-pb0-f45.google.com [209.85.160.45]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 7E2AA27F4 for ; Tue, 19 Nov 2013 01:16:02 +0000 (UTC) Received: by mail-pb0-f45.google.com with SMTP id rp16so1112934pbb.4 for ; Mon, 18 Nov 2013 17:16:01 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=RQQILa8yhjYqKyU1vK7bp3Rpwo1XEo9hsjXZUtVhUxw=; b=KD9BZWbrG2MmTzeimYn5jtVnlOlN/vmEcBd4N1UmWADQZnFq07THq38U0vwCp8YiIy ktJy9XkS1K4CWWoiqyyJI67F1TmKH0/1TztuS+VBvIHKXqT1+7vF/s9YcSdXTd5/UR/k pmxeebP86LMu69bzb01+coUXBeOQTt07d9rZs4j5p+4rHfbZ1tKEJAhNbg8IUX7Epjtv pw4gIKTiPv/evo6ZvoH55waNnJvweTLKsrf+O5FEFEuWPPbXZg1MDPfI+g8Axmd6NPYC S1fZct2eouSH/BkpntVuSZQWyxiryt1uygrVyoknK6o+OEmB+/5qPaI4psKIpJAMQ5uq ONOQ== X-Gm-Message-State: ALoCoQlG6DrOLYYXnVuL2Cl374Gp1cZS57lEOz+glnYXXJ5LtvGJGsQFMNoyZYZ0o2f1PG2tvcdx MIME-Version: 1.0 X-Received: by 10.68.189.197 with SMTP id gk5mr23692420pbc.37.1384823761585; Mon, 18 Nov 2013 17:16:01 -0800 (PST) Received: by 10.70.102.133 with HTTP; Mon, 18 Nov 2013 17:16:01 -0800 (PST) Date: Mon, 18 Nov 2013 18:16:01 -0700 Message-ID: Subject: Performance difference between UFS and ZFS with NFS From: Eric Browning To: FreeBSD FS Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.16 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Nov 2013 01:16:02 -0000 Some background: -Two identical servers, dual AMD Athlon 6220's 16 cores total @ 3Ghz, -64GB ram each server -Four Intel DC S3700 800GB SSDs for primary storage, each server. -FreeBSD 9 stable as of 902503 -ZFS v28 and later updated to feature flags (v29?) -LSI 9200-8i controller -Intel I350T4 nic (only one port being used currently) using all four in LACP overtaxed the server's NFS queue from what we found out making the server basically unusable. There is definitely something going on between NFS and ZFS when used as a file server (random workload) for mac home directories. They do not jive well at all and pretty much drag down these beefy servers and cause 20-30 second delays when just attempting to list a directory on Mac 10.7, 10.8 clients although throughput seems fast when copying files. This server's NFS was sitting north of 700% (7+ cores) all day long when using ZFSv28 raidz1. I have also tried stripe, compression on/off, sync enabled/disabled, and no dedup with 56GB of ram dedicated to ARC. I've tried just 100% stock settings in loader.conf and and some recommended tuning from various sources on the freebsd lists and other sites including the freebsd handbook. This is my mountpoint creation: zfs create -o mountpoint=/users -o sharenfs=on -o casesensitivity=insensitive -o aclmode=passthrough -o compression=lz4 -o atime=off -o aclinherit=passthrough tank/users This last weekend I switched one of these servers over to a UFS raid 0 setup and NFS now only eats about 36% of one core during the initial login phase of 150-ish users over about 10 minutes and sits under 1-3% during normal usage and directories all list instantly even when drilling down 10 or so directories on the client's home files. The same NFS config on server and clients are still active. Right now I'm going to have to abandon ZFS until it works with NFS. I don't want to get into a finger pointing game, I'd just like to help get this fixed, I have one old i386 server I can try things out on if that helps and it's already on 9 stable and ZFS v28. Thanks, -- Eric Browning Systems Administrator 801-984-7623 Skaggs Catholic Center Juan Diego Catholic High School Saint John the Baptist Middle Saint John the Baptist Elementary