From owner-freebsd-stable@FreeBSD.ORG Fri Apr 29 00:13:57 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 105E3106564A for ; Fri, 29 Apr 2011 00:13:57 +0000 (UTC) (envelope-from doconnor@gsoft.com.au) Received: from cain.gsoft.com.au (unknown [IPv6:2001:44b8:7c07:5581:266:e1ff:fe0c:8f16]) by mx1.freebsd.org (Postfix) with ESMTP id 6172B8FC12 for ; Fri, 29 Apr 2011 00:13:54 +0000 (UTC) Received: from ur.dons.net.au (ppp208-191.lns1.adl2.internode.on.net [203.122.208.191] (may be forged)) (authenticated bits=0) by cain.gsoft.com.au (8.14.4/8.14.3) with ESMTP id p3T0DlrU096129 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Fri, 29 Apr 2011 09:43:48 +0930 (CST) (envelope-from doconnor@gsoft.com.au) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: "Daniel O'Connor" In-Reply-To: <20110428195601.GA31807@icarus.home.lan> Date: Fri, 29 Apr 2011 09:43:47 +0930 Content-Transfer-Encoding: quoted-printable Message-Id: References: <537A8F4F-A302-40F9-92DF-403388D99B4B@gsoft.com.au> <20110428195601.GA31807@icarus.home.lan> To: Jeremy Chadwick X-Mailer: Apple Mail (2.1084) X-Spam-Score: -0.272 () BAYES_00,RDNS_NONE X-Scanned-By: MIMEDefang 2.67 on 203.31.81.10 Cc: freebsd-stable List Subject: Re: ZFS vs OSX Time Machine X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2011 00:13:57 -0000 On 29/04/2011, at 5:26, Jeremy Chadwick wrote: >> I have the following ZFS related tunables >>=20 >> vfs.zfs.arc_max=3D"3072M" >> vfs.zfs.prefetch_disable=3D"1"=20 >> vfs.zfs.txg.timeout=3D5 >> vfs.zfs.cache_flush_disable=3D1 >=20 > Are the last two actually *working* in /boot/loader.conf? Can you > verify by looking at them via sysctl? AFAIK they shouldn't work, = since > they lack double-quotes around the values. Parsing errors are = supposed > to throw you back to the loader prompt. See loader.conf(5) for the > syntax. Yep, they're working :) > I'm also not sure why you're setting cache_flush_disable at all. I think I was wondering if it would help the abysmal write performance = of these disks.. >> Any help appreciated, thanks :) >=20 > Others seem to be battling stating that "NFS doesn't work for TM", but > that isn't what you're complaining about. You're complaining that > FreeBSD with ZFS + NFS performs extremely poorly when trying to do > backups from an OS X client using TM (writing to the NFS mount). Yes, and also TM is over AFP not NFS (I forgot to mention that..) > I have absolutely no experience with TM or OS X, so if it's actually a > client-level problem (which I'm doubting) I can't help you there. >=20 > Just sort of a ramble here at different things... >=20 > It would be useful to provide ZFS ARC sysctl data from the FreeBSD > system where you're seeing performance issues. "sysctl -a > kstat.zfs.misc.arcstats" should suffice. kstat.zfs.misc.arcstats.hits: 236092077 kstat.zfs.misc.arcstats.misses: 6451964 kstat.zfs.misc.arcstats.demand_data_hits: 98087637 kstat.zfs.misc.arcstats.demand_data_misses: 1220891 kstat.zfs.misc.arcstats.demand_metadata_hits: 138004440 kstat.zfs.misc.arcstats.demand_metadata_misses: 5231073 kstat.zfs.misc.arcstats.prefetch_data_hits: 0 kstat.zfs.misc.arcstats.prefetch_data_misses: 0 kstat.zfs.misc.arcstats.prefetch_metadata_hits: 0 kstat.zfs.misc.arcstats.prefetch_metadata_misses: 0 kstat.zfs.misc.arcstats.mru_hits: 15041670 kstat.zfs.misc.arcstats.mru_ghost_hits: 956048 kstat.zfs.misc.arcstats.mfu_hits: 221050407 kstat.zfs.misc.arcstats.mfu_ghost_hits: 3269042 kstat.zfs.misc.arcstats.allocated: 15785717 kstat.zfs.misc.arcstats.deleted: 4690878 kstat.zfs.misc.arcstats.stolen: 4990300 kstat.zfs.misc.arcstats.recycle_miss: 2142423 kstat.zfs.misc.arcstats.mutex_miss: 518 kstat.zfs.misc.arcstats.evict_skip: 2251705 kstat.zfs.misc.arcstats.evict_l2_cached: 0 kstat.zfs.misc.arcstats.evict_l2_eligible: 470396116480 kstat.zfs.misc.arcstats.evict_l2_ineligible: 2048 kstat.zfs.misc.arcstats.hash_elements: 482679 kstat.zfs.misc.arcstats.hash_elements_max: 503063 kstat.zfs.misc.arcstats.hash_collisions: 19593315 kstat.zfs.misc.arcstats.hash_chains: 116103 kstat.zfs.misc.arcstats.hash_chain_max: 16 kstat.zfs.misc.arcstats.p: 1692798721 kstat.zfs.misc.arcstats.c: 3221225472 kstat.zfs.misc.arcstats.c_min: 402653184 kstat.zfs.misc.arcstats.c_max: 3221225472 kstat.zfs.misc.arcstats.size: 3221162968 kstat.zfs.misc.arcstats.hdr_size: 103492088 kstat.zfs.misc.arcstats.data_size: 2764591616 kstat.zfs.misc.arcstats.other_size: 353079264 kstat.zfs.misc.arcstats.l2_hits: 0 kstat.zfs.misc.arcstats.l2_misses: 0 kstat.zfs.misc.arcstats.l2_feeds: 0 kstat.zfs.misc.arcstats.l2_rw_clash: 0 kstat.zfs.misc.arcstats.l2_read_bytes: 0 kstat.zfs.misc.arcstats.l2_write_bytes: 0 kstat.zfs.misc.arcstats.l2_writes_sent: 0 kstat.zfs.misc.arcstats.l2_writes_done: 0 kstat.zfs.misc.arcstats.l2_writes_error: 0 kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 0 kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0 kstat.zfs.misc.arcstats.l2_evict_reading: 0 kstat.zfs.misc.arcstats.l2_free_on_write: 0 kstat.zfs.misc.arcstats.l2_abort_lowmem: 0 kstat.zfs.misc.arcstats.l2_cksum_bad: 0 kstat.zfs.misc.arcstats.l2_io_error: 0 kstat.zfs.misc.arcstats.l2_size: 0 kstat.zfs.misc.arcstats.l2_hdr_size: 0 kstat.zfs.misc.arcstats.memory_throttle_count: 19 kstat.zfs.misc.arcstats.l2_write_trylock_fail: 0 kstat.zfs.misc.arcstats.l2_write_passed_headroom: 0 kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 0 kstat.zfs.misc.arcstats.l2_write_in_l2: 0 kstat.zfs.misc.arcstats.l2_write_io_in_progress: 0 kstat.zfs.misc.arcstats.l2_write_not_cacheable: 1 kstat.zfs.misc.arcstats.l2_write_full: 0 kstat.zfs.misc.arcstats.l2_write_buffer_iter: 0 kstat.zfs.misc.arcstats.l2_write_pios: 0 kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 0 kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 0 kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 0 > You should also try executing "zpool iostat -v 1" during the TM backup > to see if there's a particular device which is behaving poorly. There > have been reports of ZFS pools behaving poorly when a single device > within the pool has slow I/O (e.g. 5 hard disks, one of which has > internal issues, resulting in the entire pool performing horribly). = You > should let this run for probably 60-120 seconds to get an idea. Given > your parameters above (assuming vfs.zfs.txg.timeout IS in fact 5!), = you > should see "bursts" of writes every 5 seconds. OK. > I know that there are some things on ZFS that perform badly overall. > Anything that involves excessive/large numbers of files (not file = sizes, > but actual files themselves) seems to perform not-so-great with ZFS. > For example, Maildir on ZFS =3D piss-poor performance. There are ways = to > work around this issue (if I remember correctly, by adding a dedicated > "log" device to your ZFS pool, but be aware your log devices need to > be reliable (if you have a single log device and it fails the entire > pool is damaged, if I remember right)), but I don't consider it > feasible. So if TM is creating tons of files on the NFS mount (backed > by ZFS), then I imagine the performance isn't so great. Hmm, the sparse disk image does have ~80000 files in a single = directory.. > Could you please provide the following sysctl values? Thanks. >=20 > kern.maxvnodes > kern.minvnodes > vfs.freevnodes > vfs.numvnodes kern.maxvnodes: 204477 kern.minvnodes: 51119 vfs.freevnodes: 51118 vfs.numvnodes: 66116 > If the FreeBSD machine has a wireless card in it, if at all possible > could you try ruling that out by hooking up wired Ethernet instead? > It's probably not the cause, but worth trying anyway. If you have a > home router or something doing 802.11, don't bother with this idea. The FreeBSD box is wired, although it's using an re card as the em card = died(!!). The OSX box is connected via an Airport Express (11n). > Next, you COULD try using Samba/CIFS on the FreeBSD box to see if you > can narrow the issue down to bad NFS performance. Please see this = post > of mine about tuning Samba on FreeBSD (backed by ZFS) to get extremely > good performance. Many people responded and said their performance > drastically improved (you can see the thread yourself). The trick is > AIO. You can ignore the part about setting vm.kmem_size in = loader.conf; > that advice is now old/deprecated (does not pertain to you given the > date of your kernel), and vfs.zfs.txg.write_limit_override is = something > you shouldn't mess with unless absolutely needed to leave it default: >=20 > = http://lists.freebsd.org/pipermail/freebsd-stable/2011-February/061642.htm= l OK. I don't think TM can use CIFS, I will try ISCSI as someone else = suggested, perhaps it will help. > Finally, when was the last time this FreeBSD machine was rebooted? = Some > people have seen horrible performance that goes away after a reboot. > There's some speculation that memory fragmentation has something to do > with it. I simply don't know. I'm not telling you to reboot the box > (please don't; it would be more useful if it could be kept up in case > folks want to do analysis of it). I think performance does improve after a reboot :( top looks like.. last pid: 16112; load averages: 0.24, 0.22, 0.23 = up 8+16:11:50 09:43:19 653 processes: 1 running, 652 sleeping CPU: 3.6% user, 0.0% nice, 3.4% system, 0.6% interrupt, 92.5% idle Mem: 1401M Active, 578M Inact, 4143M Wired, 4904K Cache, 16M Buf, 1658M = Free Swap: 4096M Total, 160M Used, 3936M Free, 3% Inuse although free does go down very low (~250MB) at times. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C