From owner-freebsd-stable@FreeBSD.ORG Mon Oct 4 07:27:58 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AF121106564A for ; Mon, 4 Oct 2010 07:27:58 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (mail.vx.sk [IPv6:2a01:4f8:100:1043::3]) by mx1.freebsd.org (Postfix) with ESMTP id 3E3C88FC0A for ; Mon, 4 Oct 2010 07:27:58 +0000 (UTC) Received: from core.vx.sk (localhost [127.0.0.1]) by mail.vx.sk (Postfix) with ESMTP id 6DED612630E; Mon, 4 Oct 2010 09:27:57 +0200 (CEST) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk ([127.0.0.1]) by core.vx.sk (mail.vx.sk [127.0.0.1]) (amavisd-new, port 10024) with LMTP id J9AUw0oX4rwC; Mon, 4 Oct 2010 09:27:55 +0200 (CEST) Received: from [10.9.8.1] (188-167-78-139.dynamic.chello.sk [188.167.78.139]) by mail.vx.sk (Postfix) with ESMTPSA id 4A530126306; Mon, 4 Oct 2010 09:27:55 +0200 (CEST) Message-ID: <4CA981FC.80405@FreeBSD.org> Date: Mon, 04 Oct 2010 09:27:56 +0200 From: Martin Matuska User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; sk; rv:1.8.1.23) Gecko/20090812 Lightning/0.9 Thunderbird/2.0.0.23 Mnenhy/0.7.5.0 MIME-Version: 1.0 To: Artem Belevich References: <45cfd27021fb93f9b0877a1596089776.squirrel@nyi.unixathome.org> <4C511EF8-591C-4BB9-B7AA-30D5C3DDC0FF@langille.org> <4CA68BBD.6060601@langille.org> <4CA929A8.6000708@langille.org> In-Reply-To: X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: freebsd-stable , Dan Langille Subject: Re: zfs send/receive: is this slow? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2010 07:27:58 -0000 Try using zfs receive with the -v flag (gives you some stats at the end): # zfs send storage/bacula@transfer | zfs receive -v storage/compressed/bacula And use the following sysctl (you may set that in /boot/loader.conf, too): # sysctl vfs.zfs.txg.write_limit_override=805306368 I have good results with the 768MB writelimit on systems with at least 8GB RAM. With 4GB ram, you might want to try to set the TXG write limit to a lower threshold (e.g. 256MB): # sysctl vfs.zfs.txg.write_limit_override=268435456 You can experiment with that setting to get the best results on your system. A value of 0 means using calculated default (which is very high). During the operation you can observe what your disks actually do: a) via ZFS pool I/O statistics: # zpool iostat -v 1 b) via GEOM: # gstat -a mm Dňa 4. 10. 2010 4:06, Artem Belevich wrote / napísal(a): > On Sun, Oct 3, 2010 at 6:11 PM, Dan Langille wrote: >> I'm rerunning my test after I had a drive go offline[1]. But I'm not >> getting anything like the previous test: >> >> time zfs send storage/bacula@transfer | mbuffer | zfs receive >> storage/compressed/bacula-buffer >> >> $ zpool iostat 10 10 >> capacity operations bandwidth >> pool used avail read write read write >> ---------- ----- ----- ----- ----- ----- ----- >> storage 6.83T 5.86T 8 31 1.00M 2.11M >> storage 6.83T 5.86T 207 481 25.7M 17.8M > > It may be worth checking individual disk activity using gstat -f 'da.$' > > Some time back I had one drive that was noticeably slower than the > rest of the drives in RAID-Z2 vdev and was holding everything back. > SMART looked OK, there were no obvious errors and yet performance was > much worse than what I'd expect. gstat clearly showed that one drive > was almost constantly busy with much lower number of reads and writes > per second than its peers. > > Perhaps previously fast transfer rates were due to caching effects. > I.e. if all metadata already made it into ARC, subsequent "zfs send" > commands would avoid a lot of random seeks and would show much better > throughput. > > --Artem > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"