From owner-freebsd-fs@freebsd.org Wed Aug 19 15:29:50 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 84FA49BCFD6 for ; Wed, 19 Aug 2015 15:29:50 +0000 (UTC) (envelope-from paul@pk1048.com) Received: from cpanel61.fastdnsservers.com (server61.fastdnsservers.com [216.51.232.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 63B51B94 for ; Wed, 19 Aug 2015 15:29:49 +0000 (UTC) (envelope-from paul@pk1048.com) Received: from mail.thecreativeadvantage.com ([96.236.20.34]:62541 helo=mbp-1.thecreativeadvantage.com) by cpanel61.fastdnsservers.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.85) (envelope-from ) id 1ZS5J9-001Wxh-CJ; Wed, 19 Aug 2015 10:29:43 -0500 Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Optimizing performance with SLOG/L2ARC From: PK1048 In-Reply-To: Date: Wed, 19 Aug 2015 11:29:44 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <023F881D-CCC5-4FCA-B09D-EB92C3BFBC03@pk1048.com> References: To: javocado , FreeBSD Filesystems X-Mailer: Apple Mail (2.1878.6) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cpanel61.fastdnsservers.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - pk1048.com X-Get-Message-Sender-Via: cpanel61.fastdnsservers.com: authenticated_id: info@pk1048.com X-Source: X-Source-Args: X-Source-Dir: X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Aug 2015 15:29:50 -0000 Lots of good comments already posted, more inline below... On Aug 18, 2015, at 20:28, javocado wrote: > I've been trying to optimize and enhance my ZFS filesystem performance > (running FreeBSD 8.3amd) which has been sluggish at times. Thus far I = have > added RAM (256GB) and I've added an SLOG (SSD mirror). The RAM seems = to > have helped a bit, but not sure if the SLOG was of much help. My vdev = is > decently busy, with writes and reads averaging at 100 per second with > spikes as high as 500. What is the system doing _besides_ being a file server ? =46rom the = arcstats, the ARC is not getting all of the 256 GB RAM, what else is = running ? What is the pool layout (zpool status)=85 =46rom a strictly a = performance standpoint, each vdev gives you ONE drives worth of WRITE = performance. It does not much matter if the vdev is a mirror or = RAIDz. So a pool consisting of one vdev which is a 12 drive RAIDz2 = will have much slower write performance that a pool consisting of 3 = vdevs each of which is a 4 drive RAIDz2 =85 Yes, you are balancing off = capacity vs. performance vs. reliability. > 1. based on the output above, I believe a larger ARC may not = necessarily > benefit me at this point. True? The arcstats look good, are you having issues with reads or writes ? If = reads, then you need to now look at your disk layout. If writes, then = are they sync or async, if sync then a ZIL can help, if async, then more = ARC can help and you need to look at your disk layout. > 2. Is more (L2)ARC always better? ARC is _aways_ better than L2ARC. Adding L2ARC also consumes ARC space = with L2ARC metadata, and that is ARC space that cannot now be used to = cache file metadata and data. So adding L2ARC (which is slower than RAM = based ARC) _may_ actually make a system slower. > 3. I know it's a good idea to mirror the SLOG (and I have). Do I = understand > correctly that I do not need to mirror the L2ARC since it's just a = read > cache, nothing to lose if the SSD goes down? L2ARC is just read cache, so loosing it causes no data loss. You _may_ = lose performance _if_ the L2ARC was increasing your performance. > 4. Is there a better way than looking at zpool iostat -v to determine = the > SLOG utilization and usefulness? The ZIL/SLOG is used to cache SYNC write operations (if you test with = iozone, use the -o option to force sync writes). Using iostat -x you can = see the activity to the SSDs, both write and read. This can tell you how = much you are using them. I am not aware of any way to track how FULL = your ZIL/SLOG device is. Please note that, depending on your workload, an SSD may _not_ be any = faster than a HDD. I am in the process of rebuilding a file server that = exhibited poor NFS SYNC write performance. Yet it had a mirrored pair of = SSDs. Unfortunately, those SSDs had _worse_ write performance than an = HDD for small (4 KB) writes. Based on recommendations from the OpenZFS = list I have a pair of Intel 3710 SSDs coming in to try, they are = supposed to have much better write performance (at all block sizes) and = much better reliability long term (10x full disk writes per day for 5 = years). I=92ll know more once they arrive and I can test with them. Someone commented on the size of the ZIL/SLOG=85 it needs to hold all of = the write data that arrives between TXG commits, which happen at least = every 5 seconds (it used to be 30 seconds, but that scared too many = people :-). SU a sync write arrives and it _must_ be committed to = permanent storage, so ZFS writes it to the ZFS Intent Log (ZIL) which = may or may not be a separate device (vdev). When the TXG that contains = that data is committed to the pool itself the data can be flushed from = the ZIL. If your source of sync writes is network shares, and you have a = 1 Gbps link, then your maximum ZIL will be 5 seconds x 1 Gbps or 5 = Gigabits. > I'd like to test-see if adding L2ARC yields any performance boost. = Since > SLOG isn't doing much for me, I'm thinking I could easily repurpose my = SLOG > into an L2ARC. If a ZIL/SLOG device is not adding any performance, then either your = workload is NOT sync write bound -or- your ZIL/SLOG device is no faster = than your native pool performance. If the ZIL/SLOG device is not very = fast, then reusing it as an L2ARC -may- actually make the system slower. >=20 > Questions: >=20 > 5. In testing, it seemed fine to remove the SLOG from a live/running = system > (zpool remove pool mirror-3). Is this in fact a safe thing to do to a > live/running system? ZFS knows that it should flush the ZIL, then = remove > the device? Is it better or necessary to shut down the system and = remove > the SLOG in "read only" mode? There was a time when you _could_not_ remove a LOG device once added to = a pool. A specific revision of the zpool added the ability to remove a = LOG device. As long as your zpool is a late enough version. zpool = upgrade -v will tell you what version / features you have. It looks like = version 19 added log removal, while 7 added the log itself :-) So if = your system and zpool are version 19 or later (and I think all FreeBSD = ZFS code is) then it is completely safe to remove a log device and = repurpose it. > 6. Am I missing something about how the SLOG and L2ARC play together = that I > would miss by running my proposed test. i.e. if I take down the SLOG = and > repurpose as an L2ARC might I be shooting myself in the foot cause the = SLOG > and L2ARC combo is much more powerful than the L2ARC alone (or SLOG = alone)? > My hope here is to see if the L2ARC improves performance, after which = I > will proceed with buying the SSD(s) for both the SLOG and L2ARC. It all depends on your workload=85 Sync Writes are helped by the = ZIL/SLOG, it does nothing for reads. ARC helps both async writes and all = reads. L2ARC helps all reads.