Date: Wed, 19 Aug 2015 11:29:44 -0400 From: PK1048 <paul@pk1048.com> To: javocado <javocado@gmail.com>, FreeBSD Filesystems <freebsd-fs@freebsd.org> Subject: Re: Optimizing performance with SLOG/L2ARC Message-ID: <023F881D-CCC5-4FCA-B09D-EB92C3BFBC03@pk1048.com> In-Reply-To: <CAP1HOmTidC3%2BG4XfhvkQxieo%2BSYMq-JWiXF9Cs4FSW2VqkktWA@mail.gmail.com> References: <CAP1HOmTidC3%2BG4XfhvkQxieo%2BSYMq-JWiXF9Cs4FSW2VqkktWA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Lots of good comments already posted, more inline below... On Aug 18, 2015, at 20:28, javocado <javocado@gmail.com> wrote: > I've been trying to optimize and enhance my ZFS filesystem performance > (running FreeBSD 8.3amd) which has been sluggish at times. Thus far I = have > added RAM (256GB) and I've added an SLOG (SSD mirror). The RAM seems = to > have helped a bit, but not sure if the SLOG was of much help. My vdev = is > decently busy, with writes and reads averaging at 100 per second with > spikes as high as 500. What is the system doing _besides_ being a file server ? =46rom the = arcstats, the ARC is not getting all of the 256 GB RAM, what else is = running ? What is the pool layout (zpool status)=85 =46rom a strictly a = performance standpoint, each vdev gives you ONE drives worth of WRITE = performance. It does not much matter if the vdev is a mirror or = RAIDz<n>. So a pool consisting of one vdev which is a 12 drive RAIDz2 = will have much slower write performance that a pool consisting of 3 = vdevs each of which is a 4 drive RAIDz2 =85 Yes, you are balancing off = capacity vs. performance vs. reliability. <snip> > 1. based on the output above, I believe a larger ARC may not = necessarily > benefit me at this point. True? The arcstats look good, are you having issues with reads or writes ? If = reads, then you need to now look at your disk layout. If writes, then = are they sync or async, if sync then a ZIL can help, if async, then more = ARC can help and you need to look at your disk layout. > 2. Is more (L2)ARC always better? ARC is _aways_ better than L2ARC. Adding L2ARC also consumes ARC space = with L2ARC metadata, and that is ARC space that cannot now be used to = cache file metadata and data. So adding L2ARC (which is slower than RAM = based ARC) _may_ actually make a system slower. > 3. I know it's a good idea to mirror the SLOG (and I have). Do I = understand > correctly that I do not need to mirror the L2ARC since it's just a = read > cache, nothing to lose if the SSD goes down? L2ARC is just read cache, so loosing it causes no data loss. You _may_ = lose performance _if_ the L2ARC was increasing your performance. > 4. Is there a better way than looking at zpool iostat -v to determine = the > SLOG utilization and usefulness? The ZIL/SLOG is used to cache SYNC write operations (if you test with = iozone, use the -o option to force sync writes). Using iostat -x you can = see the activity to the SSDs, both write and read. This can tell you how = much you are using them. I am not aware of any way to track how FULL = your ZIL/SLOG device is. Please note that, depending on your workload, an SSD may _not_ be any = faster than a HDD. I am in the process of rebuilding a file server that = exhibited poor NFS SYNC write performance. Yet it had a mirrored pair of = SSDs. Unfortunately, those SSDs had _worse_ write performance than an = HDD for small (4 KB) writes. Based on recommendations from the OpenZFS = list I have a pair of Intel 3710 SSDs coming in to try, they are = supposed to have much better write performance (at all block sizes) and = much better reliability long term (10x full disk writes per day for 5 = years). I=92ll know more once they arrive and I can test with them. Someone commented on the size of the ZIL/SLOG=85 it needs to hold all of = the write data that arrives between TXG commits, which happen at least = every 5 seconds (it used to be 30 seconds, but that scared too many = people :-). SU a sync write arrives and it _must_ be committed to = permanent storage, so ZFS writes it to the ZFS Intent Log (ZIL) which = may or may not be a separate device (vdev). When the TXG that contains = that data is committed to the pool itself the data can be flushed from = the ZIL. If your source of sync writes is network shares, and you have a = 1 Gbps link, then your maximum ZIL will be 5 seconds x 1 Gbps or 5 = Gigabits. > I'd like to test-see if adding L2ARC yields any performance boost. = Since > SLOG isn't doing much for me, I'm thinking I could easily repurpose my = SLOG > into an L2ARC. If a ZIL/SLOG device is not adding any performance, then either your = workload is NOT sync write bound -or- your ZIL/SLOG device is no faster = than your native pool performance. If the ZIL/SLOG device is not very = fast, then reusing it as an L2ARC -may- actually make the system slower. >=20 > Questions: >=20 > 5. In testing, it seemed fine to remove the SLOG from a live/running = system > (zpool remove pool mirror-3). Is this in fact a safe thing to do to a > live/running system? ZFS knows that it should flush the ZIL, then = remove > the device? Is it better or necessary to shut down the system and = remove > the SLOG in "read only" mode? There was a time when you _could_not_ remove a LOG device once added to = a pool. A specific revision of the zpool added the ability to remove a = LOG device. As long as your zpool is a late enough version. zpool = upgrade -v will tell you what version / features you have. It looks like = version 19 added log removal, while 7 added the log itself :-) So if = your system and zpool are version 19 or later (and I think all FreeBSD = ZFS code is) then it is completely safe to remove a log device and = repurpose it. > 6. Am I missing something about how the SLOG and L2ARC play together = that I > would miss by running my proposed test. i.e. if I take down the SLOG = and > repurpose as an L2ARC might I be shooting myself in the foot cause the = SLOG > and L2ARC combo is much more powerful than the L2ARC alone (or SLOG = alone)? > My hope here is to see if the L2ARC improves performance, after which = I > will proceed with buying the SSD(s) for both the SLOG and L2ARC. It all depends on your workload=85 Sync Writes are helped by the = ZIL/SLOG, it does nothing for reads. ARC helps both async writes and all = reads. L2ARC helps all reads.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?023F881D-CCC5-4FCA-B09D-EB92C3BFBC03>