From owner-freebsd-questions@freebsd.org Thu Dec 15 18:47:21 2016 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 311C1C81C9A for ; Thu, 15 Dec 2016 18:47:21 +0000 (UTC) (envelope-from lobo@bsd.com.br) Received: from mail-yw0-x233.google.com (mail-yw0-x233.google.com [IPv6:2607:f8b0:4002:c05::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E90C213BF for ; Thu, 15 Dec 2016 18:47:20 +0000 (UTC) (envelope-from lobo@bsd.com.br) Received: by mail-yw0-x233.google.com with SMTP id i145so18941300ywg.2 for ; Thu, 15 Dec 2016 10:47:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsd.com.br; s=capeta; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=o94nECPMLGaGEzZ38/tDtrWWECWtP3n/0QRq0pq5+eI=; b=CW5fhfJAdjpqRUeSu/6J8ZsvybdMYt/Msd90dJh7MIC23JHIlrbyw8GsJgvDHPIK0m NOIdDzjtKIoHn+5wF44tVEUaQ4xpIeO/yI51NAueQdEbT3pxduLCtPOOkLowLq8dlGLW YLAqK0qFdbMJ9TL5tWwIsQXfJA+Rz0vS8B9UU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=o94nECPMLGaGEzZ38/tDtrWWECWtP3n/0QRq0pq5+eI=; b=BEgmXTQZrIXXuIgCTDgWpvhZdMTs/x8BDViH3wq4m1WixVmz0kg0SgbOy3YAhAaOYX 9GyaaAzrf8zu7lzvN1m0TdnKQCrcitbvjCTlfRYEDKUaPS6Q18vGUkWdwDOme7kWDXOp Ql5kWLgvWetMf+jkfNpuxAAStPB4UCBKX+xEsumId4roYTd4pmflpbnF1XbwjgLjG2tv OITXeXvibVE3j50+foX5kdLIWwlXO4SAejTpwMLGadyz4e3+NnrCETtqgzahBfpl8Lit qrDrt6zjPQ7GvkPxAyInX9oty+oMY0Cf9OEHz+XmSYnz2l6T80KIdMqV4VH/dH7RF4SD GKBg== X-Gm-Message-State: AKaTC01kW2rjw48ON6UOazKf99s5I8l2EWhhgUMomo3tqiNcm5RwgfG5v4ssguBNnaTcMM/mrmLQqCd0ogIZxw== X-Received: by 10.129.85.9 with SMTP id j9mr2159898ywb.283.1481827639354; Thu, 15 Dec 2016 10:47:19 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.33.67 with HTTP; Thu, 15 Dec 2016 10:47:18 -0800 (PST) In-Reply-To: <8129aba8-acdc-c921-62ce-ed0f994cf2af@FreeBSD.org> References: <20161215111048.541d6745@Papi> <8129aba8-acdc-c921-62ce-ed0f994cf2af@FreeBSD.org> From: Mario Lobo Date: Thu, 15 Dec 2016 15:47:18 -0300 Message-ID: Subject: Re: gmirror/gstripe or ZFS? To: Matthew Seaman Cc: freebsd-questions Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2016 18:47:21 -0000 Thanks for replying, Matthew! Comments in between bellow; 2016-12-15 12:05 GMT-03:00 Matthew Seaman : > If your data is at all important to you and you aren't constrained by > running on tiny little devices with very limited system resources, then > it's a no-brainer: use ZFS. > > Creating a ZFS striped over two mirrored vdevs is not particularly > difficult and gives a result about equivalent to RAID10: > > zpool create tank -m /somewhere mirror ada0p3 ada1p3 mirror ada2p3 ada3p3 > > will create a new zpool called 'tank' and mount it at /somewhere. > There's a number of properties to fiddle with for tuning purposes, and > you'll want to create a heirarchy of ZFSes under zroot to suit your > purposes, but otherwise that's about it. > > Replacing drives in a ZFS is about as hard as replacing them in a > gmirror / gstripe setup. Swap out the physical device, create an > appropriate partitioning scheme on the new disk if needed[*], then run > 'zpool replace device-name' and wait for the pool to resilver. > > There are only two commands you need to achieve some familiarity with in > order to manage a ZFS setup -- zfs(8) and zpool(8). Don't be put off by > the length of the man pages: generally it's pretty obvious what > subcommand you need and you can just jump to that point in the manual to > find your answers. > > [*] The installer will create a zpool by using gpart partitions, so it > can also add bootcode and a swap area to each disk. If you're not going > to be booting off this pool and you have swap supplied elsewhere, then > all that is unnecessary. You can just tell ZFS to use the raw disk devices. > > Problems you may run into: > > * Not having enough RAM -- ZFS eats RAM like there's no tomorrow. > That's because of the agressive caching it employs: many IO requests > will be served out of RAM rather than having to go all the way to disk. > Sprinkling RAM liberally into your server will help performance. > > This could be a problem. I also run a couple VMs on that server. This is a 16Gram server but each VM uses 4G. How much RAM would be a good amount for ZFS? Can I limit its memory? * Do turn on compression, and use the lz4 algorithm. Compression is a > win in general due to reducing the size of IO requests, which gains more > than you lose in the extra work to compress and decompress the data. > lz4 is preferred because it gives pretty good compression for > compressible data, but can detect and bale out early for incompressible > data, like many image formats (JPG, PNG, GIF) -- in which case the data > is simply stored without compression at the ZFS level. > > Ok. > * Don't enable deduplication. It sounds really attractive, but for > almost all cases it leads to vastly increased memory requirements, > performance slowing to a near crawl, wailing, and gnashing of teeth. If > you have to ask, then you *don't* want it. > > Yes! I heard about that too! > * ZFS does a lot more processing than most filesystems -- calculating > all of those checksums, and doing all those copy-on-writes takes its > toll. It's the price you pay for being confident your data is > uncorrupted, but it does mean ZFS is harder on the system than many > other FSes. For a modern server, the extra processing cost is generally > not a problem, and swallowed in the time it takes to access the spinning > rust. It will hurt you if your IO characteristics are a lot of small > reads / writes randomly scattered around your storage, typical of eg. a > RDBMS. > > Well, the VMs run from the separate ufs drive, and not from the pool. The ZFS pool will be used for just graphic files only. But this extra load that ZFS puts on the system is exactly my main concern. I ran gmirror/gstripe on other systems and the load is practically not affected. > * You can add a 'SLOG' (Separate LOG) device to improve performance -- > this is typically a fast SSD. Doesn't have to be particularly big: all > it does is move some particularly hot IO caches off the main drives onto > the faster hardware. Can be used for ARC (reading data) or ZIL (writing > data) or both. However, you can add this on the fly without any > interruption of service, so I'd recommend starting without and only > adding one if it seems you need it. > > * Having both UFS and ZFS on the same machine. This is not > insurmountably bad, but the different memory requirements of the two > filesystems can lead to performance trouble. It depends on what your > server load levels are like. If it's lightly loaded, then no problem. > > Exactly my concern. This is not file-server only machine. It has other roles too. > Cheers, > > Matthew > > > For these reasons, I am leaning toward the gmirror/gstripe solution. -- Mario Lobo http://www.mallavoodoo.com.br FreeBSD since version 2.2.8 [not Pro-Audio.... YET!!] (99,7% winfoes FREE)