From owner-freebsd-questions@freebsd.org Thu Sep 24 15:44:50 2015 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A7BB1A07AFF for ; Thu, 24 Sep 2015 15:44:50 +0000 (UTC) (envelope-from paul@kraus-haus.org) Received: from mail-qg0-f44.google.com (mail-qg0-f44.google.com [209.85.192.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 688361119 for ; Thu, 24 Sep 2015 15:44:49 +0000 (UTC) (envelope-from paul@kraus-haus.org) Received: by qgez77 with SMTP id z77so47746995qge.1 for ; Thu, 24 Sep 2015 08:44:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=6zhhCLMdyuVhj7GYWf9NGQ5YfSzDe4+EAnsC4u5WJUQ=; b=PeGkUNaJx4iwdDZwLBq+pBdXli7wU/Xx9oD1Mw5V49ongFqcnPRjLnOkwUFWw/QomB BPJ8jI8f5F1w0jUmYjsVvwhyED8u8L3tdfBT7+RvX2S7Ooei3dSITEU0xqL9L7TAGNCP 1OukZqnaBeyE4UeYl01WQjC+x9gJ30F9KwVMeKoaqtOBeJgXd+xvKDmU6FJusduBsblL O51YDk6dmtkwjZVZAeq7YruYQFgoF9tqhU7AhzHtjzmjFb0ahdG+GcR+tfEjrX2+e7mO pvln7CsHCQpse3SdAOyumQaptV1sDJjtikfNOwkS0D4RuBRsqGAHd+xPrhrow/yJBv/4 61aQ== X-Gm-Message-State: ALoCoQl9/DzC0tosTSKOhM7kZ0vGCSpXaHPgJGK8XH0AV8SzIjddRFfnMOdGD+eVlSbup4YyU4Ag X-Received: by 10.140.234.195 with SMTP id f186mr573839qhc.25.1443109482850; Thu, 24 Sep 2015 08:44:42 -0700 (PDT) Received: from mbp-1.thecreativeadvantage.com (mail.thecreativeadvantage.com. [96.236.20.34]) by smtp.gmail.com with ESMTPSA id b16sm4717657qkj.1.2015.09.24.08.44.40 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 24 Sep 2015 08:44:40 -0700 (PDT) Subject: Re: zfs performance degradation Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Content-Type: text/plain; charset=windows-1252 From: Paul Kraus In-Reply-To: <560412B2.9070905@dim.lv> Date: Thu, 24 Sep 2015 11:44:37 -0400 Cc: FreeBSD Questions Content-Transfer-Encoding: quoted-printable Message-Id: <8D1FF55C-7068-4AB6-8C0E-B4E64C1BB5FA@kraus-haus.org> References: <56019211.2050307@dim.lv> <37A37E9D-9D65-4553-BBA2-C5B032163499@kraus-haus.org> <56038054.5060906@dim.lv> <782C9CEF-BE07-4E05-83ED-133B7DA96780@kraus-haus.org> <56040150.90403@dim.lv> <60BF2FC3-0342-46C9-A718-52492303522F@kraus-haus.org> <560412B2.9070905@dim.lv> To: Dmitrijs X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Sep 2015 15:44:50 -0000 On Sep 24, 2015, at 11:11, Dmitrijs wrote: > Nope, no compression, no deduplication, only pure zfs. Even no = prefetch, as it is not recommended for machines 4Gb RAM and below. I am very surprised that ZFS is CPU limited on that system. My N54L has = less CPU performance than that and I easily get 60 MB/sec via CIFS = (Samba) from a Mac or Windows client. > I've tested performance with 40Gb file on 4Gb ram machine, so cache = should not count so much. Yup. BTW, b =3D=3D bits B =3D=3D bytes (and to be pedantic, GB =3D=3D = one billion bytes while GiB =3D=3D 2^30 bytes) > I really hoped that I could get from 2HDD MIRROR at least 1.5x read = performance of a single HDD, but it's more tricky as you explained. Yup, remember that ZFS has _lots_ of metadata per fs block and it needs = to read all of that as well as the data. > Now I'm not sure what configuration will make better performance for 4 = HDD - raid10 or raid-z2? Or two separate mirrors? Need directions for = scale things up in the future. Of all the questions you have asked that one is the easiest to answer =85 = a zpool which has 2 vdevs each of which is a 2-way mirror will have = roughly double the performance of a zpool that has one vdev that is a 4 = drive RAIDz2. Performance scales with the number of vdevs, not the = number of drives. I know that is not obvious at first, but when you look = at the design of ZFS (all top level vdevs are striped across) it makes = perfect sense. So a 2 x 2-way mirror will be faster than a 4 drive RAIDz2. At a cost, = the MTTDL (Mean Time To Data Loss) will be better for the RAIDz2 than = the 2 x 2-way mirror. See Richard Ellings post here = http://blog.richardelling.com/2010/02/zfs-data-protection-comparison.html = for a comparison of relative MTTDL for ZFS configurations. Note that I use 3-way mirrors where I need _both_ performance and = reliability and RAIDz2 where I need mostly reliability and performance = is secondary. But =85 back when I was managing lots of data (2007 - = 2012), I did use RADIz2 in production for critical data, but we had 22 = top level vdevs, each a 5 drive RAIDz2 and 10 hot spares. Striping data = across 22 RAIDz2 gave us the performance we needed with the reliability. > I've got = http://ark.intel.com/products/78867/Intel-Celeron-Processor-J1900-2M-Cache= -up-to-2_42-GHz > And 4Gb RAM. That should beat the MicroServer in terms of CPU. I had 8 GB in my N36L = and upgraded to 16 GB in my later N54L. I keep referencing these as they = are more similar to your setup than my bigger SuperMicro servers. Also = note that these MicroServers were the first generation, not the later = Gen 8. I paid about US$200 to US$300 for the bare chassis with CPU and 2 = GB RAM. The Gen 8 are much more expensive and I am looking for something = cheaper for my clients now. > Thought it would be sufficient, but now I'm in doubt. I think that 4 GB is slightly low for a file server, but it should not = be too bad. The CPU should be fine. What are the drives themselves ? = [Because with only 4 GB RAM you _will_ feel the effect of drive = performance, and it is random I/Ops that really matter for ZFS] > I can live with reduced performance for my 1st NAS, but would be nice = to have clear performance requirements in mind for planing future = storage boxes. >=20 > I see QNAPs and Synology NAS, they use like 1Ghz CPU and 1Gb of RAM = for 4 HDD, so either I'm doing it wrong, either those NASes don't have = performance (or safety?) at all. Do they calculate checksums for end-to-end data integrity ? What is their performance like ? The data integrity and reliability features of ZFS do come at a cost. > HP Proliant MicroServer is nice, but i've made my diskless system 2-3 = times cheaper (200euro vs 530/650euro), so I need a reason or = recomendation to spend x2x3 money on the thing, which specification = looks the same. As I said above, I have moved away from the MicroServer Gen 8 due to = cost. The original MicroServer was a very sweet deal and I wish I had = bought a couple of them last December when I could have gotten them for = US$200 each :-(=20 -- Paul Kraus paul@kraus-haus.org