From owner-freebsd-fs@freebsd.org Sun Jan 20 10:02:13 2019 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9AB8E14ACE6A for ; Sun, 20 Jan 2019 10:02:13 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: from mail-lj1-x233.google.com (mail-lj1-x233.google.com [IPv6:2a00:1450:4864:20::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2D53D8E9FB for ; Sun, 20 Jan 2019 10:02:12 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: by mail-lj1-x233.google.com with SMTP id s5-v6so15036042ljd.12 for ; Sun, 20 Jan 2019 02:02:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=UIyA5Nz+sAYj9SsujOK5DEClCJNEublYwUpeyPKpOJ0=; b=tyl8fsQZpq/FQ3jCNtNgJXSp5mSv6IyEmP2hEpa/ntfQXi/HjfrkEpfFL+HjZMY0nS ghIQKVEeu0/YxtQhWWz5VAuaXuXX83XErLTf33iUAlZ7MM/2Q1/lz8ob+uIaIaJVHweg 32nVtlgnYz+db+N+PGbvDyQzsQAiZJrm18xN0yHrWDtjHIY5f3ERWOdPKfBIpAeGpZ7K C2V6GdyQg4cRVdDqz+nYyO2tK6eSAmpm9BMksKdX2Vn8GmcieDQgTaGCihb2R9Xq03yM r+D/F/U+v8vngt/s27OlddbGYVvJm+Gb22t4seeRqu4HF9VA4mXGAGQ9Wns4nXNpr2a7 WdoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=UIyA5Nz+sAYj9SsujOK5DEClCJNEublYwUpeyPKpOJ0=; b=jqPTer62YzbbCzAx7sVDMDuslOdgAkD6xg/vvjKl9ZCmULUPYW1m5LTIhAuFhKUjxw UpYe2ICRIhgDutSDmlqpJXz6cJhjHNcGJONMN84XPHGd8nhqxY2z/0ryzU8HkCBw9GxC +b8f7t2XJvypapBRFTsyQ6U8W78KWnUXphcoz0BzInk3NGPJjwPMTzc+Y7nOyU4G3tTO Xn+lMwkhp5BOZ8Gir6WAr6ZDdyDjP+NO/1VIzDj4tojM6MRom8TVU/dGtJQg5TZ9x6hF pbKRlK8m8vXLKknpVLymohwwVLehLaUPTkKOnWRQ82aCkgvebqoW+UzfqgjwsLv2IWqh yjdQ== X-Gm-Message-State: AJcUukdUigV2AJ1Bxm1pG/t4bGEB7fudDh5Dwo9tsrs102fs99GF/rb9 azMyO6OlPU+8wYxj+E1mlXiiLQu53HruLvzrlR63HQ== X-Google-Smtp-Source: ALg8bN6YGJz3qD6nBcKxzs3KhOl+DtZq/zMT8JkBKPm4hR77foTAleeK0qKC1QTIlSaKlO8e0cJ+EwolGVHNkRzKM/g= X-Received: by 2002:a2e:1552:: with SMTP id 18-v6mr15578056ljv.68.1547978528649; Sun, 20 Jan 2019 02:02:08 -0800 (PST) MIME-Version: 1.0 References: <1180280695.63420.1547910313494.JavaMail.zimbra@gausus.net> <92646202.63422.1547910433715.JavaMail.zimbra@gausus.net> <1691666278.63816.1547976245836.JavaMail.zimbra@gausus.net> In-Reply-To: <1691666278.63816.1547976245836.JavaMail.zimbra@gausus.net> From: Rich Date: Sun, 20 Jan 2019 05:01:56 -0500 Message-ID: Subject: Re: ZFS on Hardware RAID To: Maciej Jan Broniarz Cc: andy thomas , freebsd-fs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 2D53D8E9FB X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=tyl8fsQZ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rincebrain@gmail.com designates 2a00:1450:4864:20::233 as permitted sender) smtp.mailfrom=rincebrain@gmail.com X-Spamd-Result: default: False [-6.51 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[3.3.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_SHORT(-0.95)[-0.950,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-2.55)[ip: (-8.69), ipnet: 2a00:1450::/32(-2.12), asn: 15169(-1.84), country: US(-0.08)]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jan 2019 10:02:13 -0000 The performance penalty for using a puny CPU on some PCIe card shouldn't be that bad. :P You were suggesting there was a performance penalty for using SW RAID compared to HW RAID. In practice, shockingly, the CPUs we generally use for doing lots of operations fast are, in fact, pretty fast at doing lots of operations. If you're doing something that absolutely demands all the CPU power the machine can provide, maybe you'd benefit from the slower but unshared chip on the controller. Personally, given that a number of the RAID cards in PE2950s are the right age to have badcaps (source: I had several nasty surprises that were, in fact, very obvious once the system was opened and the bulging+popped capacitors were visible), I would highly recommend you use other cards, preferably HBAs. - Rich On Sun, Jan 20, 2019 at 4:24 AM Maciej Jan Broniarz wro= te: > > Hi, > > I am thinking about the scenario with ZFS on single disks configured to R= AID0 by hw raid. > Please correct me, if i'm wrong, but HW Raid uses a dedicated unit to pro= cess all RAID related work (eg. parity checks). > With ZFS the job is done by CPU. How significant is the performance loss = in that particular case? > > mjb > > > ----- Oryginalna wiadomo=C5=9B=C4=87 ----- > Od: "andy thomas" > Do: "Rich" > DW: "Maciej Jan Broniarz" , "freebsd-fs" > Wys=C5=82ane: niedziela, 20 stycze=C5=84 2019 9:45:21 > Temat: Re: ZFS on Hardware RAID > > I have to agree with your comment that hardware RAID controllers add > another layer of opaque complexity but for what it's worth, I have to > admit ZFS on h/w RAID does work and can work well in practice. > > I run a number of very busy webservers (Dell PowerEdge 2950 with LSI > MegaRAID SAS 1078 controllers) with the first two disks in RAID 1 as the > FreeBSD system disk and the remaining 4 disks configured as RAID 0 virtua= l > disks making up a ZFS RAIDz1 pool with 3 disks plus one hot spare. > With 6-10 jails running on each server, these have been running for > years with no problems at all. > > Andy > > On Sat, 19 Jan 2019, Rich wrote: > > > The two caveats I'd offer are: > > - RAID controllers add an opaque complexity layer if you have problems > > - e.g. if you're using single-disk RAID0s to make a RAID controller > > pretend to be an HBA, if the disk starts misbehaving, you have an > > additional layer of behavior (how the RAID controller interprets > > drives misbehaving and shows that to the OS) to figure out whether the > > drive is bad, the connection is loose, the controller is bad, ... > > - abstracting the redundancy away from ZFS means that ZFS can't > > recover if it knows there's a problem but the underlying RAID > > controller doesn't - that is, say you made a RAID-6, and ZFS sees some > > block fail checksum. There's not a way to say "hey that block was > > wrong, try that read again with different disks" to the controller, so > > you're just sad at data loss on your nominally "redundant" array. > > > > - Rich > > > > On Sat, Jan 19, 2019 at 11:44 AM Maciej Jan Broniarz wrote: > >> > >> Hi, > >> > >> I want to use ZFS on a hardware-raid array. I have no option of making= it JBOD. I know it is best to use ZFS on JBOD, but > >> that possible in that particular case. My question is - how bad of an = idea is it. I have read very different opinions on that subject, but none o= f them seems conclusive. > >> > >> Any comments and especially case studies are most welcome. > >> All best, > >> mjb > >> _______________________________________________ > >> freebsd-fs@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > > > ---------------------------- > Andy Thomas, > Time Domain Systems > > Tel: +44 (0)7866 556626 > Fax: +44 (0)20 8372 2582 > http://www.time-domain.co.uk