From owner-freebsd-stable@freebsd.org Mon Apr 29 17:06:35 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5EB3D1596BC8 for ; Mon, 29 Apr 2019 17:06:35 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lf1-f52.google.com (mail-lf1-f52.google.com [209.85.167.52]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 431FA8CAF3 for ; Mon, 29 Apr 2019 17:06:34 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lf1-f52.google.com with SMTP id j20so8615934lfh.2 for ; Mon, 29 Apr 2019 10:06:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=fX4ShCtfUBTozeat+Iyqg9LM7KC9Fh5VIaKj4Ynb20s=; b=BBTs/5pG5jBGkW3Tt8wEst5BsmzPRJSmF4mV4Tk7MH0MwNb2DFBQbl7ZcVQi8VfWHE eSCMtGe3NCHcBwcotTFqo5G5oq8wgwOw4x3RrQMpxxvxhGw8AK5axwLiQTcT4aJHaF9O jhN/SVu1FeLoXJYFRrmXd51MZVdcvnubxW73rqDrE+vWo8OtOSZLCBK9mHzmSTcaNXcR SymfSN9NQialLxWCXmUEE0jSBAa/J7dTSN//6QYW42TtMF+pfEjTFyASzS/8BAHz+AG6 1fzggMVPzbQKODjQpuiFXz71XoGxKO3LeBDo3RD51Srjh5ViPSD0PtwoPBrknV0Higrc MznQ== X-Gm-Message-State: APjAAAUyybj7bC9S2nrXyMTmLZITMfbWfbWqpZchPWWPiz3Aw8hqdUQV iHn73CNE3J6zKK8HzltsmtiCJnkIB8/8HKfWjJzq085d X-Google-Smtp-Source: APXvYqzKZM2rKZnBN/E10mDqldvy3tfx9bFlcTZ9LlCzlBfdwTRiD0S/sMo3XiKUEsUI8Y8/3crOIfiYJKCkaWFJEkA= X-Received: by 2002:a19:6d1b:: with SMTP id i27mr35577535lfc.156.1556557586863; Mon, 29 Apr 2019 10:06:26 -0700 (PDT) MIME-Version: 1.0 References: <30506b3d-64fb-b327-94ae-d9da522f3a48@sorbs.net> In-Reply-To: <30506b3d-64fb-b327-94ae-d9da522f3a48@sorbs.net> From: Alan Somers Date: Mon, 29 Apr 2019 11:06:15 -0600 Message-ID: Subject: Re: ZFS... To: Michelle Sullivan Cc: freebsd-stable Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 431FA8CAF3 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.167.52 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-3.79 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_SHORT(-0.88)[-0.883,0]; MIME_TRACE(0.00)[0:+]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[52.167.85.209.list.dnswl.org : 127.0.5.0]; SUBJ_ALL_CAPS(0.45)[6]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; IP_SCORE(-1.35)[ip: (-0.60), ipnet: 209.85.128.0/17(-3.86), asn: 15169(-2.24), country: US(-0.06)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Apr 2019 17:06:35 -0000 On Mon, Apr 29, 2019 at 10:23 AM Michelle Sullivan wrote: > > I know I'm not going to be popular for this, but I'll just drop it here > anyhow. > > http://www.michellesullivan.org/blog/1726 > > Perhaps one should reconsider either: > > 1. Looking at tools that may be able to recover corrupt ZFS metadata, or > 2. Defaulting to non ZFS filesystems on install. > > -- > Michelle Sullivan > http://www.mhix.org/ Wow, losing multiple TB sucks for anybody. I'm sorry for your loss. But I want to respond to a few points from the blog post. 1) When ZFS says that "the data is always correct and there's no need for fsck", they mean metadata as well as data. The spacemap is protected in exactly the same way as all other data and metadata. (to be pedantically correct, the labels and uberblocks are protected in a different way, but still protected). The only way to get metadata corruption is due a disk failure (3-disk failure when using RAIDZ2), or due to a software bug. Sadly, those do happen, and they're devilishly tricky to track down. The difference between ZFS and older filesystems is that older filesystems experience corruption during power loss _by_design_, not merely due to software bugs. A perfectly functioning UFS implementation will experience corruption during power loss, and that's why it needs to be fscked. It's not just theoretical, either. I use UFS on my development VMs, and they frequently experience corruption after a panic (which happens all the time because I'm working on kernel code). 2) Backups are essential with any filesystem, not just ZFS. After all, no amount of RAID will protect you from an accidental "rm -rf /". 3) ZFS hotspares can be swapped in automatically, though they don't be default. It sounds like you already figured out how to assign a spare to the pool. To use it automatically, you must set the "autoreplace" pool property and enable zfsd. The latter can be done with "sysrc zfsd_enable="YES"". 4) It sounds like you're having a lot of power trouble. Have you tried sysutils/apcupsd from ports? It's fairly handy. It can talk to a wide range of UPSes, and can be configured to do stuff like send you an email on power loss, and power down the server if the battery gets too low. Better luck next time, -Alan