From owner-freebsd-fs@freebsd.org Sun Mar 8 18:34:54 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ACDE7271BAD for ; Sun, 8 Mar 2020 18:34:54 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [IPv6:2001:6b0:17:f0a0::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48b94q5NK5z4CRJ for ; Sun, 8 Mar 2020 18:34:51 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 0508640002 for ; Sun, 8 Mar 2020 19:34:46 +0100 (CET) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id E6F4540008; Sun, 8 Mar 2020 19:34:45 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [192.168.1.132] (h-201-140.A785.priv.bahnhof.se [98.128.201.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 5731C40002 for ; Sun, 8 Mar 2020 19:34:45 +0100 (CET) From: Peter Eriksson Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Ways to "pause" ZFS resilver? Message-Id: Date: Sun, 8 Mar 2020 19:34:44 +0100 To: freebsd-fs X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 48b94q5NK5z4CRJ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 2001:6b0:17:f0a0::3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-2.97 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.95)[-0.949,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; RCVD_IN_DNSWL_NONE(0.00)[3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.a.0.f.7.1.0.0.0.b.6.0.1.0.0.2.list.dnswl.org : 127.0.11.0]; MV_CASE(0.50)[]; IP_SCORE(-1.72)[ip: (-6.72), ipnet: 2001:6b0::/32(-1.04), asn: 1653(-0.83), country: EU(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:1653, ipnet:2001:6b0::/32, country:EU]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Mar 2020 18:34:54 -0000 I=E2=80=99m looking for ideas on how to pause a running ZFS resilver on = a FreeBSD 11.3-RELEASE-p6 system. The reason is we have a system where a running such causes severe NFS = =E2=80=9Chiccups=E2=80=9D for our users (like 5-20s delays more or less = often) and thus I=E2=80=99d like to figure out some way to =E2=80=9Cpause=E2= =80=9D it during office hours until either we=E2=80=99ve found and fixed = the problem or the resilver is done (1D15H to go)... Since there isn=E2=80=99t any =E2=80=9Czfs=E2=80=9D command to pause a = running resilver I=E2=80=99m pondering alternative more =E2=80=9Ccreative=E2= =80=9D ways. /usr/src/cddl/contrib/opensolaris/uts/common/fs/zfs: > if (zio_flags & ZIO_FLAG_RESILVER) > scan_delay =3D zfs_resilver_delay; > else { > ASSERT(zio_flags & ZIO_FLAG_SCRUB); > scan_delay =3D zfs_scrub_delay; > } > > if (scan_delay && (ddi_get_lbolt64() - spa->spa_last_io <=3D = zfs_scan_idle)) > delay(MAX((int)scan_delay, 0)); Settings vfs.zfs.scan_idle to something high and then = vfs.zfs.resilver_delay to 10*60*60*kern.hz (10 hours) and hoping the = =E2=80=9Cif" statement will trigger? But that assumes nothing can/will = interrupt delay(). Hmmm... Any other suggestions? (I don=E2=80=99t want to abort the resilver). - Peter From owner-freebsd-fs@freebsd.org Sun Mar 8 18:38:10 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A9871271D82 for ; Sun, 8 Mar 2020 18:38:10 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x736.google.com (mail-qk1-x736.google.com [IPv6:2607:f8b0:4864:20::736]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48b98d1JHPz4LnX for ; Sun, 8 Mar 2020 18:38:08 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x736.google.com with SMTP id b5so7230066qkh.8 for ; Sun, 08 Mar 2020 11:38:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=a7R2IXBEuIMTJgXF+IwLh6gKZ93fL2Gs5MLEfkSPhtQ=; b=aM7ypu5VbGVnB5qWBh1b8r+wX8HsfE0F/7avtYp07QU5Ro7lSNRgQesayJaIR2sn7c qUxUFoAVtX0KeXPm881UB1w7i9/uI9OxFvOxqKkVx85QUSd6XIZB7FAYz0Kh1pRFk/J6 dUIBmJEP+DqVsG4TLDIXXJpJwdEegG5jchWXMivqzRj6TJXoMSnEhvXcOYccoLJELUo9 GYnPAxq8tUdGYcUrAwWtEPj7EbOnjDKUGP6JzO/2k0VMYf4IoEQgC+fkYMgmtH0RkBD8 GoR4fum6X8npAqPnwv3OLwgvALywnXvg8Vys9sZfhgWCmXp0hcQfRxSF5oUFiRHp74iT Mu7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=a7R2IXBEuIMTJgXF+IwLh6gKZ93fL2Gs5MLEfkSPhtQ=; b=swoU/5DF5Ah3vQdRbokqHMiURWDHKgn9AujT8IvCh0+zvT/jebwy8i4XFY15XiDDue sUjUtJh0c+PMg7DThAbMgHoErho5VSjBjpL1eUlwkyrT9ZKpUnj+HDKeXzS6IK4eg09r uIg7B1fhxGrI34VPlJAVPe4fMK4lGyk4tAidU+mORcxC/HgLrU1aeWg06AEKbcsWhDav E4xY/OHJ5XrjiYpg6+neRNolMMg0FjqCv8rHJEw1JdGmsdathkghZK04x1HJbwSJXUZk hOKrVCWC7DrDj+0aPv6t0Uyl+2EH/XO73w9vqzXl4aSzN2RsXGf4+TXNDcK2cBPzZl5m e7Gg== X-Gm-Message-State: ANhLgQ2stjoJaemzqE0I0BEkXc5ff2G33DHmZ/oSvevPwJ2xc2RgiXeM AcrjuhP/M/1x8qkUWbbxs0mdLs2yDJaBdKnZvOApGggO X-Google-Smtp-Source: ADFU+vuVaopuyAIuSlLZ7UZUczBq5qDdpWDVUkQumRtH5F6UNYjwhEgazxpZRVFSAqJatHmw58QSEFuZSiMP+oeXm78= X-Received: by 2002:a05:620a:848:: with SMTP id u8mr1777762qku.495.1583692687271; Sun, 08 Mar 2020 11:38:07 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Sun, 8 Mar 2020 12:37:56 -0600 Message-ID: Subject: Re: Ways to "pause" ZFS resilver? To: Peter Eriksson Cc: freebsd-fs X-Rspamd-Queue-Id: 48b98d1JHPz4LnX X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=aM7ypu5V; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::736) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-3.56 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; IP_SCORE(-2.56)[ip: (-9.24), ipnet: 2607:f8b0::/32(-1.86), asn: 15169(-1.65), country: US(-0.05)]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[6.3.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; SUBJECT_ENDS_QUESTION(1.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Mar 2020 18:38:10 -0000 On Sun, Mar 8, 2020 at 12:35 PM Peter Eriksson wrote: > I=E2=80=99m looking for ideas on how to pause a running ZFS resilver on a= FreeBSD > 11.3-RELEASE-p6 system. > > The reason is we have a system where a running such causes severe NFS > =E2=80=9Chiccups=E2=80=9D for our users (like 5-20s delays more or less o= ften) and thus I=E2=80=99d > like to figure out some way to =E2=80=9Cpause=E2=80=9D it during office h= ours until either > we=E2=80=99ve found and fixed the problem or the resilver is done (1D15H = to go)... > > Since there isn=E2=80=99t any =E2=80=9Czfs=E2=80=9D command to pause a ru= nning resilver I=E2=80=99m > pondering alternative more =E2=80=9Ccreative=E2=80=9D ways. > > /usr/src/cddl/contrib/opensolaris/uts/common/fs/zfs: > > > if (zio_flags & ZIO_FLAG_RESILVER) > > scan_delay =3D zfs_resilver_delay; > > else { > > ASSERT(zio_flags & ZIO_FLAG_SCRUB); > > scan_delay =3D zfs_scrub_delay; > > } > > > > if (scan_delay && (ddi_get_lbolt64() - spa->spa_last_io <=3D > zfs_scan_idle)) > > delay(MAX((int)scan_delay, 0)); > > Settings vfs.zfs.scan_idle to something high and then > vfs.zfs.resilver_delay to 10*60*60*kern.hz (10 hours) and hoping the =E2= =80=9Cif" > statement will trigger? But that assumes nothing can/will interrupt > delay(). Hmmm... > > Any other suggestions? > > (I don=E2=80=99t want to abort the resilver). > If you are dealing with SSDs, you might look to see if BIO_DELETE (trim) traffic is causing delays. If so, you can temporarily disable TRIM on the disk being resilvered. In the resilver case, trim doesn't help much anyway since you're rewriting the entire drive. If not, then I'm not sure what else to recommend... Warner From owner-freebsd-fs@freebsd.org Sun Mar 8 19:01:00 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CAC582726AA for ; Sun, 8 Mar 2020 19:01:00 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48b9fy0HmYz4MVd for ; Sun, 8 Mar 2020 19:00:57 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 07F4F40002 for ; Sun, 8 Mar 2020 20:00:55 +0100 (CET) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id E530440008; Sun, 8 Mar 2020 20:00:54 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, AWL, HTML_MESSAGE autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [IPv6:2001:9b1:28ff:d901:8541:7a72:29d1:9a64] (unknown [IPv6:2001:9b1:28ff:d901:8541:7a72:29d1:9a64]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 0E9DD40002 for ; Sun, 8 Mar 2020 20:00:53 +0100 (CET) From: Peter Eriksson Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: Ways to "pause" ZFS resilver? Date: Sun, 8 Mar 2020 20:00:52 +0100 References: To: freebsd-fs In-Reply-To: Message-Id: <73CD8E11-76D7-4286-B2C4-1D3835F910B6@lysator.liu.se> X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 48b9fy0HmYz4MVd X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 130.236.254.3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-4.42 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.85)[-0.854,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MIME_TRACE(0.00)[0:+,1:+,2:~]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[3.254.236.130.list.dnswl.org : 127.0.11.2]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; MV_CASE(0.50)[]; IP_SCORE(-3.07)[ip: (-7.92), ipnet: 130.236.0.0/16(-4.11), asn: 2843(-3.28), country: SE(-0.03)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:2843, ipnet:130.236.0.0/16, country:SE]; MID_RHS_MATCH_FROM(0.00)[] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Mar 2020 19:01:00 -0000 Data drives are 12 HGST 10TB 7200rpm spinning rust=E2=80=A6 = (2xRAIDZ2(4+2)) Well, except for the log (dual Intel DC S3700) and cache (Intel 750 = Series PCIe) devices. But I=E2=80=99m not seeing any errors on those. (The NFS-hickups seem to be happening in =E2=80=9Cnfsmsleep()=E2=80=9D = for some reason. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D244665 = Dtrace output: 27 54842 nfsrvd_dorpc:entry Start 27 47273 nfsv4_lock:entry Start(lp->nfslock_lock=3D6, = iwantlock=3D0) 27 37590 nfsmsleep:entry Start(ffffffff81e9982c, = ffffffff81e998a0, 99) 27 54396 _sleep:entry Start(prio=3D99, timeo=3D0) 7 54397 _sleep:return 7171965 =C2=B5s 7 37591 nfsmsleep:return 7171972 =C2=B5s 7 47274 nfsv4_lock:return 7171979 =C2=B5s But it would be really nice to be able to have some way to temporarily = pause a running resilver while I=E2=80=99m investigating this issue) - Peter > On 8 Mar 2020, at 19:37, Warner Losh wrote: >=20 >=20 >=20 > On Sun, Mar 8, 2020 at 12:35 PM Peter Eriksson > wrote: > I=E2=80=99m looking for ideas on how to pause a running ZFS resilver = on a FreeBSD 11.3-RELEASE-p6 system. >=20 > The reason is we have a system where a running such causes severe NFS = =E2=80=9Chiccups=E2=80=9D for our users (like 5-20s delays more or less = often) and thus I=E2=80=99d like to figure out some way to =E2=80=9Cpause=E2= =80=9D it during office hours until either we=E2=80=99ve found and fixed = the problem or the resilver is done (1D15H to go)... >=20 > Since there isn=E2=80=99t any =E2=80=9Czfs=E2=80=9D command to pause a = running resilver I=E2=80=99m pondering alternative more =E2=80=9Ccreative=E2= =80=9D ways. >=20 > /usr/src/cddl/contrib/opensolaris/uts/common/fs/zfs: >=20 > > if (zio_flags & ZIO_FLAG_RESILVER) > > scan_delay =3D zfs_resilver_delay; > > else { > > ASSERT(zio_flags & ZIO_FLAG_SCRUB); > > scan_delay =3D zfs_scrub_delay; > > } > > > > if (scan_delay && (ddi_get_lbolt64() - spa->spa_last_io <=3D = zfs_scan_idle)) > > delay(MAX((int)scan_delay, 0)); >=20 > Settings vfs.zfs.scan_idle to something high and then = vfs.zfs.resilver_delay to 10*60*60*kern.hz (10 hours) and hoping the = =E2=80=9Cif" statement will trigger? But that assumes nothing can/will = interrupt delay(). Hmmm... >=20 > Any other suggestions? >=20 > (I don=E2=80=99t want to abort the resilver). >=20 > If you are dealing with SSDs, you might look to see if BIO_DELETE = (trim) traffic is causing delays. If so, you can temporarily disable = TRIM on the disk being resilvered. In the resilver case, trim doesn't = help much anyway since you're rewriting the entire drive. If not, then = I'm not sure what else to recommend... >=20 > Warner=20 From owner-freebsd-fs@freebsd.org Sat Mar 14 11:00:10 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9310A27E065 for ; Sat, 14 Mar 2020 11:00:10 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48ffjP3P3fz43Lx; Sat, 14 Mar 2020 11:00:09 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 82FB471C60; Sat, 14 Mar 2020 12:00:06 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id s2GgKviGLegh; Sat, 14 Mar 2020 12:00:05 +0100 (CET) Received: from [192.168.10.9] (vaio [192.168.10.9]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id BBD1B71C5F; Sat, 14 Mar 2020 12:00:05 +0100 (CET) Subject: Re: ZFS pools in "trouble" To: Andriy Gapon , FreeBSD Filesystems References: <71e1f22a-1261-67d9-e41d-0f326bf81469@digiware.nl> <91e1cd09-b6b8-f107-537f-ae2755aba087@FreeBSD.org> From: Willem Jan Withagen Message-ID: <15bde4a5-0a2e-9984-dfd6-fce39f079f52@digiware.nl> Date: Sat, 14 Mar 2020 12:00:05 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0 Thunderbird/68.5.0 MIME-Version: 1.0 In-Reply-To: <91e1cd09-b6b8-f107-537f-ae2755aba087@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: nl X-Rspamd-Queue-Id: 48ffjP3P3fz43Lx X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 176.74.240.9 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-5.67 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.980,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[digiware.nl]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; IP_SCORE(-3.19)[ip: (-9.78), ipnet: 176.74.224.0/19(-4.89), asn: 28878(-1.33), country: NL(0.03)]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[9.240.74.176.list.dnswl.org : 127.0.9.2]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:176.74.224.0/19, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Mar 2020 11:00:10 -0000 On 27-2-2020 09:11, Andriy Gapon wrote: > On 26/02/2020 19:09, Willem Jan Withagen wrote: >> Hi, >> >> I'm using my pools in perhaps a rather awkward way as underlying storage for my >> ceph cluster: >>     1 disk per pool, with log and cache on SSD >> >> For one reason or another one of the servers has crashed ad does not really want >> to read several of the pools: >> ---- >>   pool: osd_2 >>  state: UNAVAIL >> Assertion failed: (reason == ZPOOL_STATUS_OK), file >> /usr/src/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c, line 5098. >> Abort (core dumped) >> ---- >> >> The code there is like: >> ---- >>         default: >>                 /* >>                  * The remaining errors can't actually be generated, yet. >>                  */ >>                 assert(reason == ZPOOL_STATUS_OK); >> >> ---- >> And this on already 3 disks. >> Running: >> FreeBSD 12.1-STABLE (GENERIC) #0 r355208M: Fri Nov 29 10:43:47 CET 2019 >> >> Now this is a test cluster, so no harm there in matters of data loss. >> And the ceph cluster probably can rebuild everything if I do not lose too many >> disk. >> >> But the problem also lies in the fact that not all disk are recognized by the >> kernel, and not all disk end up mounted. So I need to remove a pool first to get >> more disks online. >> >> Is there anything I can do the get them back online? >> Or is this a lost cause? > Depends on what 'reason' is. > I mean the value of the variable. I ran into the same problem. Even though I deleted the zpool in error. Ao I augmented this code with a pringtf Error: Reason not found: 5 --WjW From owner-freebsd-fs@freebsd.org Sat Mar 14 12:40:10 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7EAE525895F for ; Sat, 14 Mar 2020 12:40:10 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48fhwk6Bh0z4YB1 for ; Sat, 14 Mar 2020 12:40:06 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 6A98840006 for ; Sat, 14 Mar 2020 13:40:02 +0100 (CET) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 5766840003; Sat, 14 Mar 2020 13:40:02 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, AWL, HTML_MESSAGE autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [IPv6:2001:9b1:28ff:d901:ece3:3af1:128f:2a35] (unknown [IPv6:2001:9b1:28ff:d901:ece3:3af1:128f:2a35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 6A72640002; Sat, 14 Mar 2020 13:40:01 +0100 (CET) From: Peter Eriksson Message-Id: <2137BCAA-03B7-4DFB-B4F3-B0B26A56D416@lysator.liu.se> Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: ZFS pools in "trouble" Date: Sat, 14 Mar 2020 13:40:00 +0100 In-Reply-To: <15bde4a5-0a2e-9984-dfd6-fce39f079f52@digiware.nl> Cc: FreeBSD Filesystems To: Willem Jan Withagen References: <71e1f22a-1261-67d9-e41d-0f326bf81469@digiware.nl> <91e1cd09-b6b8-f107-537f-ae2755aba087@FreeBSD.org> <15bde4a5-0a2e-9984-dfd6-fce39f079f52@digiware.nl> X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 48fhwk6Bh0z4YB1 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 130.236.254.3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-5.63 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[3.254.236.130.list.dnswl.org : 127.0.11.2]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:2843, ipnet:130.236.0.0/16, country:SE]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-3.13)[ip: (-8.11), ipnet: 130.236.0.0/16(-4.17), asn: 2843(-3.34), country: SE(-0.03)] Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Mar 2020 12:40:10 -0000 That probably corresponds to: ZPOOL_STATUS_BAD_GUID_SUM /* sum of device guids didn't match */ (=46rom /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs.h) Which get set in:=20 /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_status.c /* = =20 * Check that the config is complete. = =20 */ if (vs->vs_state =3D=3D VDEV_STATE_CANT_OPEN && vs->vs_aux =3D=3D VDEV_AUX_BAD_GUID_SUM) return (ZPOOL_STATUS_BAD_GUID_SUM); /usr/src/sys # find . -name '*.c' -print0 | xargs -0 fgrep = VDEV_AUX_BAD_GUID_SUM ./cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_missing.c: * = VDEV_AUX_BAD_GUID_SUM. So we pretend to succeed, knowing that we ./cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c: return = (spa_vdev_err(rvd, VDEV_AUX_BAD_GUID_SUM, ENXIO)); ./cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c: return = (spa_vdev_err(rvd, VDEV_AUX_BAD_GUID_SUM, ./cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c: = case VDEV_AUX_BAD_GUID_SUM: error =3D spa_check_for_missing_logs(spa); if (error !=3D 0) return (spa_vdev_err(rvd, VDEV_AUX_BAD_GUID_SUM, = ENXIO)); if (rvd->vdev_guid_sum !=3D spa->spa_uberblock.ub_guid_sum) { spa_load_failed(spa, "uberblock guid sum doesn't match = MOS " "guid sum (%llu !=3D %llu)", (u_longlong_t)spa->spa_uberblock.ub_guid_sum, (u_longlong_t)rvd->vdev_guid_sum); return (spa_vdev_err(rvd, VDEV_AUX_BAD_GUID_SUM, ENXIO)); } Missing the logs perhaps? - Peter >>> */ >>> assert(reason =3D=3D ZPOOL_STATUS_OK); >>>=20 >=20 > Error: Reason not found: 5 From owner-freebsd-fs@freebsd.org Sat Mar 14 14:29:52 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F2BEB25ADC0 for ; Sat, 14 Mar 2020 14:29:51 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48flMG3y6Dz47jK for ; Sat, 14 Mar 2020 14:29:45 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id A90654F6C; Sat, 14 Mar 2020 15:29:42 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IvnrGFsLerpP; Sat, 14 Mar 2020 15:29:41 +0100 (CET) Received: from [192.168.10.9] (vaio [192.168.10.9]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id D85154F6B; Sat, 14 Mar 2020 15:29:41 +0100 (CET) Subject: Re: ZFS pools in "trouble" To: Peter Eriksson Cc: FreeBSD Filesystems References: <71e1f22a-1261-67d9-e41d-0f326bf81469@digiware.nl> <91e1cd09-b6b8-f107-537f-ae2755aba087@FreeBSD.org> <15bde4a5-0a2e-9984-dfd6-fce39f079f52@digiware.nl> <2137BCAA-03B7-4DFB-B4F3-B0B26A56D416@lysator.liu.se> From: Willem Jan Withagen Message-ID: Date: Sat, 14 Mar 2020 15:29:41 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0 Thunderbird/68.5.0 MIME-Version: 1.0 In-Reply-To: <2137BCAA-03B7-4DFB-B4F3-B0B26A56D416@lysator.liu.se> Content-Language: nl X-Rspamd-Queue-Id: 48flMG3y6Dz47jK X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 176.74.240.9 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-5.68 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.983,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[digiware.nl]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[9.240.74.176.list.dnswl.org : 127.0.9.2]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-3.20)[ip: (-9.78), ipnet: 176.74.224.0/19(-4.89), asn: 28878(-1.36), country: NL(0.03)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:28878, ipnet:176.74.224.0/19, country:NL]; MID_RHS_MATCH_FROM(0.00)[] Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Mar 2020 14:29:52 -0000 On 14-3-2020 13:40, Peter Eriksson wrote: > That probably corresponds to: > >    ZPOOL_STATUS_BAD_GUID_SUM    /* sum of device guids didn't match */ > (From /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs.h) > > > Which get set in: > /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_status.c > >        /* >          * Check that the config is complete. >          */ >         if (vs->vs_state == VDEV_STATE_CANT_OPEN && >             vs->vs_aux == VDEV_AUX_BAD_GUID_SUM) >                 return (ZPOOL_STATUS_BAD_GUID_SUM); > > > /usr/src/sys # find . -name '*.c' -print0 | xargs -0 fgrep > VDEV_AUX_BAD_GUID_SUM > ./cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_missing.c:* > VDEV_AUX_BAD_GUID_SUM.  So we pretend to succeed, knowing that we > ./cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:return > (spa_vdev_err(rvd, VDEV_AUX_BAD_GUID_SUM, ENXIO)); > ./cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:return > (spa_vdev_err(rvd, VDEV_AUX_BAD_GUID_SUM, > ./cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:case > VDEV_AUX_BAD_GUID_SUM: > > >        error = spa_check_for_missing_logs(spa); >         if (error != 0) >                 return (spa_vdev_err(rvd, VDEV_AUX_BAD_GUID_SUM, ENXIO)); > >         if (rvd->vdev_guid_sum != spa->spa_uberblock.ub_guid_sum) { >                 spa_load_failed(spa, "uberblock guid sum doesn't match > MOS " >                     "guid sum (%llu != %llu)", > (u_longlong_t)spa->spa_uberblock.ub_guid_sum, > (u_longlong_t)rvd->vdev_guid_sum); >                 return (spa_vdev_err(rvd, VDEV_AUX_BAD_GUID_SUM, >                     ENXIO)); >         } > > > Missing the logs perhaps? Right that is the case here, but I was led to believe that a log failure no longer would spell desaster on the vdev??? But clearly it does in this case.... --WjW From owner-freebsd-fs@freebsd.org Sat Mar 14 15:14:17 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 020E725BE40 for ; Sat, 14 Mar 2020 15:14:17 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48fmLb4Fcmz4XP6 for ; Sat, 14 Mar 2020 15:14:15 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f196.google.com with SMTP id s13so13824626ljm.1 for ; Sat, 14 Mar 2020 08:14:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=djkD8ySo8OpxEGfdA9e/kqgEJXPxVkJE5IEITHXt4xk=; b=NIK709BvyMkTK9kEc8kMBlTv+LQ/0gF2oa9gZz52q/JRVYz27qIo6xTc7yeK3Ahe1h bSNDzPbO/V0eMq1YtpxHo/0/XoNVfMXKw6arOfzKz7RMt8QLls5xxrrwmqHQVpXESRnw OXYaWnuHMic5nsh5FCKLEM736sb/ZH6hiNdmcfokmrQGJqOlekc+ANeURXkNjaTjFWlL 0IM0VTxUu7IOh28DZHGH4RIb7hg8StX8tjHc9dUxcvRITwimTQAUXZluJzzK5aBnxBc5 JcwlJj4e4eM8BVtFnp0XSDAJfgeD4x7ILcxWIBJmKtBvEvL3PVjjFkDmvbNfri6QiowS KvAg== X-Gm-Message-State: ANhLgQ2knxb8s/48sF9cl1zpPe4l37xsB2Xrr+vkGQa90CtZLU3EieSm p9Gw916jlsBi8g9xYK1a/4/g/DUB X-Google-Smtp-Source: ADFU+vvAeNhm9snh71PdYV5PzhvockKa56CaK2eNQeZzeNAK5vhb09FV+qSFVDHz85FW8UfyD9etIg== X-Received: by 2002:a2e:91c6:: with SMTP id u6mr9637699ljg.207.1584198852416; Sat, 14 Mar 2020 08:14:12 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id u21sm1445378lfo.37.2020.03.14.08.14.10 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 14 Mar 2020 08:14:11 -0700 (PDT) Subject: Re: ZFS pools in "trouble" To: Willem Jan Withagen , FreeBSD Filesystems References: <71e1f22a-1261-67d9-e41d-0f326bf81469@digiware.nl> <91e1cd09-b6b8-f107-537f-ae2755aba087@FreeBSD.org> <15bde4a5-0a2e-9984-dfd6-fce39f079f52@digiware.nl> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <24916dd7-f22c-b55b-73ae-1a2bfe653f9c@FreeBSD.org> Date: Sat, 14 Mar 2020 17:14:10 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Firefox/60.0 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <15bde4a5-0a2e-9984-dfd6-fce39f079f52@digiware.nl> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 48fmLb4Fcmz4XP6 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.196 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-2.90 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.997,0]; RCVD_COUNT_THREE(0.00)[3]; MIME_TRACE(0.00)[0:+]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[196.208.85.209.list.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-0.90)[ip: (0.18), ipnet: 209.85.128.0/17(-2.97), asn: 15169(-1.65), country: US(-0.05)]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[196.208.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Mar 2020 15:14:17 -0000 On 14/03/2020 13:00, Willem Jan Withagen wrote: > On 27-2-2020 09:11, Andriy Gapon wrote: >> On 26/02/2020 19:09, Willem Jan Withagen wrote: >>> Hi, >>> >>> I'm using my pools in perhaps a rather awkward way as underlying storage for my >>> ceph cluster: >>>      1 disk per pool, with log and cache on SSD >>> >>> For one reason or another one of the servers has crashed ad does not really want >>> to read several of the pools: >>> ---- >>>    pool: osd_2 >>>   state: UNAVAIL >>> Assertion failed: (reason == ZPOOL_STATUS_OK), file >>> /usr/src/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c, line 5098. >>> Abort (core dumped) >>> ---- >>> >>> The code there is like: >>> ---- >>>          default: >>>                  /* >>>                   * The remaining errors can't actually be generated, yet. >>>                   */ >>>                  assert(reason == ZPOOL_STATUS_OK); >>> >>> ---- >>> And this on already 3 disks. >>> Running: >>> FreeBSD 12.1-STABLE (GENERIC) #0 r355208M: Fri Nov 29 10:43:47 CET 2019 >>> >>> Now this is a test cluster, so no harm there in matters of data loss. >>> And the ceph cluster probably can rebuild everything if I do not lose too many >>> disk. >>> >>> But the problem also lies in the fact that not all disk are recognized by the >>> kernel, and not all disk end up mounted. So I need to remove a pool first to get >>> more disks online. >>> >>> Is there anything I can do the get them back online? >>> Or is this a lost cause? >> Depends on what 'reason' is. >> I mean the value of the variable. > > I ran into the same problem. Even though I deleted the zpool in error. > > Ao I augmented this code with a pringtf > > Error: Reason not found: 5 It seems that 5 is ZPOOL_STATUS_BAD_GUID_SUM and there is a discrepancy between what the code in status_callback() expects and what actually happens. Looks like check_status() can actually return ZPOOL_STATUS_BAD_GUID_SUM: /* * Check that the config is complete. */ if (vs->vs_state == VDEV_STATE_CANT_OPEN && vs->vs_aux == VDEV_AUX_BAD_GUID_SUM) return (ZPOOL_STATUS_BAD_GUID_SUM); I think that VDEV_AUX_BAD_GUID_SUM typically means that a device is missing from the pool. E.g., a log device. Or there is some other discrepancy between expected pool vdevs and found pool vdevs. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Sat Mar 14 15:37:04 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7784525C728 for ; Sat, 14 Mar 2020 15:37:04 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48fmrs0WJcz4RX6 for ; Sat, 14 Mar 2020 15:37:00 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id E5A6440005 for ; Sat, 14 Mar 2020 16:36:50 +0100 (CET) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id D03B040003; Sat, 14 Mar 2020 16:36:50 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, AWL, HTML_MESSAGE autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [IPv6:2001:9b1:28ff:d901:ece3:3af1:128f:2a35] (unknown [IPv6:2001:9b1:28ff:d901:ece3:3af1:128f:2a35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id D87EF40003 for ; Sat, 14 Mar 2020 16:36:49 +0100 (CET) From: Peter Eriksson Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: ZFS/NFS hickups and some tools to monitor stuff... Message-Id: Date: Sat, 14 Mar 2020 16:36:49 +0100 To: FreeBSD Filesystems X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 48fmrs0WJcz4RX6 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 130.236.254.3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-5.63 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[3.254.236.130.list.dnswl.org : 127.0.11.2]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; MV_CASE(0.50)[]; IP_SCORE(-3.13)[ip: (-8.11), ipnet: 130.236.0.0/16(-4.17), asn: 2843(-3.34), country: SE(-0.03)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:2843, ipnet:130.236.0.0/16, country:SE]; MID_RHS_MATCH_FROM(0.00)[] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Mar 2020 15:37:04 -0000 The last couple of weeks I=E2=80=99ve been fighting with a severe case = of NFS users complaining about slow response times from our (5) FreeBSD = 11.3-RELEASE-p6 file servers. Now even though our SMB (Windows) users = (thankfully since they are like 500 per server vs 50 NFS users) didn=E2=80= =99t see the same slowdown (or atleast didn=E2=80=99t complain about it) = the root cause is probably ZFS-related. We=E2=80=99ve identified a number of cases where some ZFS operation can = cause severe slowdown of NFS operations, and I=E2=80=99ve been trying to = figure our what is the cause and ways to mitigate the problem=E2=80=A6 Some operations that have caused issues: 1. Resilver (basically made NFS service useless during the week it = took=E2=80=A6) with response time for NFS operations regularity up to 10 = seconds or more (vs the normal 1-10ms) 2. Snapshot recursive deferred destruction (=E2=80=9Czfs destroy -dr = DATA@snapnam=E2=80=9D). Especially bad together with filesystems at or = near quota. 3. Rsync cloning of data into the servers. Response times up to 15 = minutes was seen=E2=80=A6 Yes, 15 minutes to do a mkdir(=E2=80=9Ctest-dir=E2= =80=9D). Possibly in conjunction with #1 above=E2=80=A6. Previously #1 and #2 hasn=E2=80=99t caused that much problems, and #3 = definitely. Something has changed the last half year or so but so far I = haven=E2=80=99t been able to figure it out. In order to test/figure things out I=E2=80=99ve written a couple of = tools that other might be finding useful: A couple of Dtrace scripts to watch and monitor timing of kernel calls = related to NFS, GSS & ZFS operations (the =E2=80=9Cnfsuser=E2=80=9D = script is a simple =E2=80=9Cnfs user top=E2=80=9D tool). Also a patch = that adds a sysctl than can be set and have the NFS kernel code be = verbose about slow NFS operations: - https://github.com/ptrrkssn/freebsd-stuff = With the kernel patch (nfsd-verbose-timing-11.3.patch or = nfsd-verbose-timing-12.1.patch) then it=E2=80=99ll look like this: (Altert if op takes more than 1000us): # sysctl vfs.nfsd.verbose_timing=3D1000 nfsrvd_dorpc(vers=3D4.1, uid=3D65534, procnum=3D9, repstat=3D0) took = 2853 =C2=B5s nfsrvd_dorpc(vers=3D4.0, uid=3D1003258, procnum=3D17, repstat=3D10009) = took 5433 =C2=B5s nfsrvd_dorpc(vers=3D4.1, uid=3D65534, procnum=3D9, repstat=3D0) took = 2026 =C2=B5s (The Dtrace scripts can do similar stuff, but just in case I wanted a = real in-kernel way to see if delays are caused by something on the = machine or something else (network delays etc). A very simple C program that basically in a loop tests some simple = filesystem operations and measures the time it takes: - https://github.com/ptrrkssn/pfst Mount a number of fileservers on /mnt/filur01, /mnt/filur02, = /mnt/filur03 etc, then run this to have It complain if an operation = takes more than 100ms): (With =E2=80=9C-v=E2=80=9D will print time time each operation takes) $ ./pfst -t100ms /mnt/filur0* 2020-03-14 10:56:36 [ 109 ms]: /mnt/filur03: = mkdir("t-omnibus-615-53202") [Time limit exceeded] 2020-03-14 11:14:22 [ 536 s ]: /mnt/filur03: = mkdir("t-omnibus-615-53729") [Time limit exceeded] 2020-03-14 11:31:47 [ 13 m ]: /mnt/filur03: = mkdir("t-omnibus-615-53965") [Time limit exceeded] 2020-03-14 11:35:44 [ 182 ms]: /mnt/filur03: = mkdir("t-omnibus-615-54201") [Time limit exceeded] 2020-03-14 12:20:03 [ 15 m ]: /mnt/filur03: = mkdir("t-omnibus-615-55908") [Time limit exceeded] 2020-03-14 12:39:09 [ 15 m ]: /mnt/filur03: = mkdir("t-omnibus-615-56103") [Time limit exceeded] 2020-03-14 12:50:58 [ 466 s ]: /mnt/filur03: = mkdir("t-omnibus-615-56344") [Time limit exceeded] With =E2=80=9C-v=E2=80=9D: 2020-03-14 16:27:48 [1349 =C2=B5s]: /mnt/filur01: = mkdir("t-omnibus-637-2") 2020-03-14 16:27:48 [ 327 =C2=B5s]: /mnt/filur01: = rmdir("t-omnibus-637-2") It would be interresting to see if others too are seeing ZFS and/or NFS = slowdowns during heavy writing operations (resilver, snapshot-destroy, = rsync)=E2=80=A6 Our DATA pools are basically 2xRAIDZ2(4+2) of 10TB 7200rpm disks + 400GB = SSD:s for ZIL + 400GB SSDs for L2ARC. 256GB RAM, configured with ARC-MAX = set to 64GB (used to be 128GB but we ran into out-of-memory with the = 500+ Samba smbd daemons that would compete for the RAM=E2=80=A6) We=E2=80=99ve tried it with and without L2ARC, and replaced the SSD:s. = Disabled TRIM. Not much difference. Tried trimming various sysctls but = no difference seen so far. Annoying problem this=E2=80=A6 - Peter From owner-freebsd-fs@freebsd.org Sat Mar 14 15:58:30 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0173325D27D for ; Sat, 14 Mar 2020 15:58:30 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f43.google.com (mail-ot1-f43.google.com [209.85.210.43]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48fnKd6HbZz400P; Sat, 14 Mar 2020 15:58:29 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-ot1-f43.google.com with SMTP id a6so13319834otb.10; Sat, 14 Mar 2020 08:58:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=gpZc/QhGDbT6JhS7wXiCy83+DJWDFO9jUR4HRJvvigk=; b=c+9A7guYJz0Wa1JJnLCwhON7TMVRmksjMd0OxP9Ec5FXbe1SQnHZAdXr2DBnsq3Hlc QXJYg8x2rY/poQLJOwOSdIWLYbAlEr+xgSmgvhs8zSQggQUi/J+K8/06n3PBmKt642gA 3YpFmzaiFrPX9U/c1rp/V+IoGaNXQJOmVXoVINxTqtiRoX+VQP6j2bE+f6tRSbhmNegt Nbp9bnrW9BJCHVFzA4sueZcFsu2jgS4ZZCjlQC2GVVamtX1iqy+MmXXS3pbJRMM4txKW s6qpvPVySF+jkhrwMyDBAJd8FIi2WMQ5/3g4bhFSxqE+BUixrc9JDURXCERTFv4rlNLR VY1Q== X-Gm-Message-State: ANhLgQ1XiSO7gtQpUuk/qUd2L7yi9WfVS2Efzg7/doDalAfHpxtkqlAz /B0UCcXHw1h5mXb/5F98fTClNf8M5+IMLYs71O+62rhb X-Google-Smtp-Source: ADFU+vuNJkXHzypTb9e6btEqGJ3Ix6iR+Ira4Xvz85pZ1kgfW+EUoXO2aaYpI06MZgTQ0CvMswYu8zESv2g2/zN+J7k= X-Received: by 2002:a9d:7497:: with SMTP id t23mr11216882otk.291.1584201508192; Sat, 14 Mar 2020 08:58:28 -0700 (PDT) MIME-Version: 1.0 References: <71e1f22a-1261-67d9-e41d-0f326bf81469@digiware.nl> <91e1cd09-b6b8-f107-537f-ae2755aba087@FreeBSD.org> <15bde4a5-0a2e-9984-dfd6-fce39f079f52@digiware.nl> <24916dd7-f22c-b55b-73ae-1a2bfe653f9c@FreeBSD.org> In-Reply-To: <24916dd7-f22c-b55b-73ae-1a2bfe653f9c@FreeBSD.org> From: Alan Somers Date: Sat, 14 Mar 2020 09:58:17 -0600 Message-ID: Subject: Re: ZFS pools in "trouble" To: Andriy Gapon Cc: Willem Jan Withagen , FreeBSD Filesystems X-Rspamd-Queue-Id: 48fnKd6HbZz400P X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Mar 2020 15:58:30 -0000 On Sat, Mar 14, 2020 at 9:14 AM Andriy Gapon wrote: > On 14/03/2020 13:00, Willem Jan Withagen wrote: > > On 27-2-2020 09:11, Andriy Gapon wrote: > >> On 26/02/2020 19:09, Willem Jan Withagen wrote: > >>> Hi, > >>> > >>> I'm using my pools in perhaps a rather awkward way as underlying > storage for my > >>> ceph cluster: > >>> 1 disk per pool, with log and cache on SSD > >>> > >>> For one reason or another one of the servers has crashed ad does not > really want > >>> to read several of the pools: > >>> ---- > >>> pool: osd_2 > >>> state: UNAVAIL > >>> Assertion failed: (reason == ZPOOL_STATUS_OK), file > >>> /usr/src/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c, line 5098. > >>> Abort (core dumped) > >>> ---- > >>> > >>> The code there is like: > >>> ---- > >>> default: > >>> /* > >>> * The remaining errors can't actually be generated, > yet. > >>> */ > >>> assert(reason == ZPOOL_STATUS_OK); > >>> > >>> ---- > >>> And this on already 3 disks. > >>> Running: > >>> FreeBSD 12.1-STABLE (GENERIC) #0 r355208M: Fri Nov 29 10:43:47 CET 2019 > >>> > >>> Now this is a test cluster, so no harm there in matters of data loss. > >>> And the ceph cluster probably can rebuild everything if I do not lose > too many > >>> disk. > >>> > >>> But the problem also lies in the fact that not all disk are recognized > by the > >>> kernel, and not all disk end up mounted. So I need to remove a pool > first to get > >>> more disks online. > >>> > >>> Is there anything I can do the get them back online? > >>> Or is this a lost cause? > >> Depends on what 'reason' is. > >> I mean the value of the variable. > > > > I ran into the same problem. Even though I deleted the zpool in error. > > > > Ao I augmented this code with a pringtf > > > > Error: Reason not found: 5 > > It seems that 5 is ZPOOL_STATUS_BAD_GUID_SUM and there is a discrepancy > between > what the code in status_callback() expects and what actually happens. > Looks like check_status() can actually return ZPOOL_STATUS_BAD_GUID_SUM: > /* > * Check that the config is complete. > */ > if (vs->vs_state == VDEV_STATE_CANT_OPEN && > vs->vs_aux == VDEV_AUX_BAD_GUID_SUM) > return (ZPOOL_STATUS_BAD_GUID_SUM); > > I think that VDEV_AUX_BAD_GUID_SUM typically means that a device is > missing from > the pool. E.g., a log device. Or there is some other discrepancy between > expected pool vdevs and found pool vdevs. > This situation can also arise if you remove a device from the pool (either physically, or because the device failed), then you change the pool's configuration, such as by replacing the missing vdev, then you reinsert the original device and reboot. ZFS may pick up the old device's label, and the guid sums won't match. Another way you can get this error is by creating a ZFS pool on a disk, then repartitioning the disk and creating a new pool. It's possible that "zpool import" will see the old, invalid label before it sees the newer valid label. I've even run into a situation where "zpool import" worked fine, but the bootloader failed, because they interrogate labels in a slightly different way. -Alan