From owner-freebsd-fs@freebsd.org Sat Mar 14 11:00:10 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9310A27E065 for ; Sat, 14 Mar 2020 11:00:10 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48ffjP3P3fz43Lx; Sat, 14 Mar 2020 11:00:09 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 82FB471C60; Sat, 14 Mar 2020 12:00:06 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id s2GgKviGLegh; Sat, 14 Mar 2020 12:00:05 +0100 (CET) Received: from [192.168.10.9] (vaio [192.168.10.9]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id BBD1B71C5F; Sat, 14 Mar 2020 12:00:05 +0100 (CET) Subject: Re: ZFS pools in "trouble" To: Andriy Gapon , FreeBSD Filesystems References: <71e1f22a-1261-67d9-e41d-0f326bf81469@digiware.nl> <91e1cd09-b6b8-f107-537f-ae2755aba087@FreeBSD.org> From: Willem Jan Withagen Message-ID: <15bde4a5-0a2e-9984-dfd6-fce39f079f52@digiware.nl> Date: Sat, 14 Mar 2020 12:00:05 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0 Thunderbird/68.5.0 MIME-Version: 1.0 In-Reply-To: <91e1cd09-b6b8-f107-537f-ae2755aba087@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: nl X-Rspamd-Queue-Id: 48ffjP3P3fz43Lx X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 176.74.240.9 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-5.67 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.980,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[digiware.nl]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; IP_SCORE(-3.19)[ip: (-9.78), ipnet: 176.74.224.0/19(-4.89), asn: 28878(-1.33), country: NL(0.03)]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[9.240.74.176.list.dnswl.org : 127.0.9.2]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:176.74.224.0/19, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Mar 2020 11:00:10 -0000 On 27-2-2020 09:11, Andriy Gapon wrote: > On 26/02/2020 19:09, Willem Jan Withagen wrote: >> Hi, >> >> I'm using my pools in perhaps a rather awkward way as underlying storage for my >> ceph cluster: >>     1 disk per pool, with log and cache on SSD >> >> For one reason or another one of the servers has crashed ad does not really want >> to read several of the pools: >> ---- >>   pool: osd_2 >>  state: UNAVAIL >> Assertion failed: (reason == ZPOOL_STATUS_OK), file >> /usr/src/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c, line 5098. >> Abort (core dumped) >> ---- >> >> The code there is like: >> ---- >>         default: >>                 /* >>                  * The remaining errors can't actually be generated, yet. >>                  */ >>                 assert(reason == ZPOOL_STATUS_OK); >> >> ---- >> And this on already 3 disks. >> Running: >> FreeBSD 12.1-STABLE (GENERIC) #0 r355208M: Fri Nov 29 10:43:47 CET 2019 >> >> Now this is a test cluster, so no harm there in matters of data loss. >> And the ceph cluster probably can rebuild everything if I do not lose too many >> disk. >> >> But the problem also lies in the fact that not all disk are recognized by the >> kernel, and not all disk end up mounted. So I need to remove a pool first to get >> more disks online. >> >> Is there anything I can do the get them back online? >> Or is this a lost cause? > Depends on what 'reason' is. > I mean the value of the variable. I ran into the same problem. Even though I deleted the zpool in error. Ao I augmented this code with a pringtf Error: Reason not found: 5 --WjW