From owner-freebsd-fs@freebsd.org Wed Feb 26 17:09:53 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7297C25DBD6 for ; Wed, 26 Feb 2020 17:09:53 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48SMjq3Rlkz4Bqw for ; Wed, 26 Feb 2020 17:09:51 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id CB1FF43173; Wed, 26 Feb 2020 18:09:40 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6B3PXpfRAY1q; Wed, 26 Feb 2020 18:09:40 +0100 (CET) Received: from [192.168.101.70] (unknown [192.168.101.70]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 38D9543172 for ; Wed, 26 Feb 2020 18:09:40 +0100 (CET) To: FreeBSD Filesystems From: Willem Jan Withagen Subject: ZFS pools in "trouble" Message-ID: <71e1f22a-1261-67d9-e41d-0f326bf81469@digiware.nl> Date: Wed, 26 Feb 2020 18:09:40 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0 Thunderbird/68.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 48SMjq3Rlkz4Bqw X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 2001:4cb8:90:ffff::3 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-5.38 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[digiware.nl]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; IP_SCORE(-3.08)[ip: (-9.52), ipnet: 2001:4cb8::/29(-4.67), asn: 28878(-1.26), country: NL(0.03)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:2001:4cb8::/29, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Feb 2020 17:09:53 -0000 Hi, I'm using my pools in perhaps a rather awkward way as underlying storage for my ceph cluster: 1 disk per pool, with log and cache on SSD For one reason or another one of the servers has crashed ad does not really want to read several of the pools: ---- pool: osd_2 state: UNAVAIL Assertion failed: (reason == ZPOOL_STATUS_OK), file /usr/src/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c, line 5098. Abort (core dumped) ---- The code there is like: ---- default: /* * The remaining errors can't actually be generated, yet. */ assert(reason == ZPOOL_STATUS_OK); ---- And this on already 3 disks. Running: FreeBSD 12.1-STABLE (GENERIC) #0 r355208M: Fri Nov 29 10:43:47 CET 2019 Now this is a test cluster, so no harm there in matters of data loss. And the ceph cluster probably can rebuild everything if I do not lose too many disk. But the problem also lies in the fact that not all disk are recognized by the kernel, and not all disk end up mounted. So I need to remove a pool first to get more disks online. Is there anything I can do the get them back online? Or is this a lost cause? --WjW