From owner-freebsd-fs@freebsd.org Fri Jun 12 13:30:58 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3258C33A984 for ; Fri, 12 Jun 2020 13:30:58 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49k1ns37fCz4Vyp; Fri, 12 Jun 2020 13:30:57 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:615c:68ac:a14a:abad] ([IPv6:2607:f3e0:0:4:615c:68ac:a14a:abad]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 05CDUuHK039769 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Fri, 12 Jun 2020 09:30:56 -0400 (EDT) (envelope-from mike@sentex.net) Subject: Re: ZFS failure (vdev probe failure) To: Martin Simmons , Gordon Bergling Cc: freebsd-fs@freebsd.org References: <20200611180312.GA42266@lion.0xfce3.net> <202006121035.05CAZYmv012550@higson.cam.lispworks.com> From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= mQENBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAG0HW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+iQFUBBMBCAA+FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAlywzOYCGwMFCQHhM4AFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ eVOEFl5WrMhnPAf7Bf+ola0V9t4i8rwCMGvzkssGaxY/5zNSZO9BgSgfN0WzgmBEOy/3R4km Yn5KH94NltJYAAE5hqkFmAwK6psOqAR9cxHrRfU+gV2KO8pCDc6K/htkQcd/mclJYpCHp6Eq EVJOiAxcNaYuHZkeMdXDuvvI5Rk82VHk84BGgxIqIrhLlkguoPbXOOa+8c/Mpb1sRAGZEOuX EzKNC49+GS9gKW6ISbanyPsGEcFyP7GKMzcHBPf3cPrewZQZ6gBoNscasL6IJeAQDqzQAxbU GjO0qBSMRgnLXK7+DJlxrYdHGXqNbV6AYsmHJ6c2WWWiuRviFBqXinlgJ2FnYebZPAfWiQ== Message-ID: <6da378af-8ca5-ffc4-16f1-8da5960d1c18@sentex.net> Date: Fri, 12 Jun 2020 09:30:56 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <202006121035.05CAZYmv012550@higson.cam.lispworks.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 49k1ns37fCz4Vyp X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-0.91 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.85)[-0.849]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MIME_GOOD(-0.10)[text/plain]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; DMARC_NA(0.00)[sentex.net]; NEURAL_HAM_LONG(-0.98)[-0.983]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.08)[-0.079]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Jun 2020 13:30:58 -0000 On 6/12/2020 6:35 AM, Martin Simmons wrote: >>>>>> On Thu, 11 Jun 2020 20:03:12 +0200, Gordon Bergling said: >> Hi, >> >> has anyone seen this kind of failure on a vdev / zfs pool? >> >> Jun 11 19:17:23 lion ZFS[16825]: vdev probe failure, zpool=$boiler path=$/dev/da2 >> Jun 11 19:17:23 lion ZFS[17154]: pool I/O failure, zpool=$boiler error=$6 >> Jun 11 19:17:23 lion ZFS[17181]: catastrophic pool I/O failure, zpool=$boiler >> Jun 11 19:17:26 lion ZFS[17394]: pool I/O failure, zpool=$boiler error=$28 >> Jun 11 19:17:26 lion ZFS[17702]: pool I/O failure, zpool=$boiler error=$28 >> Jun 11 19:17:26 lion ZFS[17840]: pool I/O failure, zpool=$boiler error=$28 >> Jun 11 19:17:26 lion ZFS[18189]: pool I/O failure, zpool=$boiler error=$28 >> Jun 11 19:17:26 lion ZFS[18382]: pool I/O failure, zpool=$boiler error=$28 >> Jun 11 19:17:26 lion ZFS[18733]: pool I/O failure, zpool=$boiler error=$28 >> Jun 11 19:17:26 lion ZFS[19051]: pool I/O failure, zpool=$boiler error=$28 >> Jun 11 19:17:26 lion ZFS[19346]: catastrophic pool I/O failure, zpool=$boiler >> >> After an reboot everything seems to be okay and I do daily scrubs and none >> of them as shown an error before. >> >> --Gordon > Were there any other syslog messages about /dev/da2? It looks like it stopped > working temporarily. See if there are any incrementing  errors via smartctl  on the drive(s) in that vdev     ---Mike