From owner-freebsd-questions@freebsd.org Wed Aug 28 02:57:46 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 97D8ACE623 for ; Wed, 28 Aug 2019 02:57:46 +0000 (UTC) (envelope-from vas@mpeks.tomsk.su) Received: from admin.sibptus.ru (admin.sibptus.ru [IPv6:2001:19f0:5001:21dc::10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46J9Qd3fNBz4cyN for ; Wed, 28 Aug 2019 02:57:44 +0000 (UTC) (envelope-from vas@mpeks.tomsk.su) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sibptus.ru; s=20181118; h=Message-ID:Subject:To:From:Date:In-Reply-To; bh=NHNqzsLTNkhZadzIVlAAmOetooJLGewdJ7B0ScvtpF8=; b=D3GE674eQuKPFa5ixnGBuUlqV2 epsmWKpgIVe1hturwtEzcCXyVbhWHGXeQURN5l+ujaCbrTSUfiI4kCEQmW8ApSd5OajWKg/6WSLqZ z0LecrJNllZl1qNxzWJ6/OKDghA7QcOWMi7urP1qk9KC31DbU2xuibakbkpNoIwmQGGg=; Received: from vas by admin.sibptus.ru with local (Exim 4.92.1 (FreeBSD)) (envelope-from ) id 1i2o9I-0000PZ-8h for freebsd-questions@freebsd.org; Wed, 28 Aug 2019 09:57:28 +0700 Date: Wed, 28 Aug 2019 09:57:28 +0700 From: Victor Sudakov To: freebsd-questions@freebsd.org Subject: Kernel panic and ZFS corruption on 11.3-RELEASE Message-ID: <20190828025728.GA1441@admin.sibptus.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="/9DWx/yDrRhgMJTb" Content-Disposition: inline X-PGP-Key: http://admin.sibptus.ru/~vas/ X-PGP-Fingerprint: 10E3 1171 1273 E007 C2E9 3532 0DA4 F259 9B5E C634 User-Agent: Mutt/1.12.1 (2019-06-15) Sender: Victor Sudakov X-Rspamd-Queue-Id: 46J9Qd3fNBz4cyN X-Spamd-Bar: -------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=sibptus.ru header.s=20181118 header.b=D3GE674e; dmarc=none; spf=pass (mx1.freebsd.org: domain of vas@mpeks.tomsk.su designates 2001:19f0:5001:21dc::10 as permitted sender) smtp.mailfrom=vas@mpeks.tomsk.su X-Spamd-Result: default: False [-8.41 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[sibptus.ru:s=20181118]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[tomsk.su]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(-2.82)[ip: (-8.99), ipnet: 2001:19f0:5000::/38(-4.50), asn: 20473(-0.55), country: US(-0.05)]; DKIM_TRACE(0.00)[sibptus.ru:+]; NEURAL_HAM_SHORT(-0.99)[-0.992,0]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:20473, ipnet:2001:19f0:5000::/38, country:US]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Aug 2019 02:57:46 -0000 --/9DWx/yDrRhgMJTb Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Dear Colleagues, Shortly after upgrading to 11.3-RELEASE I had a kernel panic: Aug 28 00:01:40 vas kernel: panic: solaris assert: dmu_buf_hold_array(os, o= bject, offset, size, 0, ((char *)(uintptr_t)__func__), &numbufs, &dbp) =3D= =3D 0 (0x5 =3D=3D 0x0), file: /usr/src/sys/cddl/contrib/opensolaris/uts/com= mon/fs/zfs/dmu.c, line: 1022 Aug 28 00:01:40 vas kernel: cpuid =3D 0 Aug 28 00:01:40 vas kernel: KDB: stack backtrace: Aug 28 00:01:40 vas kernel: #0 0xffffffff80b4c4d7 at kdb_backtrace+0x67 Aug 28 00:01:40 vas kernel: #1 0xffffffff80b054ee at vpanic+0x17e Aug 28 00:01:40 vas kernel: #2 0xffffffff80b05363 at panic+0x43 Aug 28 00:01:40 vas kernel: #3 0xffffffff8260322c at assfail3+0x2c Aug 28 00:01:40 vas kernel: #4 0xffffffff822a9585 at dmu_write+0xa5 Aug 28 00:01:40 vas kernel: #5 0xffffffff82302b38 at space_map_write+0x188 Aug 28 00:01:40 vas kernel: #6 0xffffffff822e31fd at metaslab_sync+0x41d Aug 28 00:01:40 vas kernel: #7 0xffffffff8230b63b at vdev_sync+0xab Aug 28 00:01:40 vas kernel: #8 0xffffffff822f776b at spa_sync+0xb5b Aug 28 00:01:40 vas kernel: #9 0xffffffff82304420 at txg_sync_thread+0x280 Aug 28 00:01:40 vas kernel: #10 0xffffffff80ac8ac3 at fork_exit+0x83 Aug 28 00:01:40 vas kernel: #11 0xffffffff80f69d6e at fork_trampoline+0xe Aug 28 00:01:40 vas kernel: Uptime: 14d3h42m57s after which the ZFS pool became corrupt: pool: d02 state: FAULTED status: The pool metadata is corrupted and the pool cannot be opened. action: Recovery is possible, but will result in some data loss. Returning the pool to its state as of =D0=B2=D1=82=D0=BE=D1=80=D0=BD=D0=B8= =D0=BA, 27 =D0=B0=D0=B2=D0=B3=D1=83=D1=81=D1=82=D0=B0 2019 =D0=B3. 23:51:20 should correct the problem. Approximately 9 minutes of data must be discarded, irreversibly. Recovery can be attempted by executing 'zpool clear -F d02'. A scrub of the pool is strongly recommended after recovery. see: http://illumos.org/msg/ZFS-8000-72 scan: resilvered 423K in 0 days 00:00:05 with 0 errors on Sat Sep 30 04:1= 2:20 2017 config: NAME STATE READ WRITE CKSUM d02 FAULTED 0 0 2 ada2.eli ONLINE 0 0 12 However, "zpool clear -F d02" results in error: cannot clear errors for d02: I/O error Do you know if there is a way to recover the data, or should I say farewell= to several hundred Gb of anime? PS I think I do have the vmcore file if someone is interested to debug the = panic. --=20 Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ --/9DWx/yDrRhgMJTb Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJdZe2YAAoJEA2k8lmbXsY07mgH/R3KXo8Ze/EQouDJVIV5dOkx j4LOFB1TNnZ6RGsJWpbJNpofxNTvdVGEzJg4Ma1+UZZCtnDfkF3Cb3YLVpvVm/al pvz6xvxewbdzy+Gr2Son3/8R00b2CA7wJPc7Wntz/syevMkQN3SFtQDTHM3NUNhm cDgDrEeXM/xAEqj9kbRziAvCdqr5sTX63bHZ28UVeYyme4skseCLKz5u1Hyyxkld v72dDtfo47/6tXxO4hZ2HNLFq6GQM4HEZi7WGqll6Hv9HhZzIOStqqF/ShrUP/Pm wtesgN4Yz/RCv2eCLHMQXLUMzDOJpW5Osf0LJ3o/JNNmBYsAmSKMAAeJHa9esMw= =tHVW -----END PGP SIGNATURE----- --/9DWx/yDrRhgMJTb--