From owner-freebsd-questions@freebsd.org Thu Aug 29 05:16:10 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C64CEC9E0E for ; Thu, 29 Aug 2019 05:16:10 +0000 (UTC) (envelope-from mafsys1234@gmail.com) Received: from mail-pg1-x541.google.com (mail-pg1-x541.google.com [IPv6:2607:f8b0:4864:20::541]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46JrRt0txDz4CZY for ; Thu, 29 Aug 2019 05:16:09 +0000 (UTC) (envelope-from mafsys1234@gmail.com) Received: by mail-pg1-x541.google.com with SMTP id w10so940077pgj.7 for ; Wed, 28 Aug 2019 22:16:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=GKPVKunxWsd0UcIHYuoRcowyoGugsSBtE+x12TZzMko=; b=R+yiPfWlrqb8Tmcn4G+L7pVzH+CyV5cNRSlGMCXv5AMj5Ff/9JujHmtlQRd2TC4glX Qvtns51phtdBucoybyo5puhJSg061YF1dT5W2HViQHp7s7SzBhn+V0rVJaWtzFoDX7vg JoizavnElhmcb2CgICFi7bqVF3USOaXiwIbooi7R6jC+tRzzTKbesgHAoq6YgryNp1WS D2+TM7XFLXY4Bq5/OEuWfu1Sh26LCglvlkN9P+JZBb77KTcl+/Ib7r5fECw9r8gabaTu V6cUP8ObYKHDoMaYEexmc/DSVRKCYoTlkCpdaCieOy95xNJU8k/qtIBfmcEcJ7sAjx/H GHyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=GKPVKunxWsd0UcIHYuoRcowyoGugsSBtE+x12TZzMko=; b=Pi8dN6y31yEa09nNqugsP1ri8+9PEABG7k7fsGye3bzByugF62JKIXwlxhXyFNhjxl dC0Qt7en4CvIvkisOT2rAgy6CVI1WR3o3fh2hXWBCuj43Eyvi505VRnDtjwvoaSMY0iz rv/qRnUSvPdmPLn7w4ggdW2fh2QVNcIG2USc1QJlGfjOmK8pptoldipo70GtvdJMsmEQ QRRqJpzpwrwb4QZGbBBAB52cdvD6eU9+o56KLulIESOqwD+L64xc7TILGROFlu+g+LVt 2CTkXQ1Xu4jFMLGvL2alGFiFpklAyjthcI+qDqP9W99IqwfsYL8SCdZdOrw6vXhDgsKz rnWw== X-Gm-Message-State: APjAAAWMyKpORIi95Khh63yHA3PJyu9P2aTbGp4FMsGgVKnE3dTZbGB4 BufOTSJkV4niCkkR/vO/FEGGqMIg X-Google-Smtp-Source: APXvYqzm5ZQSPeuq6gfltq9fkhKEBJqbecZrOIi6+7FTHEex1tZIt1mY2yiKnh6MYg+JSkijnIUhpQ== X-Received: by 2002:a17:90a:7783:: with SMTP id v3mr7681646pjk.109.1567055768682; Wed, 28 Aug 2019 22:16:08 -0700 (PDT) Received: from [192.168.1.10] (C-59-101-129-12.mel.connect.net.au. [59.101.129.12]) by smtp.gmail.com with ESMTPSA id a189sm1717554pfa.60.2019.08.28.22.16.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Aug 2019 22:16:08 -0700 (PDT) Subject: Re: Kernel panic and ZFS corruption on 11.3-RELEASE To: Victor Sudakov , freebsd-questions@freebsd.org References: <20190828025728.GA1441@admin.sibptus.ru> From: MJ Message-ID: <2964dd94-ad99-d0b8-c5d8-5d276cf02d06@gmail.com> Date: Thu, 29 Aug 2019 15:15:58 +1000 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <20190828025728.GA1441@admin.sibptus.ru> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 46JrRt0txDz4CZY X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=R+yiPfWl; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mafsys1234@gmail.com designates 2607:f8b0:4864:20::541 as permitted sender) smtp.mailfrom=mafsys1234@gmail.com X-Spamd-Result: default: False [-3.99 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.99)[-0.994,0]; RECEIVED_SPAMHAUS_PBL(0.00)[12.129.101.59.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[1.4.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(0.00)[ip: (-0.47), ipnet: 2607:f8b0::/32(-2.85), asn: 15169(-2.32), country: US(-0.05)]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 05:16:10 -0000 On 28/08/2019 12:57 pm, Victor Sudakov wrote: > Dear Colleagues, > > Shortly after upgrading to 11.3-RELEASE I had a kernel panic: > > Aug 28 00:01:40 vas kernel: panic: solaris assert: dmu_buf_hold_array(os, object, offset, size, 0, ((char *)(uintptr_t)__func__), &numbufs, &dbp) == 0 (0x5 == 0x0), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c, line: 1022 > Aug 28 00:01:40 vas kernel: cpuid = 0 > Aug 28 00:01:40 vas kernel: KDB: stack backtrace: > Aug 28 00:01:40 vas kernel: #0 0xffffffff80b4c4d7 at kdb_backtrace+0x67 > Aug 28 00:01:40 vas kernel: #1 0xffffffff80b054ee at vpanic+0x17e > Aug 28 00:01:40 vas kernel: #2 0xffffffff80b05363 at panic+0x43 > Aug 28 00:01:40 vas kernel: #3 0xffffffff8260322c at assfail3+0x2c > Aug 28 00:01:40 vas kernel: #4 0xffffffff822a9585 at dmu_write+0xa5 > Aug 28 00:01:40 vas kernel: #5 0xffffffff82302b38 at space_map_write+0x188 > Aug 28 00:01:40 vas kernel: #6 0xffffffff822e31fd at metaslab_sync+0x41d > Aug 28 00:01:40 vas kernel: #7 0xffffffff8230b63b at vdev_sync+0xab > Aug 28 00:01:40 vas kernel: #8 0xffffffff822f776b at spa_sync+0xb5b > Aug 28 00:01:40 vas kernel: #9 0xffffffff82304420 at txg_sync_thread+0x280 > Aug 28 00:01:40 vas kernel: #10 0xffffffff80ac8ac3 at fork_exit+0x83 > Aug 28 00:01:40 vas kernel: #11 0xffffffff80f69d6e at fork_trampoline+0xe > Aug 28 00:01:40 vas kernel: Uptime: 14d3h42m57s > > after which the ZFS pool became corrupt: > > pool: d02 > state: FAULTED > status: The pool metadata is corrupted and the pool cannot be opened. > action: Recovery is possible, but will result in some data loss. > Returning the pool to its state as of вторник, 27 августа 2019 г. 23:51:20 > should correct the problem. Approximately 9 minutes of data > must be discarded, irreversibly. Recovery can be attempted > by executing 'zpool clear -F d02'. A scrub of the pool > is strongly recommended after recovery. > see: http://illumos.org/msg/ZFS-8000-72 > scan: resilvered 423K in 0 days 00:00:05 with 0 errors on Sat Sep 30 04:12:20 2017 > config: > > NAME STATE READ WRITE CKSUM > d02 FAULTED 0 0 2 > ada2.eli ONLINE 0 0 12 > > However, "zpool clear -F d02" results in error: > cannot clear errors for d02: I/O error > > Do you know if there is a way to recover the data, or should I say farewell to several hundred Gb of anime? > > PS I think I do have the vmcore file if someone is interested to debug the panic. Do you have a backup? Then restore it. If you don't, have you tried zpool import -F d02 Some references you might like to read: https://docs.oracle.com/cd/E19253-01/819-5461/gbctt/index.html Take note of this section: "If the damaged pool is in the zpool.cache file, the problem is discovered when the system is booted, and the damaged pool is reported in the zpool status command. If the pool isn't in the zpool.cache file, it won't successfully import or open and you'll see the damaged pool messages when you attempt to import the pool." I've not had your exact error, but in the case of disk corruption/failure, I've used import as the sledgehammer approach. Regards, Mark.