From owner-freebsd-stable@freebsd.org Mon Jun 11 10:26:52 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5094100DC3E for ; Mon, 11 Jun 2018 10:26:52 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 4AFA77C01A for ; Mon, 11 Jun 2018 10:26:52 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: by mailman.ysv.freebsd.org (Postfix) id 0568F100DC0F; Mon, 11 Jun 2018 10:26:52 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E6E21100DC0A for ; Mon, 11 Jun 2018 10:26:51 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8597D7C017; Mon, 11 Jun 2018 10:26:51 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 20166403A7; Mon, 11 Jun 2018 12:26:49 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6DpKPM2h4U4N; Mon, 11 Jun 2018 12:26:48 +0200 (CEST) Received: from [192.168.11.152] (unknown [192.168.11.152]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 4C2B4403A6; Mon, 11 Jun 2018 12:26:48 +0200 (CEST) Subject: Re: Continuous crashing ZFS server To: Andriy Gapon , "stable@freebsd.org" References: <17446f39-97a1-8603-11a0-32176e8cb833@FreeBSD.org> From: Willem Jan Withagen Message-ID: Date: Mon, 11 Jun 2018 12:26:47 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: <17446f39-97a1-8603-11a0-32176e8cb833@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: nl Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jun 2018 10:26:52 -0000 On 11/06/2018 12:13, Andriy Gapon wrote: > On 08/06/2018 13:02, Willem Jan Withagen wrote: >> My file server is crashing about every 15 minutes at the moment. >> The panic looks like: >> >> Jun 8 11:48:43 zfs kernel: panic: Solaris(panic): zfs: allocating >> allocated segment(offset=12922221670400 size=24576) >> Jun 8 11:48:43 zfs kernel: >> Jun 8 11:48:43 zfs kernel: cpuid = 1 >> Jun 8 11:48:43 zfs kernel: KDB: stack backtrace: >> Jun 8 11:48:43 zfs kernel: #0 0xffffffff80aada57 at kdb_backtrace+0x67 >> Jun 8 11:48:43 zfs kernel: #1 0xffffffff80a6bb36 at vpanic+0x186 >> Jun 8 11:48:43 zfs kernel: #2 0xffffffff80a6b9a3 at panic+0x43 >> Jun 8 11:48:43 zfs kernel: #3 0xffffffff82488192 at vcmn_err+0xc2 >> Jun 8 11:48:43 zfs kernel: #4 0xffffffff821f73ba at zfs_panic_recover+0x5a >> Jun 8 11:48:43 zfs kernel: #5 0xffffffff821dff8f at range_tree_add+0x20f >> Jun 8 11:48:43 zfs kernel: #6 0xffffffff821deb06 at metaslab_free_dva+0x276 >> Jun 8 11:48:43 zfs kernel: #7 0xffffffff821debc1 at metaslab_free+0x91 >> Jun 8 11:48:43 zfs kernel: #8 0xffffffff8222296a at zio_dva_free+0x1a >> Jun 8 11:48:43 zfs kernel: #9 0xffffffff8221f6cc at zio_execute+0xac >> Jun 8 11:48:43 zfs kernel: #10 0xffffffff80abe827 at >> taskqueue_run_locked+0x127 >> Jun 8 11:48:43 zfs kernel: #11 0xffffffff80abf9c8 at >> taskqueue_thread_loop+0xc8 >> Jun 8 11:48:43 zfs kernel: #12 0xffffffff80a2f7d5 at fork_exit+0x85 >> Jun 8 11:48:43 zfs kernel: #13 0xffffffff80ec4abe at fork_trampoline+0xe >> Jun 8 11:48:43 zfs kernel: Uptime: 9m7s >> >> Maybe a known bug? >> Is there anything I can do about this? >> Any debugging needed? > > Sorry to inform you but your on-disk data got corrupted. > The most straightforward thing you can do is try to save data from the pool in > readonly mode. Hi Andriy, Auch, that is a first in 12 years of using ZFS. "Fortunately" it was of a test ZVOL->iSCSI->Win10 disk on which I spool my CAMs. Removing the ZVOL actually fixed the rebooting, but now the question is: Is the remainder of the zpools on the same disks in danger? --WjW