From owner-freebsd-stable@freebsd.org Mon Jun 11 11:01:39 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E9A1D10122E2 for ; Mon, 11 Jun 2018 11:01:38 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 6D23F7D96E for ; Mon, 11 Jun 2018 11:01:38 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 3212F10122D9; Mon, 11 Jun 2018 11:01:38 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0D4FA10122D2 for ; Mon, 11 Jun 2018 11:01:38 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f43.google.com (mail-lf0-f43.google.com [209.85.215.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2E9427D964 for ; Mon, 11 Jun 2018 11:01:37 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f43.google.com with SMTP id 36-v6so29836725lfr.11 for ; Mon, 11 Jun 2018 04:01:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=2ZIQA4B+9nlzo5tpI79PfTdPuBf2NDH0wfTbqdtITEc=; b=Y1Pi2xKnw11bEyTiwJ6WGG25Um7cI038Dji01NVnd1G/pSMCzmw/rooI8Kt8nxwTrw 2s8keAItBHTNibiEbQS4az2vGeNw03vna3k7E74Gavg1uZ8EubhspbsRluJlXblUhcF8 eA8ea9sQruWBaPdsmqf2fiLrQs6093uSnkiVhaAbBNvMLik9RolBLSDQXInPXQaAMgbL wpkm2WX2GuJ/2CRrvUjuL7/GpywgH1rxi/YwCy+Ij6Z6X/BY+nlXfWz5e4a8C4wPPGs4 VSpM5vfmrjvJjA+D9W7UyuzhEZJHTa9H4V0pFhYZkka91ZJ77+d2T/r+XSkXqDTHHeXW 7EHA== X-Gm-Message-State: APt69E0ISpwmXVILHUBZAYYZtyIzlyKQJEPy5WwIkt7Jrs+BxicDhXPI c57A++qfOyf3NAn3O6ry/8i6o0H6 X-Google-Smtp-Source: ADUXVKLIuQ3e7VdrsxCbleDFL8Vq7nvzUXborA2PYdSwmzXuYFD25iR6/GOTeptQt+6JN3QQDCKJzg== X-Received: by 2002:a2e:6548:: with SMTP id z69-v6mr8924901ljb.116.1528714387377; Mon, 11 Jun 2018 03:53:07 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id t15-v6sm7362359lfd.79.2018.06.11.03.53.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 11 Jun 2018 03:53:06 -0700 (PDT) Subject: Re: Continuous crashing ZFS server To: Willem Jan Withagen , "stable@freebsd.org" References: <17446f39-97a1-8603-11a0-32176e8cb833@FreeBSD.org> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABzR5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz7CwZQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryM7BTQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAcLBfAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <100ea6d0-5cf4-1a00-0e3a-dfad6175df6c@FreeBSD.org> Date: Mon, 11 Jun 2018 13:53:05 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jun 2018 11:01:39 -0000 On 11/06/2018 13:26, Willem Jan Withagen wrote: > On 11/06/2018 12:13, Andriy Gapon wrote: >> On 08/06/2018 13:02, Willem Jan Withagen wrote: >>> My file server is crashing about every 15 minutes at the moment. >>> The panic looks like: >>> >>> Jun  8 11:48:43 zfs kernel: panic: Solaris(panic): zfs: allocating >>> allocated segment(offset=12922221670400 size=24576) >>> Jun  8 11:48:43 zfs kernel: >>> Jun  8 11:48:43 zfs kernel: cpuid = 1 >>> Jun  8 11:48:43 zfs kernel: KDB: stack backtrace: >>> Jun  8 11:48:43 zfs kernel: #0 0xffffffff80aada57 at kdb_backtrace+0x67 >>> Jun  8 11:48:43 zfs kernel: #1 0xffffffff80a6bb36 at vpanic+0x186 >>> Jun  8 11:48:43 zfs kernel: #2 0xffffffff80a6b9a3 at panic+0x43 >>> Jun  8 11:48:43 zfs kernel: #3 0xffffffff82488192 at vcmn_err+0xc2 >>> Jun  8 11:48:43 zfs kernel: #4 0xffffffff821f73ba at zfs_panic_recover+0x5a >>> Jun  8 11:48:43 zfs kernel: #5 0xffffffff821dff8f at range_tree_add+0x20f >>> Jun  8 11:48:43 zfs kernel: #6 0xffffffff821deb06 at metaslab_free_dva+0x276 >>> Jun  8 11:48:43 zfs kernel: #7 0xffffffff821debc1 at metaslab_free+0x91 >>> Jun  8 11:48:43 zfs kernel: #8 0xffffffff8222296a at zio_dva_free+0x1a >>> Jun  8 11:48:43 zfs kernel: #9 0xffffffff8221f6cc at zio_execute+0xac >>> Jun  8 11:48:43 zfs kernel: #10 0xffffffff80abe827 at >>> taskqueue_run_locked+0x127 >>> Jun  8 11:48:43 zfs kernel: #11 0xffffffff80abf9c8 at >>> taskqueue_thread_loop+0xc8 >>> Jun  8 11:48:43 zfs kernel: #12 0xffffffff80a2f7d5 at fork_exit+0x85 >>> Jun  8 11:48:43 zfs kernel: #13 0xffffffff80ec4abe at fork_trampoline+0xe >>> Jun  8 11:48:43 zfs kernel: Uptime: 9m7s >>> >>> Maybe a known bug? >>> Is there anything I can do about this? >>> Any debugging needed? >> >> Sorry to inform you but your on-disk data got corrupted. >> The most straightforward thing you can do is try to save data from the pool in >> readonly mode. > > Hi Andriy, > > Auch, that is a first in 12 years of using ZFS. "Fortunately" it was of a test > ZVOL->iSCSI->Win10 disk on which I spool my CAMs. > > Removing the ZVOL actually fixed the rebooting, but now the question is: >     Is the remainder of the zpools on the same disks in danger? You can try to check with zdb -b on an idle (better exported) pool. And zpool scrub. -- Andriy Gapon