From owner-freebsd-stable@freebsd.org Thu Jul 21 12:26:00 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB006B9E131 for ; Thu, 21 Jul 2016 12:26:00 +0000 (UTC) (envelope-from karl@denninger.net) Received: from mail.denninger.net (denninger.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9523F1773 for ; Thu, 21 Jul 2016 12:26:00 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (Karl-Desktop.Denninger.net [192.168.1.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.denninger.net (Postfix) with ESMTPSA id D0888895B6 for ; Thu, 21 Jul 2016 07:25:57 -0500 (CDT) Subject: Re: Panic on BETA1 in the ZFS subsystem To: freebsd-stable@freebsd.org References: <8f44bc09-1237-44d0-fe7a-7eb9cf4fe85b@denninger.net> <54e5974c-312e-c33c-ab83-9e1148618ddc@FreeBSD.org> <97cf5283-683b-83fd-c484-18c14973b065@denninger.net> From: Karl Denninger Message-ID: <1f064549-fa72-fe9b-d66d-85923437bb9b@denninger.net> Date: Thu, 21 Jul 2016 07:25:56 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms090705090700070007050505" X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jul 2016 12:26:00 -0000 This is a cryptographically signed message in MIME format. --------------ms090705090700070007050505 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 7/21/2016 03:07, Andriy Gapon wrote: > On 21/07/2016 00:54, Karl Denninger wrote: >> io_type =3D ZIO_TYPE_FREE, >> io_child_type =3D ZIO_CHILD_VDEV, >> io_cmd =3D 0, >> io_priority =3D ZIO_PRIORITY_TRIM, >> io_flags =3D 789633, >> io_stage =3D ZIO_STAGE_VDEV_IO_DONE, >> io_pipeline =3D 3080192, >> io_orig_flags =3D 525441, >> io_orig_stage =3D ZIO_STAGE_OPEN, >> io_orig_pipeline =3D 3080192, >> io_error =3D 45, >> vdev_notrim =3D 1, >> vdev_queue =3D { >> vq_vdev =3D 0xfffff804d8683000, >> vq_class =3D 0xfffff804d86833e8, >> vq_active_tree =3D { >> avl_root =3D 0xfffff80290a71240, >> avl_compar =3D 0xffffffff8220b8a0 , >> avl_offset =3D 576, >> avl_numnodes =3D 64, >> avl_size =3D 952 >> }, >> vq_read_offset_tree =3D { >> avl_root =3D 0x0, >> avl_compar =3D 0xffffffff8220b8a0 , >> avl_offset =3D 600, >> avl_numnodes =3D 0, >> avl_size =3D 952 >> }, >> vq_write_offset_tree =3D { >> avl_root =3D 0x0, >> avl_compar =3D 0xffffffff8220b8a0 , >> avl_offset =3D 600, >> avl_numnodes =3D 0, >> avl_size =3D 952 >> }, >> }, > Karl, > > thank you for the data. > Was this a freshly imported pool? Or a pool that was not written to > since the import until shortly before the crash? > The crash occurred during a backup script operating, which is (roughly) the following: zpool import -N backup (mount the pool to copy to) iterate over a list of zfs filesystems and... zfs rename fs@zfs-base fs@zfs-old zfs snapshot fs@zfs-base zfs send -RI fs@zfs-old fs@zfs-base | zfs receive -Fudv backup zfs destroy -vr fs@zfs-old The first filesystem to be done is the rootfs, that is when it panic'd, and from the traceback it appears that the Zio's in there are from the backup volume, so the answer to your question is "yes". This is a different panic that I used to get on 10.2 (the other one was always in dounmount) and the former symptom was also not immediately reproducable; whatever was blowing it up before was in-core, and a reboot would clear it. This one is not; I (foolishly) believed that the operation would succeed after the reboot and re-attempted it, only to get an immediate repeat of the same panic (with an essentially-identical traceback.) What allowed the operation to succeed was removing *all* of the snapshots (other than the base filesystem, of course) from both the source *and* backup destination zpools, then re-running the operation.=20 That causes a "base" copy to be taken (zfs snapshot fs@zfs-base and then just a straight send of that instead of an incremental), which was successful. The only thing that was odd about the zfs filesystem in question was that as a boot environment that was my roll-forward to 11.0 its "origin" was a clone of 10.2 before the install was done, so that snapshot was present in the zfs snapshot list. However, it had been present for several days without incident, so I doubt its presence was involved in the creation of the circumstances leading to the panic. --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms090705090700070007050505 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjA3MjExMjI1NTZaME8GCSqGSIb3DQEJBDFCBEAH EQee6fi3Ih92tnX4ppPvB7HFEwrbZm0CgJNQLPs6PA6YcObKUB8xWTihjDMRfK81bDUAAbzO xn1HflQdtgTvMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIAt5wtCRzK qp2P4Za/2UwMi6i7NsuZTb3FjI/cIJ/2NE1f1BnRVMW3PbTkGvaD8tQJRGivpYTccfErHxg7 Xkn6YAHvaGRg9x+2FgkASqA87OKnnyivBuRa2Z2IZ3ApbG09dUaGeUP5P35TFszhn35mMhOR WEnuROViEovk4rpGWmCa95VMt8mK0sv/bUfDckkzmbkoQ5UKENlPWl8YqpDN+SoCU3XDAimz pLyBjGms39Fp5aGv3OhTJxLRogzZcfNiVlbbHPjnbBEMF2RdpewOm80z9mXhFHpGos0DMZl7 wEyoK5hlC1kDem6eMZNcNWq5nmq72W73UF2SwuEzgzu4KOiGlfqt1COmblMKh9BInytviM8U 7t7Slg0327gf1aWTHCfDNWxWF4FBiyOtzsI07vk7a9osthNzogqjcjhGn2gVSq3KS7a8MADP IhaAnSlobldhcDs+IJuMfka4LGO+QqnacgKSNGS7CdEDRBvisdwz0tY6PnoOyAc6tap8kEHB hZ9j8u3I62igPszkaqQUVY40HhbOrWpppYQ00zORwDGxKFQRi2GEclD2aPR7NH/RViVvE7Uj Zgf+k5UM8HoQGjg9fdJuwMB5GwtKdfVv7Lmse2btw0ST8FYaQWkAwapvtW4WIsDIDb6ATJsq blXQ9DRICQFhJS+1pisWtlAPZmQAAAAAAAA= --------------ms090705090700070007050505--