From owner-freebsd-stable@freebsd.org Wed Jan 11 14:40:04 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B9B5CAAEF3 for ; Wed, 11 Jan 2017 14:40:04 +0000 (UTC) (envelope-from karl@denninger.net) Received: from mail.denninger.net (denninger.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 124B41FAF for ; Wed, 11 Jan 2017 14:40:03 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (Karl-Desktop.Denninger.net [192.168.1.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.denninger.net (Postfix) with ESMTPSA id EC4A61110D4 for ; Wed, 11 Jan 2017 08:31:52 -0600 (CST) To: FreeBSD-STABLE Mailing List From: Karl Denninger Subject: Ugh -- attempted to update this morning, and got a nasty panic in ZFS.... Message-ID: Date: Wed, 11 Jan 2017 08:31:41 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.6.0 MIME-Version: 1.0 Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms060807040908090103090707" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jan 2017 14:40:04 -0000 This is a cryptographically signed message in MIME format. --------------ms060807040908090103090707 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable During the reboot, immediately after the daemons started up on the machine (the boot got beyond mounting all the disks and was well into starting up all the background stuff it runs), I got a double-fault. =2E.... (there were a LOT more of this same; it pretty clearly was a recursive call sequence that ran the system out of stack space) #294 0xffffffff822fdcfd in zio_execute (zio=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1666= #295 0xffffffff8230130e in zio_vdev_io_start (zio=3D0xfffff8010c8f27b0) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3127= #296 0xffffffff822fdcfd in zio_execute (zio=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1666= #297 0xffffffff822e464d in vdev_queue_io_done (zio=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 #298 0xffffffff823014c9 in zio_vdev_io_done (zio=3D0xfffff8010cff0b88) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3152= #299 0xffffffff822fdcfd in zio_execute (zio=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1666= #300 0xffffffff8230130e in zio_vdev_io_start (zio=3D0xfffff8010cff0b88) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3127= #301 0xffffffff822fdcfd in zio_execute (zio=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1666= #302 0xffffffff822e464d in vdev_queue_io_done (zio=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 #303 0xffffffff823014c9 in zio_vdev_io_done (zio=3D0xfffff8010c962000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3152= #304 0xffffffff822fdcfd in zio_execute (zio=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1666= #305 0xffffffff8230130e in zio_vdev_io_start (zio=3D0xfffff8010c962000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3127= #306 0xffffffff822fdcfd in zio_execute (zio=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1666= #307 0xffffffff822e464d in vdev_queue_io_done (zio=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 #308 0xffffffff823014c9 in zio_vdev_io_done (zio=3D0xfffff80102175000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3152= #309 0xffffffff822fdcfd in zio_execute (zio=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1666= #310 0xffffffff80b2585a in taskqueue_run_locked (queue=3D) at /usr/src/sys/kern/subr_taskqueue.c:454 #311 0xffffffff80b26a48 in taskqueue_thread_loop (arg=3D) at /usr/src/sys/kern/subr_taskqueue.c:724 #312 0xffffffff80a7eb05 in fork_exit ( callout=3D0xffffffff80b26960 , arg=3D0xfffff800b8824c30, frame=3D0xfffffe0667430c00) at /usr/src/sys/kern/kern_fork.c:1040 #313 0xffffffff80f87c3e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:611 #314 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb) =2E.... NewFS.denninger.net dumped core - see /var/crash/vmcore.3 Wed Jan 11 08:15:33 CST 2017 FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #14 r311927M: Wed Ja n 11 07:55:20 CST 2017 =20 karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64 panic: double fault GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you = are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for detail= s. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal double fault rip =3D 0xffffffff822e3c5d rsp =3D 0xfffffe066742af90 rbp =3D 0xfffffe066742b420 cpuid =3D 15; apic id =3D 35 panic: double fault cpuid =3D 15 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0649ddee30 vpanic() at vpanic+0x186/frame 0xfffffe0649ddeeb0 panic() at panic+0x43/frame 0xfffffe0649ddef10 dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649ddef30 Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649ddef30 --- trap 0x17, rip =3D 0xffffffff822e3c5d, rsp =3D 0xfffffe066742af90, rb= p =3D 0xfffff e066742b420 --- # Work around for this CPU from 11.x eratta vm.pmap.pcid_enabled=3D0 # # # Try to avoid kernel stack exhaustion due to TRIM storms. kern.kstack_pages=3D"6" I have kstack_pages set to "6" to try to avoid another panic that I got occasionally during zfs backup operations which appeared to be linked to "too many" TRIMs, and looks very similar to this one. I rebooted back to kernel.old, which was built in October, and the machine came up normally. I'll try the newer build again and see if this was transient and related to delayed TRIM operations on the disks related to the installworld/installkernel. But if it is then it remains a problem -- and setting stackpages didn't help! I've got the dump if anything in particular would be of help. The prompt to do this in the first place was the openssh CVE that was recently issued..... --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms060807040908090103090707 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC BlwwggZYMIIEQKADAgECAgE9MA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE2MTIxODE5NDUzNVoXDTIxMTIxNzE5NDUzNVowVzEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxGzAZBgNVBAMUEmthcmxAZGVubmluZ2VyLm5ldDCCAiIwDQYJKoZIhvcNAQEBBQADggIP ADCCAgoCggIBAM2N5maxs7NkoY9g5NMxFWll0TYiO7gXrGZTo3q25ZJgNdPMwrntLz/5ewE9 07TEbwJ3ah/Ep9BfZm7JF9vTtE1HkgKtXNKi0pawNGm1Yn26Dz5AbUr1byby6dFtDJr14E07 trzDCtRRvTkOVSBj6PQPal0fAnDtkIYQBVcuMkXkuMCtyfE95pjm8g4K9l7lAcKii3T1/3rE hCc1o2nBnb7EN1/XwBeCDGB+I2SN/ftZDbKQqGAF5q9dUn+iXU7Z/CVSfUWmhVh6cVZA4Ftv TglUqj410OuPx+cUQch3h1kFgsuhQR63HiJc3HbRJllHsV0rihvL1CjeARQkhnA6uY9NLFST p5I/PfzBzW2MSmtN/tGZvmfKKnmtbfUNgkzbIR1K3lsum+yEL71kB93Xtz/4f1demEx5c8TJ RBIniDHjDeLGK1aoBu8nfnvXAvgthFNTWBOEoR49AHEPjC3kZj0l8JQml1Y8bTQD5gtC5txl klO60WV0EufU7Hy9CmynMuFtjiA2v71pm097rXeCdrAKgisdYeEESB+SFrlY65rLiLv4n8o1 PX7DqRfqKkOYIakZ0ug/yHVKcq2EM3RiJxwzls5gT70CoOBlKbrC98O8TA6teON0Jq30M06t NTI2HhvNbJDLbBH+Awf4h1UKB+0ufENwjVvF5Jfz8Ww/FaSDAgMBAAGjgfQwgfEwNwYIKwYB BQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgwCQYD VR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIBDQQf Fh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUpfAI3y+751pp9A0w 6vJHx8RoR/MwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYwFIES a2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBiB6MlugxYJdccD8boZ/u8 d8VxmLkJCtbfyYHRjYdyoABLW5hE3k3xSpYCM9L7vzWyV/UWwDYKi4ZzxHo4g+jG/GQZfKhx v38BQjL2G9xD0Hn2d+cygOq3UPjVYlbbfQoew6JbyCFXrrZ7/0jvRMLAN2+bRC7ynaFUixPH Whnj9JSH7ieYdzak8KN+G2coIC2t2iyfXVKehzi5gdNQ0vJ7+ypbGsRm4gE8Mdo9N/WgFPvZ HPFqR9Dwas7Z+aHwOabpk5r/336SyjOaZsn3MqKJQZL6GqDKusVOCWt+9uFAD8kadg7FetZe atIoD9I+zbp59oVoMnkMDMx7Hi85faU03csusqMGsjSsAzWSI1N8PJytZlchLiykokLKc3OL G87QKlErotlou7cfPX2BbEAH5wmkj9oiqZhxIL/wwAUA+PkiTbEmksKBNompSjUq/6UsR8EA s74gnu17lmijv8mrg2qMlwRirE7qG8pnE8egLtCDxcjd0Of9WMi2NJskn0/ovC7P+J60Napl m3ZIgPJst1piYSE0Zc1FIat4fFphMfK5v4iLblo1tFSlkdx1UNDGdg/U+LaXkNVXlMp8fyPm R80V6cIrCAlEWnBJNxG1UyfbbsvNMCCZBM4faGGsR/hhQOiydlruxhjL6P8J2WV8p11DdeGx KymWoil2s1J5WTGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxv cmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRww GgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5c3Rl bXMgTExDIENBAgE9MA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcN AQcBMBwGCSqGSIb3DQEJBTEPFw0xNzAxMTExNDMxNDFaME8GCSqGSIb3DQEJBDFCBECRL4ZN +LxpNeb0BsuqGt3hb82auRNiQiUQit+4RwwW70JZ3QZFda3F9oBrA+46+xyai9haslBg1slE E/iIeI/AMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggq hkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZI hvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNVBAgT B0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExM QzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3VkYSBT eXN0ZW1zIExMQyBDQQIBPTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYTAlVT MRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEg U3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0B CQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECAT0wDQYJKoZIhvcNAQEBBQAEggIAhlJJSzf02a1X wLAWkuBXOF0Kt4LXdbcISV0Eg9EjPHfG4GU/rT1y4kbTmgzm1pm3N+W/vvtUZGP1KQYIrpPN o1YVdq/eviiBOIlP/pxFrOg/RnBhcUeE1QW55jeQgvzkWrqA/pEoPkrYC3dPOrqqcCkL7Ffi ZyjqipAWiGVSiha1+wcqHKvwmZSq0yHmjb7WRWVKJkOzQC1zHdS2DRXWVMfp90LcELcQ3i8t r0Qsbo6MY/UWV6JLxTkD8mlbr/7/Q1Fw9ZpeoyozBOeik1T7CXK45mGcPhOD9Vz/NYxUvgJL BSb8c3EANmceqeQsrES9+EqOPm+9g9FmKCIGuCJSUFQTWzYQYgC+fzvrwqTQcail27YDUBvb xgv16+tIwHNFyX0UV5BZnaFdY6N129ZscHT7D16C/tij8F37eA+g6rrsKEibxPoqmCMI6JBw J8TV1QNEp6gZcCgpGC+702269oWvIiTzUYHOCmwYDo0spteMAi0sJIXvMZ6/N0NshTy3VQNr QW+0jyIJ3rRcfUWdrHy2Jum1rlvdt2DsuLzCqIYn2DevNn6LUrYQ+EeHEw39hLjDgOJFE+eZ Wea7tGMgHXVxKUS5NbjdiXl8e88nLc+TImw6OcdKuWAay5IJDf5RB0GZ+05zolfKwqjSFFyf 0BT4vp31QMTHgvmkneaI3IkAAAAAAAA= --------------ms060807040908090103090707--