Date: Wed, 11 Jan 2017 08:50:39 -0600 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: Re: Ugh -- attempted to update this morning, and got a nasty panic in ZFS.... Message-ID: <a5a4bd1e-c7e5-8d16-6398-469e1f0bb11a@denninger.net> In-Reply-To: <f05fcab3-ec17-17b3-3459-73256f35fbc7@denninger.net> References: <f05fcab3-ec17-17b3-3459-73256f35fbc7@denninger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms040201020101040609020401 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable A second attempt to come up on the new kernel was successful -- so this had to be due to queued I/Os that were pending at the time of the shutdown.... On 1/11/2017 08:31, Karl Denninger wrote: > During the reboot, immediately after the daemons started up on the > machine (the boot got beyond mounting all the disks and was well into > starting up all the background stuff it runs), I got a double-fault. > > ..... (there were a LOT more of this same; it pretty clearly was a > recursive call sequence that ran the system out of stack space) > > #294 0xffffffff822fdcfd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 66 > #295 0xffffffff8230130e in zio_vdev_io_start (zio=3D0xfffff8010c8f27b0)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 27 > #296 0xffffffff822fdcfd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 66 > #297 0xffffffff822e464d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #298 0xffffffff823014c9 in zio_vdev_io_done (zio=3D0xfffff8010cff0b88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 52 > #299 0xffffffff822fdcfd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 66 > #300 0xffffffff8230130e in zio_vdev_io_start (zio=3D0xfffff8010cff0b88)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 27 > #301 0xffffffff822fdcfd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 66 > #302 0xffffffff822e464d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #303 0xffffffff823014c9 in zio_vdev_io_done (zio=3D0xfffff8010c962000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 52 > #304 0xffffffff822fdcfd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 66 > #305 0xffffffff8230130e in zio_vdev_io_start (zio=3D0xfffff8010c962000)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 27 > #306 0xffffffff822fdcfd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 66 > #307 0xffffffff822e464d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #308 0xffffffff823014c9 in zio_vdev_io_done (zio=3D0xfffff80102175000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 52 > #309 0xffffffff822fdcfd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 66 > #310 0xffffffff80b2585a in taskqueue_run_locked (queue=3D<value optimiz= ed > out>) > at /usr/src/sys/kern/subr_taskqueue.c:454 > #311 0xffffffff80b26a48 in taskqueue_thread_loop (arg=3D<value optimize= d out>) > at /usr/src/sys/kern/subr_taskqueue.c:724 > #312 0xffffffff80a7eb05 in fork_exit ( > callout=3D0xffffffff80b26960 <taskqueue_thread_loop>, > arg=3D0xfffff800b8824c30, frame=3D0xfffffe0667430c00) > at /usr/src/sys/kern/kern_fork.c:1040 > #313 0xffffffff80f87c3e in fork_trampoline () > at /usr/src/sys/amd64/amd64/exception.S:611 > #314 0x0000000000000000 in ?? () > Current language: auto; currently minimal > (kgdb) > > ..... > > > NewFS.denninger.net dumped core - see /var/crash/vmcore.3 > > Wed Jan 11 08:15:33 CST 2017 > > FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #14 > r311927M: Wed Ja > n 11 07:55:20 CST 2017 =20 > karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP > amd64 > > panic: double fault > > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and yo= u are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for deta= ils. > This GDB was configured as "amd64-marcel-freebsd"... > > Unread portion of the kernel message buffer: > > Fatal double fault > rip =3D 0xffffffff822e3c5d > rsp =3D 0xfffffe066742af90 > rbp =3D 0xfffffe066742b420 > cpuid =3D 15; apic id =3D 35 > panic: double fault > cpuid =3D 15 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfffffe0649ddee30 > vpanic() at vpanic+0x186/frame 0xfffffe0649ddeeb0 > panic() at panic+0x43/frame 0xfffffe0649ddef10 > dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649ddef30 > Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649ddef30 > --- trap 0x17, rip =3D 0xffffffff822e3c5d, rsp =3D 0xfffffe066742af90, = rbp =3D > 0xfffff > e066742b420 --- > > # Work around for this CPU from 11.x eratta > vm.pmap.pcid_enabled=3D0 > # > # > # Try to avoid kernel stack exhaustion due to TRIM storms. > kern.kstack_pages=3D"6" > > I have kstack_pages set to "6" to try to avoid another panic that I got= > occasionally during zfs backup operations which appeared to be linked t= o > "too many" TRIMs, and looks very similar to this one. > > I rebooted back to kernel.old, which was built in October, and the > machine came up normally. I'll try the newer build again and see if > this was transient and related to delayed TRIM operations on the disks > related to the installworld/installkernel. But if it is then it remain= s > a problem -- and setting stackpages didn't help! > > I've got the dump if anything in particular would be of help. > > The prompt to do this in the first place was the openssh CVE that was > recently issued..... > > --=20 Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms040201020101040609020401 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC BlwwggZYMIIEQKADAgECAgE9MA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE2MTIxODE5NDUzNVoXDTIxMTIxNzE5NDUzNVowVzEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxGzAZBgNVBAMUEmthcmxAZGVubmluZ2VyLm5ldDCCAiIwDQYJKoZIhvcNAQEBBQADggIP ADCCAgoCggIBAM2N5maxs7NkoY9g5NMxFWll0TYiO7gXrGZTo3q25ZJgNdPMwrntLz/5ewE9 07TEbwJ3ah/Ep9BfZm7JF9vTtE1HkgKtXNKi0pawNGm1Yn26Dz5AbUr1byby6dFtDJr14E07 trzDCtRRvTkOVSBj6PQPal0fAnDtkIYQBVcuMkXkuMCtyfE95pjm8g4K9l7lAcKii3T1/3rE hCc1o2nBnb7EN1/XwBeCDGB+I2SN/ftZDbKQqGAF5q9dUn+iXU7Z/CVSfUWmhVh6cVZA4Ftv TglUqj410OuPx+cUQch3h1kFgsuhQR63HiJc3HbRJllHsV0rihvL1CjeARQkhnA6uY9NLFST p5I/PfzBzW2MSmtN/tGZvmfKKnmtbfUNgkzbIR1K3lsum+yEL71kB93Xtz/4f1demEx5c8TJ RBIniDHjDeLGK1aoBu8nfnvXAvgthFNTWBOEoR49AHEPjC3kZj0l8JQml1Y8bTQD5gtC5txl klO60WV0EufU7Hy9CmynMuFtjiA2v71pm097rXeCdrAKgisdYeEESB+SFrlY65rLiLv4n8o1 PX7DqRfqKkOYIakZ0ug/yHVKcq2EM3RiJxwzls5gT70CoOBlKbrC98O8TA6teON0Jq30M06t NTI2HhvNbJDLbBH+Awf4h1UKB+0ufENwjVvF5Jfz8Ww/FaSDAgMBAAGjgfQwgfEwNwYIKwYB BQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgwCQYD VR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIBDQQf Fh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUpfAI3y+751pp9A0w 6vJHx8RoR/MwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYwFIES a2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBiB6MlugxYJdccD8boZ/u8 d8VxmLkJCtbfyYHRjYdyoABLW5hE3k3xSpYCM9L7vzWyV/UWwDYKi4ZzxHo4g+jG/GQZfKhx v38BQjL2G9xD0Hn2d+cygOq3UPjVYlbbfQoew6JbyCFXrrZ7/0jvRMLAN2+bRC7ynaFUixPH Whnj9JSH7ieYdzak8KN+G2coIC2t2iyfXVKehzi5gdNQ0vJ7+ypbGsRm4gE8Mdo9N/WgFPvZ HPFqR9Dwas7Z+aHwOabpk5r/336SyjOaZsn3MqKJQZL6GqDKusVOCWt+9uFAD8kadg7FetZe atIoD9I+zbp59oVoMnkMDMx7Hi85faU03csusqMGsjSsAzWSI1N8PJytZlchLiykokLKc3OL G87QKlErotlou7cfPX2BbEAH5wmkj9oiqZhxIL/wwAUA+PkiTbEmksKBNompSjUq/6UsR8EA s74gnu17lmijv8mrg2qMlwRirE7qG8pnE8egLtCDxcjd0Of9WMi2NJskn0/ovC7P+J60Napl m3ZIgPJst1piYSE0Zc1FIat4fFphMfK5v4iLblo1tFSlkdx1UNDGdg/U+LaXkNVXlMp8fyPm R80V6cIrCAlEWnBJNxG1UyfbbsvNMCCZBM4faGGsR/hhQOiydlruxhjL6P8J2WV8p11DdeGx KymWoil2s1J5WTGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxv cmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRww GgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5c3Rl bXMgTExDIENBAgE9MA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcN AQcBMBwGCSqGSIb3DQEJBTEPFw0xNzAxMTExNDUwMzlaME8GCSqGSIb3DQEJBDFCBEDCIM2n cX1tewY0iP3pR4zDeqkuvjQF04VOEj84c0NB/WgkN7h4xJvjq0nLqlm8REn0tWQNhVsxImTg UoioKO1xMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggq hkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZI hvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNVBAgT B0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExM QzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3VkYSBT eXN0ZW1zIExMQyBDQQIBPTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYTAlVT MRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEg U3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0B CQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECAT0wDQYJKoZIhvcNAQEBBQAEggIAtgjmXpsrriXc o3oXYlQ07zpf371emoDkU/jGasKy6HT+uo89je3BS5ZOReKtnT3cA0Iwm5r9W//1ZegN7kTy a45o8Y9aAPZXE84EFDLnGRlIMMkb35v7f+P9V0D9ImU4UjhJxXegajNrqpnEeDRKxGwcz/ck SVwEECf21b4kgYNq1jZn2D9sJ5IGTu4d+CIUY3dzJwIzWo0DH9+MxaV5VuEIWC8Dgwj0ZSQ+ BAK31UlCLVSW8jODZRURBprIT1t2/mDBVzQkwF7izIQoUHEV2DWOYpprStj2QShhZyBeNTRn i0pW6z98yaRgGTSlSNtKW2RpldgNPixcmwfaAZoWkAASo57yz+p5gybF0h68vSE3NVGK4ltt 1Xo7a/J+IKs4f5ZZKgQJv8FKAB8M30AAxhibJLqjeJ14oGNTJCiMmdxWFHl+r+u1qbTGTVfS E7J3Wxj5UudIKEbQ8bNddt6BjrJU9Y+4qwId1eiTSksw3d58y/LKXEVYyQyUQ8jLxsezJCTM 2WLvEu9NYoyrWsBbU+wNNzzcGZKJ3MHEfziNpgYwyoThivnlBMQSZrLBByWDk97XkMNVMiab kfGEUrN0Yr6nEckYvCsSWho9ovZ4CNilYHLpcUn2jdvW+cSjtiRfEuOTUp6/3ENsiyZUSh/8 9ewJ3IeuL4PoYOriNSgmGUQAAAAAAAA= --------------ms040201020101040609020401--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?a5a4bd1e-c7e5-8d16-6398-469e1f0bb11a>