Date: Wed, 11 Jan 2017 08:31:41 -0600 From: Karl Denninger <karl@denninger.net> To: FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org> Subject: Ugh -- attempted to update this morning, and got a nasty panic in ZFS.... Message-ID: <f05fcab3-ec17-17b3-3459-73256f35fbc7@denninger.net>
next in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format.
--------------ms060807040908090103090707
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
During the reboot, immediately after the daemons started up on the
machine (the boot got beyond mounting all the disks and was well into
starting up all the background stuff it runs), I got a double-fault.
=2E.... (there were a LOT more of this same; it pretty clearly was a
recursive call sequence that ran the system out of stack space)
#294 0xffffffff822fdcfd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1666=
#295 0xffffffff8230130e in zio_vdev_io_start (zio=3D0xfffff8010c8f27b0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3127=
#296 0xffffffff822fdcfd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1666=
#297 0xffffffff822e464d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#298 0xffffffff823014c9 in zio_vdev_io_done (zio=3D0xfffff8010cff0b88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3152=
#299 0xffffffff822fdcfd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1666=
#300 0xffffffff8230130e in zio_vdev_io_start (zio=3D0xfffff8010cff0b88)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3127=
#301 0xffffffff822fdcfd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1666=
#302 0xffffffff822e464d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#303 0xffffffff823014c9 in zio_vdev_io_done (zio=3D0xfffff8010c962000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3152=
#304 0xffffffff822fdcfd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1666=
#305 0xffffffff8230130e in zio_vdev_io_start (zio=3D0xfffff8010c962000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3127=
#306 0xffffffff822fdcfd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1666=
#307 0xffffffff822e464d in vdev_queue_io_done (zio=3D<value optimized out=
>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
#308 0xffffffff823014c9 in zio_vdev_io_done (zio=3D0xfffff80102175000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3152=
#309 0xffffffff822fdcfd in zio_execute (zio=3D<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1666=
#310 0xffffffff80b2585a in taskqueue_run_locked (queue=3D<value optimized=
out>)
at /usr/src/sys/kern/subr_taskqueue.c:454
#311 0xffffffff80b26a48 in taskqueue_thread_loop (arg=3D<value optimized =
out>)
at /usr/src/sys/kern/subr_taskqueue.c:724
#312 0xffffffff80a7eb05 in fork_exit (
callout=3D0xffffffff80b26960 <taskqueue_thread_loop>,
arg=3D0xfffff800b8824c30, frame=3D0xfffffe0667430c00)
at /usr/src/sys/kern/kern_fork.c:1040
#313 0xffffffff80f87c3e in fork_trampoline ()
at /usr/src/sys/amd64/amd64/exception.S:611
#314 0x0000000000000000 in ?? ()
Current language: auto; currently minimal
(kgdb)
=2E....
NewFS.denninger.net dumped core - see /var/crash/vmcore.3
Wed Jan 11 08:15:33 CST 2017
FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #14
r311927M: Wed Ja
n 11 07:55:20 CST 2017 =20
karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP
amd64
panic: double fault
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you =
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for detail=
s.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
Fatal double fault
rip =3D 0xffffffff822e3c5d
rsp =3D 0xfffffe066742af90
rbp =3D 0xfffffe066742b420
cpuid =3D 15; apic id =3D 35
panic: double fault
cpuid =3D 15
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfffffe0649ddee30
vpanic() at vpanic+0x186/frame 0xfffffe0649ddeeb0
panic() at panic+0x43/frame 0xfffffe0649ddef10
dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649ddef30
Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649ddef30
--- trap 0x17, rip =3D 0xffffffff822e3c5d, rsp =3D 0xfffffe066742af90, rb=
p =3D
0xfffff
e066742b420 ---
# Work around for this CPU from 11.x eratta
vm.pmap.pcid_enabled=3D0
#
#
# Try to avoid kernel stack exhaustion due to TRIM storms.
kern.kstack_pages=3D"6"
I have kstack_pages set to "6" to try to avoid another panic that I got
occasionally during zfs backup operations which appeared to be linked to
"too many" TRIMs, and looks very similar to this one.
I rebooted back to kernel.old, which was built in October, and the
machine came up normally. I'll try the newer build again and see if
this was transient and related to delayed TRIM operations on the disks
related to the installworld/installkernel. But if it is then it remains
a problem -- and setting stackpages didn't help!
I've got the dump if anything in particular would be of help.
The prompt to do this in the first place was the openssh CVE that was
recently issued.....
--=20
Karl Denninger
karl@denninger.net <mailto:karl@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
--------------ms060807040908090103090707
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC
BlwwggZYMIIEQKADAgECAgE9MA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G
A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl
bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND
dWRhIFN5c3RlbXMgTExDIENBMB4XDTE2MTIxODE5NDUzNVoXDTIxMTIxNzE5NDUzNVowVzEL
MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM
TEMxGzAZBgNVBAMUEmthcmxAZGVubmluZ2VyLm5ldDCCAiIwDQYJKoZIhvcNAQEBBQADggIP
ADCCAgoCggIBAM2N5maxs7NkoY9g5NMxFWll0TYiO7gXrGZTo3q25ZJgNdPMwrntLz/5ewE9
07TEbwJ3ah/Ep9BfZm7JF9vTtE1HkgKtXNKi0pawNGm1Yn26Dz5AbUr1byby6dFtDJr14E07
trzDCtRRvTkOVSBj6PQPal0fAnDtkIYQBVcuMkXkuMCtyfE95pjm8g4K9l7lAcKii3T1/3rE
hCc1o2nBnb7EN1/XwBeCDGB+I2SN/ftZDbKQqGAF5q9dUn+iXU7Z/CVSfUWmhVh6cVZA4Ftv
TglUqj410OuPx+cUQch3h1kFgsuhQR63HiJc3HbRJllHsV0rihvL1CjeARQkhnA6uY9NLFST
p5I/PfzBzW2MSmtN/tGZvmfKKnmtbfUNgkzbIR1K3lsum+yEL71kB93Xtz/4f1demEx5c8TJ
RBIniDHjDeLGK1aoBu8nfnvXAvgthFNTWBOEoR49AHEPjC3kZj0l8JQml1Y8bTQD5gtC5txl
klO60WV0EufU7Hy9CmynMuFtjiA2v71pm097rXeCdrAKgisdYeEESB+SFrlY65rLiLv4n8o1
PX7DqRfqKkOYIakZ0ug/yHVKcq2EM3RiJxwzls5gT70CoOBlKbrC98O8TA6teON0Jq30M06t
NTI2HhvNbJDLbBH+Awf4h1UKB+0ufENwjVvF5Jfz8Ww/FaSDAgMBAAGjgfQwgfEwNwYIKwYB
BQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgwCQYD
VR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIBDQQf
Fh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUpfAI3y+751pp9A0w
6vJHx8RoR/MwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYwFIES
a2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBiB6MlugxYJdccD8boZ/u8
d8VxmLkJCtbfyYHRjYdyoABLW5hE3k3xSpYCM9L7vzWyV/UWwDYKi4ZzxHo4g+jG/GQZfKhx
v38BQjL2G9xD0Hn2d+cygOq3UPjVYlbbfQoew6JbyCFXrrZ7/0jvRMLAN2+bRC7ynaFUixPH
Whnj9JSH7ieYdzak8KN+G2coIC2t2iyfXVKehzi5gdNQ0vJ7+ypbGsRm4gE8Mdo9N/WgFPvZ
HPFqR9Dwas7Z+aHwOabpk5r/336SyjOaZsn3MqKJQZL6GqDKusVOCWt+9uFAD8kadg7FetZe
atIoD9I+zbp59oVoMnkMDMx7Hi85faU03csusqMGsjSsAzWSI1N8PJytZlchLiykokLKc3OL
G87QKlErotlou7cfPX2BbEAH5wmkj9oiqZhxIL/wwAUA+PkiTbEmksKBNompSjUq/6UsR8EA
s74gnu17lmijv8mrg2qMlwRirE7qG8pnE8egLtCDxcjd0Of9WMi2NJskn0/ovC7P+J60Napl
m3ZIgPJst1piYSE0Zc1FIat4fFphMfK5v4iLblo1tFSlkdx1UNDGdg/U+LaXkNVXlMp8fyPm
R80V6cIrCAlEWnBJNxG1UyfbbsvNMCCZBM4faGGsR/hhQOiydlruxhjL6P8J2WV8p11DdeGx
KymWoil2s1J5WTGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxv
cmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRww
GgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5c3Rl
bXMgTExDIENBAgE9MA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcN
AQcBMBwGCSqGSIb3DQEJBTEPFw0xNzAxMTExNDMxNDFaME8GCSqGSIb3DQEJBDFCBECRL4ZN
+LxpNeb0BsuqGt3hb82auRNiQiUQit+4RwwW70JZ3QZFda3F9oBrA+46+xyai9haslBg1slE
E/iIeI/AMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggq
hkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZI
hvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNVBAgT
B0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExM
QzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3VkYSBT
eXN0ZW1zIExMQyBDQQIBPTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYTAlVT
MRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEg
U3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0B
CQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECAT0wDQYJKoZIhvcNAQEBBQAEggIAhlJJSzf02a1X
wLAWkuBXOF0Kt4LXdbcISV0Eg9EjPHfG4GU/rT1y4kbTmgzm1pm3N+W/vvtUZGP1KQYIrpPN
o1YVdq/eviiBOIlP/pxFrOg/RnBhcUeE1QW55jeQgvzkWrqA/pEoPkrYC3dPOrqqcCkL7Ffi
ZyjqipAWiGVSiha1+wcqHKvwmZSq0yHmjb7WRWVKJkOzQC1zHdS2DRXWVMfp90LcELcQ3i8t
r0Qsbo6MY/UWV6JLxTkD8mlbr/7/Q1Fw9ZpeoyozBOeik1T7CXK45mGcPhOD9Vz/NYxUvgJL
BSb8c3EANmceqeQsrES9+EqOPm+9g9FmKCIGuCJSUFQTWzYQYgC+fzvrwqTQcail27YDUBvb
xgv16+tIwHNFyX0UV5BZnaFdY6N129ZscHT7D16C/tij8F37eA+g6rrsKEibxPoqmCMI6JBw
J8TV1QNEp6gZcCgpGC+702269oWvIiTzUYHOCmwYDo0spteMAi0sJIXvMZ6/N0NshTy3VQNr
QW+0jyIJ3rRcfUWdrHy2Jum1rlvdt2DsuLzCqIYn2DevNn6LUrYQ+EeHEw39hLjDgOJFE+eZ
Wea7tGMgHXVxKUS5NbjdiXl8e88nLc+TImw6OcdKuWAay5IJDf5RB0GZ+05zolfKwqjSFFyf
0BT4vp31QMTHgvmkneaI3IkAAAAAAAA=
--------------ms060807040908090103090707--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?f05fcab3-ec17-17b3-3459-73256f35fbc7>