Date: Thu, 18 Jul 2019 15:35:17 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: Re: Kernel panic in zfs code; 12-STABLE Message-ID: <501734b7-22ec-5c01-eea5-26b458945e7e@denninger.net> In-Reply-To: <d6cf2edf-81f2-fb63-fa39-c310fe7258a7@grosbein.net> References: <61e5debd-b440-16c9-2a70-0912634e52aa@denninger.net> <d6cf2edf-81f2-fb63-fa39-c310fe7258a7@grosbein.net>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms080100070401010307050607 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 7/18/2019 15:19, Eugene Grosbein wrote: > 19.07.2019 3:13, Karl Denninger wrote: > >> FreeBSD 12.0-STABLE #2 r349024M: Thu Jun 13 18:01:16 CDT 2019 =20 >> karl@NewFS.denninger.net:/usr/obj/usr/src/amd64.amd64/sys/KSD-SMP >> >> Note -- no patches of any sort in the ZFS code; I am NOT running any o= f >> my former patch set. >> >> NewFS.denninger.net dumped core - see /var/crash/vmcore.8 >> >> Thu Jul 18 15:02:54 CDT 2019 >> >> FreeBSD NewFS.denninger.net 12.0-STABLE FreeBSD 12.0-STABLE #2 r349024= M: >> Thu Jun 13 18:01:16 CDT 2019 =20 >> karl@NewFS.denninger.net:/usr/obj/usr/src/amd64.amd64/sys/KSD-SMP amd= 64 >> >> panic: double fault > [skip] > >> #283 0xffffffff82748d91 in zio_vdev_io_done (zio=3D0xfffff8000b8b8000)= >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3= 376 >> #284 0xffffffff82744eac in zio_execute (zio=3D0xfffff8000b8b8000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1= 786 >> #285 0xffffffff80c3b7f4 in taskqueue_run_locked (queue=3D0xfffff801a8b= 35100) >> at /usr/src/sys/kern/subr_taskqueue.c:467 >> #286 0xffffffff80c3cb28 in taskqueue_thread_loop (arg=3D<value optimiz= ed out>) >> at /usr/src/sys/kern/subr_taskqueue.c:773 >> #287 0xffffffff80b9ab23 in fork_exit ( >> callout=3D0xffffffff80c3ca90 <taskqueue_thread_loop>, >> arg=3D0xfffff801a0577520, frame=3D0xfffffe009d4edc00) >> at /usr/src/sys/kern/kern_fork.c:1063 >> #288 0xffffffff810b367e in fork_trampoline () >> at /usr/src/sys/amd64/amd64/exception.S:996 >> #289 0x0000000000000000 in ?? () >> Current language: auto; currently minimal >> (kgdb) > You have "double fault" and completely insane number of stack frames in= the trace. > This is obviously infinite recursion resulting in kernel stack overflow= and panic. Yes, but.... why and how? What's executing at the time is this command: zfs send -RI $i@zfs-old $i@zfs-base | zfs receive -Fudv $BACKUP Which in turn results in the old snapshots on the target not on the source being deleted, then the new ones being sent.=C2=A0 It never gets t= o the sending part; it blows up during the delete of the OLD snapshots. The one(s) it deletes, however, it DOES delete.=C2=A0 When the box is rebooted those two snapshots on the target are indeed gone. That is, it is NOT getting "stuck" on one (which would imply there's an un-detected fault in the filesystem on the target in the metadata for that snapshot, resulting in a recursive call that blows up the stack) and it never gets to send the new snapshot, so whatever is going on is NOT on the source filesystem.=C2=A0 Neither source or destination shows a= ny errors on the filesystem; both pools are healthy with zero error counts. Therefore the question -- is the system queueing enough work to blow the stack *BUT* the work it queues is all legitimate?=C2=A0 If so there's a serious problem in the way the code now functions in that an "ordinary" operation can result in what amounts to kernel stack exhaustion. One note -- I haven't run this backup for the last five days, as I do it manually and I've been out of town.=C2=A0 Previous running it on a daily basis completed without trouble.=C2=A0 This smells like a backlog of "thi= ngs to do" when the send runs that results in the allegedly-infinite recursion (that isn't really infinite) that runs the stack out of space -- and THAT implies that the system is trying to queue a crazy amount of work on a recursive basis for what is a perfectly-legitimate operation -- which it should *NOT* do. --=20 Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms080100070401010307050607 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTkwNzE4MjAzNTE3 WjBPBgkqhkiG9w0BCQQxQgRAgkI8Gt5YMWiM1S+CvGBkSSIRwDwD26GM1lJaJvtG1SHZPNE2 8WvtVYIPCtZwgBk59z16OIgmH0p6nHZe8vGyFjBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgBkTByTrYxxIzbUVZeJr1p3EDq59Vgugys4C6YlpR52arKHE6DqXAeWo5T32w1V4yFk uP1sobF4s1DFagtqc0ISrY0/NLt2jZX0IypH69MF2L12RwGb/QNAmTk3FVrzsyZMTCIMC+Qq ftvzGrPi/91hZLaWZArd8gAz005T8N1Ve6zke+Uau4uJ69eBzpAjt5GZh/7pfOXOqi0PYFr0 48xMh1+25Bcu+UvRY6xmlWWmD3LbehA6nO+aRkOtl9BqVElvLg1Hb78nkUsYA0yj+rmmnJm6 gYjY6VQp0soN4nheILBB4lGzEopJuywcPfsTJMhTUECCPtet+DyVww8+tU9dqeaEmLHM00Do X2kWjdGbhZsmcjRORkDUASBpGZ4d0xqpwc56V+9OAidhLpkqjmgt4odkGfnaa0h9fNiSrqgG lfKHqYCsgM7YESB3Jtf6Nv30jP9z1Ex6b2b9l7C8HmzlvIunHaPDnx5oohBIo5dx5zk125ej UsCHmNGUF3FJdJSQQWMSQpceetLrrV2qNVKfMxUAmm3c4dbrQnQxa/QxzrkceXbcfjPDcyTG uLK3saYZbAUp6mM6lOF9Ju2Fh+eKhcjr82pCYmXileU6jnwJkeaYF6emBZDQli5FaUW2pmUG nsQWwxLwm5i8GAnCPFzc+WyskgvGTqVLjySo3E/gawAAAAAAAA== --------------ms080100070401010307050607--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?501734b7-22ec-5c01-eea5-26b458945e7e>