Date: Fri, 9 Jul 2021 22:41:00 -0400 From: Karl Denninger <karl@denninger.net> To: stable@freebsd.org Subject: Re: 12.2 Splay Tree ipfw potential panic source Message-ID: <7bfda38b-cf81-d8be-7691-e18946e6b56e@denninger.net> In-Reply-To: <ca5beb7c-db38-1d3c-0f3c-b1b6a12c311e@denninger.net> References: <2e3dcd4d-c8e6-8381-0010-d0844c99901e@denninger.net> <20210708221134.GA32658@belenus.iks-jena.de> <a6a9c220-fee6-a0ea-7721-f88ff865a6a8@denninger.net> <CAFMmRNy9K-1mTDoqQhgdChWV5f_n4QhNesz%2B6xWywn_TQ43xng@mail.gmail.com> <ca5beb7c-db38-1d3c-0f3c-b1b6a12c311e@denninger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms060608010600000502000206 Content-Type: multipart/alternative; boundary="------------4FA32C454518EECB922BCACF" Content-Language: en-US This is a multi-part message in MIME format. --------------4FA32C454518EECB922BCACF Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable On 7/9/2021 18:06, Karl Denninger wrote: > On 7/9/2021 16:17, Ryan Stone wrote: >> On Thu, Jul 8, 2021 at 8:54 PM Karl Denninger <karl@denninger.net>=20 >> wrote: >>> I will see if I can get at least a panic backtrace, although the >>> impacted box is a pcEngines firewall that boots of an SD card. >> Have you checked whether netdump supports your NICs?=C2=A0 You should = be >> able to get a full vmcore off if so. > > Yes; the box in question is in heavy production and I will not be able = > to get an isolated period of time to pull a core (assuming the remote=20 > dump works) until sometime this weekend. > > Will advise once I (hopefully) have it. > Ok, so I have good news and bad news. I have the trap and it is definitely in libalias which appears to come=20 about as a result of a NAT translation attempt. Fatal trap 18: integer divide fault while in kernel mode cpuid =3D 1; apic id =3D 01 instruction pointer=C2=A0=C2=A0=C2=A0=C2=A0 =3D 0x20:0xffffffff8275b7cc stack pointer=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =3D 0x28:0xfffffe0017b6b310 frame pointer=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =3D 0x28:0xfffffe0017b6b320 code segment=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 =3D base 0x0, limit 0xfffff, type 0x1b =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D DP= L 0, pres 1, long 1, def32 0, gran 1 processor eflags=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D interrupt = enabled, resume, IOPL =3D 0 current process=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D 0 (if= _io_tqg_1) trap number=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 =3D 18 panic: integer divide fault cpuid =3D 1 time =3D 1625883072 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame=20 0xfffffe0017b6b020 vpanic() at vpanic+0x17b/frame 0xfffffe0017b6b070 panic() at panic+0x43/frame 0xfffffe0017b6b0d0 trap_fatal() at trap_fatal+0x391/frame 0xfffffe0017b6b130 trap() at trap+0x67/frame 0xfffffe0017b6b240 calltrap() at calltrap+0x8/frame 0xfffffe0017b6b240 --- trap 0x12, rip =3D 0xffffffff8275b7cc, rsp =3D 0xfffffe0017b6b310, rb= p =3D=20 0xfffffe0017b6b320 --- HouseKeeping() at HouseKeeping+0x1c/frame 0xfffffe0017b6b320 LibAliasInLocked() at LibAliasInLocked+0x2f/frame 0xfffffe0017b6b3e0 LibAliasIn() at LibAliasIn+0x46/frame 0xfffffe0017b6b410 ipfw_nat() at ipfw_nat+0x234/frame 0xfffffe0017b6b460 ipfw_chk() at ipfw_chk+0x1350/frame 0xfffffe0017b6b670 ipfw_check_packet() at ipfw_check_packet+0xf0/frame 0xfffffe0017b6b760 pfil_run_hooks() at pfil_run_hooks+0xb0/frame 0xfffffe0017b6b7f0 ip_input() at ip_input+0x427/frame 0xfffffe0017b6b8a0 netisr_dispatch_src() at netisr_dispatch_src+0xca/frame 0xfffffe0017b6b8f= 0 ether_demux() at ether_demux+0x138/frame 0xfffffe0017b6b920 ether_nh_input() at ether_nh_input+0x33b/frame 0xfffffe0017b6b980 netisr_dispatch_src() at netisr_dispatch_src+0xca/frame 0xfffffe0017b6b9d= 0 ether_input() at ether_input+0x4b/frame 0xfffffe0017b6ba00 iflib_rxeof() at iflib_rxeof+0xad6/frame 0xfffffe0017b6bae0 _task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe0017b6bb20 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x121/frame=20 0xfffffe0017b6bb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb6/frame=20 0xfffffe0017b6bbb0 fork_exit() at fork_exit+0x7e/frame 0xfffffe0017b6bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0017b6bbf0 --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- Uptime: 7m23s netdump: overwriting mbuf zone pointers netdump in progress. searching for server... netdumping to 192.168.10.100 (ac:1f:6b:ad:d8:cb) Dumping 190 out of 1882 MB:. . . . . . . . . . . . . ** DUMP FAILED (ERROR 60) ** Now the bad news -- as you can see, an attempted remote dump fails,=20 possibly because the network code at that point is hosed. I get a 69632=20 length file (exactly and repeatedly) on the remote machine where the=20 dump is set to go; it looks like the first piece of it is indeed=20 received but that's it and then the panic'd unit reboots. On the server (remote) end I have this in the "info" file: Dump from IpGw [192.168.10.200] Dump incomplete: client timed out So it looks like it got the first part of it, the server replied but the = crashed box never sent anything else. -rw-------=C2=A0=C2=A0 1 root=C2=A0 wheel=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2= Jul=C2=A0 9 22:11 bounds.IpGw -rw-------=C2=A0=C2=A0 1 root=C2=A0 wheel=C2=A0=C2=A0=C2=A0=C2=A0 66 Jul=C2= =A0 9 22:10 info.IpGw.0 -rw-------=C2=A0=C2=A0 1 root=C2=A0 wheel=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0= Jul=C2=A0 9 22:11 info.IpGw.1 -rw-------=C2=A0=C2=A0 1 root=C2=A0 wheel=C2=A0 69632 Jul=C2=A0 9 22:00 v= mcore.IpGw.0 -rw-------=C2=A0=C2=A0 1 root=C2=A0 wheel=C2=A0 69632 Jul=C2=A0 9 22:11 v= mcore.IpGw.1 Without a complete core I can't give you a good traceback.=C2=A0 I may be= =20 able to get a local device on this unit sometime over the weekend=20 sometime -- not sure as of yet as it is in production use. This is an extremely reliable panic -- uptime is only a few minutes=20 before it blows up. --=20 Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------4FA32C454518EECB922BCACF-- --------------ms060608010600000502000206 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMjEwNzEwMDI0MTAw WjBPBgkqhkiG9w0BCQQxQgRAP3Klgmpa1kctKVqfx0bF5eclItSpkCmzQxzVTlpg2PJWlPEJ eZH3YzQPSBuIVYBZA+rn5Qya0RSg+sfRT5RMzzBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgBxBKyMqAhZqnrr836e9z28E7bm2V3Im3h/ZneaxFDmqk/MvU6MvVcjOuRgKmo8SUTn IvPUGdP5FkgpVxJkXN8lvabooiEh95C6y6BcDgcI7r9QhCP9OCdi++7l4vBcE3qWYKkI7EEK dM8i4LKgMo4UYDKEZO+sh6Redx9AyUPtNJmReKOPE1CylgtgF7hAKnwFQhO7UG2PZdNzOvrZ big5THGKU5aBTZ2zFdH8HHmWG7DT/I+oqXndtx5chIZg2M3wvWyQoL9jHsGOR1OMUr0QB4w5 /iZMWbn2ve48CMmUbDRM8BuK0w5M3Um1vvcraezSqaohP6WZs7xa0YUdv4HkJuy0ukoBRJZn wF1F/ZY5sejR653p4+CkYZE4QjVBAxHqQkysHhFM9TjRLsJ8rts4WuwX553op/loW/cVdGAT gs0P7H1IihV3apP2LvIxMNwcSXhRXQNuQHfZ+7hZgI/iCfJqNQs0ufn9U94NSJCFJ6Of4wlk CPtd6aG2aV2oLvpya5rSuu8f38NY/2MH8hqlUpForpSlj37UqCG6mlunuF9az9gGZnmI6cVe ef3h61Ow5uoguNQ31fap1mChGMxKvCwUHDDc8/9ztUK+Q88ZJZlrd2xo0h+QPi2qe22giW7J jjLzwkoDP5Z81FjftQAOvF7IwTFeOfuc92BMff1OMwAAAAAAAA== --------------ms060608010600000502000206--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7bfda38b-cf81-d8be-7691-e18946e6b56e>