From owner-freebsd-current@FreeBSD.ORG Wed Aug 11 13:28:10 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4A74F16A4CE; Wed, 11 Aug 2004 13:28:10 +0000 (GMT) Received: from mediamonks.com (siripandita.mediamonks.net [62.192.127.138]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5BF5E43D39; Wed, 11 Aug 2004 13:28:08 +0000 (GMT) (envelope-from root@mediamonks.net) Received: from manrikigusari [217.19.28.156] by mediamonks.com with ESMTP (SMTPD32-8.12) id AED934800BC; Wed, 11 Aug 2004 15:27:53 +0200 From: "Terrence Koeman" To: "'John Baldwin'" , Date: Wed, 11 Aug 2004 15:28:10 +0200 Organization: MediaMonks B.V. MIME-Version: 1.0 X-Mailer: Microsoft Office Outlook, Build 11.0.5510 Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=SHA1; boundary="----=_NextPart_000_000C_01C47FB7.CF14DC10" Thread-Index: AcR/FuTtYoq92di4TM6OknOtZEtqwwAFx9EgAB3h/bA= In-Reply-To: <200408110104380.SM01804@manrikigusari> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 Message-Id: <200408111527566.SM01804@manrikigusari> X-Info: This e-mail was scanned for spam and viruses by mail.mediamonks.net. X-Info: Please send abuse reports about this e-mail to abuse@mediamonks.net. Subject: RE: Lock order reversal in 5.2-CURRENT X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: root@mediamonks.net List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Aug 2004 13:28:10 -0000 This is a multi-part message in MIME format. ------=_NextPart_000_000C_01C47FB7.CF14DC10 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I think something else is wrong, as I get different lock order reversals = and some other errors that all lockup the box. Earlier I had a corrupted cc binary after a buildworld. Everything points to a hardware failure somewhere, but I already = switched the hardware before this happened, I swapped RAID arrays in identical machines, and the machine where -CURRENT runs on now was a production = server that ran 4.9/4.10-STABLE for months under heavy load without any = problems whatsoever. The following is what I got today: Second bad /: bad dir ino 16110954 at offset 24: mangled entry panic: ufs_dirbad: bad dir KDB: stack backtrace: kdb_backtrace(c05fb5b8,c0646f80,c060ae08,de301814,100) at = kdb_backtrace+0x2e panic(c060ae08,c1738200,f5d56a,18,c060adc2) at panic+0xb7 ufs_dirbad(c2180118,18,c060adc2,0,de301890) at ufs_dirbad+0x50 ufs_lookup(de301950,de30198c,c0501c99,de301950,de301bf0) at = ufs_lookup+0x457 ufs_vnoperate(de301950,de301bf0,de301c04,c1c61420,c1aa0b00) at ufs_vnoperate+0x18 vfs_cache_lookup(de3019d0,de3019ec,c05072c2,de3019d0,20002) at vfs_cache_lookup+0xe9 ufs_vnoperate(de3019d0,20002,c1aa0b00,de3019d0,c1aa0b00) at ufs_vnoperate+0x18 lookup(de301bdc,0,c0602e3d,a4,c1aa0b00) at lookup+0x332 namei(de301bdc,c064ab40,246,9,c1aa0b00) at namei+0x2ae vn_open_cred(de301bdc,de301cdc,1a4,c1c2e600,3) at vn_open_cred+0x24b vn_open(de301bdc,de301cdc,1a4,3,c04d1ea0) at vn_open+0x33 kern_open(c1aa0b00,bfbfd440,0,1,1b6) at kern_open+0xf2 open(c1aa0b00,de301d14,c,c04d02b2,3) at open+0x30 syscall(2f,2f,2f,8084d7b,4) at syscall+0x2e0 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip =3D 0x80662a7, esp =3D = 0xbfbfd3ec, ebp =3D 0xbfbfd418 --- KDB: enter: panic [thread 100096] Stopped at kdb_enter+0x30: leave db>=20 panic: mtx_lock() of spin mutex =A0=A5 @ = /usr/src/sys/vm/vm_object.c:1587 KDB: stack backtrace: kdb_backtrace(c05fb5b8,c0646f80,c05fa793,d4da6bb0,100) at = kdb_backtrace+0x2e panic(c05fa793,c1009ca0,c060c69a,633,c064c098) at panic+0xb7 _mtx_lock_flags(c100939c,0,c060c69a,633,c06729c0) at = _mtx_lock_flags+0x69 vm_object_collapse(c1a0f000,0,c060bde0,900,c058e2fe) at vm_object_collapse+0x5b vm_map_copy_entry(c154a940,c154bb90,c19b9618,c1aa0348,c04d2787) at vm_map_copy_entry+0x9f vmspace_fork(c154a940,1,c060b912,26e,c1542aa0) at vmspace_fork+0x30f vm_forkproc(c157ab00,c1a64000,c178c000,14,753) at vm_forkproc+0xee fork1(c157ab00,14,0,d4da6cdc,c157ab00) at fork1+0xfd9 fork(c157ab00,d4da6d14,c04a8825,c1539e00,0) at fork+0x29 syscall(280b002f,284c002f,bfbf002f,1,2) at syscall+0x2e0 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (2, FreeBSD ELF32, fork), eip =3D 0x283dd42f, esp =3D = 0xbfbfecdc, ebp =3D 0xbfbfed08 --- KDB: enter: panic [thread 100041] Stopped at kdb_enter+0x30: leave db>=20 lock order reversal 1st 0xc0645aa0 sched lock (sched lock) @ /usr/src/sys/kern/subr_sleepqueue.c:623 2nd 0xc06486ec sleepq chain (sleepq chain) @ /usr/src/sys/kern/subr_sleepqueue.c:223 KDB: stack backtrace: kdb_backtrace(c05fe96f,c06486ec,c05fdc8a,c05fdc8a,c05fdc97) at kdb_backtrace+0x2e witness_checkorder(c06486ec,9,c05fdc97,df,67e) at = witness_checkorder+0x6a6 _mtx_lock_spin_flags(c06486ec,0,c05fdc97,df,0) at = _mtx_lock_spin_flags+0x8d sleepq_lookup(c0641d80,c05fe5e3,687,c0645aa0,d3bbc9ec) at = sleepq_lookup+0x57 sleepq_broadcast(c0641d80,0,ffffffff,d3bbca14,c04b4152) at sleepq_broadcast+0x31 wakeup(c0641d80,1,c05fbc09,179,c1a632c0) at wakeup+0x21 setrunnable(c1a632c0,0,c05fdc97,26f,c066ef84) at setrunnable+0xb2 sleepq_resume_thread(c1a632c0,ffffffff,c05fdc97,31e,c2259540) at sleepq_resume_thread+0xa0 sleepq_remove(c1a632c0,c066ef84,c05fef7d,464,c2259540) at sleepq_remove+0x117 doselwakeup(c2259540,58,d3bbcaa8,c04f1771,c2259540) at doselwakeup+0x110 selwakeuppri(c2259540,58,c060126c,18e,c20f18c0) at selwakeuppri+0x18 sowakeup(c22594f0,c2259540,c0606e75,4fd,0) at sowakeup+0x41 tcp_input(c1915500,14,f,0,14) at tcp_input+0x1350 ip_input(c1915500,0,c0605194,1d0,c1791318) at ip_input+0x712 transmit_event(c1791300,0,c0605194,300,c0670300) at transmit_event+0x128 dummynet(0,0,c05fc4ec,fd,0) at dummynet+0x138 softclock(0,0,c05f8d63,263,c1545534) at softclock+0x20e ithread_loop(c1539580,d3bbcd48,c05f8b5a,32b,0) at ithread_loop+0x172 fork_exit(c04944a0,c1539580,d3bbcd48) at fork_exit+0xc7 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip =3D 0, esp =3D 0xd3bbcd7c, ebp =3D 0 --- KDB: enter: witness_checkorder [thread 100024] Stopped at kdb_enter+0x30: leave db>=20 Fatal trap 18: integer divide fault while in kernel mode instruction pointer =3D 0x8:0xc056b238 stack pointer =3D 0x10:0xde08c91c frame pointer =3D 0x10:0xde08c980 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, def32 1, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 14276 (httpd) [thread 100058] Stopped at softdep_setup_freeblocks+0x408: divl 0xb8(%ecx),%eax db> Does someone have any ideas? --=20 Regards, Terrence Koeman =20 MediaMonks B.V. (www.mediamonks.com) Please quote all replies in correspondence. =20 ------=_NextPart_000_000C_01C47FB7.CF14DC10 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIKUzCCAj0w ggGmAhEAzbp/VvDf5LxU/iKss3KqVTANBgkqhkiG9w0BAQIFADBfMQswCQYDVQQGEwJVUzEXMBUG A1UEChMOVmVyaVNpZ24sIEluYy4xNzA1BgNVBAsTLkNsYXNzIDEgUHVibGljIFByaW1hcnkgQ2Vy dGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNOTYwMTI5MDAwMDAwWhcNMjgwODAxMjM1OTU5WjBfMQsw CQYDVQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xNzA1BgNVBAsTLkNsYXNzIDEgUHVi bGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwgZ8wDQYJKoZIhvcNAQEBBQADgY0A MIGJAoGBAOUZv22jVmEtmUhx9mfeuY3rt56GgAqRDvo4Ja9GiILlc6igmyRdDR/MZW4MsNBWhBiH mgabEKFz37RYOWtuwfYV1aioP6oSBo0xrH+wNNePNGeICc0UEeJORVZpH3gCgNrcR5EpuzbJY1zF 4Ncth3uhtzKwezC6Ki8xqu6jZ9rbAgMBAAEwDQYJKoZIhvcNAQECBQADgYEATD+4i8Zo3+5DMw5d 6abLB4RNejP/khv0Nq3YlSI2aBFsfELM85wuxAc/FLAPT/+Qknb54rxK6Y/NoIAK98Up8YIiXbix 3YEjo3slFUYweRb46gVLlH8dwhzI47f0EEA8E8NfH1PoSOSGtHuhNbB7Jbq4046rPzidADQAmPPR cZQwggNmMIICz6ADAgECAhANi0/uqtIYW/R1ap0p4X/7MA0GCSqGSIb3DQEBAgUAMF8xCzAJBgNV BAYTAlVTMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjE3MDUGA1UECxMuQ2xhc3MgMSBQdWJsaWMg UHJpbWFyeSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw05ODA1MTIwMDAwMDBaFw0wODA1MTIy MzU5NTlaMIHMMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1 c3QgTmV0d29yazFGMEQGA1UECxM9d3d3LnZlcmlzaWduLmNvbS9yZXBvc2l0b3J5L1JQQSBJbmNv cnAuIEJ5IFJlZi4sTElBQi5MVEQoYyk5ODFIMEYGA1UEAxM/VmVyaVNpZ24gQ2xhc3MgMSBDQSBJ bmRpdmlkdWFsIFN1YnNjcmliZXItUGVyc29uYSBOb3QgVmFsaWRhdGVkMIGfMA0GCSqGSIb3DQEB AQUAA4GNADCBiQKBgQC7WkSKBBa7Vf0DeootlE8VeDa4DUqyb5xUv7zodyqdufBou5XZMUFweoFL uUgTVi3HCOGEQqvAopKrRFyqQvCCDgLpL/vCO7u+yScKXbawNkIztW5UiE+HSr8Z2vkV6A+Hthzj zMaajn9qJJLj/OBluqexfu/J2zdqyErICQbkmQIDAQABo4G0MIGxMBEGCWCGSAGG+EIBAQQEAwIB BjA1BgNVHR8ELjAsMCqgKKAmhiRodHRwOi8vY3JsLnZlcmlzaWduLmNvbS9wY2ExLjEuMS5jcmww RwYDVR0gBEAwPjA8BgtghkgBhvhFAQcBATAtMCsGCCsGAQUFBwIBFh93d3cudmVyaXNpZ24uY29t L3JlcG9zaXRvcnkvUlBBMA8GA1UdEwQIMAYBAf8CAQAwCwYDVR0PBAQDAgEGMA0GCSqGSIb3DQEB AgUAA4GBAEJ8Dt+MeUysvwjsTVUvUImgxV5OLl6VMpt5rWURCxxKUsTVqDEhjt4Qm2wIxQfmA7nn yDR4CQnyvAZC+FqMg9GK3qoi9dnjIdLPZYwGM7DNILIzzQq9PuGdwTWpZLCnpSRb6fFo6xPEfDf0 lGQNmsW9MxfvgzOgPuWqPq7Ycx+tMIIEpDCCBA2gAwIBAgIQZx6EJ4oHSmQocGtEpogzPzANBgkq hkiG9w0BAQQFADCBzDEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZlcmlTaWdu IFRydXN0IE5ldHdvcmsxRjBEBgNVBAsTPXd3dy52ZXJpc2lnbi5jb20vcmVwb3NpdG9yeS9SUEEg SW5jb3JwLiBCeSBSZWYuLExJQUIuTFREKGMpOTgxSDBGBgNVBAMTP1ZlcmlTaWduIENsYXNzIDEg Q0EgSW5kaXZpZHVhbCBTdWJzY3JpYmVyLVBlcnNvbmEgTm90IFZhbGlkYXRlZDAeFw0wNDAyMDQw MDAwMDBaFw0wNTAyMDMyMzU5NTlaMIIBFjEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNV BAsTFlZlcmlTaWduIFRydXN0IE5ldHdvcmsxRjBEBgNVBAsTPXd3dy52ZXJpc2lnbi5jb20vcmVw b3NpdG9yeS9SUEEgSW5jb3JwLiBieSBSZWYuLExJQUIuTFREKGMpOTgxHjAcBgNVBAsTFVBlcnNv bmEgTm90IFZhbGlkYXRlZDE0MDIGA1UECxMrRGlnaXRhbCBJRCBDbGFzcyAxIC0gTWljcm9zb2Z0 IEZ1bGwgU2VydmljZTEYMBYGA1UEAxQPVGVycmVuY2UgS29lbWFuMSIwIAYJKoZIhvcNAQkBFhNy b290QG1lZGlhbW9ua3MubmV0MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDKUqZ0P8RUaCCe aemq1V8OtPimM3jxXZY6yuWtQz74POz82LhrkctttQoqujz5SubFWsQ15KsVnK4Nt+gTmtUELQVa I4uUebypbChhf1fxFerA7Rx8SYL4Ez9Fvco0tn89FRht6C/xoc/Ms/1YkQQcfnu2sqohx53S5Pm0 Ffo/ZQIDAQABo4IBODCCATQwCQYDVR0TBAIwADCBrAYDVR0gBIGkMIGhMIGeBgtghkgBhvhFAQcB ATCBjjAoBggrBgEFBQcCARYcaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL0NQUzBiBggrBgEFBQcC AjBWMBUWDlZlcmlTaWduLCBJbmMuMAMCAQEaPVZlcmlTaWduJ3MgQ1BTIGluY29ycC4gYnkgcmVm ZXJlbmNlIGxpYWIuIGx0ZC4gKGMpOTcgVmVyaVNpZ24wEQYJYIZIAYb4QgEBBAQDAgeAMDAGCmCG SAGG+EUBBgcEIhYgMWE5NjkyOTM3Y2MyOTFhMzZkZjAxN2Q4NDBjNDRiMTYwMwYDVR0fBCwwKjAo oCagJIYiaHR0cDovL2NybC52ZXJpc2lnbi5jb20vY2xhc3MxLmNybDANBgkqhkiG9w0BAQQFAAOB gQAmpan0cUGODHHRIbkVFBHbTOumbMmEO3TR6d9z3LVO9cU5YDC/8BI9e5DZ7Kv43p/ldaGVo2ua ZwtvwL6o/iDcP8UDw+WmjSwzp6tck5cVRDEi+q9nCe3JAYBeRgSJlP5JGOirFpLuDTwL4UPdE0em p9ELBC+sXACm2C3u19hlKTGCBD4wggQ6AgEBMIHhMIHMMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5j LjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazFGMEQGA1UECxM9d3d3LnZlcmlzaWdu LmNvbS9yZXBvc2l0b3J5L1JQQSBJbmNvcnAuIEJ5IFJlZi4sTElBQi5MVEQoYyk5ODFIMEYGA1UE AxM/VmVyaVNpZ24gQ2xhc3MgMSBDQSBJbmRpdmlkdWFsIFN1YnNjcmliZXItUGVyc29uYSBOb3Qg VmFsaWRhdGVkAhBnHoQnigdKZChwa0SmiDM/MAkGBSsOAwIaBQCgggKyMBgGCSqGSIb3DQEJAzEL BgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTA0MDgxMTEzMjgxMFowIwYJKoZIhvcNAQkEMRYE FBNL2TvJdMsx5+G5zMcIlbYPwSPOMGcGCSqGSIb3DQEJDzFaMFgwCgYIKoZIhvcNAwcwDgYIKoZI hvcNAwICAgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMAcGBSsOAwIa MAoGCCqGSIb3DQIFMIHyBgkrBgEEAYI3EAQxgeQwgeEwgcwxFzAVBgNVBAoTDlZlcmlTaWduLCBJ bmMuMR8wHQYDVQQLExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMUYwRAYDVQQLEz13d3cudmVyaXNp Z24uY29tL3JlcG9zaXRvcnkvUlBBIEluY29ycC4gQnkgUmVmLixMSUFCLkxURChjKTk4MUgwRgYD VQQDEz9WZXJpU2lnbiBDbGFzcyAxIENBIEluZGl2aWR1YWwgU3Vic2NyaWJlci1QZXJzb25hIE5v dCBWYWxpZGF0ZWQCEGcehCeKB0pkKHBrRKaIMz8wgfQGCyqGSIb3DQEJEAILMYHkoIHhMIHMMRcw FQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazFG MEQGA1UECxM9d3d3LnZlcmlzaWduLmNvbS9yZXBvc2l0b3J5L1JQQSBJbmNvcnAuIEJ5IFJlZi4s TElBQi5MVEQoYyk5ODFIMEYGA1UEAxM/VmVyaVNpZ24gQ2xhc3MgMSBDQSBJbmRpdmlkdWFsIFN1 YnNjcmliZXItUGVyc29uYSBOb3QgVmFsaWRhdGVkAhBnHoQnigdKZChwa0SmiDM/MA0GCSqGSIb3 DQEBAQUABIGAI9g8ZC550YJGdQWt+jmIa8/80QhT9J8i7ZxMXtEfIW1sqCQjA16ujcubRG42gyym O+F7bFIA3cPCEXipvQVcV2NKEcNGPQ9+1GvzFZxh96KNUa1MzHoJePN3Nu4TIphNE6a0zcAUxDQ1 9+4ySHvQyyHzkyJYm7VGr6zDkCOCwgoAAAAAAAA= ------=_NextPart_000_000C_01C47FB7.CF14DC10--