From owner-freebsd-current@FreeBSD.ORG Thu Aug 7 17:06:48 2003 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 973D237B401; Thu, 7 Aug 2003 17:06:48 -0700 (PDT) Received: from boreas.isi.edu (boreas.isi.edu [128.9.160.161]) by mx1.FreeBSD.org (Postfix) with ESMTP id 04CF343FE0; Thu, 7 Aug 2003 17:06:48 -0700 (PDT) (envelope-from larse@ISI.EDU) Received: from isi.edu (c-24-130-112-121.we.client2.attbi.com [24.130.112.121]) by boreas.isi.edu (8.11.6p2/8.11.2) with ESMTP id h7806lX20084; Thu, 7 Aug 2003 17:06:47 -0700 (PDT) Message-ID: <3F32E98C.6090007@isi.edu> Date: Thu, 07 Aug 2003 17:06:36 -0700 From: Lars Eggert Organization: USC Information Sciences Institute User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.5a) Gecko/20030718 X-Accept-Language: en-us, en MIME-Version: 1.0 To: John Baldwin References: In-Reply-To: Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary="------------ms010100080307030107020403" cc: current Subject: Re: spin lock sched lock held for > 5 seconds X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Aug 2003 00:06:48 -0000 This is a cryptographically signed message in MIME format. --------------ms010100080307030107020403 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit John Baldwin wrote: > On 01-Aug-2003 Lars Eggert wrote: > >>Hi, >> >>got the following panic overnight running with all debugging options on >>(WITNESS, MUTEX_DEBUG, DIAGNOSTIC, INVARIANTS; WITNESS_SKIPSPIN off): >> >>panic: spin lock sched lock held by 0xc658e130 for > 5 seconds >>cpuid = 0; lapic.id = 00000000 >>Stack backtrace: >>backtrace(c031d030,0,c031c4c5,df0dab8c,100) at backtrace+0x17 >>panic(c031c4c5,c031c62e,c658e130,c036f160,18b) at panic+0x13d >>_mtx_lock_spin(c036f160,2,c031a229,18b,c21b2ab0) at _mtx_lock_spin+0x83 >>_mtx_lock_spin_flags(c036f160,2,c031a229,18b,df0dac0c) at >>_mtx_lock_spin_flags+0xb9 >>statclock(df0dac00,df0dac44,c02d8a9c,0,c2198d00) at statclock+0x39 >>rtcintr(0) at rtcintr+0x4f >>Xfastunpend8(df0dacb8,c02d1f05,8,608,c0372e60) at Xfastunpend8+0x1c >>call_fast_unpend(8,608,c0372e60,ffc00034,0) at call_fast_unpend+0xd >>i386_unpend(c036f160,c21b0790,df0dacd0,c01b1ae0,df0dacec) at >>i386_unpend+0x8d >>cpu_unpend(df0dacec,c01a2534,c036f160,1,c031c2e2) at cpu_unpend+0x2d >>critical_exit(c036f160,1,c031c2e2,1bc,1) at critical_exit+0x2d >>_mtx_unlock_spin_flags(c036f160,0,c031ac9f,7c,c21b2ab0) at >>_mtx_unlock_spin_flags+0xbb >>idle_proc(0,df0dad48,c031ab89,312,c21b2ab0) at idle_proc+0xb0 >>fork_exit(c0199972,0,df0dad48) at fork_exit+0xc3 >>fork_trampoline() at fork_trampoline+0x1a >>--- trap 0x1, eip = 0, esp = 0xdf0dad7c, ebp = 0 --- >>Debugger("panic") >>timeout stopping cpus >>Stopped at Debugger+0x4f: xchgl %ebx,in_Debugger.0 >>db> >> >>The machine is still in ddb, let me know if I can provide additional info. > > Try updating to a more recent current. I recently added some extra > debugging here that will attempt to better show who owns the lock and > where it was acquired. Same panic string, but a different call chain: spin lock sched lock held by 0xc21b2d10 for > 5 seconds exclusive spin mutex sched lock r = 0 (0xc036efc0) locked @ /usr/src/sys/kern/kern_mutex.c:512 panic: spin lock held too long cpuid = 3; lapic.id = 03000000 Stack backtrace: backtrace(c031ce60,3000000,c031c305,df0d0c30,100) at backtrace+0x17 panic(c031c305,c21b2d10,c21b2d10,c036efc0,bc) at panic+0x13d _mtx_lock_spin(c036efc0,2,c031a037,bc,c21b2720) at _mtx_lock_spin+0xb4 _mtx_lock_spin_flags(c036efc0,2,c031a037,bc,c21b2720) at _mtx_lock_spin_flags+0xb9 hardclock_process(df0d0ca0,df0d0ce4,c02d7062,0,c0390018) at hardclock_process+0x3c forwarded_hardclock(0) at forwarded_hardclock+0x11 Xhardclock(df0d0d10,c01998d7,c036f000,2,c031aaad) at Xhardclock+0x52 cpu_idle(c036f000,2,c031aaad,5f,c21b2720) at cpu_idle+0x8 idle_proc(0,df0d0d48,c031a997,30e,0) at idle_proc+0x45 fork_exit(c0199892,0,df0d0d48) at fork_exit+0xc3 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xdf0d0d7c, ebp = 0 --- Debugger("panic") timeout stopping cpus Stopped at Debugger+0x4f: xchgl %ebx,in_Debugger.0 db> Machine is still in ddb, in case you'd like me to poke around some more. Lars -- Lars Eggert USC Information Sciences Institute --------------ms010100080307030107020403 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJtjCC AzgwggKhoAMCAQICEGZFcrfMdPXPY3ZFhNAukQEwDQYJKoZIhvcNAQEEBQAwgdExCzAJBgNV BAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUgVG93bjEaMBgG A1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAmBgNVBAsTH0NlcnRpZmljYXRpb24gU2Vydmlj ZXMgRGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBDQTErMCkG CSqGSIb3DQEJARYccGVyc29uYWwtZnJlZW1haWxAdGhhd3RlLmNvbTAeFw0wMDA4MzAwMDAw MDBaFw0wNDA4MjcyMzU5NTlaMIGSMQswCQYDVQQGEwJaQTEVMBMGA1UECBMMV2VzdGVybiBD YXBlMRIwEAYDVQQHEwlDYXBlIFRvd24xDzANBgNVBAoTBlRoYXd0ZTEdMBsGA1UECxMUQ2Vy dGlmaWNhdGUgU2VydmljZXMxKDAmBgNVBAMTH1BlcnNvbmFsIEZyZWVtYWlsIFJTQSAyMDAw LjguMzAwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAN4zMqZjxwklRT7SbngnZ4HF2ogZ gpcO40QpimM1Km1wPPrcrvfudG8wvDOQf/k0caCjbZjxw0+iZdsN+kvx1t1hpfmFzVWaNRqd knWoJ67Ycvm6AvbXsJHeHOmr4BgDqHxDQlBRh4M88Dm0m1SKE4f/s5udSWYALQmJ7JRr6aFp AgMBAAGjTjBMMCkGA1UdEQQiMCCkHjAcMRowGAYDVQQDExFQcml2YXRlTGFiZWwxLTI5NzAS BgNVHRMBAf8ECDAGAQH/AgEAMAsGA1UdDwQEAwIBBjANBgkqhkiG9w0BAQQFAAOBgQAxsUtH XfkBceX1U2xdedY9mMAmE2KBIqcS+CKV6BtJtyd7BDm6/ObyJOuR+r3sDSo491BVqGz3Da1M G7wD9LXrokefbKIMWI0xQgkRbLAaadErErJAXWr5edDqLiXdiuT82w0fnQLzWtvKPPZE6iZp h39Ins6ln+eE2MliYq0FxjCCAzkwggKioAMCAQICAwp2bzANBgkqhkiG9w0BAQQFADCBkjEL MAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3du MQ8wDQYDVQQKEwZUaGF3dGUxHTAbBgNVBAsTFENlcnRpZmljYXRlIFNlcnZpY2VzMSgwJgYD VQQDEx9QZXJzb25hbCBGcmVlbWFpbCBSU0EgMjAwMC44LjMwMB4XDTAzMDgwMTE3MjkyOVoX DTA0MDczMTE3MjkyOVowVDEPMA0GA1UEBBMGRWdnZXJ0MQ0wCwYDVQQqEwRMYXJzMRQwEgYD VQQDEwtMYXJzIEVnZ2VydDEcMBoGCSqGSIb3DQEJARYNbGFyc2VAaXNpLmVkdTCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBAMb7PuLXnwV+45vwlkgogdSijd5HVqUB14bWvoK0 MjWPnkLPMDMDEezdsMG1BPiZyNeqXlJJtEgdAK8H2Mc9/qLeJUq3CoAeD6Wrjq4QaxJBXgdS KcGDeQAZSDgwUJS9vx9+cXJVfLyOYxJ+CLBcO/eu8PvSi17lk6oeAbrskSGDu/Xi1o2SC4Qm l69k8xcZQEMQDodkIk/U5SJmsCRGGYdy7opHZb58yXI8eiIGp5MlgryFmmgrp1pg3OYzPOR9 zJjn7Pu1vsd97LM5hLnKrmNuYt02jLNSjr8HmpLyWCDZq4Jlfq1YgNYZZ4KOSxipia7Bxjcs nMOsxEWiolkVVT8CAwEAAaNWMFQwKgYFK2UBBAEEITAfAgEAMBowGAIBBAQTTDJ1TXlmZkJO VWJOSkpjZFoyczAYBgNVHREEETAPgQ1sYXJzZUBpc2kuZWR1MAwGA1UdEwEB/wQCMAAwDQYJ KoZIhvcNAQEEBQADgYEANRaPsUtrdJzTW0AMj/EQamqxOkZnzwnPWGryqskMKIf+OKa+eaXp zlBv8CHdffv9hrYpvzWUxk0WW+YJ2LRdd4fFiVGXZCGU60eYeZGf7Z8ORoexylJpvUuKZCE4 aPGY2/QZXDfOs1NE82Bhgltx59dpWfH2K0dxbpHslO8/IbowggM5MIICoqADAgECAgMKdm8w DQYJKoZIhvcNAQEEBQAwgZIxCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUx EjAQBgNVBAcTCUNhcGUgVG93bjEPMA0GA1UEChMGVGhhd3RlMR0wGwYDVQQLExRDZXJ0aWZp Y2F0ZSBTZXJ2aWNlczEoMCYGA1UEAxMfUGVyc29uYWwgRnJlZW1haWwgUlNBIDIwMDAuOC4z MDAeFw0wMzA4MDExNzI5MjlaFw0wNDA3MzExNzI5MjlaMFQxDzANBgNVBAQTBkVnZ2VydDEN MAsGA1UEKhMETGFyczEUMBIGA1UEAxMLTGFycyBFZ2dlcnQxHDAaBgkqhkiG9w0BCQEWDWxh cnNlQGlzaS5lZHUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDG+z7i158FfuOb 8JZIKIHUoo3eR1alAdeG1r6CtDI1j55CzzAzAxHs3bDBtQT4mcjXql5SSbRIHQCvB9jHPf6i 3iVKtwqAHg+lq46uEGsSQV4HUinBg3kAGUg4MFCUvb8ffnFyVXy8jmMSfgiwXDv3rvD70ote 5ZOqHgG67JEhg7v14taNkguEJpevZPMXGUBDEA6HZCJP1OUiZrAkRhmHcu6KR2W+fMlyPHoi BqeTJYK8hZpoK6daYNzmMzzkfcyY5+z7tb7HfeyzOYS5yq5jbmLdNoyzUo6/B5qS8lgg2auC ZX6tWIDWGWeCjksYqYmuwcY3LJzDrMRFoqJZFVU/AgMBAAGjVjBUMCoGBStlAQQBBCEwHwIB ADAaMBgCAQQEE0wydU15ZmZCTlViTkpKY2RaMnMwGAYDVR0RBBEwD4ENbGFyc2VAaXNpLmVk dTAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBBAUAA4GBADUWj7FLa3Sc01tADI/xEGpqsTpG Z88Jz1hq8qrJDCiH/jimvnml6c5Qb/Ah3X37/Ya2Kb81lMZNFlvmCdi0XXeHxYlRl2QhlOtH mHmRn+2fDkaHscpSab1LimQhOGjxmNv0GVw3zrNTRPNgYYJbcefXaVnx9itHcW6R7JTvPyG6 MYID1TCCA9ECAQEwgZowgZIxCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUx EjAQBgNVBAcTCUNhcGUgVG93bjEPMA0GA1UEChMGVGhhd3RlMR0wGwYDVQQLExRDZXJ0aWZp Y2F0ZSBTZXJ2aWNlczEoMCYGA1UEAxMfUGVyc29uYWwgRnJlZW1haWwgUlNBIDIwMDAuOC4z MAIDCnZvMAkGBSsOAwIaBQCgggIPMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZI hvcNAQkFMQ8XDTAzMDgwODAwMDYzN1owIwYJKoZIhvcNAQkEMRYEFJAERy2cam+UQ+jof+7P TRtAEdgHMFIGCSqGSIb3DQEJDzFFMEMwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0G CCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGrBgkrBgEEAYI3EAQxgZ0w gZowgZIxCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNh cGUgVG93bjEPMA0GA1UEChMGVGhhd3RlMR0wGwYDVQQLExRDZXJ0aWZpY2F0ZSBTZXJ2aWNl czEoMCYGA1UEAxMfUGVyc29uYWwgRnJlZW1haWwgUlNBIDIwMDAuOC4zMAIDCnZvMIGtBgsq hkiG9w0BCRACCzGBnaCBmjCBkjELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2Fw ZTESMBAGA1UEBxMJQ2FwZSBUb3duMQ8wDQYDVQQKEwZUaGF3dGUxHTAbBgNVBAsTFENlcnRp ZmljYXRlIFNlcnZpY2VzMSgwJgYDVQQDEx9QZXJzb25hbCBGcmVlbWFpbCBSU0EgMjAwMC44 LjMwAgMKdm8wDQYJKoZIhvcNAQEBBQAEggEAppeds2qC3fEhaKugv/mBWEXE0GTNwgroI3vW xHxt7Tf1OiF7GAfieRWk+gcGzcCEY1+/Q/oOYuOi6+JqcTk2akxxDe4F7ZqF7yhENk+l71dG Ka4WMTN/k6IK51LQMvUZaCpaepQf2e3Da7U8xJjAx+KQ54NA2cJTpvIamofYKk2GjvHnCpzg UsQmG4HsGSHIeWJMzFxnDsn716a0PY6IemnZ0Rh6moX8zHxyZrKNFh/gFNxKYxKYaBgne/Un Wc8edHFpir7uNzGyEOxcfTccj2YcpCrGDwWH4lw7wAjGHQS1QtvKHo/2P9icQ8GRKK3xVuci 7CZPjyzevvUIhEydLgAAAAAAAA== --------------ms010100080307030107020403--