From owner-freebsd-current@FreeBSD.ORG Wed Sep 12 08:49:46 2007 Return-Path: Delivered-To: FreeBSD-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6291816A46C for ; Wed, 12 Sep 2007 08:49:46 +0000 (UTC) (envelope-from lioux-list@uol.com.br) Received: from goat.gigo.com (goat.gigo.com [216.218.228.114]) by mx1.freebsd.org (Postfix) with ESMTP id 48C1A13C4B3 for ; Wed, 12 Sep 2007 08:49:46 +0000 (UTC) (envelope-from lioux-list@uol.com.br) Received: from 201.24.13.31 (201-24-13-31.bsace703.dsl.brasiltelecom.net.br [201.24.13.31]) by goat.gigo.com (Postfix) with ESMTP id 3C25FB8EC for ; Wed, 12 Sep 2007 01:49:45 -0700 (PDT) Received: (qmail 10068 invoked by uid 1001); 12 Sep 2007 05:49:14 -0300 Message-ID: <20070912084914.9985.qmail@exxodus.fedaykin.here> Date: Wed, 12 Sep 2007 05:49:14 -0259 From: Mario Sergio Fujikawa Ferreira To: FreeBSD-current@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="zhXaljGHf11kAtnf" Content-Disposition: inline User-Agent: Mutt/1.5.16 (2007-06-09) Cc: Subject: panic: Trying sleep, but thread marked as sleeping prohibited X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Sep 2007 08:49:46 -0000 --zhXaljGHf11kAtnf Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi, I had 3 panics today. All seem to be related to ipf code. I have disabled ipf for the time being. I am attaching the kgdb backtraces. I am saving the cores. Let me know if there is anything I can do to help. FreeBSD exxodus.home.home 7.0-CURRENT FreeBSD 7.0-CURRENT #2: Mon Sep 10 15:01:49 BRT 2007 root@exxodus.home.home:/usr/obj/usr/src/sys/LIOUX i386 -current as of Sep 10 2007. Regards, -- Mario S F Ferreira - DF - Brazil - "I guess this is a signature." feature, n: a documented bug | bug, n: an undocumented feature --zhXaljGHf11kAtnf Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="vmcore.1.log" Script started on Tue Sep 11 23:20:45 2007 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: panic: Trying sleep, but thread marked as sleeping prohibited cpuid = 0 KDB: enter: panic Uptime: 3h53m15s Physical memory: 2030 MB Dumping 260 MB: 245 229 213 197 181 165 149 133 117 101 85 69 53 37 21 5 #0 doadump () at pcpu.h:195 195 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) backtrace #0 doadump () at pcpu.h:195 #1 0x805a1f9f in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0x805a226b in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #3 0x805d2e82 in sleepq_add (wchan=0x8086cc20, lock=0x0, wmesg=0x807dbf57 "ipf IP state rwlock", flags=3, queue=0) at /usr/src/sys/kern/subr_sleepqueue.c:289 #4 0x805a9e66 in _sx_xlock_hard (sx=0x8086cc20, tid=2236636352, opts=0, file=0x807dbe72 "/usr/src/sys/contrib/ipfilter/netinet/ip_state.c", line=3013) at /usr/src/sys/kern/kern_sx.c:548 #5 0x805a9f1e in _sx_xlock (sx=0x8086cc20, opts=0, file=0x807dbe72 "/usr/src/sys/contrib/ipfilter/netinet/ip_state.c", line=3013) at sx.h:153 #6 0x8047591c in fr_timeoutstate () at /usr/src/sys/contrib/ipfilter/netinet/ip_state.c:3013 #7 0x80462059 in fr_slowtimer (ptr=0x0) at /usr/src/sys/contrib/ipfilter/netinet/ip_frag.c:859 #8 0x805b3d99 in softclock (dummy=0x0) at /usr/src/sys/kern/kern_timeout.c:281 #9 0x80585eb5 in ithread_loop (arg=0x85503310) at /usr/src/sys/kern/kern_intr.c:1036 #10 0x80583338 in fork_exit (callout=0x80585d00 , arg=0x85503310, frame=0xd6318d38) at /usr/src/sys/kern/kern_fork.c:797 #11 0x8078a840 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 (kgdb) quit Script done on Tue Sep 11 23:23:24 2007 --zhXaljGHf11kAtnf Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="vmcore.2.log" Script started on Tue Sep 11 23:23:30 2007 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: panic: Trying sleep, but thread marked as sleeping prohibited cpuid = 0 KDB: enter: panic Uptime: 17m22s Physical memory: 2030 MB Dumping 231 MB: 216 200 184 168 152 136 120 104 88 72 56 40 24 8 #0 doadump () at pcpu.h:195 195 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) backtrace #0 doadump () at pcpu.h:195 #1 0x805a1f9f in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0x805a226b in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #3 0x805d2e82 in sleepq_add (wchan=0x8086cc20, lock=0x0, wmesg=0x807dbf57 "ipf IP state rwlock", flags=3, queue=0) at /usr/src/sys/kern/subr_sleepqueue.c:289 #4 0x805a9e66 in _sx_xlock_hard (sx=0x8086cc20, tid=2236636352, opts=0, file=0x807dbe72 "/usr/src/sys/contrib/ipfilter/netinet/ip_state.c", line=3013) at /usr/src/sys/kern/kern_sx.c:548 #5 0x805a9f1e in _sx_xlock (sx=0x8086cc20, opts=0, file=0x807dbe72 "/usr/src/sys/contrib/ipfilter/netinet/ip_state.c", line=3013) at sx.h:153 #6 0x8047591c in fr_timeoutstate () at /usr/src/sys/contrib/ipfilter/netinet/ip_state.c:3013 #7 0x80462059 in fr_slowtimer (ptr=0x0) at /usr/src/sys/contrib/ipfilter/netinet/ip_frag.c:859 #8 0x805b3d99 in softclock (dummy=0x0) at /usr/src/sys/kern/kern_timeout.c:281 #9 0x80585eb5 in ithread_loop (arg=0x85503310) at /usr/src/sys/kern/kern_intr.c:1036 #10 0x80583338 in fork_exit (callout=0x80585d00 , arg=0x85503310, frame=0xd6318d38) at /usr/src/sys/kern/kern_fork.c:797 #11 0x8078a840 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 (kgdb) quit Script done on Tue Sep 11 23:23:49 2007 --zhXaljGHf11kAtnf Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="vmcore.3.log" Script started on Tue Sep 11 23:36:45 2007 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: panic: Trying sleep, but thread marked as sleeping prohibited cpuid = 0 KDB: enter: panic Uptime: 48m49s Physical memory: 2030 MB Dumping 249 MB: 234 218 202 186 170 154 138 122 106 90 74 58 42 26 10 #0 doadump () at pcpu.h:195 195 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) backtrace #0 doadump () at pcpu.h:195 #1 0x805a1f9f in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0x805a226b in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #3 0x805d2e82 in sleepq_add (wchan=0x8086cc20, lock=0x0, wmesg=0x807dbf57 "ipf IP state rwlock", flags=3, queue=0) at /usr/src/sys/kern/subr_sleepqueue.c:289 #4 0x805a9e66 in _sx_xlock_hard (sx=0x8086cc20, tid=2236636352, opts=0, file=0x807dbe72 "/usr/src/sys/contrib/ipfilter/netinet/ip_state.c", line=3013) at /usr/src/sys/kern/kern_sx.c:548 #5 0x805a9f1e in _sx_xlock (sx=0x8086cc20, opts=0, file=0x807dbe72 "/usr/src/sys/contrib/ipfilter/netinet/ip_state.c", line=3013) at sx.h:153 #6 0x8047591c in fr_timeoutstate () at /usr/src/sys/contrib/ipfilter/netinet/ip_state.c:3013 #7 0x80462059 in fr_slowtimer (ptr=0x0) at /usr/src/sys/contrib/ipfilter/netinet/ip_frag.c:859 #8 0x805b3d99 in softclock (dummy=0x0) at /usr/src/sys/kern/kern_timeout.c:281 #9 0x80585eb5 in ithread_loop (arg=0x85503310) at /usr/src/sys/kern/kern_intr.c:1036 #10 0x80583338 in fork_exit (callout=0x80585d00 , arg=0x85503310, frame=0xd6318d38) at /usr/src/sys/kern/kern_fork.c:797 #11 0x8078a840 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 (kgdb) list 205 call fork_exit 206 addl $12,%esp 207 /* cut from syscall */ 208 209 /* 210 * Return via doreti to handle ASTs. 211 */ 212 MEXITCOUNT 213 jmp doreti 214 (kgdb) up #1 0x805a1f9f in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 409 doadump(); (kgdb) list 404 405 /* XXX This doesn't disable interrupts any more. Reconsider? */ 406 splhigh(); 407 408 if ((howto & (RB_HALT|RB_DUMP)) == RB_DUMP && !cold && !dumping) 409 doadump(); 410 411 /* Now that we're going to really halt the system... */ 412 EVENTHANDLER_INVOKE(shutdown_final, howto); 413 (kgdb) up #2 0x805a226b in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 563 boot(bootopt); (kgdb) list 558 /*thread_lock(td); */ 559 td->td_flags |= TDF_INPANIC; 560 /* thread_unlock(td); */ 561 if (!sync_on_panic) 562 bootopt |= RB_NOSYNC; 563 boot(bootopt); 564 } 565 566 /* 567 * Support for poweroff delay. (kgdb) up #3 0x805d2e82 in sleepq_add (wchan=0x8086cc20, lock=0x0, wmesg=0x807dbf57 "ipf IP state rwlock", flags=3, queue=0) at /usr/src/sys/kern/subr_sleepqueue.c:289 289 KASSERT(!(td->td_pflags & TDP_NOSLEEPING), (kgdb) list 284 MPASS(td->td_sleepqueue != NULL); 285 MPASS(wchan != NULL); 286 MPASS((queue >= 0) && (queue < NR_SLEEPQS)); 287 288 /* If this thread is not allowed to sleep, die a horrible death. */ 289 KASSERT(!(td->td_pflags & TDP_NOSLEEPING), 290 ("Trying sleep, but thread marked as sleeping prohibited")); 291 292 /* Look up the sleep queue associated with the wait channel 'wchan'. */ 293 sq = sleepq_lookup(wchan); (kgdb) up #4 0x805a9e66 in _sx_xlock_hard (sx=0x8086cc20, tid=2236636352, opts=0, file=0x807dbe72 "/usr/src/sys/contrib/ipfilter/netinet/ip_state.c", line=3013) at /usr/src/sys/kern/kern_sx.c:548 548 sleepq_add(&sx->lock_object, NULL, sx->lock_object.lo_name, (kgdb) list 543 __func__, sx); 544 545 GIANT_SAVE(); 546 lock_profile_obtain_lock_failed(&sx->lock_object, &contested, 547 &waittime); 548 sleepq_add(&sx->lock_object, NULL, sx->lock_object.lo_name, 549 SLEEPQ_SX | ((opts & SX_INTERRUPTIBLE) ? 550 SLEEPQ_INTERRUPTIBLE : 0), SQ_EXCLUSIVE_QUEUE); 551 if (!(opts & SX_INTERRUPTIBLE)) 552 sleepq_wait(&sx->lock_object); (kgdb) up #5 0x805a9f1e in _sx_xlock (sx=0x8086cc20, opts=0, file=0x807dbe72 "/usr/src/sys/contrib/ipfilter/netinet/ip_state.c", line=3013) at sx.h:153 153 error = _sx_xlock_hard(sx, tid, opts, file, line); (kgdb) list 148 { 149 uintptr_t tid = (uintptr_t)td; 150 int error = 0; 151 152 if (!atomic_cmpset_acq_ptr(&sx->sx_lock, SX_LOCK_UNLOCKED, tid)) 153 error = _sx_xlock_hard(sx, tid, opts, file, line); 154 else 155 lock_profile_obtain_lock_success(&sx->lock_object, 0, 0, file, 156 line); 157 (kgdb) up #6 0x8047591c in fr_timeoutstate () at /usr/src/sys/contrib/ipfilter/netinet/ip_state.c:3013 3013 WRITE_ENTER(&ipf_state); (kgdb) list 3008 ipftqent_t *tqe, *tqn; 3009 ipstate_t *is; 3010 SPL_INT(s); 3011 3012 SPL_NET(s); 3013 WRITE_ENTER(&ipf_state); 3014 for (ifq = ips_tqtqb; ifq != NULL; ifq = ifq->ifq_next) 3015 for (tqn = ifq->ifq_head; ((tqe = tqn) != NULL); ) { 3016 if (tqe->tqe_die > fr_ticks) 3017 break; (kgdb) up #7 0x80462059 in fr_slowtimer (ptr=0x0) at /usr/src/sys/contrib/ipfilter/netinet/ip_frag.c:859 859 fr_timeoutstate(); (kgdb) list 854 { 855 READ_ENTER(&ipf_global); 856 857 ipf_expiretokens(); 858 fr_fragexpire(); 859 fr_timeoutstate(); 860 fr_natexpire(); 861 fr_authexpire(); 862 fr_ticks++; 863 if (fr_running <= 0) (kgdb) up #8 0x805b3d99 in softclock (dummy=0x0) at /usr/src/sys/kern/kern_timeout.c:281 281 c_func(c_arg); (kgdb) list 276 } 277 #ifdef DIAGNOSTIC 278 binuptime(&bt1); 279 #endif 280 THREAD_NO_SLEEPING(); 281 c_func(c_arg); 282 THREAD_SLEEPING_OK(); 283 #ifdef DIAGNOSTIC 284 binuptime(&bt2); 285 bintime_sub(&bt2, &bt1); (kgdb) up #9 0x80585eb5 in ithread_loop (arg=0x85503310) at /usr/src/sys/kern/kern_intr.c:1036 1036 ih->ih_handler(ih->ih_argument); (kgdb) list 1031 __func__, p->p_pid, (void *)ih->ih_handler, 1032 ih->ih_argument, ih->ih_name, ih->ih_flags); 1033 1034 if (!(ih->ih_flags & IH_MPSAFE)) 1035 mtx_lock(&Giant); 1036 ih->ih_handler(ih->ih_argument); 1037 if (!(ih->ih_flags & IH_MPSAFE)) 1038 mtx_unlock(&Giant); 1039 } 1040 if (!(ie->ie_flags & IE_SOFT)) (kgdb) up #10 0x80583338 in fork_exit (callout=0x80585d00 , arg=0x85503310, frame=0xd6318d38) at /usr/src/sys/kern/kern_fork.c:797 797 callout(arg, frame); (kgdb) list 792 * cpu_set_fork_handler intercepts this function call to 793 * have this call a non-return function to stay in kernel mode. 794 * initproc has its own fork handler, but it does return. 795 */ 796 KASSERT(callout != NULL, ("NULL callout in fork_exit")); 797 callout(arg, frame); 798 799 /* 800 * Check if a kernel thread misbehaved and returned from its main 801 * function. (kgdb) up #11 0x8078a840 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 205 call fork_exit Current language: auto; currently asm (kgdb) list 200 201 ENTRY(fork_trampoline) 202 pushl %esp /* trapframe pointer */ 203 pushl %ebx /* arg1 */ 204 pushl %esi /* function */ 205 call fork_exit 206 addl $12,%esp 207 /* cut from syscall */ 208 209 /* (kgdb) quit Script done on Tue Sep 11 23:37:21 2007 --zhXaljGHf11kAtnf--