From owner-freebsd-current@FreeBSD.ORG Sun Jul 22 01:36:38 2007 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A8EE916A418 for ; Sun, 22 Jul 2007 01:36:38 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from ns.trinitel.com (186.161.36.72.static.reverse.layeredtech.com [72.36.161.186]) by mx1.freebsd.org (Postfix) with ESMTP id 8BFB013C458 for ; Sun, 22 Jul 2007 01:36:38 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from neutrino.vnode.org (r74-193-81-203.pfvlcmta01.grtntx.tl.dh.suddenlink.net [74.193.81.203]) (authenticated bits=0) by ns.trinitel.com (8.14.1/8.14.1) with ESMTP id l6M1abs8040941 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO) for ; Sat, 21 Jul 2007 20:36:37 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <46A2B49F.4030307@freebsd.org> Date: Sat, 21 Jul 2007 20:36:31 -0500 From: Eric Anderson User-Agent: Thunderbird 2.0.0.4 (X11/20070629) MIME-Version: 1.0 To: freebsd-current@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=failed version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on ns.trinitel.com Subject: LOR's, and a panic (ipf NAT related?) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Jul 2007 01:36:38 -0000 Today, on a -CURRENT from a few days ago (running ULE 3.0), I got a panic: panic: Trying sleep, but thread marked as sleeping prohibited cpuid = 0 #8 0xc074a474 in panic (fmt=0xc0a979a9 "Trying sleep, but thread marked as sleeping prohibited") at /usr/src/sys/kern/kern_shutdown.c:547 #9 0xc077abd2 in sleepq_add (wchan=0xc0e20980, lock=0x0, wmesg=0xc0e1b709 "ipf IP NAT rwlock", flags=3, queue=0) at /usr/src/sys/kern/subr_sleepqueue.c:289 #10 0xc07519ee in _sx_xlock_hard (sx=0xc0e20980, tid=3306307584, opts=0, file=0xc0e1b65e "/usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/ip_nat.c", line=4384) at /usr/src/sys/kern/kern_sx.c:548 #11 0xc0751d18 in _sx_xlock (sx=0xc0e20980, opts=0, file=0xc0e1b65e "/usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/ip_nat.c", line=4384) at sx.h:153 I also see these (maybe related) LOR's on bootup: lock order reversal: (sleepable after non-sleepable) 1st 0xc0e207f0 ipf nat io mutex (ipf nat io mutex) @ /usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/ip_nat.c:710 2nd 0xc0e20980 ipf IP NAT rwlock (ipf IP NAT rwlock) @ /usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/ip_nat.c:1062 KDB: stack backtrace: db_trace_self_wrapper(c0a96ace,e7a6ddc8,c0782f8e,c0a98fa0,c0e20980,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a98fa0,c0e20980,c0e1b709,c0e1b709,c0e1b65e,...) at kdb_backtrace+0x29 witness_checkorder(c0e20980,9,c0e1b65e,426,4,...) at witness_checkorder+0x6de _sx_xlock(c0e20980,0,c0e1b65e,426,0,...) at _sx_xlock+0x7d fr_nat_ioctl(c55d26c0,8034723c,3,0,c561d880,...) at fr_nat_ioctl+0x6ab fr_ioctlswitch(1,c55d26c0,8034723c,3,0,...) at fr_ioctlswitch+0x79 iplioctl(c53b3a00,8034723c,c55d26c0,3,c561d880,...) at iplioctl+0xd8 devfs_ioctl_f(c5643af8,8034723c,c55d26c0,c5740a00,c561d880,...) at devfs_ioctl_f+0xc9 kern_ioctl(c561d880,3,8034723c,c55d26c0,0,...) at kern_ioctl+0x243 ioctl(c561d880,e7a6ecfc,c,c,c0b40130,...) at ioctl+0x134 syscall(e7a6ed38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (54, FreeBSD ELF32, ioctl), eip = 0x28167e53, esp = 0xbfbfec9c, ebp = 0xbfbfed08 --- lock order reversal: (sleepable after non-sleepable) 1st 0xc0bfb798 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:73 2nd 0xc0e20880 ipf filter load/unload mutex (ipf filter load/unload mutex) @ /usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/fil.c:2419 KDB: stack backtrace: db_trace_self_wrapper(c0a96ace,e3cc299c,c0782f8e,c0a98fa0,c0e20880,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a98fa0,c0e20880,c0e1b400,c0e1b400,c0e1c7c1,...) at kdb_backtrace+0x29 witness_checkorder(c0e20880,1,c0e1c7c1,973,c53f1000,...) at witness_checkorder+0x6de _sx_slock(c0e20880,0,c0e1c7c1,973,c5126880,...) at _sx_slock+0x7d fr_check(c57be024,14,c5269400,0,e3cc2ad0,...) at fr_check+0x5a fr_check_wrapper(0,e3cc2ad0,c5269400,1,0,...) at fr_check_wrapper+0x3f pfil_run_hooks(c0bfb780,e3cc2b24,c5269400,1,0,...) at pfil_run_hooks+0x88 ip_input(c5782200,c0652b32,800,c5269400,800,...) at ip_input+0x24d netisr_dispatch(2,c5782200,46a266ac,89ca3,c5269400,...) at netisr_dispatch+0x73 ether_demux(c5269400,c5782200,3,0,3,...) at ether_demux+0x1f1 ether_input(c5269400,c5782200,18,c0780d4e,c5782200,...) at ether_input+0x37f ieee80211_deliver_data(c526a22c,c57df000,c5782200,18,c0aa34ce,...) at ieee80211_deliver_data+0x13e ieee80211_input(c526a22c,c5782200,c57df000,21,ffffffaa,...) at ieee80211_input+0x1159 ath_rx_proc(c526a000,1,c0a97ce7,52,c527bb9c,...) at ath_rx_proc+0x52d taskqueue_run(c527bb80,c527bb9c,0,c0a8b649,0,...) at taskqueue_run+0x10b taskqueue_thread_loop(c526b65c,e3cc2d38,c0a9069e,315,c5223000,...) at taskqueue_thread_loop+0x68 fork_exit(c077c220,c526b65c,e3cc2d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3cc2d70, ebp = 0 --- I have a vmcore if anyone wants me to debug further.. Eric