From owner-freebsd-current@FreeBSD.ORG Sat Sep 13 12:35:12 2008 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 055BA10656FB; Sat, 13 Sep 2008 12:35:12 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (unknown [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 990158FC15; Sat, 13 Sep 2008 12:35:11 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTP id 35D9B46BC1; Sat, 13 Sep 2008 06:06:35 -0400 (EDT) Date: Sat, 13 Sep 2008 11:06:35 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Giorgos Keramidas In-Reply-To: <87prnjh80z.fsf@kobe.laptop> Message-ID: References: <87prnjh80z.fsf@kobe.laptop> User-Agent: Alpine 1.10 (BSF 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-current@freebsd.org, julian@FreeBSD.org, jhb@FreeBSD.org Subject: Re: panic in rt_check_fib() X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Sep 2008 12:35:12 -0000 On Fri, 5 Sep 2008, Giorgos Keramidas wrote: > A kernel that I built last night to test Ed's "packet mode" for ptys > included all the changes up to 182743 panics with: I had an identical panic on 7-STABLE last night: db> bt Tracing pid 782 tid 100091 td 0xc4496440 kdb_enter_why(c0b25ea1,c0b25ea1,c0b24c19,e6772978,0,...) at kdb_enter_why+0x3a panic(c0b24c19,c0b32d59,c0b32d7a,633,c436c9b0,...) at panic+0x12c _mtx_lock_sleep(c436ddf4,c4496440,0,c0b32d7a,633,...) at _mtx_lock_sleep+0x4a _mtx_lock_flags(c436ddf4,0,c0b32d7a,633,c436ca14,...) at _mtx_lock_flags+0xd1 rt_check_fib(e6772a0c,e6772a28,c424ea90,0,e6772a1c,...) at rt_check_fib+0x2b4 in_rt_check(e6772a0c,e6772a28,c424ea90,0,0,...) at in_rt_check+0x26 arpresolve(c4040000,c436c9b0,c4240800,c424ea90,e6772a42,...) at arpresolve+0xb9 ether_output(c4040000,c4240800,c424ea90,c436c9b0,c450b9d8,...) at ether_output+0x7e ip_output(c4240800,0,e6772ab0,0,0,...) at ip_output+0xa34 udp_send(c44f74b0,0,c4240800,c4514ac0,0,...) at udp_send+0x58b sosend_dgram(c44f74b0,c4514ac0,e6772bd4,c4240800,0,...) at sosend_dgram+0x352 sosend(c44f74b0,c4514ac0,e6772bd4,0,0,...) at sosend+0x54 kern_sendit(c4496440,20,e6772c58,0,0,...) at kern_sendit+0x106 sendit(0,1,e6772c54,28,c426a090,...) at sendit+0x162 sendmsg(c4496440,e6772cfc,c,c4496630,c0bd53c0,...) at sendmsg+0x78 syscall(e6772d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 Unfortunately, I was unable to successfully get a crashdump -- not entirely sure why as it seemed to go to disk ok. Robert N M Watson Computer Laboratory University of Cambridge > > ======================================================================== > > root@kobe:/var/crash# kgdb /boot/kernel/kernel vmcore.5 > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i386-marcel-freebsd"... > > Unread portion of the kernel message buffer: > panic: _mtx_lock_sleep: recursed on non-recursive mutex rtentry @ /home/build/src/sys/net/route.c:1742 > > cpuid = 0 > Uptime: 5m26s > Physical memory: 2026 MB > Dumping 80 MB: 65 49 33 17 1 > > Reading symbols from /boot/kernel/snd_hda.ko...Reading symbols from /boot/kernel/snd_hda.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/snd_hda.ko > Reading symbols from /boot/kernel/sound.ko...Reading symbols from /boot/kernel/sound.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/sound.ko > Reading symbols from /boot/kernel/if_iwn.ko...Reading symbols from /boot/kernel/if_iwn.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/if_iwn.ko > Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/acpi.ko > Reading symbols from /boot/kernel/snake_saver.ko...Reading symbols from /boot/kernel/snake_saver.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/snake_saver.ko > #0 doadump () at pcpu.h:221 > 221 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) list > 216 in pcpu.h > (kgdb) bt > #0 doadump () at pcpu.h:221 > #1 0xc05e13ac in boot (howto=260) at /home/build/src/sys/kern/kern_shutdown.c:418 > #2 0xc05e1678 in panic (fmt=Variable "fmt" is not available. > ) at /home/build/src/sys/kern/kern_shutdown.c:572 > #3 0xc05d3fda in _mtx_lock_sleep (m=0xc573eba4, tid=3314466816, opts=0, file=0xc08f457a "/home/build/src/sys/net/route.c", line=1742) at /home/build/src/sys/kern/kern_mutex.c:310 > #4 0xc05d422f in _mtx_lock_flags (m=0xc573eba4, opts=0, file=0xc08f457a "/home/build/src/sys/net/route.c", line=1742) at /home/build/src/sys/kern/kern_mutex.c:182 > #5 0xc0694ad8 in rt_check_fib (lrt=0xe7c299ec, lrt0=0xe7c29a08, dst=0xc5550710, fibnum=0) at /home/build/src/sys/net/route.c:1742 > #6 0xc06caf36 in in_rt_check (lrt=0xe7c299ec, lrt0=0xe7c29a08, dst=0xc5550710, fibnum=0) at /home/build/src/sys/netinet/in_rmx.c:472 > #7 0xc06c0ecd in arpresolve (ifp=0xc51fd800, rt0=0xc573eca8, m=0xc59c2200, dst=0xc5550710, desten=0xe7c29a22 "") at /home/build/src/sys/netinet/if_ether.c:388 > #8 0xc0689a9e in ether_output (ifp=0xc51fd800, m=0xc59c2200, dst=0xc5550710, rt0=0xc573eca8) at /home/build/src/sys/net/if_ethersubr.c:183 > #9 0xc06d1bf1 in ip_output (m=0xc59c2200, opt=0x0, ro=0xe7c29aa8, flags=Variable "flags" is not available. > ) at /home/build/src/sys/netinet/ip_output.c:563 > #10 0xc073ecfb in udp_send (so=0xc573b498, flags=0, m=0xc59c2200, addr=0xc597e2f0, control=0x0, td=0xc58ec000) at /home/build/src/sys/netinet/udp_usrreq.c:1060 > #11 0xc064530f in sosend_dgram (so=0xc573b498, addr=0xc597e2f0, uio=0xe7c29bd4, top=0xc59c2200, control=0x0, flags=Variable "flags" is not available. > ) at /home/build/src/sys/kern/uipc_socket.c:1059 > #12 0xc0643054 in sosend (so=0xc573b498, addr=0xc597e2f0, uio=0xe7c29bd4, top=0x0, control=0x0, flags=0, td=0xc58ec000) at /home/build/src/sys/kern/uipc_socket.c:1292 > #13 0xc064bf15 in kern_sendit (td=0xc58ec000, s=516, mp=0xe7c29c54, flags=0, control=0x0, segflg=UIO_USERSPACE) at /home/build/src/sys/kern/uipc_syscalls.c:782 > #14 0xc064c121 in sendit (td=0xc58ec000, s=516, mp=0xe7c29c54, flags=0) at /home/build/src/sys/kern/uipc_syscalls.c:719 > #15 0xc064c1d1 in sendmsg (td=0xc58ec000, uap=0xe7c29cf8) at /home/build/src/sys/kern/uipc_syscalls.c:915 > #16 0xc0884d13 in syscall (frame=0xe7c29d38) at /home/build/src/sys/i386/i386/trap.c:1090 > #17 0xc0869020 in Xint0x80_syscall () at /home/build/src/sys/i386/i386/exception.s:261 > #18 0x00000033 in ?? () > Previous frame inner to this frame (corrupt stack?) > (kgdb) > > ======================================================================== > > From the limited testing I could do today it seems that the following > changes might be useful to track down why this is happening: > > /head@182698 -> ok so far > /head@182743 -> panic > > I don't see any rt_check_fib() changes in this commit range, so it may > be false that /head@182698 is ok. It just doesn't panic immediately > when I try to bring up my re0 interface and set the default route. > > - Giorgos > >