From owner-freebsd-current@FreeBSD.ORG  Fri Nov  7 07:50:37 2003
Return-Path: <owner-freebsd-current@FreeBSD.ORG>
Delivered-To: freebsd-current@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 01E7316A4CE
	for <freebsd-current@freebsd.org>;
	Fri,  7 Nov 2003 07:50:37 -0800 (PST)
Received: from www.premsoft.co.za (www.rune.za.net [196.37.142.13])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 6B03D43FE9
	for <freebsd-current@freebsd.org>;
	Fri,  7 Nov 2003 07:50:28 -0800 (PST)
	(envelope-from freebsd-questions@premsoft.co.za)
Received: from jaco[196.33.196.21] by www.premsoft.co.za;
     Fri, 07 Nov 2003 18:12:24 +0200
Message-ID: <070401c3a546$d7e92360$3635a8c0@jaco>
From: "Jaco H. van Tonder" <freebsd-questions@premsoft.co.za>
To: "FreeBSD-Hackers" <freebsd-hackers@freebsd.org>
Date: Fri, 7 Nov 2003 17:49:35 +0200
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 8bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2800.1158
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165
cc: freebsd-current@freebsd.org
Subject: [CURRENT] Panic in -CURRENT of 20031105
X-BeenThere: freebsd-current@freebsd.org
X-Mailman-Version: 2.1.1
Precedence: list
List-Id: Discussions about the use of FreeBSD-current
	<freebsd-current.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>,
	<mailto:freebsd-current-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current>
List-Post: <mailto:freebsd-current@freebsd.org>
List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>,
	<mailto:freebsd-current-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 07 Nov 2003 15:50:37 -0000

Hi All,

I get panics at random times of the day with -CURRENT from 20031105, with
absolutely no load on the machine.

The machine acts as a dial-up server/gateway/firewall for my local lan. I
managed to get a coredump.

The contents of the rt pointer passed to RTFREE() does really not look right
to me. These in particular:
rt_llinfo = 0xc0f95880 "\220ǼÁ\200qùÀ"
rn_Key = 0xc1bcc7a0 "0AùÀ8AùÀ"

Anyone got any ideas?

Thanks
Jaco

firewall# uname -a

FreeBSD firewall.symphiano 5.1-CURRENT FreeBSD 5.1-CURRENT #0: Fri Nov 7
15:31:34 SAST 2003
jacovt@firewall.symphiano:/usr/src/sys/i386/compile/FIREWALL i386

firewall# dmesg
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 5.1-CURRENT #0: Fri Nov 7 15:31:34 SAST 2003
jacovt@firewall.symphiano:/usr/src/sys/i386/compile/FIREWALL
Preloaded elf kernel "/boot/kernel/kernel" at 0xc06cf000.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Pentium II/Pentium II Xeon/Celeron (501.14-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x665 Stepping = 5
Features=0x183f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,
PAT,PSE36,MMX,FXSR>
real memory = 134217728 (128 MB)
avail memory = 124952576 (119 MB)
Pentium Pro MTRR support enabled
npx0: [FAST]
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcibios: BIOS version 2.10
Using $PIR table, 7 entries at 0xc00fded0
pcib0: <Host to PCI bridge> at pcibus 0 on motherboard
pci0: <PCI bus> on pcib0
pci_cfgintr: 0:8 INTA BIOS irq 11
pci_cfgintr: 0:10 INTA BIOS irq 10
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
pci_cfgintr: 0:1 INTA routed to irq 11
pcib1: slot 0 INTA is routed to irq 11
pci1: <display, VGA> at device 0.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <VIA 82C596 UDMA33 controller> port 0xe000-0xe00f at device 7.1 on
pci0
ata0: at 0x1f0 irq 14 on atapci0
ata0: [MPSAFE]
ata1: at 0x170 irq 15 on atapci0
ata1: [MPSAFE]
pci0: <serial bus, USB> at device 7.2 (no driver attached)
pci0: <bridge, HOST-PCI> at device 7.3 (no driver attached)
xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0xe800-0xe87f mem
0xdf800000-0xdf80007f irq 11 at device 8.0 on pci0
xl0: Ethernet address: 00:50:da:45:92:81
miibus0: <MII bus> on xl0
xlphy0: <3c905C 10/100 internal PHY> on miibus0
xlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
pci0: <multimedia, audio> at device 10.0 (no driver attached)
orm0: <Option ROMs> at iomem 0xc8000-0xc87ff,0xc0000-0xc7fff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x64,0x60 on isa0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
fdc0: <Enhanced floppy controller (i82077, NE72065 or clone)> at port
0x3f7,0x3f0-0x3f5 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
ppi0: <Parallel I/O> on ppbus0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
unknown: <PNP0303> can't assign resources (port)
unknown: <PNP0501> can't assign resources (port)
unknown: <PNP0700> can't assign resources (port)
unknown: <PNP0400> can't assign resources (port)
unknown: <PNP0501> can't assign resources (port)
Timecounter "TSC" frequency 501139304 Hz quality 800
Timecounters tick every 10.000 msec
ipfw2 initialized, divert enabled, rule-based forwarding enabled, default to
deny, logging limited to 1000 packets/entry by default
GEOM: create disk ad0 dp=0xc1c48370
ad0: 2014MB <ST32122A> [4092/16/63] at ata0-master UDMA33
acd0: CDROM <CD-ROM 50X> at ata1-slave PIO4
Mounting root from ufs:/dev/ad0s1a
WARNING: / was not properly dismounted
WARNING: /tmp was not properly dismounted
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted

firewall# cat /usr/src/sys/i386/conf/FIREWALL
machine         i386
cpu             I686_CPU
ident           FIREWALL
maxusers        0

options         SCHED_4BSD              #4BSD scheduler
options         INET                    #InterNETworking
#options        INET6                   #IPv6 communications protocols
options         FFS                     #Berkeley Fast Filesystem
options         SOFTUPDATES             #Enable FFS soft updates support
options         UFS_ACL                 #Support for access control lists
options         UFS_DIRHASH             #Improve performance on big
directories
options         CD9660                  #ISO 9660 Filesystem
options         PROCFS                  #Process filesystem (requires
PSEUDOFS)
options         PSEUDOFS                #Pseudo-filesystem framework
options         COMPAT_43               #Compatible with BSD 4.3 [KEEP
THIS!]
options         COMPAT_FREEBSD4         #Compatible with FreeBSD4
#options        SCSI_DELAY=5000         #Delay (in ms) before probing SCSI
options         KTRACE                  #ktrace(1) support
options         SYSVSHM                 #SYSV-style shared memory
options         SYSVMSG                 #SYSV-style message queues
options         SYSVSEM                 #SYSV-style semaphores
options         _KPOSIX_PRIORITY_SCHEDULING #Posix P1003_1B real-time
extensions
options         KBD_INSTALL_CDEV        # install a CDEV entry in /dev
#options        AHC_REG_PRETTY_PRINT    # Print register bitfields in debug
                                        # output.  Adds ~128k to driver.
#options        AHD_REG_PRETTY_PRINT    # Print register bitfields in debug
                                        # output.  Adds ~215k to driver.

# Debugging for use in -current
makeoptions     DEBUG=-g
options         DDB                     #Enable the kernel debugger
options         DDB_UNATTENDED
#options        INVARIANTS              #Enable calls of extra sanity
checking
#options        INVARIANT_SUPPORT       #Extra sanity checks of internal
structures, required by INVARIANTS
#options        WITNESS                 #Enable checks to detect deadlocks
and cycles
#options        WITNESS_SKIPSPIN        #Don't run witness on spinlocks for
speed

device          isa
device          pci

# Floppy drives
device          fdc

# ATA and ATAPI devices
device          ata
device          atadisk                 # ATA disk drives
device          atapicd                 # ATAPI CDROM drives
device          atapifd                 # ATAPI floppy drives
#device         atapist                 # ATAPI tape drives
options         ATA_STATIC_ID           #Static device numbering

# atkbdc0 controls both the keyboard and the PS/2 mouse
device          atkbdc          # AT keyboard controller
device          atkbd           # AT keyboard
device          psm             # PS/2 mouse

device          vga             # VGA video card driver

# syscons is the default console driver, resembling an SCO console
device          sc

#device         agp             # support several AGP chipsets

# Floating point support - do not disable.
device          npx

# Serial (COM) ports
device          sio             # 8250, 16[45]50 based serial ports

# Parallel port
device          ppc
device          ppbus           # Parallel port bus (required)
device          ppi             # Parallel port interface device


# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device          miibus          # MII bus support
device          xl              # 3Com 3c90x (``Boomerang'', ``Cyclone'')

# Pseudo devices - the number indicates how many units to allocate.
device          random          # Entropy device
device          loop            # Network loopback
device          ether           # Ethernet support
device          tun             # Packet tunnel.
device          pty             # Pseudo-ttys (telnet etc)
#device         gif             # IPv6 and IPv4 tunneling
#device         faith           # IPv6-to-IPv4 relaying (translation)

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
device          bpf             # Berkeley packet filter

# Firewall Stuff
options         IPFIREWALL
options         IPFIREWALL_VERBOSE
options         IPFIREWALL_VERBOSE_LIMIT=1000
options         IPDIVERT
firewall#


firewall# nm /var/crash/kernel.debug | grep c049d
c049d270 T _mtx_lock_spin
c049d330 T _mtx_unlock_sleep
c049d5f0 T mtx_destroy
c049d540 T mtx_init
c049d920 T mtx_pool_alloc
c049d7c0 T mtx_pool_create
c049d850 T mtx_pool_destroy
c049d6f0 T mtx_pool_find
c049d720 t mtx_pool_initialize
c049d8f0 t mtx_pool_setup_dynamic
c049d8b0 t mtx_pool_setup_static
c049d510 T mtx_sysinit
c049d620 T mutex_init
c049d9e0 T ntp_adjtime
c049d940 t ntp_sysctl
c049de30 T ntp_update_second

firewall# gdb -k /var/crash/kernel.debug ~jacovt/vmcore.1
GNU gdb 5.2.1 (FreeBSD)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-undermydesk-freebsd"...
panic: page fault
panic messages:
---
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x68
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc049d182
stack pointer = 0x10:0xc8da4a14
frame pointer = 0x10:0xc8da4a3c
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 29 (swi1: net)
trap number = 12
panic: page fault
syncing disks, buffers remaining... 502 502 502 502 502 502 502 502 502 502
502 502 502 502 502 502 502 502 502 502
giving up on 379 buffers
Uptime: 1h36m46s
Dumping 128 MB
16 32 48 64 80 96 112
---
#0 doadump () at ../../../kern/kern_shutdown.c:240
240 dumping++;
(kgdb) where
#0 doadump () at ../../../kern/kern_shutdown.c:240
#1 0xc04a7b60 in boot (howto=256) at ../../../kern/kern_shutdown.c:372
#2 0xc04a7f48 in panic () at ../../../kern/kern_shutdown.c:550
#3 0xc05cda26 in trap_fatal (frame=0xc8da49d4, eva=0)
at ../../../i386/i386/trap.c:821
#4 0xc05cd043 in trap (frame=
{tf_fs = -925237224, tf_es = -1068302320, tf_ds = -1044381680, tf_edi
= -1042780416, tf_esi = -1042847856, tf_ebp = -925218244, tf_isp
= -925218304, f_ebx = -1057406672, tf_edx = 2, tf_ecx = 0, tf_eax = 1,
tf_trapno = 12, tf_err = 0, tf_eip = -1068904062, tf_cs = 8, tf_eflags =
65606, tf_esp = -043482368, f_ss = -1043482368}) at
../../../i386/i386/trap.c:250
#5 0xc05be6f8 in calltrap () at {standard input}:94
#6 0xc05339cb in ip_forward_cacheinval () at ../../../netinet/ip_input.c:343
#7 0xc052bcee in in_addroute (v_arg=0xc1d86f00, n_arg=0x0, head=0xc1c48600,
treenodes=0xc1cdb900) at ../../../netinet/in_rmx.c:142
#8 0xc05238ea in rtrequest1 (req=11, info=0xc8da4b00, ret_nrt=0xc8da4b5c)
at ../../../net/route.c:773
#9 0xc052302a in rtrequest (req=0, dst=0x0, gateway=0x0, netmask=0x0,
flags=0, ret_nrt=0x0) at ../../../net/route.c:486
#10 0xc05227bd in rtalloc1 (dst=0xc8da4bc4, report=1, ignflags=0)
at ../../../net/route.c:148
#11 0xc0527de5 in arplookup (addr=0, create=1, proxy=0)
at ../../../netinet/if_ether.c:928
#12 0xc052749e in in_arpinput (m=0xc0fa5a00) at
../../../netinet/if_ether.c:664
---Type <return> to continue, or q <return> to quit---
#13 0xc05271d6 in arpintr (m=0xc0fa5a00) at ../../../netinet/if_ether.c:543
#14 0xc0520359 in netisr_processqueue (ni=0xc0640230)
at ../../../net/netisr.c:140
#15 0xc0520848 in swi_net (dummy=0x0) at ../../../net/netisr.c:246
#16 0xc0491948 in ithread_loop (arg=0xc0f90480)
at ../../../kern/kern_intr.c:544
#17 0xc04905c0 in fork_exit (callout=0xc0491770 <ithread_loop>, arg=0x0,
frame=0x0) at ../../../kern/kern_fork.c:793
(kgdb) up 6
#6 0xc05339cb in ip_forward_cacheinval () at ../../../netinet/ip_input.c:343
343 RTFREE(rt);
(kgdb) p rt
$1 = (struct rtentry *) 0xc0f94130
(kgdb) p *rt
$2 = {rt_nodes = {{rn_mklist = 0xc1bcc790, rn_parent = 0xc0f97180, rn_bit
=0,
rn_bmask = 0 '\0', rn_flags = 0 '\0', rn_u = {rn_leaf = {
rn_Key = 0xc1bcc7a0 "0AùÀ8AùÀ", rn_Mask = 0x0,
rn_Dupedkey = 0xc0f9719c}, rn_node = {rn_Off = -1044592736,
rn_L = 0x0, rn_R = 0xc0f9719c}}}, {rn_mklist = 0x0, rn_parent = 0x0,
rn_bit = 0, rn_bmask = 0 '\0', rn_flags = 0 '\0', rn_u = {rn_leaf = {
rn_Key = 0xc063c0a4 "", rn_Mask = 0x0, rn_Dupedkey = 0x0},
rn_node = {rn_Off = -1067204444, rn_L = 0x0, rn_R = 0x0}}}},
rt_gateway = 0x0, rt_refcnt = 0, rt_flags = 65538, rt_ifp = 0x0,
rt_ifa = 0x0, rt_genmask = 0xc0f95880,
rt_llinfo = 0xc0f95880 "\220ǼÁ\200qùÀ", rt_rmx = {rmx_locks = 0,
rmx_mtu = 0, rmx_hopcount = 0, rmx_expire = 0, rmx_recvpipe = 0,
rmx_sendpipe = 3237545088, rmx_ssthresh = 3227453828, rmx_rtt = 0,
rmx_rttvar = 0, rmx_pksent = 0, rmx_filler = {0, 3237514880, 0, 0}},
rt_gwroute = 0x0, rt_output = 0, rt_parent = 0x0, rt_mtx = {mtx_object = {
lo_class = 0x0, lo_name = 0x0, lo_type = 0x0, lo_flags = 0, lo_list = {
tqe_next = 0x0, tqe_prev = 0x0}, lo_witness = 0x0}, mtx_lock = 0,
mtx_recurse = 0, mtx_blocked = {tqh_first = 0x0, tqh_last = 0x0},
mtx_contested = {le_next = 0x0, le_prev = 0x0}}}
(kgdb) q

firewall# exit

Jaco van Tonder
Magic Developer :: Premsoft Development (Pty) Ltd
Direct: +27 11 312 2122 :: Fax: +27 11 312 2122 :: Mobile: +27 83 417 5424
Email: jaco@premsoft.co.za :: Web: http://www.premsoft.co.za/
PGP: http://www.premsoft.co.za/jaco/jaco_premsoft.asc