Date: Fri, 12 Jul 2002 20:41:24 +0400 (MSD) From: Dmitry Morozovsky <marck@rinet.ru> To: Mike Tancsa <mike@sentex.net> Cc: noc@rinet.ru, <stable@freebsd.org> Subject: Re: 4.6-STABLE panics wirh MROUTING Message-ID: <20020712203045.O75573-100000@woozle.rinet.ru> In-Reply-To: <5.1.0.14.0.20020711154823.0379dd28@marble.sentex.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 11 Jul 2002, Mike Tancsa wrote:
MT> Can you build a debug kernel and then capture a crash dump when the machine
MT> panics ? Then load the files in gdb and show where it crashed. Post that
MT> to the list and I am sure someone might see something at issue.
Just did it. Here is the offending data (panic reproduced exactly by
just running mrouted)
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x40
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc019508e
stack pointer = 0x10:0xc9fcab48
frame pointer = 0x10:0xc9fcabe8
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 418 (mrouted)
interrupt mask = net
trap number = 12
panic: page fault
syncing disks... 11 2 1 1
done
Uptime: 8m18s
dumping to dev #ad/0x20001, offset 262272
dump ata0: resetting devices .. ata0: mask=03 ostat0=50 ostat2=00
ad0: ATAPI 00 00
ata0-slave: ATAPI 00 00
ata0: mask=03 stat0=50 stat1=00
ad0: ATA 01 a5
ata0: devices=01
ad0: success setting PIO4 on generic chip
done
63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39
38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14
13 12 11 10 9 8 7 6 5 4 3 2 1 0
---
#0 dumpsys () at /lh/src/sys/kern/kern_shutdown.c:487
487 if (dumping++) {
(kgdb) bt
#0 dumpsys () at /lh/src/sys/kern/kern_shutdown.c:487
#1 0xc01445ba in boot (howto=256) at /lh/src/sys/kern/kern_shutdown.c:316
#2 0xc01449f0 in poweroff_wait (junk=0xc0212f4c, howto=-1071568273)
at /lh/src/sys/kern/kern_shutdown.c:595
#3 0xc01e9fdf in trap_fatal (frame=0xc9fcab08, eva=64)
at /lh/src/sys/i386/i386/trap.c:974
#4 0xc01e9ca1 in trap_pfault (frame=0xc9fcab08, usermode=0, eva=64)
at /lh/src/sys/i386/i386/trap.c:867
#5 0xc01e987f in trap (frame={tf_fs = -1072168944, tf_es = 16, tf_ds =
16,
tf_edi = 0, tf_esi = -1066864128, tf_ebp = -906187800,
tf_isp = -906187980, tf_ebx = 0, tf_edx = -1066863904,
tf_ecx = -1015757268, tf_eax = 0, tf_trapno = 12, tf_err = 0,
tf_eip = -1072082802, tf_cs = 8, tf_eflags = 66054,
tf_esp = -1015757312, tf_ss = 0}) at
/lh/src/sys/i386/i386/trap.c:466
#6 0xc019508e in ip_output (m0=0xc068f200, opt=0x0, ro=0xc374c62c,
flags=0,
imo=0x0) at /lh/src/sys/netinet/ip_output.c:189
#7 0xc018ba0e in transmit_event (pipe=0xc3651f00)
at /lh/src/sys/netinet/ip_dummynet.c:425
#8 0xc018bcc3 in ready_event (q=0xc36ef680)
at /lh/src/sys/netinet/ip_dummynet.c:577
#9 0xc018cab4 in dummynet_io (m=0xc068f200, pipe_nr=1000, dir=1,
fwa=0xc9fcad00) at /lh/src/sys/netinet/ip_dummynet.c:1175
#10 0xc01958b8 in ip_output (m0=0xc068f200, opt=0xc0602f00, ro=0xc0234598,
---Type <return> to continue, or q <return> to quit---
flags=0, imo=0xc9fcad60) at /lh/src/sys/netinet/ip_output.c:851
#11 0xc018892c in igmp_sendpkt (inm=0xc36f1840, type=22, addr=0)
at /lh/src/sys/netinet/igmp.c:482
#12 0xc0188695 in igmp_joingroup (inm=0xc36f1840)
at /lh/src/sys/netinet/igmp.c:354
#13 0xc0189b07 in in_addmulti (ap=0xc9fcae44, ifp=0xc35c4e00)
at /lh/src/sys/netinet/in.c:863
#14 0xc019692a in ip_setmoptions (sopt=0xc9fcaf10, imop=0xc99acf3c)
at /lh/src/sys/netinet/ip_output.c:1744
#15 0xc0196239 in ip_ctloutput (so=0xc88d5b80, sopt=0xc9fcaf10)
at /lh/src/sys/netinet/ip_output.c:1233
#16 0xc0197277 in rip_ctloutput (so=0xc88d5b80, sopt=0xc9fcaf10)
at /lh/src/sys/netinet/raw_ip.c:382
#17 0xc0164385 in sosetopt (so=0xc88d5b80, sopt=0xc9fcaf10)
at /lh/src/sys/kern/uipc_socket.c:1125
#18 0xc0167613 in setsockopt (p=0xc84a4560, uap=0xc9fcaf80)
at /lh/src/sys/kern/uipc_syscalls.c:1113
#19 0xc01ea252 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47,
tf_edi = 1052784323, tf_esi = 67109088, tf_ebp = -1077939076,
tf_isp = -906186796, tf_ebx = 1052784323, tf_edx = 0, tf_ecx = 6,
tf_eax = 105, tf_trapno = 12, tf_err = 2, tf_eip = 671830616,
tf_cs = 31, tf_eflags = 643, tf_esp = -1077939136, tf_ss = 47})
at /lh/src/sys/i386/i386/trap.c:1175
---Type <return> to continue, or q <return> to quit---
#20 0xc01de445 in Xint0x80_syscall ()
#21 0x804e978 in ?? ()
#22 0x804e3c5 in ?? ()
#23 0x804b129 in ?? ()
#24 0x8048fc3 in ?? ()
#6 0xc019508e in ip_output (m0=0xc068f200, opt=0x0, ro=0xc374c62c,
flags=0,
imo=0x0) at /lh/src/sys/netinet/ip_output.c:189
189 ia = ifatoia(ro->ro_rt->rt_ifa);
(kgdb) l
184 (void)ipsec_setsocket(m, NULL);
185 #endif
186 if (args.rule != NULL) { /* dummynet already saw us */
187 ip = mtod(m, struct ip *);
188 hlen = IP_VHL_HL(ip->ip_vhl) << 2 ;
189 ia = ifatoia(ro->ro_rt->rt_ifa);
190 goto sendit;
191 }
192
193 if (opt) {
(kgdb) p *ro
$2 = {ro_rt = 0x0, ro_dst = {sa_len = 16 '\020', sa_family = 2 '\002',
sa_data = "\000\000à\000\000\004\000\000\000\000\000\000\000"}}
so, ro->rt is NULL, and it's producing panic.
MT>
MT> ---Mike
MT>
MT> At 10:55 PM 11/07/2002 +0400, Dmitry Morozovsky wrote:
MT> >Hello there colleagues.
MT> >
MT> >After upgrading one of our gateways I'd encountered reproducible
MT> >kernel panics in multicast-related situations. processes involved are
MT> >mrouted (from base system) and ospfd from zebra-0.92a_1
MT> >
MT> >FreeBSD gw-f.rinet.ru 4.6-STABLE FreeBSD 4.6-STABLE #2: Wed Jul 10
MT> >15:05:26 MSD 2002 root@gw-f.rinet.ru:/var/obj/lh/src/sys/gwfn i386
MT> >
MT> >I suspect options MROUTING being "The Evil" [tm], but had no time to check
MT> >it thoroughly (it's production gateway, you know, so now I'd simply stick
MT> >it out with about 40 statics both there and on border Cisco ;-)
MT> >
MT> >kernel config file follows (it's 4-ethernet router + COM multiport console
MT> >server)
MT> >
MT> >Any thoughts?
MT> >
MT> >Sincerely,
MT> >D.Marck [DM5020, DM268-RIPE, DM3-RIPN]
MT> >------------------------------------------------------------------------
MT> >*** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru ***
MT> >------------------------------------------------------------------------
MT> >
MT> >
MT> >
MT> >machine i386
MT> >cpu I686_CPU
MT> >ident GWF
MT> >maxusers 0
MT> >
MT> >#makeoptions DEBUG=-g #Build kernel with gdb(1) debug
MT> >symbols
MT> >
MT> >options INET #InterNETworking
MT> >options FFS #Berkeley Fast Filesystem
MT> >options FFS_ROOT #FFS usable as root device [keep
MT> >this!]
MT> >options SOFTUPDATES #Enable FFS soft updates support
MT> >options NFS_NOSERVER #Completely disable server code
MT> >options PROCFS #Process filesystem
MT> >options COMPAT_43 #Compatible with BSD 4.3 [KEEP
MT> >THIS!]
MT> >options KTRACE #ktrace(1) support
MT> >options SYSVSHM #SYSV-style shared memory
MT> >options SYSVMSG #SYSV-style message queues
MT> >options SYSVSEM #SYSV-style semaphores
MT> >options P1003_1B #Posix P1003_1B real-time
MT> >extensions
MT> >options _KPOSIX_PRIORITY_SCHEDULING
MT> >options ICMP_BANDLIM #Rate limit bad replies
MT> >options KBD_INSTALL_CDEV # install a CDEV entry in /dev
MT> >options NO_F00F_HACK # we are not on buggy Pentium
MT> >
MT> >options CLK_USE_I8254_CALIBRATION
MT> >options CLK_USE_TSC_CALIBRATION
MT> >
MT> >options MROUTING # Multicast routing
MT> >options IPFIREWALL #firewall
MT> >options IPFIREWALL_VERBOSE #print information about
MT> > # dropped packets
MT> >#options IPFIREWALL_FORWARD #enable transparent proxy support
MT> >#options IPFIREWALL_VERBOSE_LIMIT=100 #limit verbosity
MT> >options DUMMYNET #dummynet shaper
MT> >options HZ=1000
MT> >
MT> >options NMBCLUSTERS=16384
MT> >
MT> >device isa
MT> >device pci
MT> >
MT> ># Floppy drives
MT> >device fdc0 at isa? port IO_FD1 irq 6 drq 2
MT> >device fd0 at fdc0 drive 0
MT> >
MT> ># ATA and ATAPI devices
MT> >device ata
MT> >device atadisk # ATA disk drives
MT> >options ATA_STATIC_ID #Static device numbering
MT> >
MT> ># atkbdc0 controls both the keyboard and the PS/2 mouse
MT> >device atkbdc0 at isa? port IO_KBD
MT> >device atkbd0 at atkbdc? irq 1 flags 0x1
MT> >
MT> >device vga0 at isa?
MT> >
MT> ># splash screen/screen saver
MT> >#pseudo-device splash
MT> >
MT> ># syscons is the default console driver, resembling an SCO console
MT> >device sc0 at isa? flags 0x100
MT> >
MT> ># Floating point support - do not disable.
MT> >device npx0 at nexus? port IO_NPX irq 13
MT> >
MT> ># Serial (COM) ports
MT> >device sio0 at isa? port IO_COM1 flags 0x10 irq 4
MT> >device sio1 at isa? port IO_COM2 irq 3
MT> >
MT> >options COM_MULTIPORT #code for some cards with shared
MT> >IRQs
MT> >
MT> ># 4-port AST
MT> >device sio2 at isa? port 0x2a0 flags 0x501
MT> >device sio3 at isa? port 0x2a8 flags 0x501
MT> >device sio4 at isa? port 0x2b0 flags 0x501
MT> >device sio5 at isa? port 0x2b8 flags 0x501 irq 7
MT> >
MT> ># 8-port Omega
MT> >device sio6 at isa? port 0x100 flags 0x605 irq 12
MT> >device sio7 at isa? port 0x108 flags 0x605
MT> >device sio8 at isa? port 0x110 flags 0x605
MT> >device sio9 at isa? port 0x118 flags 0x605
MT> >device sio10 at isa? port 0x120 flags 0x605
MT> >device sio11 at isa? port 0x128 flags 0x605
MT> >device sio12 at isa? port 0x130 flags 0x605
MT> >device sio13 at isa? port 0x138 flags 0x605
MT> >
MT> ># PCI Ethernet NICs.
MT> >#device ed # NE1000/2000 and compatibles
MT> >#device ed0 at isa? port 0x300 irq 10 iomem 0xd8000
MT> >
MT> >device miibus
MT> >device rl # NE1000/2000 and compatibles
MT> >device fxp # Intel EtherExpress PRO/100B (82557,
MT> >82558)
MT> >
MT> >
MT> ># Pseudo devices - the number indicates how many units to allocated.
MT> >pseudo-device loop # Network loopback
MT> >pseudo-device ether # Ethernet support
MT> >pseudo-device pty # Pseudo-ttys (telnet etc)
MT> >pseudo-device bpf #Berkeley packet filter
MT> >
MT> >
MT> >
MT> >
MT> >To Unsubscribe: send mail to majordomo@FreeBSD.org
MT> >with "unsubscribe freebsd-stable" in the body of the message
MT>
Sincerely,
D.Marck [DM5020, DM268-RIPE, DM3-RIPN]
------------------------------------------------------------------------
*** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru ***
------------------------------------------------------------------------
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20020712203045.O75573-100000>