Date: Sat, 13 Jul 2002 00:01:25 +0400 (MSD) From: Dmitry Morozovsky <marck@rinet.ru> To: Mike Tancsa <mike@sentex.net>, <luigi@freebsd.org> Cc: noc@rinet.ru, <stable@freebsd.org> Subject: Re: 4.6-STABLE panics wirh MROUTING Message-ID: <20020712235613.G75573-100000@woozle.rinet.ru> In-Reply-To: <20020712203045.O75573-100000@woozle.rinet.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, 12 Jul 2002, Dmitry Morozovsky wrote:
DM> On Thu, 11 Jul 2002, Mike Tancsa wrote:
DM>
DM> MT> Can you build a debug kernel and then capture a crash dump when the machine
DM> MT> panics ? Then load the files in gdb and show where it crashed. Post that
DM> MT> to the list and I am sure someone might see something at issue.
DM>
DM> Just did it. Here is the offending data (panic reproduced exactly by
DM> just running mrouted)
Well, some hours of digging, rebuilding kernels and stohastic reboots
"under my fingers"...
now i'm almost certainly sources this error as a result of Luigi's commit
at 09.07.2002. All crashdump tracebacks have a step thru dummynet, and
test machine with the same kernel, but without active ipfw pipe rules
does NOT crash.
Now i'm rebuilding kernel from RELENG_4_6. Will keep you informed.
DM>
DM> Fatal trap 12: page fault while in kernel mode
DM> fault virtual address = 0x40
DM> fault code = supervisor read, page not present
DM> instruction pointer = 0x8:0xc019508e
DM> stack pointer = 0x10:0xc9fcab48
DM> frame pointer = 0x10:0xc9fcabe8
DM> code segment = base 0x0, limit 0xfffff, type 0x1b
DM> = DPL 0, pres 1, def32 1, gran 1
DM> processor eflags = interrupt enabled, resume, IOPL = 0
DM> current process = 418 (mrouted)
DM> interrupt mask = net
DM> trap number = 12
DM> panic: page fault
DM>
DM> syncing disks... 11 2 1 1
DM> done
DM> Uptime: 8m18s
DM>
DM> dumping to dev #ad/0x20001, offset 262272
DM> dump ata0: resetting devices .. ata0: mask=03 ostat0=50 ostat2=00
DM> ad0: ATAPI 00 00
DM> ata0-slave: ATAPI 00 00
DM> ata0: mask=03 stat0=50 stat1=00
DM> ad0: ATA 01 a5
DM> ata0: devices=01
DM> ad0: success setting PIO4 on generic chip
DM> done
DM> 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39
DM> 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14
DM> 13 12 11 10 9 8 7 6 5 4 3 2 1 0
DM> ---
DM> #0 dumpsys () at /lh/src/sys/kern/kern_shutdown.c:487
DM> 487 if (dumping++) {
DM> (kgdb) bt
DM> #0 dumpsys () at /lh/src/sys/kern/kern_shutdown.c:487
DM> #1 0xc01445ba in boot (howto=256) at /lh/src/sys/kern/kern_shutdown.c:316
DM> #2 0xc01449f0 in poweroff_wait (junk=0xc0212f4c, howto=-1071568273)
DM> at /lh/src/sys/kern/kern_shutdown.c:595
DM> #3 0xc01e9fdf in trap_fatal (frame=0xc9fcab08, eva=64)
DM> at /lh/src/sys/i386/i386/trap.c:974
DM> #4 0xc01e9ca1 in trap_pfault (frame=0xc9fcab08, usermode=0, eva=64)
DM> at /lh/src/sys/i386/i386/trap.c:867
DM> #5 0xc01e987f in trap (frame={tf_fs = -1072168944, tf_es = 16, tf_ds =
DM> 16,
DM> tf_edi = 0, tf_esi = -1066864128, tf_ebp = -906187800,
DM> tf_isp = -906187980, tf_ebx = 0, tf_edx = -1066863904,
DM> tf_ecx = -1015757268, tf_eax = 0, tf_trapno = 12, tf_err = 0,
DM> tf_eip = -1072082802, tf_cs = 8, tf_eflags = 66054,
DM> tf_esp = -1015757312, tf_ss = 0}) at
DM> /lh/src/sys/i386/i386/trap.c:466
DM> #6 0xc019508e in ip_output (m0=0xc068f200, opt=0x0, ro=0xc374c62c,
DM> flags=0,
DM> imo=0x0) at /lh/src/sys/netinet/ip_output.c:189
DM> #7 0xc018ba0e in transmit_event (pipe=0xc3651f00)
DM> at /lh/src/sys/netinet/ip_dummynet.c:425
DM> #8 0xc018bcc3 in ready_event (q=0xc36ef680)
DM> at /lh/src/sys/netinet/ip_dummynet.c:577
DM> #9 0xc018cab4 in dummynet_io (m=0xc068f200, pipe_nr=1000, dir=1,
DM> fwa=0xc9fcad00) at /lh/src/sys/netinet/ip_dummynet.c:1175
DM> #10 0xc01958b8 in ip_output (m0=0xc068f200, opt=0xc0602f00, ro=0xc0234598,
DM> ---Type <return> to continue, or q <return> to quit---
DM> flags=0, imo=0xc9fcad60) at /lh/src/sys/netinet/ip_output.c:851
DM> #11 0xc018892c in igmp_sendpkt (inm=0xc36f1840, type=22, addr=0)
DM> at /lh/src/sys/netinet/igmp.c:482
DM> #12 0xc0188695 in igmp_joingroup (inm=0xc36f1840)
DM> at /lh/src/sys/netinet/igmp.c:354
DM> #13 0xc0189b07 in in_addmulti (ap=0xc9fcae44, ifp=0xc35c4e00)
DM> at /lh/src/sys/netinet/in.c:863
DM> #14 0xc019692a in ip_setmoptions (sopt=0xc9fcaf10, imop=0xc99acf3c)
DM> at /lh/src/sys/netinet/ip_output.c:1744
DM> #15 0xc0196239 in ip_ctloutput (so=0xc88d5b80, sopt=0xc9fcaf10)
DM> at /lh/src/sys/netinet/ip_output.c:1233
DM> #16 0xc0197277 in rip_ctloutput (so=0xc88d5b80, sopt=0xc9fcaf10)
DM> at /lh/src/sys/netinet/raw_ip.c:382
DM> #17 0xc0164385 in sosetopt (so=0xc88d5b80, sopt=0xc9fcaf10)
DM> at /lh/src/sys/kern/uipc_socket.c:1125
DM> #18 0xc0167613 in setsockopt (p=0xc84a4560, uap=0xc9fcaf80)
DM> at /lh/src/sys/kern/uipc_syscalls.c:1113
DM> #19 0xc01ea252 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47,
DM> tf_edi = 1052784323, tf_esi = 67109088, tf_ebp = -1077939076,
DM> tf_isp = -906186796, tf_ebx = 1052784323, tf_edx = 0, tf_ecx = 6,
DM> tf_eax = 105, tf_trapno = 12, tf_err = 2, tf_eip = 671830616,
DM> tf_cs = 31, tf_eflags = 643, tf_esp = -1077939136, tf_ss = 47})
DM> at /lh/src/sys/i386/i386/trap.c:1175
DM> ---Type <return> to continue, or q <return> to quit---
DM> #20 0xc01de445 in Xint0x80_syscall ()
DM> #21 0x804e978 in ?? ()
DM> #22 0x804e3c5 in ?? ()
DM> #23 0x804b129 in ?? ()
DM> #24 0x8048fc3 in ?? ()
DM>
DM>
DM>
DM> #6 0xc019508e in ip_output (m0=0xc068f200, opt=0x0, ro=0xc374c62c,
DM> flags=0,
DM> imo=0x0) at /lh/src/sys/netinet/ip_output.c:189
DM> 189 ia = ifatoia(ro->ro_rt->rt_ifa);
DM> (kgdb) l
DM> 184 (void)ipsec_setsocket(m, NULL);
DM> 185 #endif
DM> 186 if (args.rule != NULL) { /* dummynet already saw us */
DM> 187 ip = mtod(m, struct ip *);
DM> 188 hlen = IP_VHL_HL(ip->ip_vhl) << 2 ;
DM> 189 ia = ifatoia(ro->ro_rt->rt_ifa);
DM> 190 goto sendit;
DM> 191 }
DM> 192
DM> 193 if (opt) {
DM>
DM> (kgdb) p *ro
DM> $2 = {ro_rt = 0x0, ro_dst = {sa_len = 16 '\020', sa_family = 2 '\002',
DM> sa_data = "\000\000à\000\000\004\000\000\000\000\000\000\000"}}
DM>
DM> so, ro->rt is NULL, and it's producing panic.
DM>
DM> MT>
DM> MT> ---Mike
DM> MT>
DM> MT> At 10:55 PM 11/07/2002 +0400, Dmitry Morozovsky wrote:
DM> MT> >Hello there colleagues.
DM> MT> >
DM> MT> >After upgrading one of our gateways I'd encountered reproducible
DM> MT> >kernel panics in multicast-related situations. processes involved are
DM> MT> >mrouted (from base system) and ospfd from zebra-0.92a_1
DM> MT> >
DM> MT> >FreeBSD gw-f.rinet.ru 4.6-STABLE FreeBSD 4.6-STABLE #2: Wed Jul 10
DM> MT> >15:05:26 MSD 2002 root@gw-f.rinet.ru:/var/obj/lh/src/sys/gwfn i386
DM> MT> >
DM> MT> >I suspect options MROUTING being "The Evil" [tm], but had no time to check
DM> MT> >it thoroughly (it's production gateway, you know, so now I'd simply stick
DM> MT> >it out with about 40 statics both there and on border Cisco ;-)
DM> MT> >
DM> MT> >kernel config file follows (it's 4-ethernet router + COM multiport console
DM> MT> >server)
DM> MT> >
DM> MT> >Any thoughts?
DM> MT> >
DM> MT> >Sincerely,
DM> MT> >D.Marck [DM5020, DM268-RIPE, DM3-RIPN]
DM> MT> >------------------------------------------------------------------------
DM> MT> >*** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru ***
DM> MT> >------------------------------------------------------------------------
DM> MT> >
DM> MT> >
DM> MT> >
DM> MT> >machine i386
DM> MT> >cpu I686_CPU
DM> MT> >ident GWF
DM> MT> >maxusers 0
DM> MT> >
DM> MT> >#makeoptions DEBUG=-g #Build kernel with gdb(1) debug
DM> MT> >symbols
DM> MT> >
DM> MT> >options INET #InterNETworking
DM> MT> >options FFS #Berkeley Fast Filesystem
DM> MT> >options FFS_ROOT #FFS usable as root device [keep
DM> MT> >this!]
DM> MT> >options SOFTUPDATES #Enable FFS soft updates support
DM> MT> >options NFS_NOSERVER #Completely disable server code
DM> MT> >options PROCFS #Process filesystem
DM> MT> >options COMPAT_43 #Compatible with BSD 4.3 [KEEP
DM> MT> >THIS!]
DM> MT> >options KTRACE #ktrace(1) support
DM> MT> >options SYSVSHM #SYSV-style shared memory
DM> MT> >options SYSVMSG #SYSV-style message queues
DM> MT> >options SYSVSEM #SYSV-style semaphores
DM> MT> >options P1003_1B #Posix P1003_1B real-time
DM> MT> >extensions
DM> MT> >options _KPOSIX_PRIORITY_SCHEDULING
DM> MT> >options ICMP_BANDLIM #Rate limit bad replies
DM> MT> >options KBD_INSTALL_CDEV # install a CDEV entry in /dev
DM> MT> >options NO_F00F_HACK # we are not on buggy Pentium
DM> MT> >
DM> MT> >options CLK_USE_I8254_CALIBRATION
DM> MT> >options CLK_USE_TSC_CALIBRATION
DM> MT> >
DM> MT> >options MROUTING # Multicast routing
DM> MT> >options IPFIREWALL #firewall
DM> MT> >options IPFIREWALL_VERBOSE #print information about
DM> MT> > # dropped packets
DM> MT> >#options IPFIREWALL_FORWARD #enable transparent proxy support
DM> MT> >#options IPFIREWALL_VERBOSE_LIMIT=100 #limit verbosity
DM> MT> >options DUMMYNET #dummynet shaper
DM> MT> >options HZ=1000
DM> MT> >
DM> MT> >options NMBCLUSTERS=16384
DM> MT> >
DM> MT> >device isa
DM> MT> >device pci
DM> MT> >
DM> MT> ># Floppy drives
DM> MT> >device fdc0 at isa? port IO_FD1 irq 6 drq 2
DM> MT> >device fd0 at fdc0 drive 0
DM> MT> >
DM> MT> ># ATA and ATAPI devices
DM> MT> >device ata
DM> MT> >device atadisk # ATA disk drives
DM> MT> >options ATA_STATIC_ID #Static device numbering
DM> MT> >
DM> MT> ># atkbdc0 controls both the keyboard and the PS/2 mouse
DM> MT> >device atkbdc0 at isa? port IO_KBD
DM> MT> >device atkbd0 at atkbdc? irq 1 flags 0x1
DM> MT> >
DM> MT> >device vga0 at isa?
DM> MT> >
DM> MT> ># splash screen/screen saver
DM> MT> >#pseudo-device splash
DM> MT> >
DM> MT> ># syscons is the default console driver, resembling an SCO console
DM> MT> >device sc0 at isa? flags 0x100
DM> MT> >
DM> MT> ># Floating point support - do not disable.
DM> MT> >device npx0 at nexus? port IO_NPX irq 13
DM> MT> >
DM> MT> ># Serial (COM) ports
DM> MT> >device sio0 at isa? port IO_COM1 flags 0x10 irq 4
DM> MT> >device sio1 at isa? port IO_COM2 irq 3
DM> MT> >
DM> MT> >options COM_MULTIPORT #code for some cards with shared
DM> MT> >IRQs
DM> MT> >
DM> MT> ># 4-port AST
DM> MT> >device sio2 at isa? port 0x2a0 flags 0x501
DM> MT> >device sio3 at isa? port 0x2a8 flags 0x501
DM> MT> >device sio4 at isa? port 0x2b0 flags 0x501
DM> MT> >device sio5 at isa? port 0x2b8 flags 0x501 irq 7
DM> MT> >
DM> MT> ># 8-port Omega
DM> MT> >device sio6 at isa? port 0x100 flags 0x605 irq 12
DM> MT> >device sio7 at isa? port 0x108 flags 0x605
DM> MT> >device sio8 at isa? port 0x110 flags 0x605
DM> MT> >device sio9 at isa? port 0x118 flags 0x605
DM> MT> >device sio10 at isa? port 0x120 flags 0x605
DM> MT> >device sio11 at isa? port 0x128 flags 0x605
DM> MT> >device sio12 at isa? port 0x130 flags 0x605
DM> MT> >device sio13 at isa? port 0x138 flags 0x605
DM> MT> >
DM> MT> ># PCI Ethernet NICs.
DM> MT> >#device ed # NE1000/2000 and compatibles
DM> MT> >#device ed0 at isa? port 0x300 irq 10 iomem 0xd8000
DM> MT> >
DM> MT> >device miibus
DM> MT> >device rl # NE1000/2000 and compatibles
DM> MT> >device fxp # Intel EtherExpress PRO/100B (82557,
DM> MT> >82558)
DM> MT> >
DM> MT> >
DM> MT> ># Pseudo devices - the number indicates how many units to allocated.
DM> MT> >pseudo-device loop # Network loopback
DM> MT> >pseudo-device ether # Ethernet support
DM> MT> >pseudo-device pty # Pseudo-ttys (telnet etc)
DM> MT> >pseudo-device bpf #Berkeley packet filter
DM> MT> >
DM> MT> >
DM> MT> >
DM> MT> >
DM> MT> >To Unsubscribe: send mail to majordomo@FreeBSD.org
DM> MT> >with "unsubscribe freebsd-stable" in the body of the message
DM> MT>
DM>
DM> Sincerely,
DM> D.Marck [DM5020, DM268-RIPE, DM3-RIPN]
DM> ------------------------------------------------------------------------
DM> *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru ***
DM> ------------------------------------------------------------------------
DM>
Sincerely,
D.Marck [DM5020, DM268-RIPE, DM3-RIPN]
------------------------------------------------------------------------
*** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru ***
------------------------------------------------------------------------
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20020712235613.G75573-100000>