From owner-freebsd-questions Thu Jan 13 4:35:47 2000 Delivered-To: freebsd-questions@freebsd.org Received: from gecko.eric.net.au (gecko.eric.net.au [203.36.118.3]) by hub.freebsd.org (Postfix) with ESMTP id 923C414DD2 for ; Thu, 13 Jan 2000 04:35:29 -0800 (PST) (envelope-from tfrank@eric.net.au) Received: from random.n2-au (ppp150.eric.net.au [203.36.118.150]) by gecko.eric.net.au (8.9.3/8.8.7) with ESMTP id WAA03630 for ; Thu, 13 Jan 2000 22:29:35 +1100 Received: (from tony@localhost) by random.n2-au (8.9.3/8.9.3) id XAA01059 for freebsd-questions@freebsd.org; Thu, 13 Jan 2000 23:35:22 +1100 (EST) (envelope-from tony) From: Tony Frank Date: Thu, 13 Jan 2000 23:35:22 +1100 To: freebsd-questions@freebsd.org Subject: 3.4-STABLE crashes intermittently - need suggestions Message-ID: <20000113233522.A1035@random.n2-au> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 0.95.4i Sender: owner-freebsd-questions@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Hi, I have a system running at 3.4-STABLE that seems to occasionally freeze or hang for no obvious reason. What I'm after are some pointers on what to do to try and isolate the problem, hopefully with the end result being a system that stays up for more than about 3 days at a time. I will describe the symptoms, my configuration, and also include my attempts at a post mortem below... The problem: Most of the time the system just appears to freeze, with no error message on the console - at this time the IDE HD LED is usually on solid (however no audiable disk activity) the keyboard offers no response, the system no longer responds to any form of network probing (arp, ping etc) and requires physical intervention in the form of a power on/off (no reset button) This seems to occur most often when the system is under high load, but also appears to occur when system is mostly idle. I can do a make world (and also make -j4 world) with no problems, which indicates to me that the hardware should be pretty much ok. I have not swapped the various components around or tried parts from other systems in there, but I am reasonably confident in the hardware - certainly the harddisk and ethernet cards were working in my other PC for 6months+ with no problems. The most recent time it occured, I had some moderate NFS traffic (cvs update across 10mb ethernet, frozen system as NFS server) and at the same time some fairly heavy local disk activity (local cvs update from local repository) and also routing light http traffic to/from ISP (multilink userppp - 2x56k modems at about 8k/s sustained traffic) After the reboot, there was no coredump generated. I have the following output of an attempt to debug a panic from several days ago(my first time attempting to use gdb so I didn't get very far) I basically followed the example in the faq/handbook but didn't know what to make of the output... kernel post mortem output: 17:08:18 tony@random (/usr/src/sys/compile/RAND)$ gdb -kernel kernel.debug /var /crash/vmcore.5 17:10:19 tony@random (/usr/src/sys/compile/RAND)$ gdb -kernel GNU gdb 4.18 Copyright 1998 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-unknown-freebsd". (kgdb) symbol-file kernel.debug Reading symbols from kernel.debug...done. (kgdb) exec-file /var/crash/kernel.5 (kgdb) core-file /var/crash/vmcore.5 IdlePTD 2670592 initial pcb at 222250 panicstr: page fault panic messages: --- Fatal trap 12: page fault while in kernel mode fault virtual address = 0x0 fault code = supervisor read, page not present instruction pointer = 0x8:0x0 stack pointer = 0x10:0xc33b2acc frame pointer = 0x10:0xc33b2ad4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 167 (nfsd) interrupt mask = trap number = 12 panic: page fault syncing disks... Fatal trap 12: page fault while in kernel mode fault virtual address = 0x0 fault code = supervisor read, page not present instruction pointer = 0x8:0x0 stack pointer = 0x10:0xc33b28d0 frame pointer = 0x10:0xc33b28d8 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 167 (nfsd) interrupt mask = bio trap number = 12 panic: page fault dumping to dev 20001, offset 88064 dump 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 --- #0 boot (howto=260) at ../../kern/kern_shutdown.c:285 285 dumppcb.pcb_cr3 = rcr3(); (kgdb) where #0 boot (howto=260) at ../../kern/kern_shutdown.c:285 #1 0xc012c1b8 in at_shutdown ( function=0xc020a6de <__set_sysinit_set_sym_memdev_sys_init+1050>, arg=0xc3376c80, queue=-1019762752) at ../../kern/kern_shutdown.c:446 #2 0xc01e6099 in trap_fatal (frame=0xc33b2894, eva=0) at ../../i386/i386/trap.c:942 #3 0xc01e5d77 in trap_pfault (frame=0xc33b2894, usermode=0, eva=0) at ../../i386/i386/trap.c:835 #4 0xc01e59ee in trap (frame={tf_es = 16, tf_ds = 16, tf_edi = -1053105680, tf_esi = -1019799296, tf_ebp = -1019533096, tf_isp = -1019533124, tf_ebx = 6144, tf_edx = -1019533040, tf_ecx = 40, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = 0, tf_cs = 8, tf_eflags = 66198, tf_esp = -1072316847, tf_ss = -1019533040}) at ../../i386/i386/trap.c:437 #5 0x0 in ?? () (kgdb) up 4 #4 0xc01e59ee in trap (frame={tf_es = 16, tf_ds = 16, tf_edi = -1053105680, tf_esi = -1019799296, tf_ebp = -1019533096, tf_isp = -1019533124, tf_ebx = 6144, tf_edx = -1019533040, tf_ecx = 40, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = 0, tf_cs = 8, tf_eflags = 66198, tf_esp = -1072316847, tf_ss = -1019533040}) at ../../i386/i386/trap.c:437 437 (void) trap_pfault(&frame, FALSE, eva); (kgdb) frame frame->tf_ebp frame->tf_eip #0 0x0 in ?? () (kgdb) list 432 #endif 433 /* kernel trap */ 434 435 switch (type) { 436 case T_PAGEFLT: /* page fault */ 437 (void) trap_pfault(&frame, FALSE, eva); 438 return; 439 440 case T_DNA: 441 #if NNPX > 0 (kgdb) up #1 0xc012bf42 in boot (howto=-1019533040) at ../../kern/kern_shutdown.c:287 287 dumpsys(); (kgdb) up #2 0xc01be691 in ufs_vnoperatespec (ap=0xc33b2910) at ../../ufs/ufs/ufs_vnops.c:2318 2318 return (VOCALL(ufs_specop_p, ap->a_desc->vdesc_offset, ap)); (kgdb) up #3 0xc014a3a3 in vfs_bio_awrite (bp=0xc13ae1f0) at vnode_if.h:1145 1145 return (VCALL((bp)->b_vp, VOFFSET(vop_bwrite), &a)); (kgdb) up #4 0xc01b85ea in ffs_fsync (ap=0xc33b2998) at ../../ufs/ffs/ffs_vnops.c:205 205 vfs_bio_awrite(bp); (kgdb) up #5 0xc01b6a93 in ffs_sync (mp=0xc0772600, waitfor=2, cred=0xc04da680, p=0xc0236f94) at vnode_if.h:499 499 return (VCALL(vp, VOFFSET(vop_fsync), &a)); (kgdb) up #6 0xc015291f in sync (p=0xc0236f94, uap=0x0) at ../../kern/vfs_syscalls.c:549 549 VFS_SYNC(mp, MNT_NOWAIT, (kgdb) up #7 0xc012bd79 in boot (howto=256) at ../../kern/kern_shutdown.c:203 203 sync(&proc0, NULL); (kgdb) up #8 0xc012c1b8 in at_shutdown ( function=0xc020a6de <__set_sysinit_set_sym_memdev_sys_init+1050>, arg=0xc3376c80, queue=-1019762752) at ../../kern/kern_shutdown.c:446 446 boot(bootopt); (kgdb) up #9 0xc01e6099 in trap_fatal (frame=0xc33b2a90, eva=0) at ../../i386/i386/trap.c:942 942 panic(trap_msg[type]); (kgdb) up #10 0xc01e5d77 in trap_pfault (frame=0xc33b2a90, usermode=0, eva=0) at ../../i386/i386/trap.c:835 835 trap_fatal(frame, eva); (kgdb) up #11 0xc01e59ee in trap (frame={tf_es = 16, tf_ds = 16, tf_edi = -1063922432, tf_esi = 0, tf_ebp = -1019532588, tf_isp = -1019532616, tf_ebx = -1053129064, tf_edx = -1019532548, tf_ecx = 34, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = 0, tf_cs = 8, tf_eflags = 66071, tf_esp = -1072316847, tf_ss = -1019532548}) at ../../i386/i386/trap.c:437 437 (void) trap_pfault(&frame, FALSE, eva); (kgdb) up #12 0x0 in ?? () (kgdb) up Initial frame selected; you cannot go up. (kgdb) list 432 #endif 433 /* kernel trap */ 434 435 switch (type) { 436 case T_PAGEFLT: /* page fault */ 437 (void) trap_pfault(&frame, FALSE, eva); 438 return; 439 440 case T_DNA: 441 #if NNPX > 0 (kgdb) quit System details: Hardware: IBM PC340 (Intel p100/32mbRAM/3G IDE HDD) Also Netgear FA310TX(pn0) and NE2000(ed0), and IBM Auto 16/4 Token Ring card (not used/probed etc) Software: 3.4-STABLE, kernel based on GENERIC with everything but pn0 and ed0 removed and BRIDGE+SOFTUPDATES added (see included file at end) Also, I patched the pn0 driver to support bridging, however the problem appears to exist whether I apply this patch or not. So far the bridging seems to work as I would expect it to, but I may be missing something here too... Dmesg output and kernel config are included below, along with the patch for pn0. *** start --- if_pn.c.original Thu Jan 13 22:41:52 2000 +++ if_pn.c Thu Jan 13 22:43:12 2000 @@ -77,6 +77,11 @@ #include #endif +#include "opt_bdg.h" +#ifdef BRIDGE +#include +#endif + #include /* for vtophys */ #include /* for vtophys */ #include /* for DELAY */ @@ -1586,6 +1591,24 @@ } } #endif + +#ifdef BRIDGE + + /* Copied from if_xl.c and placed in about the same spot */ + + if (do_bridge) { + struct ifnet *bdg_ifp; + bdg_ifp = bridge_in(m); + if (bdg_ifp != BDG_LOCAL && bdg_ifp != BDG_DROP) + bdg_forward(&m, bdg_ifp); + if (((bdg_ifp != BDG_LOCAL) && (bdg_ifp != BDG_BCAST) && + (bdg_ifp != BDG_MCAST)) || bdg_ifp == BDG_DROP) { + m_freem(m); + continue; + } + } +#endif + /* Remove header from mbuf and pass it on. */ m_adj(m, sizeof(struct ether_header)); ether_input(ifp, eh, m); *** end dmesg output: Copyright (c) 1992-1999 FreeBSD Inc. Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. FreeBSD 3.4-STABLE #0: Thu Jan 13 21:18:29 EST 2000 tony@random.n2-au:/usr1/src/sys/compile/RAND Timecounter "i8254" frequency 1193182 Hz CPU: Pentium/P54C (99.47-MHz 586-class CPU) Origin = "GenuineIntel" Id = 0x525 Stepping = 5 Features=0x1bf real memory = 33554432 (32768K bytes) avail memory = 30281728 (29572K bytes) Preloaded elf kernel "kernel" at 0xc0278000. Probing for devices on PCI bus 0: chip0: rev 0x00 on pci0.0.0 chip1: rev 0x01 on pci0.1.0 ide_pci0: rev 0x08 int a irq 0 on pci0.1.1 pn0: <82c169 PNIC 10/100BaseTX> rev 0x21 int a irq 10 on pci0.14.0 pn0: Ethernet address: 00:a0:cc:3c:d1:bf pn0: autoneg complete, link status good (half-duplex, 10Mbps) vga0: rev 0x00 on pci0.20.0 Probing for devices on the ISA bus: sc0 on isa sc0: VGA color <16 virtual consoles, flags=0x0> ed0 at 0x340-0x35f irq 5 on isa ed0: address 00:40:c7:11:c5:62, type NE2000 (16 bit) atkbdc0 at 0x60-0x6f on motherboard atkbd0 irq 1 on isa psm0 not found sio0 at 0x3f8-0x3ff irq 4 flags 0x10 on isa sio0: type 16550A sio1 at 0x2f8-0x2ff irq 3 on isa sio1: type 16550A wdc0 at 0x1f0-0x1f7 irq 14 flags 0xa0ffa0ff on isa wdc0: unit 0 (wd0): , 32-bit, multi-block-16 wd0: 3093MB (6335280 sectors), 6704 cyls, 15 heads, 63 S/T, 512 B/S ppc0 at 0x378 irq 7 flags 0x40 on isa ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/15 bytes threshold lpt0: on ppbus 0 lpt0: Interrupt-driven port ppi0: on ppbus 0 plip0: on ppbus 0 vga0 at 0x3b0-0x3df maddr 0xa0000 msize 131072 on isa npx0 on motherboard npx0: INT 16 interface apm0 flags 0x31 on isa apm: found APM BIOS version 1.2 Intel Pentium detected, installing workaround for F00F bug BRIDGE 990810, have 6 interfaces -- index 1 type 6 phy 0 addrl 6 addr 00.a0.cc.3c.d1.bf -- index 2 type 6 phy 0 addrl 6 addr 00.40.c7.11.c5.62 changing root device to wd0s1a WARNING: / was not properly dismounted Kernel Config: machine "i386" cpu "I486_CPU" cpu "I586_CPU" ident RAND maxusers 32 makeoptions DEBUG="-g" options SOFTUPDATES options INET #InterNETworking options FFS #Berkeley Fast Filesystem options FFS_ROOT #FFS usable as root device [keep this!] options NFS #Network Filesystem options PROCFS #Process filesystem options "COMPAT_43" #Compatible with BSD 4.3 [KEEP THIS!] options UCONSOLE #Allow users to grab the console options FAILSAFE #Be conservative options USERCONFIG #boot -c editor options KTRACE #ktrace(1) syscall trace support options SYSVSHM #SYSV-style shared memory options SYSVMSG #SYSV-style message queues options SYSVSEM #SYSV-style semaphores config kernel root on wd0 controller isa0 controller pci0 # IDE controller and disks controller wdc0 at isa? flags 0xa0ffa0ff port "IO_WD1" bio irq 14 disk wd0 at wdc0 drive 0 # atkbdc0 controls both the keyboard and the PS/2 mouse controller atkbdc0 at isa? port IO_KBD tty device atkbd0 at isa? tty irq 1 device psm0 at isa? tty irq 12 device vga0 at isa? port ? conflicts # splash screen/screen saver pseudo-device splash # syscons is the default console driver, resembling an SCO console device sc0 at isa? tty # Floating point support - do not disable. device npx0 at isa? port IO_NPX irq 13 # Power management support (see LINT for more options) device apm0 at isa? flags 0x31 # Advanced Power Management # Serial (COM) ports device sio0 at isa? port "IO_COM1" flags 0x10 tty irq 4 device sio1 at isa? port "IO_COM2" tty irq 3 # Parallel port device ppc0 at isa? port? flags 0x40 net irq 7 controller ppbus0 # Parallel port bus (required) device lpt0 at ppbus? # Printer device plip0 at ppbus? # TCP/IP over parallel device ppi0 at ppbus? # Parallel port interface device options BRIDGE # ISA Ethernet NICs. device ed0 at isa? port 0x340 net irq 5 iomem 0xd8000 # PCI Ethernet NICs. device pn0 # Pseudo devices - the number indicates how many units to allocated. pseudo-device loop # Network loopback pseudo-device ether # Ethernet support pseudo-device tun 2 # User-PPP pseudo-device pty 16 # Pseudo-ttys (telnet etc) pseudo-device gzip # Exec gzipped a.out's pseudo-device vn pseudo-device bpfilter 4 #Berkeley packet filter -- Tony Frank _ __ ___ ___ ___ tfrank@eric.net.au _ __ ___ | _ ) __| \ http://www.freebsd.org/ _ __ ___ ____ | _ \__ \ |) | FreeBSD: The Power to Serve! _ __ ___ ____ _____ |___/___/___/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message