From owner-freebsd-bugs@FreeBSD.ORG Mon Aug 25 23:50:17 2003 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1AC1916A4BF for ; Mon, 25 Aug 2003 23:50:17 -0700 (PDT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8A2D743FFD for ; Mon, 25 Aug 2003 23:50:15 -0700 (PDT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.12.9/8.12.9) with ESMTP id h7Q6oFUp011375 for ; Mon, 25 Aug 2003 23:50:15 -0700 (PDT) (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.12.9/8.12.9/Submit) id h7Q6oFHI011374; Mon, 25 Aug 2003 23:50:15 -0700 (PDT) Resent-Date: Mon, 25 Aug 2003 23:50:15 -0700 (PDT) Resent-Message-Id: <200308260650.h7Q6oFHI011374@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, eugen@grosbein.pp.ru Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2C41716A4BF for ; Mon, 25 Aug 2003 23:48:15 -0700 (PDT) Received: from swd.svznov.kemerovo.su (swd.svznov.kemerovo.su [213.184.64.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id AF54B43FDF for ; Mon, 25 Aug 2003 23:48:10 -0700 (PDT) (envelope-from eugen@swd.svznov.kemerovo.su) Received: from swd.svznov.kemerovo.su (smmsp@localhost [127.0.0.1]) h7Q6m3nj008715 for ; Tue, 26 Aug 2003 14:48:03 +0800 (KRAST) (envelope-from eugen@swd.svznov.kemerovo.su) Received: (from root@localhost) by swd.svznov.kemerovo.su (8.12.9/8.12.9/Submit) id h7Q6m2cB008714; Tue, 26 Aug 2003 14:48:02 +0800 (KRAST) Message-Id: <200308260648.h7Q6m2cB008714@swd.svznov.kemerovo.su> Date: Tue, 26 Aug 2003 14:48:02 +0800 (KRAST) From: Eugene Grosbein To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Subject: kern/55986: kernel panic in 4.8-STABLE X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: eugen@grosbein.pp.ru List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Aug 2003 06:50:17 -0000 >Number: 55986 >Category: kern >Synopsis: kernel panic in 4.8-STABLE >Confidential: no >Severity: serious >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon Aug 25 23:50:11 PDT 2003 >Closed-Date: >Last-Modified: >Originator: Eugene Grosbein >Release: FreeBSD 4.8-STABLE i386 >Organization: Svyaz Service JSC >Environment: System: FreeBSD swd.svznov.kemerovo.su 4.8-STABLE FreeBSD 4.8-STABLE #13: Mon Aug 11 19:40:47 KRAST 2003 root@swd.svznov.kemerovo.su:/usr/obj/usr/src/sys/SWD i386 CPUTYPE=p3 and no other optimizations >Description: Today this server crashed in time of high load - in process of full warm backup of 6G Oracle database it runs. Btw, database was corrupted after reboot. With luck, it was easy to recover this time. This system works nice over a year. For long time it ran 4.8-RC. This August I've upgraded it upto 4.8-STABLE. It is first crash after upgrade. Here comes GDB backtrace. Script started on Tue Aug 26 14:36:02 2003 GNU gdb 4.18 (FreeBSD) Copyright 1998 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-unknown-freebsd"...Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 2627 in elfstab_build_psymtabs Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 933 in fill_symbuf IdlePTD at phsyical address 0x00381000 initial pcb at physical address 0x002e85e0 panicstr: page fault panic messages: --- Fatal trap 12: page fault while in kernel mode fault virtual address = 0x11 fault code = supervisor read, page not present instruction pointer = 0x8:0xc0241970 stack pointer = 0x10:0xc9a07c9c frame pointer = 0x10:0xc9a07ca4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 67184 (oracle) interrupt mask = bio trap number = 12 panic: page fault syncing disks... 95 33 30 30 30 30 30 30 30 34 28 28 28 28 28 28 28 31 28 28 28 28 28 28 28 28 28 28 28 28 28 28 28 28 28 28 28 28 giving up on 19 buffers Uptime: 3d14h3m59s dumping to dev #ad/0x20001, offset 264704 dump ata0: resetting devices .. done 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 --- #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 487 if (dumping++) { (kgdb) bt #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 #1 0xc016450c in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316 #2 0xc0164959 in panic (fmt=0xc02b73ac "%s") at /usr/src/sys/kern/kern_shutdown.c:595 #3 0xc026e868 in trap_fatal (frame=0xc9a07c5c, eva=17) at /usr/src/sys/i386/i386/trap.c:974 #4 0xc026e4fd in trap_pfault (frame=0xc9a07c5c, usermode=0, eva=17) at /usr/src/sys/i386/i386/trap.c:867 #5 0xc026e08f in trap (frame={tf_fs = -1015283696, tf_es = 16, tf_ds = 16, tf_edi = 0, tf_esi = 56468, tf_ebp = -912229212, tf_isp = -912229240, tf_ebx = -912762076, tf_edx = -3, tf_ecx = 383843407, tf_eax = 65529, tf_trapno = 12, tf_err = 0, tf_eip = -1071376016, tf_cs = 8, tf_eflags = 66178, tf_esp = 4, tf_ss = 56468}) at /usr/src/sys/i386/i386/trap.c:466 #6 0xc0241970 in vm_page_lookup (object=0xc9985b24, pindex=56468) at /usr/src/sys/vm/vm_page.c:515 #7 0xc018d382 in allocbuf (bp=0xc37aeb84, size=16384) at /usr/src/sys/kern/vfs_bio.c:2510 #8 0xc018cf46 in getblk (vp=0xc998a8c0, blkno=14117, size=16384, slpflag=0, slptimeo=0) at /usr/src/sys/kern/vfs_bio.c:2292 #9 0xc018f8ba in cluster_rbuild (vp=0xc998a8c0, filesize=2061502464, lbn=14111, blkno=5167264, size=16384, run=8, fbp=0x0) at /usr/src/sys/kern/vfs_cluster.c:400 #10 0xc018f51a in cluster_read (vp=0xc998a8c0, filesize=2061502464, ---Type to continue, or q to quit--- lblkno=14111, size=16384, cred=0x0, totread=18432, seqcount=127, bpp=0xc9a07e48) at /usr/src/sys/kern/vfs_cluster.c:228 #11 0xc022e7b7 in ffs_read (ap=0xc9a07e6c) at /usr/src/sys/ufs/ufs/ufs_readwrite.c:267 #12 0xc019ac30 in vn_read (fp=0xc153bf40, uio=0xc9a07f14, cred=0xc1323180, flags=0, p=0xc995ca40) at vnode_if.h:334 #13 0xc0173ba4 in readv (p=0xc995ca40, uap=0xc9a07f80) at /usr/src/sys/sys/file.h:147 #14 0xc026ead9 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 1486, tf_esi = 16, tf_ebp = -1077968124, tf_isp = -912228396, tf_ebx = 1486, tf_edx = 16, tf_ecx = -1077967992, tf_eax = 145, tf_trapno = 0, tf_err = 2, tf_eip = 685391979, tf_cs = 31, tf_eflags = 514, tf_esp = -1077968148, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1175 #15 0xc0262275 in Xint0x80_syscall () #16 0x86a9018 in ?? () #17 0x86a7e48 in ?? () #18 0x869e2c1 in ?? () #19 0x869ba34 in ?? () #20 0x86a2ea3 in ?? () #21 0x86a2457 in ?? () #22 0x83a9cc8 in ?? () #23 0x817515e in ?? () #24 0x816544a in ?? () ---Type to continue, or q to quit--- #25 0x80f7db6 in ?? () #26 0x83378b7 in ?? () #27 0x846c247 in ?? () #28 0x848299e in ?? () #29 0x842f516 in ?? () #30 0x842f7ca in ?? () #31 0x830341b in ?? () #32 0x8306612 in ?? () #33 0x804dc88 in ?? () #34 0x804f705 in ?? () #35 0x86b08a0 in ?? () #36 0x804f839 in ?? () #37 0x804f41a in ?? () #38 0x8050600 in ?? () #39 0x8830b8a in ?? () #40 0x8831410 in ?? () #41 0x879067a in ?? () #42 0x8779c67 in ?? () #43 0x86c2855 in ?? () #44 0x8303c93 in ?? () #45 0x82f5efd in ?? () #46 0x82dba54 in ?? () #47 0x804dc88 in ?? () #48 0x85d322e in ?? () ---Type to continue, or q to quit--- #49 0x804cb1c in ?? () #50 0x82f4875 in ?? () #51 0x804dc88 in ?? () #52 0x804c1ca in ?? () #53 0x804b60e in ?? () #54 0x804b449 in ?? () (kgdb) frame 6 #6 0xc0241970 in vm_page_lookup (object=0xc9985b24, pindex=56468) at /usr/src/sys/vm/vm_page.c:515 515 for (m = *bucket; m != NULL; m = m->hnext) { (kgdb) p m $1 = 0x0 (kgdb) p *bucket $2 = {kb_next = 0x0, kb_last = 0x0, kb_calls = 0, kb_total = 0, kb_elmpercl = 4096, kb_totalfree = 0, kb_highwat = 20480, kb_couldfree = 0} (kgdb) quit Script done on Tue Aug 26 14:36:42 2003 Now kernel config file. machine i386 #cpu I386_CPU #cpu I486_CPU #cpu I586_CPU cpu I686_CPU ident SWD maxusers 128 makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols #options MATH_EMULATE #Support for x87 emulation options INET #InterNETworking options FFS #Berkeley Fast Filesystem options FFS_ROOT #FFS usable as root device [keep this!] options SOFTUPDATES #Enable FFS soft updates support #options MFS #Memory Filesystem #options MD_ROOT #MD is a potential root device options NFS #Network Filesystem options NFS_ROOT #NFS usable as root device, NFS required options MSDOSFS #MSDOS Filesystem options CD9660 #ISO 9660 Filesystem options CD9660_ROOT #CD-ROM usable as root, CD9660 required #options PROCFS #Process filesystem options COMPAT_43 #Compatible with BSD 4.3 [KEEP THIS!] options SCSI_DELAY=15000 #Delay (in ms) before probing SCSI options UCONSOLE #Allow users to grab the console options USERCONFIG #boot -c editor options VISUAL_USERCONFIG #visual boot -c editor options KTRACE #ktrace(1) support options SYSVSHM #SYSV-style shared memory options SYSVMSG #SYSV-style message queues options SYSVSEM #SYSV-style semaphores options P1003_1B #Posix P1003_1B real-time extensions options _KPOSIX_PRIORITY_SCHEDULING options ICMP_BANDLIM #Rate limit bad replies options KBD_INSTALL_CDEV # install a CDEV entry in /dev options SHMALL=4097 options SHMMAXPGS=10000 options SHMMNI=100 options SHMSEG=10 options SEMMNS=255 options SEMMNI=235 options SEMMSL=255 options SEMMAP=255 options SEMMNU=235 options SEMOPM=201 options SEMUME=201 options IPFIREWALL options IPFIREWALL_VERBOSE #print information about options IPFIREWALL_FORWARD options IPDIVERT options DUMMYNET device isa device eisa device pci # Floppy drives device fdc0 at isa? port IO_FD1 irq 6 drq 2 device fd0 at fdc0 drive 0 # ATA and ATAPI devices device ata0 at isa? port IO_WD1 irq 14 device ata1 at isa? port IO_WD2 irq 15 device ata device atadisk # ATA disk drives device atapicd # ATAPI CDROM drives device atapifd # ATAPI floppy drives options ATA_STATIC_ID #Static device numbering # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc0 at isa? port IO_KBD device atkbd0 at atkbdc? irq 1 flags 0x0 device psm0 at atkbdc? irq 12 device vga0 at isa? # splash screen/screen saver pseudo-device splash # syscons is the default console driver, resembling an SCO console device sc0 at isa? flags 0x100 # Enable this and PCVT_FREEBSD for pcvt vt220 compatible console driver #device vt0 at isa? #options XSERVER # support for X server on a vt console #options FAT_CURSOR # start with block cursor # If you have a ThinkPAD, uncomment this along with the rest of the PCVT lines #options PCVT_SCANSET=2 # IBM keyboards are non-std # Floating point support - do not disable. device npx0 at nexus? port IO_NPX irq 13 # Serial (COM) ports device sio0 at isa? port IO_COM1 flags 0x10 irq 4 device sio1 at isa? port IO_COM2 irq 3 #device sio2 at isa? disable port IO_COM3 irq 5 #device sio3 at isa? disable port IO_COM4 irq 9 # Parallel port device ppc0 at isa? irq 7 device ppbus # Parallel port bus (required) device lpt # Printer device plip # TCP/IP over parallel device ppi # Parallel port interface device #device vpo # Requires scbus and da # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs! device miibus # MII bus support device fxp # Pseudo devices - the number indicates how many units to allocate. pseudo-device loop # Network loopback pseudo-device ether # Ethernet support pseudo-device sl 1 # Kernel SLIP pseudo-device ppp 1 # Kernel PPP pseudo-device tun # Packet tunnel. pseudo-device pty # Pseudo-ttys (telnet etc) #pseudo-device md # Memory "disks" # The `bpf' pseudo-device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! pseudo-device bpf #Berkeley packet filter # This allows you to actually store this configuration file into # the kernel binary itself, where it may be later read by saying: # strings -n 3 /kernel | sed -n 's/^___//p' > MYKERNEL # options INCLUDE_CONFIG_FILE # Include this file in kernel options CPU_ENABLE_SSE options RANDOM_IP_ID options UFS_DIRHASH options QUOTA pseudo-device gif #IPv6 and IPv4 tunneling pseudo-device speaker #Play IBM BASIC-style noises out your speaker pseudo-device gzip #Exec gzipped a.out's pseudo-device vn #Vnode driver (turns a file into a device) pseudo-device snp 6 #Snoop device - to look at pty/vty/etc.. options SC_HISTORY_SIZE=1000 options SC_MOUSE_CHAR=0x3 options SC_TWOBUTTON_MOUSE options DDB options DDB_UNATTENDED options IPSEC options IPSEC_ESP options IPSEC_FILTERGIF And now dmesg.boot: Copyright (c) 1992-2003 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 4.8-STABLE #13: Mon Aug 11 19:40:47 KRAST 2003 root@swd.svznov.kemerovo.su:/usr/obj/usr/src/sys/SWD Timecounter "i8254" frequency 1193182 Hz Timecounter "TSC" frequency 730965706 Hz CPU: Intel Pentium III (730.97-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x683 Stepping = 3 Features=0x383f9ff real memory = 132907008 (129792K bytes) config> q avail memory = 125652992 (122708K bytes) Preloaded elf kernel "kernel" at 0xc0362000. Preloaded userconfig_script "/boot/kernel.conf" at 0xc036209c. Pentium Pro MTRR support enabled Using $PIR table, 12 entries at 0xc00f2dd0 npx0: on motherboard npx0: INT 16 interface pcib0: on motherboard pci0: on pcib0 pci0: at 2.0 irq 11 pcib1: at device 30.0 on pci0 pci1: on pcib1 pci1: (vendor=0x1274, dev=0x1371) at 7.0 irq 10 fxp0: port 0xdf40-0xdf5f mem 0xff600000-0xff6fffff,0xf6afe000-0xf6afefff irq 11 at device 9.0 on pci1 fxp0: Ethernet address 00:a0:c9:29:4b:83 nsphy0: on miibus0 nsphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp1: port 0xdf80-0xdf9f mem 0xff800000-0xff8fffff,0xf6aff000-0xf6afffff irq 9 at device 11.0 on pci1 fxp1: Ethernet address 00:a0:c9:41:d6:f1 inphy0: on miibus1 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0xffa0-0xffaf at device 31.1 on pci0 ata0: at 0x1f0 irq 14 on atapci0 ata1: at 0x170 irq 15 on atapci0 pci0: at 31.2 irq 11 pci0: (vendor=0x8086, dev=0x2443) at 31.3 irq 10 pci0: at 31.4 irq 9 orm0: