Date: Thu, 19 Apr 2007 14:33:29 +0200 From: Kai <kai@xs4all.nl> To: stable@freebsd.org Subject: Fatal trap 12: page fault while in kernel mode Message-ID: <20070419123329.GA10189@xs4all.nl> In-Reply-To: <20070411105332.GC7847@xs4all.nl> References: <20070411105332.GC7847@xs4all.nl>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Apr 11, 2007 at 12:53:32PM +0200, Kai wrote:
>
> Hello all,
>
> We're running into regular panics on our webserver after upgrading
> from 4.x to 6.2-stable:
Hi Again,
The panics keep happening, so I'm trying alternate kernel setups. This is a
trace of a panic on a default SMP kernel with debugging symbols.
I'm At a loss on how to progress at this point, perhaps someone can help me
please?
System info:
uname -a reports FreeBSD webserver 6.2-RELEASE-p3 FreeBSD 6.2-RELEASE-p3 #0:
Thu Apr 19 11:37:34 CEST 2007 kai@buildserver:/usr/src/sys/i386/compile/SMP
i386
Backtrace:
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:
Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x34
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc06bdefa
stack pointer = 0x28:0xeb9cf938
frame pointer = 0x28:0xeb9cf944
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 13577 (perl5.8.8)
trap number = 12
panic: page fault
cpuid = 0
Uptime: 1h37m29s
Dumping 3070 MB (2 chunks)
chunk 0: 1MB (156 pages) ... ok
chunk 1: 3071MB (786016 pages) 3055 3039 3023 3007 2991 2975 2959 2943
2927 2911 2895 2879 2863 2847 2831 2815 2799 2783 2767 2751 2735 2719 2703
2687 2671 2655 2639 2623 2607 2591 2575 2559 2543 2527 2511 2495 2479 2463
2447 2431 2415 2399 2383 2367 2351 2335 2319 2303 2287 2271 2255 2239 2223
2207 2191 2175 2159 2143 2127 2111 2095 2079 2063 2047 2031 2015 1999 1983
1967 1951 1935 1919 1903 1887 1871 1855 1839 1823 1807 1791 1775 1759 1743
1727 1711 1695 1679 1663 1647 1631 1615 1599 1583 1567 1551 1535 1519 1503
1487 1471 1455 1439 1423 1407 1391 1375 1359 1343 1327 1311 1295 1279 1263
1247 1231 1215 1199 1183 1167 1151 1135 1119 1103 1087 1071 1055 1039 1023
1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735
719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447
431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159
143 127 111 95 79 63 47 31 15
#0 doadump () at pcpu.h:165
165 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0 doadump () at pcpu.h:165
#1 0xc067550a in boot (howto=260) at ../../../kern/kern_shutdown.c:409
#2 0xc0675831 in panic (fmt=0xc08e46dd "%s")
at ../../../kern/kern_shutdown.c:565
#3 0xc088e29c in trap_fatal (frame=0xeb9cf8f8, eva=52)
at ../../../i386/i386/trap.c:837
#4 0xc088dfdb in trap_pfault (frame=0xeb9cf8f8, usermode=0, eva=52)
at ../../../i386/i386/trap.c:745
#5 0xc088dc15 in trap (frame=
{tf_fs = 8, tf_es = -606142424, tf_ds = -926089176, tf_edi =
#-605280928, tf_esi = -605280928, tf_ebp = -342034108, tf_isp = -342034140,
tf_ebx = -605280928, tf_edx = 4, tf_ecx = -926082048, tf_eax = 0, tf_trapno
#= 12, tf_err = 0, tf_eip = -1066672390, tf_cs = 32, tf_eflags = 66050,
tf_esp = -605280928, tf_ss = -605280928}) at ../../../i386/i386/trap.c:435
#6 0xc0879d4a in calltrap () at ../../../i386/i386/exception.s:139
#7 0xc06bdefa in vfs_vmio_release (bp=0xdbec2560) at atomic.h:146
#8 0xc06be728 in getnewbuf (slpflag=0, slptimeo=0, size=6585, maxsize=8192)
at ../../../kern/vfs_bio.c:1779
#9 0xc06bfccc in getblk (vp=0xca2cfdd0, blkno=8438, size=6585, slpflag=0,
slptimeo=0, flags=0) at ../../../kern/vfs_bio.c:2497
#10 0xc075ad41 in nfs_getcacheblk (vp=0xca2cfdd0, bn=8438, size=6585,
td=0xc8cd1c00) at ../../../nfsclient/nfs_bio.c:1261
#11 0xc075a978 in nfs_write (ap=0x0) at ../../../nfsclient/nfs_bio.c:1069
#12 0xc089fde6 in VOP_WRITE_APV (vop=0xc0984440, a=0xeb9cfbec)
at vnode_if.c:698
#13 0xc06dbb26 in vn_write (fp=0xc8940e10, uio=0xeb9cfcbc,
active_cred=0xc89ee880, flags=0, td=0xc8cd1c00) at vnode_if.h:372
#14 0xc0698f63 in dofilewrite (td=0xc8cd1c00, fd=5, fp=0xc8940e10,
auio=0xeb9cfcbc, offset=Unhandled dwarf expression opcode 0x93
) at file.h:252
#15 0xc0698e07 in kern_writev (td=0xc8cd1c00, fd=5, auio=0xeb9cfcbc)
at ../../../kern/sys_generic.c:402
#16 0xc0698d2d in write (td=0xc8cd1c00, uap=0xc8cd1c00)
at ../../../kern/sys_generic.c:326
#17 0xc088e5e3 in syscall (frame=
{tf_fs = 136970299, tf_es = 673775675, tf_ds = -1078001605, tf_edi =
#137019392, tf_esi = 673803768, tf_ebp = -1077942616, tf_isp = -342033052,
tf_ebx = 673713536, tf_edx = 4096, tf_ecx = 0, tf_eax = 4, tf_trapno = 0,
#tf_err = 2, tf_eip = 673653551, tf_cs = 51, tf_eflags = 530, tf_esp =
-1077942644, tf_ss = 59})
at ../../../i386/i386/trap.c:983
#18 0xc0879d9f in Xint0x80_syscall () at ../../../i386/i386/exception.s:200
#19 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
Dmesg reports:
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RELEASE-p3 #0: Thu Apr 19 11:37:34 CEST 2007
kai@buildserver:/usr/src/sys/i386/compile/SMP
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU 5130 @ 2.00GHz (2000.08-MHz 686-class
CPU)
Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4e33d<SSE3,RSVD2,MON,DS_CPL,VMX,TM2,<b9>,CX16,<b14>,<b15>,<b18>>
AMD Features=0x20000000<LM>
AMD Features2=0x1<LAHF>
Cores per package: 2
real memory = 3220570112 (3071 MB)
avail memory = 3150516224 (3004 MB)
ACPI APIC Table: <PTLTD APIC >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <PTLTD RSDT> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
cpu1: <ACPI CPU> on acpi0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
cpu2: <ACPI CPU> on acpi0
acpi_throttle2: <ACPI CPU Throttling> on cpu2
acpi_throttle2: failed to attach P_CNT
device_attach: acpi_throttle2 attach returned 6
cpu3: <ACPI CPU> on acpi0
acpi_throttle3: <ACPI CPU Throttling> on cpu3
acpi_throttle3: failed to attach P_CNT
device_attach: acpi_throttle3 attach returned 6
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci4: <ACPI PCI bus> on pcib4
em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x2000-0x201f mem 0xc8200000-0xc821ffff irq 18 at device 0.0 on pci4
em0: Ethernet address: 00:30:48:32:94:ba
em1: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x2020-0x203f mem 0xc8220000-0xc823ffff irq 19 at device 0.1 on pci4
em1: Ethernet address: 00:30:48:32:94:bb
pcib5: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci5: <ACPI PCI bus> on pcib5
aac0: <Adaptec SCSI RAID 2020ZCR> mem
0xc9000000-0xc9ffffff,0xc8400000-0xc85fffff,0xcc000000-0xcfffffff irq 28 at
device 2.0 on pci5
aac0: New comm. interface enabled
aac0: Adaptec Raid Controller 2.0.0-1
aacp0: <SCSI Passthrough Bus> on aac0
aacp1: <SCSI Passthrough Bus> on aac0
pcib6: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci7: <ACPI PCI bus> on pcib7
pci0: <base peripheral> at device 8.0 (no driver attached)
uhci0: <UHCI (generic) USB controller> port 0x1800-0x181f irq 17 at device
29.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <UHCI (generic) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <UHCI (generic) USB controller> port 0x1820-0x183f irq 19 at device
29.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <UHCI (generic) USB controller> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <UHCI (generic) USB controller> port 0x1840-0x185f irq 18 at device
29.2 on pci0
uhci2: [GIANT-LOCKED]
usb2: <UHCI (generic) USB controller> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3: <UHCI (generic) USB controller> port 0x1860-0x187f irq 16 at device
29.3 on pci0
uhci3: [GIANT-LOCKED]
usb3: <UHCI (generic) USB controller> on uhci3
usb3: USB revision 1.0
uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xc8000000-0xc80003ff irq 17
at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <EHCI (generic) USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
pcib8: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci8: <ACPI PCI bus> on pcib8
pci8: <display, VGA> at device 1.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1880-0x188f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A, console
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xcb000-0xcf7ff on isa0
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
acd0: DMA limited to UDMA33, controller found non-ATA66 cable
acd0: DVDROM <MATSHITADVD-ROM SR-8178/PZ16> at ata0-slave UDMA33
aacd0: <RAID 1 (Mirror)> on aac0
aacd0: 139981MB (286681856 sectors)
ses0 at aacp0 bus 0 target 6 lun 0
ses0: <SUPER GEM318 0> Fixed unknown SCSI-2 device
ses0: 3.300MB/s transfers
ses0: SAF-TE Compliant Device
pass0 at aacp0 bus 0 target 0 lun 0
pass0: <SEAGATE ST3146707LC 0005> Fixed unknown SCSI-3 device
pass0: 3.300MB/s transfers
pass1 at aacp0 bus 0 target 1 lun 0
pass1: <SEAGATE ST3146707LC 0005> Fixed unknown SCSI-3 device
pass1: 3.300MB/s transfers
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Trying to mount root from ufs:/dev/aacd0s1a
WARNING: / was not properly dismounted
WARNING: /var was not properly dismounted
WARNING: /usr was not properly dismounted
WARNING: /usr/local was not properly dismounted
nfs_getpages: error 70
vm_fault: pager read error, pid 5510 (httpd)
pid 5574 (httpd), uid 65534: exited on signal 10
pid 5811 (httpd), uid 65534: exited on signal 10
nfs_getpages: error 70
vm_fault: pager read error, pid 7560 (httpd)
Kernel modules:
# kldstat
Id Refs Address Size Name
1 3 0xc0400000 70792c kernel
2 1 0xc0b08000 59f20 acpi.ko
Thanks in advance,
Kai Storbeck
--
This was an above the .signature production
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070419123329.GA10189>