From owner-freebsd-i386@FreeBSD.ORG Wed Dec 10 15:30:01 2008 Return-Path: Delivered-To: freebsd-i386@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D2109106567F for ; Wed, 10 Dec 2008 15:30:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id A77B68FC2A for ; Wed, 10 Dec 2008 15:30:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id mBAFU1Np092120 for ; Wed, 10 Dec 2008 15:30:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id mBAFU14l092117; Wed, 10 Dec 2008 15:30:01 GMT (envelope-from gnats) Resent-Date: Wed, 10 Dec 2008 15:30:01 GMT Resent-Message-Id: <200812101530.mBAFU14l092117@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-i386@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Mike Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8A5851065677 for ; Wed, 10 Dec 2008 15:25:21 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 7ABF28FC25 for ; Wed, 10 Dec 2008 15:25:21 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id mBAFPKsF070075 for ; Wed, 10 Dec 2008 15:25:20 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id mBAFPKoC070074; Wed, 10 Dec 2008 15:25:20 GMT (envelope-from nobody) Message-Id: <200812101525.mBAFPKoC070074@www.freebsd.org> Date: Wed, 10 Dec 2008 15:25:20 GMT From: Mike To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: i386/129550: crash with PAE kernel X-BeenThere: freebsd-i386@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: I386-specific issues for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Dec 2008 15:30:02 -0000 >Number: 129550 >Category: i386 >Synopsis: crash with PAE kernel >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-i386 >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Wed Dec 10 15:30:01 UTC 2008 >Closed-Date: >Last-Modified: >Originator: Mike >Release: 7 >Organization: Nux >Environment: FreeBSD redbus-lingo1950.nux.co.uk 7.0-RELEASE-p6 FreeBSD 7.0-RELEASE-p6 #7: Tue Dec 2 00:48:03 GMT 2008 razvan@redbus-lingo1950.nux.co.uk:/usr/obj/usr/src/sys/PAE i386 >Description: Hi, We have a problem with dell poweredge 1950 server and freebsd 7. The machine is used as a jail host with one jail running (with ezjail-admin), other than the standard stuff like syslog and ssh the main progam causing any siginificant work is dovecot - this has been compiled with kqueue support. Other than the below is there any other information we can provide to diagnose this problem - the machine is a live server with about 250gb of Maildirs on it with about 200 users connected at the same time. >From /var/log/messages: Dec 10 10:00:39 redbus-lingo1950 su: razvan to root on /dev/ttyp0 Dec 10 14:21:49 redbus-lingo1950 kernel: Dec 10 14:21:49 redbus-lingo1950 kernel: Dec 10 14:21:49 redbus-lingo1950 kernel: Fatal trap 12: page fault while in kernel mode Dec 10 14:21:49 redbus-lingo1950 kernel: cpuid = 3; apic id = 03 Dec 10 14:21:49 redbus-lingo1950 kernel: fault virtual address = 0x0 Dec 10 14:21:49 redbus-lingo1950 kernel: fault code = supervisor read, page not Dec 10 14:21:49 redbus-lingo1950 kernel: present Dec 10 14:21:49 redbus-lingo1950 kernel: instruction pointer = 0x20:0x0 Dec 10 14:21:49 redbus-lingo1950 kernel: stack pointer = 0x28:0xf4955a28 Dec 10 14:21:49 redbus-lingo1950 kernel: frame pointer = 0x28:0xf4955a48 Dec 10 14:21:49 redbus-lingo1950 kernel: code segment = base 0x0, limit 0xfffff, type 0x1b Dec 10 14:21:49 redbus-lingo1950 kernel: = DPL 0, Dec 10 14:21:49 redbus-lingo1950 kernel: Dec 10 14:38:57 redbus-lingo1950 syslogd: kernel boot file is /boot/kernel/kernel redbus-lingo1950# kgdb kernel.debug /var/crash/vmcore.0 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 57762 (imap) trap number = 12 panic: page fault cpuid = 3 Uptime: 8d13h27m30s Physical memory: 8175 MB Dumping 475 MB: 460 444 428 412 396 380 364 348 332 316 300 284 268 252 236 220 204 188 172 156 140 124 108 92 76 60 44 28 12 #0 doadump () at pcpu.h:195 195 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) list *0x20:0x0 A syntax error in expression, near `:0x0'. (kgdb) list "0x20:0x0" No source file named 0x20. (kgdb) backtrace #0 doadump () at pcpu.h:195 #1 0xc04ed227 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc04ed4e9 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #3 0xc072f24c in trap_fatal (frame=0xf49559e8, eva=0) at /usr/src/sys/i386/i386/trap.c:899 #4 0xc072f4e0 in trap_pfault (frame=0xf49559e8, usermode=0, eva=0) at /usr/src/sys/i386/i386/trap.c:812 #5 0xc072fe8c in trap (frame=0xf49559e8) at /usr/src/sys/i386/i386/trap.c:490 #6 0xc071482b in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0x00000000 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) quit Other info: redbus-lingo1950# uname -a FreeBSD redbus-lingo1950.nux.co.uk 7.0-RELEASE-p6 FreeBSD 7.0-RELEASE-p6 #7: Tue Dec 2 00:48:03 GMT 2008 razvan@redbus-lingo1950.nux.co.uk:/usr/obj/usr/src/sys/PAE i386 redbus-lingo1950# mount /dev/mfid0s1a on / (ufs, local) devfs on /dev (devfs, local) /dev/mfid0s1g on /home (ufs, local, soft-updates) /dev/mfid0s1e on /tmp (ufs, local, soft-updates) /dev/mfid0s1f on /usr (ufs, local, soft-updates) /dev/mfid0s1d on /var (ufs, local, soft-updates) /home/jails/basejail on /home/jails/mail.lingo24.com/basejail (nullfs, local, read-only) devfs on /home/jails/mail.lingo24.com/dev (devfs, local) redbus-lingo1950# sysctl kern.sched.name kern.sched.name: ULE Kernel config is standard PAE but with ULE and the following added: # make pf in kernel as wwell device pf device pflog # need to add nullfs device nullfs redbus-lingo1950# dmesg Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-RELEASE-p6 #7: Tue Dec 2 00:48:03 GMT 2008 razvan@redbus-lingo1950.nux.co.uk:/usr/obj/usr/src/sys/PAE Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU L5335 @ 2.00GHz (1997.03-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6fb Stepping = 11 Features=0xbfebfbff Features2=0x4e33d AMD Features=0x20100000 AMD Features2=0x1 Cores per package: 4 real memory = 9395240960 (8960 MB) avail memory = 8394002432 (8005 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 ioapic0: Changing APIC ID to 8 ioapic0 irqs 0-23 on motherboard registered firmware set registered firmware set registered firmware set registered firmware set registered firmware set registered firmware set registered firmware set registered firmware set registered firmware set registered firmware set registered firmware set kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 cpu0: on acpi0 p4tcc0: on cpu0 cpu1: on acpi0 p4tcc1: on cpu1 cpu2: on acpi0 p4tcc2: on cpu2 cpu3: on acpi0 p4tcc3: on cpu3 cpu4: on acpi0 p4tcc4: on cpu4 cpu5: on acpi0 p4tcc5: on cpu5 cpu6: on acpi0 p4tcc6: on cpu6 cpu7: on acpi0 p4tcc7: on cpu7 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci4: on pcib1 pcib2: at device 0.0 on pci4 pci5: on pcib2 pcib3: at device 0.0 on pci5 pci6: on pcib3 pcib4: at device 0.0 on pci6 pci7: on pcib4 bce0: mem 0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci7 miibus0: on bce0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bce0: Ethernet address: 00:1d:09:65:93:41 bce0: [ITHREAD] bce0: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W (0x03050C05); Flags( MFW MSI ) pcib5: at device 1.0 on pci5 pci8: on pcib5 pcib6: at device 0.3 on pci4 pci9: on pcib6 pcib7: at device 3.0 on pci0 pci1: on pcib7 mfi0: port 0xec00-0xecff mem 0xfc480000-0xfc4bffff,0xfc440000-0xfc47ffff irq 16 at device 0.0 on pci1 mfi0: Megaraid SAS driver Ver 2.00 mfi0: 2433 (282234152s/0x0020/0) - Shutdown command received from host mfi0: 2434 (4278190083s/0x0020/0) - PCI 0x041000 0x0460 0x041028 0x041f0c: Firmware initialization started (PCI ID 0060/1000/1f0c/1028) mfi0: 2435 (4278190083s/0x0020/0) - Type 18: Firmware version 1.11.52-0349 mfi0: 2436 (4278190083s/0x0008/0) - Battery Present mfi0: 2437 (4278190083s/0x0020/0) - Type 18: Package version 6.0.1-0080 mfi0: 2438 (4278190101s/0x0004/0) - PD 32(e0/s255) event: Enclosure (SES) discovered on PD 20(c None/p0) mfi0: 2439 (4278190101s/0x0002/0) - PD 32(e32/s255) event: Inserted: Encl PD 20 mfi0: 2440 (4278190101s/0x0002/0) - Type 29: Inserted: PD 20(c None/p0) Info: enclPd=20, scsiType=d, portMap=09, sasAddr=5001c030dd3f2b00,0000000000000000 mfi0: 2441 (4278190101s/0x0002/0) - PD 00(e32/s0) event: Inserted: PD 00(e0x20/s0) mfi0: 2442 (4278190101s/0x0002/0) - Type 29: Inserted: PD 00(e0x20/s0) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=5000c5000336be59,0000000000000000 mfi0: 2443 (4278190101s/0x0002/0) - PD 01(e32/s1) event: Inserted: PD 01(e0x20/s1) mfi0: 2444 (4278190101s/0x0002/0) - Type 29: Inserted: PD 01(e0x20/s1) Info: enclPd=20, scsiType=0, portMap=01, sasAddr=5000c5000336b375,0000000000000000 mfi0: 2445 (282234233s/0x0020/0) - Adapter ticks 282234233 elapsed 59s: Time established as 12/10/08 14:23:53; (59 seconds since power on) mfi0: 2446 (282234242s/0x0008/0) - Battery temperature is normal mfi0: 2447 (282234242s/0x0008/0) - Current capacity of the battery is above threshold mfi0: [ITHREAD] pcib8: at device 4.0 on pci0 pci10: on pcib8 pcib9: at device 5.0 on pci0 pci11: on pcib9 pcib10: at device 6.0 on pci0 pci12: on pcib10 pcib11: at device 7.0 on pci0 pci13: on pcib11 pcib12: at device 28.0 on pci0 pci2: on pcib12 pcib13: at device 0.0 on pci2 pci3: on pcib13 bce1: mem 0xf8000000-0xf9ffffff irq 16 at device 0.0 on pci3 miibus1: on bce1 brgphy1: PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bce1: Ethernet address: 00:1d:09:65:93:3f bce1: [ITHREAD] bce1: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W (0x03050C05); Flags( MFW MSI ) pci0: at device 29.0 (no driver attached) pci0: at device 29.1 (no driver attached) pci0: at device 29.2 (no driver attached) pci0: at device 29.3 (no driver attached) pci0: at device 29.7 (no driver attached) pcib14: at device 30.0 on pci0 pci14: on pcib14 vgapci0: port 0xdc00-0xdcff mem 0xd8000000-0xdfffffff,0xfc2d0000-0xfc2dffff irq 19 at device 13.0 on pci14 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 31.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: does not respond device_attach: fdc0 attach returned 6 sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A, console sio0: [FILTER] sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A sio1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: does not respond device_attach: fdc0 attach returned 6 pmtimer0 on isa0 orm0: at iomem 0xc0000-0xc8fff,0xc9000-0xc9fff,0xd2000-0xd2fff,0xec000-0xeffff pnpid ORM0000 on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] ppc0: parallel port not found. sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x100> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec acd0: CDROM at ata0-master UDMA33 mfid0: on mfi0 mfid0: 380928MB (780140544 sectors) RAID volume '' is optimal SMP: AP CPU #1 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #3 Launched! Trying to mount root from ufs:/dev/mfid0s1a WARNING: / was not properly dismounted redbus-lingo1950# >How-To-Repeat: unknown >Fix: unknown >Release-Note: >Audit-Trail: >Unformatted: