From owner-freebsd-bugs@FreeBSD.ORG Thu Apr 28 16:00:24 2011 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 521D81065675 for ; Thu, 28 Apr 2011 16:00:24 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 37C088FC2C for ; Thu, 28 Apr 2011 16:00:23 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p3SG0N9n038441 for ; Thu, 28 Apr 2011 16:00:23 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p3SG0NrP038439; Thu, 28 Apr 2011 16:00:23 GMT (envelope-from gnats) Date: Thu, 28 Apr 2011 16:00:23 GMT Message-Id: <201104281600.p3SG0NrP038439@freefall.freebsd.org> To: freebsd-bugs@FreeBSD.org From: "Helmut Schneider" Cc: Subject: Re: kern/156691: panic when using hard disks as RAW devices within VMWare ESXi 4.1-u1 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Helmut Schneider List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2011 16:00:24 -0000 The following reply was made to PR kern/156691; it has been noted by GNATS. From: "Helmut Schneider" To: Cc: Subject: Re: kern/156691: panic when using hard disks as RAW devices within VMWare ESXi 4.1-u1 Date: Thu, 28 Apr 2011 17:59:33 +0200 OK, I managed to get a backtrace. As a sidenote, I know that the mangled entry might be caused by hardware failures but - as said before, Windows 7, Ubuntu and OpenBSD do fine with the same configuration - The system is equiped with FBDIMM (ECC) and neither memtest nor the ECC check report errors - I tested 3 different disks - While FreeBSD crashes 4 more VMs continue to run on the same ESXi server, 2 of them FreeBSD 8.2 on virtuak disks BSDTest# kgdb kernel.debug /root/vmcore.1 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... Unread portion of the kernel message buffer: panic: ufs_dirbad: /mnt/da1: bad dir ino 117760 at offset 257: mangled entry cpuid = 0 KDB: stack backtrace: #0 0xc08e0d07 at kdb_backtrace+0x47 #1 0xc08b1dc7 at panic+0x117 #2 0xc0af9c56 at ufs_dirbad+0x46 #3 0xc0afae21 at ufs_lookup_+0x371 #4 0xc0afb5ee at ufs_lookup+0x1e #5 0xc0c02c62 at VOP_CACHEDLOOKUP_APV+0x42 #6 0xc092b056 at vfs_cache_lookup+0xd6 #7 0xc0c04a46 at VOP_LOOKUP_APV+0x46 #8 0xc093290e at lookup+0x69e #9 0xc093394b at namei+0x63b #10 0xc094b376 at vn_open_cred+0x96 #11 0xc094b96b at vn_open+0x3b #12 0xc0948e15 at kern_openat+0x125 #13 0xc09491f5 at kern_open+0x35 #14 0xc0949230 at open+0x30 #15 0xc08eca39 at syscallenter+0x329 #16 0xc0be4e14 at syscall+0x34 #17 0xc0bcbf21 at Xint0x80_syscall+0x21 Uptime: 8m29s Physical memory: 1011 MB Dumping 157 MB: 142 126 110 94 78 62 46 30 14 #0 doadump () at pcpu.h:231 231 __asm("movl %%fs:0,%0" : "=r" (td)); (kgdb) bt #0 doadump () at pcpu.h:231 #1 0xc08b1b63 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419 #2 0xc08b1e00 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:592 #3 0xc0af9c56 in ufs_dirbad (ip=0x0, offset=257, how=0xc0cd1a30 "mangled entry") at /usr/src/sys/ufs/ufs/ufs_lookup.c:775 #4 0xc0afae21 in ufs_lookup_ (vdp=0xc4c67430, vpp=0xe6879b90, cnp=0xe6879ba4, dd_ino=0x0) at /usr/src/sys/ufs/ufs/ufs_lookup.c:395 #5 0xc0afb5ee in ufs_lookup (ap=0xe6879944) at /usr/src/sys/ufs/ufs/ufs_lookup.c:218 #6 0xc0c02c62 in VOP_CACHEDLOOKUP_APV (vop=0xc0dadee0, a=0xe6879944) at vnode_if.c:187 #7 0xc092b056 in vfs_cache_lookup (ap=0xe68799cc) at vnode_if.h:80 #8 0xc0c04a46 in VOP_LOOKUP_APV (vop=0xc0dae380, a=0xe68799cc) at vnode_if.c:123 #9 0xc093290e in lookup (ndp=0xe6879b78) at vnode_if.h:54 #10 0xc093394b in namei (ndp=0xe6879b78) at /usr/src/sys/kern/vfs_lookup.c:269 #11 0xc094b376 in vn_open_cred (ndp=0xe6879b78, flagp=0xe6879c2c, cmode=384, vn_open_flags=0, cred=0xc4913700, fp=0xc495a930) at /usr/src/sys/kern/vfs_vnops.c:130 #12 0xc094b96b in vn_open (ndp=0xe6879b78, flagp=0xe6879c2c, cmode=384, fp=0xc495a930) at /usr/src/sys/kern/vfs_vnops.c:94 #13 0xc0948e15 in kern_openat (td=0xc491bb40, fd=-100, path=0x805b72c
, pathseg=UIO_USERSPACE, flags=1538, mode=384) at /usr/src/sys/kern/vfs_syscalls.c:1083 #14 0xc09491f5 in kern_open (td=0xc491bb40, path=0x805b72c
, pathseg=UIO_USERSPACE, flags=1537, mode=384) at /usr/src/sys/kern/vfs_syscalls.c:1039 #15 0xc0949230 in open (td=0xc491bb40, uap=0xe6879cec) at /usr/src/sys/kern/vfs_syscalls.c:1015 #16 0xc08eca39 in syscallenter (td=0xc491bb40, sa=0xe6879ce4) at /usr/src/sys/kern/subr_trap.c:315 #17 0xc0be4e14 in syscall (frame=0xe6879d28) at /usr/src/sys/i386/i386/trap.c:1061 #18 0xc0bcbf21 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:264 #19 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) list 264 call syscall 265 add $4, %esp 266 MEXITCOUNT 267 jmp doreti 268 269 ENTRY(fork_trampoline) 270 pushl %esp /* trapframe pointer */ 271 pushl %ebx /* arg1 */ 272 pushl %esi /* function */ 273 call fork_exit (kgdb) After the reboot the (GPT partitioned) disk looks like this: BSDTest# gpart show da1 => 34 156301421 da1 GPT (75G) 34 128 1 freebsd-boot (64K) 162 4194304 2 freebsd-ufs (2.0G) 4194466 2097152 3 freebsd-swap (1.0G) 6291618 16777216 4 freebsd-ufs (8.0G) 23068834 4194304 5 freebsd-ufs (2.0G) 27263138 129038317 6 freebsd-ufs (62G) BSDTest# fsck_ufs /dev/da1p1 ** /dev/da1p1 CANNOT READ BLK: 128 CONTINUE? [yn] y THE FOLLOWING DISK SECTORS COULD NOT BE READ: 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, ioctl (GCINFO): Inappropriate ioctl for device fsck_ufs: /dev/da1p1: can't read disk label BSDTest# fsck_ufs /dev/da1p2 ** /dev/da1p2 ** Last Mounted on /mnt/da1 ** Phase 1 - Check Blocks and Sizes CYLINDER GROUP 1: BAD MAGIC NUMBER REBUILD CYLINDER GROUP? [yn] y CYLINDER GROUP 3: BAD MAGIC NUMBER REBUILD CYLINDER GROUP? [yn] y UNKNOWN FILE TYPE I=70656 CLEAR? [yn] y UNKNOWN FILE TYPE I=70657 CLEAR? [yn] y UNKNOWN FILE TYPE I=70658 CLEAR? [yn] ^C ***** FILE SYSTEM STILL DIRTY ***** BSDTest# fsck_ufs /dev/da1p3 ** /dev/da1p3 ** Last Mounted on ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 2 files, 2 used, 506485 free (21 frags, 63308 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** BSDTest# fsck_ufs /dev/da1p4 ** /dev/da1p4 ** Last Mounted on /mnt/da1/usr ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 2 files, 2 used, 4061061 free (21 frags, 507630 blocks, 0.0% fragmentation) ***** FILE SYSTEM MARKED CLEAN ***** BSDTest# fsck_ufs /dev/da1p5 ** /dev/da1p5 ** Last Mounted on /mnt/da1/var ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 2 files, 2 used, 1013013 free (21 frags, 126624 blocks, 0.0% fragmentation) ***** FILE SYSTEM MARKED CLEAN ***** BSDTest# fsck_ufs /dev/da1p6 ** /dev/da1p6 ** Last Mounted on /mnt/da1/tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 2 files, 2 used, 31244254 free (22 frags, 3905529 blocks, 0.0% fragmentation) ***** FILE SYSTEM MARKED CLEAN ***** BSDTest# And here comes the dmesg: BSDTest# dmesg -a Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-RELEASE-p1 #0: Thu Apr 28 15:23:57 UTC 2011 root@BSDTest.charlieroot.de:/usr/obj/usr/src/sys/GENERIC i386 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz (2574.73-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x10676 Family = 6 Model = 17 Stepping = 6 Features=0xfe3fbff Features2=0x80082201> AMD Features=0x20100000 AMD Features2=0x1 TSC: P-state invariant real memory = 1073741824 (1024 MB) avail memory = 1036292096 (988 MB) ACPI APIC Table: MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x10c0-0x10cf at device 7.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] pci0: at device 7.3 (no driver attached) pci0: at device 7.7 (no driver attached) vgapci0: port 0x10d0-0x10df mem 0xd4000000-0xd7ffffff,0xd8000000-0xd87fffff irq 16 at device 15.0 on pci0 mpt0: port 0x1400-0x14ff mem 0xd8820000-0xd883ffff,0xd8800000-0xd881ffff irq 17 at device 16.0 on pci0 mpt0: [ITHREAD] mpt0: MPI Version=1.2.0.0 pcib2: at device 17.0 on pci0 pci2: on pcib2 em0: port 0x2000-0x203f mem 0xd8920000-0xd893ffff,0xd8900000-0xd890ffff irq 18 at device 0.0 on pci2 em0: Memory Access and/or Bus Master bits were not set! em0: [FILTER] em0: Ethernet address: 00:0c:29:b6:be:a1 pcib3: at device 21.0 on pci0 pci3: on pcib3 pcib4: at device 21.1 on pci0 pci4: on pcib4 pcib5: at device 21.2 on pci0 pci5: on pcib5 pcib6: at device 21.3 on pci0 pci6: on pcib6 pcib7: at device 21.4 on pci0 pci7: on pcib7 pcib8: at device 21.5 on pci0 pci8: on pcib8 pcib9: at device 21.6 on pci0 pci9: on pcib9 pcib10: at device 21.7 on pci0 pci10: on pcib10 pcib11: at device 22.0 on pci0 pci11: on pcib11 pcib12: at device 22.1 on pci0 pci12: on pcib12 pcib13: at device 22.2 on pci0 pci13: on pcib13 pcib14: at device 22.3 on pci0 pci14: on pcib14 pcib15: at device 22.4 on pci0 pci15: on pcib15 pcib16: at device 22.5 on pci0 pci16: on pcib16 pcib17: at device 22.6 on pci0 pci17: on pcib17 pcib18: at device 22.7 on pci0 pci18: on pcib18 pcib19: at device 23.0 on pci0 pci19: on pcib19 pcib20: at device 23.1 on pci0 pci20: on pcib20 pcib21: at device 23.2 on pci0 pci21: on pcib21 pcib22: at device 23.3 on pci0 pci22: on pcib22 pcib23: at device 23.4 on pci0 pci23: on pcib23 pcib24: at device 23.5 on pci0 pci24: on pcib24 pcib25: at device 23.6 on pci0 pci25: on pcib25 pcib26: at device 23.7 on pci0 pci26: on pcib26 pcib27: at device 24.0 on pci0 pci27: on pcib27 pcib28: at device 24.1 on pci0 pci28: on pcib28 pcib29: at device 24.2 on pci0 pci29: on pcib29 pcib30: at device 24.3 on pci0 pci30: on pcib30 pcib31: at device 24.4 on pci0 pci31: on pcib31 pcib32: at device 24.5 on pci0 pci32: on pcib32 pcib33: at device 24.6 on pci0 pci33: on pcib33 pcib34: at device 24.7 on pci0 pci34: on pcib34 acpi_acad0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 pmtimer0 on isa0 orm0: at iomem 0xc0000-0xc7fff,0xca000-0xcafff,0xdc000-0xdffff,0xe0000-0xe3fff pnpid ORM0000 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 acpi_throttle0: on cpu0 Timecounter "TSC" frequency 2574732390 Hz quality 800 Timecounters tick every 10.000 msec acd0: CDROM at ata1-master UDMA33 da0 at mpt0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-2 device da0: 6.600MB/s transfers (16bit) da0: Command Queueing enabled da0: 16384MB (33554432 512 byte sectors: 255H 63S/T 2088C) da1 at mpt0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SCSI-5 device da1: 6.600MB/s transfers (16bit) da1: 76319MB (156301488 512 byte sectors: 255H 63S/T 9729C) Trying to mount root from ufs:/dev/da0s1a Setting hostuuid: 564d5d6b-3d26-341a-4124-f977dfb6bea1. Setting hostid: 0x0e78eea9. Entropy harvesting: interrupts ethernet point_to_point kickstart . Starting file system checks: /dev/da0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/da0s1a: clean, 768087 free (3655 frags, 95554 blocks, 0.4% fragmentation) /dev/da0s1g: DEFER FOR BACKGROUND CHECKING /dev/da0s1f: DEFER FOR BACKGROUND CHECKING /dev/da0s1d: DEFER FOR BACKGROUND CHECKING /dev/da0s1e: DEFER FOR BACKGROUND CHECKING Mounting local file systems: WARNING: /home was not properly dismounted WARNING: /tmp was not properly dismounted WARNING: /usr was not properly dismounted WARNING: /var was not properly dismounted . Setting hostname: BSDTest.charlieroot.de . Starting Network: lo0 em0. lo0: flags=8049 metric 0 mtu 16384 options=3 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 nd6 options=3 em0: flags=8843 metric 0 mtu 1500 options=9b ether 00:0c:29:b6:be:a1 inet6 fe80::20c:29ff:feb6:bea1%em0 prefixlen 64 tentative scopeid 0x1 nd6 options=3 media: Ethernet autoselect (1000baseT ) status: active add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 net.inet6.ip6.forwarding: 0 -> 0 plip0: flags=8810 metric 0 mtu 1500 lo0: flags=8049 metric 0 mtu 16384 options=3 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 net.inet6.ip6.accept_rtadv: 0 -> 1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 IPv4 mapped IPv6 address support=NO Starting devd. DHCPREQUEST on em0 to 255.255.255.255 port 67 DHCPACK from 192.168.124.254 bound to 192.168.124.69 -- renewal in 14400 seconds. Creating and/or trimming log files . Starting syslogd. ELF ldconfig path: /lib /usr/lib /usr/lib/compat a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Clearing /tmp (X related). Updating motd: . Configuring syscons: keymap blanktime . Starting sshd. Starting cron. Starting background file system checks in 60 seconds. Thu Apr 28 15:41:34 UTC 2011 Apr 28 15:42:38 BSDTest savecore: reboot after panic: ufs_dirbad: /mnt/da1: bad dir ino 117760 at offset 257: mangled entry BSDTest# uname -a FreeBSD BSDTest.charlieroot.de 8.2-RELEASE-p1 FreeBSD 8.2-RELEASE-p1 #0: Thu Apr 28 15:23:57 UTC 2011 root@BSDTest.charlieroot.de:/usr/obj/usr/src/sys/GENERIC i386 BSDTest#