Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 27 Nov 2013 13:11:35 -0800 (PST)
From:      Don Lewis <truckman@FreeBSD.org>
To:        kostikbel@gmail.com
Cc:        freebsd-current@FreeBSD.org
Subject:   Re: panic: double fault with 11.0-CURRENT r258504
Message-ID:  <201311272111.rARLBZk9042868@gw.catspoiler.org>
In-Reply-To: <20131127200050.GE59496@kib.kiev.ua>

next in thread | previous in thread | raw e-mail | index | archive | help
On 27 Nov, Konstantin Belousov wrote:
> On Wed, Nov 27, 2013 at 11:35:19AM -0800, Don Lewis wrote:
>> On 27 Nov, Konstantin Belousov wrote:
>> > On Wed, Nov 27, 2013 at 11:02:57AM -0800, Don Lewis wrote:
>> >> On 27 Nov, Konstantin Belousov wrote:
>> >> > On Wed, Nov 27, 2013 at 10:33:30AM -0800, Don Lewis wrote:
>> >> >> On 27 Nov, Konstantin Belousov wrote:
>> >> >> > On Wed, Nov 27, 2013 at 09:41:36AM -0800, Don Lewis wrote:
>> >> >> >> On 27 Nov, Konstantin Belousov wrote:
>> >> >> >> > On Wed, Nov 27, 2013 at 02:49:12AM -0800, Don Lewis wrote:
>> >> >> >> >> <http://people.freebsd.org/~truckman/doublefault2.JPG>;
>> >> >> >> > 
>> >> >> >> > What is the instruction at cpu_switch+0x9b ?
>> >> >> >> 
>> >> >> >> movl 0x8(%edx),%eax
>> >> >> > So it is line 176 in swtch.s. Is machine still in ddb, or did you
>> >> >> > obtained the core ? If yes, please print out the content of words at
>> >> >> > 0xe4f62bb0 + 4, +8 (*), +16. Please print the content of the word at
>> >> >> > address (*) + 8.
>> >> >> 
>> >> >> It is still in ddb.
>> >> >> 
>> >> >> <http://people.freebsd.org/~truckman/doublefault3.JPG>, though not in
>> >> >> the above order.
>> >> > Uhm, sorry, I mistyped the last part of the instructions.
>> >> > 
>> >> > The new thread pointer is 0xd2f4e000, there is nothing incriminating.
>> >> > Please print the word at 0xd2f4e000+0x254 == 0xd2f4e254, which would be
>> >> > the address of the new thread pcb. It is load from the pcb + 8 which
>> >> > faults.
>> >> 
>> >> 0xf3d44d60
>> > Again, the pointer looks fine, and its tail is 0xd60, which is correct for
>> > the pcb offset in the last page of the thread stack.
>> > 
>> > Please do 'show thread 0xd2f4e000' before trying below instructions.
>> 
>> Ok, see below:
>>  
>> > What happens if you try to read word at 0xf3d44d68 ?
>> 
>> Nothing bad ...
>> 
>> <http://people.freebsd.org/~truckman/doublefault4.JPG>;
>> 
> So the thread structure looks sane, the stack region is in place where
> it is supposed to be, all the gathered data looks self-consistent. And,
> the access to the faulted address from ddb does not fault.
> 
> Thread stacks can only be invalidated when the process is swapped out and
> kernel stack is written to swap.  Your thread flags indicate that it is
> in memory, and TDF_CANSWAP is not set.  I do not believe that our swapout
> code would invalidate stack mapping in such situation, otherwise we would
> have too many complaints already.
> 
> Just in case, do you use swap on this box ?

I do.

> And, as the last resort, I do understand that this sounds as giving up,
> do you monitor the temperature of the CPUs ? BTW, which CPUs are that,
> please show the cpu identification lines from the boot dmesg.

I don't monitor the temperature, but I do hear the CPU fan speed ramping
up and down when I'm building ports like this.  Even though I'm pretty
much keeping one core busy the whole time, the temperature must drop
enough at times to let the fan speed drop.

I can run math/mprime on this machine for a while to see if anything
shows up.  I also have a very similar machine (same motherboard but
different CPU) that I can move the drive over to and test.

Here's the full dmesg.boot:

Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #63 r258614M: Tue Nov 26 00:29:01 PST 2013
    dl@scratch.catspoiler.org:/usr/obj/usr/src/sys/GENERICSMB i386
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
WARNING: WITNESS option enabled, expect reduced performance.
CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 4800+ (2500.06-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x60fb1  Family = 0xf  Model = 0x6b  Stepping = 1
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x2001<SSE3,CX16>
  AMD Features=0xea500800<SYSCALL,NX,MMX+,FFXSR,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x11f<LAHF,CMP,SVM,ExtAPIC,CR8,Prefetch>
real memory  = 4294967296 (4096 MB)
avail memory = 3589320704 (3423 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <Nvidia AWRDACPI>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ACPI BIOS Warning (bug): Optional FADT field Pm2ControlBlock has zero address or length: 0x0000000000000000/0x1 (20130823/tbfadt-630)
ioapic0: Changing APIC ID to 2
ioapic0 <Version 1.1> irqs 0-23 on motherboard
random: <Software, Yarrow> initialized
kbd1 at kbdmux0
acpi0: <Nvidia AWRDACPI> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, dbdf0000 (3) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfeff0000-0xfeff03ff irq 0,8 on acpi0
device_attach: hpet0 attach returned 12
atrtc0: <AT realtime clock> port 0x70-0x73 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: Length mismatch for 4 range: 81 vs 7f
pci0: <ACPI PCI bus> on pcib0
pci0: <memory, RAM> at device 0.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
pci0: <serial bus, SMBus> at device 1.1 (no driver attached)
pci0: <memory, RAM> at device 1.2 (no driver attached)
ohci0: <OHCI (generic) USB controller> mem 0xfe02f000-0xfe02ffff irq 21 at device 2.0 on pci0
usbus0 on ohci0
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfe02e000-0xfe02e0ff irq 22 at device 2.1 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
ohci1: <OHCI (generic) USB controller> mem 0xfe02d000-0xfe02dfff irq 23 at device 4.0 on pci0
usbus2 on ohci1
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfe02c000-0xfe02c0ff irq 20 at device 4.1 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci1
atapci0: <nVidia nForce MCP67 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 6.0 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
hdac0: <NVIDIA MCP67 HDA Controller> mem 0xfe020000-0xfe023fff irq 21 at device 7.0 on pci0
pcib1: <ACPI PCI-PCI bridge> at device 8.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pci1: <serial bus, FireWire> at device 7.0 (no driver attached)
ahc0: <Adaptec 19160B Ultra160 SCSI adapter> port 0xcc00-0xccff mem 0xfd0fe000-0xfd0fefff irq 17 at device 9.0 on pci1
aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
ahci0: <NVIDIA MCP67 AHCI SATA controller> port 0x9f0-0x9f7,0xbf0-0xbf3,0x970-0x977,0xb70-0xb73,0xdc00-0xdc0f mem 0xfe026000-0xfe027fff irq 22 at device 9.0 on pci0
ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported
ahci0: quirks=0x200<NOAA>
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
nfe0: <NVIDIA nForce MCP67 Networking Adapter> port 0xd800-0xd807 mem 0xfe02b000-0xfe02bfff,0xfe02a000-0xfe02a0ff,0xfe029000-0xfe02900f irq 23 at device 10.0 on pci0
miibus0: <MII bus> on nfe0
e1000phy0: <Marvell 88E1116 Gigabit PHY> PHY 1 on miibus0
e1000phy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
nfe0: Ethernet address: 00:50:8d:9f:6d:e3
pcib2: <ACPI PCI-PCI bridge> at device 11.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 12.0 on pci0
pci3: <ACPI PCI bus> on pcib3
ahci1: <ASMedia ASM1061 AHCI SATA controller> port 0xac00-0xac07,0xa800-0xa803,0xa400-0xa407,0xa000-0xa003,0x9c00-0x9c1f mem 0xfdcff000-0xfdcff1ff irq 16 at device 0.0 on pci3
ahci1: AHCI v1.20 with 2 6Gbps ports, Port Multiplier supported
ahcich4: <AHCI channel> at channel 0 on ahci1
ahcich5: <AHCI channel> at channel 1 on ahci1
pcib4: <ACPI PCI-PCI bridge> at device 13.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 14.0 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 15.0 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 16.0 on pci0
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 17.0 on pci0
pci8: <ACPI PCI bus> on pcib8
vgapci0: <VGA-compatible display> mem 0xfb000000-0xfbffffff,0xe0000000-0xefffffff,0xfc000000-0xfcffffff irq 20 at device 18.0 on pci0
NVRM: The NVIDIA GeForce 7050 PV / nForce 630a GPU installed in this system is
NVRM:  supported through the NVIDIA 304.xx Legacy drivers. Please
NVRM:  visit http://www.nvidia.com/object/unix.html for more
NVRM:  information.  The 319.32 NVIDIA driver will ignore
NVRM:  this GPU.  Continuing probe...
acpi_tz0: <Thermal Zone> on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model MouseMan+, device ID 0
pmtimer0 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: parallel port not found.
powernow0: <PowerNow! K8> on cpu0
powernow1: <PowerNow! K8> on cpu1
Timecounters tick every 1.000 msec
xpt_config: xpt_create_path() failed for debug target 9:0:0, debugging disabled
hdacc0: <Realtek ALC888 HDA CODEC> at cad 0 on hdac0
hdaa0: <Realtek ALC888 Audio Function Group> at nid 1 on hdacc0
pcm0: <Realtek ALC888 (Analog 7.1+HP/2.0)> at nid 20,22,21,23,27 and 24,26,28,25 on hdaa0
pcm1: <Realtek ALC888 (Rear Digital)> at nid 30 on hdaa0
hdacc1: <NVIDIA MCP67 HDA CODEC> at cad 3 on hdac0
hdaa1: <NVIDIA MCP67 Audio Function Group> at nid 1 on hdacc1
pcm2: <NVIDIA MCP67 (Digital)> at nid 5 on hdaa1
random: unblocking device.
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
ugen0.1: <nVidia> at usbus0
uhub0: <nVidia OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen3.1: <nVidia> at usbus3
uhub1: <nVidia EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen2.1: <nVidia> at usbus2
uhub2: <nVidia OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen1.1: <nVidia> at usbus1
uhub3: <nVidia EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub0: 6 ports with 6 removable, self powered
uhub2: 6 ports with 6 removable, self powered
uhub1: 6 ports with 6 removable, self powered
uhub3: 6 ports with 6 removable, self powered
ada0 at ahcich1 bus 0 scbus4 target 0 lun 0
ada0: <SAMSUNG HD501LJ CR100-13> ATA-8 SATA 2.x device
ada0: Serial Number 401411CQC11039
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad6
da0 at ahc0 bus 0 scbus2 target 0 lun 0
da0: <SEAGATE ST336706LW 010A> Fixed Direct Access SCSI-3 device 
da0: Serial Number 3FD13QC700002243GFXL
da0: 160.000MB/s transfers (80.000MHz DT, offset 63, 16bit)
da0: Command Queueing enabled
da0: 35003MB (71687370 512 byte sectors: 255H 63S/T 4462C)
SMP: AP CPU #1 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/da0s1a [rw]...
lock order reversal:
 1st 0xe4d836e8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3081
 2nd 0xc9628400 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284
KDB: stack backtrace:
db_trace_self_wrapper(c11334ac,323a632e,a3438,ee33f790,c0b660ba,...) at db_trace_self_wrapper+0x2d/frame 0xee33f760
kdb_backtrace(c1136ff9,c9628400,c116a0dd,c8da3b58,c1169d15,...) at kdb_backtrace+0x30/frame 0xee33f7c8
witness_checkorder(c9628400,9,c1169d15,11c,0,...) at witness_checkorder+0xc8a/frame 0xee33f818
_sx_xlock(c9628400,0,c1169d15,11c,c973d1d0,...) at _sx_xlock+0x77/frame 0xee33f848
ufsdirhash_add(c973d1d0,ee33f940,6f38,ee33f8c8,ee33f8cc,...) at ufsdirhash_add+0x4a/frame 0xee33f878
ufs_direnter(c9b3d6a8,c9b3d58c,ee33f940,ee33fb64,e4d83a80,...) at ufs_direnter+0x63b/frame 0xee33f8f8
ufs_mkdir(ee33fc00,c12f9e87,c11630ce,c9b3d6a8,ee33fbfc,...) at ufs_mkdir+0x884/frame 0xee33fa98
VOP_MKDIR_APV(c13bf1a0,ee33fc00,ee33fb64,ee33fb90,88803020,...) at VOP_MKDIR_APV+0x124/frame 0xee33fac8
kern_mkdirat(c96e6c40,ffffff9c,88803020,0,1c0) at kern_mkdirat+0x236/frame 0xee33fc24
sys_mkdir(c96e6c40,ee33fcc8,14,c11314ed,7ba,...) at sys_mkdir+0x31/frame 0xee33fc40
syscall(ee33fd08) at syscall+0x2de/frame 0xee33fcfc
Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xee33fcfc
--- syscall (136, FreeBSD ELF32, sys_mkdir), eip = 0x88176097, esp = 0xbfbfd92c, ebp = 0xbfbfddcc ---
lock order reversal:
 1st 0xc9b3d6dc ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2101
 2nd 0xe4d836e8 bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_vnops.c:262
 3rd 0xc9b3d5c0 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2101
KDB: stack backtrace:
db_trace_self_wrapper(c11334ac,3236323a,ee33000a,c0a24bae,c9668390,...) at db_trace_self_wrapper+0x2d/frame 0xee33f2f8
kdb_backtrace(c1137012,c9b3d5c0,c111d2f2,c8da3af0,c11406f1,...) at kdb_backtrace+0x30/frame 0xee33f360
witness_checkorder(c9b3d5c0,9,c11406f1,835,c9b3d5e0,...) at witness_checkorder+0xc8a/frame 0xee33f3b0
__lockmgr_args(c9b3d5c0,80100,c9b3d5e0,0,0,...) at __lockmgr_args+0x842/frame 0xee33f484
ffs_lock(ee33f508,c8d8f1e0,c8d95360,c8d8f1e0,c8d95360,...) at ffs_lock+0x9a/frame 0xee33f4c4
VOP_LOCK1_APV(c13bf1a0,ee33f508,ee33f544,c0b65518,c13d40b0,...) at VOP_LOCK1_APV+0x104/frame 0xee33f4f0
_vn_lock(c9b3d58c,80100,c11406f1,835,c113f929,...) at _vn_lock+0xca/frame 0xee33f530
vget(c9b3d58c,80100,c96e6c40,57,0,...) at vget+0x74/frame 0xee33f568
vfs_hash_get(c96a5d20,27e405,80000,c96e6c40,ee33f658,...) at vfs_hash_get+0xfc/frame 0xee33f594
ffs_vgetf(c96a5d20,27e405,80000,ee33f658,1,...) at ffs_vgetf+0x47/frame 0xee33f5f0
softdep_sync_buf(c9b3d6a8,e4d83690,1,0,0,...) at softdep_sync_buf+0x2cc/frame 0xee33f668
ffs_syncvnode(c9b3d6a8,1,0,246,0,...) at ffs_syncvnode+0x287/frame 0xee33f6c0
ffs_truncate(c9b3d6a8,7800,0,880,c9681080,...) at ffs_truncate+0x6db/frame 0xee33f878
ufs_direnter(c9b3d6a8,c9b3d58c,ee33f940,ee33fb64,e4d83a80,...) at ufs_direnter+0x7ed/frame 0xee33f8f8
ufs_mkdir(ee33fc00,c12f9e87,c11630ce,c9b3d6a8,ee33fbfc,...) at ufs_mkdir+0x884/frame 0xee33fa98
VOP_MKDIR_APV(c13bf1a0,ee33fc00,ee33fb64,ee33fb90,88803020,...) at VOP_MKDIR_APV+0x124/frame 0xee33fac8
kern_mkdirat(c96e6c40,ffffff9c,88803020,0,1c0) at kern_mkdirat+0x236/frame 0xee33fc24
sys_mkdir(c96e6c40,ee33fcc8,14,c11314ed,7ba,...) at sys_mkdir+0x31/frame 0xee33fc40
syscall(ee33fd08) at syscall+0x2de/frame 0xee33fcfc
Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xee33fcfc
--- syscall (136, FreeBSD ELF32, sys_mkdir), eip = 0x88176097, esp = 0xbfbfd92c, ebp = 0xbfbfddcc ---
KLD linux_adobe.ko: depends on kernel - not available or version mismatch
lock order reversal:
 1st 0xc9665c28 filedesc structure (filedesc structure) @ /usr/src/sys/kern/kern_descrip.c:1203
 2nd 0xc9d75a30 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:4350
KDB: stack backtrace:
db_trace_self_wrapper(c11334ac,6b2f7379,2f6e7265,6e72656b,7365645f,...) at db_trace_self_wrapper+0x2d/frame 0xf3cfc8f8
kdb_backtrace(c1136ff9,c9d75a30,c111d2f2,c8da3af0,c11406f1,...) at kdb_backtrace+0x30/frame 0xf3cfc960
witness_checkorder(c9d75a30,9,c11406f1,10fe,0,...) at witness_checkorder+0xc8a/frame 0xf3cfc9b0
__lockmgr_args(c9d75a30,80400,c9d75a50,0,0,...) at __lockmgr_args+0x842/frame 0xf3cfca84
ffs_lock(f3cfcb08,f3cfcadc,c0b71c45,c9d75a30,f3cfcb08,...) at ffs_lock+0x9a/frame 0xf3cfcac4
VOP_LOCK1_APV(c13bf1a0,f3cfcb08,c9d58d00,0,c13d40b0,...) at VOP_LOCK1_APV+0x104/frame 0xf3cfcaf0
_vn_lock(c9d759fc,80400,c11406f1,10fe) at _vn_lock+0xca/frame 0xf3cfcb30
vfs_knllock(c9d759fc,0,c1127f22,778,c9d759fc,...) at vfs_knllock+0x29/frame 0xf3cfcb48
knlist_remove_kq(0,0) at knlist_remove_kq+0x87/frame 0xf3cfcb6c
knlist_remove(c9d58d00,c9b0b6c0,0,11,11,...) at knlist_remove+0x1f/frame 0xf3cfcb7c
filt_vfsdetach(c9b0b6c0,0,c1127f22,882,4,...) at filt_vfsdetach+0x2f/frame 0xf3cfcb9c
knote_fdclose(c9d6d000,11,c1127661,46a,0,...) at knote_fdclose+0xde/frame 0xf3cfcbd8
closefp(c9707888,c9d6d000,1,b1,c9665c00,...) at closefp+0x6b/frame 0xf3cfcc04
kern_close(c9d6d000,11) at kern_close+0xfb/frame 0xf3cfcc30
sys_close(c9d6d000,f3cfccc8,0,0,0,...) at sys_close+0x1a/frame 0xf3cfcc40
syscall(f3cfcd08) at syscall+0x2de/frame 0xf3cfccfc
Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf3cfccfc
--- syscall (6, FreeBSD ELF32, sys_close), eip = 0x8843f017, esp = 0xbf7fbf00, ebp = 0xbf7fbf18 ---
pid 66586 (iconvcap), uid 0: exited on signal 11 (core dumped)

Fatal double fault:
eip = 0xc0b14491
esp = 0xe4f61f54
ebp = 0xe4f62020
cpuid = 0; apic id = 00
panic: double fault
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper(c11334ac,2,10000000,c15a1cf0,c15a1ce8,...) at db_trace_self_wrapper+0x2d/frame 0xc15a1cb0
kdb_backtrace(c12f14df,0,c12f2b8a,c15a1d6c,0,...) at kdb_backtrace+0x30/frame 0xc15a1d18
vpanic(c15a1d6c,c15a1d84,c0fc159b,c12f2b8a,0,...) at vpanic+0x11f/frame 0xc15a1d54
panic(c12f2b8a,0,0,0,e4f62020,...) at panic+0x12/frame 0xc15a1d60
dblfault_handler() at dblfault_handler+0xab/frame 0xc15a1d60
--- trap 0x17, eip = 0xc0b14491, esp = 0xe4f61f54, ebp = 0xe4f62020 ---
kvprintf(c12f29a0,c0b15270,e4f62040,a,e4f6210c,...) at kvprintf+0x21/frame 0xe4f62020
vprintf(c12f29a0,e4f6210c,e4f6210c) at vprintf+0x7f/frame 0xe4f620ec
printf(c12f29a0,c,0,1050500,0,...) at printf+0x1b/frame 0xe4f62100
trap(e4f62254) at trap+0x18a/frame 0xe4f62248
calltrap() at calltrap+0x6/frame 0xe4f62248
--- trap 0xc, eip = 0xc0ac1887, esp = 0xe4f62294, ebp = 0xe4f622bc ---
__mtx_lock_spin_flags(c2490ffc,0,c11338b5,c7,c2426cac,...) at __mtx_lock_spin_flags+0x27/frame 0xe4f622bc
msgbuf_addstr(c2490fc8,ffffffff,e4f624f0,1,0,...) at msgbuf_addstr+0x5c/frame 0xe4f6236c
putchar(a,e4f624d0,a,0,7ffd3914,...) at putchar+0xd6/frame 0xe4f623dc
kvprintf(c12f29a0,c0b15270,e4f624d0,a,e4f6259c,...) at kvprintf+0xcb/frame 0xe4f624b0
vprintf(c12f29a0,e4f6259c,e4f6259c) at vprintf+0x7f/frame 0xe4f6257c
printf(c12f29a0,c,2,1,c13cc2ac,...) at printf+0x1b/frame 0xe4f62590
trap(e4f626e4) at trap+0x18a/frame 0xe4f626d8
calltrap() at calltrap+0x6/frame 0xe4f626d8
--- trap 0xc, eip = 0xc0ac1887, esp = 0xe4f62724, ebp = 0xe4f6274c ---
__mtx_lock_spin_flags(c2490ffc,0,c11338b5,c7,c0fe3ac7,...) at __mtx_lock_spin_flags+0x27/frame 0xe4f6274c
msgbuf_addstr(c2490fc8,ffffffff,e4f62980,1,1,...) at msgbuf_addstr+0x5c/frame 0xe4f627fc
putchar(a,e4f62960,a,0,c0afbcda,...) at putchar+0xd6/frame 0xe4f6286c
kvprintf(c12f29a0,c0b15270,e4f62960,a,e4f62a2c,...) at kvprintf+0xcb/frame 0xe4f62940
vprintf(c12f29a0,e4f62a2c,e4f62a2c) at vprintf+0x7f/frame 0xe4f62a0c
printf(c12f29a0,c,c11314ed,92c,e4f62a70,...) at printf+0x1b/frame 0xe4f62a20
trap(e4f62b70) at trap+0x18a/frame 0xe4f62b64
calltrap() at calltrap+0x6/frame 0xe4f62b64
--- trap 0xc, eip = 0xc0fbef73, esp = 0xe4f62bb0, ebp = 0xe4f62c00 ---
cpu_switch(c8f04000,0,608,1b6,136af,...) at cpu_switch+0x9b/frame 0xe4f62c00
mi_switch(608,0,c112f584,d3,9,...) at mi_switch+0x1c9/frame 0xe4f62c34
critical_exit(0,2,c11314ed,410,c141f108,...) at critical_exit+0xa4/frame 0xe4f62c50
sched_idletd(0,e4f62d08,c11286d4,3da,0,...) at sched_idletd+0x1d6/frame 0xe4f62ccc
fork_exit(c0afeb60,0,e4f62d08) at fork_exit+0x7f/frame 0xe4f62cf4
fork_trampoline() at fork_trampoline+0x8/frame 0xe4f62cf4
--- trap 0, eip = 0, esp = 0xe4f62d40, ebp = 0 ---
KDB: enter: panic
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #63 r258614M: Tue Nov 26 00:29:01 PST 2013
    dl@scratch.catspoiler.org:/usr/obj/usr/src/sys/GENERICSMB i386
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
WARNING: WITNESS option enabled, expect reduced performance.
CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 4800+ (2500.05-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x60fb1  Family = 0xf  Model = 0x6b  Stepping = 1
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x2001<SSE3,CX16>
  AMD Features=0xea500800<SYSCALL,NX,MMX+,FFXSR,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x11f<LAHF,CMP,SVM,ExtAPIC,CR8,Prefetch>
real memory  = 4294967296 (4096 MB)
avail memory = 3589320704 (3423 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <Nvidia AWRDACPI>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ACPI BIOS Warning (bug): Optional FADT field Pm2ControlBlock has zero address or length: 0x0000000000000000/0x1 (20130823/tbfadt-630)
ioapic0: Changing APIC ID to 2
ioapic0 <Version 1.1> irqs 0-23 on motherboard
random: <Software, Yarrow> initialized
kbd1 at kbdmux0
acpi0: <Nvidia AWRDACPI> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, dbdf0000 (3) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfeff0000-0xfeff03ff irq 0,8 on acpi0
device_attach: hpet0 attach returned 12
atrtc0: <AT realtime clock> port 0x70-0x73 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: Length mismatch for 4 range: 81 vs 7f
pci0: <ACPI PCI bus> on pcib0
pci0: <memory, RAM> at device 0.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
pci0: <serial bus, SMBus> at device 1.1 (no driver attached)
pci0: <memory, RAM> at device 1.2 (no driver attached)
ohci0: <OHCI (generic) USB controller> mem 0xfe02f000-0xfe02ffff irq 21 at device 2.0 on pci0
usbus0 on ohci0
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfe02e000-0xfe02e0ff irq 22 at device 2.1 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
ohci1: <OHCI (generic) USB controller> mem 0xfe02d000-0xfe02dfff irq 23 at device 4.0 on pci0
usbus2 on ohci1
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfe02c000-0xfe02c0ff irq 20 at device 4.1 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci1
atapci0: <nVidia nForce MCP67 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 6.0 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
hdac0: <NVIDIA MCP67 HDA Controller> mem 0xfe020000-0xfe023fff irq 21 at device 7.0 on pci0
pcib1: <ACPI PCI-PCI bridge> at device 8.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pci1: <serial bus, FireWire> at device 7.0 (no driver attached)
ahc0: <Adaptec 19160B Ultra160 SCSI adapter> port 0xcc00-0xccff mem 0xfd0fe000-0xfd0fefff irq 17 at device 9.0 on pci1
aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
ahci0: <NVIDIA MCP67 AHCI SATA controller> port 0x9f0-0x9f7,0xbf0-0xbf3,0x970-0x977,0xb70-0xb73,0xdc00-0xdc0f mem 0xfe026000-0xfe027fff irq 22 at device 9.0 on pci0
ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported
ahci0: quirks=0x200<NOAA>
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
nfe0: <NVIDIA nForce MCP67 Networking Adapter> port 0xd800-0xd807 mem 0xfe02b000-0xfe02bfff,0xfe02a000-0xfe02a0ff,0xfe029000-0xfe02900f irq 23 at device 10.0 on pci0
miibus0: <MII bus> on nfe0
e1000phy0: <Marvell 88E1116 Gigabit PHY> PHY 1 on miibus0
e1000phy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
nfe0: Ethernet address: 00:50:8d:9f:6d:e3
pcib2: <ACPI PCI-PCI bridge> at device 11.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 12.0 on pci0
pci3: <ACPI PCI bus> on pcib3
ahci1: <ASMedia ASM1061 AHCI SATA controller> port 0xac00-0xac07,0xa800-0xa803,0xa400-0xa407,0xa000-0xa003,0x9c00-0x9c1f mem 0xfdcff000-0xfdcff1ff irq 16 at device 0.0 on pci3
ahci1: AHCI v1.20 with 2 6Gbps ports, Port Multiplier supported
ahcich4: <AHCI channel> at channel 0 on ahci1
ahcich5: <AHCI channel> at channel 1 on ahci1
pcib4: <ACPI PCI-PCI bridge> at device 13.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 14.0 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 15.0 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 16.0 on pci0
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 17.0 on pci0
pci8: <ACPI PCI bus> on pcib8
vgapci0: <VGA-compatible display> mem 0xfb000000-0xfbffffff,0xe0000000-0xefffffff,0xfc000000-0xfcffffff irq 20 at device 18.0 on pci0
NVRM: The NVIDIA GeForce 7050 PV / nForce 630a GPU installed in this system is
NVRM:  supported through the NVIDIA 304.xx Legacy drivers. Please
NVRM:  visit http://www.nvidia.com/object/unix.html for more
NVRM:  information.  The 319.32 NVIDIA driver will ignore
NVRM:  this GPU.  Continuing probe...
acpi_tz0: <Thermal Zone> on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model MouseMan+, device ID 0
pmtimer0 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: parallel port not found.
powernow0: <PowerNow! K8> on cpu0
powernow1: <PowerNow! K8> on cpu1
Timecounters tick every 1.000 msec
xpt_config: xpt_create_path() failed for debug target 9:0:0, debugging disabled
hdacc0: <Realtek ALC888 HDA CODEC> at cad 0 on hdac0
hdaa0: <Realtek ALC888 Audio Function Group> at nid 1 on hdacc0
pcm0: <Realtek ALC888 (Analog 7.1+HP/2.0)> at nid 20,22,21,23,27 and 24,26,28,25 on hdaa0
pcm1: <Realtek ALC888 (Rear Digital)> at nid 30 on hdaa0
hdacc1: <NVIDIA MCP67 HDA CODEC> at cad 3 on hdac0
hdaa1: <NVIDIA MCP67 Audio Function Group> at nid 1 on hdacc1
pcm2: <NVIDIA MCP67 (Digital)> at nid 5 on hdaa1
random: unblocking device.
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
ugen0.1: <nVidia> at usbus0
uhub0: <nVidia OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen2.1: <nVidia> at usbus2
uhub1: <nVidia OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen1.1: <nVidia> at usbus1
uhub2: <nVidia EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ugen3.1: <nVidia> at usbus3
uhub3: <nVidia EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
uhub0: 6 ports with 6 removable, self powered
uhub1: 6 ports with 6 removable, self powered
uhub3: 6 ports with 6 removable, self powered
uhub2: 6 ports with 6 removable, self powered
ada0 at ahcich1 bus 0 scbus4 target 0 lun 0
ada0: <SAMSUNG HD501LJ CR100-13> ATA-8 SATA 2.x device
ada0: Serial Number 401411CQC11039
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad6
da0 at ahc0 bus 0 scbus2 target 0 lun 0
da0: <SEAGATE ST336706LW 010A> Fixed Direct Access SCSI-3 device 
da0: Serial Number 3FD13QC700002243GFXL
da0: 160.000MB/s transfers (80.000MHz DT, offset 63, 16bit)
da0: Command Queueing enabled
da0: 35003MB (71687370 512 byte sectors: 255H 63S/T 4462C)
SMP: AP CPU #1 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/da0s1a [rw]...
WARNING: / was not properly dismounted
lock order reversal:
 1st 0xe4d5d738 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3081
 2nd 0xc9684200 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284
KDB: stack backtrace:
db_trace_self_wrapper(c11334ac,323a632e,a3438,ee33f790,c0b660ba,...) at db_trace_self_wrapper+0x2d/frame 0xee33f760
kdb_backtrace(c1136ff9,c9684200,c116a0dd,c8da3b58,c1169d15,...) at kdb_backtrace+0x30/frame 0xee33f7c8
witness_checkorder(c9684200,9,c1169d15,11c,0,...) at witness_checkorder+0xc8a/frame 0xee33f818
_sx_xlock(c9684200,0,c1169d15,11c,c973d414,...) at _sx_xlock+0x77/frame 0xee33f848
ufsdirhash_add(c973d414,ee33f940,71e4,ee33f8c8,ee33f8cc,...) at ufsdirhash_add+0x4a/frame 0xee33f878
ufs_direnter(c9b5911c,c9b59000,ee33f940,ee33fb64,e4d5c9c0,...) at ufs_direnter+0x63b/frame 0xee33f8f8
ufs_mkdir(ee33fc00,c12f9e87,c11630ce,c9b5911c,ee33fbfc,...) at ufs_mkdir+0x884/frame 0xee33fa98
VOP_MKDIR_APV(c13bf1a0,ee33fc00,ee33fb64,ee33fb90,88803020,...) at VOP_MKDIR_APV+0x124/frame 0xee33fac8
kern_mkdirat(c96e7c40,ffffff9c,88803020,0,1c0) at kern_mkdirat+0x236/frame 0xee33fc24
sys_mkdir(c96e7c40,ee33fcc8,14,c11314ed,7ba,...) at sys_mkdir+0x31/frame 0xee33fc40
syscall(ee33fd08) at syscall+0x2de/frame 0xee33fcfc
Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xee33fcfc
--- syscall (136, FreeBSD ELF32, sys_mkdir), eip = 0x88176097, esp = 0xbfbfd92c, ebp = 0xbfbfddcc ---




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201311272111.rARLBZk9042868>