From owner-freebsd-hackers@freebsd.org Sun Oct 1 16:30:05 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DD784E28A58 for ; Sun, 1 Oct 2017 16:30:05 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0C8DB6590A for ; Sun, 1 Oct 2017 16:30:04 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id 940C367A06 for ; Sun, 1 Oct 2017 18:23:08 +0200 (CEST) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id z1yann5wZcHx for ; Sun, 1 Oct 2017 18:23:01 +0200 (CEST) Received: from mail.local.incore (fwintern.dmz [10.0.0.253]) by dss.incore.de (Postfix) with ESMTP id 1AF51679AA for ; Sun, 1 Oct 2017 18:23:01 +0200 (CEST) Received: from bsdmhs.longwitz (unknown [192.168.99.6]) by mail.local.incore (Postfix) with ESMTP id 937C4508D6 for ; Sun, 1 Oct 2017 18:23:00 +0200 (CEST) Message-ID: <59D11664.1060206@incore.de> Date: Sun, 01 Oct 2017 18:23:00 +0200 From: Andreas Longwitz User-Agent: Thunderbird 2.0.0.19 (X11/20090113) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: double fault on 10.3-Stable i386 during installworld Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sun, 01 Oct 2017 17:25:02 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Oct 2017 16:30:06 -0000 Hello hackers, On a server running 10.3-STABLE #2 r317936 i386 I got a double fault exception after a fresh boot in single user mode during installworld. I try to understand the magic of double fault and need help with this. The server did run with 8-Stable for many years without any problems and now has run fine also with 10.3 for a week until the double fault occured. The server has 4GB, one disk (amr) and I use UFS with SU. In the kernel for 10.3 I have NKPT=36 (default is 30) because of the problem I have described in Bug 216606. As far as I understand the debug output from console and the kerneldump, the CPU 2 gets the DOUBLE_FAULT trap (0x17). Therefore CPU 2 sends the ipi message T_NMI to the other CPU's, so they run the cpustop handler. I would like to know the reason for the DOUBLE_FAULT. What was the first exception ? My understanding is that during handling of the first exception there was another exception, what was this second exception ? The second exception was "changed" (by hardare ?) to DOUBLE_FAULT, is that correct ? Output serial console: Fatal double fault: eip = 0xc0bacac8 esp = 0xe437f000 ebp = 0xe437fafc cpuid = 2; apic id = 06 panic: double fault cpuid = 2 KDB: stack backtrace: db_trace_self_wrapper(c0cc3575,c0745e1b,10000000,fc,c0d2b220,...) at db_trace_self_wrapper+0x2d/frame 0xc0e62c50 kdb_backtrace(c0cf2f2c,2,c0cf3c11,c0e62d0c,2,...) at kdb_backtrace+0x30/frame 0xc0e62cb8 vpanic(c0cf3c11,c0e62d0c,c0e62d0c,c0e62d24,c0bc2bab,...) at vpanic+0x11b/frame 0xc0e62cec panic(c0cf3c11,6,6,2,e437fafc,...) at panic+0x1b/frame 0xc0e62d00 dblfault_handler() at dblfault_handler+0xab/frame 0xc0e62d00 --- trap 0x17, eip = 0xc0bacac8, esp = 0xe437f000, ebp = 0xe437fafc --- Xpage(c7ebd000,0,608,c08d12a5,c7ebd000,...) at Xpage/frame 0xe437fafc mi_switch(608,0,c0cc08f4,e2,c7ebd000,...) at mi_switch+0x145/frame 0xe437fb34 critical_exit(c7ebd000,0,2) at critical_exit+0x89/frame 0xe437fb50 ipi_bitmap_handler(8,28,28,c7f83200,0,...) at ipi_bitmap_handler+0x6b/frame 0xe437fb70 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x3d/frame 0xe437fb70 --- interrupt, eip = 0xc0ba87e5, esp = 0xe437fbb8, ebp = 0xe437fbb8 --- acpi_cpu_c1(16600000,16,a65c2a4c,0,c90786c0,...) at acpi_cpu_c1+0x5/frame 0xe437fbb8 acpi_cpu_idle(ffffffff,ffffffff,ffffffff,e437fc28,c0bb0e3a,...) at acpi_cpu_idle+0x15a/frame 0xe437fbf8 cpu_idle_acpi(ffffffff,ffffffff,c0e43484,c0e43488,c0e43494,...) at cpu_idle_acpi+0x3f/frame 0xe437fc0c cpu_idle(1,e437fc78,c0cc1cf3,a4c,0,...) at cpu_idle+0x9a/frame 0xe437fc28 sched_idletd(0,e437fce8,0,0,0,...) at sched_idletd+0x1dd/frame 0xe437fca4 fork_exit(c08f67e0,0,e437fce8) at fork_exit+0xa3/frame 0xe437fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe437fcd4 --- trap 0, eip = 0, esp = 0xe437fd20, ebp = 0 --- KDB: enter: panic [ thread pid 11 tid 100005 ] Stopped at kdb_enter+0x3d: movl $0,kdb_why db:0:kdb.enter.panic> watchdog No argument provided, disabling watchdog db:0:kdb.enter.panic> run ddbinfo db:1:ddbinfo> capture on db:1:on> run lockinfo db:2:lockinfo> show lockedvnods Locked vnodes db:2:lockedvnods> show lockchain thread 100005 (pid 11, idle: cpu2) can run db:2:lockchain> show sleepchain thread 100005 (pid 11, idle: cpu2) can run db:1:sleepchain> show pcpu cpuid = 2 dynamic pcpu = 0x235f1500 curthread = 0xc7ebd000: pid 11 "idle: cpu2" curpcb = 0xe437fd40 fpcurthread = none idlethread = 0xc7ebd000: tid 100005 "idle: cpu2" APIC ID = 6 currentldt = 0x50 db:1:pcpu> show allpcpu Current CPU: 2 cpuid = 0 dynamic pcpu = 0x6b2500 curthread = 0xc7f64360: pid 13 "g_up" curpcb = 0xe439dd40 fpcurthread = none idlethread = 0xc7ebd6c0: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 cpuid = 1 dynamic pcpu = 0x235ee500 curthread = 0xc7ebd360: pid 11 "idle: cpu1" curpcb = 0xe437cd40 fpcurthread = none idlethread = 0xc7ebd360: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 cpuid = 2 dynamic pcpu = 0x235f1500 curthread = 0xc7ebd000: pid 11 "idle: cpu2" curpcb = 0xe437fd40 fpcurthread = none idlethread = 0xc7ebd000: tid 100005 "idle: cpu2" APIC ID = 6 currentldt = 0x50 cpuid = 3 dynamic pcpu = 0x235f4500 curthread = 0xc90786c0: pid 12498 "sh" curpcb = 0xf0b85d40 fpcurthread = none idlethread = 0xc7ebca20: tid 100006 "idle: cpu3" APIC ID = 7 currentldt = 0x50 db:1:allpcpu> bt Tracing pid 11 tid 100005 td 0xc7ebd000 kdb_enter(c0cc0398,c0cc0398,c0cf3c11,c0e62d0c,2,...) at kdb_enter+0x3d/frame 0xc0e62cb8 vpanic(c0cf3c11,c0e62d0c,c0e62d0c,c0e62d24,c0bc2bab,...) at vpanic+0x13b/frame 0xc0e62cec panic(c0cf3c11,6,6,2,e437fafc,...) at panic+0x1b/frame 0xc0e62d00 dblfault_handler() at dblfault_handler+0xab/frame 0xc0e62d00 --- trap 0x17, eip = 0xc0bacac8, esp = 0xe437f000, ebp = 0xe437fafc --- Xpage(c7ebd000,0,608,c08d12a5,c7ebd000,...) at Xpage/frame 0xe437fafc mi_switch(608,0,c0cc08f4,e2,c7ebd000,...) at mi_switch+0x145/frame 0xe437fb34 critical_exit(c7ebd000,0,2) at critical_exit+0x89/frame 0xe437fb50 ipi_bitmap_handler(8,28,28,c7f83200,0,...) at ipi_bitmap_handler+0x6b/frame 0xe437fb70 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x3d/frame 0xe437fb70 --- interrupt, eip = 0xc0ba87e5, esp = 0xe437fbb8, ebp = 0xe437fbb8 --- acpi_cpu_c1(16600000,16,a65c2a4c,0,c90786c0,...) at acpi_cpu_c1+0x5/frame 0xe437fbb8 acpi_cpu_idle(ffffffff,ffffffff,ffffffff,e437fc28,c0bb0e3a,...) at acpi_cpu_idle+0x15a/frame 0xe437fbf8 cpu_idle_acpi(ffffffff,ffffffff,c0e43484,c0e43488,c0e43494,...) at cpu_idle_acpi+0x3f/frame 0xe437fc0c cpu_idle(1,e437fc78,c0cc1cf3,a4c,0,...) at cpu_idle+0x9a/frame 0xe437fc28 sched_idletd(0,e437fce8,0,0,0,...) at sched_idletd+0x1dd/frame 0xe437fca4 fork_exit(c08f67e0,0,e437fce8) at fork_exit+0xa3/frame 0xe437fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe437fcd4 --- trap 0, eip = 0, esp = 0xe437fd20, ebp = 0 --- db:1:bt> ps pid ppid pgrp uid state wmesg wchan cmd 12498 12262 475 0 R+ CPU 3 sh 12262 798 475 0 S+ wait 0xc9d31648 make 798 756 475 0 S+ wait 0xc90da000 sh 756 620 475 0 S+ wait 0xc8cfcc90 make 620 619 475 0 S+ wait 0xc8cfc96c make 619 480 475 0 S+ wait 0xc90da648 sh 480 475 475 0 S+ wait 0xc8646324 make 475 474 475 0 Ss+ wait 0xc87e4c90 make 474 315 21 0 S+ select 0xc8683664 script 315 309 21 0 S+ wait 0xc8d0196c sh 309 21 21 0 S+ wait 0xc87e6000 sh 171 1 171 0 Ss pause 0xc87e59d0 adjkerntz 137 136 21 0 S+ nanslp 0xc0e363f9 wait4sighup 136 1 21 0 S+ wait 0xc87b496c sh 21 1 21 0 Ss+ wait 0xc864496c sh 20 0 0 0 DL syncer 0xc0e55744 [syncer] 19 0 0 0 DL vlruwt 0xc8645000 [vnlru] 18 0 0 0 DL (threaded) [bufdaemon] 100082 D sdflush 0xc8abd884 [/var worker] 100081 D biord 0xe41ea6c0 [/usr worker] 100080 D sdflush 0xc7f84a84 [/home worker] 100075 D sdflush 0xc86c0884 [/ worker] 100062 D psleep 0xc0e54e84 [bufdaemon] 17 0 0 0 DL pgzero 0xc0e5d628 [pagezero] 16 0 0 0 DL pollid 0xc0e34fb0 [idlepoll] 9 0 0 0 DL psleep 0xc0e5d344 [vmdaemon] 8 0 0 0 DL (threaded) [pagedaemon] 100065 D umarcl 0xc0e5cf24 [uma] 100058 D psleep 0xc0eaba84 [pagedaemon] 7 0 0 0 DL ipmireq 0xc802d388 [ipmi0: kcs] 6 0 0 0 DL waiting_ 0xc0ea8488 [sctp_iterator] 5 0 0 0 DL pftm 0xc0a7f0a0 [pf purge] 4 0 0 0 DL (threaded) [ctl] 100049 D - 0xc804c584 [thresh] 100048 D - 0xc804b100 [lun] 100047 D - 0xc804b580 [work0] 3 0 0 0 DL - 0xc7f8143c [fdc0] 15 0 0 0 DL (threaded) [usb] 100041 D - 0xc806bddc [usbus1] 100040 D - 0xc806bdac [usbus1] 100039 D - 0xc806bd7c [usbus1] 100038 D - 0xc806bd4c [usbus1] 100037 D - 0xc806bd1c [usbus1] 100035 D - 0xc8018ddc [usbus0] 100034 D - 0xc8018dac [usbus0] 100033 D - 0xc8018d7c [usbus0] 100032 D - 0xc8018d4c [usbus0] 100031 D - 0xc8018d1c [usbus0] 2 0 0 0 DL (threaded) [cam] 100056 D - 0xc0d7b8ac [scanner] 100025 D - 0xc0d7ba00 [doneq0] 14 0 0 0 DL - 0xc0d98d20 [rand_harvestq] 13 0 0 0 RL (threaded) [geom] 100015 D - 0xc0ea3554 [g_down] 100014 Run CPU 0 [g_up] 100013 D - 0xc0ea354c [g_event] 12 0 0 0 RL (threaded) [intr] 100054 I [swi1: pfsync] 100052 I [swi1: pf send] 100046 I [swi0: uart uart] 100044 I [irq1: atkbd0] 100043 I [irq15: ata1] 100042 I [irq14: ata0] 100036 I [irq19: uhci1] 100030 Run CPU 255 [irq16: uhci0] 100029 I [irq24: amr0] 100024 I [swi5: fast taskq] 100022 I [swi6: Giant taskq] 100021 I [swi6: task queue] 100012 I [swi1: netisr 0] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 I [swi4: clock] 100007 I [swi3: vm] 11 0 0 0 RL (threaded) [idle] 100006 CanRun [idle: cpu3] 100005 CanRun [idle: cpu2] 100004 Run CPU 1 [idle: cpu1] 100003 CanRun [idle: cpu0] 1 0 1 0 SLs wait 0xc7eba324 [init] 10 0 0 0 DL audit_wo 0xc0eaa384 [audit] 0 0 0 0 DLs (threaded) [kernel] 100055 D - 0xc7d70100 [CAM taskq] 100050 D - 0xc8079a80 [mca taskq] 100028 D - 0xc7ec2c00 [em1 taskq] 100027 D - 0xc7fe8500 [em0 taskq] 100026 D - 0xc7feaf00 [kqueue taskq] 100023 D - 0xc7d70380 [thread taskq] 100020 D - 0xc7d70600 [acpi_task_2] 100019 D - 0xc7d70600 [acpi_task_1] 100018 D - 0xc7d70600 [acpi_task_0] 100016 D - 0xc7d70d00 [firmware taskq] 100000 D swapin 0xc0ea35a4 [swapper] db:1:ps> show mount 0xc8728d20 /dev/amrd0p3 on / (ufs) 0xc8729000 devfs on /dev (devfs) 0xc89a42a0 /dev/amrd0p6 on /home (ufs) 0xc89a4000 /dev/amrd0p5 on /usr (ufs) 0xc89a3d20 /dev/amrd0p4 on /var (ufs) 0xc89a3a80 tmpfs on /tmp (tmpfs) 0xc89a37e0 tmpfs on /tmp1 (tmpfs) More info: show mount db:1:mount> show page cnt.v_free_count: 870537 cnt.v_cache_count: 19900 cnt.v_inactive_count: 38874 cnt.v_active_count: 8050 cnt.v_wire_count: 24969 cnt.v_free_reserved: 1303 cnt.v_free_min: 6113 cnt.v_free_target: 20543 cnt.v_cache_min: 0 cnt.v_inactive_target: 30814 db:1:page> show thread Thread 100005 at 0xc7ebd000: proc (pid 11): 0xc7eba000 name: idle: cpu2 stack: 0xe437e000-0xe437ffff flags: 0x40024 pflags: 0x200000 state: CAN RUN priority: 255 container lock: sched lock 2 (0xc0e43400) db:1:thread> alltrace (some interesting threads) Tracing command sh pid 12498 tid 100099 td 0xc90786c0 cpustop_handler(3,13,f0b85830,c0bc1a2a,c90786c0,...) at cpustop_handler+0x25/frame 0xf0b856e0 ipi_nmi_handler(c90786c0,c87b9b40,f0b85728,c0b52c51,c87b9b40,...) at ipi_nmi_handler+0x37/frame 0xf0b856f0 trap(f0b8583c) at trap+0x3a/frame 0xf0b85830 calltrap() at calltrap+0x6/frame 0xf0b85830 --- trap 0x13, eip = 0xc0bb6177, esp = 0xf0b8587c, ebp = 0xf0b858a0 --- smp_tlb_shootdown(0) at smp_tlb_shootdown+0xc7/frame 0xf0b858a0 smp_invlpg(ed800000,0,ed800000,bffb6000,0,...) at smp_invlpg+0x24/frame 0xf0b858ac pmap_remove_pte(ed800000,f0b85900,c9d4c360,c9d4c698,f0b85908,...) at pmap_remove_pte+0x37/frame 0xf0b858c8 pmap_remove(c0eb7174,ed800000,ed841000,c0d53468,f0b85970,...) at pmap_remove+0x217/frame 0xf0b85918 vm_map_delete(c17ac2d0,ed800000,ed841000,f0b85bd0,c90786c0,...) at vm_map_delete+0x1bf/frame 0xf0b85978 kmap_free_wakeup(c17ac2d0,ed800000,40400,2d2,ed800000,...) at kmap_free_wakeup+0x4c/frame 0xf0b85998 kern_execve(c90786c0,f0b85bd0,0,80dcf88,288140c0,...) at kern_execve+0xeed/frame 0xf0b85bb0 sys_execve(c90786c0,f0b85ca8,bfbecc58,8048e82,c90df324,...) at sys_execve+0x6d/frame 0xf0b85c10 syscall(f0b85ce8) at syscall+0x5eb/frame 0xf0b85cdc Xint0x80_syscall() at Xint0x80_syscall+0x2e/frame 0xf0b85cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x280660b0, esp = 0xbfbfea4c, ebp = 0 --- Tracing command bufdaemon pid 18 tid 100081 td 0xc876c000 sched_switch(c876c000,0,104,0,1,...) at sched_switch+0x74f/frame 0xf0b34984 mi_switch(104,0,10222,c87e2760,c876c000,e41ea6c0) at mi_switch+0x145/frame 0xf0b349bc sleepq_switch(c876c000,0,c0cc49d2,277,11222,...) at sleepq_switch+0x15b/frame 0xf0b349e4 sleepq_wait(e41ea6c0,5c,c0cc910f,0,0,...) at sleepq_wait+0x3f/frame 0xf0b34a10 _sleep(e41ea6c0,c7d926b4,5c,c0cc910f,0,...) at _sleep+0x28f/frame 0xf0b34a58 bwait(e41ea6c0,5c,c0cc910f,0,0,...) at bwait+0xd1/frame 0xf0b34aa0 breadn_flags(c87bc58c,eea3c0,0,8000,0,...) at breadn_flags+0x140/frame 0xf0b34ad0 indir_trunc(0,fffffff4,ffffffff,c876c000,0,...) at indir_trunc+0x132/frame 0xf0b34b90 handle_workitem_freeblocks(0,f0b34c18,2,f0b34c1c,c08c37d4,...) at handle_workitem_freeblocks+0x328/frame 0xf0b34be0 process_worklist_item(0,0,0,1000000,59cbc86f,...) at process_worklist_item+0x276/frame 0xf0b34c30 softdep_process_worklist(c8abda84,c8abda00,54,c0ce279a,7fffff6c,...) at softdep_process_worklist+0xb9/frame 0xf0b34c64 softdep_flush(c89a4000,f0b34ce8,c0b10420,0,f0b34cd0,...) at softdep_flush+0x12e/frame 0xf0b34ca4 fork_exit(c0b10420,c89a4000,f0b34ce8) at fork_exit+0xa3/frame 0xf0b34cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0b34cd4 --- trap 0, eip = 0, esp = 0xf0b34d20, ebp = 0 --- Tracing command bufdaemon pid 18 tid 100080 td 0xc876c360 sched_switch(c876c360,0,104,501,0,...) at sched_switch+0x74f/frame 0xf0b31b78 mi_switch(104,0,501,0,f0b31c1c,c7f84a84) at mi_switch+0x145/frame 0xf0b31bb0 sleepq_switch(c876c360,0,c0cc49d2,29a,501,...) at sleepq_switch+0x15b/frame 0xf0b31bd8 sleepq_timedwait(c7f84a84,54,0,0,0,...) at sleepq_timedwait+0x49/frame 0xf0b31c1c _sleep(c7f84a84,c7f84a00,54,c0ce279a,7fffff6c,...) at _sleep+0x266/frame 0xf0b31c64 softdep_flush(c89a42a0,f0b31ce8,c0b10420,0,f0b31cd0,...) at softdep_flush+0x202/frame 0xf0b31ca4 fork_exit(c0b10420,c89a42a0,f0b31ce8) at fork_exit+0xa3/frame 0xf0b31cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0b31cd4 --- trap 0, eip = 0, esp = 0xf0b31d20, ebp = 0 --- Tracing command bufdaemon pid 18 tid 100075 td 0xc876c6c0 sched_switch(c876c6c0,0,104,501,0,...) at sched_switch+0x74f/frame 0xf0b21b78 mi_switch(104,0,501,0,f0b21c1c,c86c0884) at mi_switch+0x145/frame 0xf0b21bb0 sleepq_switch(c876c6c0,0,c0cc49d2,29a,501,...) at sleepq_switch+0x15b/frame 0xf0b21bd8 sleepq_timedwait(c86c0884,54,0,0,0,...) at sleepq_timedwait+0x49/frame 0xf0b21c1c _sleep(c86c0884,c86c0800,54,c0ce279a,7fffff6c,...) at _sleep+0x266/frame 0xf0b21c64 softdep_flush(c8728d20,f0b21ce8,c0b10420,0,f0b21cd0,...) at softdep_flush+0x202/frame 0xf0b21ca4 fork_exit(c0b10420,c8728d20,f0b21ce8) at fork_exit+0xa3/frame 0xf0b21cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0b21cd4 --- trap 0, eip = 0, esp = 0xf0b21d20, ebp = 0 --- Tracing command bufdaemon pid 18 tid 100062 td 0xc8622a20 sched_switch(c8622a20,0,104,501,0,...) at sched_switch+0x74f/frame 0xf0ae7b80 mi_switch(104,0,501,0,f0ae7c24,c0e54e84) at mi_switch+0x145/frame 0xf0ae7bb8 sleepq_switch(c8622a20,0,c0cc49d2,29a,501,...) at sleepq_switch+0x15b/frame 0xf0ae7be0 sleepq_timedwait(c0e54e84,54,0,0,0,...) at sleepq_timedwait+0x49/frame 0xf0ae7c24 _sleep(c0e54e84,c0e54e00,54,c0cc95b2,fffffed8,...) at _sleep+0x266/frame 0xf0ae7c6c buf_daemon(0,f0ae7ce8,0,0,0,...) at buf_daemon+0xac/frame 0xf0ae7ca4 fork_exit(c096b3d0,0,f0ae7ce8) at fork_exit+0xa3/frame 0xf0ae7cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0ae7cd4 --- trap 0, eip = 0, esp = 0xf0ae7d20, ebp = 0 --- Tracing command pagezero pid 17 tid 100061 td 0xc8623000 sched_switch(c8623000,0,104,501,0,...) at sched_switch+0x74f/frame 0xf0ae4b88 mi_switch(104,0,501,12,f0ae4c2c,c0e5d628) at mi_switch+0x145/frame 0xf0ae4bc0 sleepq_switch(c8623000,0,c0cc49d2,29a,501,...) at sleepq_switch+0x15b/frame 0xf0ae4be8 sleepq_timedwait(c0e5d628,0,12b,0,0,...) at sleepq_timedwait+0x49/frame 0xf0ae4c2c _sleep(c0e5d628,c0eaba00,0,c0ce6b6a,fffea520,...) at _sleep+0x266/frame 0xf0ae4c74 vm_pagezero(0,f0ae4ce8,0,0,0,...) at vm_pagezero+0xe2/frame 0xf0ae4ca4 fork_exit(c0b75b50,0,f0ae4ce8) at fork_exit+0xa3/frame 0xf0ae4cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0ae4cd4 --- trap 0, eip = 0, esp = 0xf0ae4d20, ebp = 0 --- Tracing command idlepoll pid 16 tid 100060 td 0xc8623360 sched_switch(c8623360,0,104,501,0,...) at sched_switch+0x74f/frame 0xf0ae1b70 mi_switch(104,0,501,0,f0ae1c18,c0e34fb0) at mi_switch+0x145/frame 0xf0ae1ba8 sleepq_switch(c8623360,0,c0cc49d2,29a,501,...) at sleepq_switch+0x15b/frame 0xf0ae1bd0 sleepq_timedwait(c0e34fb0,0,2,0,0,...) at sleepq_timedwait+0x49/frame 0xf0ae1c18 _sleep(c0e34fb0,0,0,c0cbf329,fffffc88,...) at _sleep+0x266/frame 0xf0ae1c60 poll_idle(0,f0ae1ce8,0,0,0,...) at poll_idle+0x135/frame 0xf0ae1ca4 fork_exit(c08b1a70,0,f0ae1ce8) at fork_exit+0xa3/frame 0xf0ae1cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0ae1cd4 --- trap 0, eip = 0, esp = 0xf0ae1d20, ebp = 0 --- Tracing command vmdaemon pid 9 tid 100059 td 0xc86236c0 sched_switch(c86236c0,0,104,0,0,...) at sched_switch+0x74f/frame 0xf0adeb64 mi_switch(104,0,0,0,c86236c0,c0e5d344) at mi_switch+0x145/frame 0xf0adeb9c sleepq_switch(c86236c0,0,c0cc49d2,277,0,...) at sleepq_switch+0x15b/frame 0xf0adebc4 sleepq_wait(c0e5d344,74,c0cc95b2,0,0,...) at sleepq_wait+0x3f/frame 0xf0adebf0 _sleep(c0e5d344,c0e5d32c,74,c0cc95b2,0,...) at _sleep+0x28f/frame 0xf0adec38 vm_daemon(0,f0adece8,0,0,0,...) at vm_daemon+0xd9/frame 0xf0adeca4 fork_exit(c0b6e070,0,f0adece8) at fork_exit+0xa3/frame 0xf0adecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0adecd4 --- trap 0, eip = 0, esp = 0xf0aded20, ebp = 0 --- Tracing command pagedaemon pid 8 tid 100065 td 0xc86a8360 sched_switch(c86a8360,0,104,0,0,...) at sched_switch+0x74f/frame 0xf0af0b94 mi_switch(104,0,0,0,c86a8360,c0e5cf24) at mi_switch+0x145/frame 0xf0af0bcc sleepq_switch(c86a8360,0,c0cc49d2,277,0,...) at sleepq_switch+0x15b/frame 0xf0af0bf4 sleepq_wait(c0e5cf24,54,c0ce48f9,0,0,...) at sleepq_wait+0x3f/frame 0xf0af0c20 _sleep(c0e5cf24,c0e5cf10,54,c0ce48f9,0,...) at _sleep+0x28f/frame 0xf0af0c68 uma_reclaim_worker(0,f0af0ce8,0,0,0,...) at uma_reclaim_worker+0xb6/frame 0xf0af0ca4 fork_exit(c0b50360,0,f0af0ce8) at fork_exit+0xa3/frame 0xf0af0cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0af0cd4 --- trap 0, eip = 0, esp = 0xf0af0d20, ebp = 0 --- Tracing command pagedaemon pid 8 tid 100058 td 0xc8623a20 sched_switch(c8623a20,0,104,501,0,...) at sched_switch+0x74f/frame 0xf0adbb48 mi_switch(104,0,501,0,f0adbbec,c0eaba84) at mi_switch+0x145/frame 0xf0adbb80 sleepq_switch(c8623a20,0,c0cc49d2,29a,501,...) at sleepq_switch+0x15b/frame 0xf0adbba8 sleepq_timedwait(c0eaba84,54,0,0,0,...) at sleepq_timedwait+0x49/frame 0xf0adbbec _sleep(c0eaba84,c0eaba00,54,c0cc95b2,fffffed8,...) at _sleep+0x266/frame 0xf0adbc34 vm_pageout(0,f0adbce8,0,0,0,...) at vm_pageout+0x1fb/frame 0xf0adbca4 fork_exit(c0b6eb70,0,f0adbce8) at fork_exit+0xa3/frame 0xf0adbcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0adbcd4 --- trap 0, eip = 0, esp = 0xf0adbd20, ebp = 0 --- Tracing command geom pid 13 tid 100015 td 0xc7f64000 sched_switch(c7f64000,0,104,c80418ec,0,...) at sched_switch+0x74f/frame 0xe43a0b80 mi_switch(104,0,c862d400,c7f64000,c7f64000,c0ea3554) at mi_switch+0x145/frame 0xe43a0bb8 sleepq_switch(c7f64000,0,c0cc49d2,277,c8695be0,...) at sleepq_switch+0x15b/frame 0xe43a0be0 sleepq_wait(c0ea3554,5c,c0cb46b0,0,0,...) at sleepq_wait+0x3f/frame 0xe43a0c0c _sleep(c0ea3554,c0e327e8,25c,c0cb46b0,0,...) at _sleep+0x28f/frame 0xe43a0c54 g_io_schedule_down(c7f64000,5c,c0cb6555,6b,e43a0cd4,...) at g_io_schedule_down+0x5c/frame 0xe43a0c8c g_down_procbody(0,e43a0ce8,0,0,0,...) at g_down_procbody+0x6d/frame 0xe43a0ca4 fork_exit(c0849590,0,e43a0ce8) at fork_exit+0xa3/frame 0xe43a0cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe43a0cd4 --- trap 0, eip = 0, esp = 0xe43a0d20, ebp = 0 --- Tracing command geom pid 13 tid 100014 td 0xc7f64360 cpustop_handler(0,13,e439da2c,c0bc1a2a,0,...) at cpustop_handler+0x25/frame 0xe439d8e0 ipi_nmi_handler(0,c7f64360,e439d960,0,0,...) at ipi_nmi_handler+0x37/frame 0xe439d8f0 trap(e439da38) at trap+0x3a/frame 0xe439da2c calltrap() at calltrap+0x6/frame 0xe439da2c --- trap 0x13, eip = 0xc08ad9a2, esp = 0xe439da78, ebp = 0xe439dadc --- _mtx_lock_spin_cookie(c0ea7c88,c7f64360,0,0,0,...) at _mtx_lock_spin_cookie+0x152/frame 0xe439dadc smp_tlb_shootdown(e7fb4000) at smp_tlb_shootdown+0x8d/frame 0xe439db08 smp_invlpg_range(e7fb0000,e7fb4000,e4114910,0,e439db70,...) at smp_invlpg_range+0x23/frame 0xe439db14 pmap_qremove(e7fb0000,4,c7ebd6c0,c0e51a0c,400000,...) at pmap_qremove+0x69/frame 0xe439db2c allocbuf(e4114910,0,c0917e42,c08d12a5,c7f64360,...) at allocbuf+0x5a3/frame 0xe439db70 brelse(e4114910,c872aee0,e4181ea0,e439dc10,c0b2ffed,...) at brelse+0x1a3/frame 0xe439dbcc bufdone(e4114910,80140,0,10028000,c872aee0,...) at bufdone+0x60/frame 0xe439dbe0 ffs_backgroundwritedone(e4114910,c7f64360,c8658f70,e439dc54,c084dac7,...) at ffs_backgroundwritedone+0x13d/frame 0xe439dc10 bufdone(e4114910,c0847ac3,c86d3428,c86c44c0,c7f64360,c8683080,e4114910) at bufdone+0x56/frame 0xe439dc24 g_vfs_done(c86d3428,c0e32808,25c,c0cb46b0,0,...) at g_vfs_done+0x2a7/frame 0xe439dc54 g_io_schedule_up(c7f64360,5c,c0cb6555,5e,e439dcd4,...) at g_io_schedule_up+0x1f2/frame 0xe439dc8c g_up_procbody(0,e439dce8,0,0,0,...) at g_up_procbody+0x6d/frame 0xe439dca4 fork_exit(c0849520,0,e439dce8) at fork_exit+0xa3/frame 0xe439dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe439dcd4 --- trap 0, eip = 0, esp = 0xe439dd20, ebp = 0 --- Tracing command geom pid 13 tid 100013 td 0xc7ebb000 sched_switch(c7ebb000,0,104,0,c87f1d58,...) at sched_switch+0x74f/frame 0xe439ab94 mi_switch(104,0,c08d1392,c87f1a20,c7ebb000,c0ea354c) at mi_switch+0x145/frame 0xe439abcc sleepq_switch(c7ebb000,0,c0cc49d2,277,e439ac18,...) at sleepq_switch+0x15b/frame 0xe439abf4 sleepq_wait(c0ea354c,5c,c0cb46b0,0,0,...) at sleepq_wait+0x3f/frame 0xe439ac20 _sleep(c0ea354c,c0e327c0,25c,c0cb46b0,0,...) at _sleep+0x28f/frame 0xe439ac68 g_run_events(0,e439ace8,0,0,0,...) at g_run_events+0x62/frame 0xe439aca4 fork_exit(c08494b0,0,e439ace8) at fork_exit+0xa3/frame 0xe439acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe439acd4 --- trap 0, eip = 0, esp = 0xe439ad20, ebp = 0 --- Tracing command idle pid 11 tid 100006 td 0xc7ebca20 sched_switch(c7ebca20,0,608,e4382b48,c0880793,...) at sched_switch+0x74f/frame 0xe4382afc mi_switch(608,0,c0cc08f4,e2,c7ebca20,...) at mi_switch+0x145/frame 0xe4382b34 critical_exit(c7ebca20,0,3) at critical_exit+0x89/frame 0xe4382b50 ipi_bitmap_handler(8,28,28,c7f82e00,0,...) at ipi_bitmap_handler+0x6b/frame 0xe4382b70 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x3d/frame 0xe4382b70 --- interrupt, eip = 0xc0ba87e5, esp = 0xe4382bb8, ebp = 0xe4382bb8 --- acpi_cpu_c1(13000000,16,a65c2a4c,0,c7f64360,...) at acpi_cpu_c1+0x5/frame 0xe4382bb8 acpi_cpu_idle(ffffffff,ffffffff,ffffffff,e4382c28,c0bb0e3a,...) at acpi_cpu_idle+0x15a/frame 0xe4382bf8 cpu_idle_acpi(ffffffff,ffffffff,c0e43b84,c0e43b88,c0e43b94,...) at cpu_idle_acpi+0x3f/frame 0xe4382c0c cpu_idle(1,e4382c78,c0cc1cf3,a4c,0,...) at cpu_idle+0x9a/frame 0xe4382c28 sched_idletd(0,e4382ce8,0,0,0,...) at sched_idletd+0x1dd/frame 0xe4382ca4 fork_exit(c08f67e0,0,e4382ce8) at fork_exit+0xa3/frame 0xe4382cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4382cd4 --- trap 0, eip = 0, esp = 0xe4382d20, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc7ebd000 kdb_enter(c0cc0398,c0cc0398,c0cf3c11,c0e62d0c,2,...) at kdb_enter+0x3d/frame 0xc0e62cb8 vpanic(c0cf3c11,c0e62d0c,c0e62d0c,c0e62d24,c0bc2bab,...) at vpanic+0x13b/frame 0xc0e62cec panic(c0cf3c11,6,6,2,e437fafc,...) at panic+0x1b/frame 0xc0e62d00 dblfault_handler() at dblfault_handler+0xab/frame 0xc0e62d00 --- trap 0x17, eip = 0xc0bacac8, esp = 0xe437f000, ebp = 0xe437fafc --- Xpage(c7ebd000,0,608,c08d12a5,c7ebd000,...) at Xpage/frame 0xe437fafc mi_switch(608,0,c0cc08f4,e2,c7ebd000,...) at mi_switch+0x145/frame 0xe437fb34 critical_exit(c7ebd000,0,2) at critical_exit+0x89/frame 0xe437fb50 ipi_bitmap_handler(8,28,28,c7f83200,0,...) at ipi_bitmap_handler+0x6b/frame 0xe437fb70 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x3d/frame 0xe437fb70 --- interrupt, eip = 0xc0ba87e5, esp = 0xe437fbb8, ebp = 0xe437fbb8 --- acpi_cpu_c1(16600000,16,a65c2a4c,0,c90786c0,...) at acpi_cpu_c1+0x5/frame 0xe437fbb8 acpi_cpu_idle(ffffffff,ffffffff,ffffffff,e437fc28,c0bb0e3a,...) at acpi_cpu_idle+0x15a/frame 0xe437fbf8 cpu_idle_acpi(ffffffff,ffffffff,c0e43484,c0e43488,c0e43494,...) at cpu_idle_acpi+0x3f/frame 0xe437fc0c cpu_idle(1,e437fc78,c0cc1cf3,a4c,0,...) at cpu_idle+0x9a/frame 0xe437fc28 sched_idletd(0,e437fce8,0,0,0,...) at sched_idletd+0x1dd/frame 0xe437fca4 fork_exit(c08f67e0,0,e437fce8) at fork_exit+0xa3/frame 0xe437fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe437fcd4 --- trap 0, eip = 0, esp = 0xe437fd20, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc7ebd360 cpustop_handler(1,13,e437cb6c,c0bc1a2a,0,...) at cpustop_handler+0x25/frame 0xe437ca20 ipi_nmi_handler(0,0,700,0,0,...) at ipi_nmi_handler+0x37/frame 0xe437ca30 trap(e437cb78) at trap+0x3a/frame 0xe437cb6c calltrap() at calltrap+0x6/frame 0xe437cb6c --- trap 0x13, eip = 0xc0ba87e5, esp = 0xe437cbb8, ebp = 0xe437cbb8 --- acpi_cpu_c1(e437cc28,c08f3abe,c0e528d4,37,c7ebd360,...) at acpi_cpu_c1+0x5/frame 0xe437cbb8 acpi_cpu_idle(570b83e9,0,570b83e9,e437cc28,c0bb0e3a,...) at acpi_cpu_idle+0x15a/frame 0xe437cbf8 cpu_idle_acpi(570b83e9,0,c0e42d84,c0e42d88,c0e42d94,...) at cpu_idle_acpi+0x3f/frame 0xe437cc0c cpu_idle(0,e437cc78,c0cc1cf3,a4c,0,...) at cpu_idle+0x9a/frame 0xe437cc28 sched_idletd(0,e437cce8,0,0,0,...) at sched_idletd+0x1dd/frame 0xe437cca4 fork_exit(c08f67e0,0,e437cce8) at fork_exit+0xa3/frame 0xe437ccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe437ccd4 --- trap 0, eip = 0, esp = 0xe437cd20, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc7ebd6c0 sched_switch(c7ebd6c0,0,108,c0e42600,0,...) at sched_switch+0x74f/frame 0xe4379bf0 mi_switch(108,0,c0cc1cf3,a4c,0,...) at mi_switch+0x145/frame 0xe4379c28 sched_idletd(0,e4379ce8,0,0,0,...) at sched_idletd+0x3cf/frame 0xe4379ca4 fork_exit(c08f67e0,0,e4379ce8) at fork_exit+0xa3/frame 0xe4379cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4379cd4 --- trap 0, eip = 0, esp = 0xe4379d20, ebp = 0 --- Some more information from kernel dump: Unread portion of the kernel message buffer: ..... <118>cd /usr/src/include/../sys; install -C -o root -g wheel -m 444 geom/gate/*.h /usr/include/geom/gate <118>cd /usr/src/include/../sys; install -C -o root -g wheel -m 444 geom/journal/*.h /usr/include/geom/journal <118>cd /usr/src/include/../sys; install -C -o root -g wheel -m 444 geom/label/*.h /usr/include/geom/label <118>cd /usr/src/include/../sys; install -C -o root -g wheel -m 444 geom/mirror/*.h /usr/include/geom/mirror <118>cd /usr/src/include/../sys; install -C -o root -g wheel -m 444 geom/mountver/*.h /usr/include/geom/mountver Fatal double fault: eip = 0xc0bacac8 esp = 0xe437f000 ebp = 0xe437fafc cpuid = 2; apic id = 06 panic: double fault cpuid = 2 KDB: stack backtrace: db_trace_self_wrapper(c0cc3575,c0745e1b,10000000,fc,c0d2b220,...) at db_trace_self_wrapper+0x2d/frame 0xc0e62c50 kdb_backtrace(c0cf2f2c,2,c0cf3c11,c0e62d0c,2,...) at kdb_backtrace+0x30/frame 0xc0e62cb8 vpanic(c0cf3c11,c0e62d0c,c0e62d0c,c0e62d24,c0bc2bab,...) at vpanic+0x11b/frame 0xc0e62cec panic(c0cf3c11,6,6,2,e437fafc,...) at panic+0x1b/frame 0xc0e62d00 dblfault_handler() at dblfault_handler+0xab/frame 0xc0e62d00 --- trap 0x17, eip = 0xc0bacac8, esp = 0xe437f000, ebp = 0xe437fafc --- Xpage(c7ebd000,0,608,c08d12a5,c7ebd000,...) at Xpage/frame 0xe437fafc mi_switch(608,0,c0cc08f4,e2,c7ebd000,...) at mi_switch+0x145/frame 0xe437fb34 critical_exit(c7ebd000,0,2) at critical_exit+0x89/frame 0xe437fb50 ipi_bitmap_handler(8,28,28,c7f83200,0,...) at ipi_bitmap_handler+0x6b/frame 0xe437fb70 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x3d/frame 0xe437fb70 --- interrupt, eip = 0xc0ba87e5, esp = 0xe437fbb8, ebp = 0xe437fbb8 --- acpi_cpu_c1(16600000,16,a65c2a4c,0,c90786c0,...) at acpi_cpu_c1+0x5/frame 0xe437fbb8 acpi_cpu_idle(ffffffff,ffffffff,ffffffff,e437fc28,c0bb0e3a,...) at acpi_cpu_idle+0x15a/frame 0xe437fbf8 PR1: 189 Zeilen, 12455 Zeichen. === root@dssresv1 (pts/1) -> cat PR1 Some more information from kernel dump: Unread portion of the kernel message buffer: ..... <118>cd /usr/src/include/../sys; install -C -o root -g wheel -m 444 geom/gate/*.h /usr/include/geom/gate <118>cd /usr/src/include/../sys; install -C -o root -g wheel -m 444 geom/journal/*.h /usr/include/geom/journal <118>cd /usr/src/include/../sys; install -C -o root -g wheel -m 444 geom/label/*.h /usr/include/geom/label <118>cd /usr/src/include/../sys; install -C -o root -g wheel -m 444 geom/mirror/*.h /usr/include/geom/mirror <118>cd /usr/src/include/../sys; install -C -o root -g wheel -m 444 geom/mountver/*.h /usr/include/geom/mountver Fatal double fault: eip = 0xc0bacac8 esp = 0xe437f000 ebp = 0xe437fafc cpuid = 2; apic id = 06 panic: double fault cpuid = 2 KDB: stack backtrace: db_trace_self_wrapper(c0cc3575,c0745e1b,10000000,fc,c0d2b220,...) at db_trace_self_wrapper+0x2d/frame 0xc0e62c50 kdb_backtrace(c0cf2f2c,2,c0cf3c11,c0e62d0c,2,...) at kdb_backtrace+0x30/frame 0xc0e62cb8 vpanic(c0cf3c11,c0e62d0c,c0e62d0c,c0e62d24,c0bc2bab,...) at vpanic+0x11b/frame 0xc0e62cec panic(c0cf3c11,6,6,2,e437fafc,...) at panic+0x1b/frame 0xc0e62d00 dblfault_handler() at dblfault_handler+0xab/frame 0xc0e62d00 --- trap 0x17, eip = 0xc0bacac8, esp = 0xe437f000, ebp = 0xe437fafc --- Xpage(c7ebd000,0,608,c08d12a5,c7ebd000,...) at Xpage/frame 0xe437fafc mi_switch(608,0,c0cc08f4,e2,c7ebd000,...) at mi_switch+0x145/frame 0xe437fb34 critical_exit(c7ebd000,0,2) at critical_exit+0x89/frame 0xe437fb50 ipi_bitmap_handler(8,28,28,c7f83200,0,...) at ipi_bitmap_handler+0x6b/frame 0xe437fb70 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x3d/frame 0xe437fb70 --- interrupt, eip = 0xc0ba87e5, esp = 0xe437fbb8, ebp = 0xe437fbb8 --- acpi_cpu_c1(16600000,16,a65c2a4c,0,c90786c0,...) at acpi_cpu_c1+0x5/frame 0xe437fbb8 acpi_cpu_idle(ffffffff,ffffffff,ffffffff,e437fc28,c0bb0e3a,...) at acpi_cpu_idle+0x15a/frame 0xe437fbf8 cpu_idle_acpi(ffffffff,ffffffff,c0e43484,c0e43488,c0e43494,...) at cpu_idle_acpi+0x3f/frame 0xe437fc0c cpu_idle(1,e437fc78,c0cc1cf3,a4c,0,...) at cpu_idle+0x9a/frame 0xe437fc28 sched_idletd(0,e437fce8,0,0,0,...) at sched_idletd+0x1dd/frame 0xe437fca4 fork_exit(c08f67e0,0,e437fce8) at fork_exit+0xa3/frame 0xe437fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe437fcd4 --- trap 0, eip = 0, esp = 0xe437fd20, ebp = 0 --- KDB: enter: panic #0 doadump (textdump=-1058657840) at pcpu.h:233 233 __asm("movl %%fs:%1,%0" : "=r" (td) (kgdb) list *0xc0bacac8 0xc0bacac8 is at /usr/src/sys/i386/i386/exception.s:135. 130 IDTVEC(stk) 131 TRAP(T_STKFLT) 132 IDTVEC(prot) 133 TRAP(T_PROTFLT) 134 IDTVEC(page) 135 TRAP(T_PAGEFLT) 136 IDTVEC(mchk) 137 pushl $0; TRAP(T_MCHK) 138 IDTVEC(rsvd) 139 pushl $0; TRAP(T_RESERVED) (kgdb) disas Xpage Dump of assembler code for function Xpage: 0xc0bacac8 : push $0xc 0xc0bacaca : jmp 0xc0bacaf0 (kgdb) where #0 doadump (textdump=-1058657840) at pcpu.h:233 #1 0xc053646d in db_fncall (dummy1=-13, dummy2=0, dummy3=-1061010685, dummy4=0xc0e62958 "") at /usr/src/sys/ddb/db_command.c:568 #2 0xc053614b in db_command (cmd_table=) at /usr/src/sys/ddb/db_command.c:440 #3 0xc053a0e4 in db_script_exec (scriptname=, warnifnotfound=) at /usr/src/sys/ddb/db_script.c:302 #4 0xc0539f42 in db_script_kdbenter (eventname=0xc0cc0398 "panic") at /usr/src/sys/ddb/db_script.c:324 #5 0xc053871b in db_trap (type=, code=) at /usr/src/sys/ddb/db_main.c:230 #6 0xc09088a4 in kdb_trap (type=, code=, tf=) at /usr/src/sys/kern/subr_kdb.c:656 #7 0xc0bc21ef in trap (frame=) at /usr/src/sys/i386/i386/trap.c:696 #8 0xc0bacb1a in calltrap () at /usr/src/sys/i386/i386/exception.s:173 #9 0xc090812d in kdb_enter (why=0xc0cc0398 "panic", msg=) at cpufunc.h:71 #10 0xc08c645b in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:882 #11 0xc08c631b in panic (fmt=0xc0cf3c11 "double fault") at /usr/src/sys/kern/kern_shutdown.c:818 #12 0xc0bc2bab in dblfault_handler () at /usr/src/sys/i386/i386/trap.c:1066 #13 0x00000000 in ?? () (kgdb) tid 100005 [Switching to thread 16 (Thread 100005)]#0 doadump (textdump=-1058657840) at pcpu.h:233 233 __asm("movl %%fs:%1,%0" : "=r" (td) (kgdb) where #0 doadump (textdump=-1058657840) at pcpu.h:233 #1 0xc053646d in db_fncall (dummy1=-13, dummy2=0, dummy3=-1061010685, dummy4=0xc0e62958 "") at /usr/src/sys/ddb/db_command.c:568 #2 0xc053614b in db_command (cmd_table=) at /usr/src/sys/ddb/db_command.c:440 #3 0xc053a0e4 in db_script_exec (scriptname=, warnifnotfound=) at /usr/src/sys/ddb/db_script.c:302 #4 0xc0539f42 in db_script_kdbenter (eventname=0xc0cc0398 "panic") at /usr/src/sys/ddb/db_script.c:324 #5 0xc053871b in db_trap (type=, code=) at /usr/src/sys/ddb/db_main.c:230 #6 0xc09088a4 in kdb_trap (type=, code=, tf=) at /usr/src/sys/kern/subr_kdb.c:656 #7 0xc0bc21ef in trap (frame=) at /usr/src/sys/i386/i386/trap.c:696 #8 0xc0bacb1a in calltrap () at /usr/src/sys/i386/i386/exception.s:173 #9 0xc090812d in kdb_enter (why=0xc0cc0398 "panic", msg=) at cpufunc.h:71 #10 0xc08c645b in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:882 #11 0xc08c631b in panic (fmt=0xc0cf3c11 "double fault") at /usr/src/sys/kern/kern_shutdown.c:818 #12 0xc0bc2bab in dblfault_handler () at /usr/src/sys/i386/i386/trap.c:1066 #13 0x00000000 in ?? () (kgdb) tid 100099 [Switching to thread 85 (Thread 100099)]#0 0xc0bb6825 in cpustop_handler () at /usr/src/sys/i386/i386/mp_machdep.c:1506 1506 savectx(&stoppcbs[cpu]); (kgdb) where #0 0xc0bb6825 in cpustop_handler () at /usr/src/sys/i386/i386/mp_machdep.c:1506 #1 0xc0bb67f7 in ipi_nmi_handler () at /usr/src/sys/i386/i386/mp_machdep.c:1491 #2 0xc0bc1a2a in trap (frame=0xf0b8583c) at /usr/src/sys/i386/i386/trap.c:208 #3 0xc0bacb1a in calltrap () at /usr/src/sys/i386/i386/exception.s:173 #4 0xc0bb6177 in smp_tlb_shootdown (vector=, addr1=) at /usr/src/sys/i386/i386/mp_machdep.c:1242 #5 0xc0bb6214 in smp_invlpg (addr=) at /usr/src/sys/i386/i386/mp_machdep.c:1311 #6 0xc0bb99a7 in pmap_remove_pte (pmap=0xc0eb7174, ptq=, va=3984588800, free=0x55000563) at /usr/src/sys/i386/i386/pmap.c:1003 #7 0xc0bb8f17 in pmap_remove (pmap=, sva=, eva=) at /usr/src/sys/i386/i386/pmap.c:3105 #8 0xc0b5bbbf in vm_map_delete (map=0xc17ac2d0, start=) at /usr/src/sys/vm/vm_map.c:3001 #9 0xc0b5959c in kmap_free_wakeup (map=0xc17ac2d0, addr=3984588800, size=263168) at /usr/src/sys/vm/vm_kern.c:470 #10 0xc08865cd in kern_execve (td=0xc90786c0, args=0xf0b85bd0) at /usr/src/sys/kern/kern_exec.c:1267 #11 0xc088534d in sys_execve (td=0xc90786c0, uap=) at /usr/src/sys/kern/kern_exec.c:222 #12 0xc0bc328b in syscall (frame=) at subr_syscall.c:141 #13 0xc0bacbce in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:279 #14 0x00000033 in ?? () (kgdb) tid 100003 [Switching to thread 14 (Thread 100003)]#0 sched_switch (td=0xc7ebd6c0, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1956 1956 cpuid = PCPU_GET(cpuid); (kgdb) where #0 sched_switch (td=0xc7ebd6c0, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1956 #1 0xc08d12a5 in mi_switch (flags=264, newtd=0xc0e42d00) at /usr/src/sys/kern/kern_synch.c:473 #2 0xc08f6baf in sched_idletd (dummy=0xc0cc1cf3) at /usr/src/sys/kern/sched_ule.c:1014 #3 0xc088fea3 in fork_exit (callout=0xc08f67e0 ) at /usr/src/sys/kern/kern_fork.c:1030 #4 0xc0bacbe0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:288 (kgdb) tid 100004 [Switching to thread 15 (Thread 100004)]#0 0xc0bb6825 in cpustop_handler () at /usr/src/sys/i386/i386/mp_machdep.c:1506 1506 savectx(&stoppcbs[cpu]); (kgdb) where #0 0xc0bb6825 in cpustop_handler () at /usr/src/sys/i386/i386/mp_machdep.c:1506 #1 0xc0bb67f7 in ipi_nmi_handler () at /usr/src/sys/i386/i386/mp_machdep.c:1491 #2 0xc0bc1a2a in trap (frame=0xe437cb78) at /usr/src/sys/i386/i386/trap.c:208 #3 0xc0bacb1a in calltrap () at /usr/src/sys/i386/i386/exception.s:173 #4 0xc0ba87e5 in acpi_cpu_c1 () at /usr/src/sys/i386/acpica/acpi_machdep.c:114 #5 0xc0549d3a in acpi_cpu_idle (sbt=1460372457) at /usr/src/sys/dev/acpica/acpi_cpu.c:1026 #6 0xc0bb0d8f in cpu_idle_acpi (sbt=1460372457) at /usr/src/sys/i386/i386/machdep.c:1313 #7 0xc0bb0e3a in cpu_idle (busy=0) at /usr/src/sys/i386/i386/machdep.c:1472 #8 0xc08f69bd in sched_idletd (dummy=0xc0cc1cf3) at /usr/src/sys/kern/sched_ule.c:2679 #9 0xc088fea3 in fork_exit (callout=0xc08f67e0 ) at /usr/src/sys/kern/kern_fork.c:1030 #10 0xc0bacbe0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:288 (kgdb) tid 100005 [Switching to thread 16 (Thread 100005)]#0 doadump (textdump=-1058657840) at pcpu.h:233 233 __asm("movl %%fs:%1,%0" : "=r" (td) (kgdb) where #0 doadump (textdump=-1058657840) at pcpu.h:233 #1 0xc053646d in db_fncall (dummy1=-13, dummy2=0, dummy3=-1061010685, dummy4=0xc0e62958 "") at /usr/src/sys/ddb/db_command.c:568 #2 0xc053614b in db_command (cmd_table=) at /usr/src/sys/ddb/db_command.c:440 #3 0xc053a0e4 in db_script_exec (scriptname=, warnifnotfound=) at /usr/src/sys/ddb/db_script.c:302 #4 0xc0539f42 in db_script_kdbenter (eventname=0xc0cc0398 "panic") at /usr/src/sys/ddb/db_script.c:324 #5 0xc053871b in db_trap (type=, code=) at /usr/src/sys/ddb/db_main.c:230 #6 0xc09088a4 in kdb_trap (type=, code=, tf=) at /usr/src/sys/kern/subr_kdb.c:656 #7 0xc0bc21ef in trap (frame=) at /usr/src/sys/i386/i386/trap.c:696 #8 0xc0bacb1a in calltrap () at /usr/src/sys/i386/i386/exception.s:173 #9 0xc090812d in kdb_enter (why=0xc0cc0398 "panic", msg=) at cpufunc.h:71 #10 0xc08c645b in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:882 #11 0xc08c631b in panic (fmt=0xc0cf3c11 "double fault") at /usr/src/sys/kern/kern_shutdown.c:818 #12 0xc0bc2bab in dblfault_handler () at /usr/src/sys/i386/i386/trap.c:1066 #13 0x00000000 in ?? () (kgdb) tid 100006 [Switching to thread 17 (Thread 100006)]#0 sched_switch (td=0xc7ebca20, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1956 1956 cpuid = PCPU_GET(cpuid); (kgdb) where #0 sched_switch (td=0xc7ebca20, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1956 #1 0xc08d12a5 in mi_switch (flags=1544, newtd=0xe4382b28) at /usr/src/sys/kern/kern_synch.c:473 #2 0xc08ced89 in critical_exit () at /usr/src/sys/kern/kern_switch.c:233 #3 0xc0bb659b in ipi_bitmap_handler (frame= {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = -940036608, tf_esi = 0, tf_ebp = -466080840, tf_isp = -466080860, tf_ebx = -940036580, tf_edx = 91, tf_ecx = 0, tf_eax = -1158659044, tf_trapno = 0, tf_err = 0, tf_eip = -1061517339, tf_cs = 32, tf_eflags = 582, tf_esp = -466080776, tf_ss = -1068196550}) at /usr/src/sys/i386/i386/mp_machdep.c:1402 #4 0xc0bad42d in Xipi_intr_bitmap_handler () at apic_vector.s:240 #5 0xc0ba87e5 in acpi_cpu_c1 () at /usr/src/sys/i386/acpica/acpi_machdep.c:114 #6 0xc0549d3a in acpi_cpu_idle (sbt=-1) at /usr/src/sys/dev/acpica/acpi_cpu.c:1026 #7 0xc0bb0d8f in cpu_idle_acpi (sbt=-1) at /usr/src/sys/i386/i386/machdep.c:1313 #8 0xc0bb0e3a in cpu_idle (busy=1) at /usr/src/sys/i386/i386/machdep.c:1472 #9 0xc08f69bd in sched_idletd (dummy=0xc0cc1cf3) at /usr/src/sys/kern/sched_ule.c:2679 #10 0xc088fea3 in fork_exit (callout=0xc08f67e0 ) at /usr/src/sys/kern/kern_fork.c:1030 #11 0xc0bacbe0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:288 (kgdb) quit --- Andreas Longwitz