From owner-freebsd-current@FreeBSD.ORG Mon Oct 14 08:21:39 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 319975A9 for ; Mon, 14 Oct 2013 08:21:39 +0000 (UTC) (envelope-from mexas@bristol.ac.uk) Received: from eu1sys200aog104.obsmtp.com (eu1sys200aog104.obsmtp.com [207.126.144.117]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 15D8222AC for ; Mon, 14 Oct 2013 08:21:37 +0000 (UTC) Received: from mail-wg0-f49.google.com ([74.125.82.49]) (using TLSv1) by eu1sys200aob104.postini.com ([207.126.147.11]) with SMTP ID DSNKUlupdXHnnkt5x5lzGnf/bHDv6Caxf41g@postini.com; Mon, 14 Oct 2013 08:21:38 UTC Received: by mail-wg0-f49.google.com with SMTP id x12so4096293wgg.4 for ; Mon, 14 Oct 2013 01:21:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:message-id:to:subject:cc :reply-to:in-reply-to; bh=uwxZb7PVjaWjNVMWUkK8TOj0iG9WB541xcI2oF5tAK0=; b=OiIZLhbHQmfghIRotnCRIMhDfS1fYU6gN4m4+8zGOxExIuKsJnBlVaqf9fAMqYFi57 UCitC+Xux4VxOZP+JcgRtw1SD7INx2PYNTv+IGafz8p3M46Km72eSSXj3f1GB3i28rXV kiq4jnU8cCatBc9CQnmU/aW+5O57uR9oj23LCteSj4DTEh5+luc3j9rxFkAXtDXzTW4Q B/WMv4o+RXvkviZd5wtL/ZdsYyDDCIUZ+3owe97jdQIz3/+9BpUk+Q0zaly8A7vpNIq2 tmpPqpyT2djeMK2zKWUWUu2Q27zB9WlbfI0sTmHaGq2c66DOOSqk6scti4hb+oaLpBsH rfcA== X-Gm-Message-State: ALoCoQmRGMs/Bm9sU9Muo8R+EjHptLMRuZrh/pCkpEaR1fgle1g6/U8DkGajZuR+WwC5f+Zp+it18lf0W9B4gVUzBoHqs6WdvWO159uz6TGvCmmIesNbchEv8yvQRbIvEC+Jr/+zvQZevv6U4AsfsMPBG3f8dW1kELjAecNqWCsECujtplgIidM= X-Received: by 10.180.208.97 with SMTP id md1mr13580947wic.41.1381738869056; Mon, 14 Oct 2013 01:21:09 -0700 (PDT) X-Received: by 10.180.208.97 with SMTP id md1mr13580938wic.41.1381738868955; Mon, 14 Oct 2013 01:21:08 -0700 (PDT) Received: from mech-cluster241.men.bris.ac.uk (mech-cluster241.men.bris.ac.uk. [137.222.187.241]) by mx.google.com with ESMTPSA id jf2sm31806314wic.2.1969.12.31.16.00.00 (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 14 Oct 2013 01:21:07 -0700 (PDT) Sender: Anton Shterenlikht Received: from mech-cluster241.men.bris.ac.uk (localhost [127.0.0.1]) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6) with ESMTP id r9E8L5LZ014898 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 14 Oct 2013 09:21:05 +0100 (BST) (envelope-from mexas@mech-cluster241.men.bris.ac.uk) Received: (from mexas@localhost) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6/Submit) id r9E8L55j014897; Mon, 14 Oct 2013 09:21:05 +0100 (BST) (envelope-from mexas) Date: Mon, 14 Oct 2013 09:21:05 +0100 (BST) From: Anton Shterenlikht Message-Id: <201310140821.r9E8L55j014897@mech-cluster241.men.bris.ac.uk> To: davide@freebsd.org, mexas@bris.ac.uk Subject: Re: panic: ia64 r255811: deadlkres: possible deadlock detected for 0xe000000012d07b00, blocked for 902743 ticks In-Reply-To: Cc: freebsd-current@freebsd.org, freebsd-ia64@freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: mexas@bris.ac.uk List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Oct 2013 08:21:39 -0000 >From davide.italiano@gmail.com Fri Oct 11 15:39:49 2013 > >If you're not able to get a full dump, a textdump would be enough. >In your DDB scripts just remove the 'call doadump' step and you should >be done, unless I'm missing something. >Please adjust the script as well to include all the informations >requested as mentioned in my previous link, e.g. 'show lockedvnods' is >not mentioned on the example section of textdump(4) manpage but it >could be useful to ease debugging. I still haven't got it right, but I now collected manually some (hopefully) useful info (see below). - here's my /etc/ddb.conf: script lockinfo=show locks; show alllocks; show lockedvnods script kdb.enter.panic=textdump set; capture on; run lockinfo; show locks; show alllocks; show lockedvnods; show pcpu; bt; ps; alltrace; capture off; reset script kdb.enter.witness=run lockinfo Somehow when I get into a debugger, I can only see lockinfo. Why? - I've removed "call doadump", as you see above. Still the dump is being written. I've in /etc/rc.conf: dumpdev=/dev/da4p1 dumpdir=/var/crash ddb_enable=YES Shall I remove dumpdev completely? And leave only dumpdir defined? Thanks Anton db> textdump set textdump set db> capture on db> run lockinfo db:0:lockinfo> show locks db:0:locks> show alllocks db:0:alllocks> show lockedvnods Locked vnodes 0xe0000000123799d0: tag devfs, type VCHR usecount 1, writecount 0, refcount 408 mountedhere 0xe0000000126a4c00 flags (VI_ACTIVE) v_object 0xe000000012788100 ref 0 pages 21330 lock type devfs: EXCL by thread 0xe00000001268e000 (pid 21, syncer, tid 100062) dev da5p1 0xe0000000127d5448: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0xe00000001277bc00 ref 0 pages 1 lock type ufs: EXCL by thread 0xe0000000127b5680 (pid 908, ntpd, tid 100080) ino 2889216, on dev da3p1 0xe0000002203c0ce8: tag ufs, type VREG usecount 1, writecount 1, refcount 16 mountedhere 0 flags (VI_ACTIVE) v_object 0xe000000223037700 ref 0 pages 56 lock type ufs: EXCL by thread 0xe00000001557e480 (pid 45384, pkg-static, tid 100133) ino 5780554, on dev da5p1 db> db> show pcpu cpuid = 0 dynamic pcpu = 0x40040040ff13ca80 curthread = 0xe000000011a26d80: pid 0 "deadlkres" curpcb = 0 fpcurthread = none idlethread = 0xe000000011178900: tid 100003 "idle: cpu0" MD: vhpt = 0xe0000040ff000000 MD: lid = 0 MD: clock = 0x214563ccc206 MD: clock_mode = 2 MD: clock_load = 0 MD: stats = 0x9ffc0000010ff130 MD: pmap = 0x9ffc000000eee760 spin locks held: db> db> bt Tracing pid 0 tid 100053 td 0xe000000011a26d80 kdb_enter(0x9ffc000000dc2fd0, 0x9ffc000000dc2fd0, 0x9ffc00000058e6b0, 0x40c) at kdb_enter+0x92 vpanic(0x9ffc000000db8a18, 0xa00000009dca7518) at vpanic+0x2b0 panic(0x9ffc000000db8a18, 0x9ffc000000db8c70, 0xe00000001557fb00, 0xdbd38) at panic+0x80 deadlkres(0xdbd38, 0xe00000001557fb00, 0x9ffc000000dbb648, 0x9ffc000000db89a8) at deadlkres+0x420 fork_exit(0x9ffc000000e0fca0, 0x0, 0xa00000009dca7550) at fork_exit+0x120 enter_userland() at enter_userland db> db> ps pid ppid pgrp uid state wmesg wchan cmd 47482 47176 1092 0 LV+J *pmap pv 0xe000000012402540 sh 47476 35326 1092 0 D+ vm map ( 0xe0000000110adac0 sh 47475 35326 1092 0 S+ piperd 0xe000000222b2f5a8 awk 47474 35326 1092 0 L+ *pmap pv 0xe000000012402540 awk 47335 47329 1092 0 LL+J *pmap pv 0xe000000012402540 cc1 47329 34696 1092 0 SW+J wait 0xe0000002233c3720 xgcc 47176 45981 1092 0 D+J ppwait 0xe0000000156af6d0 sh 46683 46681 1092 0 S+J piperd 0xe0000000127e3340 as 46682 46681 1092 0 LL+J *pmap pv 0xe000000012402540 cc1plus 46681 33402 1092 0 SW+J wait 0xe0000000154c4000 g++ 45981 44152 1092 0 SW+J wait 0xe0000002233c3280 pkg-static 45384 45009 1092 0 L+J *vm page 0xe00000001eff4f00 pkg-static 45009 44866 1092 0 SW+J wait 0xe0000002233c2940 sh 44866 38577 1092 0 SW+J wait 0xe00000001547cde0 make 44152 43869 1092 0 SW+J wait 0xe00000001f2dcde0 sh 43869 37427 1092 0 SW+J wait 0xe00000001549d280 make 39289 38577 1092 0 S+ piperd 0xe00000001556d0d8 tee 38577 1092 1092 0 SW+ wait 0xe0000000154f1280 sh 37813 37427 1092 0 S+ piperd 0xe000000222b2f810 tee 37427 1092 1092 0 SW+ wait 0xe000000012b3c4a0 sh 34696 89091 1092 0 SW+J wait 0xe00000001556e000 gmake 33402 33399 1092 0 SW+J wait 0xe00000001280e4a0 make 33399 33339 1092 0 SW+J wait 0xe0000000129e1720 sh 33339 33337 1092 0 SW+J wait 0xe00000001556e4a0 make 33337 88092 1092 0 SW+ wait 0xe00000001f2dd720 sh 90596 88092 1092 0 S+ piperd 0xe0000000127e3ce0 tee 89091 84462 1092 0 SW+J wait 0xe000000224d20000 gmake 88092 1092 1092 0 D+ vm map ( 0xe0000000110ad9d0 sh 84462 84441 1092 0 SW+J wait 0xe00000001efcd720 gmake 84441 84440 1092 0 SW+J wait 0xe00000001547c000 gmake 84440 84425 1092 0 SW+J wait 0xe00000001f2dc000 sh 84425 84423 1092 0 SW+J wait 0xe000000012b31720 make 84423 74621 1092 0 SW+ wait 0xe00000001280e000 sh 75146 74621 1092 0 S+ piperd 0xe000000012b54000 tee 74621 1092 1092 0 D+ vm map ( 0xe0000000110ad9d0 sh 39253 1151 39253 0 L+ *pmap pv 0xe000000012402540 top 35326 1092 1092 0 S+ wait 0xe0000000154f0000 sh 1151 1150 1151 0 SW+ wait 0xe00000001280f280 sh 1150 1146 1150 1001 SW+ wait 0xe000000012b304a0 su 1146 1145 1146 1001 SWs+ pause 0xe000000012b3c9e8 tcsh 1145 1143 1143 1001 S select 0xe0000000126d4f40 sshd 1143 968 1143 0 Ss select 0xe000000010c299c0 sshd 1092 1050 1092 0 LL+ *pmap pv 0xe000000012402540 sh 1050 1049 1050 0 SW+ wait 0xe000000012b3d720 sh 1049 1045 1049 1001 SW+ wait 0xe00000001280f720 su 1045 1044 1045 1001 SWs+ pause 0xe000000012b3ce88 tcsh 1044 1042 1042 1001 S select 0xe000000010c28e40 sshd 1042 968 1042 0 Ss select 0xe0000000126d5040 sshd 1041 1 1041 0 Ss+ ttyin 0xe00000001121c8a8 getty 1040 1 1 0 S ttydcd 0xe00000001121cce8 getty 1020 1 1020 0 Ss select 0xe000000010c290c0 inetd 979 1 979 0 LLs *pmap pv 0xe000000012402540 cron 974 1 974 25 LLs *pmap pv 0xe000000012402540 sendmail 971 1 971 0 LLs *pmap pv 0xe000000012402540 sendmail 968 1 968 0 Ss select 0xe0000000126d54c0 sshd 908 1 908 0 Ls *vm page 0xe00000001eff4f00 ntpd 827 1 827 0 Ss select 0xe0000000126d56c0 syslogd 818 1 818 0 Ss select 0xe0000000126d5740 ipmon 696 1 696 0 Ss select 0xe0000000126d57c0 devd 181 1 181 0 SWs pause 0xe00000001269a9e8 adjkerntz 23 0 0 0 DL sdflush 0x9ffc000000ed1ff0 [softdepflush] 22 0 0 0 DL vlruwt 0xe000000012409720 [vnlru] 21 0 0 0 LL *vm page 0xe00000001eff4f00 [syncer] 20 0 0 0 DL psleep 0x9ffc000000ed1390 [bufdaemon] 19 0 0 0 DL pgzero 0x9ffc000000ed239c [pagezero] 18 0 0 0 RL [vmdaemon] 17 0 0 0 LL *pmap pv 0xe000000012402540 [pagedaemon] 9 0 0 0 DL ccb_scan 0x9ffc000000ed26a0 [xpt_thrd] 6 0 0 0 DL waiting_ 0x9ffc0000010f33b0 [sctp_iterator] 5 0 0 0 RL (threaded) [zfskern] 100052 Run CPU 1 [l2arc_feed_thread] 100051 D arc_recl 0x9ffc000001538120 [arc_reclaim_thread] 4 0 0 0 DL ispf 0xe000000011a02000 [isp0: fc_thrd0] 3 0 0 0 DL idle 0xa000000000a16000 [mpt_recovery1] 2 0 0 0 DL idle 0xa0000000009fe000 [mpt_recovery0] 16 0 0 0 DL (threaded) [usb] 100039 D - 0xa0000000009fae18 [usbus2] 100038 D - 0xa0000000009fadc0 [usbus2] 100037 D - 0xa0000000009fad68 [usbus2] 100036 D - 0xa0000000009fad10 [usbus2] 100034 D - 0xa0000000009f1460 [usbus1] 100033 D - 0xa0000000009f1408 [usbus1] 100032 D - 0xa0000000009f13b0 [usbus1] 100031 D - 0xa0000000009f1358 [usbus1] 100029 D - 0xa0000000009ed460 [usbus0] 100028 D - 0xa0000000009ed408 [usbus0] 100027 D - 0xa0000000009ed3b0 [usbus0] 100026 D - 0xa0000000009ed358 [usbus0] 15 0 0 0 DL tzpoll 0x9ffc000000ecfd20 [acpi_thermal] 14 0 0 0 DL - 0x9ffc000000ed0008 [rand_harvestq] 13 0 0 0 DL (threaded) [geom] 100012 D - 0x9ffc000000ed0540 [g_down] 100011 D - 0x9ffc000000ed0538 [g_up] 100010 D - 0x9ffc000000ed0528 [g_event] 12 0 0 0 WL (threaded) [intr] 100048 I [swi0: uart uart+++] 100046 I [irq71: isp0] 100042 I [irq28: mpt1] 100040 I [irq27: mpt0] 100035 I [irq18: ehci0] 100030 I [irq17: ohci1] 100025 I [irq16: ohci0] 100022 I [swi6: task queue] 100021 I [swi6: Giant taskq] 100019 I [swi5: fast taskq] 100015 I [swi2: cambio] 100008 I [swi4: clock] 100007 I [swi4: clock] 100006 I [swi3: vm] 100005 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100004 CanRun [idle: cpu1] 100003 CanRun [idle: cpu0] 1 0 1 0 SLs wait 0xe000000011176de0 [init] 10 0 0 0 DL audit_wo 0x9ffc0000010f7a48 [audit] 0 0 0 0 RLs (threaded) [kernel] 100053 Run CPU 0 [deadlkres] 100050 D - 0xe000000011a2d300 [system_taskq_1] 100049 D - 0xe000000011a2d300 [system_taskq_0] 100045 D - 0xe000000011198500 [em1 taskq] 100044 D - 0xe000000011198600 [em0 taskq] 100023 D - 0xe00000001119b300 [ffs_trim taskq] 100020 D - 0xe00000001119b600 [thread taskq] 100018 D - 0xe00000001119b800 [acpi_task_2] 100017 D - 0xe00000001119b800 [acpi_task_1] 100016 D - 0xe00000001119b800 [acpi_task_0] 100014 D - 0xe00000001119ba00 [kqueue taskq] 100009 D - 0xe00000001119bd00 [firmware taskq] 100000 D swapin 0x9ffc000000eedd28 [swapper] db> db> alltrace Tracing command sh pid 47482 tid 100135 td 0xe00000001557fb00 cpu_switch(0xe00000001557fb00, 0xe000000015478d80, 0xe000000012402540, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe00000001557fb00, 0xe000000015478d80, 0x9ffc000000f15698, 0x9ffc000000f15680) at sched_switch+0x890 mi_switch(0x103, 0x0, 0xe00000001557fb00, 0x9ffc00000062d1f0) at mi_switch+0x3f0 turnstile_wait(0xe000000012402540, 0xe000000012400000, 0x0, 0x9ffc000000dcb698, 0xe00000001557fb00) at turnstile_wait+0x960 __rw_wlock_hard(0x9ffc0000010ffc18, 0xe00000001557fb00, 0x0, 0x9ffc000000ecc03c, 0x9ffc0000010ffc00) at __rw_wlock_hard+0x4c0 _rw_wlock_cookie(0x9ffc0000010ffc00, 0x9ffc000000e081f8, 0x68a, 0x9ffc000000ab2720, 0x815) at _rw_wlock_cookie+0x150 pmap_enter(0x9ffc0000010ffc80, 0xa0000000e2802000, 0x2, 0xe00000027edd1590, 0x7, 0x0, 0x9ffc000000e081f8, 0xe000000019c28f18) at pmap_enter+0x40 vm_fault_hold(0xe000000010000000, 0xa0000000e2802000, 0x2, 0x0, 0x0) at vm_fault_hold+0x2c70 vm_fault(0xe000000010000000, 0xa0000000e2802000, 0x2, 0x0, 0xe00000001557fb00, 0x9ffc000000abbd60, 0x716, 0x30a) at vm_fault+0xf0 trap(0x14, 0xa00000009df21000) at trap+0xa60 ivt_Data_TLB() at ivt_Data_TLB+0x1d0 --- trapframe at 0xa00000009df21000 copystr() at copystr+0x11 copyinstr(0x140c0cf70, 0xa0000000e2802000, 0x400, 0xa00000009df21350) at copyinstr+0xa0 exec_copyin_args(0xa00000009df21360, 0x140c0cf70, 0x0, 0x140c0ce18, 0x140c0ce40) at exec_copyin_args+0xf0 sys_execve(0xe00000001557fb00, 0xa00000009df214e8, 0x9ffc000000abac80, 0x48d) at sys_execve+0x40 syscall(0xe0000000156af280, 0x140c0ce18, 0x140c0ce40, 0xe00000001557fb00, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sh pid 47476 tid 100138 td 0xe000000015479b00 cpu_switch(0xe000000015479b00, 0xe00000001557e480, 0x9ffc000000f4b7b0, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe000000015479b00, 0xe00000001557e480, 0x9ffc000000f16398, 0x9ffc000000f16380) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000015479b00, 0x9ffc000000620b70) at mi_switch+0x3f0 sleepq_switch(0xe0000000110adac0, 0x0, 0xe000000015479e78, 0xe000000015479b00) at sleepq_switch+0x2e0 sleepq_wait(0xe0000000110adac0, 0x0, 0x9ffc000000dca928, 0xe000000015479b00, 0x9ffc00000059ead0) at sleepq_wait+0xb0 _sx_xlock_hard(0xe0000000110adac0, 0xe000000015479b00, 0x0, 0xe0000000110adad8, 0xe0000000110adac0) at _sx_xlock_hard+0x8d0 _sx_xlock(0xe0000000110adac0, 0x0, 0x9ffc000000dfcf30, 0x1c2) at _sx_xlock+0x170 _vm_map_lock(0xe0000000110ada40, 0x9ffc000000dfcf30, 0x1c2, 0x9ffc000000a3fa50, 0x593, 0xe000000010c54f00) at _vm_map_lock+0x80 kmap_alloc_wait(0xe0000000110ada40, 0x42000) at kmap_alloc_wait+0x90 exec_alloc_args(0xa00000009df39360, 0x9ffc0000005253e0, 0x610, 0x140308ed0) at exec_alloc_args+0x30 exec_copyin_args(0xa00000009df39360, 0x140c148b0, 0x0, 0x140c14578, 0x140c14810) at exec_copyin_args+0x60 sys_execve(0xe000000015479b00, 0xa00000009df394e8, 0x9ffc000000abac80, 0x48d) at sys_execve+0x40 syscall(0xe0000000156ae4a0, 0x140c14578, 0x140c14810, 0xe000000015479b00, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command awk pid 47475 tid 100131 td 0xe000000015478480 cpu_switch(0xe000000015478480, 0xe00000001557fb00, 0x9ffc000000f4c228, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe000000015478480, 0xe00000001557fb00, 0x9ffc000000f16398, 0x9ffc000000f16380) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000015478480, 0x9ffc000000620b70) at mi_switch+0x3f0 sleepq_switch(0xe000000222b2f5a8, 0x5c, 0xe0000000154787f8, 0xe000000015478480) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000222b2f5a8, 0x5c, 0x9ffc000000dca928, 0x9ffc000000dcace8) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000222b2f5a8, 0x5c, 0x9ffc000000dce2b0, 0x100, 0x9ffc0000005a0a90) at sleepq_wait_sig+0x30 _sleep(0xe000000222b2f5a8, 0xe000000222b2f7e8, 0x5c, 0x9ffc000000dce2b0, 0x0, 0x0, 0x100) at _sleep+0x7b0 pipe_read(0xe00000001edaa9d4, 0xa00000009df01360, 0xa00000009df01378, 0x0, 0xe000000222b2f6b0) at pipe_read+0x740 dofileread(0xe000000015478480, 0x0, 0xe00000001edaa9b0, 0xa00000009df01360, 0x0, 0x0) at dofileread+0x150 kern_readv(0xe000000015478480, 0x0, 0xa00000009df01360) at kern_readv+0xa0 sys_read(0xe000000015478480, 0xa00000009df014e8, 0x9ffc000000abac80, 0x48d) at sys_read+0x100 syscall(0xe00000001556e940, 0x140c8a000, 0x2000, 0xe000000015478480, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command awk pid 47474 tid 100165 td 0xe00000001f4b6000 cpu_switch(0xe00000001f4b6000, 0xe000000011197680, 0xe000000012402540, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe00000001f4b6000, 0xe000000011197680, 0x9ffc000000f15698, 0x9ffc000000f15680) at sched_switch+0x890 mi_switch(0x103, 0x0, 0xe00000001f4b6000, 0x9ffc00000062d1f0) at mi_switch+0x3f0 turnstile_wait(0xe000000012402540, 0xe000000012400000, 0x0, 0x9ffc000000dcb698, 0xe00000001f4b6000) at turnstile_wait+0x960 __rw_wlock_hard(0x9ffc0000010ffc18, 0xe00000001f4b6000, 0x0, 0x9ffc000000ecc03c, 0x9ffc0000010ffc00) at __rw_wlock_hard+0x4c0 _rw_wlock_cookie(0x9ffc0000010ffc00, 0x9ffc000000e081f8, 0x68a, 0x9ffc000000ab2720, 0x815) at _rw_wlock_cookie+0x150 pmap_enter(0x9ffc0000010ffc80, 0xa0000000e2c4e000, 0x2, 0xe00000027edd13d0, 0x7, 0x0, 0x9ffc000000e081f8, 0xe00000001b1a6918) at pmap_enter+0x40 vm_fault_hold(0xe000000010000000, 0xa0000000e2c4e000, 0x2, 0x0, 0x0) at vm_fault_hold+0x2c70 vm_fault(0xe000000010000000, 0xa0000000e2c4e000, 0x2, 0x0, 0xe00000001f4b6000, 0x9ffc000000abbd60, 0x716, 0xe000000222b2f5a8) at vm_fault+0xf0 trap(0x14, 0xa000000000211000) at trap+0xa60 ivt_Data_TLB() at ivt_Data_TLB+0x1d0 --- trapframe at 0xa000000000211000 bcopy() at bcopy+0xe1 copyin(0x140c70000, 0xa0000000e2c4e000, 0x10ae) at copyin+0x90 uiomove_faultflag(0xa0000000e2c4e000, 0x10ae, 0xa000000000211360, 0xa000000000211390) at uiomove_faultflag+0x2a0 uiomove(0xa0000000e2c4e000, 0x10ae, 0xa000000000211360, 0x9ffc00000064dc40, 0x1939, 0x9ffc00000064dc10) at uiomove+0x30 pipe_write(0xe0000000127220c4, 0xa000000000211360, 0xe000000222b2f6b0, 0x10ae, 0x10ae) at pipe_write+0x1bd0 dofilewrite(0xe00000001f4b6000, 0x1, 0xe0000000127220a0, 0xa000000000211360, 0xffffffffffffffff, 0x0) at dofilewrite+0x180 kern_writev(0xe00000001f4b6000, 0x1, 0xa000000000211360) at kern_writev+0xa0 sys_write(0xe00000001f4b6000, 0xa0000000002114e8, 0x9ffc000000abac80, 0x48d) at sys_write+0x100 syscall(0xe000000224d21280, 0x140c70000, 0x10ae, 0xe00000001f4b6000, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command cc1 pid 47335 tid 100146 td 0xe000000015478d80 cpu_switch(0xe000000015478d80, 0xe000000011178900, 0xe000000012402540, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe000000015478d80, 0xe000000011178900, 0x9ffc000000f15698, 0x9ffc000000f15680) at sched_switch+0x890 mi_switch(0x103, 0x0, 0xe000000015478d80, 0x9ffc00000062d1f0) at mi_switch+0x3f0 turnstile_wait(0xe000000012402540, 0xe000000012400000, 0x0, 0x9ffc000000dcb698, 0xe000000015478d80) at turnstile_wait+0x960 __rw_wlock_hard(0x9ffc0000010ffc18, 0xe000000015478d80, 0x0, 0x9ffc000000ecc03c, 0x9ffc0000010ffc00) at __rw_wlock_hard+0x4c0 _rw_wlock_cookie(0x9ffc0000010ffc00, 0x9ffc000000e081f8, 0x68a, 0x9ffc000000ab2720, 0x815) at _rw_wlock_cookie+0x150 pmap_enter(0xe000000011185210, 0x2000000043cf2000, 0x2, 0xe00000027edd1c90, 0x3, 0x0, 0x9ffc000000e081f8, 0xe000000019c0c718) at pmap_enter+0x40 vm_fault_hold(0xe0000000111850d8, 0x2000000043cf2000, 0x2, 0x0, 0x0) at vm_fault_hold+0x2c70 vm_fault(0xe0000000111850d8, 0x2000000043cf2000, 0x2, 0x0, 0xe000000015478d80, 0x9ffc000000abbce0, 0x716, 0xe00000001f2dc5a0) at vm_fault+0xf0 trap(0x14, 0xa00000009df79400) at trap+0x9e0 ivt_Data_TLB() at ivt_Data_TLB+0x1d0 Tracing command xgcc pid 47329 tid 100160 td 0xe000000222c26480 Tracing command sh pid 47176 tid 100110 td 0xe0000000154aad80 Tracing command as pid 46683 tid 100105 td 0xe000000012c3b680 Tracing command cc1plus pid 46682 tid 100115 td 0xe0000000154ba000 Tracing command g++ pid 46681 tid 100116 td 0xe0000000154b4480 Tracing command pkg-static pid 45981 tid 100161 td 0xe000000224c28d80 Tracing command pkg-static pid 45384 tid 100133 td 0xe00000001557e480 Tracing command sh pid 45009 tid 100163 td 0xe000000224c28480 Tracing command make pid 44866 tid 100104 td 0xe000000012c3bb00 Tracing command sh pid 44152 tid 100154 td 0xe00000001f4b7200 Tracing command make pid 43869 tid 100119 td 0xe0000000154abb00 Tracing command tee pid 39289 tid 100139 td 0xe000000015479680 Tracing command sh pid 38577 tid 100128 td 0xe00000001557e900 Tracing command tee pid 37813 tid 100159 td 0xe000000222c26900 Tracing command sh pid 37427 tid 100090 td 0xe000000012a7a480 Tracing command gmake pid 34696 tid 100125 td 0xe0000000154ba480 Tracing command make pid 33402 tid 100075 td 0xe0000000127b4000 Tracing command sh pid 33399 tid 100088 td 0xe000000012a7ad80 Tracing command make pid 33339 tid 100132 td 0xe000000015478000 Tracing command sh pid 33337 tid 100152 td 0xe00000001f4b7b00 Tracing command tee pid 90596 tid 100118 td 0xe0000000154b5b00 Tracing command gmake pid 89091 tid 100169 td 0xe000000224a18000 Tracing command sh pid 88092 tid 100124 td 0xe0000000154ba900 Tracing command gmake pid 84462 tid 100148 td 0xe00000001f4b6900 Tracing command gmake pid 84441 tid 100107 td 0xe000000012c3ad80 Tracing command sh pid 84440 tid 100147 td 0xe000000015478900 Tracing command make pid 84425 tid 100094 td 0xe000000012bbc900 Tracing command sh pid 84423 tid 100076 td 0xe0000000127f6900 Tracing command tee pid 75146 tid 100113 td 0xe0000000154aa000 Tracing command sh pid 74621 tid 100122 td 0xe0000000154bb200 Tracing command top pid 39253 tid 100097 td 0xe0000000127f7680 Tracing command sh pid 35326 tid 100121 td 0xe0000000154bb680 Tracing command sh pid 1151 tid 100072 td 0xe0000000127b4d80 Tracing command su pid 1150 tid 100086 td 0xe000000012a7b680 Tracing command tcsh pid 1146 tid 100099 td 0xe000000012c3a000 Tracing command sshd pid 1145 tid 100078 td 0xe0000000127f6000 Tracing command sshd pid 1143 tid 100069 td 0xe00000001268ed80 Tracing command sh pid 1092 tid 100095 td 0xe000000012bbc480 Tracing command sh pid 1050 tid 100092 td 0xe0000000127f7200 Tracing command su pid 1049 tid 100081 td 0xe000000012a7a000 Tracing command tcsh pid 1045 tid 100098 td 0xe000000012c3a480 Tracing command sshd pid 1044 tid 100085 td 0xe0000000127f6d80 Tracing command sshd pid 1042 tid 100091 td 0xe0000000127f7b00 Tracing command getty pid 1041 tid 100056 td 0xe000000012400d80 Tracing command getty pid 1040 tid 100071 td 0xe0000000127b5200 Tracing command inetd pid 1020 tid 100093 td 0xe000000012bbcd80 Tracing command cron pid 979 tid 100067 td 0xe00000001268f680 Tracing command sendmail pid 974 tid 100066 td 0xe00000001268fb00 Tracing command sendmail pid 971 tid 100073 td 0xe0000000127b4900 Tracing command sshd pid 968 tid 100055 td 0xe000000011a26480 Tracing command ntpd pid 908 tid 100080 td 0xe0000000127b5680 Tracing command syslogd pid 827 tid 100077 td 0xe0000000127f6480 Tracing command ipmon pid 818 tid 100065 td 0xe000000012401200 Tracing command devd pid 696 tid 100068 td 0xe00000001268f200 Tracing command adjkerntz pid 181 tid 100070 td 0xe00000001268e900 Tracing command softdepflush pid 23 tid 100064 td 0xe000000012401680 Tracing command vnlru pid 22 tid 100063 td 0xe000000012401b00 Tracing command syncer pid 21 tid 100062 td 0xe00000001268e000 Tracing command bufdaemon pid 20 tid 100061 td 0xe00000001268e480 Tracing command pagezero pid 19 tid 100060 td 0xe000000011a27b00 Tracing command vmdaemon pid 18 tid 100059 td 0xe000000012400000 Tracing command pagedaemon pid 17 tid 100058 td 0xe000000012400480 Tracing command xpt_thrd pid 9 tid 100057 td 0xe000000012400900 Tracing command sctp_iterator pid 6 tid 100054 td 0xe000000011a26900 Tracing command zfskern pid 5 tid 100052 td 0xe000000011a27200 Tracing command zfskern pid 5 tid 100051 td 0xe000000011a27680 Tracing command isp0: fc_thrd0 pid 4 tid 100047 td 0xe000000011931b00 Tracing command mpt_recovery1 pid 3 tid 100043 td 0xe000000011930000 Tracing command mpt_recovery0 pid 2 tid 100041 td 0xe000000011930900 Tracing command usb pid 16 tid 100039 td 0xe000000011924480 Tracing command usb pid 16 tid 100038 td 0xe000000011924900 Tracing command usb pid 16 tid 100037 td 0xe000000011924d80 Tracing command usb pid 16 tid 100036 td 0xe000000011925200 Tracing command usb pid 16 tid 100034 td 0xe000000011300d80 Tracing command usb pid 16 tid 100033 td 0xe000000011301200 Tracing command usb pid 16 tid 100032 td 0xe000000011301680 Tracing command usb pid 16 tid 100031 td 0xe000000011301b00 Tracing command usb pid 16 tid 100029 td 0xe000000011219680 Tracing command usb pid 16 tid 100028 td 0xe000000011219b00 Tracing command usb pid 16 tid 100027 td 0xe000000011300000 Tracing command usb pid 16 tid 100026 td 0xe000000011300480 Tracing command acpi_thermal pid 15 tid 100024 td 0xe000000011218000 Tracing command rand_harvestq pid 14 tid 100013 td 0xe000000011197680 Tracing command geom pid 13 tid 100012 td 0xe000000011197b00 Tracing command geom pid 13 tid 100011 td 0xe0000000111a6000 Tracing command geom pid 13 tid 100010 td 0xe000000011179680 Tracing command intr pid 12 tid 100048 td 0xe000000011931680 Tracing command intr pid 12 tid 100046 td 0xe000000011a26000 Tracing command intr pid 12 tid 100042 td 0xe000000011930480 Tracing command intr pid 12 tid 100040 td 0xe000000011924000 Tracing command intr pid 12 tid 100035 td 0xe000000011300900 Tracing command intr pid 12 tid 100030 td 0xe000000011219200 Tracing command intr pid 12 tid 100025 td 0xe0000000111a7b00 Tracing command intr pid 12 tid 100022 td 0xe000000011218900 Tracing command intr pid 12 tid 100021 td 0xe000000011218d80 Tracing command intr pid 12 tid 100019 td 0xe0000000111a6900 Tracing command intr pid 12 tid 100015 td 0xe000000011196d80 Tracing command intr pid 12 tid 100008 td 0xe000000011196000 Tracing command intr pid 12 tid 100007 td 0xe000000011196480 Tracing command intr pid 12 tid 100006 td 0xe000000011196900 Tracing command intr pid 12 tid 100005 td 0xe000000011178000 Tracing command idle pid 11 tid 100004 td 0xe000000011178480 Tracing command idle pid 11 tid 100003 td 0xe000000011178900 Tracing command init pid 1 tid 100002 td 0xe000000011178d80 Tracing command audit pid 10 tid 100001 td 0xe000000011179200 Tracing command kernel pid 0 tid 100053 td 0xe000000011a26d80 Tracing command kernel pid 0 tid 100050 td 0xe000000011930d80 Tracing command kernel pid 0 tid 100049 td 0xe000000011931200 Tracing command kernel pid 0 tid 100045 td 0xe000000011925680 Tracing command kernel pid 0 tid 100044 td 0xe000000011925b00 Tracing command kernel pid 0 tid 100023 td 0xe000000011218480 Tracing command kernel pid 0 tid 100020 td 0xe0000000111a6480 Tracing command kernel pid 0 tid 100018 td 0xe0000000111a6d80 Tracing command kernel pid 0 tid 100017 td 0xe0000000111a7200 Tracing command kernel pid 0 tid 100016 td 0xe0000000111a7680 Tracing command kernel pid 0 tid 100014 td 0xe000000011197200 Tracing command kernel pid 0 tid 100009 td 0xe000000011179b00 Tracing command kernel pid 0 tid 100000 td 0x9ffc000000eee1d0 db>