From owner-freebsd-ia64@FreeBSD.ORG Mon Oct 14 08:21:38 2013 Return-Path: Delivered-To: freebsd-ia64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 708895A6 for ; Mon, 14 Oct 2013 08:21:38 +0000 (UTC) (envelope-from mexas@bristol.ac.uk) Received: from eu1sys200aog107.obsmtp.com (eu1sys200aog107.obsmtp.com [207.126.144.123]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4BD7122A5 for ; Mon, 14 Oct 2013 08:21:37 +0000 (UTC) Received: from mail-wg0-f41.google.com ([74.125.82.41]) (using TLSv1) by eu1sys200aob107.postini.com ([207.126.147.11]) with SMTP ID DSNKUlupdXHnnkt5x5lzGnf/bHDv6Caxf41g@postini.com; Mon, 14 Oct 2013 08:21:37 UTC Received: by mail-wg0-f41.google.com with SMTP id b13so1862102wgh.4 for ; Mon, 14 Oct 2013 01:21:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:message-id:to:subject:cc :reply-to:in-reply-to; bh=uwxZb7PVjaWjNVMWUkK8TOj0iG9WB541xcI2oF5tAK0=; b=SbBJqrLGdRXVIEDewYQ+Rg/U8xa9goLBLdz2A3PioW09LX4nX0zVFr9Tc3gzdtQxYJ dXPuGJSlStQkkui1z6QAODXiiLeoqdEcUfoLr+0a6ITmiTZUJhS7eR9l7X4MShYkAfpc tQZMs8Qz2Eg4VbE2fTHExWBlZS8DbBrNPJDxUeZs7Dg7/LjSNlcrd6rAormSkQ+zNwK0 KDGZjKfXzmALurx5BuW/pFPF1ZBRD+BOgg6y84RvrUTk3Gi69bodBazOB8jv6fdsVdDI mTEgv064IKKCccD7y1z5rgLdwwg0BLOOlgc23E+87EdsSHlhwl9zjwzZeaHSHXpvMBpU m1wg== X-Gm-Message-State: ALoCoQk9oRWw+NYe0VbKQqegWeVy9hBlsq2A07qKGK3Tzqape4he/7KxsjKnWp39J6Wkj4oBUmfrk3dDiYYsVe1Kn+lr7ulIRwCixT7z3Pylp8zAzTGhIm/ainEnO2klbb08tZYgqasrFdvOaLB19KVjniKvHYfHNSBCp5/uoz+HY4LaoEd2vZk= X-Received: by 10.180.208.97 with SMTP id md1mr13580950wic.41.1381738869057; Mon, 14 Oct 2013 01:21:09 -0700 (PDT) X-Received: by 10.180.208.97 with SMTP id md1mr13580938wic.41.1381738868955; Mon, 14 Oct 2013 01:21:08 -0700 (PDT) Received: from mech-cluster241.men.bris.ac.uk (mech-cluster241.men.bris.ac.uk. [137.222.187.241]) by mx.google.com with ESMTPSA id jf2sm31806314wic.2.1969.12.31.16.00.00 (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 14 Oct 2013 01:21:07 -0700 (PDT) Sender: Anton Shterenlikht Received: from mech-cluster241.men.bris.ac.uk (localhost [127.0.0.1]) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6) with ESMTP id r9E8L5LZ014898 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 14 Oct 2013 09:21:05 +0100 (BST) (envelope-from mexas@mech-cluster241.men.bris.ac.uk) Received: (from mexas@localhost) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6/Submit) id r9E8L55j014897; Mon, 14 Oct 2013 09:21:05 +0100 (BST) (envelope-from mexas) Date: Mon, 14 Oct 2013 09:21:05 +0100 (BST) From: Anton Shterenlikht Message-Id: <201310140821.r9E8L55j014897@mech-cluster241.men.bris.ac.uk> To: davide@freebsd.org, mexas@bris.ac.uk Subject: Re: panic: ia64 r255811: deadlkres: possible deadlock detected for 0xe000000012d07b00, blocked for 902743 ticks In-Reply-To: Cc: freebsd-current@freebsd.org, freebsd-ia64@freebsd.org X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: mexas@bris.ac.uk List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Oct 2013 08:21:38 -0000 >From davide.italiano@gmail.com Fri Oct 11 15:39:49 2013 > >If you're not able to get a full dump, a textdump would be enough. >In your DDB scripts just remove the 'call doadump' step and you should >be done, unless I'm missing something. >Please adjust the script as well to include all the informations >requested as mentioned in my previous link, e.g. 'show lockedvnods' is >not mentioned on the example section of textdump(4) manpage but it >could be useful to ease debugging. I still haven't got it right, but I now collected manually some (hopefully) useful info (see below). - here's my /etc/ddb.conf: script lockinfo=show locks; show alllocks; show lockedvnods script kdb.enter.panic=textdump set; capture on; run lockinfo; show locks; show alllocks; show lockedvnods; show pcpu; bt; ps; alltrace; capture off; reset script kdb.enter.witness=run lockinfo Somehow when I get into a debugger, I can only see lockinfo. Why? - I've removed "call doadump", as you see above. Still the dump is being written. I've in /etc/rc.conf: dumpdev=/dev/da4p1 dumpdir=/var/crash ddb_enable=YES Shall I remove dumpdev completely? And leave only dumpdir defined? Thanks Anton db> textdump set textdump set db> capture on db> run lockinfo db:0:lockinfo> show locks db:0:locks> show alllocks db:0:alllocks> show lockedvnods Locked vnodes 0xe0000000123799d0: tag devfs, type VCHR usecount 1, writecount 0, refcount 408 mountedhere 0xe0000000126a4c00 flags (VI_ACTIVE) v_object 0xe000000012788100 ref 0 pages 21330 lock type devfs: EXCL by thread 0xe00000001268e000 (pid 21, syncer, tid 100062) dev da5p1 0xe0000000127d5448: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0xe00000001277bc00 ref 0 pages 1 lock type ufs: EXCL by thread 0xe0000000127b5680 (pid 908, ntpd, tid 100080) ino 2889216, on dev da3p1 0xe0000002203c0ce8: tag ufs, type VREG usecount 1, writecount 1, refcount 16 mountedhere 0 flags (VI_ACTIVE) v_object 0xe000000223037700 ref 0 pages 56 lock type ufs: EXCL by thread 0xe00000001557e480 (pid 45384, pkg-static, tid 100133) ino 5780554, on dev da5p1 db> db> show pcpu cpuid = 0 dynamic pcpu = 0x40040040ff13ca80 curthread = 0xe000000011a26d80: pid 0 "deadlkres" curpcb = 0 fpcurthread = none idlethread = 0xe000000011178900: tid 100003 "idle: cpu0" MD: vhpt = 0xe0000040ff000000 MD: lid = 0 MD: clock = 0x214563ccc206 MD: clock_mode = 2 MD: clock_load = 0 MD: stats = 0x9ffc0000010ff130 MD: pmap = 0x9ffc000000eee760 spin locks held: db> db> bt Tracing pid 0 tid 100053 td 0xe000000011a26d80 kdb_enter(0x9ffc000000dc2fd0, 0x9ffc000000dc2fd0, 0x9ffc00000058e6b0, 0x40c) at kdb_enter+0x92 vpanic(0x9ffc000000db8a18, 0xa00000009dca7518) at vpanic+0x2b0 panic(0x9ffc000000db8a18, 0x9ffc000000db8c70, 0xe00000001557fb00, 0xdbd38) at panic+0x80 deadlkres(0xdbd38, 0xe00000001557fb00, 0x9ffc000000dbb648, 0x9ffc000000db89a8) at deadlkres+0x420 fork_exit(0x9ffc000000e0fca0, 0x0, 0xa00000009dca7550) at fork_exit+0x120 enter_userland() at enter_userland db> db> ps pid ppid pgrp uid state wmesg wchan cmd 47482 47176 1092 0 LV+J *pmap pv 0xe000000012402540 sh 47476 35326 1092 0 D+ vm map ( 0xe0000000110adac0 sh 47475 35326 1092 0 S+ piperd 0xe000000222b2f5a8 awk 47474 35326 1092 0 L+ *pmap pv 0xe000000012402540 awk 47335 47329 1092 0 LL+J *pmap pv 0xe000000012402540 cc1 47329 34696 1092 0 SW+J wait 0xe0000002233c3720 xgcc 47176 45981 1092 0 D+J ppwait 0xe0000000156af6d0 sh 46683 46681 1092 0 S+J piperd 0xe0000000127e3340 as 46682 46681 1092 0 LL+J *pmap pv 0xe000000012402540 cc1plus 46681 33402 1092 0 SW+J wait 0xe0000000154c4000 g++ 45981 44152 1092 0 SW+J wait 0xe0000002233c3280 pkg-static 45384 45009 1092 0 L+J *vm page 0xe00000001eff4f00 pkg-static 45009 44866 1092 0 SW+J wait 0xe0000002233c2940 sh 44866 38577 1092 0 SW+J wait 0xe00000001547cde0 make 44152 43869 1092 0 SW+J wait 0xe00000001f2dcde0 sh 43869 37427 1092 0 SW+J wait 0xe00000001549d280 make 39289 38577 1092 0 S+ piperd 0xe00000001556d0d8 tee 38577 1092 1092 0 SW+ wait 0xe0000000154f1280 sh 37813 37427 1092 0 S+ piperd 0xe000000222b2f810 tee 37427 1092 1092 0 SW+ wait 0xe000000012b3c4a0 sh 34696 89091 1092 0 SW+J wait 0xe00000001556e000 gmake 33402 33399 1092 0 SW+J wait 0xe00000001280e4a0 make 33399 33339 1092 0 SW+J wait 0xe0000000129e1720 sh 33339 33337 1092 0 SW+J wait 0xe00000001556e4a0 make 33337 88092 1092 0 SW+ wait 0xe00000001f2dd720 sh 90596 88092 1092 0 S+ piperd 0xe0000000127e3ce0 tee 89091 84462 1092 0 SW+J wait 0xe000000224d20000 gmake 88092 1092 1092 0 D+ vm map ( 0xe0000000110ad9d0 sh 84462 84441 1092 0 SW+J wait 0xe00000001efcd720 gmake 84441 84440 1092 0 SW+J wait 0xe00000001547c000 gmake 84440 84425 1092 0 SW+J wait 0xe00000001f2dc000 sh 84425 84423 1092 0 SW+J wait 0xe000000012b31720 make 84423 74621 1092 0 SW+ wait 0xe00000001280e000 sh 75146 74621 1092 0 S+ piperd 0xe000000012b54000 tee 74621 1092 1092 0 D+ vm map ( 0xe0000000110ad9d0 sh 39253 1151 39253 0 L+ *pmap pv 0xe000000012402540 top 35326 1092 1092 0 S+ wait 0xe0000000154f0000 sh 1151 1150 1151 0 SW+ wait 0xe00000001280f280 sh 1150 1146 1150 1001 SW+ wait 0xe000000012b304a0 su 1146 1145 1146 1001 SWs+ pause 0xe000000012b3c9e8 tcsh 1145 1143 1143 1001 S select 0xe0000000126d4f40 sshd 1143 968 1143 0 Ss select 0xe000000010c299c0 sshd 1092 1050 1092 0 LL+ *pmap pv 0xe000000012402540 sh 1050 1049 1050 0 SW+ wait 0xe000000012b3d720 sh 1049 1045 1049 1001 SW+ wait 0xe00000001280f720 su 1045 1044 1045 1001 SWs+ pause 0xe000000012b3ce88 tcsh 1044 1042 1042 1001 S select 0xe000000010c28e40 sshd 1042 968 1042 0 Ss select 0xe0000000126d5040 sshd 1041 1 1041 0 Ss+ ttyin 0xe00000001121c8a8 getty 1040 1 1 0 S ttydcd 0xe00000001121cce8 getty 1020 1 1020 0 Ss select 0xe000000010c290c0 inetd 979 1 979 0 LLs *pmap pv 0xe000000012402540 cron 974 1 974 25 LLs *pmap pv 0xe000000012402540 sendmail 971 1 971 0 LLs *pmap pv 0xe000000012402540 sendmail 968 1 968 0 Ss select 0xe0000000126d54c0 sshd 908 1 908 0 Ls *vm page 0xe00000001eff4f00 ntpd 827 1 827 0 Ss select 0xe0000000126d56c0 syslogd 818 1 818 0 Ss select 0xe0000000126d5740 ipmon 696 1 696 0 Ss select 0xe0000000126d57c0 devd 181 1 181 0 SWs pause 0xe00000001269a9e8 adjkerntz 23 0 0 0 DL sdflush 0x9ffc000000ed1ff0 [softdepflush] 22 0 0 0 DL vlruwt 0xe000000012409720 [vnlru] 21 0 0 0 LL *vm page 0xe00000001eff4f00 [syncer] 20 0 0 0 DL psleep 0x9ffc000000ed1390 [bufdaemon] 19 0 0 0 DL pgzero 0x9ffc000000ed239c [pagezero] 18 0 0 0 RL [vmdaemon] 17 0 0 0 LL *pmap pv 0xe000000012402540 [pagedaemon] 9 0 0 0 DL ccb_scan 0x9ffc000000ed26a0 [xpt_thrd] 6 0 0 0 DL waiting_ 0x9ffc0000010f33b0 [sctp_iterator] 5 0 0 0 RL (threaded) [zfskern] 100052 Run CPU 1 [l2arc_feed_thread] 100051 D arc_recl 0x9ffc000001538120 [arc_reclaim_thread] 4 0 0 0 DL ispf 0xe000000011a02000 [isp0: fc_thrd0] 3 0 0 0 DL idle 0xa000000000a16000 [mpt_recovery1] 2 0 0 0 DL idle 0xa0000000009fe000 [mpt_recovery0] 16 0 0 0 DL (threaded) [usb] 100039 D - 0xa0000000009fae18 [usbus2] 100038 D - 0xa0000000009fadc0 [usbus2] 100037 D - 0xa0000000009fad68 [usbus2] 100036 D - 0xa0000000009fad10 [usbus2] 100034 D - 0xa0000000009f1460 [usbus1] 100033 D - 0xa0000000009f1408 [usbus1] 100032 D - 0xa0000000009f13b0 [usbus1] 100031 D - 0xa0000000009f1358 [usbus1] 100029 D - 0xa0000000009ed460 [usbus0] 100028 D - 0xa0000000009ed408 [usbus0] 100027 D - 0xa0000000009ed3b0 [usbus0] 100026 D - 0xa0000000009ed358 [usbus0] 15 0 0 0 DL tzpoll 0x9ffc000000ecfd20 [acpi_thermal] 14 0 0 0 DL - 0x9ffc000000ed0008 [rand_harvestq] 13 0 0 0 DL (threaded) [geom] 100012 D - 0x9ffc000000ed0540 [g_down] 100011 D - 0x9ffc000000ed0538 [g_up] 100010 D - 0x9ffc000000ed0528 [g_event] 12 0 0 0 WL (threaded) [intr] 100048 I [swi0: uart uart+++] 100046 I [irq71: isp0] 100042 I [irq28: mpt1] 100040 I [irq27: mpt0] 100035 I [irq18: ehci0] 100030 I [irq17: ohci1] 100025 I [irq16: ohci0] 100022 I [swi6: task queue] 100021 I [swi6: Giant taskq] 100019 I [swi5: fast taskq] 100015 I [swi2: cambio] 100008 I [swi4: clock] 100007 I [swi4: clock] 100006 I [swi3: vm] 100005 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100004 CanRun [idle: cpu1] 100003 CanRun [idle: cpu0] 1 0 1 0 SLs wait 0xe000000011176de0 [init] 10 0 0 0 DL audit_wo 0x9ffc0000010f7a48 [audit] 0 0 0 0 RLs (threaded) [kernel] 100053 Run CPU 0 [deadlkres] 100050 D - 0xe000000011a2d300 [system_taskq_1] 100049 D - 0xe000000011a2d300 [system_taskq_0] 100045 D - 0xe000000011198500 [em1 taskq] 100044 D - 0xe000000011198600 [em0 taskq] 100023 D - 0xe00000001119b300 [ffs_trim taskq] 100020 D - 0xe00000001119b600 [thread taskq] 100018 D - 0xe00000001119b800 [acpi_task_2] 100017 D - 0xe00000001119b800 [acpi_task_1] 100016 D - 0xe00000001119b800 [acpi_task_0] 100014 D - 0xe00000001119ba00 [kqueue taskq] 100009 D - 0xe00000001119bd00 [firmware taskq] 100000 D swapin 0x9ffc000000eedd28 [swapper] db> db> alltrace Tracing command sh pid 47482 tid 100135 td 0xe00000001557fb00 cpu_switch(0xe00000001557fb00, 0xe000000015478d80, 0xe000000012402540, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe00000001557fb00, 0xe000000015478d80, 0x9ffc000000f15698, 0x9ffc000000f15680) at sched_switch+0x890 mi_switch(0x103, 0x0, 0xe00000001557fb00, 0x9ffc00000062d1f0) at mi_switch+0x3f0 turnstile_wait(0xe000000012402540, 0xe000000012400000, 0x0, 0x9ffc000000dcb698, 0xe00000001557fb00) at turnstile_wait+0x960 __rw_wlock_hard(0x9ffc0000010ffc18, 0xe00000001557fb00, 0x0, 0x9ffc000000ecc03c, 0x9ffc0000010ffc00) at __rw_wlock_hard+0x4c0 _rw_wlock_cookie(0x9ffc0000010ffc00, 0x9ffc000000e081f8, 0x68a, 0x9ffc000000ab2720, 0x815) at _rw_wlock_cookie+0x150 pmap_enter(0x9ffc0000010ffc80, 0xa0000000e2802000, 0x2, 0xe00000027edd1590, 0x7, 0x0, 0x9ffc000000e081f8, 0xe000000019c28f18) at pmap_enter+0x40 vm_fault_hold(0xe000000010000000, 0xa0000000e2802000, 0x2, 0x0, 0x0) at vm_fault_hold+0x2c70 vm_fault(0xe000000010000000, 0xa0000000e2802000, 0x2, 0x0, 0xe00000001557fb00, 0x9ffc000000abbd60, 0x716, 0x30a) at vm_fault+0xf0 trap(0x14, 0xa00000009df21000) at trap+0xa60 ivt_Data_TLB() at ivt_Data_TLB+0x1d0 --- trapframe at 0xa00000009df21000 copystr() at copystr+0x11 copyinstr(0x140c0cf70, 0xa0000000e2802000, 0x400, 0xa00000009df21350) at copyinstr+0xa0 exec_copyin_args(0xa00000009df21360, 0x140c0cf70, 0x0, 0x140c0ce18, 0x140c0ce40) at exec_copyin_args+0xf0 sys_execve(0xe00000001557fb00, 0xa00000009df214e8, 0x9ffc000000abac80, 0x48d) at sys_execve+0x40 syscall(0xe0000000156af280, 0x140c0ce18, 0x140c0ce40, 0xe00000001557fb00, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sh pid 47476 tid 100138 td 0xe000000015479b00 cpu_switch(0xe000000015479b00, 0xe00000001557e480, 0x9ffc000000f4b7b0, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe000000015479b00, 0xe00000001557e480, 0x9ffc000000f16398, 0x9ffc000000f16380) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000015479b00, 0x9ffc000000620b70) at mi_switch+0x3f0 sleepq_switch(0xe0000000110adac0, 0x0, 0xe000000015479e78, 0xe000000015479b00) at sleepq_switch+0x2e0 sleepq_wait(0xe0000000110adac0, 0x0, 0x9ffc000000dca928, 0xe000000015479b00, 0x9ffc00000059ead0) at sleepq_wait+0xb0 _sx_xlock_hard(0xe0000000110adac0, 0xe000000015479b00, 0x0, 0xe0000000110adad8, 0xe0000000110adac0) at _sx_xlock_hard+0x8d0 _sx_xlock(0xe0000000110adac0, 0x0, 0x9ffc000000dfcf30, 0x1c2) at _sx_xlock+0x170 _vm_map_lock(0xe0000000110ada40, 0x9ffc000000dfcf30, 0x1c2, 0x9ffc000000a3fa50, 0x593, 0xe000000010c54f00) at _vm_map_lock+0x80 kmap_alloc_wait(0xe0000000110ada40, 0x42000) at kmap_alloc_wait+0x90 exec_alloc_args(0xa00000009df39360, 0x9ffc0000005253e0, 0x610, 0x140308ed0) at exec_alloc_args+0x30 exec_copyin_args(0xa00000009df39360, 0x140c148b0, 0x0, 0x140c14578, 0x140c14810) at exec_copyin_args+0x60 sys_execve(0xe000000015479b00, 0xa00000009df394e8, 0x9ffc000000abac80, 0x48d) at sys_execve+0x40 syscall(0xe0000000156ae4a0, 0x140c14578, 0x140c14810, 0xe000000015479b00, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command awk pid 47475 tid 100131 td 0xe000000015478480 cpu_switch(0xe000000015478480, 0xe00000001557fb00, 0x9ffc000000f4c228, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe000000015478480, 0xe00000001557fb00, 0x9ffc000000f16398, 0x9ffc000000f16380) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000015478480, 0x9ffc000000620b70) at mi_switch+0x3f0 sleepq_switch(0xe000000222b2f5a8, 0x5c, 0xe0000000154787f8, 0xe000000015478480) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000222b2f5a8, 0x5c, 0x9ffc000000dca928, 0x9ffc000000dcace8) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000222b2f5a8, 0x5c, 0x9ffc000000dce2b0, 0x100, 0x9ffc0000005a0a90) at sleepq_wait_sig+0x30 _sleep(0xe000000222b2f5a8, 0xe000000222b2f7e8, 0x5c, 0x9ffc000000dce2b0, 0x0, 0x0, 0x100) at _sleep+0x7b0 pipe_read(0xe00000001edaa9d4, 0xa00000009df01360, 0xa00000009df01378, 0x0, 0xe000000222b2f6b0) at pipe_read+0x740 dofileread(0xe000000015478480, 0x0, 0xe00000001edaa9b0, 0xa00000009df01360, 0x0, 0x0) at dofileread+0x150 kern_readv(0xe000000015478480, 0x0, 0xa00000009df01360) at kern_readv+0xa0 sys_read(0xe000000015478480, 0xa00000009df014e8, 0x9ffc000000abac80, 0x48d) at sys_read+0x100 syscall(0xe00000001556e940, 0x140c8a000, 0x2000, 0xe000000015478480, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command awk pid 47474 tid 100165 td 0xe00000001f4b6000 cpu_switch(0xe00000001f4b6000, 0xe000000011197680, 0xe000000012402540, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe00000001f4b6000, 0xe000000011197680, 0x9ffc000000f15698, 0x9ffc000000f15680) at sched_switch+0x890 mi_switch(0x103, 0x0, 0xe00000001f4b6000, 0x9ffc00000062d1f0) at mi_switch+0x3f0 turnstile_wait(0xe000000012402540, 0xe000000012400000, 0x0, 0x9ffc000000dcb698, 0xe00000001f4b6000) at turnstile_wait+0x960 __rw_wlock_hard(0x9ffc0000010ffc18, 0xe00000001f4b6000, 0x0, 0x9ffc000000ecc03c, 0x9ffc0000010ffc00) at __rw_wlock_hard+0x4c0 _rw_wlock_cookie(0x9ffc0000010ffc00, 0x9ffc000000e081f8, 0x68a, 0x9ffc000000ab2720, 0x815) at _rw_wlock_cookie+0x150 pmap_enter(0x9ffc0000010ffc80, 0xa0000000e2c4e000, 0x2, 0xe00000027edd13d0, 0x7, 0x0, 0x9ffc000000e081f8, 0xe00000001b1a6918) at pmap_enter+0x40 vm_fault_hold(0xe000000010000000, 0xa0000000e2c4e000, 0x2, 0x0, 0x0) at vm_fault_hold+0x2c70 vm_fault(0xe000000010000000, 0xa0000000e2c4e000, 0x2, 0x0, 0xe00000001f4b6000, 0x9ffc000000abbd60, 0x716, 0xe000000222b2f5a8) at vm_fault+0xf0 trap(0x14, 0xa000000000211000) at trap+0xa60 ivt_Data_TLB() at ivt_Data_TLB+0x1d0 --- trapframe at 0xa000000000211000 bcopy() at bcopy+0xe1 copyin(0x140c70000, 0xa0000000e2c4e000, 0x10ae) at copyin+0x90 uiomove_faultflag(0xa0000000e2c4e000, 0x10ae, 0xa000000000211360, 0xa000000000211390) at uiomove_faultflag+0x2a0 uiomove(0xa0000000e2c4e000, 0x10ae, 0xa000000000211360, 0x9ffc00000064dc40, 0x1939, 0x9ffc00000064dc10) at uiomove+0x30 pipe_write(0xe0000000127220c4, 0xa000000000211360, 0xe000000222b2f6b0, 0x10ae, 0x10ae) at pipe_write+0x1bd0 dofilewrite(0xe00000001f4b6000, 0x1, 0xe0000000127220a0, 0xa000000000211360, 0xffffffffffffffff, 0x0) at dofilewrite+0x180 kern_writev(0xe00000001f4b6000, 0x1, 0xa000000000211360) at kern_writev+0xa0 sys_write(0xe00000001f4b6000, 0xa0000000002114e8, 0x9ffc000000abac80, 0x48d) at sys_write+0x100 syscall(0xe000000224d21280, 0x140c70000, 0x10ae, 0xe00000001f4b6000, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command cc1 pid 47335 tid 100146 td 0xe000000015478d80 cpu_switch(0xe000000015478d80, 0xe000000011178900, 0xe000000012402540, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe000000015478d80, 0xe000000011178900, 0x9ffc000000f15698, 0x9ffc000000f15680) at sched_switch+0x890 mi_switch(0x103, 0x0, 0xe000000015478d80, 0x9ffc00000062d1f0) at mi_switch+0x3f0 turnstile_wait(0xe000000012402540, 0xe000000012400000, 0x0, 0x9ffc000000dcb698, 0xe000000015478d80) at turnstile_wait+0x960 __rw_wlock_hard(0x9ffc0000010ffc18, 0xe000000015478d80, 0x0, 0x9ffc000000ecc03c, 0x9ffc0000010ffc00) at __rw_wlock_hard+0x4c0 _rw_wlock_cookie(0x9ffc0000010ffc00, 0x9ffc000000e081f8, 0x68a, 0x9ffc000000ab2720, 0x815) at _rw_wlock_cookie+0x150 pmap_enter(0xe000000011185210, 0x2000000043cf2000, 0x2, 0xe00000027edd1c90, 0x3, 0x0, 0x9ffc000000e081f8, 0xe000000019c0c718) at pmap_enter+0x40 vm_fault_hold(0xe0000000111850d8, 0x2000000043cf2000, 0x2, 0x0, 0x0) at vm_fault_hold+0x2c70 vm_fault(0xe0000000111850d8, 0x2000000043cf2000, 0x2, 0x0, 0xe000000015478d80, 0x9ffc000000abbce0, 0x716, 0xe00000001f2dc5a0) at vm_fault+0xf0 trap(0x14, 0xa00000009df79400) at trap+0x9e0 ivt_Data_TLB() at ivt_Data_TLB+0x1d0 Tracing command xgcc pid 47329 tid 100160 td 0xe000000222c26480 Tracing command sh pid 47176 tid 100110 td 0xe0000000154aad80 Tracing command as pid 46683 tid 100105 td 0xe000000012c3b680 Tracing command cc1plus pid 46682 tid 100115 td 0xe0000000154ba000 Tracing command g++ pid 46681 tid 100116 td 0xe0000000154b4480 Tracing command pkg-static pid 45981 tid 100161 td 0xe000000224c28d80 Tracing command pkg-static pid 45384 tid 100133 td 0xe00000001557e480 Tracing command sh pid 45009 tid 100163 td 0xe000000224c28480 Tracing command make pid 44866 tid 100104 td 0xe000000012c3bb00 Tracing command sh pid 44152 tid 100154 td 0xe00000001f4b7200 Tracing command make pid 43869 tid 100119 td 0xe0000000154abb00 Tracing command tee pid 39289 tid 100139 td 0xe000000015479680 Tracing command sh pid 38577 tid 100128 td 0xe00000001557e900 Tracing command tee pid 37813 tid 100159 td 0xe000000222c26900 Tracing command sh pid 37427 tid 100090 td 0xe000000012a7a480 Tracing command gmake pid 34696 tid 100125 td 0xe0000000154ba480 Tracing command make pid 33402 tid 100075 td 0xe0000000127b4000 Tracing command sh pid 33399 tid 100088 td 0xe000000012a7ad80 Tracing command make pid 33339 tid 100132 td 0xe000000015478000 Tracing command sh pid 33337 tid 100152 td 0xe00000001f4b7b00 Tracing command tee pid 90596 tid 100118 td 0xe0000000154b5b00 Tracing command gmake pid 89091 tid 100169 td 0xe000000224a18000 Tracing command sh pid 88092 tid 100124 td 0xe0000000154ba900 Tracing command gmake pid 84462 tid 100148 td 0xe00000001f4b6900 Tracing command gmake pid 84441 tid 100107 td 0xe000000012c3ad80 Tracing command sh pid 84440 tid 100147 td 0xe000000015478900 Tracing command make pid 84425 tid 100094 td 0xe000000012bbc900 Tracing command sh pid 84423 tid 100076 td 0xe0000000127f6900 Tracing command tee pid 75146 tid 100113 td 0xe0000000154aa000 Tracing command sh pid 74621 tid 100122 td 0xe0000000154bb200 Tracing command top pid 39253 tid 100097 td 0xe0000000127f7680 Tracing command sh pid 35326 tid 100121 td 0xe0000000154bb680 Tracing command sh pid 1151 tid 100072 td 0xe0000000127b4d80 Tracing command su pid 1150 tid 100086 td 0xe000000012a7b680 Tracing command tcsh pid 1146 tid 100099 td 0xe000000012c3a000 Tracing command sshd pid 1145 tid 100078 td 0xe0000000127f6000 Tracing command sshd pid 1143 tid 100069 td 0xe00000001268ed80 Tracing command sh pid 1092 tid 100095 td 0xe000000012bbc480 Tracing command sh pid 1050 tid 100092 td 0xe0000000127f7200 Tracing command su pid 1049 tid 100081 td 0xe000000012a7a000 Tracing command tcsh pid 1045 tid 100098 td 0xe000000012c3a480 Tracing command sshd pid 1044 tid 100085 td 0xe0000000127f6d80 Tracing command sshd pid 1042 tid 100091 td 0xe0000000127f7b00 Tracing command getty pid 1041 tid 100056 td 0xe000000012400d80 Tracing command getty pid 1040 tid 100071 td 0xe0000000127b5200 Tracing command inetd pid 1020 tid 100093 td 0xe000000012bbcd80 Tracing command cron pid 979 tid 100067 td 0xe00000001268f680 Tracing command sendmail pid 974 tid 100066 td 0xe00000001268fb00 Tracing command sendmail pid 971 tid 100073 td 0xe0000000127b4900 Tracing command sshd pid 968 tid 100055 td 0xe000000011a26480 Tracing command ntpd pid 908 tid 100080 td 0xe0000000127b5680 Tracing command syslogd pid 827 tid 100077 td 0xe0000000127f6480 Tracing command ipmon pid 818 tid 100065 td 0xe000000012401200 Tracing command devd pid 696 tid 100068 td 0xe00000001268f200 Tracing command adjkerntz pid 181 tid 100070 td 0xe00000001268e900 Tracing command softdepflush pid 23 tid 100064 td 0xe000000012401680 Tracing command vnlru pid 22 tid 100063 td 0xe000000012401b00 Tracing command syncer pid 21 tid 100062 td 0xe00000001268e000 Tracing command bufdaemon pid 20 tid 100061 td 0xe00000001268e480 Tracing command pagezero pid 19 tid 100060 td 0xe000000011a27b00 Tracing command vmdaemon pid 18 tid 100059 td 0xe000000012400000 Tracing command pagedaemon pid 17 tid 100058 td 0xe000000012400480 Tracing command xpt_thrd pid 9 tid 100057 td 0xe000000012400900 Tracing command sctp_iterator pid 6 tid 100054 td 0xe000000011a26900 Tracing command zfskern pid 5 tid 100052 td 0xe000000011a27200 Tracing command zfskern pid 5 tid 100051 td 0xe000000011a27680 Tracing command isp0: fc_thrd0 pid 4 tid 100047 td 0xe000000011931b00 Tracing command mpt_recovery1 pid 3 tid 100043 td 0xe000000011930000 Tracing command mpt_recovery0 pid 2 tid 100041 td 0xe000000011930900 Tracing command usb pid 16 tid 100039 td 0xe000000011924480 Tracing command usb pid 16 tid 100038 td 0xe000000011924900 Tracing command usb pid 16 tid 100037 td 0xe000000011924d80 Tracing command usb pid 16 tid 100036 td 0xe000000011925200 Tracing command usb pid 16 tid 100034 td 0xe000000011300d80 Tracing command usb pid 16 tid 100033 td 0xe000000011301200 Tracing command usb pid 16 tid 100032 td 0xe000000011301680 Tracing command usb pid 16 tid 100031 td 0xe000000011301b00 Tracing command usb pid 16 tid 100029 td 0xe000000011219680 Tracing command usb pid 16 tid 100028 td 0xe000000011219b00 Tracing command usb pid 16 tid 100027 td 0xe000000011300000 Tracing command usb pid 16 tid 100026 td 0xe000000011300480 Tracing command acpi_thermal pid 15 tid 100024 td 0xe000000011218000 Tracing command rand_harvestq pid 14 tid 100013 td 0xe000000011197680 Tracing command geom pid 13 tid 100012 td 0xe000000011197b00 Tracing command geom pid 13 tid 100011 td 0xe0000000111a6000 Tracing command geom pid 13 tid 100010 td 0xe000000011179680 Tracing command intr pid 12 tid 100048 td 0xe000000011931680 Tracing command intr pid 12 tid 100046 td 0xe000000011a26000 Tracing command intr pid 12 tid 100042 td 0xe000000011930480 Tracing command intr pid 12 tid 100040 td 0xe000000011924000 Tracing command intr pid 12 tid 100035 td 0xe000000011300900 Tracing command intr pid 12 tid 100030 td 0xe000000011219200 Tracing command intr pid 12 tid 100025 td 0xe0000000111a7b00 Tracing command intr pid 12 tid 100022 td 0xe000000011218900 Tracing command intr pid 12 tid 100021 td 0xe000000011218d80 Tracing command intr pid 12 tid 100019 td 0xe0000000111a6900 Tracing command intr pid 12 tid 100015 td 0xe000000011196d80 Tracing command intr pid 12 tid 100008 td 0xe000000011196000 Tracing command intr pid 12 tid 100007 td 0xe000000011196480 Tracing command intr pid 12 tid 100006 td 0xe000000011196900 Tracing command intr pid 12 tid 100005 td 0xe000000011178000 Tracing command idle pid 11 tid 100004 td 0xe000000011178480 Tracing command idle pid 11 tid 100003 td 0xe000000011178900 Tracing command init pid 1 tid 100002 td 0xe000000011178d80 Tracing command audit pid 10 tid 100001 td 0xe000000011179200 Tracing command kernel pid 0 tid 100053 td 0xe000000011a26d80 Tracing command kernel pid 0 tid 100050 td 0xe000000011930d80 Tracing command kernel pid 0 tid 100049 td 0xe000000011931200 Tracing command kernel pid 0 tid 100045 td 0xe000000011925680 Tracing command kernel pid 0 tid 100044 td 0xe000000011925b00 Tracing command kernel pid 0 tid 100023 td 0xe000000011218480 Tracing command kernel pid 0 tid 100020 td 0xe0000000111a6480 Tracing command kernel pid 0 tid 100018 td 0xe0000000111a6d80 Tracing command kernel pid 0 tid 100017 td 0xe0000000111a7200 Tracing command kernel pid 0 tid 100016 td 0xe0000000111a7680 Tracing command kernel pid 0 tid 100014 td 0xe000000011197200 Tracing command kernel pid 0 tid 100009 td 0xe000000011179b00 Tracing command kernel pid 0 tid 100000 td 0x9ffc000000eee1d0 db> From owner-freebsd-ia64@FreeBSD.ORG Mon Oct 14 08:44:39 2013 Return-Path: Delivered-To: freebsd-ia64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 8C997C04 for ; Mon, 14 Oct 2013 08:44:39 +0000 (UTC) (envelope-from mexas@bristol.ac.uk) Received: from eu1sys200aog111.obsmtp.com (eu1sys200aog111.obsmtp.com [207.126.144.131]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C0BD32408 for ; Mon, 14 Oct 2013 08:44:38 +0000 (UTC) Received: from mail-wg0-f43.google.com ([74.125.82.43]) (using TLSv1) by eu1sys200aob111.postini.com ([207.126.147.11]) with SMTP ID DSNKUluu7zONS/qXr9O4+Z3aV34Jptf9ZLtc@postini.com; Mon, 14 Oct 2013 08:44:38 UTC Received: by mail-wg0-f43.google.com with SMTP id b13so2706899wgh.34 for ; Mon, 14 Oct 2013 01:44:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:message-id:to:subject:reply-to; bh=2E6rMLSY6Ur4pvRQn3h6lX1kOOtQD3S/oaoQDfByWvc=; b=N/1F2zUs++/Ga4PZIdP9tvtiErE9ZisCYr/VKI+vS5vk++923tjdfS3t8gc0eswh8N 5uB4vsxZ5Rf9P82FUlfvrL5sQXBxUmbzaK0EazvNFbqVNBTshmPuo86Nx0JdUKkUtvAf R4N983pJ3vJQF/1TV4zSpVbqagZmywU07pgJVEyiwHxGLLfv6cTB/zgLEEJzXOQZZtZJ SyLtWB5IsxSFp1SJ0+H8DpRZSWbiz+fEQ+yXQ7b/cL2P2b+0wcvdXUpGTQHvsauQfpQq zlgAMtz90UZy8mWzr4f0L/lm5RAxO1Rbu9+FWm4OafOhXibzj/vhzvPRAMTrkO5+wUan h8ow== X-Gm-Message-State: ALoCoQnf38aKU5x2f3Ki/Y3Dim1Lm2aQZkzA3E+E3HzvqRzmz/6GVoT2n3SEccytDvWLpjkBHG3u4gophXFQw4bh3y4MlCrouilN+OITWcSfvWuhyvfLeV5splXI4LQylANHbvmPHy8gARX17fJ1FgnUJl83aFOvQlGdIOHiUONOCwZmn1huqco= X-Received: by 10.180.73.134 with SMTP id l6mr13694123wiv.16.1381740271533; Mon, 14 Oct 2013 01:44:31 -0700 (PDT) X-Received: by 10.180.73.134 with SMTP id l6mr13694119wiv.16.1381740271454; Mon, 14 Oct 2013 01:44:31 -0700 (PDT) Received: from mech-cluster241.men.bris.ac.uk (mech-cluster241.men.bris.ac.uk. [137.222.187.241]) by mx.google.com with ESMTPSA id q17sm31960706wiv.10.1969.12.31.16.00.00 (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 14 Oct 2013 01:44:30 -0700 (PDT) Sender: Anton Shterenlikht Received: from mech-cluster241.men.bris.ac.uk (localhost [127.0.0.1]) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6) with ESMTP id r9E8iSXr015099 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 14 Oct 2013 09:44:28 +0100 (BST) (envelope-from mexas@mech-cluster241.men.bris.ac.uk) Received: (from mexas@localhost) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6/Submit) id r9E8iSYM015098; Mon, 14 Oct 2013 09:44:28 +0100 (BST) (envelope-from mexas) Date: Mon, 14 Oct 2013 09:44:28 +0100 (BST) From: Anton Shterenlikht Message-Id: <201310140844.r9E8iSYM015098@mech-cluster241.men.bris.ac.uk> To: freebsd-current@freebsd.org, freebsd-ia64@freebsd.org Subject: panic: uma_zfree: Freeing to non free bucket index. X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: mexas@bris.ac.uk List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Oct 2013 08:44:39 -0000 BTW, I see in dmesg: Starting ddb. ddb: sysctl: debug.ddb.scripting.scripts: Invalid argument /etc/rc: WARNING: failed to start ddb What is that about? panic: uma_zfree: Freeing to non free bucket index. cpuid = 0 KDB: stack backtrace: db_trace_self(0x9ffc000000158380) at db_trace_self+0x40 db_trace_self_wrapper(0x9ffc000000607370) at db_trace_self_wrapper+0x70 kdb_backtrace(0x9ffc000000ed0e10, 0x9ffc00000058e660, 0x40c, 0x9ffc0000010a44a0) at kdb_backtrace+0xc0 vpanic(0x9ffc000000dfc468, 0xa0000000e26e0fd8, 0x9ffc000000ef9670, 0x9ffc000000ed0bc0) at vpanic+0x260 kassert_panic(0x9ffc000000dfc468, 0xe000000015e25f90, 0xe000000015e243e0, 0xe00000027ffd5200) at kassert_panic+0x120 uma_zfree_arg(0xe00000027ffccfc0, 0xe000000015e243e0, 0x0) at uma_zfree_arg+0x2d0 g_destroy_bio(0xe000000015e243e0, 0x9ffc0000004ad4a0, 0x30a, 0x30a) at g_destroy_bio+0x30 g_disk_done(0xe000000015e243e0, 0xe000000015e15d10, 0xe000000012672700, 0x9ffc0000006b18c0) at g_disk_done+0x140 biodone(0xe000000015e243e0, 0x9ffc000000e0e150, 0xe000000010c24030, 0x0, 0x0, 0x0, 0x9ffc000000066890, 0x614) at biodone+0x180 dadone(0xe000000012672600, 0xe000000012541000, 0xe000000015e243e0, 0x7) at dadone+0x620 camisr_runqueue(0xe000000011a2dc00, 0xe000000012541054, 0xe000000012541000, 0x135d) at camisr_runqueue+0x6c0 camisr(0xe000000011a2dc20, 0xe000000011a2dc00, 0x9ffc000000bee9d0, 0xa0000000e26e1548) at camisr+0x260 intr_event_execute_handlers(0xe0000000111764a0, 0xe00000001118d998, 0xe000000011191c00, 0x0) at intr_event_execute_handlers+0x280 ithread_loop(0xe000000011192f00, 0xa0000000e26e1550, 0xe000000011192f14, 0xe00000001118d99c) at ithread_loop+0x1b0 fork_exit(0x9ffc000000e12a90, 0xe000000011192f00, 0xa0000000e26e1550) at fork_exit+0x120 enter_userland() at enter_userland KDB: enter: panic [ thread pid 12 tid 100015 ] Stopped at kdb_enter+0x92: [I2] addl r14=0xffffffffffe2c990,gp ;; db> db> scripts lockinfo=show locks; show alllocks; show lockedvnods db> run lockinfo db:0:lockinfo> show locks db:0:locks> show alllocks db:0:alllocks> show lockedvnods Locked vnodes 0xe00000001ab39ba8: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0xe00000001cd30900 ref 0 pages 0 lock type ufs: EXCL by thread 0xe0000000183d9680 (pid 41389, cpdup, tid 100121) ino 5467932, on dev da5p1 0xe000000015ed3ba8: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0xe00000001cd33e00 ref 0 pages 0 lock type ufs: EXCL by thread 0xe000000012a28900 (pid 41421, cpdup, tid 100092) ino 5467948, on dev da5p1 0xe00000001ab16938: tag ufs, type VREG usecount 1, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0xe00000001cd98a00 ref 0 pages 1 lock type ufs: EXCL by thread 0xe000000018494000 (pid 41337, cpdup, tid 100137) ino 5469420, on dev da5p1 0xe00000001b2503b0: tag ufs, type VREG usecount 1, writecount 0, refcount 1 mountedhere 0 flags (VI_ACTIVE) lock type ufs: EXCL by thread 0xe000000012a28900 (pid 41421, cpdup, tid 100092) ino 5469421, on dev da5p1 0xe00000001ab2a760: tag ufs, type VREG usecount 1, writecount 0, refcount 1 mountedhere 0 flags (VI_ACTIVE) lock type ufs: EXCL by thread 0xe0000000183d9680 (pid 41389, cpdup, tid 100121) ino 5469422, on dev da5p1 db> db> script zzz=textdump set; capture on; run lockinfo; show pcpu; bt; ps; alltrace; capture off; reset db> run zzz Then I see lots of info on the console. However, on reboot, there's nothing: from dmesg: *skip* Starting syslogd. No core dumps found. ^^^^^^^^^^^^^^^^^^^ Clearing /tmp (X related). *skip* # savecore -C -v unable to open bounds file, using 0 checking for kernel dump on device /dev/da4p1 mediasize = 72834820096 sectorsize = 512 magic mismatch on last dump header on /dev/da4p1 No dump exists # ls -al /var/crash/ total 16 drwxr-x--- 2 root wheel 512 Oct 13 21:55 . drwxr-xr-x 28 root wheel 512 Oct 14 09:37 .. -rw-r--r-- 1 root wheel 2 Oct 13 21:54 bounds -rw-r--r-- 1 root wheel 5 Jun 12 2009 minfree # Where did my textdump go? Thanks Anton From owner-freebsd-ia64@FreeBSD.ORG Mon Oct 14 11:03:32 2013 Return-Path: Delivered-To: freebsd-ia64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 020FABFE; Mon, 14 Oct 2013 11:03:30 +0000 (UTC) (envelope-from davide.italiano@gmail.com) Received: from mail-vc0-x22a.google.com (mail-vc0-x22a.google.com [IPv6:2607:f8b0:400c:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 54FB82BC9; Mon, 14 Oct 2013 11:03:30 +0000 (UTC) Received: by mail-vc0-f170.google.com with SMTP id lc6so4563325vcb.1 for ; Mon, 14 Oct 2013 04:03:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=VIarfLrSItDa0I1UHnua40yktAvSk4My/jyoISGCW94=; b=yvEgTxObOzxAY39fgp93ola9tpKy55YMsjBAJ6FjPvwEJjBNrkaKY0rUL+JLIMB1yu 5b7RB8inDc2d7re47XiyUNyXBx0HsASFOJn4YxcAiv5GBbg5gb2HGTkaLm9KJbx9eJkz jcOTK06QJbBypNVR7YZCaPk/BW5tHNgbeY5l+8PX8QXhIjUXORa8cHWKM8UmBXcwFtcm yjAJpX15szI+7EnRkXL8khiT3xVyelafuwIl+Ql6LjfbGirAnt9U2e/1p2G11nRZlmB3 uHn/WVstuy1+ZmsI96Hk5kqiRGu05TYEM2xW9DXuPXu7nk2ow6YfaaUD9PoIzAxLwKiQ aXQQ== MIME-Version: 1.0 X-Received: by 10.52.52.137 with SMTP id t9mr598589vdo.22.1381748609331; Mon, 14 Oct 2013 04:03:29 -0700 (PDT) Sender: davide.italiano@gmail.com Received: by 10.220.94.71 with HTTP; Mon, 14 Oct 2013 04:03:29 -0700 (PDT) In-Reply-To: <201310140821.r9E8L55j014897@mech-cluster241.men.bris.ac.uk> References: <201310140821.r9E8L55j014897@mech-cluster241.men.bris.ac.uk> Date: Mon, 14 Oct 2013 13:03:29 +0200 X-Google-Sender-Auth: iTdbcUU87dE3NdiYBpiwJnWXRTw Message-ID: Subject: Re: panic: ia64 r255811: deadlkres: possible deadlock detected for 0xe000000012d07b00, blocked for 902743 ticks From: Davide Italiano To: mexas@bris.ac.uk Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-current , freebsd-ia64@freebsd.org X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Oct 2013 11:03:32 -0000 On Mon, Oct 14, 2013 at 10:21 AM, Anton Shterenlikht wrote: > >From davide.italiano@gmail.com Fri Oct 11 15:39:49 2013 >> >>If you're not able to get a full dump, a textdump would be enough. >>In your DDB scripts just remove the 'call doadump' step and you should >>be done, unless I'm missing something. >>Please adjust the script as well to include all the informations >>requested as mentioned in my previous link, e.g. 'show lockedvnods' is >>not mentioned on the example section of textdump(4) manpage but it >>could be useful to ease debugging. > > I still haven't got it right, but I now collected manually > some (hopefully) useful info (see below). > This is fair enough -- If you're still at the ddb prompt, please print the whole panic message (or at least the address of the lock reported as deadlocked by DEADLKRES), so that we can at least have a candidate. -- Davide "There are no solved problems; there are only problems that are more or less solved" -- Henri Poincare From owner-freebsd-ia64@FreeBSD.ORG Mon Oct 14 11:06:49 2013 Return-Path: Delivered-To: freebsd-ia64@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C0B0F4B5 for ; Mon, 14 Oct 2013 11:06:49 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id ADC0F2C78 for ; Mon, 14 Oct 2013 11:06:49 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r9EB6nq6035223 for ; Mon, 14 Oct 2013 11:06:49 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r9EB6nTf035221 for freebsd-ia64@FreeBSD.org; Mon, 14 Oct 2013 11:06:49 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 14 Oct 2013 11:06:49 GMT Message-Id: <201310141106.r9EB6nTf035221@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-ia64@FreeBSD.org Subject: Current problem reports assigned to freebsd-ia64@FreeBSD.org X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Oct 2013 11:06:49 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o ia64/182578 ia64 GRUB does not recognize Windows 7 p ia64/169926 ia64 [ia64] netstat -r: kvm_read: Bad address o ia64/169781 ia64 /usr/bin/csup ends in sigwai state and never exits o ia64/167618 ia64 [ia64] Implement revision 232832 o ia64/158547 ia64 [ia64] Epilog counter (ar.ec) needs to be saved and re o ia64/156900 ia64 ia64 -current r221488 panic if kern.maxssiz=536748033 6 problems total. From owner-freebsd-ia64@FreeBSD.ORG Tue Oct 15 08:36:31 2013 Return-Path: Delivered-To: freebsd-ia64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 4B351BB6 for ; Tue, 15 Oct 2013 08:36:31 +0000 (UTC) (envelope-from mexas@bristol.ac.uk) Received: from eu1sys200aog117.obsmtp.com (eu1sys200aog117.obsmtp.com [207.126.144.143]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9D2CB2CC6 for ; Tue, 15 Oct 2013 08:36:30 +0000 (UTC) Received: from mail-we0-f172.google.com ([74.125.82.172]) (using TLSv1) by eu1sys200aob117.postini.com ([207.126.147.11]) with SMTP ID DSNKUlz+ct12D82Z7wGKuWKweB9cV3aHZqWF@postini.com; Tue, 15 Oct 2013 08:36:30 UTC Received: by mail-we0-f172.google.com with SMTP id q58so8246930wes.3 for ; Tue, 15 Oct 2013 01:36:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:message-id:to:subject:cc :reply-to:in-reply-to; bh=Ou4YbHxnVaCL/YU6ZS0DdIh3k55xIIg4DZs6C/5sJRE=; b=CDs71ymotyF19P77XjtvV+d3rFcxZeW4tgTCMoX6lUaiy11+ejL+5xQBbqlSKk7Mzu i+5m7nsvH9tOFVjS+8poUohdIEmnkd1vEeEm6x6TblXxpSnvaXl0MY8oQxMOzgCuJLe+ ig1ZSKhLqD7Ykk6qsHQS4uHIyQw/85PxGKG8b0T8SNuwJ2/1Cj8R4GPw0O6I00yDd1XD irRrjkikXoVIIXGEIEG7A4XTHlI4Al+epqWd7CQP9+Gpb6pVMW57n3PZj61dR+iEBRQH xRAt1XvhFtRyRScoQMOD5MUeamETTft6aSx1blJ0rijdE+5vKwwlHP/Ml2zdibQIwZ3+ dESQ== X-Gm-Message-State: ALoCoQmygwZO58VQKXHJE9jgxz+weQqQis+hAhaneMU9FI3nBYXuEMPEqOQTQa9aMgHq029EEbNss8zVC6YgA+pSrSoQxc3ggxGqXCIwKsj/xCdBDvgFyRO9egV/ehzAYzt/EfCLua8mx7UKZnCMg6nzjX/ujX6y3UnZ//rsJGSNdc3H9uxJrLE= X-Received: by 10.180.95.202 with SMTP id dm10mr13692831wib.62.1381826162044; Tue, 15 Oct 2013 01:36:02 -0700 (PDT) X-Received: by 10.180.95.202 with SMTP id dm10mr13692817wib.62.1381826161932; Tue, 15 Oct 2013 01:36:01 -0700 (PDT) Received: from mech-cluster241.men.bris.ac.uk (mech-cluster241.men.bris.ac.uk. [137.222.187.241]) by mx.google.com with ESMTPSA id mw9sm376220wib.0.2013.10.15.01.35.59 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Oct 2013 01:36:00 -0700 (PDT) Sender: Anton Shterenlikht Received: from mech-cluster241.men.bris.ac.uk (localhost [127.0.0.1]) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6) with ESMTP id r9F8ZwlK043225 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 15 Oct 2013 09:35:58 +0100 (BST) (envelope-from mexas@mech-cluster241.men.bris.ac.uk) Received: (from mexas@localhost) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6/Submit) id r9F8ZwZk043224; Tue, 15 Oct 2013 09:35:58 +0100 (BST) (envelope-from mexas) Date: Tue, 15 Oct 2013 09:35:58 +0100 (BST) From: Anton Shterenlikht Message-Id: <201310150835.r9F8ZwZk043224@mech-cluster241.men.bris.ac.uk> To: davide@freebsd.org, mexas@bris.ac.uk Subject: Re: panic: ia64 r255811: deadlkres: possible deadlock detected for 0xe000000012d07b00, blocked for 902743 ticks In-Reply-To: Cc: freebsd-current@freebsd.org, freebsd-ia64@freebsd.org X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: mexas@bris.ac.uk List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 08:36:31 -0000 >From davide.italiano@gmail.com Fri Oct 11 15:39:49 2013 > >If you're not able to get a full dump, a textdump would be enough. >In your DDB scripts just remove the 'call doadump' step and you should >be done, unless I'm missing something. >Please adjust the script as well to include all the informations >requested as mentioned in my previous link, e.g. 'show lockedvnods' is >not mentioned on the example section of textdump(4) manpage but it >could be useful to ease debugging. It seems "call doadump" is always needed. At least it is included in textdump(4) examples. Also, this tutorial incluldes it: http://www.etinc.com/122/Using-FreeBSD-Text-Dumps I think I still haven't got textdump right, because instead of savecore: writing core to textdump.tar.0 I see: savecore: writing core to /var/crash/vmcore.9 Thanks Anton From owner-freebsd-ia64@FreeBSD.ORG Tue Oct 15 08:44:03 2013 Return-Path: Delivered-To: freebsd-ia64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id ED0A7E87 for ; Tue, 15 Oct 2013 08:44:02 +0000 (UTC) (envelope-from mexas@bristol.ac.uk) Received: from eu1sys200aog120.obsmtp.com (eu1sys200aog120.obsmtp.com [207.126.144.149]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9B52B2D30 for ; Tue, 15 Oct 2013 08:44:01 +0000 (UTC) Received: from mail-wg0-f51.google.com ([74.125.82.51]) (using TLSv1) by eu1sys200aob120.postini.com ([207.126.147.11]) with SMTP ID DSNKUl0ASamvSHv8ND3C6dkFNxh7x0przGNz@postini.com; Tue, 15 Oct 2013 08:44:01 UTC Received: by mail-wg0-f51.google.com with SMTP id l18so4128055wgh.18 for ; Tue, 15 Oct 2013 01:43:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:message-id:to:subject:cc :reply-to:in-reply-to; bh=qIZOnKs6cocoqi24maySE41oy63uT+WI8F4kkYMQedY=; b=BT+MH+Nj9y50eOSliRV6FwZO3VGFsXx9UeDNhs3W78b4McD649fq7XvkcSjxS1NSxS 16pL7HAWHDnRiwWslm780PosGQZagBGKTRDjO0FnXm+enyRSvkwaOPV0GyaL6yQC/PMK VhX1pTo5APxRxVNBw/1UToqSTTAsKYfV+KB6Okj8gadTjVn3v0nZokSY6EI3O5vON1Q/ NKSSmAx68ceNtN0bJWZapzqyctX8rMnt+315tPaccz6p0Id6wKN6G6VX0PmNow4ukMPe qp57TvyftlXLXnpCtELLzV78BOkqxDEU5PscfGS+4ACGM10WbAMyfuv5J2YfS6VNLAy1 K+NA== X-Gm-Message-State: ALoCoQm7KScT2bxYiaXMsFVs+go4zSHs9dZx2LfimLP8/GurgCw4p24jYe1ai4/IHU7o/HwnylsbPMYLhWNZQsDHYjkJvI1Az0lKwzRWhghEkAiQn5kD1qeZR3zGkGn9MqtzzssinjSGCrQcDNUxtMiEe27XYXgk0Re2PEYVjr6oGtJj7NizKlQ= X-Received: by 10.180.187.169 with SMTP id ft9mr18481614wic.14.1381826633645; Tue, 15 Oct 2013 01:43:53 -0700 (PDT) X-Received: by 10.180.187.169 with SMTP id ft9mr18481606wic.14.1381826633515; Tue, 15 Oct 2013 01:43:53 -0700 (PDT) Received: from mech-cluster241.men.bris.ac.uk (mech-cluster241.men.bris.ac.uk. [137.222.187.241]) by mx.google.com with ESMTPSA id om10sm3947130wic.5.2013.10.15.01.43.51 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Oct 2013 01:43:52 -0700 (PDT) Sender: Anton Shterenlikht Received: from mech-cluster241.men.bris.ac.uk (localhost [127.0.0.1]) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6) with ESMTP id r9F8hok0043236 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 15 Oct 2013 09:43:50 +0100 (BST) (envelope-from mexas@mech-cluster241.men.bris.ac.uk) Received: (from mexas@localhost) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6/Submit) id r9F8ho1c043235; Tue, 15 Oct 2013 09:43:50 +0100 (BST) (envelope-from mexas) Date: Tue, 15 Oct 2013 09:43:50 +0100 (BST) From: Anton Shterenlikht Message-Id: <201310150843.r9F8ho1c043235@mech-cluster241.men.bris.ac.uk> To: davide@freebsd.org, mexas@bris.ac.uk Subject: panic: wrong page state m 0xe00000027a9adb40 + savecore deadlock In-Reply-To: Cc: freebsd-current@freebsd.org, freebsd-ia64@freebsd.org X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: mexas@bris.ac.uk List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 08:44:03 -0000 >From davide.italiano@gmail.com Mon Oct 14 12:50:44 2013 > >This is fair enough -- If you're still at the ddb prompt, please print >the whole panic message (or at least the address of the lock reported >as deadlocked by DEADLKRES), so that we can at least have a candidate. > Here's another one, followed by savecore deadlock. ia64 r255488 panic: wrong page state m 0xe00000027a9adb40 cpuid = 0 KDB: stack backtrace: db_trace_self(0x9ffc000000158380) at db_trace_self+0x40 db_trace_self_wrapper(0x9ffc000000607370) at db_trace_self_wrapper+0x70 kdb_backtrace(0x9ffc000000ed0e10, 0x9ffc00000058e660, 0x40c, 0x9ffc0000010a44a0) at kdb_backtrace+0xc0 vpanic(0x9ffc000000dd3fe0, 0xa00000009de61118, 0x9ffc000000ef9670, 0x9ffc000000ed0bc0) at vpanic+0x260 kassert_panic(0x9ffc000000dd3fe0, 0xe00000027a9adb40, 0x81f, 0xe0000002013cf400, 0x9ffc0000006a0220, 0x2c60, 0xe0000002013cf400, 0xe0000002013cf418) at kassert_panic+0x120 vn_sendfile(0x8df, 0xd, 0x0, 0x0, 0x0, 0x8df, 0x7fffffffffffdfe0, 0x0) at vn_sendfile+0x15d0 sys_sendfile(0xe000000012aef200, 0xa00000009de614e8, 0x10, 0xa00000009de61360) at sys_sendfile+0x2b0 syscall(0xe0000000154f2940, 0xd, 0x0, 0xe000000012aef200, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return KDB: enter: panic [ thread pid 5989 tid 100111 ] Stopped at kdb_enter+0x92: [I2] addl r14=0xffffffffffe2c990,gp ;; db> db> scripts lockinfo=show locks; show alllocks; show lockedvnods zzz=textdump set; capture on; run lockinfo; show pcpu; bt; ps; alltrace; capture off; call doadump; reset db> db> run zzz I get to db:0:alltrace> capture off db:0:off> call doadump Dumping 10220 MB (25 chunks) chunk 0: 1 pages ... ok chunk 1: 159 pages ... ok chunk 2: 256 pages ... ok chunk 3: 7680 pages ... ok chunk 4: 8192 pages ... ok chunk 5: 239734 pages ... ok chunk 6: 748 pages ... ok chunk 7: 533 pages ... ok chunk 8: 21 pages ... ok chunk 9: 1572862 pages ... ok chunk 10: 781683 pages ... ok chunk 11: 512 pages ... ok chunk 12: 139 pages ... ok chunk 13: 484 pages ... ok chunk 14: 1565 pages ... ok chunk 15: 1 pages ... ok chunk 16: 506 pages ... ok chunk 17: 1 pages ... ok chunk 18: 3 pages ... ok chunk 19: 566 pages ... ok chunk 20: 66 pages ... ok chunk 21: 1 pages ... ok chunk 22: 285 pages ... ok chunk 23: 6 pages ... ok chunk 24: 354 pages ... ok Dump complete = 0 db:0:doadump> reset So far, so good. On reboot I get: Starting ddb. ddb: sysctl: debug.ddb.scripting.scripts: Invalid argument /etc/rc: WARNING: failed to start ddb This probably already indicates some problem? Eventually I get to: savecore: reboot after panic: wrong page state m 0xe00000027a9adb40 Oct 15 09:05:50 mech-as28 savecore: reboot after panic: wrong page state m 0xe00000027a9adb40 savecore: writing core to /var/crash/vmcore.9 So here I'm confused. I think I set up textdump as in the man page. So I think the core should not be written. Instead I was expecting ddb.txt, config.txt, etc., as in textdump(4). Anyway, savecore eventually deadlocks: panic: deadlkres: possible deadlock detected for 0xe0000000127b7b00, blocked for 901401 ticks cpuid = 0 KDB: stack backtrace: db_trace_self(0x9ffc000000158380) at db_trace_self+0x40 db_trace_self_wrapper(0x9ffc000000607370) at db_trace_self_wrapper+0x70 kdb_backtrace(0x9ffc000000ed0e10, 0x9ffc00000058e660, 0x40c, 0x9ffc0000010a44a0) at kdb_backtrace+0xc0 vpanic(0x9ffc000000db8a18, 0xa00000009dca7518) at vpanic+0x260 panic(0x9ffc000000db8a18, 0x9ffc000000db8c70, 0xe0000000127b7b00, 0xdc119) at panic+0x80 deadlkres(0xdc119, 0xe0000000127b7b00, 0x9ffc000000dbb648, 0x9ffc000000db89a8) at deadlkres+0x420 fork_exit(0x9ffc000000e0fca0, 0x0, 0xa00000009dca7550) at fork_exit+0x120 enter_userland() at enter_userland KDB: enter: panic [ thread pid 0 tid 100053 ] Stopped at kdb_enter+0x92: [I2] addl r14=0xffffffffffe2c990,gp ;; db> db> scripts lockinfo=show locks; show alllocks; show lockedvnods db> run lockinfo db:0:lockinfo> show locks db:0:locks> show alllocks db:0:alllocks> show lockedvnods Locked vnodes 0xe0000000127cbba8: tag devfs, type VCHR usecount 1, writecount 0, refcount 19 mountedhere 0xe0000000126ab200 flags (VI_ACTIVE) v_object 0xe0000000127c2b00 ref 0 pages 422 lock type devfs: EXCL by thread 0xe000000012690000 (pid 21, syncer, tid 100062) dev da3p1 0xe0000000127f4ec0: tag ufs, type VREG usecount 1, writecount 1, refcount 32934 mountedhere 0 flags (VI_ACTIVE) v_object 0xe0000000127f7200 ref 0 pages 1242850 lock type ufs: EXCL by thread 0xe0000000127b7b00 (pid 805, savecore, tid 100079) ino 6500740, on dev da3p1 db> db> ps pid ppid pgrp uid state wmesg wchan cmd 805 803 24 0 L+ *vm page 0xe000000012402fc0 savecore 803 24 24 0 DL+ vm map ( 0xe00000001285fa88 sh 801 1 801 0 Ss select 0xe000000010c296c0 syslogd 792 1 792 0 Ss select 0xe000000010c29840 ipmon 670 1 670 0 Ss select 0xe0000000126d5740 devd 155 1 155 0 SWs pause 0xe0000000127fe9e8 adjkerntz 24 1 24 0 LLs+ *vm activ 0xe000000012403bc0 sh 23 0 0 0 DL sdflush 0x9ffc000000ed1ff0 [softdepflush] 22 0 0 0 DL vlruwt 0xe000000012409720 [vnlru] 21 0 0 0 LL *vm objec 0xe000000010c30240 [syncer] 20 0 0 0 DL psleep 0x9ffc000000ed1390 [bufdaemon] 19 0 0 0 DL pgzero 0x9ffc000000ed239c [pagezero] 18 0 0 0 RL [vmdaemon] 17 0 0 0 LL *pmap pv 0xe000000010c2e180 [pagedaemon] 9 0 0 0 DL ccb_scan 0x9ffc000000ed26a0 [xpt_thrd] 6 0 0 0 DL waiting_ 0x9ffc0000010f33b0 [sctp_iterator] 5 0 0 0 DL (threaded) [zfskern] 100052 D l2arc_fe 0x9ffc000001547f40 [l2arc_feed_thread] 100051 D arc_recl 0x9ffc000001538120 [arc_reclaim_thread] --More-- 4 0 0 0 DL ispf 0xe000000011a02000 [isp0: fc_thrd0] 3 0 0 0 DL idle 0xa000000000a16000 [mpt_recovery1] 2 0 0 0 DL idle 0xa0000000009fe000 [mpt_recovery0] 16 0 0 0 DL (threaded) [usb] 100039 D - 0xa0000000009fae18 [usbus2] 100038 D - 0xa0000000009fadc0 [usbus2] 100037 D - 0xa0000000009fad68 [usbus2] 100036 D - 0xa0000000009fad10 [usbus2] 100034 D - 0xa0000000009f1460 [usbus1] 100033 D - 0xa0000000009f1408 [usbus1] 100032 D - 0xa0000000009f13b0 [usbus1] 100031 D - 0xa0000000009f1358 [usbus1] 100029 D - 0xa0000000009ed460 [usbus0] 100028 D - 0xa0000000009ed408 [usbus0] 100027 D - 0xa0000000009ed3b0 [usbus0] 100026 D - 0xa0000000009ed358 [usbus0] 15 0 0 0 DL tzpoll 0x9ffc000000ecfd20 [acpi_thermal] 14 0 0 0 DL - 0x9ffc000000ed0008 [rand_harvestq] 13 0 0 0 LL (threaded) [geom] 100012 D - 0x9ffc000000ed0540 [g_down] --More-- 100011 L *vm objec 0xe000000010c30240 [g_up] 100010 D - 0x9ffc000000ed0528 [g_event] 12 0 0 0 RL (threaded) [intr] 100048 I [swi0: uart uart+++] 100046 I [irq71: isp0] 100042 I [irq28: mpt1] 100040 I [irq27: mpt0] 100035 I [irq18: ehci0] 100030 I [irq17: ohci1] 100025 I [irq16: ohci0] 100022 I [swi6: task queue] 100021 I [swi6: Giant taskq] 100019 I [swi5: fast taskq] 100015 I [swi2: cambio] 100008 I [swi4: clock] 100007 Run CPU 1 [swi4: clock] 100006 I [swi3: vm] 100005 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100004 CanRun [idle: cpu1] 100003 CanRun [idle: cpu0] 1 0 1 0 SLs wait 0xe000000011176de0 [init] 10 0 0 0 DL audit_wo 0x9ffc0000010f7a48 [audit] 0 0 0 0 RLs (threaded) [kernel] 100053 Run CPU 0 [deadlkres] 100050 D - 0xe000000011a2d300 [system_taskq_1] 100049 D - 0xe000000011a2d300 [system_taskq_0] 100045 D - 0xe000000011198500 [em1 taskq] 100044 D - 0xe000000011198600 [em0 taskq] 100023 D - 0xe00000001119b300 [ffs_trim taskq] 100020 D - 0xe00000001119b600 [thread taskq] 100018 D - 0xe00000001119b800 [acpi_task_2] 100017 D - 0xe00000001119b800 [acpi_task_1] 100016 D - 0xe00000001119b800 [acpi_task_0] 100014 D - 0xe00000001119ba00 [kqueue taskq] 100009 D - 0xe00000001119bd00 [firmware taskq] 100000 D swapin 0x9ffc000000eedd28 [swapper] db> db> show pcpu cpuid = 0 dynamic pcpu = 0x40040040ff13ca80 curthread = 0xe000000011a26d80: pid 0 "deadlkres" curpcb = 0 fpcurthread = none idlethread = 0xe000000011178900: tid 100003 "idle: cpu0" MD: vhpt = 0xe0000040ff000000 MD: lid = 0 MD: clock = 0x2351ad02dce MD: clock_mode = 2 MD: clock_load = 0 MD: stats = 0x9ffc0000010ff130 MD: pmap = 0x9ffc000000eee760 spin locks held: db> db> bt Tracing pid 0 tid 100053 td 0xe000000011a26d80 kdb_enter(0x9ffc000000dc2fd0, 0x9ffc000000dc2fd0, 0x9ffc00000058e6b0, 0x40c) at kdb_enter+0x92 vpanic(0x9ffc000000db8a18, 0xa00000009dca7518) at vpanic+0x2b0 panic(0x9ffc000000db8a18, 0x9ffc000000db8c70, 0xe0000000127b7b00, 0xdc119) at panic+0x80 deadlkres(0xdc119, 0xe0000000127b7b00, 0x9ffc000000dbb648, 0x9ffc000000db89a8) at deadlkres+0x420 fork_exit(0x9ffc000000e0fca0, 0x0, 0xa00000009dca7550) at fork_exit+0x120 enter_userland() at enter_userland db> db> alltrace Tracing command savecore pid 805 tid 100079 td 0xe0000000127b7b00 cpu_switch(0xe0000000127b7b00, 0xe000000011178900, 0xe000000012402fc0, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe0000000127b7b00, 0xe000000011178900, 0x9ffc000000f15698, 0x9ffc000000f15680) at sched_switch+0x890 mi_switch(0x103, 0x0, 0xe0000000127b7b00, 0x9ffc00000062d1f0) at mi_switch+0x3f0 turnstile_wait(0xe000000012402fc0, 0xe000000012400480, 0x0, 0x9ffc000000dcb698) at turnstile_wait+0x960 __mtx_lock_sleep(0x9ffc0000010f9998, 0xe0000000127b7b00, 0xe000000012402fc0, 0x9ffc000000dc0558, 0x742) at __mtx_lock_sleep+0x2f0 __mtx_lock_flags(0x9ffc0000010f9980, 0x0, 0x9ffc000000dd4a90, 0x742) at __mtx_lock_flags+0x1e0 vfs_vmio_release(0xa00000009ebe72f0, 0xe00000027ed2ab70, 0x3, 0xa00000009ebe736c, 0xa00000009ebe7498, 0xa00000009ebe72f8, 0x9ffc000000dd4a90, 0x9ffc0000010f9680) at vfs_vmio_release+0x290 getnewbuf(0xe0000000127f4ec0, 0x0, 0x0, 0x8000, 0xa00000009ebe99a8, 0x0, 0x9ffc0000010f0798, 0xa00000009ebe72f0) at getnewbuf+0x7e0 getblk(0xe0000000127f4ec0, 0x4cbaa, 0x8000, 0x0, 0x0, 0x0, 0x0, 0x0) at getblk+0xee0 ffs_balloc_ufs2(0xe0000000127f4ec0, 0x4cbaa, 0xa0000000c60ba000, 0xe000000011165a00, 0x7f050000, 0xa00000009dd79160) at ffs_balloc_ufs2+0x2950 ffs_write(0xa00000009dd79248, 0x3000, 0x265d50000) at ffs_write+0x5c0 VOP_WRITE_APV(0x9ffc000000e94ac0, 0xa00000009dd79248, 0x0, 0x0) at VOP_WRITE_APV+0x330 vn_write(0xe0000000129ae820, 0xa00000009dd79360, 0xe000000011165a00, 0x0, 0xe0000000129ae830, 0xe0000000127f4ec0) at vn_write+0x450 vn_io_fault(0xe0000000129ae820, 0xa00000009dd79360, 0xe000000011165a00, 0x0, 0xe0000000127b7b00) at vn_io_fault+0x330 dofilewrite(0xe0000000127b7b00, 0x7, 0xe0000000129ae820, 0xa00000009dd79360, 0xffffffffffffffff, 0x0) at dofilewrite+0x180 kern_writev(0xe0000000127b7b00, 0x7, 0xa00000009dd79360) at kern_writev+0xa0 sys_write(0xe0000000127b7b00, 0xa00000009dd794e8, 0x9ffc000000abac80, 0x48d) at sys_write+0x100 syscall(0xe0000000129d04a0, 0x140857000, 0x8000, 0xe0000000127b7b00, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 --More-- epc_syscall_return() at epc_syscall_return Tracing command sh pid 803 tid 100075 td 0xe0000000127b6000 cpu_switch(0xe0000000127b6000, 0xe000000011178900, 0x9ffc000000f4c570, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe0000000127b6000, 0xe000000011178900, 0x9ffc000000f15698, 0x9ffc000000f15680) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000127b6000, 0x9ffc000000620b70) at mi_switch+0x3f0 sleepq_switch(0xe00000001285fa88, 0x0, 0xe0000000127b6378, 0xe0000000127b6000) at sleepq_switch+0x2e0 sleepq_wait(0xe00000001285fa88, 0x0, 0x9ffc000000dca928, 0xe0000000127b6000, 0x9ffc00000059d350) at sleepq_wait+0xb0 _sx_slock_hard(0xe00000001285fa88, 0x0, 0xe00000001285faa0, 0xe000000012400000, 0xe00000001285fa88) at _sx_slock_hard+0x5f0 _sx_slock(0xe00000001285fa88, 0x0, 0x9ffc000000dfd218, 0xefa) at _sx_slock+0x190 _vm_map_lock_read(0xe00000001285fa08, 0x9ffc000000dfd218, 0xefa) at _vm_map_lock_read+0x80 vm_map_lookup(0xa00000009dd58928, 0x100046000, 0x1, 0xa00000009dd58930, 0xa00000009dd58918, 0xa00000009dd58920, 0xa00000009dd588f0, 0xa00000009dd588f4) at vm_map_lookup+0x70 vm_fault_hold(0xe00000001285fa08, 0x100046000, 0x1, 0x0, 0x0) at vm_fault_hold+0x280 vm_fault(0xe00000001285fa08, 0x100046000, 0x1, 0x0, 0xe0000000127b6000, 0x9ffc000000abbce0, 0x716, 0xe0000000127fe5a0) at vm_fault+0xf0 trap(0x14, 0xa00000009dd59400) at trap+0x9e0 ivt_Data_TLB() at ivt_Data_TLB+0x1d0 --- trapframe at 0xa00000009dd59400 (null)(...) at 0x8000000000000120 Tracing command syslogd pid 801 tid 100078 td 0xe00000001282c000 cpu_switch(0xe00000001282c000, 0xe000000011178480, 0x9ffc000000f4c110, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe00000001282c000, 0xe000000011178480, 0x9ffc000000f16398, 0x9ffc000000f16380) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001282c000, 0x9ffc000000620b70) at mi_switch+0x3f0 sleepq_switch(0xe000000010c296c0, 0x0, 0xe00000001282c378, 0xe00000001282c000) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000010c296c0, 0x0, 0x9ffc000000dca928, 0x9ffc00000056ab00) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000010c296c0, 0x0, 0x9ffc0000004f7230, 0x611, 0x9ffc0000004f7270) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe000000010c296c0, 0xe000000010c296a0, 0xe00000001282c000, 0x0, 0x9ffc000000ef0710) at _cv_wait_sig+0x4d0 seltdwait(0xe000000010c29680, 0xffffffffffffffff, 0x0, 0xe000000010c296b8, 0x9ffc000000dcdfe0, 0x9ffc000000645bf0) at seltdwait+0x110 kern_select(0xe00000001282c000, 0x0, 0x140866180, 0x0, 0x0, 0xffffffffffffffff, 0x9) at kern_select+0xb40 sys_select(0xe00000001282c000, 0xa00000009dd714e8, 0x9ffc000000abac80, 0x48d) at sys_select+0xb0 syscall(0xe0000000129d0940, 0x140866180, 0x0, 0xe00000001282c000, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command ipmon pid 792 tid 100077 td 0xe00000001282c480 cpu_switch(0xe00000001282c480, 0xe000000011178900, 0x9ffc000000f4d560, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe00000001282c480, 0xe000000011178900, 0x9ffc000000f15698, 0x9ffc000000f15680) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001282c480, 0x9ffc000000620b70) at mi_switch+0x3f0 sleepq_switch(0xe000000010c29840, 0x0, 0xe00000001282c7f8, 0xe00000001282c480) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000010c29840, 0x0, 0x9ffc000000dca928, 0x9ffc00000056ab00) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000010c29840, 0x0, 0x9ffc0000004f7230, 0x611, 0x9ffc0000004f7270) at sleepq_wait_sig+0x30 --More-- _cv_wait_sig(0xe000000010c29840, 0xe000000010c29820, 0xe00000001282c480, 0x0, 0x9ffc000000ef0710) at _cv_wait_sig+0x4d0 seltdwait(0xe000000010c29800, 0xffffffffffffffff, 0x0, 0xe000000010c29838, 0x9ffc000000dcdfe0, 0x9ffc000000645bf0) at seltdwait+0x110 kern_select(0xe00000001282c480, 0x0, 0x7fffffffffff6b50, 0x0, 0x0, 0xffffffffffffffff, 0x4) at kern_select+0xb40 sys_select(0xe00000001282c480, 0xa00000009dd694e8, 0x9ffc000000abac80, 0x48d) at sys_select+0xb0 syscall(0xe0000000129d0de0, 0x7fffffffffff6b50, 0x0, 0xe00000001282c480, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command devd pid 670 tid 100056 td 0xe000000012400d80 cpu_switch(0xe000000012400d80, 0xe000000011178480, 0x9ffc000000f4b738, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe000000012400d80, 0xe000000011178480, 0x9ffc000000f16398, 0x9ffc000000f16380) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012400d80, 0x9ffc000000620b70) at mi_switch+0x3f0 sleepq_switch(0xe0000000126d5740, 0x0, 0xe0000000124010f8, 0xe000000012400d80) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000000126d5740, 0x0, 0x9ffc000000dca928, 0xe0000000126d5720) at sleepq_catch_signals+0x190 sleepq_timedwait_sig(0xe0000000126d5740, 0x0, 0x9ffc0000004f6430, 0x9ffc0000004f6470, 0x713) at sleepq_timedwait_sig+0x30 _cv_timedwait_sig_sbt(0xe0000000126d5740, 0xe0000000126d5720, 0x5bbba38f388, 0x3c0000000, 0x200) at _cv_timedwait_sig_sbt+0x4f0 seltdwait(0xe0000000126d5700, 0x5bbba38f388, 0x3c0000000, 0xe0000000126d5738, 0x9ffc000000dcdfe0, 0x9ffc000000645bf0) at seltdwait+0xf0 kern_select(0xe000000012400d80, 0x0, 0x7fffffffffffcd00, 0x0, 0x0, 0x5bbba38f388, 0x5) at kern_select+0xb40 sys_select(0xe000000012400d80, 0xa00000009dcc14e8, 0x9ffc000000abac80, 0x48d) at sys_select+0xb0 syscall(0xe0000000111cb280, 0x7fffffffffffcd00, 0x0, 0xe000000012400d80, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return --More-- Tracing command adjkerntz pid 155 tid 100074 td 0xe0000000127b6480 Tracing command sh pid 24 tid 100065 td 0xe000000012401200 Tracing command softdepflush pid 23 tid 100064 td 0xe000000012401680 Tracing command vnlru pid 22 tid 100063 td 0xe000000012401b00 Tracing command syncer pid 21 tid 100062 td 0xe000000012690000 Tracing command bufdaemon pid 20 tid 100061 td 0xe000000012690480 Tracing command pagezero pid 19 tid 100060 td 0xe000000011a27b00 Tracing command vmdaemon pid 18 tid 100059 td 0xe000000012400000 Tracing command pagedaemon pid 17 tid 100058 td 0xe000000012400480 Tracing command xpt_thrd pid 9 tid 100057 td 0xe000000012400900 --More-- Tracing command sctp_iterator pid 6 tid 100054 td 0xe000000011a26900 Tracing command zfskern pid 5 tid 100052 td 0xe000000011a27200 Tracing command zfskern pid 5 tid 100051 td 0xe000000011a27680 Tracing command isp0: fc_thrd0 pid 4 tid 100047 td 0xe000000011931b00 Tracing command mpt_recovery1 pid 3 tid 100043 td 0xe000000011930000 Tracing command mpt_recovery0 pid 2 tid 100041 td 0xe000000011930900 Tracing command usb pid 16 tid 100039 td 0xe000000011924480 Tracing command usb pid 16 tid 100038 td 0xe000000011924900 Tracing command usb pid 16 tid 100037 td 0xe000000011924d80 Tracing command usb pid 16 tid 100036 td 0xe000000011925200 --More-- Tracing command usb pid 16 tid 100034 td 0xe000000011300d80 Tracing command usb pid 16 tid 100033 td 0xe000000011301200 Tracing command usb pid 16 tid 100032 td 0xe000000011301680 Tracing command usb pid 16 tid 100031 td 0xe000000011301b00 Tracing command usb pid 16 tid 100029 td 0xe000000011219680 Tracing command usb pid 16 tid 100028 td 0xe000000011219b00 Tracing command usb pid 16 tid 100027 td 0xe000000011300000 Tracing command usb pid 16 tid 100026 td 0xe000000011300480 Tracing command acpi_thermal pid 15 tid 100024 td 0xe000000011218000 Tracing command rand_harvestq pid 14 tid 100013 td 0xe000000011197680 --More-- Tracing command geom pid 13 tid 100012 td 0xe000000011197b00 Tracing command geom pid 13 tid 100011 td 0xe0000000111a6000 Tracing command geom pid 13 tid 100010 td 0xe000000011179680 Tracing command intr pid 12 tid 100048 td 0xe000000011931680 Tracing command intr pid 12 tid 100046 td 0xe000000011a26000 Tracing command intr pid 12 tid 100042 td 0xe000000011930480 Tracing command intr pid 12 tid 100040 td 0xe000000011924000 Tracing command intr pid 12 tid 100035 td 0xe000000011300900 Tracing command intr pid 12 tid 100030 td 0xe000000011219200 Tracing command intr pid 12 tid 100025 td 0xe0000000111a7b00 --More-- Tracing command intr pid 12 tid 100022 td 0xe000000011218900 Tracing command intr pid 12 tid 100021 td 0xe000000011218d80 Tracing command intr pid 12 tid 100019 td 0xe0000000111a6900 Tracing command intr pid 12 tid 100015 td 0xe000000011196d80 Tracing command intr pid 12 tid 100008 td 0xe000000011196000 Tracing command intr pid 12 tid 100007 td 0xe000000011196480 Tracing command intr pid 12 tid 100006 td 0xe000000011196900 Tracing command intr pid 12 tid 100005 td 0xe000000011178000 Tracing command idle pid 11 tid 100004 td 0xe000000011178480 Tracing command idle pid 11 tid 100003 td 0xe000000011178900 --More-- Tracing command init pid 1 tid 100002 td 0xe000000011178d80 Tracing command audit pid 10 tid 100001 td 0xe000000011179200 Tracing command kernel pid 0 tid 100053 td 0xe000000011a26d80 Tracing command kernel pid 0 tid 100050 td 0xe000000011930d80 Tracing command kernel pid 0 tid 100049 td 0xe000000011931200 Tracing command kernel pid 0 tid 100045 td 0xe000000011925680 Tracing command kernel pid 0 tid 100044 td 0xe000000011925b00 Tracing command kernel pid 0 tid 100023 td 0xe000000011218480 Tracing command kernel pid 0 tid 100020 td 0xe0000000111a6480 Tracing command kernel pid 0 tid 100018 td 0xe0000000111a6d80 --More-- Tracing command kernel pid 0 tid 100017 td 0xe0000000111a7200 Tracing command kernel pid 0 tid 100016 td 0xe0000000111a7680 Tracing command kernel pid 0 tid 100014 td 0xe000000011197200 Tracing command kernel pid 0 tid 100009 td 0xe000000011179b00 Tracing command kernel pid 0 tid 100000 td 0x9ffc000000eee1d0 db> I'll go check textdump setup again... Many thanks Anton From owner-freebsd-ia64@FreeBSD.ORG Tue Oct 15 10:26:03 2013 Return-Path: Delivered-To: freebsd-ia64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 17EC122C; Tue, 15 Oct 2013 10:26:03 +0000 (UTC) (envelope-from davide.italiano@gmail.com) Received: from mail-vb0-x236.google.com (mail-vb0-x236.google.com [IPv6:2607:f8b0:400c:c02::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B9AC8231B; Tue, 15 Oct 2013 10:26:02 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id q14so5160573vbe.13 for ; Tue, 15 Oct 2013 03:26:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=s7DQ0MDelyv7nT0h1Ggu1i7oanK1teHvtlyCywi/RoQ=; b=CSPZoscpGX/+C0xTxUeAJBSJQRcuWxgldkv78q2IgYtUypOtFjFiij4cMY3yjxqnVy NcqLdoa6FovWXP3aP0TYTLmiKUH0xoLlYbHRCtJ9lHm0A3NiyJCF4b5gA9CT77VhwLJH KAPyc32oBgBmMLuLFakJoeToNIl6DKsodtOfKz/TH0P+i3E6taF5hwo5cs0R73/LrJrp fFp4S4ziJetvt495XIvxWI6TeRHbRe2HIRTWU+ufGM3j4d7+Hp76A+AXHwusLYR5RiiT 4Ebh2xKE/bl//L/Y6DkXgS/rYpFDIp58aFcK8bxLpobYOF6+5Ot6Foc2uPrA5d9/7JF6 VpzQ== MIME-Version: 1.0 X-Received: by 10.220.1.203 with SMTP id 11mr11251898vcg.15.1381832761735; Tue, 15 Oct 2013 03:26:01 -0700 (PDT) Sender: davide.italiano@gmail.com Received: by 10.220.94.71 with HTTP; Tue, 15 Oct 2013 03:26:01 -0700 (PDT) In-Reply-To: <201310150843.r9F8ho1c043235@mech-cluster241.men.bris.ac.uk> References: <201310150843.r9F8ho1c043235@mech-cluster241.men.bris.ac.uk> Date: Tue, 15 Oct 2013 12:26:01 +0200 X-Google-Sender-Auth: J1uWHL85Gj2CzKF49DGZReO-iA8 Message-ID: Subject: Re: panic: wrong page state m 0xe00000027a9adb40 + savecore deadlock From: Davide Italiano To: mexas@bris.ac.uk Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-current , freebsd-ia64@freebsd.org X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 10:26:03 -0000 On Tue, Oct 15, 2013 at 10:43 AM, Anton Shterenlikht wrote: > Anyway, savecore eventually deadlocks: > > panic: deadlkres: possible deadlock detected for 0xe0000000127b7b00, blocked for 901401 ticks > [trim] > > Tracing command savecore pid 805 tid 100079 td 0xe0000000127b7b00 > cpu_switch(0xe0000000127b7b00, 0xe000000011178900, 0xe000000012402fc0, 0x9ffc0000005e7e80) at cpu_switch+0xd0 > sched_switch(0xe0000000127b7b00, 0xe000000011178900, 0x9ffc000000f15698, 0x9ffc000000f15680) at sched_switch+0x890 > mi_switch(0x103, 0x0, 0xe0000000127b7b00, 0x9ffc00000062d1f0) at mi_switch+0x3f0 > turnstile_wait(0xe000000012402fc0, 0xe000000012400480, 0x0, 0x9ffc000000dcb698) at turnstile_wait+0x960 > __mtx_lock_sleep(0x9ffc0000010f9998, 0xe0000000127b7b00, 0xe000000012402fc0, 0x9ffc000000dc0558, 0x742) at __mtx_lock_sleep+0x2f0 > __mtx_lock_flags(0x9ffc0000010f9980, 0x0, 0x9ffc000000dd4a90, 0x742) at __mtx_lock_flags+0x1e0 > vfs_vmio_release(0xa00000009ebe72f0, 0xe00000027ed2ab70, 0x3, 0xa00000009ebe736c, 0xa00000009ebe7498, 0xa00000009ebe72f8, 0x9ffc000000dd4a90, 0x9ffc0000010f9680) at vfs_vmio_release+0x290 > getnewbuf(0xe0000000127f4ec0, 0x0, 0x0, 0x8000, 0xa00000009ebe99a8, 0x0, 0x9ffc0000010f0798, 0xa00000009ebe72f0) at getnewbuf+0x7e0 > getblk(0xe0000000127f4ec0, 0x4cbaa, 0x8000, 0x0, 0x0, 0x0, 0x0, 0x0) at getblk+0xee0 > ffs_balloc_ufs2(0xe0000000127f4ec0, 0x4cbaa, 0xa0000000c60ba000, 0xe000000011165a00, 0x7f050000, 0xa00000009dd79160) at ffs_balloc_ufs2+0x2950 > ffs_write(0xa00000009dd79248, 0x3000, 0x265d50000) at ffs_write+0x5c0 > VOP_WRITE_APV(0x9ffc000000e94ac0, 0xa00000009dd79248, 0x0, 0x0) at VOP_WRITE_APV+0x330 > vn_write(0xe0000000129ae820, 0xa00000009dd79360, 0xe000000011165a00, 0x0, 0xe0000000129ae830, 0xe0000000127f4ec0) at vn_write+0x450 > vn_io_fault(0xe0000000129ae820, 0xa00000009dd79360, 0xe000000011165a00, 0x0, 0xe0000000127b7b00) at vn_io_fault+0x330 > dofilewrite(0xe0000000127b7b00, 0x7, 0xe0000000129ae820, 0xa00000009dd79360, 0xffffffffffffffff, 0x0) at dofilewrite+0x180 > kern_writev(0xe0000000127b7b00, 0x7, 0xa00000009dd79360) at kern_writev+0xa0 > sys_write(0xe0000000127b7b00, 0xa00000009dd794e8, 0x9ffc000000abac80, 0x48d) at sys_write+0x100 > syscall(0xe0000000129d04a0, 0x140857000, 0x8000, 0xe0000000127b7b00, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 > --More-- I'm not commenting on the first panic you got -- but on the deadlock reported by DEADLKRES. I think that's the vm_page lock. You can run kgdb /boot/${KERNEL}/kernel where ${KERNEL} is the incrimined one then l *vfs_vmio_release+0x290 to get the exact point where it fails. I'm unsure here because 'show alllocks' and 'show locks' outputs are empty -- are you building your kernel with WITNESS etc..? Thanks, -- Davide "There are no solved problems; there are only problems that are more or less solved" -- Henri Poincare From owner-freebsd-ia64@FreeBSD.ORG Tue Oct 15 12:19:30 2013 Return-Path: Delivered-To: freebsd-ia64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 6370DB9D for ; Tue, 15 Oct 2013 12:19:30 +0000 (UTC) (envelope-from mexas@bristol.ac.uk) Received: from eu1sys200aog110.obsmtp.com (eu1sys200aog110.obsmtp.com [207.126.144.129]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 783162CE7 for ; Tue, 15 Oct 2013 12:19:29 +0000 (UTC) Received: from mail-wi0-f181.google.com ([209.85.212.181]) (using TLSv1) by eu1sys200aob110.postini.com ([207.126.147.11]) with SMTP ID DSNKUl0ytgsBH4yu31YxwPlUK1U2LGqZ4tWI@postini.com; Tue, 15 Oct 2013 12:19:29 UTC Received: by mail-wi0-f181.google.com with SMTP id l12so4903627wiv.14 for ; Tue, 15 Oct 2013 05:19:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:message-id:to:subject:cc :reply-to:in-reply-to; bh=RfOk2gtNSoGNWHOJ16mbrs3U17bKsA1qKO3HI+8RIRE=; b=Ng+8qHzpZLY8nMXdaBEwuaDg31rBQgEO5nVKRZgfhfXXLpm4K5auBlU2jCrOrMNGpy OgjmKRRxPIrcHwMd7o08kVPFN/zNkngPZ8e/shkfkjSXA03+Lh+DV0c1LIEdW3GxSO/p mLamzRX77jwW+dQfv7L3RnMbxzoGQdnkmvZfRX55oqg/Zy6xdyEg8xrwTONdA0h7lwY1 MXU3NcNHi4VePuVw5cQMEqlOgR/m2G/XekCuU8tbA1tXPVUNnR/MN5yCH3sXYvu6cxVb MIovB0AfjiQeL9wBLpvYqcIWwmDCAbGWFFU1sQrA4S2vGL28OGn6VoTWgLf5biwVMurg j0gA== X-Gm-Message-State: ALoCoQlT8dSd33hSeLnj5UXCtS80lF45bnIXzZbj2P7U7N0wRdPqbV+gK55Sx0a0Q6xzZmtj5DRgWj6ldXhqGMC8xVzw0zKnjYWFW3xekNv4hSQe9l9+XI3rQAXVZBNjxjX0tJbyN3i/G8tig9P6GFEzGKlH9MW6lSdx3E4ka7eG8Ejxup9C9oo= X-Received: by 10.194.20.202 with SMTP id p10mr7276493wje.39.1381839541875; Tue, 15 Oct 2013 05:19:01 -0700 (PDT) X-Received: by 10.194.20.202 with SMTP id p10mr7276485wje.39.1381839541766; Tue, 15 Oct 2013 05:19:01 -0700 (PDT) Received: from mech-cluster241.men.bris.ac.uk (mech-cluster241.men.bris.ac.uk. [137.222.187.241]) by mx.google.com with ESMTPSA id i8sm5676961wiy.6.1969.12.31.16.00.00 (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 15 Oct 2013 05:19:00 -0700 (PDT) Sender: Anton Shterenlikht Received: from mech-cluster241.men.bris.ac.uk (localhost [127.0.0.1]) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6) with ESMTP id r9FCIwYh043809 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 15 Oct 2013 13:18:58 +0100 (BST) (envelope-from mexas@mech-cluster241.men.bris.ac.uk) Received: (from mexas@localhost) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6/Submit) id r9FCIwBx043808; Tue, 15 Oct 2013 13:18:58 +0100 (BST) (envelope-from mexas) Date: Tue, 15 Oct 2013 13:18:58 +0100 (BST) From: Anton Shterenlikht Message-Id: <201310151218.r9FCIwBx043808@mech-cluster241.men.bris.ac.uk> To: davide@freebsd.org, mexas@bris.ac.uk Subject: Re: panic: wrong page state m 0xe00000027a9adb40 + savecore deadlock In-Reply-To: Cc: freebsd-current@freebsd.org, freebsd-ia64@freebsd.org X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: mexas@bris.ac.uk List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 12:19:30 -0000 >From davide.italiano@gmail.com Tue Oct 15 11:30:07 2013 > >On Tue, Oct 15, 2013 at 10:43 AM, Anton Shterenlikht wrote: > >> Anyway, savecore eventually deadlocks: >> >> panic: deadlkres: possible deadlock detected for 0xe0000000127b7b00, blocked for 901401 ticks >> > >[trim] > >> >> Tracing command savecore pid 805 tid 100079 td 0xe0000000127b7b00 >> cpu_switch(0xe0000000127b7b00, 0xe000000011178900, 0xe000000012402fc0, 0x9ffc0000005e7e80) at cpu_switch+0xd0 >> sched_switch(0xe0000000127b7b00, 0xe000000011178900, 0x9ffc000000f15698, 0x9ffc000000f15680) at sched_switch+0x890 >> mi_switch(0x103, 0x0, 0xe0000000127b7b00, 0x9ffc00000062d1f0) at mi_switch+0x3f0 >> turnstile_wait(0xe000000012402fc0, 0xe000000012400480, 0x0, 0x9ffc000000dcb698) at turnstile_wait+0x960 >> __mtx_lock_sleep(0x9ffc0000010f9998, 0xe0000000127b7b00, 0xe000000012402fc0, 0x9ffc000000dc0558, 0x742) at __mtx_lock_sleep+0x2f0 >> __mtx_lock_flags(0x9ffc0000010f9980, 0x0, 0x9ffc000000dd4a90, 0x742) at __mtx_lock_flags+0x1e0 >> vfs_vmio_release(0xa00000009ebe72f0, 0xe00000027ed2ab70, 0x3, 0xa00000009ebe736c, 0xa00000009ebe7498, 0xa00000009ebe72f8, 0x9ffc000000dd4a90, 0x9ffc0000010f9680) at vfs_vmio_release+0x290 >> getnewbuf(0xe0000000127f4ec0, 0x0, 0x0, 0x8000, 0xa00000009ebe99a8, 0x0, 0x9ffc0000010f0798, 0xa00000009ebe72f0) at getnewbuf+0x7e0 >> getblk(0xe0000000127f4ec0, 0x4cbaa, 0x8000, 0x0, 0x0, 0x0, 0x0, 0x0) at getblk+0xee0 >> ffs_balloc_ufs2(0xe0000000127f4ec0, 0x4cbaa, 0xa0000000c60ba000, 0xe000000011165a00, 0x7f050000, 0xa00000009dd79160) at ffs_balloc_ufs2+0x2950 >> ffs_write(0xa00000009dd79248, 0x3000, 0x265d50000) at ffs_write+0x5c0 >> VOP_WRITE_APV(0x9ffc000000e94ac0, 0xa00000009dd79248, 0x0, 0x0) at VOP_WRITE_APV+0x330 >> vn_write(0xe0000000129ae820, 0xa00000009dd79360, 0xe000000011165a00, 0x0, 0xe0000000129ae830, 0xe0000000127f4ec0) at vn_write+0x450 >> vn_io_fault(0xe0000000129ae820, 0xa00000009dd79360, 0xe000000011165a00, 0x0, 0xe0000000127b7b00) at vn_io_fault+0x330 >> dofilewrite(0xe0000000127b7b00, 0x7, 0xe0000000129ae820, 0xa00000009dd79360, 0xffffffffffffffff, 0x0) at dofilewrite+0x180 >> kern_writev(0xe0000000127b7b00, 0x7, 0xa00000009dd79360) at kern_writev+0xa0 >> sys_write(0xe0000000127b7b00, 0xa00000009dd794e8, 0x9ffc000000abac80, 0x48d) at sys_write+0x100 >> syscall(0xe0000000129d04a0, 0x140857000, 0x8000, 0xe0000000127b7b00, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 >> --More-- > >I'm not commenting on the first panic you got -- but on the deadlock >reported by DEADLKRES. I think that's the vm_page lock. >You can run kgdb /boot/${KERNEL}/kernel where ${KERNEL} is the incrimined one >then l *vfs_vmio_release+0x290 >to get the exact point where it fails. Like this? # kgdb /boot/kernel/kernel GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "ia64-marcel-freebsd"... (kgdb) l *vfs_vmio_release+0x290 0x9ffc0000006b8830 is in vfs_vmio_release (/usr/src/sys/kern/vfs_bio.c:1859). 1854 /* 1855 * In order to keep page LRU ordering consistent, put 1856 * everything on the inactive queue. 1857 */ 1858 vm_page_lock(m); 1859 vm_page_unwire(m, 0); 1860 1861 /* 1862 * Might as well free the page if we can and it has 1863 * no valid data. We also free the page if the (kgdb) >I'm unsure here because 'show alllocks' and 'show locks' outputs are >empty -- are you building your kernel with WITNESS etc..? I think so: # Debugging support. Always need this: options KDB # Enable kernel debugger support. options KDB_TRACE # Print a stack trace for a panic. # For full debugger support use (turn off in stable branch): options DDB # Support DDB options GDB # Support remote GDB options DEADLKRES # Enable the deadlock resolver options INVARIANTS # Enable calls of extra sanity checking options INVARIANT_SUPPORT # required by INVARIANTS options WITNESS # Enable checks to detect deadlocks and cycles options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed options MALLOC_DEBUG_MAXZONES=8 # Separate malloc(9) zones # textdump(4) options TEXTDUMP_PREFERRED options TEXTDUMP_VERBOSE # http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-deadlocks.html options DEBUG_LOCKS options DEBUG_VFS_LOCKS options DIAGNOSTIC Also, does this look right: $ sysctl -a | grep kdb debug.ddb.scripting.scripts: kdb.enter.panic=textdump set; capture on; run lockinfo; show pcpu; bt; ps; alltrace; capture off; call doadump; reset kdb.enter.witness=run lockinfo debug.kassert.do_kdb: 0 debug.kdb.alt_break_to_debugger: 0 debug.kdb.break_to_debugger: 0 debug.kdb.trap_code: 0 debug.kdb.trap: 0 debug.kdb.panic: 0 debug.kdb.enter: 0 debug.kdb.current: ddb debug.kdb.available: ddb gdb debug.witness.kdb: 0 $ Thank you Anton From owner-freebsd-ia64@FreeBSD.ORG Tue Oct 15 15:10:16 2013 Return-Path: Delivered-To: freebsd-ia64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id EFBF5EAF for ; Tue, 15 Oct 2013 15:10:16 +0000 (UTC) (envelope-from mexas@bristol.ac.uk) Received: from eu1sys200aog106.obsmtp.com (eu1sys200aog106.obsmtp.com [207.126.144.121]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 57D1B2D28 for ; Tue, 15 Oct 2013 15:10:15 +0000 (UTC) Received: from mail-wi0-f177.google.com ([209.85.212.177]) (using TLSv1) by eu1sys200aob106.postini.com ([207.126.147.11]) with SMTP ID DSNKUl1az8cewRQ22613PR2+G15O0LTygpBE@postini.com; Tue, 15 Oct 2013 15:10:15 UTC Received: by mail-wi0-f177.google.com with SMTP id h11so944293wiv.16 for ; Tue, 15 Oct 2013 08:10:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:message-id:to:subject:reply-to; bh=KIaFojNslQIooXIGB9biXraDvdn0yCcCK5EVeJtf3Zw=; b=fmr3FpaVv35AhkjJrVtwyW7SCljMVwvtJHH7DXX23VVKPEFYF5fiyjkejGgtv/UIy0 nCiIdxbb5pcrW60emaVRgO8jLVhn8UCxZI5LcJhahmI4cJ7eeuT3/wZ+w1/MuECodldt 2cWI/rtWQIvU7prmVxqxdehNbt+rQygTWhCze6mGneIJD8SyO2wjoWnc8smHSfeG7SkK jQyYI+ZVsp1pIE/v2vA2GEh2MvOBB+8Qan7OjDZVLWLxr3BU0VA8JI96tnhUtyb6u1H2 L+lejJzE2MJHd2A1Dwt9uusKvvQyGMlGe7eOeh6eDodT2fhn2e346KLAjea94nGPgnfK 0S6Q== X-Gm-Message-State: ALoCoQmycQIh+PVfwoifqtpAKqgZ68MTwPz8xjbSy73pbfsnAarUAr+QmpP1TKZvtaQFetPuvUJj7qfDHTCScQs2l22CcWTUxkU9nQLfAiVB1B4K1qwGyBDnc9g7XYekq8bUGrLweUodX7zLKp2Ljuux2NhfTpkND0SAZWN9e3y1vfL9lIdoxcU= X-Received: by 10.194.201.202 with SMTP id kc10mr35081730wjc.1.1381849807804; Tue, 15 Oct 2013 08:10:07 -0700 (PDT) X-Received: by 10.194.201.202 with SMTP id kc10mr35081710wjc.1.1381849807638; Tue, 15 Oct 2013 08:10:07 -0700 (PDT) Received: from mech-cluster241.men.bris.ac.uk (mech-cluster241.men.bris.ac.uk. [137.222.187.241]) by mx.google.com with ESMTPSA id fr4sm48650wib.0.2013.10.15.08.10.05 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Oct 2013 08:10:06 -0700 (PDT) Sender: Anton Shterenlikht Received: from mech-cluster241.men.bris.ac.uk (localhost [127.0.0.1]) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6) with ESMTP id r9FFA4MT047274 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 15 Oct 2013 16:10:04 +0100 (BST) (envelope-from mexas@mech-cluster241.men.bris.ac.uk) Received: (from mexas@localhost) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6/Submit) id r9FFA4fH047273; Tue, 15 Oct 2013 16:10:04 +0100 (BST) (envelope-from mexas) Date: Tue, 15 Oct 2013 16:10:04 +0100 (BST) From: Anton Shterenlikht Message-Id: <201310151510.r9FFA4fH047273@mech-cluster241.men.bris.ac.uk> To: freebsd-current@freebsd.org, freebsd-ia64@freebsd.org Subject: ia64: panic: wrong page state m 0xe00000027fcc1900 X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: mexas@bris.ac.uk List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 15:10:17 -0000 This panic is always reproducible by starting nginx, and directing the browser to poudriere logs/bulk/ia64-default/latest/. I tried to do manually: show proc show threads run lockinfo show pcpu bt ps alltrace since I cannot get savecore to run successfully. Thanks Anton db> show proc Process 1035 (nginx) at 0xe0000000128c1280: state: NORMAL uid: 80 gids: 80 parent: pid 1034 at 0xe00000001273e940 ABI: FreeBSD ELF64 arguments: nginx: worker process threads: 1 100097 Run CPU 0 nginx db> db> show threads 100097 (0xe000000012800000) (stack 0xa00000009df7e000) kdb_enter(0x9ffc000000dcfac8, 0x9ffc000000dcfac8, 0x9ffc000000592a90, 0x40c) at kdb_enter+0x92 100078 (0xe0000000124bc000) (stack 0xa00000009defe000) cpu_switch(0xe0000000124bc000, 0xe000000011182900, 0x9ffc000000f5c148, 0x9ffc0000005ec410) at cpu_switch+0xd0 100072 (0xe000000012500d80) (stack 0xa00000009dece000) cpu_switch(0xe000000012500d80, 0xe000000011182900, 0x9ffc000000f5d160, 0x9ffc0000005ec410) at cpu_switch+0xd0 100071 (0xe000000012501200) (stack 0xa00000009dec6000) cpu_switch(0xe000000012501200, 0xe000000011182900, 0x9ffc000000f5dfc0, 0x9ffc0000005ec410) at cpu_switch+0xd0 100069 (0xe0000000123fed80) (stack 0xa00000009deb6000) cpu_switch(0xe0000000123fed80, 0xe000000011182480, 0x9ffc000000f5e510, 0x9ffc0000005ec410) at cpu_switch+0xd0 100081 (0xe0000000127a0000) (stack 0xa00000009df16000) cpu_switch(0xe0000000127a0000, 0xe000000011182480, 0x9ffc000000f5cdf0, 0x9ffc0000005ec410) at cpu_switch+0xd0 100073 (0xe000000012500900) (stack 0xa00000009ded6000) cpu_switch(0xe000000012500900, 0xe000000011182900, 0x9ffc000000f5e1c8, 0x9ffc0000005ec410) at cpu_switch+0xd0 100099 (0xe000000012890480) (stack 0xa00000009df8e000) cpu_switch(0xe000000012890480, 0xe000000011182900, 0x9ffc000000f5c620, 0x9ffc0000005ec410) at cpu_switch+0xd0 100098 (0xe0000000127a1b00) (stack 0xa00000009df86000) cpu_switch(0xe0000000127a1b00, 0xe000000011182900, 0x9ffc000000f5d0c0, 0x9ffc0000005ec410) at cpu_switch+0xd0 100088 (0xe0000000124bcd80) (stack 0xa00000009df36000) cpu_switch(0xe0000000124bcd80, 0xe0000000127a1680, 0x9ffc000000f5de08, 0x9ffc0000005ec410) at cpu_switch+0xd0 100096 (0xe000000012800480) (stack 0xa00000009df76000) cpu_switch(0xe000000012800480, 0xe000000011182480, 0x9ffc000000f5e5b0, 0x9ffc0000005ec410) at cpu_switch+0xd0 100089 (0xe0000000127a1680) (stack 0xa00000009df3e000) cpu_switch(0xe0000000127a1680, 0xe000000011182900, 0x9ffc000000f5d2f0, 0x9ffc0000005ec410) at cpu_switch+0xd0 100075 (0xe000000012500000) (stack 0xa00000009dee6000) cpu_switch(0xe000000012500000, 0xe000000011182900, 0x9ffc000000f5cd78, 0x9ffc0000005ec410) at cpu_switch+0xd0 100076 (0xe0000000124bc900) (stack 0xa00000009deee000) cpu_switch(0xe0000000124bc900, 0xe000000011182480, 0x9ffc000000f5dde0, 0x9ffc0000005ec410) at cpu_switch+0xd0 100079 (0xe000000012501b00) (stack 0xa00000009df06000) cpu_switch(0xe000000012501b00, 0xe0000000127a1200, 0x9ffc000000f5dc28, 0x9ffc0000005ec410) at cpu_switch+0xd0 100080 (0xe000000012501680) (stack 0xa00000009df0e000) cpu_switch(0xe000000012501680, 0xe000000011182480, 0x9ffc000000f5e290, 0x9ffc0000005ec410) at cpu_switch+0xd0 100067 (0xe0000000123ff680) (stack 0xa00000009dea6000) cpu_switch(0xe0000000123ff680, 0xe000000011182900, 0x9ffc000000f5ceb8, 0x9ffc0000005ec410) at cpu_switch+0xd0 100077 (0xe0000000124bc480) (stack 0xa00000009def6000) cpu_switch(0xe0000000124bc480, 0xe000000011182480, 0x9ffc000000f5cee0, 0x9ffc0000005ec410) at cpu_switch+0xd0 100068 (0xe0000000123ff200) (stack 0xa00000009deae000) cpu_switch(0xe0000000123ff200, 0xe000000011182900, 0x9ffc000000f5ce18, 0x9ffc0000005ec410) at cpu_switch+0xd0 100056 (0xe000000012168d80) (stack 0xa00000009de4e000) cpu_switch(0xe000000012168d80, 0xe000000011182480, 0x9ffc000000f5e308, 0x9ffc0000005ec410) at cpu_switch+0xd0 100055 (0xe00000001178c480) (stack 0xa00000009de46000) cpu_switch(0xe00000001178c480, 0xe000000011182480, 0x9ffc000000f5e358, 0x9ffc0000005ec410) at cpu_switch+0xd0 100074 (0xe000000012500480) (stack 0xa00000009dede000) cpu_switch(0xe000000012500480, 0xe000000011182900, 0x9ffc000000f5d548, 0x9ffc0000005ec410) at cpu_switch+0xd0 100064 (0xe000000012169680) (stack 0xa00000009de8e000) cpu_switch(0xe000000012169680, 0xe00000001178d680, 0x9ffc000000f5c5a8, 0x9ffc0000005ec410) at cpu_switch+0xd0 100063 (0xe000000012169b00) (stack 0xa00000009de86000) cpu_switch(0xe000000012169b00, 0xe000000012169680, 0x9ffc000000f5c9b8, 0x9ffc0000005ec410) at cpu_switch+0xd0 100062 (0xe0000000123fe000) (stack 0xa00000009de7e000) cpu_switch(0xe0000000123fe000, 0xe000000011182480, 0x9ffc000000f5d6b0, 0x9ffc0000005ec410) at cpu_switch+0xd0 100061 (0xe0000000123fe480) (stack 0xa00000009de76000) cpu_switch(0xe0000000123fe480, 0xe0000000111a1680, 0x9ffc000000f5e560, 0x9ffc0000005ec410) at cpu_switch+0xd0 100060 (0xe00000001178db00) (stack 0xa00000009de6e000) cpu_switch(0xe00000001178db00, 0xe000000011182480, 0x9ffc000000f5e4c0, 0x9ffc0000005ec410) at cpu_switch+0xd0 100059 (0xe000000012168000) (stack 0xa00000009de66000) cpu_switch(0xe000000012168000, 0xe0000000123fe480, 0x9ffc000000f5c300, 0x9ffc0000005ec410) at cpu_switch+0xd0 100058 (0xe000000012168480) (stack 0xa00000009de5e000) cpu_switch(0xe000000012168480, 0xe0000000123fe480, 0x9ffc000000f5cf80, 0x9ffc0000005ec410) at cpu_switch+0xd0 100057 (0xe000000012168900) (stack 0xa00000009de56000) cpu_switch(0xe000000012168900, 0xe000000011182480, 0x9ffc000000f5d818, 0x9ffc0000005ec410) at cpu_switch+0xd0 100054 (0xe00000001178c900) (stack 0xa00000009de3e000) cpu_switch(0xe00000001178c900, 0xe000000011182480, 0x9ffc000000f5d2c8, 0x9ffc0000005ec410) at cpu_switch+0xd0 100052 (0xe00000001178d200) (stack 0xa00000009de2c000) cpu_switch(0xe00000001178d200, 0xe000000011182900, 0x9ffc000000f5ded0, 0x9ffc0000005ec410) at cpu_switch+0xd0 100051 (0xe00000001178d680) (stack 0xa00000009de24000) cpu_switch(0xe00000001178d680, 0xe0000000123fe000, 0x9ffc000000f5c620, 0x9ffc0000005ec410) at cpu_switch+0xd0 100047 (0xe00000001174db00) (stack 0xa00000009de04000) cpu_switch(0xe00000001174db00, 0xe000000011425b00, 0x9ffc000000f5d520, 0x9ffc0000005ec410) at cpu_switch+0xd0 100043 (0xe00000001174c000) (stack 0xa00000009ddc6000) cpu_switch(0xe00000001174c000, 0xe00000001174db00, 0x9ffc000000f5d020, 0x9ffc0000005ec410) at cpu_switch+0xd0 100041 (0xe00000001174c900) (stack 0xa00000009dd68000) cpu_switch(0xe00000001174c900, 0xe00000001174c000, 0x9ffc000000f5e420, 0x9ffc0000005ec410) at cpu_switch+0xd0 100039 (0xe000000011744480) (stack 0xa00000009dd0a000) cpu_switch(0xe000000011744480, 0xe00000001174c900, 0x9ffc000000f5dd90, 0x9ffc0000005ec410) at cpu_switch+0xd0 100038 (0xe000000011744900) (stack 0xa00000009dd02000) cpu_switch(0xe000000011744900, 0xe000000011182900, 0x9ffc000000f5d228, 0x9ffc0000005ec410) at cpu_switch+0xd0 100037 (0xe000000011744d80) (stack 0xa00000009dcfa000) cpu_switch(0xe000000011744d80, 0xe000000011744900, 0x9ffc000000f5dfe8, 0x9ffc0000005ec410) at cpu_switch+0xd0 100036 (0xe000000011745200) (stack 0xa00000009dcf2000) cpu_switch(0xe000000011745200, 0xe000000011744d80, 0x9ffc000000f5dea8, 0x9ffc0000005ec410) at cpu_switch+0xd0 100034 (0xe000000011480d80) (stack 0xa0000000e2fe6000) cpu_switch(0xe000000011480d80, 0xe000000011745200, 0x9ffc000000f5d340, 0x9ffc0000005ec410) at cpu_switch+0xd0 100033 (0xe000000011481200) (stack 0xa0000000e2fde000) cpu_switch(0xe000000011481200, 0xe000000011182900, 0x9ffc000000f5c580, 0x9ffc0000005ec410) at cpu_switch+0xd0 100032 (0xe000000011481680) (stack 0xa0000000e2fd6000) cpu_switch(0xe000000011481680, 0xe000000011182480, 0x9ffc000000f5da98, 0x9ffc0000005ec410) at cpu_switch+0xd0 100031 (0xe000000011481b00) (stack 0xa0000000e2fce000) cpu_switch(0xe000000011481b00, 0xe000000011182480, 0x9ffc000000f5ccd8, 0x9ffc0000005ec410) at cpu_switch+0xd0 100029 (0xe000000011425680) (stack 0xa0000000e2f36000) cpu_switch(0xe000000011425680, 0xe000000011182480, 0x9ffc000000f5dd40, 0x9ffc0000005ec410) at cpu_switch+0xd0 100028 (0xe000000011425b00) (stack 0xa0000000e2f2e000) cpu_switch(0xe000000011425b00, 0xe000000011182480, 0x9ffc000000f5e380, 0x9ffc0000005ec410) at cpu_switch+0xd0 100027 (0xe000000011480000) (stack 0xa0000000e2f26000) cpu_switch(0xe000000011480000, 0xe000000011425b00, 0x9ffc000000f5d098, 0x9ffc0000005ec410) at cpu_switch+0xd0 100026 (0xe000000011480480) (stack 0xa0000000e2f1e000) cpu_switch(0xe000000011480480, 0xe000000011480000, 0x9ffc000000f5d6d8, 0x9ffc0000005ec410) at cpu_switch+0xd0 100024 (0xe000000011424000) (stack 0xa0000000e2e8a000) cpu_switch(0xe000000011424000, 0xe000000011182480, 0x9ffc000000f5cf30, 0x9ffc0000005ec410) at cpu_switch+0xd0 100013 (0xe0000000111a1680) (stack 0xa0000000a07f6000) cpu_switch(0xe0000000111a1680, 0xe000000011182480, 0x9ffc000000f5c968, 0x9ffc0000005ec410) at cpu_switch+0xd0 100012 (0xe0000000111a1b00) (stack 0xa0000000a07ee000) cpu_switch(0xe0000000111a1b00, 0xe000000011182480, 0x9ffc000000f5d070, 0x9ffc0000005ec410) at cpu_switch+0xd0 100011 (0xe0000000111b0000) (stack 0xa0000000a07e6000) cpu_switch(0xe0000000111b0000, 0xe000000011182480, 0x9ffc000000f5d430, 0x9ffc0000005ec410) at cpu_switch+0xd0 100010 (0xe000000011183680) (stack 0xa0000000a07de000) cpu_switch(0xe000000011183680, 0xe000000011182900, 0x9ffc000000f5ccb0, 0x9ffc0000005ec410) at cpu_switch+0xd0 100048 (0xe00000001174d680) (stack 0xa00000009de0c000) cpu_switch(0xe00000001174d680, 0xe000000012500d80, 0x9ffc000000f27100, 0x9ffc0000005ec410) at cpu_switch+0xd0 100046 (0xe00000001178c000) (stack 0xa00000009ddde000) cpu_switch(0xe00000001178c000, 0xe000000011182480, 0x9ffc000000f27100, 0x9ffc0000005ec410) at cpu_switch+0xd0 100042 (0xe00000001174c480) (stack 0xa00000009dd70000) cpu_switch(0xe00000001174c480, 0xe000000011182900, 0x9ffc000000f26400, 0x9ffc0000005ec410) at cpu_switch+0xd0 100040 (0xe000000011744000) (stack 0xa00000009dd12000) cpu_switch(0xe000000011744000, 0xe000000011182480, 0x9ffc000000f27100, 0x9ffc0000005ec410) at cpu_switch+0xd0 100035 (0xe000000011480900) (stack 0xa00000009dcea000) fork_trampoline() at fork_trampoline 100030 (0xe000000011425200) (stack 0xa0000000e2fc6000) fork_trampoline() at fork_trampoline 100025 (0xe0000000111b1b00) (stack 0xa0000000e2f16000) fork_trampoline() at fork_trampoline 100022 (0xe000000011424900) (stack 0xa0000000e2e7a000) cpu_switch(0xe000000011424900, 0xe000000011182900, 0x9ffc000000f26400, 0x9ffc0000005ec410) at cpu_switch+0xd0 100021 (0xe000000011424d80) (stack 0xa0000000e2e72000) cpu_switch(0xe000000011424d80, 0xe000000011182900, 0x9ffc000000f26400, 0x9ffc0000005ec410) at cpu_switch+0xd0 100019 (0xe0000000111b0900) (stack 0xa0000000e2e62000) cpu_switch(0xe0000000111b0900, 0xe000000011182480, 0x9ffc000000f27100, 0x9ffc0000005ec410) at cpu_switch+0xd0 100018 (0xe0000000111b0d80) (stack 0xa0000000e2e5a000) cpu_switch(0xe0000000111b0d80, 0xe000000011182900, 0x9ffc000000f26400, 0x9ffc0000005ec410) at cpu_switch+0xd0 100008 (0xe0000000111a0000) (stack 0xa0000000a07ce000) fork_trampoline() at fork_trampoline 100007 (0xe0000000111a0480) (stack 0xa0000000a07c6000) cpu_switch(0xe0000000111a0480, 0xe000000011182480, 0x9ffc000000f27100, 0x9ffc0000005ec410) at cpu_switch+0xd0 100006 (0xe0000000111a0900) (stack 0xa0000000a07be000) fork_trampoline() at fork_trampoline 100005 (0xe000000011182000) (stack 0xa0000000a07b6000) cpu_switch(0xe000000011182000, 0xe000000011182480, 0x9ffc000000f27100, 0x9ffc0000005ec410) at cpu_switch+0xd0 100004 (0xe000000011182480) (stack 0xa0000000a07ae000) ia64_ih_stop(0x9ffc000000aa9070, 0x40b) at ia64_ih_stop+0x80 100003 (0xe000000011182900) (stack 0xa0000000a07a6000) cpu_switch(0xe000000011182900, 0xe000000012800000, 0x9ffc000000f26400, 0x9ffc0000005ec410) at cpu_switch+0xd0 100002 (0xe000000011182d80) (stack 0xa0000000a079e000) cpu_switch(0xe000000011182d80, 0xe000000011182480, 0x9ffc000000f5e628, 0x9ffc0000005ec410) at cpu_switch+0xd0 100001 (0xe000000011183200) (stack 0xa0000000a0796000) cpu_switch(0xe000000011183200, 0xe000000011183680, 0x9ffc000000f5cd78, 0x9ffc0000005ec410) at cpu_switch+0xd0 100053 (0xe00000001178cd80) (stack 0xa00000009de34000) cpu_switch(0xe00000001178cd80, 0xe0000000124bc480, 0x9ffc000000f5def8, 0x9ffc0000005ec410) at cpu_switch+0xd0 100050 (0xe00000001174cd80) (stack 0xa00000009de1c000) cpu_switch(0xe00000001174cd80, 0xe00000001178cd80, 0x9ffc000000f5ce18, 0x9ffc0000005ec410) at cpu_switch+0xd0 100049 (0xe00000001174d200) (stack 0xa00000009de14000) cpu_switch(0xe00000001174d200, 0xe00000001174cd80, 0x9ffc000000f5ce18, 0x9ffc0000005ec410) at cpu_switch+0xd0 100045 (0xe000000011745680) (stack 0xa00000009ddd6000) cpu_switch(0xe000000011745680, 0xe000000011182480, 0x9ffc000000f5c6e8, 0x9ffc0000005ec410) at cpu_switch+0xd0 100044 (0xe000000011745b00) (stack 0xa00000009ddce000) cpu_switch(0xe000000011745b00, 0xe000000011182900, 0x9ffc000000f5c710, 0x9ffc0000005ec410) at cpu_switch+0xd0 100023 (0xe000000011424480) (stack 0xa0000000e2e82000) cpu_switch(0xe000000011424480, 0xe000000011182900, 0x9ffc000000f5ce18, 0x9ffc0000005ec410) at cpu_switch+0xd0 100020 (0xe0000000111b0480) (stack 0xa0000000e2e6a000) cpu_switch(0xe0000000111b0480, 0xe000000011182480, 0x9ffc000000f5ce90, 0x9ffc0000005ec410) at cpu_switch+0xd0 100017 (0xe0000000111b1200) (stack 0xa0000000e2e52000) cpu_switch(0xe0000000111b1200, 0xe0000000111b0480, 0x9ffc000000f5cf08, 0x9ffc0000005ec410) at cpu_switch+0xd0 111b0480, 0x9ffc000000f5cf08, 0x9ffc0000005ec410) at cpu_switch+0xd0 100016 (0xe0000000111b1680) (stack 0xa0000000e2e4a000) cpu_switch(0xe0000000111b1680, 0xe0000000111b1200, 0x9ffc000000f5cf08, 0x9ffc0000005ec410) at cpu_switch+0xd0 100015 (0xe0000000111a0d80) (stack 0xa0000000e2e42000) cpu_switch(0xe0000000111a0d80, 0xe0000000111b1680, 0x9ffc000000f5cf08, 0x9ffc0000005ec410) at cpu_switch+0xd0 100014 (0xe0000000111a1200) (stack 0xa0000000e2e3a000) cpu_switch(0xe0000000111a1200, 0xe0000000111a0d80, 0x9ffc000000f5cf30, 0x9ffc0000005ec410) at cpu_switch+0xd0 100009 (0xe000000011183b00) (stack 0xa0000000a07d6000) cpu_switch(0xe000000011183b00, 0xe000000011182480, 0x9ffc000000f5cfa8, 0x9ffc0000005ec410) at cpu_switch+0xd0 100000 (0x9ffc000000efef70) (stack 0xe0000040fffc0000) cpu_switch(0x9ffc000000efef70, 0xe000000011182900, 0x9ffc000000f5c670, 0x9ffc0000005ec410) at cpu_switch+0xd0 db> db> show msgbuf msgbufp = 0xe0000040fffdffb8 magic = 63062, size = 98232, r= 10746, w = 11693, ptr = 0xe0000040fffc8000, cksum= 890849 panic: wrong page state m 0xe00000027fcc1900 cpuid = 0 KDB: stack backtrace: db_trace_self(0x9ffc000000158380) at db_trace_self+0x40 db_trace_self_wrapper(0x9ffc00000060b900) at db_trace_self_wrapper+0x70 kdb_backtrace(0x9ffc000000ee1b48, 0x9ffc000000592a40, 0x40c, 0x9ffc0000010b48a0) at kdb_backtrace+0xc0 vpanic(0x9ffc000000de0c20, 0xa00000009df85118, 0x9ffc000000f0a410, 0x9ffc000000ee18f0) at vpanic+0x260 kassert_panic(0x9ffc000000de0c20, 0xe00000027fcc1900, 0x81f, 0xe0000000127a7800, 0x9ffc0000006a4f10, 0x2c60, 0xe0000000127a7800, 0xe0000000127a7818) at kassert_panic+0x120 vn_sendfile(0x8df, 0x3, 0x0, 0x0, 0x0, 0x8df, 0x7fffffffffffe000, 0x0) at vn_sendfile+0x15d0 sys_sendfile(0xe000000012800000, 0xa00000009df854e8, 0x9, 0xa00000009df85360) at sys_sendfile+0x2b0 syscall(0xe0000000128c1280, 0x3, 0x0, 0xe000000012800000, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return KDB: enter: panic GDB: debug ports: uart GDB: current port: uart KDB: debugger backends: ddb gdb KDB: current backend: ddb Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #7 r255488: Tue Oct 15 10:09:07 BST 2013 root@mech-as28.men.bris.ac.uk:/usr/obj/usr/src/sys/ZEEV ia64 gcc version 4.2.1 20070831 patched [FreeBSD] WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: Madison II (1600 MHz Itanium 2) Origin = "GenuineIntel" Revision = 2 Features = 0x1 real memory = 10737418240 (10240 MB) avail memory = 10484695040 (9998 MB) FPSWA Revision = 0x10012, Entry = 0xe0000040ffcc4050 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0: ACPI Id=0, SAPIC Id=0, SAPIC Eid=0 (BSP) cpu1: ACPI Id=1, SAPIC Id=1, SAPIC Eid=0 random: initialized Event timer "ITC" frequency 1600000000 Hz quality 1000 ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20130823/tbfadt-603) ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20130823/tbfadt-603) acpi0: on motherboard acpi0: Power Button (fixed) acpi0: Sleep Button (fixed) cpu0: on acpi0 cpu1: on acpi0 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <32-bit timer at 3.579545MHz> iomem 0xff5c1004-0xff5c1007 on acpi0 acpi_tz0: on acpi0 pcib0: on acpi0 pcib0: could not evaluate _ADR - AE_NOT_FOUND pci0: on pcib0 ohci0: mem 0x80002000-0x80002fff irq 16 at device 1.0 on pci0 usbus0 on ohci0 ohci1: mem 0x80001000-0x80001fff irq 17 at device 1.1 on pci0 usbus1 on ohci1 ehci0: mem 0x80000000-0x800000ff irq 18 at device 1.2 on pci0 usbus2: EHCI version 0.95 usbus2 on ehci0 pci0: at device 2.0 (no driver attached) pcib1: on acpi0 pcib1: could not evaluate _ADR - AE_NOT_FOUND pci32: on pcib1 mpt0: port 0x2100-0x21ff mem 0x903a0000-0x903bffff,0x90380000-0x9039ffff irq 27 at device 1.0 on pci32 mpt0: MPI Version=1.2.12.0 mpt1: port 0x2000-0x20ff mem 0x90360000-0x9037ffff,0x90340000-0x9035ffff irq 28 at device 1.1 on pci32 mpt1: MPI Version=1.2.12.0 em0: port 0x2240-0x227f mem 0x90320000-0x9033ffff,0x90280000-0x902fffff irq 29 at device 2.0 on pci32 em0: Ethernet address: 00:13:21:5b:05:1c em1: port 0x2200-0x223f mem 0x90300000-0x9031ffff irq 30 at device 2.1 on pci32 em1: Ethernet address: 00:13:21:5b:05:1d pcib2: on acpi0 pcib2: could not evaluate _ADR - AE_NOT_FOUND pci64: on pcib2 pcib3: on acpi0 pcib3: could not evaluate _ADR - AE_NOT_FOUND pci96: on pcib3 pcib4: on acpi0 pcib4: could not evaluate _ADR - AE_NOT_FOUND pci128: on pcib4 pcib5: on acpi0 pcib5: could not evaluate _ADR - AE_NOT_FOUND pci192: on pcib5 isp0: port 0xc000-0xc0ff mem 0xe0040000-0xe0040fff irq 71 at device 1.0 on pci192 pcib6: on acpi0 pcib6: could not evaluate _ADR - AE_NOT_FOUND pci224: on pcib6 uart0: mem 0xf8051000-0xf805100f irq 82 at device 1.0 on pci224 puc0: mem 0xf8050000-0xf8050fff,0xf8020000-0xf803ffff irq 82 at device 1.1 on pci224 uart1: at port 1 on puc0 uart1: console (9600,n,8,1) uart2: at port 3 on puc0 vgapci0: port 0xe000-0xe0ff mem 0xf0000000-0xf7ffffff,0xf8040000-0xf804ffff at device 2.0 on pci224 uart3: <16550 or compatible> iomem 0xff5e0000-0xff5e0007 irq 34 on acpi0 uart4: <16550 or compatible> iomem 0xff5e2000-0xff5e2007 irq 35 on acpi0 uart4: debug port (9600,n,8,1) ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 IP Filter: v5.1.2 initialized. Default = block all, Logging = enabled usbus1: 12Mbps Full Speed USB v1.0 usbus2: 480Mbps High Speed USB v2.0 ugen2.1: at usbus2 uhub0: on usbus2 ugen0.1: at usbus0 uhub1: on usbus0 ugen1.1: at usbus1 uhub2: on usbus1 uhub2: 2 ports with 2 removable, self powered uhub1: 3 ports with 3 removable, self powered uhub0: 5 ports with 5 removable, self powered da0 at mpt0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-3 device da0: 320.000MB/s transfers (160.000MHz, offset 63, 16bit) da0: Command Queueing enabled da0: 70007MB (143374738 512 byte sectors: 255H 63S/T 8924C) da1 at mpt0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SCSI-3 device da1: 160.000MB/s transfers (80.000MHz, offset 63, 16bit) da1: Command Queueing enabled da1: 17366MB (35566478 512 byte sectors: 255H 63S/T 2213C) da3 at isp0 bus 0 scbus2 target 0 lun 1 da3: Fixed Direct Access SCSI-4 device da3: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000 da3: Command Queueing enabled da3: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C) da4 at isp0 bus 0 scbus2 target 0 lun 2 da4: Fixed Direct Access SCSI-4 device da4: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000 da4: Command Queueing enabled da4: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C) da5 at isp0 bus 0 scbus2 target 0 lun 3 da5: Fixed Direct Access SCSI-4 device da5: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000 da5: Command Queueing enabled da5: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C) da6 at isp0 bus 0 scbus2 target 0 lun 5 da6: Fixed Direct Access SCSI-4 device da6: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000 da6: Command Queueing enabled da6: 140011MB (286744185 512 byte sectors: 255H 63S/T 17849C) da2 at mpt1 bus 0 scbus1 target 2 lun 0 da2: Fixed Direct Access SCSI-3 device da2: 160.000MB/s transfers (80.000MHz, offset 63, 16bit) da2: Command Queueing enabled da2: 17366MB (35566478 512 byte sectors: 255H 63S/T 2213C) pass3 at isp0 bus 0 scbus2 target 0 lun 0 pass3: Fixed Storage Array SCSI-4 device pass3: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000 pass3: Command Queueing enabled WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Trying to mount root from ufs:/dev/da0p2 [rw]... WARNING: / was not properly dismounted WITNESS: unable to allocate a new witness object <118>Setting hostuuid: ee652761-3dcc-11df-9074-0013215b051c. <118>Setting hostid: 0x8968724b. <118>Starting ddb. <118>Entropy harvesting: interrupts ethernet point_to_point Expensive timeout(9) function: 0x9ffc000000e222e0(0xa0000000009f1320) 0.002119543 s <118> kickstart. <118>Starting file system checks: <118>/dev/da0p2: 582356 files, 3927896 used, 30688528 free (183320 frags, 3813151 blocks, 0.5% fragmentation) <118>/dev/da3p1: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) <118>/dev/da3p1: SUMMARY INFORMATION BAD (SALVAGED) <118>/dev/da3p1: BLK(S) MISSING IN BIT MAPS (SALVAGED) <118>/dev/da3p1: 1968 files, 7223676 used, 9999685 free (245 frags, 1249930 blocks, 0.0% fragmentation) <118>/dev/da5p1: 436860 files, 1247548 used, 15975813 free (36405 frags, 1992426 blocks, 0.2% fragmentation) <118>/dev/da6p1: 5481 files, 1323234 used, 68104269 free (5773 frags, 8512312 blocks, 0.0% fragmentation) <118>Mounting local file systems:. <118>Writing entropy file:. <118>Setting hostname: mech-as28.men.bris.ac.uk. <118>Enabling ipfilter. <118>Starting Network: lo0 em0 em1. <118>lo0: flags=8049 metric 0 mtu 16384 <118> options=600003 <118> inet6 ::1 prefixlen 128 <118> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 <118> inet 127.0.0.1 netmask 0xff000000 <118> nd6 options=21 <118>em0: flags=8843 metric 0 mtu 1500 <118> options=209b <118> ether 00:13:21:5b:05:1c <118> inet 137.222.187.28 netmask 0xffffff00 broadcast 137.222.187.255 <118> inet6 fe80::213:21ff:fe5b:51c%em0 prefixlen 64 scopeid 0x1 <118> nd6 options=29 <118> media: Ethernet autoselect <118> status: no carrier <118>em1: flags=8843 metric 0 mtu 1500 <118> options=209b <118> ether 00:13:21:5b:05:1d <118> inet 10.10.10.14 netmask 0xffffff00 broadcast 10.10.10.255 <118> inet6 fe80::213:21ff:fe5b:51d%em1 prefixlen 64 scopeid 0x2 <118> nd6 options=29 <118> media: Ethernet autoselect (100baseTX ) <118> status: active <118>filter sync'd <118>Starting devd. <118>add net default: gateway 137.222.187.250 <118>add net fe80::: gateway ::1 <118>add net ff02::: gateway ::1 <118>add net ::ffff:0.0.0.0: gateway ::1 <118>add net ::0.0.0.0: gateway ::1 <118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib <118>Starting ipmon. <118>Creating and/or trimming log files. <118>Starting syslogd. <118>No core dumps found. <118>Clearing /tmp (X related). <118>Recovering vi editor sessions:. <118>Updating motd:. <118>Mounting late file systems:. <118>Starting ntpd. <118>Starting bsdstats. <118>Posting monthly OS statistics to rpt.bsdstats.org <118>Performing sanity check on sshd configuration. <118>Starting sshd. <118>Starting cron. <118>Local package initialization:cd: /var/portbuild/: No such file or directory <118>. <118>Starting inetd. <118>Starting background file system checks in 60 seconds. <118> <118>Tue Oct 15 15:38:25 BST 2013 <118>Oct 15 15:39:44 mech-as28 su: mexas to root on /dev/pts/0 db> db> run lockinfo db:0:lockinfo> show locks db:0:locks> show alllocks db:0:alllocks> show lockedvnods Locked vnodes 0xe0000000129789c0: tag ufs, type VREG usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0xe0000000127a7800 ref 1 pages 1 lock type ufs: SHARED (count 1) ino 5220225, on dev da5p1 db> db> show pcpu cpuid = 0 dynamic pcpu = 0x40040040ff12c680 curthread = 0xe000000012800000: pid 1035 "nginx" curpcb = 0 fpcurthread = none idlethread = 0xe000000011182900: tid 100003 "idle: cpu0" MD: vhpt = 0xe0000040ff000000 MD: lid = 0 MD: clock = 0x64261bc9ce MD: clock_mode = 2 MD: clock_load = 0 MD: stats = 0x9ffc00000110feb0 MD: pmap = 0xe00000001118ef00 spin locks held: db> db> bt Tracing pid 1035 tid 100097 td 0xe000000012800000 kdb_enter(0x9ffc000000dcfac8, 0x9ffc000000dcfac8, 0x9ffc000000592a90, 0x40c) at kdb_enter+0x92 vpanic(0x9ffc000000de0c20, 0xa00000009df85118, 0x100, 0x9ffc000000ee18f0) at vpanic+0x2b0 kassert_panic(0x9ffc000000de0c20, 0xe00000027fcc1900, 0x81f, 0xe0000000127a7800, 0x9ffc0000006a4f10, 0x2c60, 0xe0000000127a7800, 0xe0000000127a7818) at kassert_panic+0x120 vn_sendfile(0x8df, 0x3, 0x0, 0x0, 0x0, 0x8df, 0x7fffffffffffe000, 0x0) at vn_sendfile+0x15d0 sys_sendfile(0xe000000012800000, 0xa00000009df854e8, 0x9, 0xa00000009df85360) at sys_sendfile+0x2b0 syscall(0xe0000000128c1280, 0x3, 0x0, 0xe000000012800000, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return db> db> ps pid ppid pgrp uid state wmesg wchan cmd 1035 1034 1034 80 R CPU 0 nginx 1034 1 1034 0 Ss pause 0xe00000001273e9e8 nginx 1032 1031 1032 0 S+ ttyin 0xe00000001253c0a8 sh 1031 1027 1031 1001 S+ wait 0xe0000000124064a0 su 1027 1026 1027 1001 Ss+ pause 0xe000000012406e88 tcsh 1026 1024 1024 1001 S select 0xe0000000124392c0 sshd 1024 926 1024 0 Ss select 0xe000000010c29140 sshd 999 1 999 0 Ss+ ttyin 0xe0000000114288a8 getty 998 1 1 0 S ttydcd 0xe000000011428ce8 getty 994 992 24 0 S+ nanslp 0x9ffc000000f0c178 sleep 993 1 24 0 S+ piperd 0xe000000012848268 logger 992 1 24 0 S+ wait 0xe00000001273f280 sh 976 1 976 0 Ss select 0xe000000010c28fc0 inetd 937 1 937 0 Ss nanslp 0x9ffc000000f0c179 cron 932 1 932 25 Ss pause 0xe00000001273e548 sendmail 929 1 929 0 Ss select 0xe000000012439640 sendmail 926 1 926 0 Ss select 0xe0000000124397c0 sshd 866 1 866 0 Ss select 0xe0000000124398c0 ntpd 786 1 786 0 Ss select 0xe000000010c293c0 syslogd 777 1 777 0 Ss select 0xe000000012439940 ipmon 655 1 655 0 Ss select 0xe000000012439b40 devd 140 1 140 0 Ss pause 0xe0000000125469e8 adjkerntz 23 0 0 0 DL sdflush 0x9ffc000000ee2d30 [softdepflush] 22 0 0 0 DL vlruwt 0xe000000012171720 [vnlru] 21 0 0 0 DL syncer 0x9ffc000001101a90 [syncer] 20 0 0 0 DL psleep 0x9ffc000000ee20c8 [bufdaemon] 19 0 0 0 DL pgzero 0x9ffc000000ee30d4 [pagezero] 18 0 0 0 DL psleep 0x9ffc000000ee303c [vmdaemon] 17 0 0 0 DL psleep 0x9ffc000000ee306c [pagedaemon] 9 0 0 0 DL ccb_scan 0x9ffc000000ee33a0 [xpt_thrd] 6 0 0 0 DL waiting_ 0x9ffc000001104130 [sctp_iterator] 5 0 0 0 DL (threaded) [zfskern] 100052 D l2arc_fe 0x9ffc000001557ec0 [l2arc_feed_thread] 100051 D arc_recl 0x9ffc0000015480a0 [arc_reclaim_thread] 4 0 0 0 DL ispf 0xe000000011768000 [isp0: fc_thrd0] 3 0 0 0 DL idle 0xa000000000a16000 [mpt_recovery1] 2 0 0 0 DL idle 0xa0000000009fe000 [mpt_recovery0] 16 0 0 0 DL (threaded) [usb] 100039 D - 0xa0000000009fae18 [usbus2] 100038 D - 0xa0000000009fadc0 [usbus2] 100037 D - 0xa0000000009fad68 [usbus2] 100036 D - 0xa0000000009fad10 [usbus2] 100034 D - 0xa0000000009f1460 [usbus1] 100033 D - 0xa0000000009f1408 [usbus1] 100032 D - 0xa0000000009f13b0 [usbus1] 100031 D - 0xa0000000009f1358 [usbus1] 100029 D - 0xa0000000009ed460 [usbus0] 100028 D - 0xa0000000009ed408 [usbus0] 100027 D - 0xa0000000009ed3b0 [usbus0] 100026 D - 0xa0000000009ed358 [usbus0] 15 0 0 0 DL tzpoll 0x9ffc000000ee0a50 [acpi_thermal] 14 0 0 0 DL - 0x9ffc000000ee0d38 [rand_harvestq] 13 0 0 0 DL (threaded) [geom] 100012 D - 0x9ffc000000ee1270 [g_down] 100011 D - 0x9ffc000000ee1268 [g_up] 100010 D - 0x9ffc000000ee1258 [g_event] 12 0 0 0 WL (threaded) [intr] 100048 I [swi0: uart uart+++] 100046 I [irq71: isp0] 100042 I [irq28: mpt1] 100040 I [irq27: mpt0] 100035 I [irq18: ehci0] 100030 I [irq17: ohci1] 100025 I [irq16: ohci0] 100022 I [swi6: task queue] 100021 I [swi6: Giant taskq] 100019 I [swi5: fast taskq] 100018 I [swi2: cambio] 100008 I [swi4: clock] 100007 I [swi4: clock] 100006 I [swi3: vm] 100005 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100004 Run CPU 1 [idle: cpu1] 100003 CanRun [idle: cpu0] 1 0 1 0 SLs wait 0xe000000011180de0 [init] 10 0 0 0 DL audit_wo 0x9ffc0000011087c8 [audit] 0 0 0 0 DLs (threaded) [kernel] 100053 D - 0x9ffc000000f0af10 [deadlkres] 100050 D - 0xe000000011795300 [system_taskq_1] 100049 D - 0xe000000011795300 [system_taskq_0] 100045 D - 0xe0000000111a2500 [em1 taskq] 100044 D - 0xe0000000111a2600 [em0 taskq] 100023 D - 0xe0000000111a5300 [ffs_trim taskq] 100020 D - 0xe0000000111a5600 [thread taskq] 100017 D - 0xe0000000111a5900 [acpi_task_2] 100016 D - 0xe0000000111a5900 [acpi_task_1] 100015 D - 0xe0000000111a5900 [acpi_task_0] 100014 D - 0xe0000000111a5a00 [kqueue taskq] 100009 D - 0xe0000000111a5d00 [firmware taskq] 100000 D swapin 0x9ffc000000efeac8 [swapper] db> db> alltrace Tracing command nginx pid 1035 tid 100097 td 0xe000000012800000 kdb_enter(0x9ffc000000dcfac8, 0x9ffc000000dcfac8, 0x9ffc000000592a90, 0x40c) at kdb_enter+0x92 vpanic(0x9ffc000000de0c20, 0xa00000009df85118, 0x100, 0x9ffc000000ee18f0) at vpanic+0x2b0 kassert_panic(0x9ffc000000de0c20, 0xe00000027fcc1900, 0x81f, 0xe0000000127a7800, 0x9ffc0000006a4f10, 0x2c60, 0xe0000000127a7800, 0xe0000000127a7818) at kassert_panic+0x120 vn_sendfile(0x8df, 0x3, 0x0, 0x0, 0x0, 0x8df, 0x7fffffffffffe000, 0x0) at vn_sendfile+0x15d0 sys_sendfile(0xe000000012800000, 0xa00000009df854e8, 0x9, 0xa00000009df85360) at sys_sendfile+0x2b0 syscall(0xe0000000128c1280, 0x3, 0x0, 0xe000000012800000, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command nginx pid 1034 tid 100078 td 0xe0000000124bc000 cpu_switch(0xe0000000124bc000, 0xe000000011182900, 0x9ffc000000f5c148, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000124bc000, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000124bc000, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe00000001273e9e8, 0x74, 0xe0000000124bc378, 0xe0000000124bc000) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe00000001273e9e8, 0x74, 0x9ffc000000dd7480, 0x9ffc000000dd7840) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe00000001273e9e8, 0x74, 0x9ffc000000dcff18, 0x100, 0x9ffc0000005a4ea0) at sleepq_wait_sig+0x30 _sleep(0xe00000001273e9e8, 0xe00000001273ea40, 0x74, 0x9ffc000000dcff18, 0x0, 0x0, 0x100) at _sleep+0x7b0 kern_sigsuspend(0xe0000000124bc000, 0x0, 0xe00000001273e9e8) at kern_sigsuspend+0x140 sys_sigsuspend(0xe0000000124bc000, 0xa00000009df054e8, 0x9ffc000000ac2c80, 0x48d) at sys_sigsuspend+0x60 syscall(0xe00000001273e940, 0x7fffffffffffead0, 0x0, 0xe0000000124bc000, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sh pid 1032 tid 100072 td 0xe000000012500d80 cpu_switch(0xe000000012500d80, 0xe000000011182900, 0x9ffc000000f5d160, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000012500d80, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012500d80, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe00000001253c0a8, 0x0, 0xe0000000125010f8, 0xe000000012500d80) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe00000001253c0a8, 0x0, 0x9ffc000000dd7480, 0x9ffc00000056eef0) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe00000001253c0a8, 0x0, 0x9ffc0000004fb550, 0x611, 0x9ffc0000004fb590) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe00000001253c0a8, 0xe00000001253c008, 0xe000000012500d80, 0x0) at _cv_wait_sig+0x4d0 tty_wait(0xe00000001253c000, 0xe00000001253c0a8, 0x0, 0x9ffc000000dddc28, 0x9ffc0000006756c0) at tty_wait+0xb0 ttydisc_read(0xe00000001253c000, 0xa00000009ded5360, 0x0, 0x0) at ttydisc_read+0x4c0 ttydev_read(0xe00000001253c000, 0xa00000009ded5360, 0x0, 0x9ffc000000378070) at ttydev_read+0xe0 devfs_read_f(0xe0000000124aa000, 0xa00000009ded5360, 0xe000000012431400, 0x0, 0xe000000012500d80) at devfs_read_f+0xf0 dofileread(0xe000000012500d80, 0x0, 0xe0000000124aa000, 0xa00000009ded5360, 0x0, 0x0) at dofileread+0x150 kern_readv(0xe000000012500d80, 0x0, 0xa00000009ded5360) at kern_readv+0xa0 sys_read(0xe000000012500d80, 0xa00000009ded54e8, 0x9ffc000000ac2c80, 0x48d) at sys_read+0x100 syscall(0xe000000012547280, 0x7fffffffffffec60, 0x1, 0xe000000012500d80, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command su pid 1031 tid 100071 td 0xe000000012501200 cpu_switch(0xe000000012501200, 0xe000000011182900, 0x9ffc000000f5dfc0, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000012501200, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012501200, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe0000000124064a0, 0x6c, 0xe000000012501578, 0xe000000012501200) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000000124064a0, 0x6c, 0x9ffc000000dd7480, 0x9ffc000000dd7840) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe0000000124064a0, 0x6c, 0x9ffc000000e06798, 0x100, 0x9ffc0000005a4ea0) at sleepq_wait_sig+0x30 _sleep(0xe0000000124064a0, 0xe0000000124065a0, 0x6c, 0x9ffc000000e06798, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0 kern_wait6(0xe000000012501200, 0x0, 0x408, 0xa00000009decd390, 0x32, 0x0, 0x0) at kern_wait6+0xba0 kern_wait(0xe000000012501200, 0x408, 0xa00000009decd390, 0x2, 0x0) at kern_wait+0xe0 sys_wait4(0xe000000012501200, 0xa00000009decd4e8, 0xa00000009decd390, 0x9ffc000000ac2c80) at sys_wait4+0x70 syscall(0xe0000000124064a0, 0x7fffffffffffe640, 0x2, 0xe000000012501200, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command tcsh pid 1027 tid 100069 td 0xe0000000123fed80 cpu_switch(0xe0000000123fed80, 0xe000000011182480, 0x9ffc000000f5e510, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000123fed80, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000123fed80, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe000000012406e88, 0x74, 0xe0000000123ff0f8, 0xe0000000123fed80) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000012406e88, 0x74, 0x9ffc000000dd7480, 0x9ffc000000dd7840) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000012406e88, 0x74, 0x9ffc000000dcff18, 0x100, 0x9ffc0000005a4ea0) at sleepq_wait_sig+0x30 _sleep(0xe000000012406e88, 0xe000000012406ee0, 0x74, 0x9ffc000000dcff18, 0x0, 0x0, 0x100) at _sleep+0x7b0 kern_sigsuspend(0xe0000000123fed80, 0x0, 0xe000000012406e88) at kern_sigsuspend+0x140 sys_sigsuspend(0xe0000000123fed80, 0xa00000009debd4e8, 0x9ffc000000ac2c80, 0x48d) at sys_sigsuspend+0x60 syscall(0xe000000012406de0, 0x0, 0x10004f5d0, 0xe0000000123fed80, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sshd pid 1026 tid 100081 td 0xe0000000127a0000 cpu_switch(0xe0000000127a0000, 0xe000000011182480, 0x9ffc000000f5cdf0, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000127a0000, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000127a0000, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe0000000124392c0, 0x0, 0xe0000000127a0378, 0xe0000000127a0000) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000000124392c0, 0x0, 0x9ffc000000dd7480, 0xe0000000124392a0) at sleepq_catch_signals+0x190 sleepq_timedwait_sig(0xe0000000124392c0, 0x0, 0x9ffc0000004fa750, 0x9ffc0000004fa790, 0x713) at sleepq_timedwait_sig+0x30 _cv_timedwait_sig_sbt(0xe0000000124392c0, 0xe0000000124392a0, 0x1e0fa3e4187, 0x12c0000000, 0x200) at _cv_timedwait_sig_sbt+0x4f0 seltdwait(0xe000000012439280, 0x1e0fa3e4187, 0x12c0000000, 0xe0000000124392b8, 0x9ffc000000ddac78, 0x9ffc00000064a8d0) at seltdwait+0xf0 kern_select(0xe0000000127a0000, 0x0, 0x141416230, 0x141416238, 0x0, 0x1e0fa3e4187, 0xa) at kern_select+0xb40 sys_select(0xe0000000127a0000, 0xa00000009df1d4e8, 0x9ffc000000ac2c80, 0x48d) at sys_select+0xb0 syscall(0xe000000012547720, 0x141416230, 0x141416238, 0xe0000000127a0000, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sshd pid 1024 tid 100073 td 0xe000000012500900 cpu_switch(0xe000000012500900, 0xe000000011182900, 0x9ffc000000f5e1c8, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000012500900, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012500900, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe000000010c29140, 0x0, 0xe000000012500c78, 0xe000000012500900) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000010c29140, 0x0, 0x9ffc000000dd7480, 0x9ffc00000056eef0) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000010c29140, 0x0, 0x9ffc0000004fb550, 0x611, 0x9ffc0000004fb590) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe000000010c29140, 0xe000000010c29120, 0xe000000012500900, 0x0, 0x9ffc000000f014b0) at _cv_wait_sig+0x4d0 seltdwait(0xe000000010c29100, 0xffffffffffffffff, 0x0, 0xe000000010c29138) at seltdwait+0x110 sys_poll(0xe000000012500900, 0xa00000009dedd4e8, 0xe000000012352170, 0xa00000009dedd290) at sys_poll+0x620 syscall(0xe000000012546de0, 0x1, 0xffffffffffffffff, 0xe000000012500900, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command getty pid 999 tid 100099 td 0xe000000012890480 cpu_switch(0xe000000012890480, 0xe000000011182900, 0x9ffc000000f5c620, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000012890480, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012890480, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe0000000114288a8, 0x0, 0xe0000000128907f8, 0xe000000012890480) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000000114288a8, 0x0, 0x9ffc000000dd7480, 0x9ffc00000056eef0) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe0000000114288a8, 0x0, 0x9ffc0000004fb550, 0x611, 0x9ffc0000004fb590) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe0000000114288a8, 0xe000000011428808, 0xe000000012890480, 0x0) at _cv_wait_sig+0x4d0 tty_wait(0xe000000011428800, 0xe0000000114288a8, 0x2, 0x9ffc000000dddc28, 0x9ffc0000006756c0) at tty_wait+0xb0 ttydisc_read(0xe000000011428800, 0xa00000009df95360, 0x0, 0x0) at ttydisc_read+0x4c0 ttydev_read(0xe000000011428800, 0xa00000009df95360, 0x0, 0x9ffc000000378070) at ttydev_read+0xe0 devfs_read_f(0xe000000012486690, 0xa00000009df95360, 0xe00000001116e700, 0x0, 0xe000000012890480) at devfs_read_f+0xf0 dofileread(0xe000000012890480, 0x0, 0xe000000012486690, 0xa00000009df95360, 0x0, 0x0) at dofileread+0x150 kern_readv(0xe000000012890480, 0x0, 0xa00000009df95360) at kern_readv+0xa0 sys_read(0xe000000012890480, 0xa00000009df954e8, 0x9ffc000000ac2c80, 0x48d) at sys_read+0x100 syscall(0xe0000000128c0940, 0x7fffffffffffed20, 0x1, 0xe000000012890480, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command getty pid 998 tid 100098 td 0xe0000000127a1b00 cpu_switch(0xe0000000127a1b00, 0xe000000011182900, 0x9ffc000000f5d0c0, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000127a1b00, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000127a1b00, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe000000011428ce8, 0x0, 0xe0000000127a1e78, 0xe0000000127a1b00) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000011428ce8, 0x0, 0x9ffc000000dd7480, 0x9ffc00000056eef0) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000011428ce8, 0x0, 0x9ffc0000004fb550, 0x611, 0x9ffc0000004fb590) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe000000011428ce8, 0xe000000011428c08, 0xe0000000127a1b00, 0x0) at _cv_wait_sig+0x4d0 tty_wait(0xe000000011428c00, 0xe000000011428ce8, 0x1, 0x9ffc000000dddc28, 0x9ffc00000066a8f0, 0x694, 0xe0000000128df500) at tty_wait+0xb0 ttydev_open(0xe000000011785600, 0x9ffc000000dddc60, 0x2000, 0x9ffc000000dddd48, 0xe000000011428c00) at ttydev_open+0x710 devfs_open(0xa00000009df8d158, 0x0, 0xa00000009df8d118) at devfs_open+0x2c0 VOP_OPEN_APV(0x9ffc000000e684f0, 0xa00000009df8d158, 0x0, 0x9ffc000000e18760) at VOP_OPEN_APV+0x270 vn_open_vnode(0xe000000012872c30, 0x3, 0xe00000001116e100, 0xe0000000127a1b00, 0xe000000012487900) at vn_open_vnode+0x2c0 vn_open_cred(0xa00000009df8d288, 0xa00000009df8d390, 0x980, 0x0, 0xe00000001116e100, 0xe000000012487900) at vn_open_cred+0x520 vn_open(0xa00000009df8d288, 0xa00000009df8d390, 0x980, 0xe000000012487900, 0x9ffc000000703760, 0x695, 0x0) at vn_open+0x50 kern_openat(0xe0000000127a1b00, 0xffffffffffffff9c, 0x10001ca90, 0x0, 0x0, 0x7980) at kern_openat+0x490 kern_open(0xe0000000127a1b00, 0x10001ca90, 0x0, 0x2, 0x7980) at kern_open+0x40 sys_open(0xe0000000127a1b00, 0xa00000009df8d4e8, 0x9ffc000000ac2c80, 0x48d) at sys_open+0x40 syscall(0xe0000000128c0de0, 0x2, 0x100007980, 0xe0000000127a1b00, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sleep pid 994 tid 100088 td 0xe0000000124bcd80 cpu_switch(0xe0000000124bcd80, 0xe0000000127a1680, 0x9ffc000000f5de08, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000124bcd80, 0xe0000000127a1680, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000124bcd80, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000000f0c178, 0x6c, 0xe0000000124bd0f8, 0xe0000000124bcd80) at sleepq_switch+0x2e0 sleepq_catch_signals(0x9ffc000000f0c178, 0x6c, 0x9ffc000000dd7480, 0xe0000000124bd110) at sleepq_catch_signals+0x190 sleepq_timedwait_sig(0x9ffc000000f0c178, 0x6c, 0x3c0000000, 0x9ffc0000005a4e40, 0x998) at sleepq_timedwait_sig+0x30 _sleep(0x9ffc000000f0c178, 0x0, 0x6c, 0x9ffc000000dd1750, 0x1, 0x3c0000000, 0x200) at _sleep+0x750 kern_nanosleep(0xe0000000124bcd80, 0x3c00000000, 0xa00000009df3d380) at kern_nanosleep+0x2a0 sys_nanosleep(0xe0000000124bcd80, 0xa00000009df3d4e8, 0x0, 0x9ffc000000ac2c80) at sys_nanosleep+0xb0 syscall(0xe00000001273f720, 0x7fffffffffffed60, 0x7fffffffffffed30, 0xe0000000124bcd80, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command logger pid 993 tid 100096 td 0xe000000012800480 cpu_switch(0xe000000012800480, 0xe000000011182480, 0x9ffc000000f5e5b0, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000012800480, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012800480, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe000000012848268, 0x5c, 0xe0000000128007f8, 0xe000000012800480) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000012848268, 0x5c, 0x9ffc000000dd7480, 0x9ffc000000dd7840) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000012848268, 0x5c, 0x9ffc000000ddaf48, 0x100, 0x9ffc0000005a4ea0) at sleepq_wait_sig+0x30 _sleep(0xe000000012848268, 0xe0000000128484a8, 0x5c, 0x9ffc000000ddaf48, 0x0, 0x0, 0x100) at _sleep+0x7b0 pipe_read(0xe000000012487e74, 0xa00000009df7d360, 0xa00000009df7d378, 0x0, 0xe000000012848370) at pipe_read+0x740 dofileread(0xe000000012800480, 0x0, 0xe000000012487e50, 0xa00000009df7d360, 0x0, 0x0) at dofileread+0x150 kern_readv(0xe000000012800480, 0x0, 0xa00000009df7d360) at kern_readv+0xa0 sys_read(0xe000000012800480, 0xa00000009df7d4e8, 0x9ffc000000ac2c80, 0x48d) at sys_read+0x100 syscall(0xe0000000128c1720, 0x140810000, 0x2000, 0xe000000012800480, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sh pid 992 tid 100089 td 0xe0000000127a1680 cpu_switch(0xe0000000127a1680, 0xe000000011182900, 0x9ffc000000f5d2f0, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000127a1680, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000127a1680, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe00000001273f280, 0x6c, 0xe0000000127a19f8, 0xe0000000127a1680) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe00000001273f280, 0x6c, 0x9ffc000000dd7480, 0x9ffc000000dd7840) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe00000001273f280, 0x6c, 0x9ffc000000e06798, 0x100, 0x9ffc0000005a4ea0) at sleepq_wait_sig+0x30 _sleep(0xe00000001273f280, 0xe00000001273f380, 0x6c, 0x9ffc000000e06798, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0 kern_wait6(0xe0000000127a1680, 0x7, 0x0, 0xa00000009df45390, 0x30, 0x0, 0x0) at kern_wait6+0xba0 kern_wait(0xe0000000127a1680, 0xffffffff, 0xa00000009df45390, 0x0, 0x0) at kern_wait+0xe0 sys_wait4(0xe0000000127a1680, 0xa00000009df454e8, 0xa00000009df45390, 0x9ffc000000ac2c80) at sys_wait4+0x70 syscall(0xe00000001273f280, 0x7fffffffffffd100, 0x0, 0xe0000000127a1680, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command inetd pid 976 tid 100075 td 0xe000000012500000 cpu_switch(0xe000000012500000, 0xe000000011182900, 0x9ffc000000f5cd78, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000012500000, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012500000, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe000000010c28fc0, 0x0, 0xe000000012500378, 0xe000000012500000) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000010c28fc0, 0x0, 0x9ffc000000dd7480, 0x9ffc00000056eef0) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000010c28fc0, 0x0, 0x9ffc0000004fb550, 0x611, 0x9ffc0000004fb590) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe000000010c28fc0, 0xe000000010c28fa0, 0xe000000012500000, 0x0, 0x9ffc000000f014b0) at _cv_wait_sig+0x4d0 seltdwait(0xe000000010c28f80, 0xffffffffffffffff, 0x0, 0xe000000010c28fb8, 0x9ffc000000ddac78, 0x9ffc00000064a8d0) at seltdwait+0x110 kern_select(0xe000000012500000, 0x0, 0x7fffffffffffe0e0, 0x0, 0x0, 0xffffffffffffffff, 0x6) at kern_select+0xb40 sys_select(0xe000000012500000, 0xa00000009deed4e8, 0x9ffc000000ac2c80, 0x48d) at sys_select+0xb0 syscall(0xe0000000125464a0, 0x7fffffffffffe0e0, 0x0, 0xe000000012500000, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command cron pid 937 tid 100076 td 0xe0000000124bc900 cpu_switch(0xe0000000124bc900, 0xe000000011182480, 0x9ffc000000f5dde0, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000124bc900, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000124bc900, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000000f0c179, 0x6c, 0xe0000000124bcc78, 0xe0000000124bc900) at sleepq_switch+0x2e0 sleepq_catch_signals(0x9ffc000000f0c179, 0x6c, 0x9ffc000000dd7480, 0xe0000000124bcc90) at sleepq_catch_signals+0x190 sleepq_timedwait_sig(0x9ffc000000f0c179, 0x6c, 0x206df9538, 0x9ffc0000005a4e40, 0x998) at sleepq_timedwait_sig+0x30 _sleep(0x9ffc000000f0c179, 0x0, 0x6c, 0x9ffc000000dd1750, 0x1, 0x206df9538, 0x200) at _sleep+0x750 kern_nanosleep(0xe0000000124bc900, 0x206df9538f, 0xa00000009def5380) at kern_nanosleep+0x2a0 sys_nanosleep(0xe0000000124bc900, 0xa00000009def54e8, 0x0, 0x9ffc000000ac2c80) at sys_nanosleep+0xb0 syscall(0xe000000012546000, 0x7fffffffffffec28, 0x4, 0xe0000000124bc900, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sendmail pid 932 tid 100079 td 0xe000000012501b00 cpu_switch(0xe000000012501b00, 0xe0000000127a1200, 0x9ffc000000f5dc28, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000012501b00, 0xe0000000127a1200, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012501b00, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe00000001273e548, 0x74, 0xe000000012501e78, 0xe000000012501b00) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe00000001273e548, 0x74, 0x9ffc000000dd7480, 0x9ffc000000dd7840) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe00000001273e548, 0x74, 0x9ffc000000dcff18, 0x100, 0x9ffc0000005a4ea0) at sleepq_wait_sig+0x30 _sleep(0xe00000001273e548, 0xe00000001273e5a0, 0x74, 0x9ffc000000dcff18, 0x0, 0x0, 0x100) at _sleep+0x7b0 kern_sigsuspend(0xe000000012501b00, 0x0, 0xe00000001273e548) at kern_sigsuspend+0x140 sys_sigsuspend(0xe000000012501b00, 0xa00000009df0d4e8, 0x9ffc000000ac2c80, 0x48d) at sys_sigsuspend+0x60 syscall(0xe00000001273e4a0, 0xe000000012501b08, 0x9ffc000000abebb0, 0xe000000012501b00, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sendmail pid 929 tid 100080 td 0xe000000012501680 cpu_switch(0xe000000012501680, 0xe000000011182480, 0x9ffc000000f5e290, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000012501680, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012501680, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe000000012439640, 0x0, 0xe0000000125019f8, 0xe000000012501680) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000012439640, 0x0, 0x9ffc000000dd7480, 0xe000000012439620) at sleepq_catch_signals+0x190 sleepq_timedwait_sig(0xe000000012439640, 0x0, 0x9ffc0000004fa750, 0x9ffc0000004fa790, 0x713) at sleepq_timedwait_sig+0x30 _cv_timedwait_sig_sbt(0xe000000012439640, 0xe000000012439620, 0xbd71ec5aba, 0x50000000, 0x200) at _cv_timedwait_sig_sbt+0x4f0 seltdwait(0xe000000012439600, 0xbd71ec5aba, 0x50000000, 0xe000000012439638, 0x9ffc000000ddac78, 0x9ffc00000064a8d0) at seltdwait+0xf0 kern_select(0xe000000012501680, 0x0, 0x7fffffffffffc3c8, 0x0, 0x0, 0xbd71ec5aba, 0x4) at kern_select+0xb40 sys_select(0xe000000012501680, 0xa00000009df154e8, 0x9ffc000000ac2c80, 0x48d) at sys_select+0xb0 syscall(0xe00000001273e000, 0x7fffffffffffc3c8, 0x0, 0xe000000012501680, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sshd pid 926 tid 100067 td 0xe0000000123ff680 cpu_switch(0xe0000000123ff680, 0xe000000011182900, 0x9ffc000000f5ceb8, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000123ff680, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000123ff680, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe0000000124397c0, 0x0, 0xe0000000123ff9f8, 0xe0000000123ff680) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000000124397c0, 0x0, 0x9ffc000000dd7480, 0x9ffc00000056eef0) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe0000000124397c0, 0x0, 0x9ffc0000004fb550, 0x611, 0x9ffc0000004fb590) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe0000000124397c0, 0xe0000000124397a0, 0xe0000000123ff680, 0x0, 0x9ffc000000f014b0) at _cv_wait_sig+0x4d0 seltdwait(0xe000000012439780, 0xffffffffffffffff, 0x0, 0xe0000000124397b8, 0x9ffc000000ddac78, 0x9ffc00000064a8d0) at seltdwait+0x110 kern_select(0xe0000000123ff680, 0x0, 0x1414161a0, 0x0, 0x0, 0xffffffffffffffff, 0x5) at kern_select+0xb40 sys_select(0xe0000000123ff680, 0xa00000009dead4e8, 0x9ffc000000ac2c80, 0x48d) at sys_select+0xb0 syscall(0xe000000012407720, 0x1414161a0, 0x0, 0xe0000000123ff680, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command ntpd pid 866 tid 100077 td 0xe0000000124bc480 cpu_switch(0xe0000000124bc480, 0xe000000011182480, 0x9ffc000000f5cee0, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000124bc480, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000124bc480, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe0000000124398c0, 0x0, 0xe0000000124bc7f8, 0xe0000000124bc480) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000000124398c0, 0x0, 0x9ffc000000dd7480, 0x9ffc00000056eef0) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe0000000124398c0, 0x0, 0x9ffc0000004fb550, 0x611, 0x9ffc0000004fb590) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe0000000124398c0, 0xe0000000124398a0, 0xe0000000124bc480, 0x0, 0x9ffc000000f014b0) at _cv_wait_sig+0x4d0 seltdwait(0xe000000012439880, 0xffffffffffffffff, 0x0, 0xe0000000124398b8, 0x9ffc000000ddac78, 0x9ffc00000064a8d0) at seltdwait+0x110 kern_select(0xe0000000124bc480, 0x0, 0x7fffffffffffec88, 0x0, 0x0, 0xffffffffffffffff, 0x1e) at kern_select+0xb40 sys_select(0xe0000000124bc480, 0xa00000009defd4e8, 0x9ffc000000ac2c80, 0x48d) at sys_select+0xb0 syscall(0xe00000001273ede0, 0x7fffffffffffec88, 0x0, 0xe0000000124bc480, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command syslogd pid 786 tid 100068 td 0xe0000000123ff200 cpu_switch(0xe0000000123ff200, 0xe000000011182900, 0x9ffc000000f5ce18, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000123ff200, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000123ff200, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe000000010c293c0, 0x0, 0xe0000000123ff578, 0xe0000000123ff200) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000010c293c0, 0x0, 0x9ffc000000dd7480, 0x9ffc00000056eef0) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000010c293c0, 0x0, 0x9ffc0000004fb550, 0x611, 0x9ffc0000004fb590) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe000000010c293c0, 0xe000000010c293a0, 0xe0000000123ff200, 0x0, 0x9ffc000000f014b0) at _cv_wait_sig+0x4d0 seltdwait(0xe000000010c29380, 0xffffffffffffffff, 0x0, 0xe000000010c293b8, 0x9ffc000000ddac78, 0x9ffc00000064a8d0) at seltdwait+0x110 kern_select(0xe0000000123ff200, 0x0, 0x140866180, 0x0, 0x0, 0xffffffffffffffff, 0x9) at kern_select+0xb40 sys_select(0xe0000000123ff200, 0xa00000009deb54e8, 0x9ffc000000ac2c80, 0x48d) at sys_select+0xb0 syscall(0xe000000012407280, 0x140866180, 0x0, 0xe0000000123ff200, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command ipmon pid 777 tid 100056 td 0xe000000012168d80 cpu_switch(0xe000000012168d80, 0xe000000011182480, 0x9ffc000000f5e308, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000012168d80, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012168d80, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe000000012439940, 0x0, 0xe0000000121690f8, 0xe000000012168d80) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000012439940, 0x0, 0x9ffc000000dd7480, 0x9ffc00000056eef0) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000012439940, 0x0, 0x9ffc0000004fb550, 0x611, 0x9ffc0000004fb590) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe000000012439940, 0xe000000012439920, 0xe000000012168d80, 0x0, 0x9ffc000000f014b0) at _cv_wait_sig+0x4d0 seltdwait(0xe000000012439900, 0xffffffffffffffff, 0x0, 0xe000000012439938, 0x9ffc000000ddac78, 0x9ffc00000064a8d0) at seltdwait+0x110 kern_select(0xe000000012168d80, 0x0, 0x7fffffffffff6b50, 0x0, 0x0, 0xffffffffffffffff, 0x4) at kern_select+0xb40 sys_select(0xe000000012168d80, 0xa00000009de554e8, 0x9ffc000000ac2c80, 0x48d) at sys_select+0xb0 syscall(0xe0000000111d7280, 0x7fffffffffff6b50, 0x0, 0xe000000012168d80, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command devd pid 655 tid 100055 td 0xe00000001178c480 cpu_switch(0xe00000001178c480, 0xe000000011182480, 0x9ffc000000f5e358, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe00000001178c480, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001178c480, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe000000012439b40, 0x0, 0xe00000001178c7f8, 0xe00000001178c480) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000012439b40, 0x0, 0x9ffc000000dd7480, 0xe000000012439b20) at sleepq_catch_signals+0x190 sleepq_timedwait_sig(0xe000000012439b40, 0x0, 0x9ffc0000004fa750, 0x9ffc0000004fa790, 0x713) at sleepq_timedwait_sig+0x30 _cv_timedwait_sig_sbt(0xe000000012439b40, 0xe000000012439b20, 0xe321aaea70, 0x3c0000000, 0x200) at _cv_timedwait_sig_sbt+0x4f0 seltdwait(0xe000000012439b00, 0xe321aaea70, 0x3c0000000, 0xe000000012439b38, 0x9ffc000000ddac78, 0x9ffc00000064a8d0) at seltdwait+0xf0 kern_select(0xe00000001178c480, 0x0, 0x7fffffffffffcd00, 0x0, 0x0, 0xe321aaea70, 0x5) at kern_select+0xb40 sys_select(0xe00000001178c480, 0xa00000009de4d4e8, 0x9ffc000000ac2c80, 0x48d) at sys_select+0xb0 syscall(0xe0000000111d7720, 0x7fffffffffffcd00, 0x0, 0xe00000001178c480, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command adjkerntz pid 140 tid 100074 td 0xe000000012500480 cpu_switch(0xe000000012500480, 0xe000000011182900, 0x9ffc000000f5d548, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000012500480, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012500480, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe0000000125469e8, 0x74, 0xe0000000125007f8, 0xe000000012500480) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000000125469e8, 0x74, 0x9ffc000000dd7480, 0x9ffc000000dd7840) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe0000000125469e8, 0x74, 0x9ffc000000dcff18, 0x100, 0x9ffc0000005a4ea0) at sleepq_wait_sig+0x30 _sleep(0xe0000000125469e8, 0xe000000012546a40, 0x74, 0x9ffc000000dcff18, 0x0, 0x0, 0x100) at _sleep+0x7b0 kern_sigsuspend(0xe000000012500480, 0x0, 0xe0000000125469e8) at kern_sigsuspend+0x140 sys_sigsuspend(0xe000000012500480, 0xa00000009dee54e8, 0x9ffc000000ac2c80, 0x48d) at sys_sigsuspend+0x60 syscall(0xe000000012546940, 0x0, 0x0, 0xe000000012500480, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command softdepflush pid 23 tid 100064 td 0xe000000012169680 cpu_switch(0xe000000012169680, 0xe00000001178d680, 0x9ffc000000f5c5a8, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000012169680, 0xe00000001178d680, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012169680, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000000ee2d30, 0x54, 0xe0000000121699f8, 0xe000000012169680) at sleepq_switch+0x2e0 sleepq_timedwait(0x9ffc000000ee2d30, 0x54, 0x0, 0x9ffc000000dd7480, 0xe000000012169680) at sleepq_timedwait+0xb0 _sleep(0x9ffc000000ee2d30, 0x9ffc0000011091e0, 0x54, 0x9ffc000000e06260, 0x1, 0x0, 0x100) at _sleep+0x780 softdep_flush(0x9ffc0000011091f8, 0x0, 0xe00000001255daa0, 0x0) at softdep_flush+0x390 fork_exit(0x9ffc000000e24940, 0x0, 0xa00000009de95550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command vnlru pid 22 tid 100063 td 0xe000000012169b00 cpu_switch(0xe000000012169b00, 0xe000000012169680, 0x9ffc000000f5c9b8, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000012169b00, 0xe000000012169680, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012169b00, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe000000012171720, 0x60, 0xe000000012169e78, 0xe000000012169b00) at sleepq_switch+0x2e0 sleepq_timedwait(0xe000000012171720, 0x60, 0x0, 0x9ffc000000dd7480, 0xe000000012169b00) at sleepq_timedwait+0xb0 _sleep(0xe000000012171720, 0x9ffc000001101a50, 0x60, 0x9ffc000000de6420, 0x1, 0x0, 0x100) at _sleep+0x780 vnlru_proc(0x0, 0xa00000009de8d550, 0x9ffc000000dc8140, 0x9ffc000000de4db0) at vnlru_proc+0x2c0 fork_exit(0x9ffc000000e1ee40, 0x0, 0xa00000009de8d550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command syncer pid 21 tid 100062 td 0xe0000000123fe000 cpu_switch(0xe0000000123fe000, 0xe000000011182480, 0x9ffc000000f5d6b0, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000123fe000, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000123fe000, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000001101a90, 0x0, 0xe0000000123fe378, 0xe0000000123fe000) at sleepq_switch+0x2e0 sleepq_timedwait(0x9ffc000001101a90, 0x0, 0x9ffc0000004fae60, 0x9ffc000000dd7480, 0xe0000000123fe000) at sleepq_timedwait+0xb0 _cv_timedwait_sbt(0x9ffc000001101a90, 0x9ffc000001101a70, 0xfffffed8, 0x0, 0x100) at _cv_timedwait_sbt+0x4f0 sched_sync(0xba, 0xe000000012579008, 0xe000000012578ea0, 0xe0000000117952a0) at sched_sync+0xba0 fork_exit(0x9ffc000000e202a0, 0x0, 0xa00000009de85550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command bufdaemon pid 20 tid 100061 td 0xe0000000123fe480 cpu_switch(0xe0000000123fe480, 0xe0000000111a1680, 0x9ffc000000f5e560, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000123fe480, 0xe0000000111a1680, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000123fe480, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000000ee20c8, 0x54, 0xe0000000123fe7f8, 0xe0000000123fe480) at sleepq_switch+0x2e0 sleepq_timedwait(0x9ffc000000ee20c8, 0x54, 0x0, 0x9ffc000000dd7480, 0xe0000000123fe480) at sleepq_timedwait+0xb0 _sleep(0x9ffc000000ee20c8, 0x9ffc000001101600, 0x54, 0x9ffc000000de20d0, 0x1, 0x0, 0x100) at _sleep+0x780 buf_daemon(0x203b, 0x9ffc000000ee2094, 0x9ffc000000ee20c8, 0x9ffc000000de1700) at buf_daemon+0x2f0 fork_exit(0x9ffc000000e1ad90, 0x0, 0xa00000009de7d550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command pagezero pid 19 tid 100060 td 0xe00000001178db00 cpu_switch(0xe00000001178db00, 0xe000000011182480, 0x9ffc000000f5e4c0, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe00000001178db00, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001178db00, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000000ee30d4, 0x0, 0xe00000001178de78, 0xe00000001178db00) at sleepq_switch+0x2e0 sleepq_timedwait(0x9ffc000000ee30d4, 0x0, 0x0, 0x9ffc000000dd7480, 0xe00000001178db00) at sleepq_timedwait+0xb0 _sleep(0x9ffc000000ee30d4, 0x9ffc00000110a380, 0x0, 0x9ffc000000e0e998, 0x1, 0x0, 0x100) at _sleep+0x780 vm_pagezero(0x9ffc000000ee30d4, 0x9ffc000000ee1ba8, 0x9ffc000000ee1bc0, 0x9ffc000000e0e998) at vm_pagezero+0x280 fork_exit(0x9ffc000000e21ce0, 0x0, 0xa00000009de75550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command vmdaemon pid 18 tid 100059 td 0xe000000012168000 cpu_switch(0xe000000012168000, 0xe0000000123fe480, 0x9ffc000000f5c300, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000012168000, 0xe0000000123fe480, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012168000, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000000ee303c, 0x74, 0xe000000012168378, 0xe000000012168000) at sleepq_switch+0x2e0 sleepq_wait(0x9ffc000000ee303c, 0x74, 0x9ffc000000dd7480, 0xe000000012168000, 0x9ffc0000005a4ec0) at sleepq_wait+0xb0 _sleep(0x9ffc000000ee303c, 0x9ffc00000110b400, 0x74, 0x9ffc000000de20d0, 0x0, 0x0, 0x100) at _sleep+0x7d0 vm_daemon(0x0, 0xa00000009de6d550, 0x9ffc000000e0d7d0, 0x3d5) at vm_daemon+0xd0 fork_exit(0x9ffc000000e22f10, 0x0, 0xa00000009de6d550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command pagedaemon pid 17 tid 100058 td 0xe000000012168480 cpu_switch(0xe000000012168480, 0xe0000000123fe480, 0x9ffc000000f5cf80, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000012168480, 0xe0000000123fe480, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012168480, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000000ee306c, 0x54, 0xe0000000121687f8, 0xe000000012168480) at sleepq_switch+0x2e0 sleepq_timedwait(0x9ffc000000ee306c, 0x54, 0x0, 0x9ffc000000dd7480, 0xe000000012168480) at sleepq_timedwait+0xb0 _sleep(0x9ffc000000ee306c, 0x9ffc00000110a380, 0x54, 0x9ffc000000de20d0, 0x1, 0x0, 0x100) at _sleep+0x780 vm_pageout(0x9ffc000000edfd98, 0xe00000027d37f5b0, 0x0, 0x0) at vm_pageout+0x8a0 fork_exit(0x9ffc000000e274d0, 0x0, 0xa00000009de65550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command xpt_thrd pid 9 tid 100057 td 0xe000000012168900 cpu_switch(0xe000000012168900, 0xe000000011182480, 0x9ffc000000f5d818, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000012168900, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012168900, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000000ee33a0, 0x5c, 0xe000000012168c78, 0xe000000012168900) at sleepq_switch+0x2e0 sleepq_wait(0x9ffc000000ee33a0, 0x5c, 0x9ffc000000dd7480, 0xe000000012168900, 0x9ffc0000005a4ec0) at sleepq_wait+0xb0 _sleep(0x9ffc000000ee33a0, 0x9ffc000000ee3420, 0x5c, 0x9ffc000000bfb200, 0x0, 0x0, 0x100) at _sleep+0x7d0 xpt_scanner_thread(0xe000000011795c20, 0xe00000001219f000, 0x9ffc000000ee33a0, 0x9ffc000000bfb040) at xpt_scanner_thread+0xb0 fork_exit(0x9ffc000000e1a910, 0x0, 0xa00000009de5d550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command sctp_iterator pid 6 tid 100054 td 0xe00000001178c900 cpu_switch(0xe00000001178c900, 0xe000000011182480, 0x9ffc000000f5d2c8, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe00000001178c900, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001178c900, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000001104130, 0x0, 0xe00000001178cc78, 0xe00000001178c900) at sleepq_switch+0x2e0 sleepq_wait(0x9ffc000001104130, 0x0, 0x9ffc000000dd7480, 0xe00000001178c900, 0x9ffc0000005a4ec0) at sleepq_wait+0xb0 _sleep(0x9ffc000001104130, 0x9ffc0000011040d0, 0x0, 0x9ffc000000df1610, 0x0, 0x0, 0x100) at _sleep+0x7d0 sctp_iterator_thread(0x9ffc000000df1610, 0x9ffc0000011040d0, 0x9ffc000001104130, 0x9ffc0000005335f0) at sctp_iterator_thread+0x90 fork_exit(0x9ffc000000e1e730, 0x0, 0xa00000009de45550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command zfskern pid 5 tid 100052 td 0xe00000001178d200 cpu_switch(0xe00000001178d200, 0xe000000011182900, 0x9ffc000000f5ded0, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe00000001178d200, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001178d200, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000001557ec0, 0x0, 0xe00000001178d578, 0xe00000001178d200) at sleepq_switch+0x2e0 sleepq_timedwait(0x9ffc000001557ec0, 0x0, 0x713, 0x9ffc000000dd7480, 0xe00000001178d200) at sleepq_timedwait+0xb0 _cv_timedwait_sbt(0x9ffc000001557ec0, 0x9ffc000001557ea0, 0xfffffed8, 0x0, 0x100, 0xe00000001178d200, 0x1, 0x9ffc000000e84138) at _cv_timedwait_sbt+0x4f0 l2arc_feed_thread(0x9ffc000001546de0, 0x9ffc000001546dd8, 0x9ffc000001546dd0, 0x9ffc0000014e9468) at l2arc_feed_thread+0x800 fork_exit(0x9ffc000000e3e7b0, 0x0, 0xa00000009de33550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command zfskern pid 5 tid 100051 td 0xe00000001178d680 cpu_switch(0xe00000001178d680, 0xe0000000123fe000, 0x9ffc000000f5c620, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe00000001178d680, 0xe0000000123fe000, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001178d680, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc0000015480a0, 0x0, 0xe00000001178d9f8, 0xe00000001178d680) at sleepq_switch+0x2e0 sleepq_timedwait(0x9ffc0000015480a0, 0x0, 0x713, 0x9ffc000000dd7480, 0xe00000001178d680) at sleepq_timedwait+0xb0 _cv_timedwait_sbt(0x9ffc0000015480a0, 0x9ffc000001548080, 0xfffffed8, 0x0, 0x100) at _cv_timedwait_sbt+0x4f0 arc_reclaim_thread(0x0, 0xa00000009de2b550, 0x9ffc000000dc8140, 0x3d5) at arc_reclaim_thread+0x610 fork_exit(0x9ffc000000e3e870, 0x0, 0xa00000009de2b550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command isp0: fc_thrd0 pid 4 tid 100047 td 0xe00000001174db00 cpu_switch(0xe00000001174db00, 0xe000000011425b00, 0x9ffc000000f5d520, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe00000001174db00, 0xe000000011425b00, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001174db00, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe000000011768000, 0x5c, 0xe00000001174de78, 0xe00000001174db00) at sleepq_switch+0x2e0 sleepq_wait(0xe000000011768000, 0x5c, 0x9ffc000000dd7480, 0xe00000001174db00, 0x9ffc0000005a4ec0) at sleepq_wait+0xb0 _sleep(0xe000000011768000, 0xe000000011786400, 0x5c, 0x9ffc000000c27460, 0x0, 0x0, 0x100) at _sleep+0x7d0 isp_kthread(0xe000000011768000, 0xe000000011786400, 0x0, 0xe000000011768040) at isp_kthread+0x580 fork_exit(0x9ffc000000e1f480, 0xe000000011768000, 0xa00000009de0b550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command mpt_recovery1 pid 3 tid 100043 td 0xe00000001174c000 cpu_switch(0xe00000001174c000, 0xe00000001174db00, 0x9ffc000000f5d020, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe00000001174c000, 0xe00000001174db00, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001174c000, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xa000000000a16000, 0x78, 0xe00000001174c378, 0xe00000001174c000) at sleepq_switch+0x2e0 sleepq_wait(0xa000000000a16000, 0x78, 0x9ffc000000dd7480, 0xe00000001174c000, 0x9ffc0000005a4ec0) at sleepq_wait+0xb0 _sleep(0xa000000000a16000, 0xa000000000a16008, 0x78, 0x9ffc000000da3878, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7d0 mpt_recovery_thread(0xa000000000a16000, 0xa00000009ddcd550, 0x9ffc000000dc8140, 0xa000000000a16440) at mpt_recovery_thread+0x170 fork_exit(0x9ffc000000e1d050, 0xa000000000a16000, 0xa00000009ddcd550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command mpt_recovery0 pid 2 tid 100041 td 0xe00000001174c900 cpu_switch(0xe00000001174c900, 0xe00000001174c000, 0x9ffc000000f5e420, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe00000001174c900, 0xe00000001174c000, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001174c900, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xa0000000009fe000, 0x78, 0xe00000001174cc78, 0xe00000001174c900) at sleepq_switch+0x2e0 sleepq_wait(0xa0000000009fe000, 0x78, 0x9ffc000000dd7480, 0xe00000001174c900, 0x9ffc0000005a4ec0) at sleepq_wait+0xb0 _sleep(0xa0000000009fe000, 0xa0000000009fe008, 0x78, 0x9ffc000000da3878, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7d0 mpt_recovery_thread(0xa0000000009fe000, 0xa00000009dd6f550, 0x9ffc000000dc8140, 0xa0000000009fe440) at mpt_recovery_thread+0x170 fork_exit(0x9ffc000000e1d050, 0xa0000000009fe000, 0xa00000009dd6f550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command usb pid 16 tid 100039 td 0xe000000011744480 cpu_switch(0xe000000011744480, 0xe00000001174c900, 0x9ffc000000f5dd90, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011744480, 0xe00000001174c900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011744480, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xa0000000009fae18, 0x0, 0xe0000000117447f8, 0xe000000011744480) at sleepq_switch+0x2e0 sleepq_wait(0xa0000000009fae18, 0x0, 0x9ffc000000dd7480, 0xe000000011744480, 0x9ffc0000004fc2c0) at sleepq_wait+0xb0 _cv_wait(0xa0000000009fae18, 0xa0000000009fb040, 0xe000000011744480, 0x0) at _cv_wait+0x4d0 usb_process(0xa0000000009fae08, 0x0, 0xa0000000009fae55, 0x9ffc000000ee0ea4) at usb_process+0x380 fork_exit(0x9ffc000000e20c30, 0xa0000000009fae08, 0xa00000009dd11550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command usb pid 16 tid 100038 td 0xe000000011744900 cpu_switch(0xe000000011744900, 0xe000000011182900, 0x9ffc000000f5d228, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011744900, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011744900, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xa0000000009fadc0, 0x0, 0xe000000011744c78, 0xe000000011744900) at sleepq_switch+0x2e0 sleepq_wait(0xa0000000009fadc0, 0x0, 0x9ffc000000dd7480, 0xe000000011744900, 0x9ffc0000004fc2c0) at sleepq_wait+0xb0 _cv_wait(0xa0000000009fadc0, 0xa0000000009fb040, 0xe000000011744900, 0x0) at _cv_wait+0x4d0 usb_process(0xa0000000009fadb0, 0x0, 0xa0000000009fadfd, 0x9ffc000000ee0ea4) at usb_process+0x380 fork_exit(0x9ffc000000e20c30, 0xa0000000009fadb0, 0xa00000009dd09550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command usb pid 16 tid 100037 td 0xe000000011744d80 cpu_switch(0xe000000011744d80, 0xe000000011744900, 0x9ffc000000f5dfe8, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011744d80, 0xe000000011744900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011744d80, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xa0000000009fad68, 0x0, 0xe0000000117450f8, 0xe000000011744d80) at sleepq_switch+0x2e0 sleepq_wait(0xa0000000009fad68, 0x0, 0x9ffc000000dd7480, 0xe000000011744d80, 0x9ffc0000004fc2c0) at sleepq_wait+0xb0 _cv_wait(0xa0000000009fad68, 0xa0000000009fb040, 0xe000000011744d80, 0x0) at _cv_wait+0x4d0 usb_process(0xa0000000009fad58, 0x0, 0xa0000000009fada5, 0x9ffc000000ee0ea4) at usb_process+0x380 fork_exit(0x9ffc000000e20c30, 0xa0000000009fad58, 0xa00000009dd01550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command usb pid 16 tid 100036 td 0xe000000011745200 cpu_switch(0xe000000011745200, 0xe000000011744d80, 0x9ffc000000f5dea8, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011745200, 0xe000000011744d80, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011745200, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xa0000000009fad10, 0x0, 0xe000000011745578, 0xe000000011745200) at sleepq_switch+0x2e0 sleepq_wait(0xa0000000009fad10, 0x0, 0x9ffc000000dd7480, 0xe000000011745200, 0x9ffc0000004fc2c0) at sleepq_wait+0xb0 _cv_wait(0xa0000000009fad10, 0xa0000000009fb040, 0xe000000011745200, 0x0) at _cv_wait+0x4d0 usb_process(0xa0000000009fad00, 0x0, 0xa0000000009fad4d, 0x9ffc000000ee0ea4) at usb_process+0x380 fork_exit(0x9ffc000000e20c30, 0xa0000000009fad00, 0xa00000009dcf9550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command usb pid 16 tid 100034 td 0xe000000011480d80 cpu_switch(0xe000000011480d80, 0xe000000011745200, 0x9ffc000000f5d340, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011480d80, 0xe000000011745200, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011480d80, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xa0000000009f1460, 0x0, 0xe0000000114810f8, 0xe000000011480d80) at sleepq_switch+0x2e0 sleepq_wait(0xa0000000009f1460, 0x0, 0x9ffc000000dd7480, 0xe000000011480d80, 0x9ffc0000004fc2c0) at sleepq_wait+0xb0 _cv_wait(0xa0000000009f1460, 0xa0000000009f1688, 0xe000000011480d80, 0x0) at _cv_wait+0x4d0 usb_process(0xa0000000009f1450, 0x0, 0xa0000000009f149d, 0x9ffc000000ee0ea4) at usb_process+0x380 fork_exit(0x9ffc000000e20c30, 0xa0000000009f1450, 0xa0000000e2fed550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command usb pid 16 tid 100033 td 0xe000000011481200 cpu_switch(0xe000000011481200, 0xe000000011182900, 0x9ffc000000f5c580, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011481200, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011481200, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xa0000000009f1408, 0x0, 0xe000000011481578, 0xe000000011481200) at sleepq_switch+0x2e0 sleepq_wait(0xa0000000009f1408, 0x0, 0x9ffc000000dd7480, 0xe000000011481200, 0x9ffc0000004fc2c0) at sleepq_wait+0xb0 _cv_wait(0xa0000000009f1408, 0xa0000000009f1688, 0xe000000011481200, 0x0) at _cv_wait+0x4d0 usb_process(0xa0000000009f13f8, 0x0, 0xa0000000009f1445, 0x9ffc000000ee0ea4) at usb_process+0x380 fork_exit(0x9ffc000000e20c30, 0xa0000000009f13f8, 0xa0000000e2fe5550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command usb pid 16 tid 100032 td 0xe000000011481680 cpu_switch(0xe000000011481680, 0xe000000011182480, 0x9ffc000000f5da98, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011481680, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011481680, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xa0000000009f13b0, 0x0, 0xe0000000114819f8, 0xe000000011481680) at sleepq_switch+0x2e0 sleepq_wait(0xa0000000009f13b0, 0x0, 0x9ffc000000dd7480, 0xe000000011481680, 0x9ffc0000004fc2c0) at sleepq_wait+0xb0 _cv_wait(0xa0000000009f13b0, 0xa0000000009f1688, 0xe000000011481680, 0x9ffc000000efef70) at _cv_wait+0x4d0 usb_process(0xa0000000009f13a0, 0x0, 0xa0000000009f13ed, 0x9ffc000000ee0ea4) at usb_process+0x380 fork_exit(0x9ffc000000e20c30, 0xa0000000009f13a0, 0xa0000000e2fdd550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command usb pid 16 tid 100031 td 0xe000000011481b00 cpu_switch(0xe000000011481b00, 0xe000000011182480, 0x9ffc000000f5ccd8, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011481b00, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011481b00, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xa0000000009f1358, 0x0, 0xe000000011481e78, 0xe000000011481b00) at sleepq_switch+0x2e0 sleepq_wait(0xa0000000009f1358, 0x0, 0x9ffc000000dd7480, 0xe000000011481b00, 0x9ffc0000004fc2c0) at sleepq_wait+0xb0 _cv_wait(0xa0000000009f1358, 0xa0000000009f1688, 0xe000000011481b00, 0x9ffc000000efef70) at _cv_wait+0x4d0 usb_process(0xa0000000009f1348, 0x0, 0xa0000000009f1395, 0x9ffc000000ee0ea4) at usb_process+0x380 fork_exit(0x9ffc000000e20c30, 0xa0000000009f1348, 0xa0000000e2fd5550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command usb pid 16 tid 100029 td 0xe000000011425680 cpu_switch(0xe000000011425680, 0xe000000011182480, 0x9ffc000000f5dd40, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011425680, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011425680, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xa0000000009ed460, 0x0, 0xe0000000114259f8, 0xe000000011425680) at sleepq_switch+0x2e0 sleepq_wait(0xa0000000009ed460, 0x0, 0x9ffc000000dd7480, 0xe000000011425680, 0x9ffc0000004fc2c0) at sleepq_wait+0xb0 _cv_wait(0xa0000000009ed460, 0xa0000000009ed688, 0xe000000011425680, 0x9ffc000000efef70) at _cv_wait+0x4d0 usb_process(0xa0000000009ed450, 0x0, 0xa0000000009ed49d, 0x9ffc000000ee0ea4) at usb_process+0x380 fork_exit(0x9ffc000000e20c30, 0xa0000000009ed450, 0xa0000000e2f3d550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command usb pid 16 tid 100028 td 0xe000000011425b00 cpu_switch(0xe000000011425b00, 0xe000000011182480, 0x9ffc000000f5e380, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011425b00, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011425b00, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xa0000000009ed408, 0x0, 0xe000000011425e78, 0xe000000011425b00) at sleepq_switch+0x2e0 sleepq_wait(0xa0000000009ed408, 0x0, 0x9ffc000000dd7480, 0xe000000011425b00, 0x9ffc0000004fc2c0) at sleepq_wait+0xb0 _cv_wait(0xa0000000009ed408, 0xa0000000009ed688, 0xe000000011425b00, 0x0) at _cv_wait+0x4d0 usb_process(0xa0000000009ed3f8, 0x0, 0xa0000000009ed445, 0x9ffc000000ee0ea4) at usb_process+0x380 fork_exit(0x9ffc000000e20c30, 0xa0000000009ed3f8, 0xa0000000e2f35550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command usb pid 16 tid 100027 td 0xe000000011480000 cpu_switch(0xe000000011480000, 0xe000000011425b00, 0x9ffc000000f5d098, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011480000, 0xe000000011425b00, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011480000, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xa0000000009ed3b0, 0x0, 0xe000000011480378, 0xe000000011480000) at sleepq_switch+0x2e0 sleepq_wait(0xa0000000009ed3b0, 0x0, 0x9ffc000000dd7480, 0xe000000011480000, 0x9ffc0000004fc2c0) at sleepq_wait+0xb0 _cv_wait(0xa0000000009ed3b0, 0xa0000000009ed688, 0xe000000011480000, 0x9ffc000000efef70) at _cv_wait+0x4d0 usb_process(0xa0000000009ed3a0, 0x0, 0xa0000000009ed3ed, 0x9ffc000000ee0ea4) at usb_process+0x380 fork_exit(0x9ffc000000e20c30, 0xa0000000009ed3a0, 0xa0000000e2f2d550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command usb pid 16 tid 100026 td 0xe000000011480480 cpu_switch(0xe000000011480480, 0xe000000011480000, 0x9ffc000000f5d6d8, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011480480, 0xe000000011480000, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011480480, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xa0000000009ed358, 0x0, 0xe0000000114807f8, 0xe000000011480480) at sleepq_switch+0x2e0 sleepq_wait(0xa0000000009ed358, 0x0, 0x9ffc000000dd7480, 0xe000000011480480, 0x9ffc0000004fc2c0) at sleepq_wait+0xb0 _cv_wait(0xa0000000009ed358, 0xa0000000009ed688, 0xe000000011480480, 0x9ffc000000efef70) at _cv_wait+0x4d0 usb_process(0xa0000000009ed348, 0x0, 0xa0000000009ed395, 0x9ffc000000ee0ea4) at usb_process+0x380 fork_exit(0x9ffc000000e20c30, 0xa0000000009ed348, 0xa0000000e2f25550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command acpi_thermal pid 15 tid 100024 td 0xe000000011424000 cpu_switch(0xe000000011424000, 0xe000000011182480, 0x9ffc000000f5cf30, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011424000, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011424000, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000000ee0a50, 0x64, 0xe000000011424378, 0xe000000011424000) at sleepq_switch+0x2e0 sleepq_timedwait(0x9ffc000000ee0a50, 0x64, 0x0, 0x9ffc000000dd7480, 0xe000000011424000) at sleepq_timedwait+0xb0 _sleep(0x9ffc000000ee0a50, 0x9ffc000000ee9f80, 0x64, 0x9ffc000000c1e9c8, 0x1, 0x0, 0x100) at _sleep+0x780 acpi_tz_thread(0x1, 0xe000000011177a00, 0xe0000000111bed68, 0x1) at acpi_tz_thread+0x4e0 fork_exit(0x9ffc000000e20700, 0x0, 0xa0000000e2e91550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command rand_harvestq pid 14 tid 100013 td 0xe0000000111a1680 cpu_switch(0xe0000000111a1680, 0xe000000011182480, 0x9ffc000000f5c968, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000111a1680, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000111a1680, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000000ee0d38, 0x0, 0xe0000000111a19f8, 0xe0000000111a1680) at sleepq_switch+0x2e0 sleepq_timedwait(0x9ffc000000ee0d38, 0x0, 0x713, 0x9ffc000000dd7480, 0xe0000000111a1680) at sleepq_timedwait+0xb0 msleep_spin_sbt(0x9ffc000000ee0d38, 0x9ffc000000eea5f0, 0x9ffc000000dab938, 0x19999999, 0x0, 0x4) at msleep_spin_sbt+0x420 random_kthread(0x9ffc000000e3bb70, 0x0, 0x9ffc000000eea518, 0x3) at random_kthread+0x370 fork_exit(0x9ffc000000e264c0, 0x9ffc000000e3bb70, 0xa0000000a07fd550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command geom pid 13 tid 100012 td 0xe0000000111a1b00 cpu_switch(0xe0000000111a1b00, 0xe000000011182480, 0x9ffc000000f5d070, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000111a1b00, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000111a1b00, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000000ee1270, 0x5c, 0xe0000000111a1e78, 0xe0000000111a1b00) at sleepq_switch+0x2e0 sleepq_wait(0x9ffc000000ee1270, 0x5c, 0x9ffc000000dd7480, 0xe0000000111a1b00, 0x9ffc0000005a4ec0) at sleepq_wait+0xb0 _sleep(0x9ffc000000ee1270, 0x9ffc000000efe8c8, 0x5c, 0x9ffc000000dab938, 0x0, 0x0, 0x100) at _sleep+0x7d0 g_io_schedule_down(0x0, 0x9ffc000000ee120c, 0x9fe3a7400, 0x9ffc000000efe8b8) at g_io_schedule_down+0x190 g_down_procbody(0x9ffc000000ee1228, 0x9ffc000000dc1218, 0x9ffc0000005335f0, 0x40c) at g_down_procbody+0xb0 fork_exit(0x9ffc000000e1cbe0, 0x0, 0xa0000000a07f5550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command geom pid 13 tid 100011 td 0xe0000000111b0000 cpu_switch(0xe0000000111b0000, 0xe000000011182480, 0x9ffc000000f5d430, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000111b0000, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000111b0000, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000000ee1268, 0x5c, 0xe0000000111b0378, 0xe0000000111b0000) at sleepq_switch+0x2e0 sleepq_wait(0x9ffc000000ee1268, 0x5c, 0x9ffc000000dd7480, 0xe0000000111b0000, 0x9ffc0000005a4ec0) at sleepq_wait+0xb0 _sleep(0x9ffc000000ee1268, 0x9ffc000000efe900, 0x5c, 0x9ffc000000dab938, 0x0, 0x0, 0x100) at _sleep+0x7d0 g_io_schedule_up(0x0, 0xe0000000111b0350, 0x9ffc000000efe8f0, 0xe0000000111b0350) at g_io_schedule_up+0x1c0 g_up_procbody(0x9ffc000000ee1220, 0x9ffc000000dc1218, 0x9ffc0000005335f0, 0x40c) at g_up_procbody+0xb0 fork_exit(0x9ffc000000e1a640, 0x0, 0xa0000000a07ed550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command geom pid 13 tid 100010 td 0xe000000011183680 cpu_switch(0xe000000011183680, 0xe000000011182900, 0x9ffc000000f5ccb0, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011183680, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011183680, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000000ee1258, 0x5c, 0xe0000000111839f8, 0xe000000011183680) at sleepq_switch+0x2e0 sleepq_wait(0x9ffc000000ee1258, 0x5c, 0x9ffc000000dd7480, 0xe000000011183680, 0x9ffc0000005a4ec0) at sleepq_wait+0xb0 _sleep(0x9ffc000000ee1258, 0x9ffc000000efe898, 0x5c, 0x9ffc000000dab938, 0x0, 0x0, 0x100) at _sleep+0x7d0 g_run_events(0x9ffc000000efe8b0, 0x0, 0x0, 0x0) at g_run_events+0xb00 g_event_procbody(0x9ffc000000ee1230, 0x9ffc000000dc1218, 0x9ffc0000005335f0, 0x40c) at g_event_procbody+0xb0 fork_exit(0x9ffc000000e1a620, 0x0, 0xa0000000a07e5550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command intr pid 12 tid 100048 td 0xe00000001174d680 cpu_switch(0xe00000001174d680, 0xe000000012500d80, 0x9ffc000000f27100, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe00000001174d680, 0xe000000012500d80, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x109, 0x0, 0xe00000001174d680, 0x9ffc00000053c9d0) at mi_switch+0x3f0 ithread_loop(0xe00000001119cc60, 0xa00000009de13550, 0xe00000001119cc74, 0xe00000001119689c) at ithread_loop+0x420 fork_exit(0x9ffc000000e20f50, 0xe00000001119cc60, 0xa00000009de13550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command intr pid 12 tid 100046 td 0xe00000001178c000 cpu_switch(0xe00000001178c000, 0xe000000011182480, 0x9ffc000000f27100, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe00000001178c000, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x109, 0x0, 0xe00000001178c000, 0x9ffc00000053c9d0) at mi_switch+0x3f0 ithread_loop(0xe00000001119cd00, 0xa00000009dde5550, 0xe00000001119cd14, 0xe000000011196a9c) at ithread_loop+0x420 fork_exit(0x9ffc000000e20f50, 0xe00000001119cd00, 0xa00000009dde5550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command intr pid 12 tid 100042 td 0xe00000001174c480 cpu_switch(0xe00000001174c480, 0xe000000011182900, 0x9ffc000000f26400, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe00000001174c480, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x109, 0x0, 0xe00000001174c480, 0x9ffc00000053c9d0) at mi_switch+0x3f0 ithread_loop(0xe00000001119cd20, 0xa00000009dd77550, 0xe00000001119cd34, 0xe000000011196f9c) at ithread_loop+0x420 fork_exit(0x9ffc000000e20f50, 0xe00000001119cd20, 0xa00000009dd77550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command intr pid 12 tid 100040 td 0xe000000011744000 cpu_switch(0xe000000011744000, 0xe000000011182480, 0x9ffc000000f27100, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011744000, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x109, 0x0, 0xe000000011744000, 0x9ffc00000053c9d0) at mi_switch+0x3f0 ithread_loop(0xe00000001119ccc0, 0xa00000009dd19550, 0xe00000001119ccd4, 0xe000000011196d9c) at ithread_loop+0x420 fork_exit(0x9ffc000000e20f50, 0xe00000001119ccc0, 0xa00000009dd19550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command intr pid 12 tid 100035 td 0xe000000011480900 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100030 td 0xe000000011425200 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100025 td 0xe0000000111b1b00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100022 td 0xe000000011424900 cpu_switch(0xe000000011424900, 0xe000000011182900, 0x9ffc000000f26400, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011424900, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x109, 0x0, 0xe000000011424900, 0x9ffc00000053c9d0) at mi_switch+0x3f0 ithread_loop(0xe00000001119cea0, 0xa0000000e2e81550, 0xe00000001119ceb4, 0xe00000001119769c) at ithread_loop+0x420 fork_exit(0x9ffc000000e20f50, 0xe00000001119cea0, 0xa0000000e2e81550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command intr pid 12 tid 100021 td 0xe000000011424d80 cpu_switch(0xe000000011424d80, 0xe000000011182900, 0x9ffc000000f26400, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011424d80, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x109, 0x0, 0xe000000011424d80, 0x9ffc00000053c9d0) at mi_switch+0x3f0 ithread_loop(0xe00000001119cec0, 0xa0000000e2e79550, 0xe00000001119ced4, 0xe00000001119779c) at ithread_loop+0x420 fork_exit(0x9ffc000000e20f50, 0xe00000001119cec0, 0xa0000000e2e79550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command intr pid 12 tid 100019 td 0xe0000000111b0900 cpu_switch(0xe0000000111b0900, 0xe000000011182480, 0x9ffc000000f27100, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000111b0900, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x109, 0x0, 0xe0000000111b0900, 0x9ffc00000053c9d0) at mi_switch+0x3f0 ithread_loop(0xe00000001119cee0, 0xa0000000e2e69550, 0xe00000001119cef4, 0xe00000001119789c) at ithread_loop+0x420 fork_exit(0x9ffc000000e20f50, 0xe00000001119cee0, 0xa0000000e2e69550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command intr pid 12 tid 100018 td 0xe0000000111b0d80 cpu_switch(0xe0000000111b0d80, 0xe000000011182900, 0x9ffc000000f26400, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000111b0d80, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x109, 0x0, 0xe0000000111b0d80, 0x9ffc00000053c9d0) at mi_switch+0x3f0 ithread_loop(0xe00000001119cf00, 0xa0000000e2e61550, 0xe00000001119cf14, 0xe00000001119799c) at ithread_loop+0x420 fork_exit(0x9ffc000000e20f50, 0xe00000001119cf00, 0xa0000000e2e61550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command intr pid 12 tid 100008 td 0xe0000000111a0000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100007 td 0xe0000000111a0480 cpu_switch(0xe0000000111a0480, 0xe000000011182480, 0x9ffc000000f27100, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000111a0480, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x109, 0x0, 0xe0000000111a0480, 0x9ffc00000053c9d0) at mi_switch+0x3f0 ithread_loop(0xe00000001119cf40, 0xa0000000a07cd550, 0xe00000001119cf54, 0xe000000011197b9c) at ithread_loop+0x420 fork_exit(0x9ffc000000e20f50, 0xe00000001119cf40, 0xa0000000a07cd550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command intr pid 12 tid 100006 td 0xe0000000111a0900 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100005 td 0xe000000011182000 cpu_switch(0xe000000011182000, 0xe000000011182480, 0x9ffc000000f27100, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011182000, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x109, 0x0, 0xe000000011182000, 0x9ffc00000053c9d0) at mi_switch+0x3f0 ithread_loop(0xe00000001119cf80, 0xa0000000a07bd550, 0xe00000001119cf94, 0xe000000011197d9c) at ithread_loop+0x420 fork_exit(0x9ffc000000e20f50, 0xe00000001119cf80, 0xa0000000a07bd550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command idle pid 11 tid 100004 td 0xe000000011182480 ia64_ih_stop(0x9ffc000000aa9070, 0x40b) at ia64_ih_stop+0x80 ia64_handle_intr(0xa0000000a07b5000) at ia64_handle_intr+0x110 ivt_External_Interrupt() at ivt_External_Interrupt+0x30 --- trapframe at 0xa0000000a07b5000 acpi_cpu_c1(0x9ffc0000001779f0, 0x48b) at acpi_cpu_c1+0x71 acpi_cpu_idle(0x0, 0x9ffc000000edce20, 0x642ea6cd77, 0xe000000011177c00) at acpi_cpu_idle+0x420 cpu_idle(0x0, 0x19526e0b, 0x9ffc0000005e96d0, 0xd9f) at cpu_idle+0x130 sched_idletd(0x10, 0x10, 0x2, 0xfffffffffffffffd) at sched_idletd+0x670 fork_exit(0x9ffc000000e3bef0, 0x0, 0xa0000000a07b5550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command idle pid 11 tid 100003 td 0xe000000011182900 cpu_switch(0xe000000011182900, 0xe000000012800000, 0x9ffc000000f26400, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011182900, 0xe000000012800000, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x608, 0x0, 0xe000000011182900, 0x9ffc00000059ff20) at mi_switch+0x3f0 critical_exit(0xe000000011182900, 0xe000000011182cec, 0x9ffc000000aad1e0, 0x289) at critical_exit+0x170 cpu_idle(0x0, 0xddf3fdd, 0x9ffc0000005e96d0, 0xd9f) at cpu_idle+0x190 sched_idletd(0xd, 0xd, 0x2, 0xfffffffffffffffe) at sched_idletd+0x670 fork_exit(0x9ffc000000e3bef0, 0x0, 0xa0000000a07ad550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command init pid 1 tid 100002 td 0xe000000011182d80 cpu_switch(0xe000000011182d80, 0xe000000011182480, 0x9ffc000000f5e628, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011182d80, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011182d80, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe000000011180de0, 0x6c, 0xe0000000111830f8, 0xe000000011182d80) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000011180de0, 0x6c, 0x9ffc000000dd7480, 0x9ffc000000dd7840) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000011180de0, 0x6c, 0x9ffc000000e06798, 0x100, 0x9ffc0000005a4ea0) at sleepq_wait_sig+0x30 _sleep(0xe000000011180de0, 0xe000000011180ee0, 0x6c, 0x9ffc000000e06798, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0 kern_wait6(0xe000000011182d80, 0x7, 0x0, 0xa0000000a07a5390, 0x30, 0x0, 0x0) at kern_wait6+0xba0 kern_wait(0xe000000011182d80, 0xffffffff, 0xa0000000a07a5390, 0x0, 0x0) at kern_wait+0xe0 sys_wait4(0xe000000011182d80, 0xa0000000a07a54e8, 0xa0000000a07a5390, 0x9ffc000000ac2c80) at sys_wait4+0x70 syscall(0xe000000011180de0, 0x0, 0x0, 0xe000000011182d80, 0x0, 0x0, 0x9ffc000000abeba0, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command audit pid 10 tid 100001 td 0xe000000011183200 cpu_switch(0xe000000011183200, 0xe000000011183680, 0x9ffc000000f5cd78, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011183200, 0xe000000011183680, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011183200, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc0000011087c8, 0x0, 0xe000000011183578, 0xe000000011183200) at sleepq_switch+0x2e0 sleepq_wait(0x9ffc0000011087c8, 0x0, 0x9ffc000000dd7480, 0xe000000011183200, 0x9ffc0000004fc2c0) at sleepq_wait+0xb0 _cv_wait(0x9ffc0000011087c8, 0x9ffc000001108798, 0xe000000011183200, 0x9ffc000000efef70, 0x9ffc000000f014b0, 0x9ffc000000e81ac0, 0x0) at _cv_wait+0x4d0 audit_worker(0x0, 0xa0000000a079d550, 0xa0000000a079d538, 0x9ffc0000011087d8) at audit_worker+0x150 fork_exit(0x9ffc000000e1ef20, 0x0, 0xa0000000a079d550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command kernel pid 0 tid 100053 td 0xe00000001178cd80 cpu_switch(0xe00000001178cd80, 0xe0000000124bc480, 0x9ffc000000f5def8, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe00000001178cd80, 0xe0000000124bc480, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001178cd80, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000000f0af10, 0x0, 0xe00000001178d0f8, 0xe00000001178cd80) at sleepq_switch+0x2e0 sleepq_timedwait(0x9ffc000000f0af10, 0x0, 0x0, 0x9ffc000000dd7480, 0xe00000001178cd80) at sleepq_timedwait+0xb0 _sleep(0x9ffc000000f0af10, 0x0, 0x0, 0x9ffc000000dab938, 0x1, 0x0, 0x100) at _sleep+0x780 pause_sbt(0x9ffc000000dab938, 0x2fffffc88, 0x0, 0x100) at pause_sbt+0x1b0 deadlkres(0x9ffc000000efeac8, 0x0, 0x9ffc000000dc8140, 0x9ffc000000dc5468) at deadlkres+0x670 fork_exit(0x9ffc000000e23230, 0x0, 0xa00000009de3b550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command kernel pid 0 tid 100050 td 0xe00000001174cd80 cpu_switch(0xe00000001174cd80, 0xe00000001178cd80, 0x9ffc000000f5ce18, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe00000001174cd80, 0xe00000001178cd80, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001174cd80, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe000000011795300, 0x0, 0xe00000001174d0f8, 0xe00000001174cd80) at sleepq_switch+0x2e0 sleepq_wait(0xe000000011795300, 0x0, 0x9ffc000000dd7480, 0xe00000001174cd80, 0x9ffc0000005a4ec0) at sleepq_wait+0xb0 _sleep(0xe000000011795300, 0xe000000011795330, 0x0, 0x9ffc000000dab938, 0x0, 0x0, 0x100) at _sleep+0x7d0 taskqueue_thread_loop(0xe000000011795300, 0xe000000011795360, 0xe00000001179535c, 0xe000000011795330) at taskqueue_thread_loop+0x1d0 fork_exit(0x9ffc000000e3bff0, 0xe000000011162740, 0xa00000009de23550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command kernel pid 0 tid 100049 td 0xe00000001174d200 cpu_switch(0xe00000001174d200, 0xe00000001174cd80, 0x9ffc000000f5ce18, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe00000001174d200, 0xe00000001174cd80, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001174d200, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe000000011795300, 0x0, 0xe00000001174d578, 0xe00000001174d200) at sleepq_switch+0x2e0 sleepq_wait(0xe000000011795300, 0x0, 0x9ffc000000dd7480, 0xe00000001174d200, 0x9ffc0000005a4ec0) at sleepq_wait+0xb0 _sleep(0xe000000011795300, 0xe000000011795330, 0x0, 0x9ffc000000dab938, 0x0, 0x0, 0x100) at _sleep+0x7d0 taskqueue_thread_loop(0xe000000011795300, 0xe000000011795360, 0xe00000001179535c, 0xe000000011795330) at taskqueue_thread_loop+0x1d0 fork_exit(0x9ffc000000e3bff0, 0xe000000011162740, 0xa00000009de1b550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command kernel pid 0 tid 100045 td 0xe000000011745680 cpu_switch(0xe000000011745680, 0xe000000011182480, 0x9ffc000000f5c6e8, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011745680, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011745680, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe0000000111a2500, 0x0, 0xe0000000117459f8, 0xe000000011745680) at sleepq_switch+0x2e0 sleepq_wait(0xe0000000111a2500, 0x0, 0x9ffc000000dd7480, 0xe000000011745680, 0x9ffc0000005a4520) at sleepq_wait+0xb0 msleep_spin_sbt(0xe0000000111a2500, 0xe0000000111a2530, 0x9ffc000000dab938, 0x0, 0x0, 0x100, 0xe0000000111a2548) at msleep_spin_sbt+0x440 taskqueue_thread_loop(0xe0000000111a2500, 0xe0000000111a2560, 0xe0000000111a255c, 0xe0000000111a2530) at taskqueue_thread_loop+0x190 fork_exit(0x9ffc000000e3bff0, 0xa000000000a34830, 0xa00000009dddd550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command kernel pid 0 tid 100044 td 0xe000000011745b00 cpu_switch(0xe000000011745b00, 0xe000000011182900, 0x9ffc000000f5c710, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011745b00, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011745b00, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe0000000111a2600, 0x0, 0xe000000011745e78, 0xe000000011745b00) at sleepq_switch+0x2e0 sleepq_wait(0xe0000000111a2600, 0x0, 0x9ffc000000dd7480, 0xe000000011745b00, 0x9ffc0000005a4520) at sleepq_wait+0xb0 msleep_spin_sbt(0xe0000000111a2600, 0xe0000000111a2630, 0x9ffc000000dab938, 0x0, 0x0, 0x100, 0xe0000000111a2648) at msleep_spin_sbt+0x440 taskqueue_thread_loop(0xe0000000111a2600, 0xe0000000111a2660, 0xe0000000111a265c, 0xe0000000111a2630) at taskqueue_thread_loop+0x190 fork_exit(0x9ffc000000e3bff0, 0xa000000000a30830, 0xa00000009ddd5550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command kernel pid 0 tid 100023 td 0xe000000011424480 cpu_switch(0xe000000011424480, 0xe000000011182900, 0x9ffc000000f5ce18, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011424480, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011424480, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe0000000111a5300, 0x0, 0xe0000000114247f8, 0xe000000011424480) at sleepq_switch+0x2e0 sleepq_wait(0xe0000000111a5300, 0x0, 0x9ffc000000dd7480, 0xe000000011424480, 0x9ffc0000005a4ec0) at sleepq_wait+0xb0 _sleep(0xe0000000111a5300, 0xe0000000111a5330, 0x0, 0x9ffc000000dab938, 0x0, 0x0, 0x100) at _sleep+0x7d0 taskqueue_thread_loop(0xe0000000111a5300, 0xe0000000111a5360, 0xe0000000111a535c, 0xe0000000111a5330) at taskqueue_thread_loop+0x1d0 fork_exit(0x9ffc000000e3bff0, 0x9ffc000000ee2c68, 0xa0000000e2e89550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command kernel pid 0 tid 100020 td 0xe0000000111b0480 cpu_switch(0xe0000000111b0480, 0xe000000011182480, 0x9ffc000000f5ce90, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000111b0480, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000111b0480, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe0000000111a5600, 0x0, 0xe0000000111b07f8, 0xe0000000111b0480) at sleepq_switch+0x2e0 sleepq_wait(0xe0000000111a5600, 0x0, 0x9ffc000000dd7480, 0xe0000000111b0480, 0x9ffc0000005a4ec0) at sleepq_wait+0xb0 _sleep(0xe0000000111a5600, 0xe0000000111a5630, 0x0, 0x9ffc000000dab938, 0x0, 0x0, 0x100) at _sleep+0x7d0 taskqueue_thread_loop(0xe0000000111a5600, 0xe0000000111a5660, 0xe0000000111a565c, 0xe0000000111a5630) at taskqueue_thread_loop+0x1d0 fork_exit(0x9ffc000000e3bff0, 0x9ffc000000ee1d48, 0xa0000000e2e71550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command kernel pid 0 tid 100017 td 0xe0000000111b1200 cpu_switch(0xe0000000111b1200, 0xe0000000111b0480, 0x9ffc000000f5cf08, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000111b1200, 0xe0000000111b0480, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000111b1200, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe0000000111a5900, 0x0, 0xe0000000111b1578, 0xe0000000111b1200) at sleepq_switch+0x2e0 sleepq_wait(0xe0000000111a5900, 0x0, 0x9ffc000000dd7480, 0xe0000000111b1200, 0x9ffc0000005a4520) at sleepq_wait+0xb0 msleep_spin_sbt(0xe0000000111a5900, 0xe0000000111a5930, 0x9ffc000000dab938, 0x0, 0x0, 0x100, 0xe0000000111a5948) at msleep_spin_sbt+0x440 taskqueue_thread_loop(0xe0000000111a5900, 0xe0000000111a5960, 0xe0000000111a595c, 0xe0000000111a5930) at taskqueue_thread_loop+0x190 fork_exit(0x9ffc000000e3bff0, 0x9ffc000000ee0900, 0xa0000000e2e59550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command kernel pid 0 tid 100016 td 0xe0000000111b1680 cpu_switch(0xe0000000111b1680, 0xe0000000111b1200, 0x9ffc000000f5cf08, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000111b1680, 0xe0000000111b1200, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000111b1680, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe0000000111a5900, 0x0, 0xe0000000111b19f8, 0xe0000000111b1680) at sleepq_switch+0x2e0 sleepq_wait(0xe0000000111a5900, 0x0, 0x9ffc000000dd7480, 0xe0000000111b1680, 0x9ffc0000005a4520) at sleepq_wait+0xb0 msleep_spin_sbt(0xe0000000111a5900, 0xe0000000111a5930, 0x9ffc000000dab938, 0x0, 0x0, 0x100, 0xe0000000111a5948) at msleep_spin_sbt+0x440 taskqueue_thread_loop(0xe0000000111a5900, 0xe0000000111a5960, 0xe0000000111a595c, 0xe0000000111a5930) at taskqueue_thread_loop+0x190 fork_exit(0x9ffc000000e3bff0, 0x9ffc000000ee0900, 0xa0000000e2e51550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command kernel pid 0 tid 100015 td 0xe0000000111a0d80 cpu_switch(0xe0000000111a0d80, 0xe0000000111b1680, 0x9ffc000000f5cf08, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000111a0d80, 0xe0000000111b1680, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000111a0d80, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe0000000111a5900, 0x0, 0xe0000000111a10f8, 0xe0000000111a0d80) at sleepq_switch+0x2e0 sleepq_wait(0xe0000000111a5900, 0x0, 0x9ffc000000dd7480, 0xe0000000111a0d80, 0x9ffc0000005a4520) at sleepq_wait+0xb0 msleep_spin_sbt(0xe0000000111a5900, 0xe0000000111a5930, 0x9ffc000000dab938, 0x0, 0x0, 0x100, 0xe0000000111a5948) at msleep_spin_sbt+0x440 taskqueue_thread_loop(0xe0000000111a5900, 0xe0000000111a5960, 0xe0000000111a595c, 0xe0000000111a5930) at taskqueue_thread_loop+0x190 fork_exit(0x9ffc000000e3bff0, 0x9ffc000000ee0900, 0xa0000000e2e49550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command kernel pid 0 tid 100014 td 0xe0000000111a1200 cpu_switch(0xe0000000111a1200, 0xe0000000111a0d80, 0x9ffc000000f5cf30, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe0000000111a1200, 0xe0000000111a0d80, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000111a1200, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe0000000111a5a00, 0x0, 0xe0000000111a1578, 0xe0000000111a1200) at sleepq_switch+0x2e0 sleepq_wait(0xe0000000111a5a00, 0x0, 0x9ffc000000dd7480, 0xe0000000111a1200, 0x9ffc0000005a4ec0) at sleepq_wait+0xb0 _sleep(0xe0000000111a5a00, 0xe0000000111a5a30, 0x0, 0x9ffc000000dab938, 0x0, 0x0, 0x100) at _sleep+0x7d0 taskqueue_thread_loop(0xe0000000111a5a00, 0xe0000000111a5a60, 0xe0000000111a5a5c, 0xe0000000111a5a30) at taskqueue_thread_loop+0x1d0 fork_exit(0x9ffc000000e3bff0, 0x9ffc000000ee1488, 0xa0000000e2e41550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command kernel pid 0 tid 100009 td 0xe000000011183b00 cpu_switch(0xe000000011183b00, 0xe000000011182480, 0x9ffc000000f5cfa8, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0xe000000011183b00, 0xe000000011182480, 0x9ffc000000f27118, 0x9ffc000000f27100) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011183b00, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0xe0000000111a5d00, 0x0, 0xe000000011183e78, 0xe000000011183b00) at sleepq_switch+0x2e0 sleepq_wait(0xe0000000111a5d00, 0x0, 0x9ffc000000dd7480, 0xe000000011183b00, 0x9ffc0000005a4ec0) at sleepq_wait+0xb0 _sleep(0xe0000000111a5d00, 0xe0000000111a5d30, 0x0, 0x9ffc000000dab938, 0x0, 0x0, 0x100) at _sleep+0x7d0 taskqueue_thread_loop(0xe0000000111a5d00, 0xe0000000111a5d60, 0xe0000000111a5d5c, 0xe0000000111a5d30) at taskqueue_thread_loop+0x1d0 fork_exit(0x9ffc000000e3bff0, 0x9ffc000000ee1b20, 0xa0000000a07dd550) at fork_exit+0x120 enter_userland() at enter_userland Tracing command kernel pid 0 tid 100000 td 0x9ffc000000efef70 cpu_switch(0x9ffc000000efef70, 0xe000000011182900, 0x9ffc000000f5c670, 0x9ffc0000005ec410) at cpu_switch+0xd0 sched_switch(0x9ffc000000efef70, 0xe000000011182900, 0x9ffc000000f26418, 0x9ffc000000f26400) at sched_switch+0x890 mi_switch(0x104, 0x0, 0x9ffc000000efef70, 0x9ffc000000625100) at mi_switch+0x3f0 sleepq_switch(0x9ffc000000efeac8, 0x54, 0x9ffc000000eff2e8, 0x9ffc000000efef70) at sleepq_switch+0x2e0 sleepq_timedwait(0x9ffc000000efeac8, 0x54, 0x0, 0x9ffc000000dd7480, 0x9ffc000000efef70) at sleepq_timedwait+0xb0 _sleep(0x9ffc000000efeac8, 0x0, 0x54, 0x9ffc000000e0a9b8, 0x1, 0x0, 0x100) at _sleep+0x780 swapper(0x9ffc000000f014b0, 0x0, 0x9ffc000000e0a848, 0xffffffff80000000) at swapper+0x480 mi_startup(0x9ffc000000f014b0) at mi_startup+0x3a0 __start() at __start+0xd0 db> From owner-freebsd-ia64@FreeBSD.ORG Tue Oct 15 16:21:11 2013 Return-Path: Delivered-To: freebsd-ia64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id EC4EAD34; Tue, 15 Oct 2013 16:21:11 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4306023F5; Tue, 15 Oct 2013 16:21:11 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r9FGKhWd031017; Tue, 15 Oct 2013 19:20:43 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r9FGKhWd031017 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r9FGKhfv031016; Tue, 15 Oct 2013 19:20:43 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 15 Oct 2013 19:20:43 +0300 From: Konstantin Belousov To: Anton Shterenlikht Subject: Re: ia64: panic: wrong page state m 0xe00000027fcc1900 Message-ID: <20131015162043.GG3865@kib.kiev.ua> References: <201310151510.r9FFA4fH047273@mech-cluster241.men.bris.ac.uk> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ahP6B03r4gLOj5uD" Content-Disposition: inline In-Reply-To: <201310151510.r9FFA4fH047273@mech-cluster241.men.bris.ac.uk> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-current@freebsd.org, freebsd-ia64@freebsd.org X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 16:21:12 -0000 --ahP6B03r4gLOj5uD Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Oct 15, 2013 at 04:10:04PM +0100, Anton Shterenlikht wrote: > This panic is always reproducible by > starting nginx, and directing the browser > to poudriere logs/bulk/ia64-default/latest/. =46rom ddb, do 'show pginfo
'. --ahP6B03r4gLOj5uD Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJSXWtaAAoJEJDCuSvBvK1BswgQAKMLIIYinz2naNH/hURuh+OM z1c9DXmszb0GwjldHmtOViOd74FNpzVQGF/dkR6OA+QO6YtzvBBYHeg1RtVrPtiK xbaKXxso0ca3m31KLzYPMGWg8RzmMrieljU56JXn1i+nFCup2pBQZGl4DsH1Y93M VQKKMwjrFAvV9E1v36GW+ziRd3QdmRag6toim0M6KuybXJ6y3FcMmANAKIYhjSZt GrXNTu2XVEdIegrTEdFGa7bk7k/Ux8sRDxn85GMmE6nVlwbhJFGjbC4j1Wb362j2 39cT6fNcZ20KgnXykOxnL/r5QfuJNBOLS571NWUmmnilWhe9w3l9xgASyYcypHnC tYy/QGjsdNRPjGC2szAxws6FfAWxzRxMZ4U1Y0Zx3ePJChJrDuUs/XYozSTGw1gh ey+/TzswfpuvWJKvUAn16NKo2avNk6kXgtvpPNP9xGaEqcoChFlsTYDGgOM0KjZq h1vGu1ghGGK+nocaiYDrqKmOTHJAnnFfypTxdihxpKJTye/oQ5UTmgHav3lei3Z6 +QEc9dTLgk+d4OJ8KLzT4ITVyh3IDTdX5AnZhUNlzaxpG7kpahPzfpM60G92XMsH BtYZwZKSDNDqw870GOsfby9abDraO7kJeyjZOpMJQ92hNEA0iTYnOZ2oVwfN6Fxs 6vk8Rj7K6IAft8LmblCR =U/qT -----END PGP SIGNATURE----- --ahP6B03r4gLOj5uD-- From owner-freebsd-ia64@FreeBSD.ORG Tue Oct 15 19:29:09 2013 Return-Path: Delivered-To: freebsd-ia64@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 227441F4; Tue, 15 Oct 2013 19:29:09 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id EA5392202; Tue, 15 Oct 2013 19:29:08 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r9FJT8tE029157; Tue, 15 Oct 2013 19:29:08 GMT (envelope-from jhb@freefall.freebsd.org) Received: (from jhb@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r9FJT8xS029156; Tue, 15 Oct 2013 19:29:08 GMT (envelope-from jhb) Date: Tue, 15 Oct 2013 19:29:08 GMT Message-Id: <201310151929.r9FJT8xS029156@freefall.freebsd.org> To: bipolar009@gmail.com, jhb@FreeBSD.org, freebsd-ia64@FreeBSD.org From: jhb@FreeBSD.org Subject: Re: ia64/182578: GRUB does not recognize Windows 7 X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 19:29:09 -0000 Synopsis: GRUB does not recognize Windows 7 State-Changed-From-To: open->closed State-Changed-By: jhb State-Changed-When: Tue Oct 15 19:28:27 UTC 2013 State-Changed-Why: Not a FreeBSD issue. http://www.freebsd.org/cgi/query-pr.cgi?pr=182578 From owner-freebsd-ia64@FreeBSD.ORG Tue Oct 15 21:19:39 2013 Return-Path: Delivered-To: freebsd-ia64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id AC55F165; Tue, 15 Oct 2013 21:19:39 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 7DD32296C; Tue, 15 Oct 2013 21:19:39 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 612EDB980; Tue, 15 Oct 2013 17:19:38 -0400 (EDT) From: John Baldwin To: freebsd-ia64@freebsd.org, mexas@bris.ac.uk Subject: Re: panic: uma_zfree: Freeing to non free bucket index. Date: Tue, 15 Oct 2013 17:10:02 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20130906; KDE/4.5.5; amd64; ; ) References: <201310140844.r9E8iSYM015098@mech-cluster241.men.bris.ac.uk> In-Reply-To: <201310140844.r9E8iSYM015098@mech-cluster241.men.bris.ac.uk> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201310151710.02551.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 15 Oct 2013 17:19:38 -0400 (EDT) Cc: freebsd-current@freebsd.org X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 21:19:39 -0000 On Monday, October 14, 2013 4:44:28 am Anton Shterenlikht wrote: > > BTW, I see in dmesg: > > Starting ddb. > ddb: sysctl: debug.ddb.scripting.scripts: Invalid argument > /etc/rc: WARNING: failed to start ddb > > What is that about? > > > panic: uma_zfree: Freeing to non free bucket index. > cpuid = 0 > KDB: stack backtrace: > db_trace_self(0x9ffc000000158380) at db_trace_self+0x40 > db_trace_self_wrapper(0x9ffc000000607370) at db_trace_self_wrapper+0x70 > kdb_backtrace(0x9ffc000000ed0e10, 0x9ffc00000058e660, 0x40c, 0x9ffc0000010a44a0) at kdb_backtrace+0xc0 > vpanic(0x9ffc000000dfc468, 0xa0000000e26e0fd8, 0x9ffc000000ef9670, 0x9ffc000000ed0bc0) at vpanic+0x260 > kassert_panic(0x9ffc000000dfc468, 0xe000000015e25f90, 0xe000000015e243e0, 0xe00000027ffd5200) at kassert_panic+0x120 > uma_zfree_arg(0xe00000027ffccfc0, 0xe000000015e243e0, 0x0) at uma_zfree_arg+0x2d0 > g_destroy_bio(0xe000000015e243e0, 0x9ffc0000004ad4a0, 0x30a, 0x30a) at g_destroy_bio+0x30 > g_disk_done(0xe000000015e243e0, 0xe000000015e15d10, 0xe000000012672700, 0x9ffc0000006b18c0) at g_disk_done+0x140 > biodone(0xe000000015e243e0, 0x9ffc000000e0e150, 0xe000000010c24030, 0x0, 0x0, 0x0, 0x9ffc000000066890, 0x614) at biodone+0x180 > dadone(0xe000000012672600, 0xe000000012541000, 0xe000000015e243e0, 0x7) at dadone+0x620 > camisr_runqueue(0xe000000011a2dc00, 0xe000000012541054, 0xe000000012541000, 0x135d) at camisr_runqueue+0x6c0 > camisr(0xe000000011a2dc20, 0xe000000011a2dc00, 0x9ffc000000bee9d0, 0xa0000000e26e1548) at camisr+0x260 > intr_event_execute_handlers(0xe0000000111764a0, 0xe00000001118d998, 0xe000000011191c00, 0x0) at intr_event_execute_handlers+0x280 > ithread_loop(0xe000000011192f00, 0xa0000000e26e1550, 0xe000000011192f14, 0xe00000001118d99c) at ithread_loop+0x1b0 > fork_exit(0x9ffc000000e12a90, 0xe000000011192f00, 0xa0000000e26e1550) at fork_exit+0x120 > enter_userland() at enter_userland > KDB: enter: panic > [ thread pid 12 tid 100015 ] > Stopped at kdb_enter+0x92: [I2] addl r14=0xffffffffffe2c990,gp ;; > db> > > db> scripts > lockinfo=show locks; show alllocks; show lockedvnods > db> run lockinfo > db:0:lockinfo> show locks > db:0:locks> show alllocks > db:0:alllocks> show lockedvnods > Locked vnodes > > 0xe00000001ab39ba8: tag ufs, type VDIR > usecount 1, writecount 0, refcount 3 mountedhere 0 > flags (VI_ACTIVE) > v_object 0xe00000001cd30900 ref 0 pages 0 > lock type ufs: EXCL by thread 0xe0000000183d9680 (pid 41389, cpdup, tid 100121) > ino 5467932, on dev da5p1 > > 0xe000000015ed3ba8: tag ufs, type VDIR > usecount 1, writecount 0, refcount 3 mountedhere 0 > flags (VI_ACTIVE) > v_object 0xe00000001cd33e00 ref 0 pages 0 > lock type ufs: EXCL by thread 0xe000000012a28900 (pid 41421, cpdup, tid 100092) > ino 5467948, on dev da5p1 > > 0xe00000001ab16938: tag ufs, type VREG > usecount 1, writecount 0, refcount 3 mountedhere 0 > flags (VI_ACTIVE) > v_object 0xe00000001cd98a00 ref 0 pages 1 > lock type ufs: EXCL by thread 0xe000000018494000 (pid 41337, cpdup, tid 100137) > ino 5469420, on dev da5p1 > > 0xe00000001b2503b0: tag ufs, type VREG > usecount 1, writecount 0, refcount 1 mountedhere 0 > flags (VI_ACTIVE) > lock type ufs: EXCL by thread 0xe000000012a28900 (pid 41421, cpdup, tid 100092) > ino 5469421, on dev da5p1 > > 0xe00000001ab2a760: tag ufs, type VREG > usecount 1, writecount 0, refcount 1 mountedhere 0 > flags (VI_ACTIVE) > lock type ufs: EXCL by thread 0xe0000000183d9680 (pid 41389, cpdup, tid 100121) > ino 5469422, on dev da5p1 > db> > db> script zzz=textdump set; capture on; run lockinfo; show pcpu; bt; ps; alltrace; capture off; reset > db> run zzz I think 'reset' is going to reset without doing a dump? -- John Baldwin From owner-freebsd-ia64@FreeBSD.ORG Wed Oct 16 08:02:25 2013 Return-Path: Delivered-To: freebsd-ia64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 11731B71 for ; Wed, 16 Oct 2013 08:02:25 +0000 (UTC) (envelope-from mexas@bristol.ac.uk) Received: from eu1sys200aog119.obsmtp.com (eu1sys200aog119.obsmtp.com [207.126.144.147]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 68D9B28BD for ; Wed, 16 Oct 2013 08:02:24 +0000 (UTC) Received: from mail-wi0-f175.google.com ([209.85.212.175]) (using TLSv1) by eu1sys200aob119.postini.com ([207.126.147.11]) with SMTP ID DSNKUl5IDozbxXneN+SNK9ZDnzUf3k61+AId@postini.com; Wed, 16 Oct 2013 08:02:24 UTC Received: by mail-wi0-f175.google.com with SMTP id hm4so3898692wib.8 for ; Wed, 16 Oct 2013 01:02:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:message-id:to:subject:reply-to :in-reply-to; bh=oHA9i1H80oXi/8v0NjKqPT6f53tUtyhZonIIQz6jT1E=; b=W9Neq8zn0WYZe/ro1DrHXD1pMEjqrWyP3jOWQ9uZjRJ7rJj2hVCiCBbASPoypNGU7y tpOZVGL1aAKt5VslZeAKR5etIVpU2kBxYMxUejZHv0DxLPpV6NWT6wRuh+x01cLrYZS8 iJUeniQlOiaDHXWbpni3DGeceOVfvCtMxXQfUmQZQ9LxPE16TKsGwa4hd6DhhD8QemAk 5s33Q9iGb2UXZvvpe4oSw02b7C4On7wx2c7XdNSmbS2M5srp3ZgXgK3Fj17h9h9ldvH4 2ylxCx8LhrjnQ+/+IOFQ8z2fmIheRpba14MuBiuV4ecuNUNcxBvf3EQPf2MFCbbSZ3FZ 7kyA== X-Gm-Message-State: ALoCoQlqUUksgoe96hU3zhZQZ8ZjzhqhHpgnhIeN2Kk4Ka+TTO4pp+hUY6zPHDzTNWA9Dpm2dkIhEKPtak7z926xmOlgK215boP5/aXYT6l4aop0wyB/OM+67tFukqxcgh8Zj2h2eXFYgvFfYx+FNOpmKG+biEUuTBm2NxwpWxJDStzy691gBho= X-Received: by 10.180.38.72 with SMTP id e8mr1183100wik.0.1381910542625; Wed, 16 Oct 2013 01:02:22 -0700 (PDT) X-Received: by 10.180.38.72 with SMTP id e8mr1183091wik.0.1381910542558; Wed, 16 Oct 2013 01:02:22 -0700 (PDT) Received: from mech-cluster241.men.bris.ac.uk (mech-cluster241.men.bris.ac.uk. [137.222.187.241]) by mx.google.com with ESMTPSA id jf2sm3363417wic.2.1969.12.31.16.00.00 (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 16 Oct 2013 01:02:21 -0700 (PDT) Sender: Anton Shterenlikht Received: from mech-cluster241.men.bris.ac.uk (localhost [127.0.0.1]) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6) with ESMTP id r9G82Ja6010220 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 16 Oct 2013 09:02:19 +0100 (BST) (envelope-from mexas@mech-cluster241.men.bris.ac.uk) Received: (from mexas@localhost) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6/Submit) id r9G82JvJ010219; Wed, 16 Oct 2013 09:02:19 +0100 (BST) (envelope-from mexas) Date: Wed, 16 Oct 2013 09:02:19 +0100 (BST) From: Anton Shterenlikht Message-Id: <201310160802.r9G82JvJ010219@mech-cluster241.men.bris.ac.uk> To: freebsd-current@freebsd.org, freebsd-ia64@freebsd.org Subject: Re: panic: wrong page state m 0xe00000027a9adb40 + savecore deadlock In-Reply-To: X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: mexas@bris.ac.uk List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 08:02:25 -0000 panic: http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/182999 deadlock: http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/183007 Anton From owner-freebsd-ia64@FreeBSD.ORG Wed Oct 16 11:55:34 2013 Return-Path: Delivered-To: freebsd-ia64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 004C9CEF; Wed, 16 Oct 2013 11:55:33 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 61CA02766; Wed, 16 Oct 2013 11:55:33 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r9GBtQR1084440; Wed, 16 Oct 2013 14:55:26 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r9GBtQR1084440 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r9GBtQ38084438; Wed, 16 Oct 2013 14:55:26 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 16 Oct 2013 14:55:26 +0300 From: Konstantin Belousov To: Anton Shterenlikht Subject: Re: panic: wrong page state m 0xe00000027a9adb40 + savecore deadlock Message-ID: <20131016115526.GR3865@kib.kiev.ua> References: <201310160802.r9G82JvJ010219@mech-cluster241.men.bris.ac.uk> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Ep5m4srWGXPl6O+g" Content-Disposition: inline In-Reply-To: <201310160802.r9G82JvJ010219@mech-cluster241.men.bris.ac.uk> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-current@freebsd.org, freebsd-ia64@freebsd.org X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 11:55:34 -0000 --Ep5m4srWGXPl6O+g Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Wed, Oct 16, 2013 at 09:02:19AM +0100, Anton Shterenlikht wrote: > panic: > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/182999 db> show pginfo 0xe00000027d352600 page 0xe00000027d352600 obj 0xe0000000128fda00 pidx 0x0 phys 0x275dc6000 q 255 hold 0 wire 1 af 0x0 of 0x0 f 0x0 act 0 busy 1 valid 0xff dirty 0x0 AFAIR ia64 uses 8K pages. Please do the following: 1. apply the patch at the end of this message, reproduce the problem and show me both exact panic message from the patched kernel and 'show pginfo addr' again. 2. show me the ls -la output for the file which was accessed through nginx, also what is the filesystem where the file resides on ? --Ep5m4srWGXPl6O+g Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJSXn6tAAoJEJDCuSvBvK1B1G8P/jYZZautxz7J+0z2omas2EKf HoN/4w7YhyevQnVWaxLkhrbR3uG6Tyy3cMT33vSQ9eiUjWuLbz+ezBKktYZw15N4 Wnc7YdK63OYX0/ZfUnqAwxKo30MN3vYJj+BtcIYnCJID+ktQDRBF4cpJzBqjn2iF Os3geAahzNsPP/iguPUCp+Uf7eG1SlxKBNc17xdMb20Y5feBbq1DZxgUREP3ngHO VDXG4GWxYP4u4+RwI4nTB8W/yBxDWETuV6NmcbgTgjNn0Fxj5QLm5SUQijPYw7w1 HnmW0VxEhrAaqvRICJhRIMdVfiaOUPXT3ZLqPnJpqTkgYcbZNaQ4m2wVaesvHZ7q D2C4a+nmzkSG5DuacPezpAFQoBn61j/Iolw3uPPcUmmJ61+DgnT+gfvi2T8jf4Y9 l1W1/MJXHTfH23irCmMy7IzRfxV6MWtHeuO+BPxqit3BhOefGaQsTOVhxi5yAWi1 BouJlzotkExP6sZeNcTBgNcMlTG1UT9oXL0QVQAFgJeaKx8BUbyhbNRF9rw2PoM+ WT+KK6nBkLRR2ovL+ZkRJULHKbcDRdpZMPBYpVJYZfMihqRqSz63LYCpSJPaKm6p BMZJyEft5nqLMhyuumGKrkDnGz3mWQVhvqme7qGfE32jz5mKhTRWtfi27DJqm+of uf0/XiFgjnABOZRhZSr6 =5+WG -----END PGP SIGNATURE----- --Ep5m4srWGXPl6O+g-- From owner-freebsd-ia64@FreeBSD.ORG Wed Oct 16 12:01:49 2013 Return-Path: Delivered-To: freebsd-ia64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2AF3F1CA; Wed, 16 Oct 2013 12:01:49 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BC6DE280E; Wed, 16 Oct 2013 12:01:48 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r9GC1gMR086226; Wed, 16 Oct 2013 15:01:42 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r9GC1gMR086226 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r9GC1gqq086225; Wed, 16 Oct 2013 15:01:42 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 16 Oct 2013 15:01:42 +0300 From: Konstantin Belousov To: Anton Shterenlikht Subject: Re: panic: wrong page state m 0xe00000027a9adb40 + savecore deadlock Message-ID: <20131016120142.GS3865@kib.kiev.ua> References: <201310160802.r9G82JvJ010219@mech-cluster241.men.bris.ac.uk> <20131016115526.GR3865@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="s9pXJW6w71JX4l3T" Content-Disposition: inline In-Reply-To: <20131016115526.GR3865@kib.kiev.ua> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-current@freebsd.org, freebsd-ia64@freebsd.org X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 12:01:49 -0000 --s9pXJW6w71JX4l3T Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Oct 16, 2013 at 02:55:26PM +0300, Konstantin Belousov wrote: > On Wed, Oct 16, 2013 at 09:02:19AM +0100, Anton Shterenlikht wrote: > > panic: > > http://www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/182999 >=20 > db> show pginfo 0xe00000027d352600 > page 0xe00000027d352600 obj 0xe0000000128fda00 pidx 0x0 phys 0x275dc6000 = q 255 hold 0 wire 1 > af 0x0 of 0x0 f 0x0 act 0 busy 1 valid 0xff dirty 0x0 >=20 > AFAIR ia64 uses 8K pages. >=20 > Please do the following: > 1. apply the patch at the end of this message, reproduce the problem > and show me both exact panic message from the patched kernel and 'show > pginfo addr' again. > 2. show me the ls -la output for the file which was accessed > through nginx, also what is the filesystem where the file resides on ? Sure, I forgot the patch. diff --git a/sys/kern/uipc_syscalls.c b/sys/kern/uipc_syscalls.c index 322550b..9d46dc7 100644 --- a/sys/kern/uipc_syscalls.c +++ b/sys/kern/uipc_syscalls.c @@ -2070,7 +2070,7 @@ free_page: } KASSERT(error !=3D 0 || (m->wire_count > 0 && vm_page_is_valid(m, off & PAGE_MASK, xfsize)), - ("wrong page state m %p", m)); + ("wrong page state m %p off %#jx xfsize %d", m, off, xfsize)); VM_OBJECT_WUNLOCK(obj); return (error); } --s9pXJW6w71JX4l3T Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJSXoAlAAoJEJDCuSvBvK1BPAEP/AzDZwOD7zXNuAdVzhG7bIi4 gRSOTCF+CIAu6ADTHnvgCHKOSyBX6IFnG4/bHVVnbv+pHalzWpdFqO2Ji+5150OK gjCP5rKIlgpzJpV8eklsw3pmcW06hEhyvujfkJnFX8uoJ90M3PMyQNt0GYq1Hm4y vh9elf7QeJmN4GuGDdHxdfzcW3z2FaqnB8S5IijebcwhIrcvMUluO5SnvRMlB99q 9o+q1wxEZRFx4/pGugT+PnGC3iZqBVN8rhbdbnMaOr1dRvQN+3zQJZU41Nfl3Csy CHf8C9HuP6VnDxed2nMRa1lVVvystuYxLQqdpFrLyEuyN3hP0X1M0htrxevXDn+q B7Ua9Jf/taueI+w0TuBkqMqc9p0x+SQa076NAVCpazQ/6xLJJma8xLYgQxPasW53 ej0p9CpD2B5rqO3I2hEXtu7qMbea/w4YVDp4eZVLKHZX7uea4QH/mV136wUc+JA+ lLWOOKa6xwgXQyWPI0mVlbQcfJUxrKiwFiPRAMvZpRAUEcdnuwVfvbVEcPqwynjk I1+gc19abrvHMKQLTzIjJ3o4rChKqkBAinUQkfgQCRX21jgt34sYKi/4mFJGwgp7 PRcHkbzqw5WSglunvjw1gThrPsORTrFuoHb9cVEIu+Z53W2j926BA7xleGuN7uUF 2QVi0BdpW8ATScMTfWPZ =AZIr -----END PGP SIGNATURE----- --s9pXJW6w71JX4l3T-- From owner-freebsd-ia64@FreeBSD.ORG Wed Oct 16 14:30:38 2013 Return-Path: Delivered-To: freebsd-ia64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E6B9DBC9 for ; Wed, 16 Oct 2013 14:30:38 +0000 (UTC) (envelope-from mexas@bristol.ac.uk) Received: from eu1sys200aog107.obsmtp.com (eu1sys200aog107.obsmtp.com [207.126.144.123]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 407102342 for ; Wed, 16 Oct 2013 14:30:37 +0000 (UTC) Received: from mail-wg0-f50.google.com ([74.125.82.50]) (using TLSv1) by eu1sys200aob107.postini.com ([207.126.147.11]) with SMTP ID DSNKUl6i+NEvj5wNQByrHtNwWbHTE/k+DJZT@postini.com; Wed, 16 Oct 2013 14:30:38 UTC Received: by mail-wg0-f50.google.com with SMTP id n12so786604wgh.17 for ; Wed, 16 Oct 2013 07:30:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:message-id:to:subject:cc :reply-to:in-reply-to; bh=+qbVc/Wde1PArlFnyejvT4hhPgxfo02XKqOvtoTGf4w=; b=j4wvcKgEmZdOg9xk3K01pOmV4rCV21vOf1OlVcc7DGcxTGF4oawSA1GG8kb0sLrt1u jjGZQ7uFVwkXieEo4ZIL8EwJK2v7IddvihPCkhufFCvJKrxgmBwcnt757YuKvRE7gCNX UR5wVwcm9hvKD1ktbPQtzzoioO9W45/ZmoaxjPaFwbb6tOTAWdJrrCZ2mrxtpjWugcoj H2hXXvF1JyJrjEQx06mEMYHilHMsBb2u7oBFmwl0G2pc1up/7JPk+jeD9uQPEqbULw6s a7ZkvkvEdeemMLHYmDhtCeFxyLNbEsE33ENO+Q726ddvpLhxy1PqqU3Iy0ZWzk0Q360L PAWw== X-Gm-Message-State: ALoCoQkrTiG90A3BPbQxnTnYZFciW4Deiq2jtUbKb+7y+tp5Fp87QLkTAfvktbq1ogBqPIcJq0YtFtdVB4ZDwwoEIeCaGCLXp2pLl/BKeu1Yjl/khsKCvLmpDz9lYtbylgQYUAjCipYMKadfoXYMhqixcAeyAQDWCfcv0oFA0sqbc7gi36uxnWs= X-Received: by 10.180.72.195 with SMTP id f3mr2540435wiv.51.1381933816123; Wed, 16 Oct 2013 07:30:16 -0700 (PDT) X-Received: by 10.180.72.195 with SMTP id f3mr2540427wiv.51.1381933816012; Wed, 16 Oct 2013 07:30:16 -0700 (PDT) Received: from mech-cluster241.men.bris.ac.uk (mech-cluster241.men.bris.ac.uk. [137.222.187.241]) by mx.google.com with ESMTPSA id b13sm6424064wic.9.2013.10.16.07.30.14 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Oct 2013 07:30:15 -0700 (PDT) Sender: Anton Shterenlikht Received: from mech-cluster241.men.bris.ac.uk (localhost [127.0.0.1]) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6) with ESMTP id r9GEUD3X025962 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 16 Oct 2013 15:30:13 +0100 (BST) (envelope-from mexas@mech-cluster241.men.bris.ac.uk) Received: (from mexas@localhost) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6/Submit) id r9GEUCYo025961; Wed, 16 Oct 2013 15:30:12 +0100 (BST) (envelope-from mexas) Date: Wed, 16 Oct 2013 15:30:12 +0100 (BST) From: Anton Shterenlikht Message-Id: <201310161430.r9GEUCYo025961@mech-cluster241.men.bris.ac.uk> To: kostikbel@gmail.com, mexas@bris.ac.uk Subject: Re: panic: wrong page state m 0xe00000027a9adb40 + savecore deadlock In-Reply-To: <20131016120142.GS3865@kib.kiev.ua> Cc: freebsd-current@freebsd.org, freebsd-ia64@freebsd.org X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: mexas@bris.ac.uk List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 14:30:39 -0000 >From kostikbel@gmail.com Wed Oct 16 13:02:51 2013 > >On Wed, Oct 16, 2013 at 02:55:26PM +0300, Konstantin Belousov wrote: >> On Wed, Oct 16, 2013 at 09:02:19AM +0100, Anton Shterenlikht wrote: >> > panic: >> > http://www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/182999 >>=20 >> db> show pginfo 0xe00000027d352600 >> page 0xe00000027d352600 obj 0xe0000000128fda00 pidx 0x0 phys 0x275dc6000 = >q 255 hold 0 wire 1 >> af 0x0 of 0x0 f 0x0 act 0 busy 1 valid 0xff dirty 0x0 >>=20 >> AFAIR ia64 uses 8K pages. >>=20 >> Please do the following: >> 1. apply the patch at the end of this message, reproduce the problem >> and show me both exact panic message from the patched kernel and 'show >> pginfo addr' again. >> 2. show me the ls -la output for the file which was accessed >> through nginx, also what is the filesystem where the file resides on ? >Sure, I forgot the patch. > >diff --git a/sys/kern/uipc_syscalls.c b/sys/kern/uipc_syscalls.c >index 322550b..9d46dc7 100644 >--- a/sys/kern/uipc_syscalls.c >+++ b/sys/kern/uipc_syscalls.c >@@ -2070,7 +2070,7 @@ free_page: > } > KASSERT(error !=3D 0 || (m->wire_count > 0 && > vm_page_is_valid(m, off & PAGE_MASK, xfsize)), >- ("wrong page state m %p", m)); >+ ("wrong page state m %p off %#jx xfsize %d", m, off, xfsize)); > VM_OBJECT_WUNLOCK(obj); > return (error); > } The patch didn't apply cleanly. I concluded that my src was older that yours, so I updated to r 256624. Now I don't get this panic! I don't know whether to be happy or not. Anyway, I was on r255488 when the panic happened, and there have been a lot of changes under sys/kern. Specifically related to the patch: # svn info Path: . Working Copy Root Path: /usr/src URL: https://svn0.eu.freebsd.org/base/head/sys/kern Relative URL: ^/head/sys/kern Repository Root: https://svn0.eu.freebsd.org/base Repository UUID: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f Revision: 256624 Node Kind: directory Schedule: normal Last Changed Author: mav Last Changed Rev: 256614 Last Changed Date: 2013-10-16 10:56:40 +0100 (Wed, 16 Oct 2013) # svn diff -r 255488 uipc_syscalls.c Index: uipc_syscalls.c =================================================================== --- uipc_syscalls.c (revision 255488) +++ uipc_syscalls.c (working copy) @@ -123,21 +123,13 @@ /* * sendfile(2)-related variables and associated sysctls */ -int nsfbufs; -int nsfbufspeak; -int nsfbufsused; +static SYSCTL_NODE(_kern_ipc, OID_AUTO, sendfile, CTLFLAG_RW, 0, + "sendfile(2) tunables"); static int sfreadahead = 1; +SYSCTL_INT(_kern_ipc_sendfile, OID_AUTO, readahead, CTLFLAG_RW, + &sfreadahead, 0, "Number of sendfile(2) read-ahead MAXBSIZE blocks"); -SYSCTL_INT(_kern_ipc, OID_AUTO, nsfbufs, CTLFLAG_RDTUN, &nsfbufs, 0, - "Maximum number of sendfile(2) sf_bufs available"); -SYSCTL_INT(_kern_ipc, OID_AUTO, nsfbufspeak, CTLFLAG_RD, &nsfbufspeak, 0, - "Number of sendfile(2) sf_bufs at peak usage"); -SYSCTL_INT(_kern_ipc, OID_AUTO, nsfbufsused, CTLFLAG_RD, &nsfbufsused, 0, - "Number of sendfile(2) sf_bufs in use"); -SYSCTL_INT(_kern_ipc, OID_AUTO, sfreadahead, CTLFLAG_RW, &sfreadahead, 0, - "Number of sendfile(2) read-ahead MAXBSIZE blocks"); - static void sfstat_init(const void *unused) { @@ -2076,10 +2068,10 @@ vm_page_free(m); vm_page_unlock(m); } + KASSERT(error != 0 || (m->wire_count > 0 && + vm_page_is_valid(m, off & PAGE_MASK, xfsize)), + ("wrong page state m %p off %#jx xfsize %d", m, off, xfsize)); VM_OBJECT_WUNLOCK(obj); - KASSERT(error != 0 || (m->wire_count > 0 && m->valid == - VM_PAGE_BITS_ALL), - ("wrong page state m %p", m)); return (error); } # Please let me know if there is any other diagnostics you'd like to see. Otherwise, till next panic... Many thanks for your help Anton From owner-freebsd-ia64@FreeBSD.ORG Fri Oct 18 15:04:47 2013 Return-Path: Delivered-To: freebsd-ia64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 8351842C for ; Fri, 18 Oct 2013 15:04:47 +0000 (UTC) (envelope-from noreply+1723683720@badoo.com) Received: from cluster1095.monopost.com (cluster1095.monopost.com [159.253.178.121]) by mx1.freebsd.org (Postfix) with ESMTP id F1ECB2918 for ; Fri, 18 Oct 2013 15:04:46 +0000 (UTC) Received: from scripts41.mlan (scripts41.mlan [10.10.126.42]) by cluster1095.monopost.com (Postfix) with SMTP id 07FB26DFA6C6 for ; Fri, 18 Oct 2013 14:59:24 +0000 (UTC) X-DomainKeys: Sendmail DomainKeys Filter v1.0.2 cluster1095.monopost.com 07FB26DFA6C6 DomainKey-Signature: a=rsa-sha1; s=m23uiy45; d=badoo.com; c=nofws; q=dns; b=GzXKC6mVx7xgfRyOAkYC6yd6NfDZTLkE0biPUfjnqLp/1HkXcbDOa7jRI9Y2AIO2I weAaSaNH0S9v3Yh2OnIS++O7mJWEiHrsRZKp+96vx+C1GWbphX9KfXIMkrjgC9//ayk /qNNKxqh2XbO6TOBxks8KmSK38XvI23W5WQcFoQ= X-DKIM: Sendmail DKIM Filter v2.7.2 cluster1095.monopost.com 07FB26DFA6C6 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=badoo.com; s=m23uiy45; t=1382108364; bh=18gvIhuk10C0IlSWKeFeGxl6A5zVOrX/2bOkwj 36Wi4=; h=Date:To:MIME-Version:Reply-to:From:List-Unsubscribe: Content-Type:Subject; b=TurpcBNGj94zsiUII6vVAUYbilPsb5kF7EhWrinGn4 0zTl9AU4/jsbc8B5POdSvoBZ9qIeir2D9LqYyTUYUn//kiKu6SjaSjuyDkFw3ntqqd7 z8WzD0brnEqHD0mCRfxt/7DODF8JE9xsS9Xl2do3dtcP7YcI4YdUGWx4W4grwE= Received: by scripts41.mlan (sSMTP sendmail emulation); Fri, 18 Oct 2013 14:59:24 +0000 Date: Fri, 18 Oct 2013 14:59:24 +0000 To: freebsd-ia64@freebsd.org MIME-Version: 1.0 From: Innocent X-campaignid: badoo 003.61.2.131018145922 Subject: =?UTF-8?B?4piFIEZyZWVic2QgSWEsIElubm9jZW50IGxlZnQgYSBtZXNzYWdlIGZvciB5b3U=?= Message-Id: <20131018150447.8351842C@hub.freebsd.org> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: revcharles.cole@gmail.com List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Oct 2013 15:04:47 -0000 WW91IGNhbiBpbnN0YW50bHkgcmVwbHkgdXNpbmcgb3VyIG1lc3NhZ2UgZXhjaGFuZ2Ugc3lzdGVt Lg0KQ2hlY2sgeW91ciBtZXNzYWdlDQpodHRwOi8vZXUxLmJhZG9vLmNvbS8wMzMwNjYzMjg1L2lu LzZCOFJodmY4bXBjLz9sYW5nX2lkPTMmZz01Ny0wLTQmbT02MSZtaWQ9NTI2MTRjY2EwMDAwMDAw MDAwMDMwMDAwMDI3MWI5Y2EwMTNjZWQ2MDAwODQKCgpTb21lIG90aGVyIHBlb3BsZSBpbiB0aGUg YXJlYToKCgpJZiBjbGlja2luZyB0aGUgbGlua3MgaW4gdGhpcyBtZXNzYWdlIGRvZXMgbm90IHdv cmssDQpjb3B5IGFuZCBwYXN0ZSB0aGVtIGludG8gdGhlIGFkZHJlc3MgYmFyIG9mIHlvdXIgYnJv d3Nlci4KCgpZb3UgaGF2ZSByZWNlaXZlZCB0aGlzIGVtYWlsIGZyb20gQmFkb28gVHJhZGluZyBM aW1pdGVkIChwb3N0YWwgYWRkcmVzcyBiZWxvdykuDQpJZiB5b3UgZG8gbm90IHdpc2ggdG8gcmVj ZWl2ZSBmdXJ0aGVyIGVtYWlsIGNvbW11bmljYXRpb25zIGZyb20gQmFkb28gY2xpY2sgaGVyZSB0 byBvcHQgb3V0OiBodHRwczovL2V1MS5iYWRvby5jb20vaW1wZXJzb25hdGlvbi5waHRtbD9sYW5n X2lkPTMmZW1haWw9ZnJlZWJzZC1pYTY0JTQwZnJlZWJzZC5vcmcmYmxvY2tfY29kZT1jNDc0NmUm bT02MSZtaWQ9NTI2MTRjY2EwMDAwMDAwMDAwMDMwMDAwMDI3MWI5Y2EwMTNjZWQ2MDAwODQmZz0w LTAtNC4NCgpCYWRvbyBUcmFkaW5nIExpbWl0ZWQgaXMgYSBsaW1pdGVkIGNvbXBhbnkgcmVnaXN0 ZXJlZCBpbiBFbmdsYW5kIGFuZCBXYWxlcw0KdW5kZXIgQ1JOIDc1NDAyNTUgd2l0aCBpdHMgcmVn aXN0ZXJlZCBvZmZpY2UgYXQgTWVkaWEgVmlsbGFnZSwgMTMxIC0gMTUxIEdyZWF0IFRpdGNoZmll bGQgU3RyZWV0LCBMb25kb24sIFcxVyA1QkIu From owner-freebsd-ia64@FreeBSD.ORG Sat Oct 19 03:23:30 2013 Return-Path: Delivered-To: ia64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 8F3708E1; Sat, 19 Oct 2013 03:23:30 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-current.sentex.ca (freebsd-current.sentex.ca [64.7.128.98]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5502E2FFB; Sat, 19 Oct 2013 03:23:30 +0000 (UTC) Received: from freebsd-current.sentex.ca (localhost [127.0.0.1]) by freebsd-current.sentex.ca (8.14.5/8.14.5) with ESMTP id r9J3NK2s038122; Fri, 18 Oct 2013 23:23:20 -0400 (EDT) (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-current.sentex.ca (8.14.5/8.14.5/Submit) id r9J3NKpL038107; Sat, 19 Oct 2013 03:23:20 GMT (envelope-from tinderbox@freebsd.org) Date: Sat, 19 Oct 2013 03:23:20 GMT Message-Id: <201310190323.r9J3NKpL038107@freebsd-current.sentex.ca> X-Authentication-Warning: freebsd-current.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Subject: [head tinderbox] failure on ia64/ia64 Precedence: bulk X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Oct 2013 03:23:30 -0000 TB --- 2013-10-19 01:35:56 - tinderbox 2.20 running on freebsd-current.sentex.ca TB --- 2013-10-19 01:35:56 - FreeBSD freebsd-current.sentex.ca 8.3-PRERELEASE FreeBSD 8.3-PRERELEASE #0: Mon Mar 26 13:54:12 EDT 2012 des@freebsd-current.sentex.ca:/usr/obj/usr/src/sys/GENERIC amd64 TB --- 2013-10-19 01:35:56 - starting HEAD tinderbox run for ia64/ia64 TB --- 2013-10-19 01:35:56 - cleaning the object tree TB --- 2013-10-19 01:35:56 - /usr/local/bin/svn stat /src TB --- 2013-10-19 01:35:59 - At svn revision 256751 TB --- 2013-10-19 01:36:00 - building world TB --- 2013-10-19 01:36:00 - CROSS_BUILD_TESTING=YES TB --- 2013-10-19 01:36:00 - MAKEOBJDIRPREFIX=/obj TB --- 2013-10-19 01:36:00 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2013-10-19 01:36:00 - SRCCONF=/dev/null TB --- 2013-10-19 01:36:00 - TARGET=ia64 TB --- 2013-10-19 01:36:00 - TARGET_ARCH=ia64 TB --- 2013-10-19 01:36:00 - TZ=UTC TB --- 2013-10-19 01:36:00 - __MAKE_CONF=/dev/null TB --- 2013-10-19 01:36:00 - cd /src TB --- 2013-10-19 01:36:00 - /usr/bin/make -B buildworld >>> Building an up-to-date make(1) >>> World build started on Sat Oct 19 01:36:07 UTC 2013 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools >>> stage 4.1: building includes >>> stage 4.2: building libraries >>> stage 4.3: make dependencies >>> stage 4.4: building everything >>> World build completed on Sat Oct 19 03:10:06 UTC 2013 TB --- 2013-10-19 03:10:06 - generating LINT kernel config TB --- 2013-10-19 03:10:06 - cd /src/sys/ia64/conf TB --- 2013-10-19 03:10:06 - /usr/bin/make -B LINT TB --- 2013-10-19 03:10:06 - cd /src/sys/ia64/conf TB --- 2013-10-19 03:10:06 - /usr/sbin/config -m LINT TB --- 2013-10-19 03:10:06 - building LINT kernel TB --- 2013-10-19 03:10:06 - CROSS_BUILD_TESTING=YES TB --- 2013-10-19 03:10:06 - MAKEOBJDIRPREFIX=/obj TB --- 2013-10-19 03:10:06 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2013-10-19 03:10:06 - SRCCONF=/dev/null TB --- 2013-10-19 03:10:06 - TARGET=ia64 TB --- 2013-10-19 03:10:06 - TARGET_ARCH=ia64 TB --- 2013-10-19 03:10:06 - TZ=UTC TB --- 2013-10-19 03:10:06 - __MAKE_CONF=/dev/null TB --- 2013-10-19 03:10:06 - cd /src TB --- 2013-10-19 03:10:06 - /usr/bin/make -B buildkernel KERNCONF=LINT >>> Kernel build for LINT started on Sat Oct 19 03:10:06 UTC 2013 >>> stage 1: configuring the kernel >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3.1: making dependencies >>> stage 3.2: building everything [...] cc -c -O2 -pipe -fno-strict-aliasing -std=c99 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs -fdiagnostics-show-option -nostdinc -I. -I/src/sys -I/src/sys/contrib/altq -I/src/sys/contrib/ia64/libuwx/src -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-builtin -mconstant-gp -ffixed-r13 -mfixed-range=f32-f127 -fpic -ffreestanding -Werror /src/sys/geom/geom_vfs.c cc -c -O2 -pipe -fno-strict-aliasing -std=c99 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs -fdiagnostics-show-option -nostdinc -I. -I/src/sys -I/src/sys/contrib/altq -I/src/sys/contrib/ia64/libuwx/src -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-builtin -mconstant-gp -ffixed-r13 -mfixed-range=f32-f127 -fpic -ffreestanding -Werror /src/sys/geom/geom_vol_ffs.c cc -c -O2 -pipe -fno-strict-aliasing -std=c99 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs -fdiagnostics-show-option -nostdinc -I. -I/src/sys -I/src/sys/contrib/altq -I/src/sys/contrib/ia64/libuwx/src -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-builtin -mconstant-gp -ffixed-r13 -mfixed-range=f32-f127 -fpic -ffreestanding -Werror /src/sys/geom/journal/g_journal.c cc -c -O2 -pipe -fno-strict-aliasing -std=c99 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs -fdiagnostics-show-option -nostdinc -I. -I/src/sys -I/src/sys/contrib/altq -I/src/sys/contrib/ia64/libuwx/src -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-builtin -mconstant-gp -ffixed-r13 -mfixed-range=f32-f127 -fpic -ffreestanding -Werror /src/sys/geom/journal/g_journal_ufs.c cc -c -O2 -pipe -fno-strict-aliasing -std=c99 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs -fdiagnostics-show-option -nostdinc -I. -I/src/sys -I/src/sys/contrib/altq -I/src/sys/contrib/ia64/libuwx/src -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-builtin -mconstant-gp -ffixed-r13 -mfixed-range=f32-f127 -fpic -ffreestanding -Werror /src/sys/geom/label/g_label.c cc1: warnings being treated as errors /src/sys/geom/label/g_label.c: In function 'g_label_resize': /src/sys/geom/label/g_label.c:135: warning: null format string [-Wformat] *** Error code 1 Stop. bmake[1]: stopped in /obj/ia64.ia64/src/sys/LINT *** Error code 1 Stop. bmake: stopped in /src *** Error code 1 Stop in /src. TB --- 2013-10-19 03:23:20 - WARNING: /usr/bin/make returned exit code 1 TB --- 2013-10-19 03:23:20 - ERROR: failed to build LINT kernel TB --- 2013-10-19 03:23:20 - 5194.45 user 885.11 system 6443.45 real http://tinderbox.freebsd.org/tinderbox-head-build-HEAD-ia64-ia64.full From owner-freebsd-ia64@FreeBSD.ORG Sat Oct 19 15:10:44 2013 Return-Path: Delivered-To: ia64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E99ADA38; Sat, 19 Oct 2013 15:10:43 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-current.sentex.ca (freebsd-current.sentex.ca [64.7.128.98]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AF21C2EF3; Sat, 19 Oct 2013 15:10:43 +0000 (UTC) Received: from freebsd-current.sentex.ca (localhost [127.0.0.1]) by freebsd-current.sentex.ca (8.14.5/8.14.5) with ESMTP id r9JFAg6X011197; Sat, 19 Oct 2013 11:10:42 -0400 (EDT) (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-current.sentex.ca (8.14.5/8.14.5/Submit) id r9JFAgHI011188; Sat, 19 Oct 2013 15:10:42 GMT (envelope-from tinderbox@freebsd.org) Date: Sat, 19 Oct 2013 15:10:42 GMT Message-Id: <201310191510.r9JFAgHI011188@freebsd-current.sentex.ca> X-Authentication-Warning: freebsd-current.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Subject: [head tinderbox] failure on ia64/ia64 Precedence: bulk X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.14 List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Oct 2013 15:10:44 -0000 TB --- 2013-10-19 13:21:21 - tinderbox 2.20 running on freebsd-current.sentex.ca TB --- 2013-10-19 13:21:21 - FreeBSD freebsd-current.sentex.ca 8.3-PRERELEASE FreeBSD 8.3-PRERELEASE #0: Mon Mar 26 13:54:12 EDT 2012 des@freebsd-current.sentex.ca:/usr/obj/usr/src/sys/GENERIC amd64 TB --- 2013-10-19 13:21:21 - starting HEAD tinderbox run for ia64/ia64 TB --- 2013-10-19 13:21:21 - cleaning the object tree TB --- 2013-10-19 13:22:28 - /usr/local/bin/svn stat /src TB --- 2013-10-19 13:22:31 - At svn revision 256765 TB --- 2013-10-19 13:22:32 - building world TB --- 2013-10-19 13:22:32 - CROSS_BUILD_TESTING=YES TB --- 2013-10-19 13:22:32 - MAKEOBJDIRPREFIX=/obj TB --- 2013-10-19 13:22:32 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2013-10-19 13:22:32 - SRCCONF=/dev/null TB --- 2013-10-19 13:22:32 - TARGET=ia64 TB --- 2013-10-19 13:22:32 - TARGET_ARCH=ia64 TB --- 2013-10-19 13:22:32 - TZ=UTC TB --- 2013-10-19 13:22:32 - __MAKE_CONF=/dev/null TB --- 2013-10-19 13:22:32 - cd /src TB --- 2013-10-19 13:22:32 - /usr/bin/make -B buildworld >>> Building an up-to-date make(1) >>> World build started on Sat Oct 19 13:22:39 UTC 2013 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools >>> stage 4.1: building includes >>> stage 4.2: building libraries >>> stage 4.3: make dependencies >>> stage 4.4: building everything >>> World build completed on Sat Oct 19 14:57:26 UTC 2013 TB --- 2013-10-19 14:57:26 - generating LINT kernel config TB --- 2013-10-19 14:57:26 - cd /src/sys/ia64/conf TB --- 2013-10-19 14:57:26 - /usr/bin/make -B LINT TB --- 2013-10-19 14:57:26 - cd /src/sys/ia64/conf TB --- 2013-10-19 14:57:26 - /usr/sbin/config -m LINT TB --- 2013-10-19 14:57:26 - building LINT kernel TB --- 2013-10-19 14:57:26 - CROSS_BUILD_TESTING=YES TB --- 2013-10-19 14:57:26 - MAKEOBJDIRPREFIX=/obj TB --- 2013-10-19 14:57:26 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2013-10-19 14:57:26 - SRCCONF=/dev/null TB --- 2013-10-19 14:57:26 - TARGET=ia64 TB --- 2013-10-19 14:57:26 - TARGET_ARCH=ia64 TB --- 2013-10-19 14:57:26 - TZ=UTC TB --- 2013-10-19 14:57:26 - __MAKE_CONF=/dev/null TB --- 2013-10-19 14:57:26 - cd /src TB --- 2013-10-19 14:57:26 - /usr/bin/make -B buildkernel KERNCONF=LINT >>> Kernel build for LINT started on Sat Oct 19 14:57:26 UTC 2013 >>> stage 1: configuring the kernel >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3.1: making dependencies >>> stage 3.2: building everything [...] cc -c -O2 -pipe -fno-strict-aliasing -std=c99 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs -fdiagnostics-show-option -nostdinc -I. -I/src/sys -I/src/sys/contrib/altq -I/src/sys/contrib/ia64/libuwx/src -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-builtin -mconstant-gp -ffixed-r13 -mfixed-range=f32-f127 -fpic -ffreestanding -Werror /src/sys/geom/geom_vfs.c cc -c -O2 -pipe -fno-strict-aliasing -std=c99 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs -fdiagnostics-show-option -nostdinc -I. -I/src/sys -I/src/sys/contrib/altq -I/src/sys/contrib/ia64/libuwx/src -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-builtin -mconstant-gp -ffixed-r13 -mfixed-range=f32-f127 -fpic -ffreestanding -Werror /src/sys/geom/geom_vol_ffs.c cc -c -O2 -pipe -fno-strict-aliasing -std=c99 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs -fdiagnostics-show-option -nostdinc -I. -I/src/sys -I/src/sys/contrib/altq -I/src/sys/contrib/ia64/libuwx/src -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-builtin -mconstant-gp -ffixed-r13 -mfixed-range=f32-f127 -fpic -ffreestanding -Werror /src/sys/geom/journal/g_journal.c cc -c -O2 -pipe -fno-strict-aliasing -std=c99 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs -fdiagnostics-show-option -nostdinc -I. -I/src/sys -I/src/sys/contrib/altq -I/src/sys/contrib/ia64/libuwx/src -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-builtin -mconstant-gp -ffixed-r13 -mfixed-range=f32-f127 -fpic -ffreestanding -Werror /src/sys/geom/journal/g_journal_ufs.c cc -c -O2 -pipe -fno-strict-aliasing -std=c99 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs -fdiagnostics-show-option -nostdinc -I. -I/src/sys -I/src/sys/contrib/altq -I/src/sys/contrib/ia64/libuwx/src -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-builtin -mconstant-gp -ffixed-r13 -mfixed-range=f32-f127 -fpic -ffreestanding -Werror /src/sys/geom/label/g_label.c cc1: warnings being treated as errors /src/sys/geom/label/g_label.c: In function 'g_label_resize': /src/sys/geom/label/g_label.c:135: warning: null format string [-Wformat] *** Error code 1 Stop. bmake[1]: stopped in /obj/ia64.ia64/src/sys/LINT *** Error code 1 Stop. bmake: stopped in /src *** Error code 1 Stop in /src. TB --- 2013-10-19 15:10:42 - WARNING: /usr/bin/make returned exit code 1 TB --- 2013-10-19 15:10:42 - ERROR: failed to build LINT kernel TB --- 2013-10-19 15:10:42 - 5192.27 user 891.86 system 6560.48 real http://tinderbox.freebsd.org/tinderbox-head-build-HEAD-ia64-ia64.full