From owner-freebsd-bugs@FreeBSD.ORG Wed Oct 23 09:10:01 2013 Return-Path: Delivered-To: freebsd-bugs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C9BBACEC for ; Wed, 23 Oct 2013 09:10:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9DBC62E8B for ; Wed, 23 Oct 2013 09:10:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r9N9A0SR080762 for ; Wed, 23 Oct 2013 09:10:00 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r9N9A0aS080761; Wed, 23 Oct 2013 09:10:00 GMT (envelope-from gnats) Resent-Date: Wed, 23 Oct 2013 09:10:00 GMT Resent-Message-Id: <201310230910.r9N9A0aS080761@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Anton Shterenlikht Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 54F38B0F for ; Wed, 23 Oct 2013 09:03:59 +0000 (UTC) (envelope-from mexas@bristol.ac.uk) Received: from eu1sys200aog101.obsmtp.com (eu1sys200aog101.obsmtp.com [207.126.144.111]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D96552E31 for ; Wed, 23 Oct 2013 09:03:57 +0000 (UTC) Received: from mail-wi0-f171.google.com ([209.85.212.171]) (using TLSv1) by eu1sys200aob101.postini.com ([207.126.147.11]) with SMTP ID DSNKUmeQ4qQG1tVNR6ALRQ9DI2IWXQHJKPCh@postini.com; Wed, 23 Oct 2013 09:03:57 UTC Received: by mail-wi0-f171.google.com with SMTP id h11so6976185wiv.4 for ; Wed, 23 Oct 2013 02:03:30 -0700 (PDT) Received: from mech-cluster241.men.bris.ac.uk (mech-cluster241.men.bris.ac.uk. [137.222.187.241]) by mx.google.com with ESMTPSA id s4sm14732627wiy.1.2013.10.23.02.03.27 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Oct 2013 02:03:28 -0700 (PDT) Received: from mech-cluster241.men.bris.ac.uk (localhost [127.0.0.1]) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6) with ESMTP id r9N93Q8U078489 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 23 Oct 2013 10:03:26 +0100 (BST) (envelope-from mexas@mech-cluster241.men.bris.ac.uk) Received: (from mexas@localhost) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6/Submit) id r9N93QPm078488; Wed, 23 Oct 2013 10:03:26 +0100 (BST) (envelope-from mexas) Message-Id: <201310230903.r9N93QPm078488@mech-cluster241.men.bris.ac.uk> Date: Wed, 23 Oct 2013 10:03:26 +0100 (BST) From: Anton Shterenlikht Sender: Anton Shterenlikht To: FreeBSD-gnats-submit@freebsd.org X-Send-Pr-Version: 3.114 Subject: kern/183227: panic: uma_zfree: Freeing to non free bucket index. X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Anton Shterenlikht List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Oct 2013 09:10:01 -0000 >Number: 183227 >Category: kern >Synopsis: panic: uma_zfree: Freeing to non free bucket index. >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Wed Oct 23 09:10:00 UTC 2013 >Closed-Date: >Last-Modified: >Originator: Anton Shterenlikht >Release: FreeBSD 11.0-CURRENT ia64 >Organization: University of Bristol >Environment: FreeBSD mech-as221.men.bris.ac.uk 10.0-CURRENT FreeBSD 10.0-CURRENT #12 r255488M: Tue Oct 22 15:42:23 BST 2013 root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI ia64 >Description: I managed to obtain a texdump with # ddb scripts lockinfo=show locks; show alllocks; show lockedvnods kdb.enter.panic=textdump set; capture on; run lockinfo; show pcpu; bt; ps; alltrace; capture off; textdump dump So below are: panic.txt, version.txt, config.txt, ddb.txt, msgbuf.txt # panic.txt # uma_zfree: Freeing to non free bucket index. # version.txt # FreeBSD 10.0-CURRENT #12 r255488M: Tue Oct 22 15:42:23 BST 2013 root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI # config.txt options CONFIG_AUTOGENERATED ident UZI machine ia64 cpu ITANIUM2 makeoptions MODULES_OVERRIDE=geom/geom_part opensolaris zfs acl_nfs4 acl_posix1e makeoptions DEBUG=-g options USB_DEBUG options TEXTDUMP_VERBOSE options TEXTDUMP_PREFERRED options DIAGNOSTIC options DEBUG_VFS_LOCKS options DEBUG_LOCKS options MALLOC_DEBUG_MAXZONES=8 options WITNESS_SKIPSPIN options WITNESS options INVARIANT_SUPPORT options INVARIANTS options DEADLKRES options GDB options DDB options KDB_TRACE options KDB options _KPOSIX_PRIORITY_SCHEDULING options QUOTA options UFS_GJOURNAL options UFS_DIRHASH options UFS_ACL options SYSVSHM options SYSVSEM options SYSVMSG options STACK options SOFTUPDATES options SMP options SCTP options SCSI_DELAY=5000 options SCHED_ULE options PSEUDOFS options PROCFS options PROCDESC options PRINTF_BUFR_SIZE=128 options PREEMPTION options P1003_1B_SEMAPHORES options NFS_ROOT options NFSD options NFSLOCKD options NFSCL options MSDOSFS options MD_ROOT options MAC options KTRACE options INET6 options INET options INCLUDE_CONFIG_FILE options GEOM_LABEL options FFS options TMPFS options NULLFS options FDESCFS options COMPAT_FREEBSD7 options CD9660 options CAPABILITIES options CAPABILITY_MODE options AUDIT options NEW_PCIB options GEOM_PART_MBR options GEOM_PART_GPT options GEOM_PART_BSD device acpi device io device mem device uart_ns8250 device miibus device pci device scbus device usb device mpt device cd device da device pass device ehci device ohci device uhci device umass device bge device fxp device ether device faith device gif device loop device md device puc device random device tun device uart device vlan device firmware # ddb.txt # db:0:kdb.enter.panic> run lockinfo db:1:lockinfo> show locks shared lockmgr zfs (zfs) r = 0 (0xe000000012d87658) locked @ /usr/src/sys/kern/vfs_subr.c:2099 db:1:locks> show alllocks Process 4293 (svn) thread 0xe00000001291ed80 (100363) shared lockmgr zfs (zfs) r = 0 (0xe000000012d87658) locked @ /usr/src/sys/kern/vfs_subr.c:2099 db:1:alllocks> show lockedvnods Locked vnodes db:0:kdb.enter.panic> show pcpu cpuid = 1 dynamic pcpu = 0x400005fc2ad00 curthread = 0xe00000001291ed80: pid 4293 "svn" curpcb = 0 fpcurthread = none idlethread = 0xe000000010b92480: tid 100004 "idle: cpu1" MD: vhpt = 0xe000004040600000 MD: lid = 0x1000000 MD: clock = 0x1f81ddc9eb9c MD: clock_mode = 2 MD: clock_load = 0 MD: stats = 0xe000000010b6ea30 MD: pmap = 0xe000000010b9f9b8 spin locks held: db:0:kdb.enter.panic> bt Tracing pid 4293 tid 100363 td 0xe00000001291ed80 kdb_enter(0x9ffc000000af8d60, 0x9ffc000000af8d60, 0x9ffc0000004573f0, 0x40c) at kdb_enter+0x92 vpanic(0x9ffc000000b33428, 0xa00000009026ae88, 0x100, 0x9ffc000000bfb3c8) at vpanic+0x2b0 kassert_panic(0x9ffc000000b33428, 0x38, 0xe00000001e7eff90, 0xe0000040ffbdfb00) at kassert_panic+0x120 uma_zfree_arg(0xe0000040ffbdc380, 0xe00000001e7efac0, 0xe00000001e7eff90) at uma_zfree_arg+0x2d0 free(0xe00000001e7efac0, 0xa0000000016eda30) at free+0x170 zfs_kmem_free(0xe00000001e7efac0, 0xa00000000155f270) at zfs_kmem_free+0x30 mze_destroy(0xe00000002d458a58, 0xa00000000155fd20) at mze_destroy+0x70 zap_evict(0xe00000002fd2d110, 0xe00000002d458a00, 0xa000000001445940) at zap_evict+0x70 dbuf_evict_user(0xe00000002fd2d110, 0xe00000002fd2d250, 0xe00000002fd2d248) at dbuf_evict_user+0x130 dbuf_clear(0xe00000002fd2d110, 0xe00000002fd2d188, 0x0, 0xe00000002fcfe490, 0xe000000012dc62a0, 0xa00000000144f830) at dbuf_clear+0xc0 dbuf_hold_impl(0xe000000012dc9540, 0x0, 0x0, 0x0, 0x0, 0xa00000009026aee0) at dbuf_hold_impl+0x1d0 dbuf_hold(0xe000000012dc9540, 0x0, 0x0, 0xa000000001463fa0) at dbuf_hold+0x40 dmu_buf_hold(0xe0000000124d8800, 0xa, 0x0, 0x0, 0xa00000009026af00, 0x1) at dmu_buf_hold+0x120 zap_lockdir(0xe0000000124d8800, 0x9, 0x0, 0x1, 0x1, 0x0, 0xa00000009026af40) at zap_lockdir+0x50 zap_lookup_norm(0xe0000000124d8800, 0x9, 0xa00000009026afe0, 0x8, 0x1, 0xa00000009026af88, 0x0, 0x0) at zap_lookup_norm+0x60 zap_lookup(0xe0000000124d8800, 0x9, 0xa00000009026afe0, 0x8, 0x1, 0xa00000009026af88, 0xa0000000015b5800, 0xb1e) at zap_lookup+0x60 zfs_dirent_lock(0xa00000009026afa0, 0xe000000012699140, 0xa00000009026afe0, 0xa00000009026afa8, 0x6, 0x0, 0x0, 0xe000000021c2d880) at zfs_dirent_lock+0xbb0 zfs_dirlook(0xe000000012699140, 0xa00000009026afe0, 0xa00000009026b220, 0x0, 0x0, 0x0) at zfs_dirlook+0x240 zfs_lookup(0xe000000012d875f0, 0xa00000009026afe0, 0xa00000009026b220, 0xa00000009026b248, 0x0, 0xe000000010b80f00, 0xe00000001291ed80, 0x0) at zfs_lookup+0x7a0 zfs_freebsd_lookup(0xa00000009026b0e0, 0xa00000009026b248, 0xa00000009026b280) at zfs_freebsd_lookup+0x100 VOP_CACHEDLOOKUP_APV(0xa0000000016bda00, 0xa00000009026b0e0, 0x0, 0x9ffc000000b41b68, 0x9ffc00000058c860) at VOP_CACHEDLOOKUP_APV+0x270 vfs_cache_lookup(0xe000000012d875f0, 0x0, 0xa00000009026b248) at vfs_cache_lookup+0x1f0 VOP_LOOKUP_APV(0xa0000000016bda00, 0xa00000009026b118, 0x0, 0x9ffc000000b41bc8) at VOP_LOOKUP_APV+0x270 lookup(0xa00000009026b1c0, 0x200000, 0xe000000012d875f0, 0x0, 0xa00000009026b250) at lookup+0xe20 namei(0xa00000009026b1c0, 0xe00000001291a000, 0x4000144, 0x0, 0xa00000009026b230, 0xa00000009026b278, 0xa00000009026b270, 0xa00000009026b220) at namei+0xb50 kern_statat_vnhook(0xe00000001291ed80, 0x0, 0xffffffffffffff9c, 0x121d2f760, 0x0, 0xa00000009026b320, 0x0) at kern_statat_vnhook+0xf0 kern_statat(0xe00000001291ed80, 0x0, 0xffffffffffffff9c, 0x121d2f760, 0x0, 0xa00000009026b320) at kern_statat+0x40 kern_stat(0xe00000001291ed80, 0x121d2f760, 0x0, 0xa00000009026b320) at kern_stat+0x40 sys_stat(0xe00000001291ed80, 0xa00000009026b4e8, 0x9ffc000000986c80, 0x48d) at sys_stat+0x30 syscall(0xe00000001291a000, 0x7fffffffffffe1f0, 0x1209936e0, 0xe00000001291ed80, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return db:0:kdb.enter.panic> ps pid ppid pgrp uid state wmesg wchan cmd 4293 4265 4265 0 R+ CPU 1 svn 4265 4204 4265 0 S+ wait 0xe0000000122a3720 sh 4205 1004 4205 0 S+ ttyin 0xe000000010c348a8 sh 4204 4203 4204 0 S+ wait 0xe000000012224000 sh 4203 4187 4203 1001 S+ wait 0xe0000000122b6de0 su 4187 4186 4187 1001 Ss+ pause 0xe000000012a4f7c8 tcsh 4186 4184 4184 1001 S select 0xe0000000122eaf40 sshd 4184 935 4184 0 Ss select 0xe0000000122eae40 sshd 1004 1 1004 0 Ss+ wait 0xe000000012a6c4a0 login 1003 1 1 0 S ttydcd 0xe000000010c34ce8 getty 982 1 982 0 Ss select 0xe0000000122eb0c0 inetd 945 1 945 0 Ss nanslp 0x9ffc000000c23588 cron 941 1 941 25 Ss pause 0xe0000000122a29e8 sendmail 938 1 938 0 Ss select 0xe000000010831540 sendmail 935 1 935 0 Ss select 0xe0000000122eb6c0 sshd 876 1 876 0 Ss select 0xe0000000108316c0 ntpd 795 1 795 0 Ss select 0xe0000000122eb740 syslogd 657 1 657 0 Ss select 0xe000000010831740 devd 156 1 156 0 Ss pause 0xe000000012224548 adjkerntz 40 0 0 0 DL (threaded) [zfskern] 100348 D zio->io_ 0xe00000001245e6e8 [txg_thread_enter] 100347 D tx->tx_q 0xe000000012475a38 [txg_thread_enter] 100345 D spa->spa 0xe0000000124049f8 [trim tank] 100068 D l2arc_fe 0xa0000000016d1ec0 [l2arc_feed_thread] 100067 D arc_recl 0xa0000000016c20a0 [arc_reclaim_thread] 21 0 0 0 DL sdflush 0x9ffc000000bfc808 [softdepflush] 20 0 0 0 DL vlruwt 0xe000000012224de0 [vnlru] 19 0 0 0 DL syncer 0x9ffc000000e18e90 [syncer] 18 0 0 0 DL psleep 0x9ffc000000bfbba0 [bufdaemon] 17 0 0 0 DL pgzero 0x9ffc000000bfcbac [pagezero] 9 0 0 0 DL psleep 0x9ffc000000bfcb14 [vmdaemon] 8 0 0 0 DL psleep 0x9ffc000000bfcb44 [pagedaemon] 7 0 0 0 DL ccb_scan 0x9ffc000000bfcea0 [xpt_thrd] 4 0 0 0 DL waiting_ 0x9ffc000000e1b530 [sctp_iterator] 3 0 0 0 DL idle 0xa000000000a12000 [mpt_recovery1] 2 0 0 0 DL idle 0xa0000000009fa000 [mpt_recovery0] 16 0 0 0 DL (threaded) [usb] 100039 D - 0xa0000000009f6e18 [usbus2] 100038 D - 0xa0000000009f6dc0 [usbus2] 100037 D - 0xa0000000009f6d68 [usbus2] 100036 D - 0xa0000000009f6d10 [usbus2] 100034 D - 0xa0000000009ed460 [usbus1] 100033 D - 0xa0000000009ed408 [usbus1] 100032 D - 0xa0000000009ed3b0 [usbus1] 100031 D - 0xa0000000009ed358 [usbus1] 100029 D - 0xa0000000009e9460 [usbus0] 100028 D - 0xa0000000009e9408 [usbus0] 100027 D - 0xa0000000009e93b0 [usbus0] 100026 D - 0xa0000000009e9358 [usbus0] 15 0 0 0 DL tzpoll 0x9ffc000000bfa5c8 [acpi_thermal] 14 0 0 0 DL - 0x9ffc000000bfa840 [rand_harvestq] 13 0 0 0 DL (threaded) [geom] 100011 D - 0x9ffc000000bfad68 [g_down] 100010 D - 0x9ffc000000bfad60 [g_up] 100009 D - 0x9ffc000000bfad50 [g_event] 12 0 0 0 WL (threaded) [intr] 100046 I [swi0: uart uart+++] 100045 I [irq29: bge0] 100043 I [irq28: mpt1] 100041 I [irq27: mpt0] 100040 I [irq20: fxp0] 100035 I [irq18: ehci0] 100030 I [irq17: ohci1] 100025 I [irq16: ohci0] 100021 I [swi6: task queue] 100020 I [swi6: Giant taskq] 100018 I [swi5: fast taskq] 100017 I [swi2: cambio] 100008 I [swi4: clock] 100007 I [swi4: clock] 100006 I [swi3: vm] 100005 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100004 CanRun [idle: cpu1] 100003 Run CPU 0 [idle: cpu0] 1 0 1 0 SLs wait 0xe000000010b90de0 [init] 10 0 0 0 DL audit_wo 0x9ffc000000e1fbc8 [audit] 0 0 0 0 DLs (threaded) [kernel] 100380 D - 0xe000000012685600 [zil_clean] 100379 D - 0xe000000012686800 [zil_clean] 100356 D - 0xe0000000122f2700 [zil_clean] 100355 D - 0xe000000012686a00 [zil_clean] 100354 D - 0xe0000000122f2b00 [zil_clean] 100353 D - 0xe0000000125ed900 [zil_clean] 100352 D - 0xe000000012686d00 [zil_clean] 100346 D - 0xe0000000122f9400 [zfs_vn_rele_taskq] 100344 D - 0xe0000000122f8600 [zio_ioctl_intr] 100343 D - 0xe000000011c3d700 [zio_ioctl_issue] 100342 D - 0xe0000000122f8900 [zio_claim_intr] 100341 D - 0xe0000000122f8800 [zio_claim_issue] 100340 D - 0xe0000000122f9700 [zio_free_intr] 100339 D - 0xe0000000122f9200 [zio_free_issue_99] 100338 D - 0xe0000000122f9200 [zio_free_issue_98] 100337 D - 0xe0000000122f9200 [zio_free_issue_97] 100336 D - 0xe0000000122f9200 [zio_free_issue_96] 100335 D - 0xe0000000122f9200 [zio_free_issue_95] 100334 D - 0xe0000000122f9200 [zio_free_issue_94] 100333 D - 0xe0000000122f9200 [zio_free_issue_93] 100332 D - 0xe0000000122f9200 [zio_free_issue_92] 100331 D - 0xe0000000122f9200 [zio_free_issue_91] 100330 D - 0xe0000000122f9200 [zio_free_issue_90] 100329 D - 0xe0000000122f9200 [zio_free_issue_89] 100328 D - 0xe0000000122f9200 [zio_free_issue_88] 100327 D - 0xe0000000122f9200 [zio_free_issue_87] 100326 D - 0xe0000000122f9200 [zio_free_issue_86] 100325 D - 0xe0000000122f9200 [zio_free_issue_85] 100324 D - 0xe0000000122f9200 [zio_free_issue_84] 100323 D - 0xe0000000122f9200 [zio_free_issue_83] 100322 D - 0xe0000000122f9200 [zio_free_issue_82] 100321 D - 0xe0000000122f9200 [zio_free_issue_81] 100320 D - 0xe0000000122f9200 [zio_free_issue_80] 100319 D - 0xe0000000122f9200 [zio_free_issue_79] 100318 D - 0xe0000000122f9200 [zio_free_issue_78] 100317 D - 0xe0000000122f9200 [zio_free_issue_77] 100316 D - 0xe0000000122f9200 [zio_free_issue_76] 100315 D - 0xe0000000122f9200 [zio_free_issue_75] 100314 D - 0xe0000000122f9200 [zio_free_issue_74] 100313 D - 0xe0000000122f9200 [zio_free_issue_73] 100312 D - 0xe0000000122f9200 [zio_free_issue_72] 100311 D - 0xe0000000122f9200 [zio_free_issue_71] 100310 D - 0xe0000000122f9200 [zio_free_issue_70] 100309 D - 0xe0000000122f9200 [zio_free_issue_69] 100308 D - 0xe0000000122f9200 [zio_free_issue_68] 100307 D - 0xe0000000122f9200 [zio_free_issue_67] 100306 D - 0xe0000000122f9200 [zio_free_issue_66] 100305 D - 0xe0000000122f9200 [zio_free_issue_65] 100304 D - 0xe0000000122f9200 [zio_free_issue_64] 100303 D - 0xe0000000122f9200 [zio_free_issue_63] 100302 D - 0xe0000000122f9200 [zio_free_issue_62] 100301 D - 0xe0000000122f9200 [zio_free_issue_61] 100300 D - 0xe0000000122f9200 [zio_free_issue_60] 100299 D - 0xe0000000122f9200 [zio_free_issue_59] 100298 D - 0xe0000000122f9200 [zio_free_issue_58] 100297 D - 0xe0000000122f9200 [zio_free_issue_57] 100296 D - 0xe0000000122f9200 [zio_free_issue_56] 100295 D - 0xe0000000122f9200 [zio_free_issue_55] 100294 D - 0xe0000000122f9200 [zio_free_issue_54] 100293 D - 0xe0000000122f9200 [zio_free_issue_53] 100292 D - 0xe0000000122f9200 [zio_free_issue_52] 100291 D - 0xe0000000122f9200 [zio_free_issue_51] 100290 D - 0xe0000000122f9200 [zio_free_issue_50] 100289 D - 0xe0000000122f9200 [zio_free_issue_49] 100288 D - 0xe0000000122f9200 [zio_free_issue_48] 100287 D - 0xe0000000122f9200 [zio_free_issue_47] 100286 D - 0xe0000000122f9200 [zio_free_issue_46] 100285 D - 0xe0000000122f9200 [zio_free_issue_45] 100284 D - 0xe0000000122f9200 [zio_free_issue_44] 100283 D - 0xe0000000122f9200 [zio_free_issue_43] 100282 D - 0xe0000000122f9200 [zio_free_issue_42] 100281 D - 0xe0000000122f9200 [zio_free_issue_41] 100280 D - 0xe0000000122f9200 [zio_free_issue_40] 100279 D - 0xe0000000122f9200 [zio_free_issue_39] 100278 D - 0xe0000000122f9200 [zio_free_issue_38] 100277 D - 0xe0000000122f9200 [zio_free_issue_37] 100276 D - 0xe0000000122f9200 [zio_free_issue_36] 100275 D - 0xe0000000122f9200 [zio_free_issue_35] 100274 D - 0xe0000000122f9200 [zio_free_issue_34] 100273 D - 0xe0000000122f9200 [zio_free_issue_33] 100272 D - 0xe0000000122f9200 [zio_free_issue_32] 100271 D - 0xe0000000122f9200 [zio_free_issue_31] 100270 D - 0xe0000000122f9200 [zio_free_issue_30] 100269 D - 0xe0000000122f9200 [zio_free_issue_29] 100268 D - 0xe0000000122f9200 [zio_free_issue_28] 100267 D - 0xe0000000122f9200 [zio_free_issue_27] 100266 D - 0xe0000000122f9200 [zio_free_issue_26] 100265 D - 0xe0000000122f9200 [zio_free_issue_25] 100264 D - 0xe0000000122f9200 [zio_free_issue_24] 100263 D - 0xe0000000122f9200 [zio_free_issue_23] 100262 D - 0xe0000000122f9200 [zio_free_issue_22] 100261 D - 0xe0000000122f9200 [zio_free_issue_21] 100260 D - 0xe0000000122f9200 [zio_free_issue_20] 100259 D - 0xe0000000122f9200 [zio_free_issue_19] 100258 D - 0xe0000000122f9200 [zio_free_issue_18] 100257 D - 0xe0000000122f9200 [zio_free_issue_17] 100256 D - 0xe0000000122f9200 [zio_free_issue_16] 100255 D - 0xe0000000122f9200 [zio_free_issue_15] 100254 D - 0xe0000000122f9200 [zio_free_issue_14] 100253 D - 0xe0000000122f9200 [zio_free_issue_13] 100252 D - 0xe0000000122f9200 [zio_free_issue_12] 100251 D - 0xe0000000122f9200 [zio_free_issue_11] 100250 D - 0xe0000000122f9200 [zio_free_issue_10] 100249 D - 0xe0000000122f9200 [zio_free_issue_9] 100248 D - 0xe0000000122f9200 [zio_free_issue_8] 100247 D - 0xe0000000122f9200 [zio_free_issue_7] 100246 D - 0xe0000000122f9200 [zio_free_issue_6] 100245 D - 0xe0000000122f9200 [zio_free_issue_5] 100244 D - 0xe0000000122f9200 [zio_free_issue_4] 100243 D - 0xe0000000122f9200 [zio_free_issue_3] 100242 D - 0xe0000000122f9200 [zio_free_issue_2] 100241 D - 0xe0000000122f9200 [zio_free_issue_1] 100240 D - 0xe0000000122f9200 [zio_free_issue_0] 100239 D - 0xe0000000122f9100 [zio_write_intr_high] 100238 D - 0xe0000000122f9100 [zio_write_intr_high] 100237 D - 0xe0000000122f9100 [zio_write_intr_high] 100236 D - 0xe0000000122f9100 [zio_write_intr_high] 100235 D - 0xe0000000122f9100 [zio_write_intr_high] 100234 D - 0xe0000000122f9000 [zio_write_intr_7] 100233 D - 0xe0000000122f9000 [zio_write_intr_6] 100232 D - 0xe0000000122f9000 [zio_write_intr_5] 100231 D - 0xe0000000122f9000 [zio_write_intr_4] 100230 D - 0xe0000000122f9000 [zio_write_intr_3] 100229 D - 0xe0000000122f9000 [zio_write_intr_2] 100228 D - 0xe0000000122f9000 [zio_write_intr_1] 100227 D - 0xe0000000122f9000 [zio_write_intr_0] 100226 D - 0xe0000000122f8f00 [zio_write_issue_hig] 100225 D - 0xe0000000122f8f00 [zio_write_issue_hig] 100224 D - 0xe0000000122f8f00 [zio_write_issue_hig] 100223 D - 0xe0000000122f8f00 [zio_write_issue_hig] 100222 D - 0xe0000000122f8f00 [zio_write_issue_hig] 100221 D - 0xe0000000122f8e00 [zio_write_issue_1] 100220 D - 0xe0000000122f8e00 [zio_write_issue_0] 100219 D - 0xe0000000122f8d00 [zio_read_intr_1] 100218 D - 0xe0000000122f8d00 [zio_read_intr_0] 100217 D - 0xe0000000122f8c00 [zio_read_issue_7] 100216 D - 0xe0000000122f8c00 [zio_read_issue_6] 100215 D - 0xe0000000122f8c00 [zio_read_issue_5] 100214 D - 0xe0000000122f8c00 [zio_read_issue_4] 100213 D - 0xe0000000122f8c00 [zio_read_issue_3] 100212 D - 0xe0000000122f8c00 [zio_read_issue_2] 100211 D - 0xe0000000122f8c00 [zio_read_issue_1] 100210 D - 0xe0000000122f8c00 [zio_read_issue_0] 100209 D - 0xe0000000122f8b00 [zio_null_intr] 100208 D - 0xe0000000122f8a00 [zio_null_issue] 100066 D - 0xe0000000122f9900 [system_taskq_1] 100065 D - 0xe0000000122f9900 [system_taskq_0] 100047 D - 0x9ffc000000c22320 [deadlkres] 100023 D - 0xe000000010bc7300 [kqueue taskq] 100022 D - 0xe000000010bc7400 [ffs_trim taskq] 100019 D - 0xe000000010bc7700 [thread taskq] 100016 D - 0xe000000010bc7a00 [acpi_task_2] 100015 D - 0xe000000010bc7a00 [acpi_task_1] 100014 D - 0xe000000010bc7a00 [acpi_task_0] 100012 D - 0xe000000010bc7d00 [firmware taskq] 100000 D swapin 0x9ffc000000c15ed8 [swapper] db:0:kdb.enter.panic> alltrace Tracing command svn pid 4293 tid 100363 td 0xe00000001291ed80 kdb_enter(0x9ffc000000af8d60, 0x9ffc000000af8d60, 0x9ffc0000004573f0, 0x40c) at kdb_enter+0x92 vpanic(0x9ffc000000b33428, 0xa00000009026ae88, 0x100, 0x9ffc000000bfb3c8) at vpanic+0x2b0 kassert_panic(0x9ffc000000b33428, 0x38, 0xe00000001e7eff90, 0xe0000040ffbdfb00) at kassert_panic+0x120 uma_zfree_arg(0xe0000040ffbdc380, 0xe00000001e7efac0, 0xe00000001e7eff90) at uma_zfree_arg+0x2d0 free(0xe00000001e7efac0, 0xa0000000016eda30) at free+0x170 zfs_kmem_free(0xe00000001e7efac0, 0xa00000000155f270) at zfs_kmem_free+0x30 mze_destroy(0xe00000002d458a58, 0xa00000000155fd20) at mze_destroy+0x70 zap_evict(0xe00000002fd2d110, 0xe00000002d458a00, 0xa000000001445940) at zap_evict+0x70 dbuf_evict_user(0xe00000002fd2d110, 0xe00000002fd2d250, 0xe00000002fd2d248) at dbuf_evict_user+0x130 dbuf_clear(0xe00000002fd2d110, 0xe00000002fd2d188, 0x0, 0xe00000002fcfe490, 0xe000000012dc62a0, 0xa00000000144f830) at dbuf_clear+0xc0 dbuf_hold_impl(0xe000000012dc9540, 0x0, 0x0, 0x0, 0x0, 0xa00000009026aee0) at dbuf_hold_impl+0x1d0 dbuf_hold(0xe000000012dc9540, 0x0, 0x0, 0xa000000001463fa0) at dbuf_hold+0x40 dmu_buf_hold(0xe0000000124d8800, 0xa, 0x0, 0x0, 0xa00000009026af00, 0x1) at dmu_buf_hold+0x120 zap_lockdir(0xe0000000124d8800, 0x9, 0x0, 0x1, 0x1, 0x0, 0xa00000009026af40) at zap_lockdir+0x50 zap_lookup_norm(0xe0000000124d8800, 0x9, 0xa00000009026afe0, 0x8, 0x1, 0xa00000009026af88, 0x0, 0x0) at zap_lookup_norm+0x60 zap_lookup(0xe0000000124d8800, 0x9, 0xa00000009026afe0, 0x8, 0x1, 0xa00000009026af88, 0xa0000000015b5800, 0xb1e) at zap_lookup+0x60 zfs_dirent_lock(0xa00000009026afa0, 0xe000000012699140, 0xa00000009026afe0, 0xa00000009026afa8, 0x6, 0x0, 0x0, 0xe000000021c2d880) at zfs_dirent_lock+0xbb0 zfs_dirlook(0xe000000012699140, 0xa00000009026afe0, 0xa00000009026b220, 0x0, 0x0, 0x0) at zfs_dirlook+0x240 zfs_lookup(0xe000000012d875f0, 0xa00000009026afe0, 0xa00000009026b220, 0xa00000009026b248, 0x0, 0xe000000010b80f00, 0xe00000001291ed80, 0x0) at zfs_lookup+0x7a0 zfs_freebsd_lookup(0xa00000009026b0e0, 0xa00000009026b248, 0xa00000009026b280) at zfs_freebsd_lookup+0x100 VOP_CACHEDLOOKUP_APV(0xa0000000016bda00, 0xa00000009026b0e0, 0x0, 0x9ffc000000b41b68, 0x9ffc00000058c860) at VOP_CACHEDLOOKUP_APV+0x270 vfs_cache_lookup(0xe000000012d875f0, 0x0, 0xa00000009026b248) at vfs_cache_lookup+0x1f0 VOP_LOOKUP_APV(0xa0000000016bda00, 0xa00000009026b118, 0x0, 0x9ffc000000b41bc8) at VOP_LOOKUP_APV+0x270 lookup(0xa00000009026b1c0, 0x200000, 0xe000000012d875f0, 0x0, 0xa00000009026b250) at lookup+0xe20 namei(0xa00000009026b1c0, 0xe00000001291a000, 0x4000144, 0x0, 0xa00000009026b230, 0xa00000009026b278, 0xa00000009026b270, 0xa00000009026b220) at namei+0xb50 kern_statat_vnhook(0xe00000001291ed80, 0x0, 0xffffffffffffff9c, 0x121d2f760, 0x0, 0xa00000009026b320, 0x0) at kern_statat_vnhook+0xf0 kern_statat(0xe00000001291ed80, 0x0, 0xffffffffffffff9c, 0x121d2f760, 0x0, 0xa00000009026b320) at kern_statat+0x40 kern_stat(0xe00000001291ed80, 0x121d2f760, 0x0, 0xa00000009026b320) at kern_stat+0x40 sys_stat(0xe00000001291ed80, 0xa00000009026b4e8, 0x9ffc000000986c80, 0x48d) at sys_stat+0x30 syscall(0xe00000001291a000, 0x7fffffffffffe1f0, 0x1209936e0, 0xe00000001291ed80, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sh pid 4265 tid 100060 td 0xe0000000122ac000 cpu_switch(0xe0000000122ac000, 0xe000000010b92900, 0x9ffc000000c738b8, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe0000000122ac000, 0xe000000010b92900, 0x9ffc000000c3d818, 0x9ffc000000c3d800) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000122ac000, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0xe0000000122a3720, 0x6c, 0xe0000000122ac378, 0xe0000000122ac000) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000000122a3720, 0x6c, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe0000000122a3720, 0x6c, 0x9ffc000000b2fb20, 0x100, 0x9ffc000000469800) at sleepq_wait_sig+0x30 _sleep(0xe0000000122a3720, 0xe0000000122a3820, 0x6c, 0x9ffc000000b2fb20, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0 kern_wait6(0xe0000000122ac000, 0x7, 0x0, 0xa00000008fe3b390, 0x30, 0x0, 0x0) at kern_wait6+0xba0 kern_wait(0xe0000000122ac000, 0xffffffff, 0xa00000008fe3b390, 0x0, 0x0) at kern_wait+0xe0 sys_wait4(0xe0000000122ac000, 0xa00000008fe3b4e8, 0xa00000008fe3b390, 0x9ffc000000986c80) at sys_wait4+0x70 syscall(0xe0000000122a3720, 0x7fffffffffffe790, 0x0, 0xe0000000122ac000, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sh pid 4205 tid 100371 td 0xe00000001270ad80 cpu_switch(0xe00000001270ad80, 0xe000000010b92480, 0x9ffc000000c75820, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe00000001270ad80, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001270ad80, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0xe000000010c348a8, 0x0, 0xe00000001270b0f8, 0xe00000001270ad80) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000010c348a8, 0x0, 0x9ffc000000b00730, 0x9ffc000000433850) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000010c348a8, 0x0, 0x9ffc0000003bfeb0, 0x611, 0x9ffc0000003bfef0) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe000000010c348a8, 0xe000000010c34808, 0xe00000001270ad80, 0x0) at _cv_wait_sig+0x4d0 tty_wait(0xe000000010c34800, 0xe000000010c348a8, 0x2, 0x9ffc000000b06ef8, 0x9ffc00000053a020) at tty_wait+0xb0 ttydisc_read(0xe000000010c34800, 0xa000000090293360, 0x0, 0x0) at ttydisc_read+0x4c0 ttydev_read(0xe000000010c34800, 0xa000000090293360, 0x0, 0x9ffc000000246cc0) at ttydev_read+0xe0 devfs_read_f(0xe000000012376cd0, 0xa000000090293360, 0xe000000010b82700, 0x0, 0xe00000001270ad80) at devfs_read_f+0xf0 dofileread(0xe00000001270ad80, 0x0, 0xe000000012376cd0, 0xa000000090293360, 0x0, 0x0) at dofileread+0x150 kern_readv(0xe00000001270ad80, 0x0, 0xa000000090293360) at kern_readv+0xa0 sys_read(0xe00000001270ad80, 0xa0000000902934e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100 syscall(0xe000000012a4e000, 0x7fffffffffffec50, 0x1, 0xe00000001270ad80, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sh pid 4204 tid 100050 td 0xe000000011972d80 cpu_switch(0xe000000011972d80, 0xe000000010b92480, 0x9ffc000000c73f20, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe000000011972d80, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011972d80, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0xe000000012224000, 0x6c, 0xe0000000119730f8, 0xe000000011972d80) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000012224000, 0x6c, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000012224000, 0x6c, 0x9ffc000000b2fb20, 0x100, 0x9ffc000000469800) at sleepq_wait_sig+0x30 _sleep(0xe000000012224000, 0xe000000012224100, 0x6c, 0x9ffc000000b2fb20, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0 kern_wait6(0xe000000011972d80, 0x7, 0x0, 0xa00000008fdeb390, 0x36, 0x0, 0x0) at kern_wait6+0xba0 kern_wait(0xe000000011972d80, 0xffffffff, 0xa00000008fdeb390, 0x6, 0x0) at kern_wait+0xe0 sys_wait4(0xe000000011972d80, 0xa00000008fdeb4e8, 0xa00000008fdeb390, 0x9ffc000000986c80) at sys_wait4+0x70 syscall(0xe000000012224000, 0x7fffffffffffe8b0, 0x6, 0xe000000011972d80, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command su pid 4203 tid 100062 td 0xe0000000122adb00 cpu_switch(0xe0000000122adb00, 0xe000000010b92480, 0x9ffc000000c74b28, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe0000000122adb00, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000122adb00, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0xe0000000122b6de0, 0x6c, 0xe0000000122ade78, 0xe0000000122adb00) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000000122b6de0, 0x6c, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe0000000122b6de0, 0x6c, 0x9ffc000000b2fb20, 0x100, 0x9ffc000000469800) at sleepq_wait_sig+0x30 _sleep(0xe0000000122b6de0, 0xe0000000122b6ee0, 0x6c, 0x9ffc000000b2fb20, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0 kern_wait6(0xe0000000122adb00, 0x0, 0x106c, 0xa00000008fe4b390, 0x32, 0x0, 0x0) at kern_wait6+0xba0 kern_wait(0xe0000000122adb00, 0x106c, 0xa00000008fe4b390, 0x2, 0x0) at kern_wait+0xe0 sys_wait4(0xe0000000122adb00, 0xa00000008fe4b4e8, 0xa00000008fe4b390, 0x9ffc000000986c80) at sys_wait4+0x70 syscall(0xe0000000122b6de0, 0x7fffffffffffe4c0, 0x2, 0xe0000000122adb00, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command tcsh pid 4187 tid 100375 td 0xe00000001298c480 cpu_switch(0xe00000001298c480, 0xe000000010b92480, 0x9ffc000000c73ef8, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe00000001298c480, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001298c480, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0xe000000012a4f7c8, 0x74, 0xe00000001298c7f8, 0xe00000001298c480) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000012a4f7c8, 0x74, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000012a4f7c8, 0x74, 0x9ffc000000ac89c8, 0x100, 0x9ffc000000469800) at sleepq_wait_sig+0x30 _sleep(0xe000000012a4f7c8, 0xe000000012a4f820, 0x74, 0x9ffc000000ac89c8, 0x0, 0x0, 0x100) at _sleep+0x7b0 kern_sigsuspend(0xe00000001298c480, 0x0, 0xe000000012a4f7c8) at kern_sigsuspend+0x140 sys_sigsuspend(0xe00000001298c480, 0xa0000000902b34e8, 0x9ffc000000986c80, 0x48d) at sys_sigsuspend+0x60 syscall(0xe000000012a4f720, 0x0, 0x10004f5d0, 0xe00000001298c480, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sshd pid 4186 tid 100370 td 0xe00000001270b200 cpu_switch(0xe00000001270b200, 0xe000000010b92480, 0x9ffc000000c75a78, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe00000001270b200, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001270b200, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0xe0000000122eaf40, 0x0, 0xe00000001270b578, 0xe00000001270b200) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000000122eaf40, 0x0, 0x9ffc000000b00730, 0xe0000000122eaf20) at sleepq_catch_signals+0x190 sleepq_timedwait_sig(0xe0000000122eaf40, 0x0, 0x9ffc0000003bf0b0, 0x9ffc0000003bf0f0, 0x713) at sleepq_timedwait_sig+0x30 _cv_timedwait_sig_sbt(0xe0000000122eaf40, 0xe0000000122eaf20, 0x5a62b60182b9, 0x12c0000000, 0x200) at _cv_timedwait_sig_sbt+0x4f0 seltdwait(0xe0000000122eaf00, 0x5a62b60182b9, 0x12c0000000, 0xe0000000122eaf38, 0x9ffc000000b03f48, 0x9ffc00000050f230) at seltdwait+0xf0 kern_select(0xe00000001270b200, 0x0, 0x121416228, 0x121416230, 0x0, 0x5a62b60182b9, 0xa) at kern_select+0xb40 sys_select(0xe00000001270b200, 0xa00000009028b4e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0 syscall(0xe000000012a4e4a0, 0x121416228, 0x121416230, 0xe00000001270b200, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sshd pid 4184 tid 100061 td 0xe00000001233a000 cpu_switch(0xe00000001233a000, 0xe000000010b92480, 0x9ffc000000c75a50, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe00000001233a000, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001233a000, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0xe0000000122eae40, 0x0, 0xe00000001233a378, 0xe00000001233a000) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000000122eae40, 0x0, 0x9ffc000000b00730, 0x9ffc000000433850) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe0000000122eae40, 0x0, 0x9ffc0000003bfeb0, 0x611, 0x9ffc0000003bfef0) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe0000000122eae40, 0xe0000000122eae20, 0xe00000001233a000, 0x0, 0x9ffc000000c188c0) at _cv_wait_sig+0x4d0 seltdwait(0xe0000000122eae00, 0xffffffffffffffff, 0x0, 0xe0000000122eae38) at seltdwait+0x110 sys_poll(0xe00000001233a000, 0xa00000008fe434e8, 0xe000000012963170, 0xa00000008fe43290) at sys_poll+0x620 syscall(0xe0000000122b7280, 0x1, 0xffffffffffffffff, 0xe00000001233a000, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command login pid 1004 tid 100373 td 0xe00000001298c900 cpu_switch(0xe00000001298c900, 0xe000000010b92480, 0x9ffc000000c744c0, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe00000001298c900, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001298c900, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0xe000000012a6c4a0, 0x6c, 0xe00000001298cc78, 0xe00000001298c900) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000012a6c4a0, 0x6c, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000012a6c4a0, 0x6c, 0x9ffc000000b2fb20, 0x100, 0x9ffc000000469800) at sleepq_wait_sig+0x30 _sleep(0xe000000012a6c4a0, 0xe000000012a6c5a0, 0x6c, 0x9ffc000000b2fb20, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0 kern_wait6(0xe00000001298c900, 0x0, 0x106d, 0xa0000000902a3390, 0x30, 0x0, 0x0) at kern_wait6+0xba0 kern_wait(0xe00000001298c900, 0x106d, 0xa0000000902a3390, 0x0, 0x0) at kern_wait+0xe0 sys_wait4(0xe00000001298c900, 0xa0000000902a34e8, 0xa0000000902a3390, 0x9ffc000000986c80) at sys_wait4+0x70 syscall(0xe000000012a6c4a0, 0x7fffffffffffece0, 0x0, 0xe00000001298c900, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command getty pid 1003 tid 100358 td 0xe000000012531b00 cpu_switch(0xe000000012531b00, 0xe000000010b92480, 0x9ffc000000c74ec0, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe000000012531b00, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012531b00, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0xe000000010c34ce8, 0x0, 0xe000000012531e78, 0xe000000012531b00) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000010c34ce8, 0x0, 0x9ffc000000b00730, 0x9ffc000000433850) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000010c34ce8, 0x0, 0x9ffc0000003bfeb0, 0x611, 0x9ffc0000003bfef0) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe000000010c34ce8, 0xe000000010c34c08, 0xe000000012531b00, 0x0) at _cv_wait_sig+0x4d0 tty_wait(0xe000000010c34c00, 0xe000000010c34ce8, 0x1, 0x9ffc000000b06ef8, 0x9ffc00000052f250, 0x694, 0xe00000001235d800) at tty_wait+0xb0 ttydev_open(0xe000000010bb2200, 0x9ffc000000b06f30, 0x2000, 0x9ffc000000b07018, 0xe000000010c34c00) at ttydev_open+0x710 devfs_open(0xa000000090243158, 0x0, 0xa000000090243118) at devfs_open+0x2c0 VOP_OPEN_APV(0x9ffc000000b866e0, 0xa000000090243158, 0x0, 0x9ffc000000b41b10) at VOP_OPEN_APV+0x270 vn_open_vnode(0xe000000012a3a000, 0x3, 0xe0000000122e4200, 0xe000000012531b00, 0xe0000000123536d0) at vn_open_vnode+0x2c0 vn_open_cred(0xa000000090243288, 0xa000000090243390, 0x980, 0x0, 0xe0000000122e4200, 0xe0000000123536d0) at vn_open_cred+0x520 vn_open(0xa000000090243288, 0xa000000090243390, 0x980, 0xe0000000123536d0, 0x9ffc0000005c80d0, 0x695, 0x0) at vn_open+0x50 kern_openat(0xe000000012531b00, 0xffffffffffffff9c, 0x10001ca90, 0x0, 0x0, 0x7980) at kern_openat+0x490 kern_open(0xe000000012531b00, 0x10001ca90, 0x0, 0x2, 0x7980) at kern_open+0x40 sys_open(0xe000000012531b00, 0xa0000000902434e8, 0x9ffc000000986c80, 0x48d) at sys_open+0x40 syscall(0xe0000000126cc000, 0x2, 0x100007980, 0xe000000012531b00, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command inetd pid 982 tid 100372 td 0xe000000012a68480 cpu_switch(0xe000000012a68480, 0xe00000001291ed80, 0x9ffc000000c746a0, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe000000012a68480, 0xe00000001291ed80, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012a68480, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0xe0000000122eb0c0, 0x0, 0xe000000012a687f8, 0xe000000012a68480) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000000122eb0c0, 0x0, 0x9ffc000000b00730, 0x9ffc000000433850) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe0000000122eb0c0, 0x0, 0x9ffc0000003bfeb0, 0x611, 0x9ffc0000003bfef0) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe0000000122eb0c0, 0xe0000000122eb0a0, 0xe000000012a68480, 0x0, 0x9ffc000000c188c0) at _cv_wait_sig+0x4d0 seltdwait(0xe0000000122eb080, 0xffffffffffffffff, 0x0, 0xe0000000122eb0b8, 0x9ffc000000b03f48, 0x9ffc00000050f230) at seltdwait+0x110 kern_select(0xe000000012a68480, 0x0, 0x7fffffffffffe0e0, 0x0, 0x0, 0xffffffffffffffff, 0x6) at kern_select+0xb40 sys_select(0xe000000012a68480, 0xa00000009029b4e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0 syscall(0xe00000001291b720, 0x7fffffffffffe0e0, 0x0, 0xe000000012a68480, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command cron pid 945 tid 100359 td 0xe00000001291e000 cpu_switch(0xe00000001291e000, 0xe000000010b92900, 0x9ffc000000c752a8, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe00000001291e000, 0xe000000010b92900, 0x9ffc000000c3d818, 0x9ffc000000c3d800) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001291e000, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0x9ffc000000c23588, 0x6c, 0xe00000001291e378, 0xe00000001291e000) at sleepq_switch+0x2e0 sleepq_catch_signals(0x9ffc000000c23588, 0x6c, 0x9ffc000000b00730, 0xe00000001291e390) at sleepq_catch_signals+0x190 sleepq_timedwait_sig(0x9ffc000000c23588, 0x6c, 0x3bff63d09, 0x9ffc0000004697a0, 0x998) at sleepq_timedwait_sig+0x30 _sleep(0x9ffc000000c23588, 0x0, 0x6c, 0x9ffc000000afa9f0, 0x1, 0x3bff63d09, 0x200) at _sleep+0x750 kern_nanosleep(0xe00000001291e000, 0x3bff63d096, 0xa00000009024b380) at kern_nanosleep+0x2a0 sys_nanosleep(0xe00000001291e000, 0xa00000009024b4e8, 0x0, 0x9ffc000000986c80) at sys_nanosleep+0xb0 syscall(0xe0000000122a2de0, 0x7fffffffffffec28, 0x0, 0xe00000001291e000, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sendmail pid 941 tid 100365 td 0xe00000001270bb00 cpu_switch(0xe00000001270bb00, 0xe000000010b92480, 0x9ffc000000c75348, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe00000001270bb00, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001270bb00, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0xe0000000122a29e8, 0x74, 0xe00000001270be78, 0xe00000001270bb00) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000000122a29e8, 0x74, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe0000000122a29e8, 0x74, 0x9ffc000000ac89c8, 0x100, 0x9ffc000000469800) at sleepq_wait_sig+0x30 _sleep(0xe0000000122a29e8, 0xe0000000122a2a40, 0x74, 0x9ffc000000ac89c8, 0x0, 0x0, 0x100) at _sleep+0x7b0 kern_sigsuspend(0xe00000001270bb00, 0x0, 0xe0000000122a29e8) at kern_sigsuspend+0x140 sys_sigsuspend(0xe00000001270bb00, 0xa00000009027b4e8, 0x9ffc000000986c80, 0x48d) at sys_sigsuspend+0x60 syscall(0xe0000000122a2940, 0xe00000001270bb08, 0x9ffc000000983510, 0xe00000001270bb00, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sendmail pid 938 tid 100357 td 0xe00000001270a000 cpu_switch(0xe00000001270a000, 0xe000000010b92480, 0x9ffc000000c74268, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe00000001270a000, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001270a000, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0xe000000010831540, 0x0, 0xe00000001270a378, 0xe00000001270a000) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000010831540, 0x0, 0x9ffc000000b00730, 0xe000000010831520) at sleepq_catch_signals+0x190 sleepq_timedwait_sig(0xe000000010831540, 0x0, 0x9ffc0000003bf0b0, 0x9ffc0000003bf0f0, 0x713) at sleepq_timedwait_sig+0x30 _cv_timedwait_sig_sbt(0xe000000010831540, 0xe000000010831520, 0x59e8045668c1, 0x50000000, 0x200) at _cv_timedwait_sig_sbt+0x4f0 seltdwait(0xe000000010831500, 0x59e8045668c1, 0x50000000, 0xe000000010831538, 0x9ffc000000b03f48, 0x9ffc00000050f230) at seltdwait+0xf0 kern_select(0xe00000001270a000, 0x0, 0x7fffffffffffc3c8, 0x0, 0x0, 0x59e8045668c1, 0x4) at kern_select+0xb40 sys_select(0xe00000001270a000, 0xa00000009023b4e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0 syscall(0xe0000000126cc4a0, 0x7fffffffffffc3c8, 0x0, 0xe00000001270a000, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sshd pid 935 tid 100063 td 0xe0000000122ad680 cpu_switch(0xe0000000122ad680, 0xe000000010b92480, 0x9ffc000000c74790, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe0000000122ad680, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000122ad680, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0xe0000000122eb6c0, 0x0, 0xe0000000122ad9f8, 0xe0000000122ad680) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000000122eb6c0, 0x0, 0x9ffc000000b00730, 0x9ffc000000433850) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe0000000122eb6c0, 0x0, 0x9ffc0000003bfeb0, 0x611, 0x9ffc0000003bfef0) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe0000000122eb6c0, 0xe0000000122eb6a0, 0xe0000000122ad680, 0x0, 0x9ffc000000c188c0) at _cv_wait_sig+0x4d0 seltdwait(0xe0000000122eb680, 0xffffffffffffffff, 0x0, 0xe0000000122eb6b8, 0x9ffc000000b03f48, 0x9ffc00000050f230) at seltdwait+0x110 kern_select(0xe0000000122ad680, 0x0, 0x121416198, 0x0, 0x0, 0xffffffffffffffff, 0x5) at kern_select+0xb40 sys_select(0xe0000000122ad680, 0xa00000008fe534e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0 syscall(0xe0000000122b6940, 0x121416198, 0x0, 0xe0000000122ad680, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command ntpd pid 876 tid 100362 td 0xe00000001291f200 cpu_switch(0xe00000001291f200, 0xe000000010b92480, 0x9ffc000000c75690, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe00000001291f200, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001291f200, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0xe0000000108316c0, 0x0, 0xe00000001291f578, 0xe00000001291f200) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000000108316c0, 0x0, 0x9ffc000000b00730, 0x9ffc000000433850) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe0000000108316c0, 0x0, 0x9ffc0000003bfeb0, 0x611, 0x9ffc0000003bfef0) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe0000000108316c0, 0xe0000000108316a0, 0xe00000001291f200, 0x0, 0x9ffc000000c188c0) at _cv_wait_sig+0x4d0 seltdwait(0xe000000010831680, 0xffffffffffffffff, 0x0, 0xe0000000108316b8, 0x9ffc000000b03f48, 0x9ffc00000050f230) at seltdwait+0x110 kern_select(0xe00000001291f200, 0x0, 0x7fffffffffffec88, 0x0, 0x0, 0xffffffffffffffff, 0x1e) at kern_select+0xb40 sys_select(0xe00000001291f200, 0xa0000000902634e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0 syscall(0xe00000001291a4a0, 0x7fffffffffffec88, 0x0, 0xe00000001291f200, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command syslogd pid 795 tid 100064 td 0xe000000011c49b00 cpu_switch(0xe000000011c49b00, 0xe000000010b92900, 0x9ffc000000c75bb8, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe000000011c49b00, 0xe000000010b92900, 0x9ffc000000c3d818, 0x9ffc000000c3d800) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011c49b00, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0xe0000000122eb740, 0x0, 0xe000000011c49e78, 0xe000000011c49b00) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000000122eb740, 0x0, 0x9ffc000000b00730, 0x9ffc000000433850) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe0000000122eb740, 0x0, 0x9ffc0000003bfeb0, 0x611, 0x9ffc0000003bfef0) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe0000000122eb740, 0xe0000000122eb720, 0xe000000011c49b00, 0x0, 0x9ffc000000c188c0) at _cv_wait_sig+0x4d0 seltdwait(0xe0000000122eb700, 0xffffffffffffffff, 0x0, 0xe0000000122eb738, 0x9ffc000000b03f48, 0x9ffc00000050f230) at seltdwait+0x110 kern_select(0xe000000011c49b00, 0x0, 0x120866168, 0x0, 0x0, 0xffffffffffffffff, 0x9) at kern_select+0xb40 sys_select(0xe000000011c49b00, 0xa00000008fe5b4e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0 syscall(0xe0000000122a3280, 0x120866168, 0x0, 0xe000000011c49b00, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command devd pid 657 tid 100349 td 0xe00000001245c000 cpu_switch(0xe00000001245c000, 0xe000000010b92480, 0x9ffc000000c742b8, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe00000001245c000, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001245c000, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0xe000000010831740, 0x0, 0xe00000001245c378, 0xe00000001245c000) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000010831740, 0x0, 0x9ffc000000b00730, 0xe000000010831720) at sleepq_catch_signals+0x190 sleepq_timedwait_sig(0xe000000010831740, 0x0, 0x9ffc0000003bf0b0, 0x9ffc0000003bf0f0, 0x713) at sleepq_timedwait_sig+0x30 _cv_timedwait_sig_sbt(0xe000000010831740, 0xe000000010831720, 0x59f0a1c6c699, 0x3c0000000, 0x200) at _cv_timedwait_sig_sbt+0x4f0 seltdwait(0xe000000010831700, 0x59f0a1c6c699, 0x3c0000000, 0xe000000010831738, 0x9ffc000000b03f48, 0x9ffc00000050f230) at seltdwait+0xf0 kern_select(0xe00000001245c000, 0x0, 0x7fffffffffffcd00, 0x0, 0x0, 0x59f0a1c6c699, 0x5) at kern_select+0xb40 sys_select(0xe00000001245c000, 0xa0000000901fb4e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0 syscall(0xe0000000126cd280, 0x7fffffffffffcd00, 0x0, 0xe00000001245c000, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command adjkerntz pid 156 tid 100049 td 0xe000000011973200 cpu_switch(0xe000000011973200, 0xe000000010b92480, 0x9ffc000000c73728, 0x9ffc0000004b0d70) at cpu_switch+0xd0 sched_switch(0xe000000011973200, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000011973200, 0x9ffc0000004e9a60) at mi_switch+0x3f0 sleepq_switch(0xe000000012224548, 0x74, 0xe000000011973578, 0xe000000011973200) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000012224548, 0x74, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190 sle # msbuf.txt # GDB: debug ports: uart GDB: current port: uart KDB: debugger backends: ddb gdb KDB: current backend: ddb Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #12 r255488M: Tue Oct 22 15:42:23 BST 2013 root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI ia64 gcc version 4.2.1 20070831 patched [FreeBSD] WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: Madison (1500 MHz Itanium 2) Origin = "GenuineIntel" Revision = 5 Features = 0x1 real memory = 6442450944 (6144 MB) avail memory = 6286475264 (5995 MB) FPSWA Revision = 0x10012, Entry = 0xe0000040ffe60050 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0: ACPI Id=0, SAPIC Id=0, SAPIC Eid=0 (BSP) cpu1: ACPI Id=1, SAPIC Id=1, SAPIC Eid=0 random: initialized Event timer "ITC" frequency 1500000000 Hz quality 1000 ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20130823/tbfadt-603) ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20130823/tbfadt-603) acpi0: on motherboard acpi0: Power Button (fixed) acpi0: Sleep Button (fixed) cpu0: on acpi0 cpu1: on acpi0 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <32-bit timer at 3.579545MHz> iomem 0xff5c1004-0xff5c1007 on acpi0 acpi_tz0: on acpi0 pcib0: on acpi0 pcib0: could not evaluate _ADR - AE_NOT_FOUND pci0: on pcib0 ohci0: mem 0x80023000-0x80023fff irq 16 at device 1.0 on pci0 usbus0 on ohci0 ohci1: mem 0x80022000-0x80022fff irq 17 at device 1.1 on pci0 usbus1 on ohci1 ehci0: mem 0x80021000-0x800210ff irq 18 at device 1.2 on pci0 usbus2: EHCI version 0.95 usbus2 on ehci0 pci0: at device 2.0 (no driver attached) fxp0: port 0xd00-0xd3f mem 0x80020000-0x80020fff,0x80000000-0x8001ffff irq 20 at device 3.0 on pci0 miibus0: on fxp0 inphy0: PHY 1 on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow fxp0: Ethernet address: 00:11:0a:31:e6:0a pcib1: on acpi0 pcib1: could not evaluate _ADR - AE_NOT_FOUND pci32: on pcib1 mpt0: port 0x2100-0x21ff mem 0x90840000-0x9084ffff,0x90830000-0x9083ffff irq 27 at device 1.0 on pci32 mpt0: MPI Version=1.2.12.0 mpt1: port 0x2000-0x20ff mem 0x90820000-0x9082ffff,0x90810000-0x9081ffff irq 28 at device 1.1 on pci32 mpt1: MPI Version=1.2.12.0 bge0: mem 0x90800000-0x9080ffff irq 29 at device 2.0 on pci32 bge0: CHIP ID 0x00000105; ASIC REV 0x00; CHIP REV 0x01; PCI on PCI-X 66 MHz; 64bit miibus1: on bge0 brgphy0: PHY 1 on miibus1 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bge0: Ethernet address: 00:11:0a:31:46:06 pcib2: on acpi0 pcib2: could not evaluate _ADR - AE_NOT_FOUND pci64: on pcib2 pcib3: on acpi0 pcib3: could not evaluate _ADR - AE_NOT_FOUND pci96: on pcib3 pcib4: on acpi0 pcib4: could not evaluate _ADR - AE_NOT_FOUND pci128: on pcib4 pcib5: on acpi0 pcib5: could not evaluate _ADR - AE_NOT_FOUND pci192: on pcib5 pcib6: on acpi0 pcib6: could not evaluate _ADR - AE_NOT_FOUND pci224: on pcib6 uart0: mem 0xf4051000-0xf405100f irq 82 at device 1.0 on pci224 puc0: mem 0xf4050000-0xf4050fff,0xf4020000-0xf403ffff irq 82 at device 1.1 on pci224 uart1: at port 1 on puc0 uart1: console (9600,n,8,1) uart2: at port 3 on puc0 vgapci0: port 0xe000-0xe0ff mem 0xf0000000-0xf3ffffff,0xf4040000-0xf404ffff at device 2.0 on pci224 uart3: <16550 or compatible> iomem 0xff5e0000-0xff5e0007 irq 34 on acpi0 uart4: <16550 or compatible> iomem 0xff5e2000-0xff5e2007 irq 35 on acpi0 uart4: debug port (9600,n,8,1) Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 480Mbps High Speed USB v2.0 ugen1.1: at usbus1 ugen2.1: at usbus2 uhub0: on usbus1 uhub1: on usbus2 ugen0.1: at usbus0 uhub2: on usbus0 uhub0: 2 ports with 2 removable, self powered uhub2: 3 ports with 3 removable, self powered uhub1: 5 ports with 5 removable, self powered WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. da2 at mpt1 bus 0 scbus1 target 2 lun 0 da2: Fixed Direct Access SCSI-3 device da2: 320.000MB/s transfers (160.000MHz, offset 63, 16bit) da2: Command Queueing enabled da2: 70007MB (143374650 512 byte sectors: 255H 63S/T 8924C) da0 at mpt0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-3 device da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da0: Command Queueing enabled da0: 70007MB (143374738 512 byte sectors: 255H 63S/T 8924C) da1 at mpt0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SCSI-3 device da1: 320.000MB/s transfers (160.000MHz, offset 63, 16bit) da1: Command Queueing enabled da1: 70007MB (143374650 512 byte sectors: 255H 63S/T 8924C) Trying to mount root from ufs:/dev/da0p2 [rw]... WARNING: / was not properly dismounted <118>Setting hostuuid: a90e62bf-ce84-11e1-9fc5-00110a31e60a. <118>Setting hostid: 0x688171f3. ZFS filesystem version: 5 ZFS storage pool version: features support (5000) <118>Starting ddb. <118>Entropy harvesting: interrupts ethernet point_to_point kickstart. <118>Starting file system checks: <118>/dev/da0p2: UNREF FILE I=6288403 OWNER=root MODE=100600 <118>/dev/da0p2: SIZE=174 MTIME=Oct 22 20:59 2013 (CLEARED) <118>/dev/da0p2: UNREF FILE I=6288425 OWNER=root MODE=100600 <118>/dev/da0p2: SIZE=1290 MTIME=Oct 23 00:30 2013 (CLEARED) <118>/dev/da0p2: UNREF FILE I=6288428 OWNER=root MODE=100600 <118>/dev/da0p2: SIZE=444 MTIME=Oct 23 02:58 2013 (CLEARED) <118>/dev/da0p2: UNREF FILE I=6288429 OWNER=root MODE=100600 <118>/dev/da0p2: SIZE=1886 MTIME=Oct 23 02:41 2013 (CLEARED) <118>/dev/da0p2: UNREF FILE I=6288430 OWNER=root MODE=100600 <118>/dev/da0p2: SIZE=283 MTIME=Oct 23 02:41 2013 (CLEARED) <118>/dev/da0p2: UNREF FILE I=6288431 OWNER=root MODE=100600 <118>/dev/da0p2: SIZE=288 MTIME=Oct 23 02:43 2013 (CLEARED) <118>/dev/da0p2: UNREF FILE I=6288432 OWNER=root MODE=100600 <118>/dev/da0p2: SIZE=2052 MTIME=Oct 23 02:43 2013 (CLEARED) <118>/dev/da0p2: UNREF FILE I=6288433 OWNER=root MODE=100600 <118>/dev/da0p2: SIZE=468 MTIME=Oct 23 03:03 2013 (CLEARED) <118>/dev/da0p2: UNREF FILE I=6288436 OWNER=root MODE=100600 <118>/dev/da0p2: SIZE=0 MTIME=Oct 23 03:03 2013 (CLEARED) <118>/dev/da0p2: UNREF FILE I=6288438 OWNER=root MODE=100600 <118>/dev/da0p2: SIZE=192 MTIME=Oct 23 03:03 2013 (CLEARED) <118>/dev/da0p2: UNREF FILE I=6288443 OWNER=root MODE=100600 <118>/dev/da0p2: SIZE=0 MTIME=Oct 23 03:01 2013 (CLEARED) <118>/dev/da0p2: UNREF FILE I=6288445 OWNER=root MODE=100600 <118>/dev/da0p2: SIZE=0 MTIME=Oct 23 03:01 2013 (CLEARED) <118>/dev/da0p2: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) <118>/dev/da0p2: SUMMARY INFORMATION BAD (SALVAGED) <118>/dev/da0p2: BLK(S) MISSING IN BIT MAPS (SALVAGED) <118>/dev/da0p2: 672211 files, 6449503 used, 24016134 free (101838 frags, 2989287 blocks, 0.3% fragmentation) <118>Mounting local file systems:. <118>Writing entropy file:. <118>Setting hostname: mech-as221.men.bris.ac.uk. <118>Starting Network: lo0 fxp0 bge0. <118>lo0: flags=8049 metric 0 mtu 16384 <118> options=600003 <118> inet6 ::1 prefixlen 128 <118> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 <118> inet 127.0.0.1 netmask 0xff000000 <118> nd6 options=21 <118>fxp0: flags=8843 metric 0 mtu 1500 <118> options=4219b <118> ether 00:11:0a:31:e6:0a <118> inet 10.10.10.13 netmask 0xffffff00 broadcast 10.10.10.255 <118> inet6 fe80::211:aff:fe31:e60a%fxp0 prefixlen 64 scopeid 0x1 <118> nd6 options=29 <118> media: Ethernet autoselect (100baseTX ) <118> status: active <118>bge0: flags=8843 metric 0 mtu 1500 <118> options=8009b <118> ether 00:11:0a:31:46:06 <118> inet 137.222.187.221 netmask 0xffffff00 broadcast 137.222.187.255 <118> inet6 fe80::211:aff:fe31:4606%bge0 prefixlen 64 scopeid 0x2 <118> nd6 options=29 <118> media: Ethernet autoselect (none) <118> status: no carrier <118>Starting devd. <118>add net default: gateway 137.222.187.250 <118>add net fe80::: gateway ::1 <118>add net ff02::: gateway ::1 <118>add net ::ffff:0.0.0.0: gateway ::1 <118>add net ::0.0.0.0: gateway ::1 <118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/gcc46 /usr/local/lib/gcc48 /usr/local/lib/graphviz <118>Creating and/or trimming log files. <118>Starting syslogd. <118>savecore: reboot after panic: deadlkres: possible deadlock detected for 0xe000000012aed200, blocked for 900014 ticks <118>Oct 23 03:21:29 mech-as221 savecore: reboot after panic: deadlkres: possible deadlock detected for 0xe000000012aed200, blocked for 900014 ticks <118>savecore: writing core to /var/crash/textdump.tar.4 <118>/var/crash/vmcore.4 not found <118>Clearing /tmp (X related). <118>Recovering vi editor sessions:. <118>Updating motd:. <118>Mounting late file systems:. <118>Starting ntpd. <118>Starting bsdstats. <118>Posting monthly OS statistics to rpt.bsdstats.org <118>Performing sanity check on sshd configuration. <118>Starting sshd. <118>Starting cron. <118>Local package initialization:cd: /var/portbuild/: No such file or directory <118>. <118>Starting inetd. <118>Starting background file system checks in 60 seconds. <118> <118>Wed Oct 23 03:21:34 BST 2013 Expensive timeout(9) function: 0x9ffc000000b4e130(0) 0.030631053 s Expensive timeout(9) function: 0x9ffc000000b4e130(0) 0.092184233 s <118>Oct 23 09:22:05 mech-as221 su: BAD SU mexas to root on /dev/pts/0 <118>Oct 23 09:22:13 mech-as221 su: mexas to root on /dev/pts/0 <118>Oct 23 09:22:26 mech-as221 login: ROOT LOGIN (root) ON ttyu1 panic: uma_zfree: Freeing to non free bucket index. cpuid = 1 KDB: stack backtrace: db_trace_self(0x9ffc0000000c9ec0) at db_trace_self+0x40 db_trace_self_wrapper(0x9ffc0000004d0260) at db_trace_self_wrapper+0x70 kdb_backtrace(0x9ffc000000bfb620, 0x9ffc0000004573a0, 0x40c, 0x9ffc000000dd2220) at kdb_backtrace+0xc0 vpanic(0x9ffc000000b33428, 0xa00000009026ae88, 0x9ffc000000c21820, 0x9ffc000000bfb3c8) at vpanic+0x260 kassert_panic(0x9ffc000000b33428, 0x38, 0xe00000001e7eff90, 0xe0000040ffbdfb00) at kassert_panic+0x120 uma_zfree_arg(0xe0000040ffbdc380, 0xe00000001e7efac0, 0xe00000001e7eff90) at uma_zfree_arg+0x2d0 free(0xe00000001e7efac0, 0xa0000000016eda30) at free+0x170 zfs_kmem_free(0xe00000001e7efac0, 0xa00000000155f270) at zfs_kmem_free+0x30 mze_destroy(0xe00000002d458a58, 0xa00000000155fd20) at mze_destroy+0x70 zap_evict(0xe00000002fd2d110, 0xe00000002d458a00, 0xa000000001445940) at zap_evict+0x70 dbuf_evict_user(0xe00000002fd2d110, 0xe00000002fd2d250, 0xe00000002fd2d248) at dbuf_evict_user+0x130 dbuf_clear(0xe00000002fd2d110, 0xe00000002fd2d188, 0x0, 0xe00000002fcfe490, 0xe000000012dc62a0, 0xa00000000144f830) at dbuf_clear+0xc0 dbuf_hold_impl(0xe000000012dc9540, 0x0, 0x0, 0x0, 0x0, 0xa00000009026aee0) at dbuf_hold_impl+0x1d0 dbuf_hold(0xe000000012dc9540, 0x0, 0x0, 0xa000000001463fa0) at dbuf_hold+0x40 dmu_buf_hold(0xe0000000124d8800, 0xa, 0x0, 0x0, 0xa00000009026af00, 0x1) at dmu_buf_hold+0x120 zap_lockdir(0xe0000000124d8800, 0x9, 0x0, 0x1, 0x1, 0x0, 0xa00000009026af40) at zap_lockdir+0x50 zap_lookup_norm(0xe0000000124d8800, 0x9, 0xa00000009026afe0, 0x8, 0x1, 0xa00000009026af88, 0x0, 0x0) at zap_lookup_norm+0x60 zap_lookup(0xe0000000124d8800, 0x9, 0xa00000009026afe0, 0x8, 0x1, 0xa00000009026af88, 0xa0000000015b5800, 0xb1e) at zap_lookup+0x60 zfs_dirent_lock(0xa00000009026afa0, 0xe000000012699140, 0xa00000009026afe0, 0xa00000009026afa8, 0x6, 0x0, 0x0, 0xe000000021c2d880) at zfs_dirent_lock+0xbb0 zfs_dirlook(0xe000000012699140, 0xa00000009026afe0, 0xa00000009026b220, 0x0, 0x0, 0x0) at zfs_dirlook+0x240 zfs_lookup(0xe000000012d875f0, 0xa00000009026afe0, 0xa00000009026b220, 0xa00000009026b248, 0x0, 0xe000000010b80f00, 0xe00000001291ed80, 0x0) at zfs_lookup+0x7a0 zfs_freebsd_lookup(0xa00000009026b0e0, 0xa00000009026b248, 0xa00000009026b280) at zfs_freebsd_lookup+0x100 VOP_CACHEDLOOKUP_APV(0xa0000000016bda00, 0xa00000009026b0e0, 0x0, 0x9ffc000000b41b68, 0x9ffc00000058c860) at VOP_CACHEDLOOKUP_APV+0x270 vfs_cache_lookup(0xe000000012d875f0, 0x0, 0xa00000009026b248) at vfs_cache_lookup+0x1f0 VOP_LOOKUP_APV(0xa0000000016bda00, 0xa00000009026b118, 0x0, 0x9ffc000000b41bc8) at VOP_LOOKUP_APV+0x270 lookup(0xa00000009026b1c0, 0x200000, 0xe000000012d875f0, 0x0, 0xa00000009026b250) at lookup+0xe20 namei(0xa00000009026b1c0, 0xe00000001291a000, 0x4000144, 0x0, 0xa00000009026b230, 0xa00000009026b278, 0xa00000009026b270, 0xa00000009026b220) at namei+0xb50 kern_statat_vnhook(0xe00000001291ed80, 0x0, 0xffffffffffffff9c, 0x121d2f760, 0x0, 0xa00000009026b320, 0x0) at kern_statat_vnhook+0xf0 kern_statat(0xe00000001291ed80, 0x0, 0xffffffffffffff9c, 0x121d2f760, 0x0, 0xa00000009026b320) at kern_statat+0x40 kern_stat(0xe00000001291ed80, 0x121d2f760, 0x0, 0xa00000009026b320) at kern_stat+0x40 sys_stat(0xe00000001291ed80, 0xa00000009026b4e8, 0x9ffc000000986c80, 0x48d) at sys_stat+0x30 syscall(0xe00000001291a000, 0x7fffffffffffe1f0, 0x1209936e0, 0xe00000001291ed80, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return KDB: enter: panic The only local modification is this patch, suggested by kib@ to collect more debug data: # svn diff /usr/src/ Index: /usr/src/sys/kern/uipc_syscalls.c =================================================================== --- /usr/src/sys/kern/uipc_syscalls.c (revision 255488) +++ /usr/src/sys/kern/uipc_syscalls.c (working copy) @@ -2079,7 +2079,7 @@ VM_OBJECT_WUNLOCK(obj); KASSERT(error != 0 || (m->wire_count > 0 && m->valid == VM_PAGE_BITS_ALL), - ("wrong page state m %p", m)); + ("wrong page state m %p off %#jx xfsize %d", m, off, xfsize)); return (error); } # >How-To-Repeat: The panic is triggered by poudriere ports -c or poudriere jails -c >Fix: >Release-Note: >Audit-Trail: >Unformatted: