Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 13 Sep 2010 02:07:19 -0400
From:      Rich <rincebrain@gmail.com>
To:        freebsd-fs <freebsd-fs@freebsd.org>
Subject:   ZFS v28: panic: zfs accessing past end of object
Message-ID:  <AANLkTikis8J_x84CESMB0zYk1eVka1uFXQMP3HS9TOgG@mail.gmail.com>

next in thread | raw e-mail | index | archive | help

[-- Attachment #1 --]
My experimental v28 server panicked today when I was testing it by
pushing a backup of my laptop to it over NFS with rsync. Dedup was
enabled on the pool, NFS sharing done via sharenfs property, nothing
else of interest that I'm aware of was on.

Running on r212074 with the original v28 patch, on an amd64 system.
I've attached core.txt.0, please let me know what else  I can provide.

- Rich

[-- Attachment #2 --]
xanadu dumped core - see /var/crash/vmcore.0

Mon Sep 13 02:05:29 UTC 2010

FreeBSD xanadu 9.0-CURRENT FreeBSD 9.0-CURRENT #0 r212074M: Sun Sep 12 18:48:36 UTC 2010     root@xanadu:/usr/obj/head_old/sys/DTRACE2  amd64

panic: Solaris(panic): zfs: accessing past end of object ffffff80627acd50/57 (size=1652215463 access=0+18446743525605952880)

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: Solaris(panic): zfs: accessing past end of object ffffff80627acd50/57 (size=1652215463 access=0+18446743525605952880)

cpuid = 0
KDB: enter: panic
panic: from debugger
cpuid = 0
Uptime: 5m51s
Physical memory: 3309 MB
Dumping 1336 MB: 1321 1305 1289 1273 1257 1241 1225 1209 1193 1177 1161 1145 1129 1113 1097 1081 1065 1049 1033 1017 1001 985 969 953 937 921 905 889 873 857 841 825 809 793 777 761 745 729 713 697 681 665 649 633 617 601 585 569 553 537 521 505 489 473 457 441 425 409 393 377 361 345 329 313 297 281 265 249 233 217 201 185 169 153 137 121 105 89 73 57 41 25 9

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
#0  doadump () at /head_old/sys/kern/kern_shutdown.c:248
248		if (textdump_pending)
(kgdb) #0  doadump () at /head_old/sys/kern/kern_shutdown.c:248
#1  0xffffffff805d83c7 in boot (howto=260)
    at /head_old/sys/kern/kern_shutdown.c:416
#2  0xffffffff805d883c in panic (fmt=Variable "fmt" is not available.
)
    at /head_old/sys/kern/kern_shutdown.c:590
#3  0xffffffff80202ae7 in db_panic (addr=Variable "addr" is not available.
)
    at /head_old/sys/ddb/db_command.c:478
#4  0xffffffff80202f91 in db_command (last_cmdp=0xffffffff80cdbf00, cmd_table=Variable "cmd_table" is not available.

) at /head_old/sys/ddb/db_command.c:445
#5  0xffffffff802031e0 in db_command_loop ()
    at /head_old/sys/ddb/db_command.c:498
#6  0xffffffff80205249 in db_trap (type=Variable "type" is not available.
) at /head_old/sys/ddb/db_main.c:229
#7  0xffffffff8060cbde in kdb_trap (type=3, code=0, tf=0xffffff80627acc70)
    at /head_old/sys/kern/subr_kdb.c:535
#8  0xffffffff808deff6 in trap (frame=0xffffff80627acc70)
    at /head_old/sys/amd64/amd64/trap.c:619
#9  0xffffffff808c8978 in calltrap ()
    at /head_old/sys/amd64/amd64/exception.S:228
#10 0xffffffff8060cd7d in kdb_enter (why=0xffffffff809fcb18 "panic", 
    msg=0xa <Address 0xa out of bounds>) at cpufunc.h:63
#11 0xffffffff805d884b in panic (fmt=Variable "fmt" is not available.
)
    at /head_old/sys/kern/kern_shutdown.c:573
#12 0xffffffff8138c197 in vcmn_err (ce=18, 
    fmt=0xa <Address 0xa out of bounds>, adx=0xffffff80627acf80)
    at /head_old/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:51
#13 0xffffffff8128914a in zfs_panic_recover (fmt=Variable "fmt" is not available.
)
    at /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa_misc.c:1218
#14 0xffffffff81243a93 in dmu_buf_hold_array_by_dnode (dn=0xffffff006de63000, 
    offset=98304, length=3399, read=0, tag=0xffffffff81335510, 
    numbufsp=0xffffff80627ad134, dbpp=0xffffff80627ad128, flags=Variable "flags" is not available.
)
    at /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:372
#15 0xffffffff812444d2 in dmu_write_uio_dnode (dn=Variable "dn" is not available.
)
    at /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:1019
#16 0xffffffff81244766 in dmu_write_uio_dbuf (zdb=0xffffff006e6d0148, 
    uio=0xffffff80627ad680, size=3399, tx=0xffffff006e7eac00)
    at /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:1076
#17 0xffffffff812db82f in zfs_freebsd_write (ap=Variable "ap" is not available.
)
    at /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:909
#18 0xffffffff8093bc1d in VOP_WRITE_APV (vop=0xffffffff81361500, 
    a=0xffffff80627ad6b0) at vnode_if.c:951
#19 0xffffffff807bbf22 in nfsrv_write (nfsd=0xffffff80627ad7f0, slp=0x0, 
    mrq=0xffffff80627ad7e0) at vnode_if.h:413
#20 0xffffffff807bd376 in nfssvc_program (rqst=0xffffff006e6a7800, xprt=Variable "xprt" is not available.
)
    at /head_old/sys/nfsserver/nfs_srvkrpc.c:352
#21 0xffffffff807d67cb in svc_run_internal (pool=0xffffff000261c200, ismaster=dwarf2_read_address: Corrupted DWARF expression.

) at /head_old/sys/rpc/svc.c:895
#22 0xffffffff807d6c7f in svc_run (pool=0xffffff000261c200)
    at /head_old/sys/rpc/svc.c:1235
#23 0xffffffff807bd542 in nfssvc_nfsd (td=Variable "td" is not available.
)
    at /head_old/sys/nfsserver/nfs_srvkrpc.c:467
#24 0xffffffff807bd5fb in nfssvc_nfsserver (td=0xffffff001737f880, uap=Variable "uap" is not available.
)
    at /head_old/sys/nfsserver/nfs_srvkrpc.c:193
#25 0xffffffff807bf153 in nfssvc (td=0xffffff001737f880, 
    uap=0xffffff80627adbb0) at /head_old/sys/nfs/nfs_nfssvc.c:98
#26 0xffffffff8061a09b in syscallenter (td=0xffffff001737f880, 
    sa=0xffffff80627adba0) at /head_old/sys/kern/subr_trap.c:319
#27 0xffffffff808de91c in syscall (frame=0xffffff80627adc40)
    at /head_old/sys/amd64/amd64/trap.c:939
#28 0xffffffff808c8c52 in Xfast_syscall ()
    at /head_old/sys/amd64/amd64/exception.S:381
#29 0x00000008006a2bec in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) 

------------------------------------------------------------------------
ps -axl

  UID   PID  PPID CPU PRI NI    VSZ    RSS MWCHAN STAT  TT       TIME COMMAND
    0     0     0   0  -8  0      0      0 -      DLs   ??  67408929:15.00 [kernel]
    0     1     0   0  76  0   7296    604 wait   DLs   ??  293523:24.00 [init]
    0     2     0   0  -8  0      0      0 -      DL    ??  805517:44.00 [g_event]
    0     3     0   0  -8  0      0      0 -      DL    ??  187516358:27.00 [g_up]
    0     4     0   0  -8  0      0      0 -      DL    ??  234271484:34.00 [g_down]
    0     5     0   0 -16  0      0      0 waitin DL    ??  3645:10.00 [sctp_ite
    0     6     0   0 -60  0      0      0 ccb_sc DL    ??    0:00.00 [xpt_thrd
    0     7     0   0  76  0      0      0 -      RL    ??  19153:07.00 [pagedaem
    0     8     0   0 -16  0      0      0 psleep DL    ??  500:13.00 [vmdaemon
    0     9     0   0  76  0      0      0 pgzero DL    ??  429:23.00 [pagezero
    0    10     0   0 -16  0      0      0 audit_ DL    ??    0:00.00 [audit]
    0    11     0   0 171  0      0      0 -      RL    ??  24690575099:18.00 [idle]
    0    12     0   0 -60  0      0      0 -      WL    ??  89758842:34.00 [intr]
    0    13     0   0  44  0      0      0 -      DL    ??  4941052:02.00 [yarrow]
    0    14     0   0 -64  0      0      0 -      DL    ??  138973:30.00 [usb]
    0    15     0   0 -16  0      0      0 psleep DL    ??  62694:36.00 [bufdaemo
    0    16     0   0 -16  0      0      0 vlruwt DL    ??  61477:56.00 [vnlru]
    0    17     0   0 -16  0      0      0 syncer DL    ??  169999:10.00 [syncer]
    0    18     0   0 -16  0      0      0 sdflus DL    ??  70042:21.00 [softdepf
    0    19     0   0 -16  0      0      0 flowcl DL    ??  36610:37.00 [flowclea
    0    39     0   0  -8  0      0      0 zio->i DL    ??  644441:11.00 [zfskern]
    0   636     1   0  76  0   7296    852 select Ds    ??  85117:58.00 [devd]
    0   783     1   0  44  0  11276   1672 select Ds    ??  463344:52.00 [syslogd]
    0   913     1   0  44  0  12204   1808 select Ds    ??  317812:24.00 [rpcbind]
    0   917     1   0  76  0  11144   1960 select Ds    ??  136662:12.00 [mountd]
    0   934     1   0  44  0  10044   1724 select Ds    ??  1031892:03.00 [nfsd]
    0   935   934   0  47  0  10044   1720 rpcsvc D     ??  5666521:04.00 [nfsd]
    0  1095     1   0  76  0  10040   1684 select Ds    ??  58947:28.00 [dhclient
   65  1144     1   0  76  0  10040   1808 select Ds    ??  15367:01.00 [dhclient
    0  1145     1   0  44  0  30540   4640 select Ds    ??  187410:34.00 [sshd]
    0  1154     1   0  44  0  16404   4172 select Ds    ??  142232:44.00 [sendmail
   25  1161     1   0  44  0  16404   4232 pause  Ds    ??  93368:12.00 [sendmail
    0  1168     1   0  50  0  12204   1792 nanslp Ds    ??  132706:11.00 [cron]
    0  1228     1   0  76  0  12536   1780 wait   D     ??    0:00.00 [sh]
    0  1229     1   0  76  0  10044   1136 piperd D     ??    0:00.00 [logger]
    0  1230  1228   0  76  0   2880    996 nanslp D     ??    0:00.00 [sleep]
    0  1232     1   0  76  0  11144   1428 ttyin  Ds+   ??  156840:12.00 [getty]
    0  1233     1   0  76  0  11144   1428 ttyin  Ds+   ??  156240:51.00 [getty]
    0  1234     1   0  76  0  11144   1428 ttyin  Ds+   ??  153097:43.00 [getty]
    0  1235     1   0  76  0  11144   1428 ttyin  Ds+   ??  133735:07.00 [getty]
    0  1236     1   0  76  0  11144   1428 ttyin  Ds+   ??  162378:40.00 [getty]
    0  1237     1   0  76  0  11144   1428 ttyin  Ds+   ??  170143:30.00 [getty]
    0  1238     1   0  76  0  11144   1428 ttyin  Ds+   ??  173020:09.00 [getty]
    0  1239     1   0  76  0  11144   1428 ttyin  Ds+   ??  142691:05.00 [getty]

------------------------------------------------------------------------
vmstat -s

  2407850 cpu context switches
   339852 device interrupts
    64027 software interrupts
   127735 traps
   662866 system calls
       20 kernel threads created
     1218  fork() calls
        2 vfork() calls
        0 rfork() calls
        0 swap pager pageins
        0 swap pager pages paged in
        0 swap pager pageouts
        0 swap pager pages paged out
      356 vnode pager pageins
     2475 vnode pager pages paged in
        0 vnode pager pageouts
        0 vnode pager pages paged out
        0 page daemon wakeups
        0 pages examined by the page daemon
      108 pages reactivated
    43921 copy-on-write faults
       45 copy-on-write optimized faults
    51631 zero fill pages zeroed
        0 zero fill pages prezeroed
       13 intransit blocking page faults
   131386 total VM faults taken
        0 pages affected by kernel thread creation
   935865 pages affected by  fork()
     1592 pages affected by vfork()
        0 pages affected by rfork()
        0 pages cached
   257206 pages freed
        0 pages freed by daemon
        0 pages freed by exiting processes
     2780 pages active
     1922 pages inactive
        2 pages in VM cache
    52427 pages wired down
   765005 pages free
     4096 bytes per page
    17489 total name lookups
          cache hits (77% pos + 6% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
    proc-args    24     2K       -      366  32,64,128
     acpitask     1     2K       -        1  2048
      ithread    84    14K       -       84  32,128,256
     ppbusdev     3     1K       -        3  256
       KTRACE   100    13K       -      100  128
      entropy  1024    64K       -     1024  64
       linker   145  1053K       -      164  16,32,64,128,256,512,1024,2048,4096
        lockf    23     3K       -       43  64,128
      CAM XPT    12     3K       -       26  32,64,128,2048
       ip6ndp     6     1K       -        7  64,128
         temp    23     5K       -     9497  16,32,64,128,256,512,1024,2048,4096
       devbuf  1062  2102K       -     1085  16,32,64,128,256,512,1024,2048,4096
       module   419    53K       -      419  128
     mtx_pool     2    16K       -        2  
         UART     6     4K       -        6  16,512,1024
          osd     3     1K       -        4  16,64
      acpisem    21     3K       -       21  128
      subproc   117   213K       -     1314  512,4096
         proc     2    16K       -        2  
      session    21     3K       -       24  128
         pgrp    21     3K       -       24  128
         cred    26     5K       -    28906  64,256
      uidinfo     4     3K       -        4  128,2048
       plimit    12     3K       -      134  256
    sysctltmp     0     0K       -      311  16,32,64,128,4096
    sysctloid  4363   215K       -     4470  16,32,64,128
       sysctl     0     0K       -      382  16,32,64
      callout     1   512K       -        1  
         umtx   516    65K       -      516  128
     p1003.1b     1     1K       -        1  16
         SWAP     2   685K       -        2  64
       bus-sc    81   133K       -     3506  16,32,64,128,256,512,1024,2048,4096
          bus  1157   102K       -     6341  16,32,64,128,256,512,1024
      devstat     6    13K       -        6  32,4096
 eventhandler    83     7K       -       83  64,128
         kobj   291  1164K       -      350  4096
      Per-cpu     1     1K       -        1  32
CAM dev queue     1     1K       -        1  128
    CAM queue     3     1K       -        7  16
         rman   179    22K       -      644  16,32,128
         sbuf     0     0K       -      426  16,32,64,128,256,512,1024,2048,4096
      CAM SIM     1     1K       -        1  256
        stack     0     0K       -        2  256
    taskqueue    45     5K       -       75  16,32,64,128,1024
       Unitno    10     1K       -      292  32,64
       USBdev     8     3K       -        8  64,128,1024
          USB    14     5K       -       14  16,32,64,128,2048
  ata_generic     5     5K       -        5  1024
      Witness     1   128K       -        1  
          iov     0     0K       -      648  16,64,128,256,512
       select    15     2K       -       15  128
     ioctlops     0     0K       -      782  16,32,64,128,256,512,1024
          msg     4    30K       -        4  2048,4096
          sem     4   106K       -        4  2048,4096
          shm     1    20K       -        1  
          tty    21    21K       -       23  1024,2048
     mbuf_tag     0     0K       -        9  32
         ksem     1     8K       -        1  
        shmfd     1     8K       -        1  
          pcb    24   157K       -       40  16,32,128,1024,2048,4096
       soname     4     1K       -      299  16,32,128
       biobuf     4     8K       -        6  2048
     vfscache     1  2048K       -        1  
   cl_savebuf     0     0K       -        2  64
  export_host     1     1K       -        1  256
     vfs_hash     1  1024K       -        1  
       vnodes     2     1K       -        2  256
  vnodemarker     0     0K       -       31  1024
        mount    46     3K       -      233  16,32,64,128,256,512
          BPF     9    10K       -        9  128,512,4096
  ether_multi    40     3K       -       52  16,32,64
       ifaddr    51    14K       -       52  32,64,128,256,512,4096
        ifnet     5     9K       -        5  128,2048
    ad_driver     5     1K       -        5  32
        clone     6    24K       -        6  4096
       arpcom     2     1K       -        2  16
      lltable    14     6K       -       14  256,512
    ar_driver     0     0K       -       30  512,2048
       kbdmux     6    18K       -        6  16,512,1024,2048
     routetbl    33  1045K       -      116  32,64,128,256,512
         igmp     4     1K       -        4  256
       DEVFS1    85    43K       -       87  512
       DEVFS3    99    25K       -      102  256
     in_multi     2     1K       -        3  256
    sctp_iter     0     0K       -        4  256
     sctp_ifn     2     1K       -        2  128
     sctp_ifa     5     1K       -        5  128
     sctp_vrf     1     1K       -        1  64
    sctp_a_it     0     0K       -        4  16
    hostcache     1    28K       -        1  
     syncache     1    96K       -        1  
        DEVFS    12     1K       -       13  16,128
       DEVFSP     1     1K       -        1  64
 ip6_moptions     2     1K       -        2  32,256
    in6_multi    22     3K       -       22  32,256
  in6_mfilter     1     1K       -        1  1024
          mld     4     1K       -        4  128
      NFS FHA     2     3K       -     8637  64,2048
          rpc  1051   547K       -    52974  32,64,128,256,512,2048
audit_evclass   172     6K       -      211  32
       newblk     1   128K       -        1  
    bmsafemap     1     8K       -        1  
     inodedep     1  1024K       -        1  
      pagedep     1   128K       -        1  
  ufs_dirhash    30     6K       -       30  16,32,64,128,512
    ufs_mount     3    31K       -        3  512,2048
    pfs_nodes    21     6K       -       21  256
    vm_pgdata     2   129K       -        2  128
       apmdev     1     1K       -        1  128
   CAM periph     2     1K       -       12  16,32,64,128,256
         GEOM    73    22K       -      385  16,32,64,128,256,512,1024
      acpidev    76     5K       -       76  64
      memdesc     1     4K       -        1  4096
     nexusdev     3     1K       -        3  16
     atkbddev     2     1K       -        2  64
     pci_link    30     3K       -       30  16,128
    acpi_perf     2     1K       -        2  128
       isadev     5     1K       -        5  128
  ddb_capture     1    48K       -        1  
         cdev     8     2K       -        8  256
       acpica  1695   169K       -    73810  16,32,64,128,256,512,1024
        sigio     1     1K       -        1  64
     filedesc    44    22K       -     1241  512
      kdtrace   287    64K       -     1622  64,256
      io_apic     3     3K       -        3  512,2048
         kenv    65    11K       -       69  16,32,64,128
       kqueue     0     0K       -       28  256,2048
      solaris 18272 120355K       -   107850  16,32,64,128,256,512,1024,2048,4096
   kstat_data     4     1K       -        4  64

------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,     251,       4,     251,   0,   0
UMA Zones:              640,      0,     251,       1,     251,   0,   0
UMA Slabs:              568,      0,    2661,     678,    9473,   0,   0
UMA RCntSlabs:          568,      0,     953,       6,     953,   0,   0
UMA Hash:               256,      0,      81,       9,      81,   0,   0
16 Bucket:              152,      0,     190,      10,     190,   0,   0
32 Bucket:              280,      0,     108,       4,     108,   1,   0
64 Bucket:              536,      0,     124,       2,     124,  66,   0
128 Bucket:            1048,      0,     193,       2,     193,   0,   0
VM OBJECT:              216,      0,    5987,      79,   22488,   0,   0
MAP:                    232,      0,       7,      25,       7,   0,   0
KMAP ENTRY:             120, 127441,     191,     429,   17315,   0,   0
MAP ENTRY:              120,      0,     437,     152,   32509,   0,   0
DP fakepg:              120,      0,       0,       0,       0,   0,   0
SG fakepg:              120,      0,       0,       0,       0,   0,   0
mt_zone:               2064,      0,     282,      23,     282,   0,   0
16:                      16,      0,      22,     314,     189,   0,   0
16:                      16,      0,    1619,     229,    2230,   0,   0
16:                      16,      0,       3,     165,       3,   0,   0
16:                      16,      0,      41,     295,   31640,   0,   0
16:                      16,      0,      28,     308,     398,   0,   0
16:                      16,      0,     310,     362,    2920,   0,   0
16:                      16,      0,       7,     329,     166,   0,   0
16:                      16,      0,     642,     534,   10265,   0,   0
32:                      32,      0,      18,     285,      51,   0,   0
32:                      32,      0,    1773,     247,    2003,   0,   0
32:                      32,      0,      10,     293,      92,   0,   0
32:                      32,      0,      47,     256,    7303,   0,   0
32:                      32,      0,      31,     272,     186,   0,   0
32:                      32,      0,      95,     410,    1812,   0,   0
32:                      32,      0,      87,     216,     561,   0,   0
32:                      32,      0,    1898,     627,   13372,   0,   0
64:                      64,      0,      13,     155,     360,   0,   0
64:                      64,      0,     279,     113,     321,   0,   0
64:                      64,      0,      15,     153,     256,   0,   0
64:                      64,      0,     733,     107,   30585,   0,   0
64:                      64,      0,      79,     145,   16067,   0,   0
64:                      64,      0,     696,     144,     989,   0,   0
64:                      64,      0,    1118,     114,    9774,   0,   0
64:                      64,      0,    2936,     312,   25408,   0,   0
128:                    128,      0,     561,      48,     678,   0,   0
128:                    128,      0,    1092,      68,    1123,   0,   0
128:                    128,      0,       1,      86,      20,   0,   0
128:                    128,      0,    1020,      82,    3095,   0,   0
128:                    128,      0,     141,      62,     171,   0,   0
128:                    128,      0,     430,      63,     658,   0,   0
128:                    128,      0,     217,      44,     724,   0,   0
128:                    128,      0,     897,      60,   14491,   0,   0
256:                    256,      0,       6,      39,      37,   0,   0
256:                    256,      0,       5,      40,      20,   0,   0
256:                    256,      0,       0,       0,       0,   0,   0
256:                    256,      0,       4,      41,      15,   0,   0
256:                    256,      0,     318,      72,   15110,   0,   0
256:                    256,      0,     673,      62,    1080,   0,   0
256:                    256,      0,      45,      30,      55,   0,   0
256:                    256,      0,    3132,    1638,    7457,   0,   0
512:                    512,      0,    1038,      33,   17363,   0,   0
512:                    512,      0,       6,      15,       6,   0,   0
512:                    512,      0,       1,      13,      16,   0,   0
512:                    512,      0,       2,      12,       9,   0,   0
512:                    512,      0,      16,      12,      47,   0,   0
512:                    512,      0,     250,      51,    2455,   0,   0
512:                    512,      0,      14,      14,      23,   0,   0
512:                    512,      0,    7442,     111,   23356,   0,   0
1024:                  1024,      0,       0,      12,      17,   0,   0
1024:                  1024,      0,       2,       6,       3,   0,   0
1024:                  1024,      0,       2,      10,      33,   0,   0
1024:                  1024,      0,      10,      10,    1353,   0,   0
1024:                  1024,      0,      23,      13,      32,   0,   0
1024:                  1024,      0,      13,     167,    1307,   0,   0
1024:                  1024,      0,       0,       8,       4,   0,   0
1024:                  1024,      0,      73,    1255,    5153,   0,   0
2048:                  2048,      0,      12,      40,   40253,   0,   0
2048:                  2048,      0,      11,       7,      27,   0,   0
2048:                  2048,      0,       2,       2,      12,   0,   0
2048:                  2048,      0,       4,       4,       4,   0,   0
2048:                  2048,      0,       4,       8,       6,   0,   0
2048:                  2048,      0,       6,       2,      52,   0,   0
2048:                  2048,      0,       2,       2,       6,   0,   0
2048:                  2048,      0,     215,     129,    1781,   0,   0
4096:                  4096,      0,       3,       2,    4124,   0,   0
4096:                  4096,      0,       2,       3,       4,   0,   0
4096:                  4096,      0,     291,       4,     350,   0,   0
4096:                  4096,      0,       3,       2,       3,   0,   0
4096:                  4096,      0,       0,       0,       0,   0,   0
4096:                  4096,      0,      76,      28,    1517,   0,   0
4096:                  4096,      0,       1,       4,       9,   0,   0
4096:                  4096,      0,     289,      38,    1100,   0,   0
Files:                   80,      0,      75,     150,    5028,   0,   0
TURNSTILE:              136,      0,     259,      41,     259,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1120,      0,      43,      29,    1240,   0,   0
THREAD:                1080,      0,     242,      16,     380,   0,   0
SLEEPQUEUE:              88,      0,     259,      31,     259,   0,   0
VMSPACE:                400,      0,      24,      30,    1222,   0,   0
cpuset:                  72,      0,       2,      98,       2,   0,   0
audit_record:           952,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,      0,     829,     328,  115504,   0,   0
mbuf:                   256,      0,    1024,     389,  126550,   0,   0
mbuf_cluster:          2048,  25600,    1156,     750,    2604,   0,   0
mbuf_jumbo_page:       4096,  12800,       0,       0,       0,   0,   0
mbuf_jumbo_9k:         9216,  19200,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,  12800,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
g_bio:                  232,      0,       4,     220, 1311935,   0,   0
ttyinq:                 160,      0,    1440,      48,    1575,   0,   0
ttyoutq:                256,      0,     744,      36,     816,   0,   0
ata_request:            328,      0,       3,     105,  531216,   0,   0
ata_composite:          336,      0,       0,       0,       0,   0,   0
VNODE:                  632,      0,    7671,      33,    7702,   0,   0
VNODEPOLL:              112,      0,       0,       0,       0,   0,   0
NAMEI:                 1024,      0,       0,      36,    8887,   0,   0
S VFS Cache:            108,      0,     497,     328,    1431,   0,   0
L VFS Cache:            328,      0,    1170,      78,    1237,   0,   0
NFSMOUNT:               616,      0,       0,       0,       0,   0,   0
NFSNODE:                640,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,      40,       8,      40,   0,   0
pipe:                   728,      0,       2,      38,     824,   0,   0
ksiginfo:               112,      0,      52,    1004,      58,   0,   0
itimer:                 344,      0,       0,       0,       0,   0,   0
KNOTE:                  128,      0,       0,      87,      26,   0,   0
socket:                 680,  25602,      30,      18,     320,   0,   0
ipq:                     56,    819,       0,       0,       0,   0,   0
udp_inpcb:              336,  25608,      11,      22,     235,   0,   0
udpcb:                   16,  25704,      11,     325,     235,   0,   0
tcp_inpcb:              336,  25608,      10,      23,      15,   0,   0
tcpcb:                  880,  25600,      10,      10,      15,   0,   0
tcptw:                   72,   5150,       0,       0,       0,   0,   0
syncache:               144,  15366,       0,      52,       4,   0,   0
hostcache:              136,  15372,       0,       0,       0,   0,   0
tcpreass:                40,   1680,       0,       0,       0,   0,   0
sackhole:                32,      0,       0,       0,       0,   0,   0
sctp_ep:               1272,  25602,       0,       0,       0,   0,   0
sctp_asoc:             2240,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80064,       0,     216,       4,   0,   0
sctp_raddr:             616,  80004,       0,       0,       0,   0,   0
sctp_chunk:             136, 400008,       0,       0,       0,   0,   0
sctp_readq:             104, 400032,       0,       0,       0,   0,   0
sctp_stream_msg_out:     96, 400026,       0,       0,       0,   0,   0
sctp_asconf:             40, 400008,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400032,       0,       0,       0,   0,   0
ripcb:                  336,  25608,       1,      21,       1,   0,   0
unpcb:                  240,  25600,       7,      41,      65,   0,   0
rtentry:                200,      0,      13,      44,      14,   0,   0
selfd:                   56,      0,      43,     146,     916,   0,   0
SWAPMETA:               288, 116519,       0,       0,       0,   0,   0
ip4flow:                 56,  50211,       4,     311,       8,   0,   0
ip6flow:                 80,  50220,       0,     135,       1,   0,   0
Mountpoints:            912,      0,       3,       9,       3,   0,   0
FFS inode:              168,      0,     412,      72,     441,   0,   0
FFS1 dinode:            128,      0,       0,       0,       0,   0,   0
FFS2 dinode:            256,      0,     412,      38,     441,   0,   0
taskq_zone:              64,      0,       0,     168,       8,   0,   0
reference_cache:         40,      0,       0,       0,       0,   0,   0
reference_history_cache:      8,      0,       0,       0,       0,   0,   0
zio_cache:              896,      0,     194,    3350,   36506,   0,   0
zio_link_cache:          48,      0,     191,    3409,   25282,   0,   0
zio_buf_512:            512,      0,       0,       0,       0,   0,   0
zio_data_buf_512:       512,      0,       0,       0,       0,   0,   0
zio_buf_1024:          1024,      0,       0,       0,       0,   0,   0
zio_data_buf_1024:     1024,      0,       0,       0,       0,   0,   0
zio_buf_1536:          1536,      0,       0,       0,       0,   0,   0
zio_data_buf_1536:     1536,      0,       0,       0,       0,   0,   0
zio_buf_2048:          2048,      0,       0,       0,       0,   0,   0
zio_data_buf_2048:     2048,      0,       0,       0,       0,   0,   0
zio_buf_2560:          2560,      0,       0,       0,       0,   0,   0
zio_data_buf_2560:     2560,      0,       0,       0,       0,   0,   0
zio_buf_3072:          3072,      0,       0,       0,       0,   0,   0
zio_data_buf_3072:     3072,      0,       0,       0,       0,   0,   0
zio_buf_3584:          3584,      0,       0,       0,       0,   0,   0
zio_data_buf_3584:     3584,      0,       0,       0,       0,   0,   0
zio_buf_4096:          4096,      0,       0,       0,       0,   0,   0
zio_data_buf_4096:     4096,      0,       0,       0,       0,   0,   0
zio_buf_5120:          5120,      0,       0,       0,       0,   0,   0
zio_data_buf_5120:     5120,      0,       0,       0,       0,   0,   0
zio_buf_6144:          6144,      0,       0,       0,       0,   0,   0
zio_data_buf_6144:     6144,      0,       0,       0,       0,   0,   0
zio_buf_7168:          7168,      0,       0,       0,       0,   0,   0
zio_data_buf_7168:     7168,      0,       0,       0,       0,   0,   0
zio_buf_8192:          8192,      0,       0,       0,       0,   0,   0
zio_data_buf_8192:     8192,      0,       0,       0,       0,   0,   0
zio_buf_10240:        10240,      0,       0,       0,       0,   0,   0
zio_data_buf_10240:   10240,      0,       0,       0,       0,   0,   0
zio_buf_12288:        12288,      0,       0,       0,       0,   0,   0
zio_data_buf_12288:   12288,      0,       0,       0,       0,   0,   0
zio_buf_14336:        14336,      0,       0,       0,       0,   0,   0
zio_data_buf_14336:   14336,      0,       0,       0,       0,   0,   0
zio_buf_16384:        16384,      0,       0,       0,       0,   0,   0
zio_data_buf_16384:   16384,      0,       0,       0,       0,   0,   0
zio_buf_20480:        20480,      0,       0,       0,       0,   0,   0
zio_data_buf_20480:   20480,      0,       0,       0,       0,   0,   0
zio_buf_24576:        24576,      0,       0,       0,       0,   0,   0
zio_data_buf_24576:   24576,      0,       0,       0,       0,   0,   0
zio_buf_28672:        28672,      0,       0,       0,       0,   0,   0
zio_data_buf_28672:   28672,      0,       0,       0,       0,   0,   0
zio_buf_32768:        32768,      0,       0,       0,       0,   0,   0
zio_data_buf_32768:   32768,      0,       0,       0,       0,   0,   0
zio_buf_36864:        36864,      0,       0,       0,       0,   0,   0
zio_data_buf_36864:   36864,      0,       0,       0,       0,   0,   0
zio_buf_40960:        40960,      0,       0,       0,       0,   0,   0
zio_data_buf_40960:   40960,      0,       0,       0,       0,   0,   0
zio_buf_45056:        45056,      0,       0,       0,       0,   0,   0
zio_data_buf_45056:   45056,      0,       0,       0,       0,   0,   0
zio_buf_49152:        49152,      0,       0,       0,       0,   0,   0
zio_data_buf_49152:   49152,      0,       0,       0,       0,   0,   0
zio_buf_53248:        53248,      0,       0,       0,       0,   0,   0
zio_data_buf_53248:   53248,      0,       0,       0,       0,   0,   0
zio_buf_57344:        57344,      0,       0,       0,       0,   0,   0
zio_data_buf_57344:   57344,      0,       0,       0,       0,   0,   0
zio_buf_61440:        61440,      0,       0,       0,       0,   0,   0
zio_data_buf_61440:   61440,      0,       0,       0,       0,   0,   0
zio_buf_65536:        65536,      0,       0,       0,       0,   0,   0
zio_data_buf_65536:   65536,      0,       0,       0,       0,   0,   0
zio_buf_69632:        69632,      0,       0,       0,       0,   0,   0
zio_data_buf_69632:   69632,      0,       0,       0,       0,   0,   0
zio_buf_73728:        73728,      0,       0,       0,       0,   0,   0
zio_data_buf_73728:   73728,      0,       0,       0,       0,   0,   0
zio_buf_77824:        77824,      0,       0,       0,       0,   0,   0
zio_data_buf_77824:   77824,      0,       0,       0,       0,   0,   0
zio_buf_81920:        81920,      0,       0,       0,       0,   0,   0
zio_data_buf_81920:   81920,      0,       0,       0,       0,   0,   0
zio_buf_86016:        86016,      0,       0,       0,       0,   0,   0
zio_data_buf_86016:   86016,      0,       0,       0,       0,   0,   0
zio_buf_90112:        90112,      0,       0,       0,       0,   0,   0
zio_data_buf_90112:   90112,      0,       0,       0,       0,   0,   0
zio_buf_94208:        94208,      0,       0,       0,       0,   0,   0
zio_data_buf_94208:   94208,      0,       0,       0,       0,   0,   0
zio_buf_98304:        98304,      0,       0,       0,       0,   0,   0
zio_data_buf_98304:   98304,      0,       0,       0,       0,   0,   0
zio_buf_102400:      102400,      0,       0,       0,       0,   0,   0
zio_data_buf_102400: 102400,      0,       0,       0,       0,   0,   0
zio_buf_106496:      106496,      0,       0,       0,       0,   0,   0
zio_data_buf_106496: 106496,      0,       0,       0,       0,   0,   0
zio_buf_110592:      110592,      0,       0,       0,       0,   0,   0
zio_data_buf_110592: 110592,      0,       0,       0,       0,   0,   0
zio_buf_114688:      114688,      0,       0,       0,       0,   0,   0
zio_data_buf_114688: 114688,      0,       0,       0,       0,   0,   0
zio_buf_118784:      118784,      0,       0,       0,       0,   0,   0
zio_data_buf_118784: 118784,      0,       0,       0,       0,   0,   0
zio_buf_122880:      122880,      0,       0,       0,       0,   0,   0
zio_data_buf_122880: 122880,      0,       0,       0,       0,   0,   0
zio_buf_126976:      126976,      0,       0,       0,       0,   0,   0
zio_data_buf_126976: 126976,      0,       0,       0,       0,   0,   0
zio_buf_131072:      131072,      0,       0,       0,       0,   0,   0
zio_data_buf_131072: 131072,      0,       0,       0,       0,   0,   0
sa_cache:                80,      0,    7230,     105,    7231,   0,   0
dnode_t:               1064,      0,    7280,      19,    7333,   0,   0
dmu_buf_impl_t:         328,      0,    8063,     397,    9862,   0,   0
arc_buf_hdr_t:          320,      0,    1108,     260,    2904,   0,   0
arc_buf_t:              104,      0,    1102,     338,    2936,   0,   0
zil_lwb_cache:          192,      0,       0,       0,       0,   0,   0
zfs_znode_cache:        408,      0,    7230,      15,    7231,   0,   0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq1: atkbd0                           1          0
irq8: atrtc0                       44909        165
irq14: ata0                       261093        963
irq22: atapci1                      1971          7
irq23: atapci2                      1950          7
irq29: bge1                        29928        110
cpu0:timer                        350873       1294
cpu1:timer                        349975       1291
Total                            1040700       3840

------------------------------------------------------------------------
pstat -T

 75/12328 files
0M/5119M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity
/dev/ad0s1b       10485504        0 10485504     0%

------------------------------------------------------------------------
iostat

iostat: kvm_read(_tk_nin): invalid address (0x0)
iostat: disabling TTY statistics
iostat: kvm_getcptime: invalid address (0x0)
iostat: disabling CPU time statistics
             ad0              ad4              ad6 
  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s 
  4.15 922  3.73  39.57   3  0.13  39.54   3  0.13 

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME   

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME   

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME   


------------------------------------------------------------------------
ipcs -T

msginfo:
	msgmax:        16384	(max characters in a message)
	msgmni:           40	(# of message queues)
	msgmnb:         2048	(max characters in a message queue)
	msgtql:           40	(max # of messages in system)
	msgssz:            8	(size of a message segment)
	msgseg:         2048	(# of message segments in system)

shminfo:
	shmmax:    536870912	(max shared memory segment size)
	shmmin:            1	(min shared memory segment size)
	shmmni:          192	(max number of shared memory identifiers)
	shmseg:          128	(max shared memory segments per process)
	shmall:       131072	(max amount of shared memory in pages)

seminfo:
	semmap:           30	(# of entries in semaphore map)
	semmni:           50	(# of semaphore identifiers)
	semmns:          340	(# of semaphores in system)
	semmnu:          150	(# of undo structures in system)
	semmsl:          340	(max # of semaphores per id)
	semopm:          100	(max # of operations per semop call)
	semume:           50	(max # of undo entries per process)
	semusz:          632	(size in bytes of undo structure)
	semvmx:        32767	(semaphore maximum value)
	semaem:        16384	(adjust on exit max value)


------------------------------------------------------------------------
nfsstat

Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0         0
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
        0         0         0         0         0         0         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses
        0         0         0         0         0         0

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
     5782        11      1656         0         0      4483         3         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        2         0         0         0         0         0         0       814
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         3
Server Ret-Failed
              444
Server Faults
            0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
        0         0         0         0
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
     4249      4483       234

------------------------------------------------------------------------
netstat -s

tcp:
	73644 packets sent
		12753 data packets (1853404 bytes)
		0 data packets (0 bytes) retransmitted
		0 data packets unnecessarily retransmitted
		0 resends initiated by MTU discovery
		49273 ack-only packets (5 delayed)
		0 URG only packets
		0 window probe packets
		11618 window update packets
		0 control packets
	112636 packets received
		12082 acks (for 1853244 bytes)
		0 duplicate acks
		0 acks for unsent data
		111937 packets (148540392 bytes) received in-sequence
		0 completely duplicate packets (0 bytes)
		0 old duplicate packets
		0 packets with some dup. data (0 bytes duped)
		0 out-of-order packets (0 bytes)
		0 packets (0 bytes) of data after window
		0 window probes
		0 window update packets
		0 packets received after close
		0 discarded for bad checksums
		0 discarded for bad header offset fields
		0 discarded because packet too short
		0 discarded due to memory problems
	0 connection requests
	4 connection accepts
	0 bad connection attempts
	0 listen queue overflows
	0 ignored RSTs in the windows
	4 connections established (including accepts)
	5 connections closed (including 3 drops)
		0 connections updated cached RTT on close
		0 connections updated cached RTT variance on close
		0 connections updated cached ssthresh on close
	0 embryonic connections dropped
	12082 segments updated rtt (of 11419 attempts)
	0 retransmit timeouts
		0 connections dropped by rexmit timeout
	0 persist timeouts
		0 connections dropped by persist timeout
	0 Connections (fin_wait_2) dropped because of timeout
	0 keepalive timeouts
		0 keepalive probes sent
		0 connections dropped by keepalive
	336 correct ACK header predictions
	99042 correct data packet header predictions
	4 syncache entries added
		0 retransmitted
		0 dupsyn
		0 dropped
		4 completed
		0 bucket overflow
		0 cache overflow
		0 reset
		0 stale
		0 aborted
		0 badack
		0 unreach
		0 zone failures
	4 cookies sent
	0 cookies received
	0 SACK recovery episodes
	0 segment rexmits in SACK recovery episodes
	0 byte rexmits in SACK recovery episodes
	0 SACK options (SACK blocks) received
	0 SACK options (SACK blocks) sent
	0 SACK scoreboard overflow
	0 packets with ECN CE bit set
	0 packets with ECN ECT(0) bit set
	0 packets with ECN ECT(1) bit set
	0 successful ECN handshakes
	0 times ECN reduced the congestion window
udp:
	13 datagrams received
	0 with incomplete header
	0 with bad data length field
	0 with bad checksum
	0 with no checksum
	0 dropped due to no socket
	0 broadcast/multicast datagrams undelivered
	0 dropped due to full socket buffers
	0 not for hashed pcb
	13 delivered
	13 datagrams output
	0 times multicast source filter matched
ip:
	112650 total packets received
	0 bad header checksums
	0 with size smaller than minimum
	0 with data size < data length
	0 with ip length > max ip packet size
	0 with header length < data size
	0 with data length < header length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 packets reassembled ok
	112649 packets for this host
	0 packets for unknown/unsupported protocol
	0 packets forwarded (0 packets fast forwarded)
	1 packet not forwardable
	0 packets received for unknown multicast group
	0 redirects sent
	73664 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	0 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 tunneling packets that can't find gif
	0 datagrams with bad address in header
icmp:
	0 calls to icmp_error
	0 errors not generated in response to an icmp message
	0 messages with bad code fields
	0 messages less than the minimum length
	0 messages with bad checksum
	0 messages with bad length
	0 multicast echo requests ignored
	0 multicast timestamp requests ignored
	0 message responses generated
	0 invalid return addresses
	0 no return routes
igmp:
	0 messages received
	0 messages received with too few bytes
	0 messages received with wrong TTL
	0 messages received with bad checksum
	0 V1/V2 membership queries received
	0 V3 membership queries received
	0 membership queries received with invalid field(s)
	0 general queries received
	0 group queries received
	0 group-source queries received
	0 group-source queries dropped
	0 membership reports received
	0 membership reports received with invalid field(s)
	0 membership reports received for groups to which we belong
	0 V3 reports received without Router Alert
	0 membership reports sent
arp:
	2 ARP requests sent
	3 ARP replies sent
	2330 ARP requests received
	1 ARP reply received
	2331 ARP packets received
	0 total packets dropped due to no ARP entry
	0 ARP entrys timed out
	0 Duplicate IPs seen
ip6:
	0 total packets received
	0 with size smaller than minimum
	0 with data size < data length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 fragments that exceeded limit
	0 packets reassembled ok
	0 packets for this host
	0 packets forwarded
	0 packets not forwardable
	0 redirects sent
	5 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	0 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 packets that violated scope rules
	0 multicast packets which we don't join
	Mbuf statistics:
		0 one mbuf
		0 one ext mbuf
		0 two or more ext mbuf
	0 packets whose headers are not continuous
	0 tunneling packets that can't find gif
	0 packets discarded because of too many headers
	0 failures of source address selection
	Source addresses selection rule applied:
icmp6:
	0 calls to icmp6_error
	0 errors not generated in response to an icmp6 message
	0 errors not generated because of rate limitation
	Output histogram:
		neighbor solicitation: 1
		MLDv2 listener report: 4
	0 messages with bad code fields
	0 messages < minimum length
	0 bad checksums
	0 messages with bad length
	Histogram of error messages to be generated:
		0 no route
		0 administratively prohibited
		0 beyond scope
		0 address unreachable
		0 port unreachable
		0 packet too big
		0 time exceed transit
		0 time exceed reassembly
		0 erroneous header field
		0 unrecognized next header
		0 unrecognized option
		0 redirect
		0 unknown
	0 message responses generated
	0 messages with too many ND options
	0 messages with bad ND options
	0 bad neighbor solicitation messages
	0 bad neighbor advertisement messages
	0 bad router solicitation messages
	0 bad router advertisement messages
	0 bad redirect messages
	0 path MTU changes
rip6:
	0 messages received
	0 checksum calculations on inbound
	0 messages with bad checksum
	0 messages dropped due to no socket
	0 multicast messages dropped due to no socket
	0 messages dropped due to full socket buffers
	0 delivered
	0 datagrams output

------------------------------------------------------------------------
netstat -m

1853/717/2570 mbufs in use (current/cache/total)
828/1078/1906/25600 mbuf clusters in use (current/cache/total/max)
829/328 mbuf+clusters out of packet secondary zone in use (current/cache)
0/0/0/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/19200 9k jumbo clusters in use (current/cache/total/max)
0/0/0/12800 16k jumbo clusters in use (current/cache/total/max)
2119K/2335K/4454K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

------------------------------------------------------------------------
netstat -id

Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll Drop
bge0*  1500 <Link#1>      00:e0:81:40:29:d2        0     0     0        0     0     0    0 
bge1   1500 <Link#2>      00:e0:81:40:29:d3   114981     0     0    73675     0     0    0 
bge1   1500 fe80:2::2e0:8 fe80:2::2e0:81ff:        0     -     -        3     -     -    - 
bge1   1500 10.42.43.0    xanadu              112649     -     -    73664     -     -    - 
plip0  1500 <Link#3>                               0     0     0        0     0     0    0 
lo0   16384 <Link#4>                               0     0     0        0     0     0    0 
lo0   16384 fe80:4::1     fe80:4::1                0     -     -        0     -     -    - 
lo0   16384 localhost     ::1                      0     -     -        0     -     -    - 
lo0   16384 your-net      localhost                0     -     -        0     -     -    - 

------------------------------------------------------------------------
netstat -anr

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            10.42.43.1         UGS         0        0   bge1
10.42.43.0/24      link#2             U           4    73664   bge1
10.42.43.10        link#2             UHS         0        0    lo0
127.0.0.1          link#4             UH          0        0    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::1                               ::1                           UH          lo0
fe80::%bge1/64                    link#2                        U          bge1
fe80::2e0:81ff:fe40:29d3%bge1     link#2                        UHS         lo0
fe80::%lo0/64                     link#4                        U           lo0
fe80::1%lo0                       link#4                        UHS         lo0
ff01:2::/32                       fe80::2e0:81ff:fe40:29d3%bge1 U          bge1
ff01:4::/32                       fe80::1%lo0                   U           lo0
ff02::%bge1/32                    fe80::2e0:81ff:fe40:29d3%bge1 U          bge1
ff02::%lo0/32                     fe80::1%lo0                   U           lo0

------------------------------------------------------------------------
netstat -anA

Active Internet connections (including servers)
Tcpcb    Proto Recv-Q Send-Q  Local Address      Foreign Address   (state)
ffffff00175ef000 tcp4   82708    164 10.42.43.10.2049   10.42.43.1.762     ESTABLISHED
ffffff0017548000 tcp4       0      0 127.0.0.1.25       *.*                LISTEN
ffffff00175496e0 tcp4       0      0 *.22               *.*                LISTEN
ffffff0017549a50 tcp6       0      0 *.22               *.*                LISTEN
ffffff00173df000 tcp6       0      0 *.2049             *.*                LISTEN
ffffff00173df370 tcp4       0      0 *.2049             *.*                LISTEN
ffffff00175486e0 tcp4       0      0 *.627              *.*                LISTEN
ffffff00173df6e0 tcp6       0      0 *.627              *.*                LISTEN
ffffff0017548a50 tcp4       0      0 *.111              *.*                LISTEN
ffffff0017549000 tcp6       0      0 *.111              *.*                LISTEN
ffffff001739b3f0 udp6       0      0 *.2049             *.*                
ffffff001739b2a0 udp4       0      0 *.2049             *.*                
ffffff001739b690 udp4       0      0 *.627              *.*                
ffffff001739bd20 udp6       0      0 *.627              *.*                
ffffff001739ad20 udp6       0      0 *.*                *.*                
ffffff001739aa80 udp4       0      0 *.720              *.*                
ffffff001739a7e0 udp4       0      0 *.111              *.*                
ffffff001739a540 udp6       0      0 *.1015             *.*                
ffffff0002a312a0 udp6       0      0 *.111              *.*                
ffffff001739a150 udp4       0      0 *.514              *.*                
ffffff001739a3f0 udp6       0      0 *.514              *.*                
Active UNIX domain sockets
Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
ffffff00173c0c30 stream      0      0 ffffff0017525c58        0        0        0 /var/run/rpcbind.sock
ffffff001739fc30 stream      0      0 ffffff0017396768        0        0        0 /var/run/devd.pipe
ffffff00173c0960 dgram       0      0        0 ffffff001739f870        0 ffffff001739f0f0
ffffff001739f0f0 dgram       0      0        0 ffffff001739f870        0 ffffff00173c0e10
ffffff00173c0e10 dgram       0      0        0 ffffff001739f870        0        0
ffffff001739f870 dgram       0      0 ffffff001757f278        0 ffffff00173c0960        0 /var/run/logpriv
ffffff00173c0d20 dgram       0      0 ffffff00175799e0        0        0        0 /var/run/log

------------------------------------------------------------------------
netstat -aL

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address         
tcp4  0/0/10         localhost.smtp         
tcp4  0/0/128        *.ssh                  
tcp6  0/0/128        *.ssh                  
tcp6  0/0/5          *.nfsd                 
tcp4  0/0/5          *.nfsd                 
tcp4  0/0/128        *.passgo-tivoli        
tcp6  0/0/128        *.passgo-tivoli        
tcp4  0/0/128        *.sunrpc               
tcp6  0/0/128        *.sunrpc               
unix  0/0/128        /var/run/rpcbind.sock
unix  0/0/4          /var/run/devd.pipe

------------------------------------------------------------------------
fstat

USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
root     getty       1239 root /             2 drwxr-xr-x    1024  r
root     getty       1239   wd /             2 drwxr-xr-x    1024  r
root     getty       1239 text /        28856334 -r-xr-xr-x   27696  r
root     getty       1239    0 /dev         59 crw-------   ttyv7 rw
root     getty       1239    1 /dev         59 crw-------   ttyv7 rw
root     getty       1239    2 /dev         59 crw-------   ttyv7 rw
root     getty       1238 root /             2 drwxr-xr-x    1024  r
root     getty       1238   wd /             2 drwxr-xr-x    1024  r
root     getty       1238 text /        28856334 -r-xr-xr-x   27696  r
root     getty       1238    0 /dev         58 crw-------   ttyv6 rw
root     getty       1238    1 /dev         58 crw-------   ttyv6 rw
root     getty       1238    2 /dev         58 crw-------   ttyv6 rw
root     getty       1237 root /             2 drwxr-xr-x    1024  r
root     getty       1237   wd /             2 drwxr-xr-x    1024  r
root     getty       1237 text /        28856334 -r-xr-xr-x   27696  r
root     getty       1237    0 /dev         57 crw-------   ttyv5 rw
root     getty       1237    1 /dev         57 crw-------   ttyv5 rw
root     getty       1237    2 /dev         57 crw-------   ttyv5 rw
root     getty       1236 root /             2 drwxr-xr-x    1024  r
root     getty       1236   wd /             2 drwxr-xr-x    1024  r
root     getty       1236 text /        28856334 -r-xr-xr-x   27696  r
root     getty       1236    0 /dev         56 crw-------   ttyv4 rw
root     getty       1236    1 /dev         56 crw-------   ttyv4 rw
root     getty       1236    2 /dev         56 crw-------   ttyv4 rw
root     getty       1235 root /             2 drwxr-xr-x    1024  r
root     getty       1235   wd /             2 drwxr-xr-x    1024  r
root     getty       1235 text /        28856334 -r-xr-xr-x   27696  r
root     getty       1235    0 /dev         55 crw-------   ttyv3 rw
root     getty       1235    1 /dev         55 crw-------   ttyv3 rw
root     getty       1235    2 /dev         55 crw-------   ttyv3 rw
root     getty       1234 root /             2 drwxr-xr-x    1024  r
root     getty       1234   wd /             2 drwxr-xr-x    1024  r
root     getty       1234 text /        28856334 -r-xr-xr-x   27696  r
root     getty       1234    0 /dev         54 crw-------   ttyv2 rw
root     getty       1234    1 /dev         54 crw-------   ttyv2 rw
root     getty       1234    2 /dev         54 crw-------   ttyv2 rw
root     getty       1233 root /             2 drwxr-xr-x    1024  r
root     getty       1233   wd /             2 drwxr-xr-x    1024  r
root     getty       1233 text /        28856334 -r-xr-xr-x   27696  r
root     getty       1233    0 /dev         53 crw-------   ttyv1 rw
root     getty       1233    1 /dev         53 crw-------   ttyv1 rw
root     getty       1233    2 /dev         53 crw-------   ttyv1 rw
root     getty       1232 root /             2 drwxr-xr-x    1024  r
root     getty       1232   wd /             2 drwxr-xr-x    1024  r
root     getty       1232 text /        28856334 -r-xr-xr-x   27696  r
root     getty       1232    0 /dev         52 crw-------   ttyv0 rw
root     getty       1232    1 /dev         52 crw-------   ttyv0 rw
root     getty       1232    2 /dev         52 crw-------   ttyv0 rw
root     sleep       1230 root /             2 drwxr-xr-x    1024  r
root     sleep       1230   wd /             2 drwxr-xr-x    1024  r
root     sleep       1230 text /        10457122 -r-xr-xr-x    5224  r
root     sleep       1230    0 /dev         20 crw-rw-rw-    null  r
root     sleep       1230    1* pipe ffffff001734b430 <-> ffffff001734b2d8      0 rw
root     sleep       1230    2* pipe ffffff001734b430 <-> ffffff001734b2d8      0 rw
root     logger      1229 root /             2 drwxr-xr-x    1024  r
root     logger      1229   wd /             2 drwxr-xr-x    1024  r
root     logger      1229 text /        28855019 -r-xr-xr-x   12024  r
root     logger      1229    0* pipe ffffff001734b2d8 <-> ffffff001734b430      0 rw
root     logger      1229    2 -         -         bad    -
root     sh          1228 root /             2 drwxr-xr-x    1024  r
root     sh          1228   wd /             2 drwxr-xr-x    1024  r
root     sh          1228 text /        10457119 -r-xr-xr-x  139024  r
root     sh          1228    0 /dev         20 crw-rw-rw-    null  r
root     sh          1228    1* pipe ffffff001734b430 <-> ffffff001734b2d8      0 rw
root     sh          1228    2* pipe ffffff001734b430 <-> ffffff001734b2d8      0 rw
root     cron        1168 root /             2 drwxr-xr-x    1024  r
root     cron        1168   wd /        7984134 drwxr-x---     512  r
root     cron        1168 text /        28856163 -r-xr-xr-x   39856  r
root     cron        1168    0 /dev         20 crw-rw-rw-    null rw
root     cron        1168    1 /dev         20 crw-rw-rw-    null rw
root     cron        1168    2 /dev         20 crw-rw-rw-    null rw
root     cron        1168    3 /        7984369 -rw-------       4  w
smmsp    sendmail    1161 root /             2 drwxr-xr-x    1024  r
smmsp    sendmail    1161   wd /        7984156 drwxrwx---     512  r
smmsp    sendmail    1161 text /        28855338 -r-xr-sr-x  698144  r
smmsp    sendmail    1161    0 /dev         20 crw-rw-rw-    null  r
smmsp    sendmail    1161    1 /dev         20 crw-rw-rw-    null  w
smmsp    sendmail    1161    2 /dev         20 crw-rw-rw-    null  w
smmsp    sendmail    1161    3* local dgram ffffff00173c0960 <-> ffffff001739f870
smmsp    sendmail    1161    4 /        7984287 -rw-------      50  w
root     sendmail    1154 root /             2 drwxr-xr-x    1024  r
root     sendmail    1154   wd /        7984153 drwxr-xr-x     512  r
root     sendmail    1154 text /        28855338 -r-xr-sr-x  698144  r
root     sendmail    1154    0 /dev         20 crw-rw-rw-    null  r
root     sendmail    1154    1 /dev         20 crw-rw-rw-    null  w
root     sendmail    1154    2 /dev         20 crw-rw-rw-    null  w
root     sendmail    1154    3* local dgram ffffff001739f0f0 <-> ffffff001739f870
root     sendmail    1154    4* internet stream tcp ffffff0017548000
root     sendmail    1154    5 /        7984355 -rw-------      79  w
root     sshd        1145 root /             2 drwxr-xr-x    1024  r
root     sshd        1145   wd /             2 drwxr-xr-x    1024  r
root     sshd        1145 text /        28852346 -r-xr-xr-x  259512  r
root     sshd        1145    0 /dev         20 crw-rw-rw-    null rw
root     sshd        1145    1 /dev         20 crw-rw-rw-    null rw
root     sshd        1145    2 /dev         20 crw-rw-rw-    null rw
root     sshd        1145    3* internet6 stream tcp ffffff0017549a50
root     sshd        1145    4* internet stream tcp ffffff00175496e0
_dhcp    dhclient    1144 root /        7984136 dr-xr-xr-x     512  r
_dhcp    dhclient    1144   wd /        7984136 dr-xr-xr-x     512  r
_dhcp    dhclient    1144 jail /        7984136 dr-xr-xr-x     512  r
_dhcp    dhclient    1144 text /        2449507 -r-xr-xr-x   91232  r
_dhcp    dhclient    1144    0 /dev         20 crw-rw-rw-    null rw
_dhcp    dhclient    1144    1 /dev         20 crw-rw-rw-    null rw
_dhcp    dhclient    1144    2 /dev         20 crw-rw-rw-    null rw
_dhcp    dhclient    1144    3* local dgram ffffff00173c0e10 <-> ffffff001739f870
_dhcp    dhclient    1144    5* route raw 0 ffffff0017399550
_dhcp    dhclient    1144    6* pipe ffffff00175549e0 <-> ffffff0017554888      0 rw
_dhcp    dhclient    1144    7 /        7984290 ----------    1000  w
_dhcp    dhclient    1144    8 /dev         18 crw-------     bpf rw
_dhcp    dhclient    1144    9* internet raw ip ffffff001760d000
root     dhclient    1095 root /             2 drwxr-xr-x    1024  r
root     dhclient    1095   wd /             2 drwxr-xr-x    1024  r
root     dhclient    1095 text /        2449507 -r-xr-xr-x   91232  r
root     dhclient    1095    0 /dev         20 crw-rw-rw-    null rw
root     dhclient    1095    1 /dev         20 crw-rw-rw-    null rw
root     dhclient    1095    2 /dev         20 crw-rw-rw-    null rw
root     dhclient    1095    3* local dgram ffffff00173c0e10 <-> ffffff001739f870
root     dhclient    1095    5* pipe ffffff0017554888 <-> ffffff00175549e0      0 rw
root     nfsd         935 root /             2 drwxr-xr-x    1024  r
root     nfsd         935   wd /             2 drwxr-xr-x    1024  r
root     nfsd         935 text /        28855173 -r-xr-xr-x   19808  r
root     nfsd         935    0 /dev         20 crw-rw-rw-    null rw
root     nfsd         935    1 /dev         20 crw-rw-rw-    null rw
root     nfsd         935    2 /dev         20 crw-rw-rw-    null rw
root     nfsd         934 root /             2 drwxr-xr-x    1024  r
root     nfsd         934   wd /             2 drwxr-xr-x    1024  r
root     nfsd         934 text /        28855173 -r-xr-xr-x   19808  r
root     nfsd         934    0 /dev         20 crw-rw-rw-    null rw
root     nfsd         934    1 /dev         20 crw-rw-rw-    null rw
root     nfsd         934    2 /dev         20 crw-rw-rw-    null rw
root     nfsd         934    3* internet stream tcp ffffff00173df370
root     nfsd         934    4* internet6 stream tcp ffffff00173df000
root     mountd       917 root /             2 drwxr-xr-x    1024  r
root     mountd       917   wd /             2 drwxr-xr-x    1024  r
root     mountd       917 text /        28855157 -r-xr-xr-x   42552  r
root     mountd       917    0 /dev         20 crw-rw-rw-    null rw
root     mountd       917    1 /dev         20 crw-rw-rw-    null rw
root     mountd       917    2 /dev         20 crw-rw-rw-    null rw
root     mountd       917    3 /        7984335 -rw-------       3  w
root     mountd       917    5* internet6 dgram udp ffffff001739bd20
root     mountd       917    6* internet6 stream tcp ffffff00173df6e0
root     mountd       917    7* internet dgram udp ffffff001739b690
root     mountd       917    8* internet stream tcp ffffff00175486e0
root     rpcbind      913 root /             2 drwxr-xr-x    1024  r
root     rpcbind      913   wd /             2 drwxr-xr-x    1024  r
root     rpcbind      913 text /        28855218 -r-xr-xr-x   47160  r
root     rpcbind      913    0 /dev         20 crw-rw-rw-    null rw
root     rpcbind      913    1 /dev         20 crw-rw-rw-    null rw
root     rpcbind      913    2 /dev         20 crw-rw-rw-    null rw
root     rpcbind      913    3 /        7984285 -r--r--r--       0  r
root     rpcbind      913    4* internet6 dgram udp ffffff001739ad20
root     rpcbind      913    5* local stream ffffff00173c0c30
root     rpcbind      913    6* internet6 dgram udp ffffff0002a312a0
root     rpcbind      913    7* internet6 dgram udp ffffff001739a540
root     rpcbind      913    8* internet6 stream tcp ffffff0017549000
root     rpcbind      913    9* internet dgram udp ffffff001739a7e0
root     rpcbind      913   10* internet dgram udp ffffff001739aa80
root     rpcbind      913   11* internet stream tcp ffffff0017548a50
root     syslogd      783 root /             2 drwxr-xr-x    1024  r
root     syslogd      783   wd /             2 drwxr-xr-x    1024  r
root     syslogd      783 text /        28855339 -r-xr-xr-x   39544  r
root     syslogd      783    0 /dev         20 crw-rw-rw-    null rw
root     syslogd      783    1 /dev         20 crw-rw-rw-    null rw
root     syslogd      783    2 /dev         20 crw-rw-rw-    null rw
root     syslogd      783    3 /        7984274 -rw-------       3  w
root     syslogd      783    4* local dgram ffffff00173c0d20
root     syslogd      783    5* local dgram ffffff001739f870
root     syslogd      783    6* internet6 dgram udp ffffff001739a3f0
root     syslogd      783    7* internet dgram udp ffffff001739a150
root     syslogd      783    8 /dev         31 crw-------    klog  r
root     syslogd      783   10 -         -         bad    -
root     syslogd      783   11 /        7984295 -rw-r--r--  208250  w
root     syslogd      783   12 /        7984270 -rw-------      55  w
root     syslogd      783   13 /        7984258 -rw-------   11794  w
root     syslogd      783   14 /        7984265 -rw-r-----    1926  w
root     syslogd      783   15 /        7984263 -rw-r--r--      55  w
root     syslogd      783   16 /        7984271 -rw-------      55  w
root     syslogd      783   17 /        7984259 -rw-------   93656  w
root     syslogd      783   18 /        7984262 -rw-------     118  w
root     syslogd      783   19 /        7984269 -rw-r-----      55  w
root     devd         636 root /             2 drwxr-xr-x    1024  r
root     devd         636   wd /             2 drwxr-xr-x    1024  r
root     devd         636 text /        2449505 -r-xr-xr-x  453040  r
root     devd         636    0 /dev         20 crw-rw-rw-    null rw
root     devd         636    1 /dev         20 crw-rw-rw-    null rw
root     devd         636    2 /dev         20 crw-rw-rw-    null rw
root     devd         636    3 /        777227 drwxr-xr-x     512  r
root     devd         636    4 /dev          4 crw-------  devctl  r
root     devd         636    5* local stream ffffff001739fc30
root     devd         636    6 /        7984272 -rw-------       3  w
root     zfskern       39 root /             2 drwxr-xr-x    1024  r
root     zfskern       39   wd /             2 drwxr-xr-x    1024  r
root     init           1 root /             2 drwxr-xr-x    1024  r
root     init           1   wd /             2 drwxr-xr-x    1024  r
root     init           1 text /        2449445 -r-xr-xr-x  779888  r
root     kernel         0 root /             2 drwxr-xr-x    1024  r
root     kernel         0   wd /             2 drwxr-xr-x    1024  r

------------------------------------------------------------------------
dmesg

Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-CURRENT #0 r212074M: Sun Sep 12 18:48:36 UTC 2010
    root@xanadu:/usr/obj/head_old/sys/DTRACE2 amd64
WARNING: WITNESS option enabled, expect reduced performance.
CPU: AMD Opteron(tm) Processor 250 (2411.16-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0x20f51  Family = f  Model = 25  Stepping = 1
  Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
  Features2=0x1<SSE3>
  AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow!>
  AMD Features2=0x1<LAHF>
real memory  = 34359738368 (32768 MB)
avail memory = 3344609280 (3189 MB)
Event timer "LAPIC" frequency 0 Hz quality 500
ACPI APIC Table: <PTLTD  	 APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 2 package(s) x 1 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ioapic0 <Version 1.1> irqs 0-23 on motherboard
ioapic1 <Version 1.1> irqs 24-27 on motherboard
ioapic2 <Version 1.1> irqs 28-31 on motherboard
kbd1 at kbdmux0
acpi0: <PTLTD   RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x8008-0x800b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <memory> at device 0.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
pci0: <serial bus, SMBus> at device 1.1 (no driver attached)
ohci0: <OHCI (generic) USB controller> mem 0xdd000000-0xdd000fff irq 20 at device 2.0 on pci0
ohci0: [ITHREAD]
usbus0: <OHCI (generic) USB controller> on ohci0
ehci0: <NVIDIA nForce4 USB 2.0 controller> mem 0xdd001000-0xdd0010ff irq 21 at device 2.1 on pci0
ehci0: [ITHREAD]
usbus1: EHCI version 1.0
usbus1: <NVIDIA nForce4 USB 2.0 controller> on ehci0
atapci0: <nVidia nForce CK804 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1400-0x140f at device 6.0 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
atapci1: <nVidia nForce CK804 SATA300 controller> port 0x1440-0x1447,0x1434-0x1437,0x1438-0x143f,0x1430-0x1433,0x1410-0x141f mem 0xdd002000-0xdd002fff irq 22 at device 7.0 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
atapci2: <nVidia nForce CK804 SATA300 controller> port 0x1458-0x145f,0x144c-0x144f,0x1450-0x1457,0x1448-0x144b,0x1420-0x142f mem 0xdd003000-0xdd003fff irq 23 at device 8.0 on pci0
atapci2: [ITHREAD]
ata4: <ATA channel 0> on atapci2
ata4: [ITHREAD]
ata5: <ATA channel 1> on atapci2
ata5: [ITHREAD]
pcib1: <ACPI PCI-PCI bridge> at device 9.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0x2000-0x20ff mem 0xde000000-0xdeffffff,0xdd100000-0xdd100fff irq 16 at device 7.0 on pci1
pcib2: <ACPI PCI-PCI bridge> at device 14.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci8: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 10.0 on pci8
pci9: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 11.0 on pci8
pci10: <ACPI PCI bus> on pcib5
bge0: <Broadcom Gigabit Ethernet Controller, ASIC rev. 0x002003> mem 0xdf310000-0xdf31ffff,0xdf300000-0xdf30ffff irq 28 at device 9.0 on pci10
miibus0: <MII bus> on bge0
brgphy0: <BCM5704 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge0: Ethernet address: 00:e0:81:40:29:d2
bge0: [ITHREAD]
bge1: <Broadcom Gigabit Ethernet Controller, ASIC rev. 0x002003> mem 0xdf330000-0xdf33ffff,0xdf320000-0xdf32ffff irq 29 at device 9.1 on pci10
miibus1: <MII bus> on bge1
brgphy1: <BCM5704 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge1: Ethernet address: 00:e0:81:40:29:d3
bge1: [ITHREAD]
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
attimer0: Can't map interrupt.
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: [FILTER]
Event timer "RTC" frequency 32768 Hz quality 0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
ppc1: <Parallel port> port 0x278-0x27f irq 5 on acpi0
ppc1: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc1: [ITHREAD]
ppbus0: <Parallel port bus> on ppc1
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xcafff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
powernow0: <Cool`n'Quiet K8> on cpu0
powernow1: <Cool`n'Quiet K8> on cpu1
Starting kernel event timers: LAPIC @ 1000Hz, RTC @ 128Hz
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
ad0: 238475MB <Seagate ST3250823A 3.03> at ata0-master UDMA100 
ugen0.1: <nVidia> at usbus0
uhub0: <nVidia OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <nVidia> at usbus1
uhub1: <nVidia EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ad4: 1430799MB <WDC WD15EARS-00Z5B1 80.00A80> at ata2-master UDMA100 SATA 3Gb/s
uhub0: 10 ports with 10 removable, self powered
ad6: 1430799MB <Seagate ST31500341AS CC1H> at ata3-master UDMA100 SATA 3Gb/s
ad8: 1907729MB <Hitachi HDS722020ALA330 JKAOA3EA> at ata4-master UDMA100 SATA 3Gb/s
ad10: 1907729MB <Hitachi HDS722020ALA330 JKAOA20N> at ata5-master UDMA100 SATA 3Gb/s
SMP: AP CPU #1 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
Root mount waiting for: usbus1
Root mount waiting for: usbus1
uhub1: 10 ports with 10 removable, self powered
Trying to mount root from ufs:/dev/ad0s1a
Setting hostuuid: 5c060463-baf1-11df-8536-00e0814029d2.
Setting hostid: 0x38de9d4e.
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version 5
ZFS storage pool version 28
unrecognized command 'volinit'
usage: zfs command args ...
where 'command' is one of the following:

	create [-p] [-o property=value] ... <filesystem>
	create [-ps] [-b blocksize] [-o property=value] ... -V <size> <volume>
	destroy [-rRf] <filesystem|volume>
	destroy [-rRd] <snapshot>

	snapshot [-r] [-o property=value] ... <filesystem@snapname|volume@snapname>
	rollback [-rRf] <snapshot>
	clone [-p] [-o property=value] ... <snapshot> <filesystem|volume>
	promote <clone-filesystem>
	rename <filesystem|volume|snapshot> <filesystem|volume|snapshot>
	rename -p <filesystem|volume> <filesystem|volume>
	rename -r <snapshot> <snapshot>

	list [-rH][-d max] [-o property[,...]] [-t type[,...]] [-s property] ...
	    [-S property] ... [filesystem|volume|snapshot] ...

	set <property=value> <filesystem|volume|snapshot> ...
	get [-rHp] [-d max] [-o "all" | field[,...]] [-s source[,...]]
	    <"all" | property[,...]> [filesystem|volume|snapshot] ...
	inherit [-rS] <property> <filesystem|volume|snapshot> ...
	upgrade [-v]
	upgrade [-r] [-V version] <-a | filesystem ...>
	userspace [-hniHp] [-o field[,...]] [-sS field] ... [-t type[,...]]
	    <filesystem|snapshot>
	groupspace [-hniHpU] [-o field[,...]] [-sS field] ... [-t type[,...]]
	    <filesystem|snapshot>

	mount
	mount [-vO] [-o opts] <-a | filesystem>
	unmount [-f] <-a | filesystem|mountpoint>
	share <-a | filesystem>
	unshare <-a | filesystem|mountpoint>

	send [-RDp] [-[iI] snapshot] <snapshot>
	receive [-vnFu] <filesystem|volume|snapshot>
	receive [-vnFu] [-d | -e] <filesystem>

	allow <filesystem|volume>
	allow [-ldug] <"everyone"|user|group>[,...] <perm|@setname>[,...]
	    <filesystem|volume>
	allow [-ld] -e <perm|@setname>[,...] <filesystem|volume>
	allow -c <perm|@setname>[,...] <filesystem|volume>
	allow -s @setname <perm|@setname>[,...] <filesystem|volume>

	unallow [-rldug] <"everyone"|user|group>[,...]
	    [<perm|@setname>[,...]] <filesystem|volume>
	unallow [-rld] -e [<perm|@setname>[,...]] <filesystem|volume>
	unallow [-r] -c [<perm|@setname>[,...]] <filesystem|volume>
	unallow [-r] -s @setname [<perm|@setname>[,...]] <filesystem|volume>

	hold [-r] <tag> <snapshot> ...
	holds [-r] <snapshot> ...
	release [-r] <tag> <snapshot> ...
	diff [-FHt] <snapshot> [snapshot|filesystem]

	jail <jailid> <filesystem>
	unjail <jailid> <filesystem>

Each dataset is of the form: pool/[dataset/]*dataset[@name]

For the property list, run: zfs set|get

For the delegated permission list, run: zfs allow|unallow
lock order reversal:
 1st 0xffffff0002ca9a98 db->db_mtx (db->db_mtx) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:2009
 2nd 0xffffff0002cb40e8 dn->dn_mtx (dn->dn_mtx) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:1174
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
dnode_rele() at dnode_rele+0x4e
dsl_deadlist_close() at dsl_deadlist_close+0x47
dsl_dataset_evict() at dsl_dataset_evict+0x7d
dbuf_evict_user() at dbuf_evict_user+0x55
dbuf_rele_and_unlock() at dbuf_rele_and_unlock+0x154
dsl_pool_open() at dsl_pool_open+0x1b5
spa_load() at spa_load+0x54f
spa_load_best() at spa_load_best+0x52
spa_open_common() at spa_open_common+0x14a
pool_status_check() at pool_status_check+0x21
zfsdev_ioctl() at zfsdev_ioctl+0x121
devfs_ioctl_f() at devfs_ioctl_f+0x7a
kern_ioctl() at kern_ioctl+0xbe
ioctl() at ioctl+0xfd
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x80110268c, rsp = 0x7fffffffccd8, rbp = 0x4000 ---
lock order reversal:
 1st 0xffffff0002ca9058 db->db_mtx (db->db_mtx) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c:396
 2nd 0xffffff0002b47660 os->os_lock (os->os_lock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:439
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
dnode_destroy() at dnode_destroy+0x3e
dnode_buf_pageout() at dnode_buf_pageout+0x9d
dbuf_evict_user() at dbuf_evict_user+0x55
dbuf_clear() at dbuf_clear+0x58
dnode_evict_dbufs() at dnode_evict_dbufs+0x98
dmu_objset_evict_dbufs() at dmu_objset_evict_dbufs+0x11c
dmu_objset_evict() at dmu_objset_evict+0xa3
dsl_pool_close() at dsl_pool_close+0x6a
spa_unload() at spa_unload+0x79
spa_load() at spa_load+0x62d
spa_load_best() at spa_load_best+0x52
spa_open_common() at spa_open_common+0x14a
pool_status_check() at pool_status_check+0x21
zfsdev_ioctl() at zfsdev_ioctl+0x121
devfs_ioctl_f() at devfs_ioctl_f+0x7a
kern_ioctl() at kern_ioctl+0xbe
ioctl() at ioctl+0xfd
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x80110268c, rsp = 0x7fffffffccd8, rbp = 0x4000 ---
Entropy harvesting:
 interrupts
 ethernet
 point_to_point
 kickstart
.
Starting file system checks:
/dev/ad0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ad0s1a: clean, 107698027 free (70491 frags, 13453442 blocks, 0.1% fragmentation)
Mounting local file systems:
.
lock order reversal:
 1st 0xffffff0002e2e100 sa->sa_lock (sa->sa_lock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:995
 2nd 0xffffff0002df7428 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:205
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_slock() at _sx_slock+0x54
dnode_verify() at dnode_verify+0x7c
dnode_hold_impl() at dnode_hold_impl+0x94
dmu_buf_hold() at dmu_buf_hold+0x48
zap_lockdir() at zap_lockdir+0x74
zap_lookup_norm() at zap_lookup_norm+0x45
zap_lookup() at zap_lookup+0x2e
sa_setup() at sa_setup+0x214
zfsvfs_create() at zfsvfs_create+0x218
zfs_mount() at zfs_mount+0xe6
vfs_donmount() at vfs_donmount+0xcde
nmount() at nmount+0x63
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (378, FreeBSD ELF64, nmount), rip = 0x801061d4c, rsp = 0x7fffffffcce8, rbp = 0x80185f10c ---
lock order reversal:
 1st 0xffffff0002e2e100 sa->sa_lock (sa->sa_lock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:995
 2nd 0xffffff0002cc4a60 os->os_lock (os->os_lock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:415
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
dnode_create() at dnode_create+0x130
dnode_hold_impl() at dnode_hold_impl+0x56a
dmu_buf_hold() at dmu_buf_hold+0x48
zap_lockdir() at zap_lockdir+0x74
zap_lookup_norm() at zap_lookup_norm+0x45
zap_lookup() at zap_lookup+0x2e
sa_setup() at sa_setup+0x214
zfsvfs_create() at zfsvfs_create+0x218
zfs_mount() at zfs_mount+0xe6
vfs_donmount() at vfs_donmount+0xcde
nmount() at nmount+0x63
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (378, FreeBSD ELF64, nmount), rip = 0x801061d4c, rsp = 0x7fffffffcce8, rbp = 0x80185f10c ---
lock order reversal:
 1st 0xffffff0002e2e100 sa->sa_lock (sa->sa_lock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:995
 2nd 0xffffff0002ca5998 zap->zap_rwlock (zap->zap_rwlock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:375
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
zap_lockdir() at zap_lockdir+0x34c
zap_lookup_norm() at zap_lookup_norm+0x45
zap_lookup() at zap_lookup+0x2e
sa_setup() at sa_setup+0x214
zfsvfs_create() at zfsvfs_create+0x218
zfs_mount() at zfs_mount+0xe6
vfs_donmount() at vfs_donmount+0xcde
nmount() at nmount+0x63
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (378, FreeBSD ELF64, nmount), rip = 0x801061d4c, rsp = 0x7fffffffcce8, rbp = 0x80185f10c ---
lock order reversal:
 1st 0xffffff0002e24098 zfs (zfs) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/gfs.c:488
 2nd 0xffffff0002dc9350 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:1116
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
zfs_zget() at zfs_zget+0x241
zfs_root() at zfs_root+0x50
zfsctl_create() at zfsctl_create+0x81
zfs_mount() at zfs_mount+0x5f4
vfs_donmount() at vfs_donmount+0xcde
nmount() at nmount+0x63
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (378, FreeBSD ELF64, nmount), rip = 0x801061d4c, rsp = 0x7fffffffcce8, rbp = 0x80185f10c ---
Setting hostname: xanadu
.
Starting Network: lo0 bge1.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 
	inet6 ::1 prefixlen 128 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
	ether 00:e0:81:40:29:d3
	inet6 fe80::2e0:81ff:fe40:29d3%bge1 prefixlen 64 tentative scopeid 0x2 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (none)
	status: no carrier
Starting devd.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
32-bit compatibility ldconfig path: /usr/lib32
Creating and/or trimming log files
.
Starting syslogd.
No core dumps found.
Clearing /tmp (X related).
Starting rpcbind.
Starting mountd.
Starting nfsd.
Updating motd:
bge1: link state changed to UP
.
Configuring syscons:
 blanktime
.
Starting sshd.
Sep 13 01:38:04 xanadu sm-mta[1149]: My unqualified host name (xanadu) unknown; sleeping for retry
Sep 13 01:39:04 xanadu sm-mta[1149]: unable to qualify my own domain name (xanadu) -- using short name
Sep 13 01:39:04 xanadu sm-msp-queue[1156]: My unqualified host name (xanadu) unknown; sleeping for retry
Sep 13 01:40:04 xanadu sm-msp-queue[1156]: unable to qualify my own domain name (xanadu) -- using short name
Starting cron.
Starting background file system checks in 60 seconds.

Mon Sep 13 01:40:04 UTC 2010
Sep 13 01:43:24 xanadu sshd[1142]: error: accept: Software caused connection abort
lock order reversal:
 1st 0xffffff0002dc9200 zfsvfs->z_teardown_inactive_lock (zfsvfs->z_teardown_inactive_lock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4503
 2nd 0xffffff0002dc93f0 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:1345
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
zfs_zinactive() at zfs_zinactive+0x8b
zfs_inactive() at zfs_inactive+0x7e
zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a
VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xd9
vinactive() at vinactive+0x90
vputx() at vputx+0x2dc
nfsrv3_access() at nfsrv3_access+0x2ca
nfssvc_program() at nfssvc_program+0x1a6
svc_run_internal() at svc_run_internal+0x5fb
svc_thread_start() at svc_thread_start+0xb
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0xc, rip = 0x8006a2bec, rsp = 0x7fffffffe698, rbp = 0x5 ---
lock order reversal:
 1st 0xffffff006268cb80 zp->z_name_lock (zp->z_name_lock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:220
 2nd 0xffffff0002dc9690 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:1116
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
zfs_zget() at zfs_zget+0x241
zfs_dirent_lock() at zfs_dirent_lock+0x4c1
zfs_dirlook() at zfs_dirlook+0x90
zfs_lookup() at zfs_lookup+0x2d4
zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xd7
vfs_cache_lookup() at vfs_cache_lookup+0xf0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xdf
lookup() at lookup+0x3d3
nfs_namei() at nfs_namei+0x3e3
nfsrv_lookup() at nfsrv_lookup+0x216
nfssvc_program() at nfssvc_program+0x1a6
svc_run_internal() at svc_run_internal+0x5fb
svc_run() at svc_run+0x8f
nfssvc_nfsd() at nfssvc_nfsd+0xa2
nfssvc_nfsserver() at nfssvc_nfsserver+0x5b
nfssvc() at nfssvc+0x73
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x8006a2bec, rsp = 0x7fffffffe698, rbp = 0x5 ---
lock order reversal:
 1st 0xffffff0002cb12e8 db->db_mtx (db->db_mtx) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1242
 2nd 0xffffff00c6127b38 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1246
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
dbuf_dirty() at dbuf_dirty+0x6ce
dbuf_dirty() at dbuf_dirty+0x52b
dbuf_dirty() at dbuf_dirty+0x52b
dbuf_dirty() at dbuf_dirty+0x52b
dbuf_dirty() at dbuf_dirty+0x52b
dbuf_dirty() at dbuf_dirty+0x52b
dnode_setdirty() at dnode_setdirty+0x1a5
dbuf_dirty() at dbuf_dirty+0x593
sa_attr_op() at sa_attr_op+0x32c
sa_bulk_update_impl() at sa_bulk_update_impl+0x7c
sa_bulk_update() at sa_bulk_update+0x50
zfs_freebsd_setattr() at zfs_freebsd_setattr+0x1b61
VOP_SETATTR_APV() at VOP_SETATTR_APV+0xd3
nfsrv_setattr() at nfsrv_setattr+0x80d
nfssvc_program() at nfssvc_program+0x1a6
svc_run_internal() at svc_run_internal+0x5fb
svc_thread_start() at svc_thread_start+0xb
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0xc, rip = 0x8006a2bec, rsp = 0x7fffffffe698, rbp = 0x5 ---
lock order reversal:
 1st 0xffffff00c61060c0 l->l_rwlock (l->l_rwlock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:522
 2nd 0xffffff00c6155000 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:611
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_slock() at _sx_slock+0x54
dbuf_read() at dbuf_read+0x319
dbuf_will_dirty() at dbuf_will_dirty+0x80
zap_get_leaf_byblk() at zap_get_leaf_byblk+0x1dd
zap_deref_leaf() at zap_deref_leaf+0xb6
fzap_add_cd() at fzap_add_cd+0x75
zap_add() at zap_add+0x105
zfs_link_create() at zfs_link_create+0x332
zfs_freebsd_create() at zfs_freebsd_create+0x762
VOP_CREATE_APV() at VOP_CREATE_APV+0xd7
nfsrv_create() at nfsrv_create+0x906
nfssvc_program() at nfssvc_program+0x1a6
svc_run_internal() at svc_run_internal+0x5fb
svc_thread_start() at svc_thread_start+0xb
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0xc, rip = 0x8006a2bec, rsp = 0x7fffffffe698, rbp = 0x5 ---
lock order reversal:
 1st 0xffffff00c61060c0 l->l_rwlock (l->l_rwlock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:522
 2nd 0xffffff0002cc4a60 os->os_lock (os->os_lock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:1228
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
dnode_setdirty() at dnode_setdirty+0xc9
dbuf_dirty() at dbuf_dirty+0x593
dbuf_dirty() at dbuf_dirty+0x52b
zap_get_leaf_byblk() at zap_get_leaf_byblk+0x1dd
zap_deref_leaf() at zap_deref_leaf+0xb6
fzap_add_cd() at fzap_add_cd+0x75
zap_add() at zap_add+0x105
zfs_link_create() at zfs_link_create+0x332
zfs_freebsd_create() at zfs_freebsd_create+0x762
VOP_CREATE_APV() at VOP_CREATE_APV+0xd7
nfsrv_create() at nfsrv_create+0x906
nfssvc_program() at nfssvc_program+0x1a6
svc_run_internal() at svc_run_internal+0x5fb
svc_thread_start() at svc_thread_start+0xb
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0xc, rip = 0x8006a2bec, rsp = 0x7fffffffe698, rbp = 0x5 ---
lock order reversal:
 1st 0xffffff00c6227a38 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:2243
 2nd 0xffffff00c621c850 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:611
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_slock() at _sx_slock+0x54
dbuf_read() at dbuf_read+0x319
dbuf_sync_list() at dbuf_sync_list+0x642
dbuf_sync_list() at dbuf_sync_list+0x17f
dnode_sync() at dnode_sync+0xe9c
dmu_objset_sync_dnodes() at dmu_objset_sync_dnodes+0x92
dmu_objset_sync() at dmu_objset_sync+0x19d
dsl_pool_sync() at dsl_pool_sync+0xe5
spa_sync() at spa_sync+0x33f
txg_sync_thread() at txg_sync_thread+0x147
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff806273fcf0, rbp = 0 ---
lock order reversal:
 1st 0xffffff00c6174000 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_tx.c:440
 2nd 0xffffff0002ca4418 zap->zap_rwlock (zap->zap_rwlock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:490
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_slock() at _sx_slock+0x54
zap_lockdir() at zap_lockdir+0x111
zap_prefetch_uint64() at zap_prefetch_uint64+0x36
ddt_prefetch() at ddt_prefetch+0xba
dsl_dataset_block_freeable() at dsl_dataset_block_freeable+0x3c
dmu_tx_hold_free() at dmu_tx_hold_free+0x56f
dmu_free_long_range_impl() at dmu_free_long_range_impl+0x114
dmu_free_long_range() at dmu_free_long_range+0x4c
zfs_rmnode() at zfs_rmnode+0x89
zfs_inactive() at zfs_inactive+0x7e
zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a
VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xd9
vinactive() at vinactive+0x90
vputx() at vputx+0x2dc
zfs_freebsd_rename() at zfs_freebsd_rename+0x11b
VOP_RENAME_APV() at VOP_RENAME_APV+0xbf
nfsrv_rename() at nfsrv_rename+0xb52
nfssvc_program() at nfssvc_program+0x1a6
svc_run_internal() at svc_run_internal+0x5fb
svc_run() at svc_run+0x8f
nfssvc_nfsd() at nfssvc_nfsd+0xa2
nfssvc_nfsserver() at nfssvc_nfsserver+0x5b
nfssvc() at nfssvc+0x73
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x8006a2bec, rsp = 0x7fffffffe698, rbp = 0x5 ---
panic: Solaris(panic): zfs: accessing past end of object ffffff804423bd50/57 (size=1143193255 access=0+18446743525096930672)

cpuid = 1
KDB: enter: panic
panic: bufwrite: buffer is not busy???
cpuid = 1
KDB: enter: panic
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-CURRENT #0 r212074M: Sun Sep 12 18:48:36 UTC 2010
    root@xanadu:/usr/obj/head_old/sys/DTRACE2 amd64
WARNING: WITNESS option enabled, expect reduced performance.
CPU: AMD Opteron(tm) Processor 250 (2411.16-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0x20f51  Family = f  Model = 25  Stepping = 1
  Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
  Features2=0x1<SSE3>
  AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow!>
  AMD Features2=0x1<LAHF>
real memory  = 34359738368 (32768 MB)
avail memory = 3344609280 (3189 MB)
Event timer "LAPIC" frequency 0 Hz quality 500
ACPI APIC Table: <PTLTD  	 APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 2 package(s) x 1 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ioapic0 <Version 1.1> irqs 0-23 on motherboard
ioapic1 <Version 1.1> irqs 24-27 on motherboard
ioapic2 <Version 1.1> irqs 28-31 on motherboard
kbd1 at kbdmux0
acpi0: <PTLTD   RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x8008-0x800b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <memory> at device 0.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
pci0: <serial bus, SMBus> at device 1.1 (no driver attached)
ohci0: <OHCI (generic) USB controller> mem 0xdd000000-0xdd000fff irq 20 at device 2.0 on pci0
ohci0: [ITHREAD]
usbus0: <OHCI (generic) USB controller> on ohci0
ehci0: <NVIDIA nForce4 USB 2.0 controller> mem 0xdd001000-0xdd0010ff irq 21 at device 2.1 on pci0
ehci0: [ITHREAD]
usbus1: EHCI version 1.0
usbus1: <NVIDIA nForce4 USB 2.0 controller> on ehci0
atapci0: <nVidia nForce CK804 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1400-0x140f at device 6.0 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
atapci1: <nVidia nForce CK804 SATA300 controller> port 0x1440-0x1447,0x1434-0x1437,0x1438-0x143f,0x1430-0x1433,0x1410-0x141f mem 0xdd002000-0xdd002fff irq 22 at device 7.0 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
atapci2: <nVidia nForce CK804 SATA300 controller> port 0x1458-0x145f,0x144c-0x144f,0x1450-0x1457,0x1448-0x144b,0x1420-0x142f mem 0xdd003000-0xdd003fff irq 23 at device 8.0 on pci0
atapci2: [ITHREAD]
ata4: <ATA channel 0> on atapci2
ata4: [ITHREAD]
ata5: <ATA channel 1> on atapci2
ata5: [ITHREAD]
pcib1: <ACPI PCI-PCI bridge> at device 9.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0x2000-0x20ff mem 0xde000000-0xdeffffff,0xdd100000-0xdd100fff irq 16 at device 7.0 on pci1
pcib2: <ACPI PCI-PCI bridge> at device 14.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci8: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 10.0 on pci8
pci9: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 11.0 on pci8
pci10: <ACPI PCI bus> on pcib5
bge0: <Broadcom Gigabit Ethernet Controller, ASIC rev. 0x002003> mem 0xdf310000-0xdf31ffff,0xdf300000-0xdf30ffff irq 28 at device 9.0 on pci10
miibus0: <MII bus> on bge0
brgphy0: <BCM5704 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge0: Ethernet address: 00:e0:81:40:29:d2
bge0: [ITHREAD]
bge1: <Broadcom Gigabit Ethernet Controller, ASIC rev. 0x002003> mem 0xdf330000-0xdf33ffff,0xdf320000-0xdf32ffff irq 29 at device 9.1 on pci10
miibus1: <MII bus> on bge1
brgphy1: <BCM5704 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge1: Ethernet address: 00:e0:81:40:29:d3
bge1: [ITHREAD]
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
attimer0: Can't map interrupt.
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: [FILTER]
Event timer "RTC" frequency 32768 Hz quality 0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
ppc1: <Parallel port> port 0x278-0x27f irq 5 on acpi0
ppc1: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc1: [ITHREAD]
ppbus0: <Parallel port bus> on ppc1
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xcafff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
powernow0: <Cool`n'Quiet K8> on cpu0
powernow1: <Cool`n'Quiet K8> on cpu1
Starting kernel event timers: LAPIC @ 1000Hz, RTC @ 128Hz
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
ad0: 238475MB <Seagate ST3250823A 3.03> at ata0-master UDMA100 
ugen0.1: <nVidia> at usbus0
uhub0: <nVidia OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <nVidia> at usbus1
uhub1: <nVidia EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ad4: 1430799MB <WDC WD15EARS-00Z5B1 80.00A80> at ata2-master UDMA100 SATA 3Gb/s
uhub0: 10 ports with 10 removable, self powered
ad6: 1430799MB <Seagate ST31500341AS CC1H> at ata3-master UDMA100 SATA 3Gb/s
ad8: 1907729MB <Hitachi HDS722020ALA330 JKAOA3EA> at ata4-master UDMA100 SATA 3Gb/s
ad10: 1907729MB <Hitachi HDS722020ALA330 JKAOA20N> at ata5-master UDMA100 SATA 3Gb/s
SMP: AP CPU #1 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
Root mount waiting for: usbus1
Root mount waiting for: usbus1
uhub1: 10 ports with 10 removable, self powered
Trying to mount root from ufs:/dev/ad0s1a
WARNING: / was not properly dismounted
Setting hostuuid: 5c060463-baf1-11df-8536-00e0814029d2.
Setting hostid: 0x38de9d4e.
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version 5
ZFS storage pool version 28
unrecognized command 'volinit'
usage: zfs command args ...
where 'command' is one of the following:

	create [-p] [-o property=value] ... <filesystem>
	create [-ps] [-b blocksize] [-o property=value] ... -V <size> <volume>
	destroy [-rRf] <filesystem|volume>
	destroy [-rRd] <snapshot>

	snapshot [-r] [-o property=value] ... <filesystem@snapname|volume@snapname>
	rollback [-rRf] <snapshot>
	clone [-p] [-o property=value] ... <snapshot> <filesystem|volume>
	promote <clone-filesystem>
	rename <filesystem|volume|snapshot> <filesystem|volume|snapshot>
	rename -p <filesystem|volume> <filesystem|volume>
	rename -r <snapshot> <snapshot>

	list [-rH][-d max] [-o property[,...]] [-t type[,...]] [-s property] ...
	    [-S property] ... [filesystem|volume|snapshot] ...

	set <property=value> <filesystem|volume|snapshot> ...
	get [-rHp] [-d max] [-o "all" | field[,...]] [-s source[,...]]
	    <"all" | property[,...]> [filesystem|volume|snapshot] ...
	inherit [-rS] <property> <filesystem|volume|snapshot> ...
	upgrade [-v]
	upgrade [-r] [-V version] <-a | filesystem ...>
	userspace [-hniHp] [-o field[,...]] [-sS field] ... [-t type[,...]]
	    <filesystem|snapshot>
	groupspace [-hniHpU] [-o field[,...]] [-sS field] ... [-t type[,...]]
	    <filesystem|snapshot>

	mount
	mount [-vO] [-o opts] <-a | filesystem>
	unmount [-f] <-a | filesystem|mountpoint>
	share <-a | filesystem>
	unshare <-a | filesystem|mountpoint>

	send [-RDp] [-[iI] snapshot] <snapshot>
	receive [-vnFu] <filesystem|volume|snapshot>
	receive [-vnFu] [-d | -e] <filesystem>

	allow <filesystem|volume>
	allow [-ldug] <"everyone"|user|group>[,...] <perm|@setname>[,...]
	    <filesystem|volume>
	allow [-ld] -e <perm|@setname>[,...] <filesystem|volume>
	allow -c <perm|@setname>[,...] <filesystem|volume>
	allow -s @setname <perm|@setname>[,...] <filesystem|volume>

	unallow [-rldug] <"everyone"|user|group>[,...]
	    [<perm|@setname>[,...]] <filesystem|volume>
	unallow [-rld] -e [<perm|@setname>[,...]] <filesystem|volume>
	unallow [-r] -c [<perm|@setname>[,...]] <filesystem|volume>
	unallow [-r] -s @setname [<perm|@setname>[,...]] <filesystem|volume>

	hold [-r] <tag> <snapshot> ...
	holds [-r] <snapshot> ...
	release [-r] <tag> <snapshot> ...
	diff [-FHt] <snapshot> [snapshot|filesystem]

	jail <jailid> <filesystem>
	unjail <jailid> <filesystem>

Each dataset is of the form: pool/[dataset/]*dataset[@name]

For the property list, run: zfs set|get

For the delegated permission list, run: zfs allow|unallow
lock order reversal:
 1st 0xffffff0002ca6be0 db->db_mtx (db->db_mtx) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:2009
 2nd 0xffffff0002caf510 dn->dn_mtx (dn->dn_mtx) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:1174
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
dnode_rele() at dnode_rele+0x4e
dsl_deadlist_close() at dsl_deadlist_close+0x47
dsl_dataset_evict() at dsl_dataset_evict+0x7d
dbuf_evict_user() at dbuf_evict_user+0x55
dbuf_rele_and_unlock() at dbuf_rele_and_unlock+0x154
dsl_pool_open() at dsl_pool_open+0x1b5
spa_load() at spa_load+0x54f
spa_load_best() at spa_load_best+0x52
spa_open_common() at spa_open_common+0x14a
pool_status_check() at pool_status_check+0x21
zfsdev_ioctl() at zfsdev_ioctl+0x121
devfs_ioctl_f() at devfs_ioctl_f+0x7a
kern_ioctl() at kern_ioctl+0xbe
ioctl() at ioctl+0xfd
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x80110268c, rsp = 0x7fffffffccd8, rbp = 0x4000 ---
lock order reversal:
 1st 0xffffff0002ca6058 db->db_mtx (db->db_mtx) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c:396
 2nd 0xffffff0002c33660 os->os_lock (os->os_lock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:439
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
dnode_destroy() at dnode_destroy+0x3e
dnode_buf_pageout() at dnode_buf_pageout+0x9d
dbuf_evict_user() at dbuf_evict_user+0x55
dbuf_clear() at dbuf_clear+0x58
dnode_evict_dbufs() at dnode_evict_dbufs+0x98
dmu_objset_evict_dbufs() at dmu_objset_evict_dbufs+0x11c
dmu_objset_evict() at dmu_objset_evict+0xa3
dsl_pool_close() at dsl_pool_close+0x6a
spa_unload() at spa_unload+0x79
spa_load() at spa_load+0x62d
spa_load_best() at spa_load_best+0x52
spa_open_common() at spa_open_common+0x14a
pool_status_check() at pool_status_check+0x21
zfsdev_ioctl() at zfsdev_ioctl+0x121
devfs_ioctl_f() at devfs_ioctl_f+0x7a
kern_ioctl() at kern_ioctl+0xbe
ioctl() at ioctl+0xfd
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x80110268c, rsp = 0x7fffffffccd8, rbp = 0x4000 ---
Entropy harvesting:
 interrupts
 ethernet
 point_to_point
 kickstart
.
Starting file system checks:
/dev/ad0s1a: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
/dev/ad0s1a: SUMMARY INFORMATION BAD (SALVAGED)
/dev/ad0s1a: BLK(S) MISSING IN BIT MAPS (SALVAGED)
/dev/ad0s1a: 1168789 files, 8017603 used, 107698023 free (70487 frags, 13453442 blocks, 0.1% fragmentation)
Mounting local file systems:
.
lock order reversal:
 1st 0xffffff0002a23e00 sa->sa_lock (sa->sa_lock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:995
 2nd 0xffffff0002cb3000 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:205
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_slock() at _sx_slock+0x54
dnode_verify() at dnode_verify+0x7c
dnode_hold_impl() at dnode_hold_impl+0x94
dmu_buf_hold() at dmu_buf_hold+0x48
zap_lockdir() at zap_lockdir+0x74
zap_lookup_norm() at zap_lookup_norm+0x45
zap_lookup() at zap_lookup+0x2e
sa_setup() at sa_setup+0x214
zfsvfs_create() at zfsvfs_create+0x218
zfs_mount() at zfs_mount+0xe6
vfs_donmount() at vfs_donmount+0xcde
nmount() at nmount+0x63
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (378, FreeBSD ELF64, nmount), rip = 0x801061d4c, rsp = 0x7fffffffcce8, rbp = 0x80185f10c ---
lock order reversal:
 1st 0xffffff0002a23e00 sa->sa_lock (sa->sa_lock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:995
 2nd 0xffffff0002d0ce60 os->os_lock (os->os_lock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:415
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
dnode_create() at dnode_create+0x130
dnode_hold_impl() at dnode_hold_impl+0x56a
dmu_buf_hold() at dmu_buf_hold+0x48
zap_lockdir() at zap_lockdir+0x74
zap_lookup_norm() at zap_lookup_norm+0x45
zap_lookup() at zap_lookup+0x2e
sa_setup() at sa_setup+0x214
zfsvfs_create() at zfsvfs_create+0x218
zfs_mount() at zfs_mount+0xe6
vfs_donmount() at vfs_donmount+0xcde
nmount() at nmount+0x63
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (378, FreeBSD ELF64, nmount), rip = 0x801061d4c, rsp = 0x7fffffffcce8, rbp = 0x80185f10c ---
lock order reversal:
 1st 0xffffff0002a23e00 sa->sa_lock (sa->sa_lock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:995
 2nd 0xffffff0002e5f198 zap->zap_rwlock (zap->zap_rwlock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:375
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
zap_lockdir() at zap_lockdir+0x34c
zap_lookup_norm() at zap_lookup_norm+0x45
zap_lookup() at zap_lookup+0x2e
sa_setup() at sa_setup+0x214
zfsvfs_create() at zfsvfs_create+0x218
zfs_mount() at zfs_mount+0xe6
vfs_donmount() at vfs_donmount+0xcde
nmount() at nmount+0x63
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (378, FreeBSD ELF64, nmount), rip = 0x801061d4c, rsp = 0x7fffffffcce8, rbp = 0x80185f10c ---
lock order reversal:
 1st 0xffffff0002a3e310 zfs (zfs) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/gfs.c:488
 2nd 0xffffff0002ee4350 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:1116
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
zfs_zget() at zfs_zget+0x241
zfs_root() at zfs_root+0x50
zfsctl_create() at zfsctl_create+0x81
zfs_mount() at zfs_mount+0x5f4
vfs_donmount() at vfs_donmount+0xcde
nmount() at nmount+0x63
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (378, FreeBSD ELF64, nmount), rip = 0x801061d4c, rsp = 0x7fffffffcce8, rbp = 0x80185f10c ---
Setting hostname: xanadu
.
Starting Network: lo0 bge1.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 
	inet6 ::1 prefixlen 128 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
	ether 00:e0:81:40:29:d3
	inet6 fe80::2e0:81ff:fe40:29d3%bge1 prefixlen 64 tentative scopeid 0x2 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (none)
	status: no carrier
Starting devd.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
32-bit compatibility ldconfig path: /usr/lib32
Creating and/or trimming log files
.
Starting syslogd.
No core dumps found.
Clearing /tmp (X related).
Starting rpcbind.
Starting mountd.
Starting nfsd.
Updating motd:
.
bge1: link state changed to UP
Configuring syscons:
 blanktime
.
Starting sshd.
Sep 13 01:52:06 xanadu sm-mta[1152]: My unqualified host name (xanadu) unknown; sleeping for retry
Sep 13 01:53:06 xanadu sm-mta[1152]: unable to qualify my own domain name (xanadu) -- using short name
Sep 13 01:53:07 xanadu sm-msp-queue[1159]: My unqualified host name (xanadu) unknown; sleeping for retry
lock order reversal:
 1st 0xffffff0002ee4200 zfsvfs->z_teardown_inactive_lock (zfsvfs->z_teardown_inactive_lock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4503
 2nd 0xffffff0002ee43f0 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:1345
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
zfs_zinactive() at zfs_zinactive+0x8b
zfs_inactive() at zfs_inactive+0x7e
zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a
VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xd9
vinactive() at vinactive+0x90
vputx() at vputx+0x2dc
nfsrv3_access() at nfsrv3_access+0x2ca
nfssvc_program() at nfssvc_program+0x1a6
svc_run_internal() at svc_run_internal+0x5fb
svc_thread_start() at svc_thread_start+0xb
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0xc, rip = 0x8006a2bec, rsp = 0x7fffffffe698, rbp = 0x5 ---
Sep 13 01:54:07 xanadu sm-msp-queue[1159]: unable to qualify my own domain name (xanadu) -- using short name
Starting cron.
Starting background file system checks in 60 seconds.

Mon Sep 13 01:54:07 UTC 2010
lock order reversal:
 1st 0xffffff0017b8c058 zp->z_name_lock (zp->z_name_lock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:220
 2nd 0xffffff0002ee4490 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:1116
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
zfs_zget() at zfs_zget+0x241
zfs_dirent_lock() at zfs_dirent_lock+0x4c1
zfs_dirlook() at zfs_dirlook+0x90
zfs_lookup() at zfs_lookup+0x2d4
zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xd7
vfs_cache_lookup() at vfs_cache_lookup+0xf0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xdf
lookup() at lookup+0x3d3
nfs_namei() at nfs_namei+0x3e3
nfsrv_lookup() at nfsrv_lookup+0x216
nfssvc_program() at nfssvc_program+0x1a6
svc_run_internal() at svc_run_internal+0x5fb
svc_thread_start() at svc_thread_start+0xb
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0xc, rip = 0x8006a2bec, rsp = 0x7fffffffe698, rbp = 0x5 ---
lock order reversal:
 1st 0xffffff0017339be0 db->db_mtx (db->db_mtx) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1242
 2nd 0xffffff006e5a7938 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1246
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
dbuf_dirty() at dbuf_dirty+0x6ce
dbuf_dirty() at dbuf_dirty+0x52b
dbuf_dirty() at dbuf_dirty+0x52b
dbuf_dirty() at dbuf_dirty+0x52b
dbuf_dirty() at dbuf_dirty+0x52b
dbuf_dirty() at dbuf_dirty+0x52b
dnode_setdirty() at dnode_setdirty+0x1a5
dbuf_dirty() at dbuf_dirty+0x593
sa_attr_op() at sa_attr_op+0x32c
sa_bulk_update_impl() at sa_bulk_update_impl+0x7c
sa_bulk_update() at sa_bulk_update+0x50
zfs_freebsd_setattr() at zfs_freebsd_setattr+0x1b61
VOP_SETATTR_APV() at VOP_SETATTR_APV+0xd3
nfsrv_setattr() at nfsrv_setattr+0x80d
nfssvc_program() at nfssvc_program+0x1a6
svc_run_internal() at svc_run_internal+0x5fb
svc_run() at svc_run+0x8f
nfssvc_nfsd() at nfssvc_nfsd+0xa2
nfssvc_nfsserver() at nfssvc_nfsserver+0x5b
nfssvc() at nfssvc+0x73
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x8006a2bec, rsp = 0x7fffffffe698, rbp = 0x5 ---
lock order reversal:
 1st 0xffffff00170171c0 l->l_rwlock (l->l_rwlock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:522
 2nd 0xffffff0017311850 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:611
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_slock() at _sx_slock+0x54
dbuf_read() at dbuf_read+0x319
dbuf_will_dirty() at dbuf_will_dirty+0x80
zap_get_leaf_byblk() at zap_get_leaf_byblk+0x1dd
zap_deref_leaf() at zap_deref_leaf+0xb6
fzap_add_cd() at fzap_add_cd+0x75
zap_add() at zap_add+0x105
zfs_link_create() at zfs_link_create+0x332
zfs_freebsd_create() at zfs_freebsd_create+0x762
VOP_CREATE_APV() at VOP_CREATE_APV+0xd7
nfsrv_create() at nfsrv_create+0x906
nfssvc_program() at nfssvc_program+0x1a6
svc_run_internal() at svc_run_internal+0x5fb
svc_run() at svc_run+0x8f
nfssvc_nfsd() at nfssvc_nfsd+0xa2
nfssvc_nfsserver() at nfssvc_nfsserver+0x5b
nfssvc() at nfssvc+0x73
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x8006a2bec, rsp = 0x7fffffffe698, rbp = 0x5 ---
lock order reversal:
 1st 0xffffff00170171c0 l->l_rwlock (l->l_rwlock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:522
 2nd 0xffffff0002d0ce60 os->os_lock (os->os_lock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:1228
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_xlock() at _sx_xlock+0x55
dnode_setdirty() at dnode_setdirty+0xc9
dbuf_dirty() at dbuf_dirty+0x593
dbuf_dirty() at dbuf_dirty+0x52b
zap_get_leaf_byblk() at zap_get_leaf_byblk+0x1dd
zap_deref_leaf() at zap_deref_leaf+0xb6
fzap_add_cd() at fzap_add_cd+0x75
zap_add() at zap_add+0x105
zfs_link_create() at zfs_link_create+0x332
zfs_freebsd_create() at zfs_freebsd_create+0x762
VOP_CREATE_APV() at VOP_CREATE_APV+0xd7
nfsrv_create() at nfsrv_create+0x906
nfssvc_program() at nfssvc_program+0x1a6
svc_run_internal() at svc_run_internal+0x5fb
svc_run() at svc_run+0x8f
nfssvc_nfsd() at nfssvc_nfsd+0xa2
nfssvc_nfsserver() at nfssvc_nfsserver+0x5b
nfssvc() at nfssvc+0x73
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x8006a2bec, rsp = 0x7fffffffe698, rbp = 0x5 ---
lock order reversal:
 1st 0xffffff006e65f238 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:2243
 2nd 0xffffff006e64e428 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:2194
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_slock() at _sx_slock+0x54
dbuf_check_blkptr() at dbuf_check_blkptr+0x210
dbuf_sync_list() at dbuf_sync_list+0x36c
dbuf_sync_list() at dbuf_sync_list+0x17f
dnode_sync() at dnode_sync+0xe9c
dmu_objset_sync_dnodes() at dmu_objset_sync_dnodes+0x92
dmu_objset_sync() at dmu_objset_sync+0x19d
dsl_pool_sync() at dsl_pool_sync+0xe5
spa_sync() at spa_sync+0x33f
txg_sync_thread() at txg_sync_thread+0x147
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff806273acf0, rbp = 0 ---
lock order reversal:
 1st 0xffffff006e5b8000 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_tx.c:440
 2nd 0xffffff0002361818 zap->zap_rwlock (zap->zap_rwlock) @ /head_old/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:490
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
_sx_slock() at _sx_slock+0x54
zap_lockdir() at zap_lockdir+0x111
zap_prefetch_uint64() at zap_prefetch_uint64+0x36
ddt_prefetch() at ddt_prefetch+0xba
dsl_dataset_block_freeable() at dsl_dataset_block_freeable+0x3c
dmu_tx_hold_free() at dmu_tx_hold_free+0x56f
dmu_free_long_range_impl() at dmu_free_long_range_impl+0x114
dmu_free_long_range() at dmu_free_long_range+0x4c
zfs_rmnode() at zfs_rmnode+0x89
zfs_inactive() at zfs_inactive+0x7e
zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a
VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xd9
vinactive() at vinactive+0x90
vputx() at vputx+0x2dc
zfs_freebsd_rename() at zfs_freebsd_rename+0x11b
VOP_RENAME_APV() at VOP_RENAME_APV+0xbf
nfsrv_rename() at nfsrv_rename+0xb52
nfssvc_program() at nfssvc_program+0x1a6
svc_run_internal() at svc_run_internal+0x5fb
svc_run() at svc_run+0x8f
nfssvc_nfsd() at nfssvc_nfsd+0xa2
nfssvc_nfsserver() at nfssvc_nfsserver+0x5b
nfssvc() at nfssvc+0x73
syscallenter() at syscallenter+0x1cb
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xe2
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x8006a2bec, rsp = 0x7fffffffe698, rbp = 0x5 ---
panic: Solaris(panic): zfs: accessing past end of object ffffff80627acd50/57 (size=1652215463 access=0+18446743525605952880)

cpuid = 0
KDB: enter: panic
panic: from debugger
cpuid = 0
Uptime: 5m51s
Physical memory: 3309 MB
Dumping 1336 MB: 1321 1305 1289 1273 1257 1241 1225 1209 1193 1177 1161 1145 1129 1113 1097 1081 1065 1049 1033 1017 1001 985 969 953 937 921 905 889 873 857 841 825 809 793 777 761 745 729 713 697 681 665 649 633 617 601 585 569 553 537 521 505 489 473 457 441 425 409 393 377 361 345 329 313 297 281 265 249 233 217 201 185 169 153 137 121 105 89 73 57 41 25 9

------------------------------------------------------------------------
kernel config

options	CONFIG_AUTOGENERATED
ident	GENERIC
machine	amd64
cpu	HAMMER
makeoptions	DEBUG=-g
options	USB_DEBUG
options	AH_SUPPORT_AR5416
options	IEEE80211_SUPPORT_MESH
options	IEEE80211_AMPDU_AGE
options	IEEE80211_DEBUG
options	AHD_REG_PRETTY_PRINT
options	AHC_REG_PRETTY_PRINT
options	ATA_STATIC_ID
options	SMP
options	DEBUG_VFS_LOCKS
options	DEBUG_LOCKS
options	MALLOC_DEBUG_MAXZONES=8
options	WITNESS_SKIPSPIN
options	WITNESS
options	INVARIANT_SUPPORT
options	INVARIANTS
options	DEADLKRES
options	GDB
options	DDB_CTF
options	DDB
options	KDB
options	INCLUDE_CONFIG_FILE
options	KDTRACE_HOOKS
options	KDTRACE_FRAME
options	FLOWTABLE
options	MAC
options	AUDIT
options	HWPMC_HOOKS
options	KBD_INSTALL_CDEV
options	PRINTF_BUFR_SIZE=128
options	_KPOSIX_PRIORITY_SCHEDULING
options	P1003_1B_SEMAPHORES
options	SYSVSEM
options	SYSVMSG
options	SYSVSHM
options	STACK
options	KTRACE
options	SCSI_DELAY=5000
options	COMPAT_FREEBSD7
options	COMPAT_FREEBSD6
options	COMPAT_FREEBSD5
options	COMPAT_FREEBSD4
options	COMPAT_FREEBSD32
options	GEOM_LABEL
options	GEOM_PART_GPT
options	PSEUDOFS
options	PROCFS
options	CD9660
options	MSDOSFS
options	NFS_ROOT
options	NFSLOCKD
options	NFSSERVER
options	NFSCLIENT
options	MD_ROOT
options	UFS_GJOURNAL
options	UFS_DIRHASH
options	UFS_ACL
options	SOFTUPDATES
options	FFS
options	SCTP
options	INET6
options	INET
options	PREEMPTION
options	SCHED_ULE
options	GEOM_PART_MBR
options	GEOM_PART_EBR_COMPAT
options	GEOM_PART_EBR
options	GEOM_PART_BSD
device	isa
device	mem
device	io
device	uart_ns8250
device	cpufreq
device	acpi
device	pci
device	fdc
device	ata
device	atadisk
device	ataraid
device	atapicd
device	atapifd
device	atapist
device	ahc
device	ahd
device	amd
device	hptiop
device	isp
device	mpt
device	sym
device	trm
device	adv
device	adw
device	aic
device	bt
device	scbus
device	ch
device	da
device	sa
device	cd
device	pass
device	ses
device	amr
device	arcmsr
device	ciss
device	dpt
device	hptmv
device	hptrr
device	iir
device	ips
device	mly
device	twa
device	aac
device	aacp
device	ida
device	mfi
device	mlx
device	twe
device	atkbdc
device	atkbd
device	psm
device	kbdmux
device	vga
device	splash
device	sc
device	agp
device	cbb
device	pccard
device	cardbus
device	uart
device	ppc
device	ppbus
device	lpt
device	plip
device	ppi
device	de
device	em
device	igb
device	ixgbe
device	le
device	ti
device	txp
device	vx
device	miibus
device	ae
device	age
device	alc
device	ale
device	bce
device	bfe
device	bge
device	dc
device	et
device	fxp
device	jme
device	lge
device	msk
device	nfe
device	nge
device	pcn
device	re
device	rl
device	sf
device	sge
device	sis
device	sk
device	ste
device	stge
device	tl
device	tx
device	vge
device	vr
device	wb
device	xl
device	cs
device	ed
device	ex
device	ep
device	fe
device	sn
device	xe
device	wlan
device	wlan_wep
device	wlan_ccmp
device	wlan_tkip
device	wlan_amrr
device	an
device	ath
device	ath_hal
device	ath_rate_sample
device	ral
device	wi
device	loop
device	random
device	ether
device	vlan
device	tun
device	pty
device	md
device	gif
device	faith
device	firmware
device	bpf
device	uhci
device	ohci
device	ehci
device	usb
device	uhid
device	ukbd
device	ulpt
device	umass
device	ums
device	urio
device	u3g
device	uark
device	ubsa
device	uftdi
device	uipaq
device	uplcom
device	uslcom
device	uvisor
device	uvscom
device	aue
device	axe
device	cdce
device	cue
device	kue
device	rue
device	udav
device	rum
device	uath
device	ural
device	zyd
device	firewire
device	sbp
device	fwe
device	fwip
device	dcons
device	dcons_crom

------------------------------------------------------------------------
ddb capture buffer



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTikis8J_x84CESMB0zYk1eVka1uFXQMP3HS9TOgG>