Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 12 Jul 2018 11:35:12 -0700
From:      Jim Long <list@museum.rain.com>
To:        Mike Tancsa <mike@sentex.net>
Cc:        freebsd-questions@freebsd.org
Subject:   Re: Disk/ZFS activity crash on 11.2-STABLE
Message-ID:  <20180712183512.GA75020@g5.umpquanet.com>
In-Reply-To: <5ebd8573-1363-06c7-cbb2-8298b0894319@sentex.net>
References:  <20180711212959.GA81029@g5.umpquanet.com> <5ebd8573-1363-06c7-cbb2-8298b0894319@sentex.net>

next in thread | previous in thread | raw e-mail | index | archive | help

--k1lZvvs/B4yU6o8G
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On Thu, Jul 12, 2018 at 11:53:31AM -0400, Mike Tancsa wrote:
> 
> prior to the crash, if you run gstat does it show the disks at 100% load ?
> 
> You have a lot of RAM, is ARC by chance using all of it ? Try putting a
> hard limit on it. So if you have 98G of RAM, place a limit at something
> like 70G for now.
> Also, what is the file system like ? Is it a few big files or MANY
> little files.  Post the output of zfs-stats -a
> 

..snip..

> And speaking
> of crash, post the contents of the
> core.txt.3
> 
> file in /var/crash
> 
> 	---Mike


Mike:

Thank you for copying me on your reply, I forgot to request that.  I
read the digest version of the list, so a direct reply is much more
timely.

The core.txt.3 file is attached.  Your suggestion of dhclient-safety.sh
makes sense, I should have thought of using a legacy rc.d script in
/usr/local/etc/rc.d/

zfs-stats.out is also attached.

Yes, ARC usage is high.  top shows:

last pid:  2215;  load averages:  1.50,  1.69,  1.69   up 0+00:44:59  11:02:00
27 processes:  2 running, 25 sleeping
CPU:  0.0% user,  0.0% nice,  9.0% system,  0.1% interrupt, 90.9% idle
Mem: 1416K Active, 11M Inact, 2788K Laundry, 91G Wired, 1236M Free
ARC: 84G Total, 68M MFU, 83G MRU, 15M Anon, 877M Header, 4078K Other
     83G Compressed, 148G Uncompressed, 1.78:1 Ratio
Swap: 16G Total, 15M Used, 16G Free

I'm re-running the test now.  I had shut off hyperthreading to go down to
24 cores, but it's back on now.  This server will be a computational resource,
not just storage, so I'd like to find a way to stablize the configuration with
hyperthreading enabled, if possible.

Thanks again,

Jim



zfs-stress.sh:

#!/usr/bin/env bash

FILE="/var/crash/zfs-stress.start"
SLEEP_TIME=10

# if this script causes a crash, then we should expect to see a
# crash dump newer than this file:

touch $FILE

logger "$0 : begin"

echo 'zfs destroy -r mendeleev@stress-test' | sh -v 2>&1 | logger

echo 'zfs snap    -r mendeleev@stress-test' | sh -v 2>&1 | logger

logger "$0 : backgrounding 'zfs send -R mendeleev@stress-test > /dev/null'"

zfs send -R mendeleev@stress-test > /dev/null &

while [ -f $FILE ] && sleep $SLEEP_TIME; do

        printf "gstat -ab:\n$(gstat -ab)\n\n" | logger
        top | head -8 | logger

done

[ -f $FILE ] || echo "$0 : $FILE has disappeared ... terminating."

-- snip




--k1lZvvs/B4yU6o8G
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="core.txt.3"

mendeleev3 dumped core - see /var/crash/vmcore.3

Wed Jul 11 13:38:41 PDT 2018

FreeBSD mendeleev3 11.2-STABLE FreeBSD 11.2-STABLE #0 r335674: Tue Jun 26 13:20:24 PDT 2018     root@jimsdesk:/usr/obj/usr/src/sys/GENERIC  amd64

panic: page fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 38; apic id = 32
fault virtual address	= 0x5a
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80dff30d
stack pointer	        = 0x28:0xfffffe17eaaa8440
frame pointer	        = 0x28:0xfffffe17eaaa8480
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 0 (system_taskq_33)
trap number		= 12
panic: page fault
cpuid = 38
KDB: stack backtrace:
#0 0xffffffff80b3d907 at kdb_backtrace+0x67
#1 0xffffffff80af6e77 at vpanic+0x177
#2 0xffffffff80af6cf3 at panic+0x43
#3 0xffffffff80f77fef at trap_fatal+0x35f
#4 0xffffffff80f78049 at trap_pfault+0x49
#5 0xffffffff80f77824 at trap+0x2d4
#6 0xffffffff80f5786c at calltrap+0x8
#7 0xffffffff80f7938b at uma_small_alloc+0x5b
#8 0xffffffff80de5b72 at keg_alloc_slab+0xe2
#9 0xffffffff80de857e at keg_fetch_slab+0x14e
#10 0xffffffff80de7db4 at zone_fetch_slab+0x64
#11 0xffffffff80de7e8f at zone_import+0x3f
#12 0xffffffff80de4599 at uma_zalloc_arg+0x3d9
#13 0xffffffff8225b692 at abd_alloc+0x112
#14 0xffffffff8226064e at arc_hdr_alloc_pabd+0x8e
#15 0xffffffff8225dbcf at arc_hdr_alloc+0x11f
#16 0xffffffff8225f84f at arc_read+0x24f
#17 0xffffffff8228342d at traverse_prefetcher+0x18d
Uptime: 29m53s
Dumping 7011 out of 96901 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/ums.ko...Reading symbols from /usr/lib/debug//boot/kernel/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ums.ko
#0  doadump (textdump=<value optimized out>) at pcpu.h:230
230	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:230
#1  0xffffffff80af6a8b in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:383
#2  0xffffffff80af6eb1 in vpanic (fmt=<value optimized out>, 
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:776
#3  0xffffffff80af6cf3 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:707
#4  0xffffffff80f77fef in trap_fatal (frame=0xfffffe17eaaa8380, eva=90)
    at /usr/src/sys/amd64/amd64/trap.c:877
#5  0xffffffff80f78049 in trap_pfault (frame=0xfffffe17eaaa8380, usermode=0)
    at pcpu.h:230
#6  0xffffffff80f77824 in trap (frame=0xfffffe17eaaa8380)
    at /usr/src/sys/amd64/amd64/trap.c:415
#7  0xffffffff80f5786c in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:231
#8  0xffffffff80dff30d in vm_page_alloc_after (object=0x0, pindex=0, 
    req=<value optimized out>, mpred=0x0) at vm_phys.h:121
#9  0xffffffff80f7938b in uma_small_alloc (zone=<value optimized out>, 
    bytes=<value optimized out>, flags=<value optimized out>, wait=2305)
    at /usr/src/sys/amd64/amd64/uma_machdep.c:52
#10 0xffffffff80de5b72 in keg_alloc_slab (keg=0xfffff80162a67780, 
    zone=<value optimized out>, wait=2305) at /usr/src/sys/vm/uma_core.c:978
#11 0xffffffff80de857e in keg_fetch_slab (keg=0xfffff80162a67780, 
    zone=0xfffffe000423a000, flags=1) at /usr/src/sys/vm/uma_core.c:2358
#12 0xffffffff80de7db4 in zone_fetch_slab (zone=0xfffffe000423a000, 
    keg=0xfffff80162a67780, flags=1) at /usr/src/sys/vm/uma_core.c:2390
#13 0xffffffff80de7e8f in zone_import (zone=0xfffffe000423a000, 
    bucket=0xfffff80311698018, max=<value optimized out>, flags=1)
    at /usr/src/sys/vm/uma_core.c:2516
#14 0xffffffff80de4599 in uma_zalloc_arg (zone=0xfffffe000423a000, 
    udata=0xfffff801619cbf40, flags=2) at /usr/src/sys/vm/uma_core.c:2546
#15 0xffffffff8225b692 in abd_alloc (size=131072, is_metadata=0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/abd.c:177
#16 0xffffffff8226064e in arc_hdr_alloc_pabd (hdr=0xfffff807718f8620)
    at cpufunc.h:98
#17 0xffffffff8225dbcf in arc_hdr_alloc (spa=773887303047704091, 
    psize=<value optimized out>, lsize=<value optimized out>, 
    compression_type=ZIO_COMPRESS_OFF, type=ARC_BUFC_DATA)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:3248
#18 0xffffffff8225f84f in arc_read (pio=0x0, spa=0xfffffe001d8c3000, 
    bp=0xfffffe002542ab00, done=0, private=0x0, 
    priority=ZIO_PRIORITY_ASYNC_READ, zio_flags=<value optimized out>, 
    arc_flags=0xfffffe17eaaa87d4, zb=0xfffffe17eaaa8970)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:5482
#19 0xffffffff8228342d in traverse_prefetcher (spa=0xfffffe001d8c3000, 
    zilog=<value optimized out>, bp=0xfffffe002542ab00, 
    zb=0xfffffe17eaaa8970, dnp=<value optimized out>, arg=0xfffffe17eb6d11e0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:520
#20 0xffffffff822828c3 in traverse_visitbp (td=0xfffffe17eaaa91b8, 
    dnp=0xfffffe00e9d62e00, bp=0xfffffe002542ab00, zb=0xfffffe17eaaa8970)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:290
#21 0xffffffff82282add in traverse_visitbp (td=<value optimized out>, 
    dnp=0xfffffe00e9d62e00, bp=0xfffffe002fe8f180, zb=0xfffffe17eaaa8a50)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:322
#22 0xffffffff82282add in traverse_visitbp (td=<value optimized out>, 
    dnp=0xfffffe00e9d62e00, bp=0xfffffe00e9d62e40, zb=0xfffffe17eaaa8ab8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:322
#23 0xffffffff822835f3 in traverse_dnode (td=0xfffffe17eaaa91b8, 
    dnp=0xfffffe00e9d62e00, objset=3530, object=381335)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:473
#24 0xffffffff82283000 in traverse_visitbp (td=<value optimized out>, 
    dnp=0xfffff801800c6800, bp=0xfffffe002a586600, zb=0xfffffe17eaaa8c80)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:344
#25 0xffffffff82282add in traverse_visitbp (td=<value optimized out>, 
    dnp=0xfffff801800c6800, bp=0xfffffe008e440580, zb=0xfffffe17eaaa8d60)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:322
#26 0xffffffff82282add in traverse_visitbp (td=<value optimized out>, 
    dnp=0xfffff801800c6800, bp=0xfffffe002bdc2000, zb=0xfffffe17eaaa8e40)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:322
#27 0xffffffff82282add in traverse_visitbp (td=<value optimized out>, 
    dnp=0xfffff801800c6800, bp=0xfffffe0044482000, zb=0xfffffe17eaaa8f20)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:322
#28 0xffffffff82282add in traverse_visitbp (td=<value optimized out>, 
    dnp=0xfffff801800c6800, bp=0xfffffe002b1a2000, zb=0xfffffe17eaaa9000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:322
#29 0xffffffff82282add in traverse_visitbp (td=<value optimized out>, 
    dnp=0xfffff801800c6800, bp=0xfffff801800c6840, zb=0xfffffe17eaaa9068)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:322
#30 0xffffffff822835f3 in traverse_dnode (td=0xfffffe17eaaa91b8, 
    dnp=0xfffff801800c6800, objset=3530, object=0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:473
#31 0xffffffff82282da8 in traverse_visitbp (td=<value optimized out>, 
    dnp=0x0, bp=0xfffff801801a8680, zb=0xfffffe17eaaa9218)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:376
#32 0xffffffff82282656 in traverse_prefetch_thread (arg=0xfffffe17eb6d1180)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:540
#33 0xffffffff82244940 in taskq_run (arg=0xfffff801dafd2ae0, 
    pending=<value optimized out>)
    at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_taskq.c:110
#34 0xffffffff80b4f114 in taskqueue_run_locked (queue=0xfffff80161a6da00)
    at /usr/src/sys/kern/subr_taskqueue.c:463
#35 0xffffffff80b50278 in taskqueue_thread_loop (arg=<value optimized out>)
    at /usr/src/sys/kern/subr_taskqueue.c:755
#36 0xffffffff80aba383 in fork_exit (
    callout=0xffffffff80b501e0 <taskqueue_thread_loop>, 
    arg=0xfffff80161a8c200, frame=0xfffffe17eaaa9340)
    at /usr/src/sys/kern/kern_fork.c:1054
#37 0xffffffff80f5881e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:972
#38 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
(kgdb) 

------------------------------------------------------------------------
ps -axlww

UID  PID PPID CPU PRI NI   VSZ   RSS MWCHAN   STAT TT      TIME COMMAND
  0    0    0   0 -16  0     0     0 vmwait   DLs   -   0:10.03 [kernel]
  0    1    0   0  20  0  5416  1092 wait     DLs   -   0:00.14 [init]
  0    2    0   0 -16  0     0     0 crypto_w DL    -   0:00.00 [crypto]
  0    3    0   0 -16  0     0     0 crypto_r DL    -   0:00.00 [crypto returns]
  0    4    0   0 -16  0     0     0 -        RL    -   0:00.55 [cam]
  0    5    0   0 -16  0     0     0 -        DL    -   0:00.00 [soaiod1]
  0    6    0   0 -16  0     0     0 -        DL    -   0:00.00 [soaiod2]
  0    7    0   0 -16  0     0     0 -        DL    -   0:00.00 [soaiod3]
  0    8    0   0 -16  0     0     0 -        DL    -   0:00.00 [soaiod4]
  0    9    0   0  -8  0     0     0 -        RL    -   0:11.15 [zfskern]
  0   10    0   0 -16  0     0     0 audit_wo DL    -   0:00.00 [audit]
  0   11    0   0 155  0     0     0 -        RL    -  52:54.37 [idle]
  0   12    0   0 -56  0     0     0 -        WL    -   0:02.28 [intr]
  0   13    0   0  -8  0     0     0 -        DL    -   0:00.01 [geom]
  0   14    0   0 -68  0     0     0 -        DL    -   0:00.06 [usb]
  0   15    0   0 -16  0     0     0 waiting_ DL    -   0:00.00 [sctp_iterator]
  0   16    0   0 -16  0     0     0 -        DL    -   0:00.04 [rand_harvestq]
  0   17    0   0 -16  0     0     0 idle     DL    -   0:00.00 [enc_daemon0]
  0   18    0   0 -16  0     0     0 idle     DL    -   0:00.02 [enc_daemon1]
  0   19    0   0 -16  0     0     0 idle     DL    -   0:00.00 [enc_daemon2]
  0   20    0   0 -16  0     0     0 arc_recl DL    -   0:00.28 [pagedaemon]
  0   21    0   0 -16  0     0     0 psleep   DL    -   0:00.00 [vmdaemon]
  0   22    0   0 155  0     0     0 pgzero   DL    -   0:00.00 [pagezero]
  0   23    0   0 -16  0     0     0 -        DL    -   0:00.00 [bufspacedaemon]
  0   24    0   0 -16  0     0     0 psleep   DL    -   0:00.00 [bufdaemon]
  0   25    0   0 -16  0     0     0 vlruwt   DL    -   0:00.00 [vnlru]
  0   26    0   0  16  0     0     0 syncer   DL    -   0:00.00 [syncer]
  0  125    1   0  52  0  6268  1972 pause    Ds    -   0:00.00 [adjkerntz]
  0  491    1   0  52  0  6524  2336 select   Ds    -   0:00.00 [dhclient]
 65  553    1   0  20  0  6524  2468 select   DCs   -   0:00.00 [dhclient]
  0  606    1   0  52  0  6580  2216 select   Ds    -   0:00.00 [moused]
  0  620    1   0  20  0  9180  5112 select   Ds    -   0:00.00 [devd]
  0  882    1   0  20  0  6412  2464 select   Ds    -   0:00.02 [syslogd]
  0 1089    1   0  20  0 12432 12512 select   Ds    -   0:01.04 [ntpd]
  0 1171    1   0  25  0 12828  7052 select   Ds    -   0:00.00 [sshd]
  0 1185    1   0  20  0 10432  6372 select   Ds    -   0:00.00 [sendmail]
 25 1196    1   0  52  0 10432  6528 pause    Ds    -   0:00.00 [sendmail]
  0 1204    1   0  20  0  6444  2484 nanslp   Ds    -   0:00.00 [cron]
  0 1283    1   0  20  0  6932  2992 wait     Ds    -   0:00.00 [login]
  0 1284    1   0  20  0  6932  2992 wait     Ds    -   0:00.00 [login]
  0 1285    1   0  52  0  6388  2200 ttyin    Ds+   -   0:00.00 [getty]
  0 1286    1   0  52  0  6388  2200 ttyin    Ds+   -   0:00.00 [getty]
  0 1287    1   0  20  0  6932  2992 wait     Ds    -   0:00.00 [login]
  0 1288    1   0  52  0  6388  2200 ttyin    Ds+   -   0:00.00 [getty]
  0 1289    1   0  52  0  6388  2200 ttyin    Ds+   -   0:00.00 [getty]
  0 1290    1   0  52  0  6388  2200 ttyin    Ds+   -   0:00.00 [getty]
  0 1291 1171   0  20  0 13140  8308 select   Ds    -   0:00.00 [sshd]
  0 1293 1291   0  20  0  7776  4040 select   Ds+   -   0:00.00 [bash]
  0 1411 1283   0  20  0  7776  4040 wait     D     -   0:00.00 [bash]
  0 1444 1411   0  20  0  7776  3636 wait     D     -   0:00.00 [bash]
  0 1450 1444   0  52  0  7768  4444 hdr->b_l D     -   0:00.00 [zfs]
  0 1477 1287   0  49  0  7776  4036 wait     D     -   0:00.00 [bash]
  0 1499 1477   0  30 10  9292  5396 select   DN+   -   0:00.00 [systat]
  0 1506 1284   0  20  0  7776  4044 select   D+    -   0:00.00 [bash]
  0 1570 1506   0  20  0  7776  4040 wait     D     -   0:00.00 [bash]
  0 1590 1411   0  20  0  6276  2072 kqread   D+    -   0:00.00 [tail]
  0 1596 1444   0  20  0  4220  2000 nanslp   DC    -   0:00.00 [sleep]
  0 1599 1570   0  20  0  4220  2000 nanslp   DC    -   0:00.00 [sleep]

------------------------------------------------------------------------
vmstat -s

 39539081 cpu context switches
  2335473 device interrupts
    17734 software interrupts
   570899 traps
  3744909 system calls
       26 kernel threads created
      912  fork() calls
      657 vfork() calls
        4 rfork() calls
        0 swap pager pageins
        0 swap pager pages paged in
        0 swap pager pageouts
        0 swap pager pages paged out
      706 vnode pager pageins
     6346 vnode pager pages paged in
        0 vnode pager pageouts
        0 vnode pager pages paged out
        0 page daemon wakeups
        0 pages examined by the page daemon
        0 clean page reclamation shortfalls
        0 pages reactivated by the page daemon
    42293 copy-on-write faults
      171 copy-on-write optimized faults
   525374 zero fill pages zeroed
       11 zero fill pages prezeroed
        0 intransit blocking page faults
   628010 total VM faults taken
      533 page faults requiring I/O
        0 pages affected by kernel thread creation
    36395 pages affected by  fork()
    22985 pages affected by vfork()
      200 pages affected by rfork()
  1210386 pages freed
        0 pages freed by daemon
        0 pages freed by exiting processes
     1036 pages active
     5628 pages inactive
     4623 pages in the laundry queue
 24162530 pages wired down
     1023 pages free
     4096 bytes per page
        0 total name lookups
          cache hits (0% pos + 0% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
       feeder     7     1K       -        7  32
     SCSI ENC   146   340K       -     3446  16,64,128,256,512,1024,2048,32768,65536
       nvlist   104    10K       -      104  16,64,128
         UART     6     5K       -        6  16,1024
     acpiintr     1     1K       -        1  64
          USB    35    54K       -       36  16,32,128,256,512,1024,2048,4096
       USBdev    25     8K       -       25  32,64,128,256,512,1024,4096
       acpica 63733  6361K       -   720376  16,32,64,128,256,512,1024,2048,4096
     acpitask     1    64K       -        1  65536
      acpisem    85    11K       -       85  128
      acpidev   293    19K       -      293  64
      CAM SIM    18     5K       -       18  256
      CAM XPT   131    12K       -      656  16,32,64,128,256,512,1024,2048,65536
      CAM DEV    29    58K       -       61  2048
      CAM CCB     0     0K       -  2756925  2048
     CAM path    40     2K       -      764  32
   CAM periph    24     6K       -       60  16,32,64,128,256
        vtbuf    24  4992K       -       46  4096
           vt    11     6K       -       11  512
CAM I/O Scheduler     8     1K       -        8  128
       DEVFS3   435   109K       -      467  256
       DEVFS1   250   125K       -      281  512
   DEVFS_RULE    55    26K       -       55  64,512
        DEVFS    57     2K       -       58  16,32,128
       DEVFSP     4     1K       -        4  64
NFSD V4client     1     1K       -        1  256
 NFSD lckfile     1     1K       -        1  256
 NFSD session     1     1K       -        1  1024
    pfs_nodes    21     6K       -       21  256
         GEOM   705   109K       -     5201  16,32,64,128,256,512,1024,2048,4096,8192,16384
    raid_data     0     0K       -      738  32,128,256
       isadev     6     1K       -        6  128
     pci_link    16     2K       -       16  64,128
         cdev     3     1K       -        3  256
     filedesc    23   181K       -      431  32,4096,16384
        sigio     0     0K       -        1  64
     filecaps     7     1K       -      124  16,32,64
      kdtrace   856   204K       -     4143  64,256
         kenv   121    13K       -      125  16,32,64,128,8192
       kqueue    60     6K       -     1602  64,256,2048
    proc-args   415    14K       -     1856  16,32,64,128,256
        hhook    13     4K       -       13  256
      ithread   337    57K       -      337  32,128,256
       KTRACE   100    13K       -      100  128
    acpi_perf    48    24K       -       48  512
       linker   196    35K       -      206  16,32,64,128,256,512,1024,2048,4096,8192
        lockf    23     3K       -      139  64,128
   loginclass     3     1K       -        3  64
       devbuf 21124 41984K       -    26728  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
         temp   208    21K       -     3659  16,32,64,128,256,512,1024,2048,4096,16384,65536
       module   505    64K       -      506  128
     mtx_pool     2    16K       -        2  8192
          osd     7     1K       -       98  16,32,64,128,256
     pmchooks     1     1K       -        1  128
         pgrp    29     4K       -      106  128
      session    22     3K       -       35  128
         proc     2   256K       -        2  
      subproc   483   445K       -     2025  512,4096
         cred    51    13K       -      427  256
    CAM queue    47   139K       -      154  16,32,64,128,512,16384
       plimit    22     6K       -      487  256
      uidinfo     4    33K       -       18  128,32768
       dumper     1     4K       -        1  4096
       sysctl     0     0K       -        1  64
    sysctloid 12625   644K       -    12914  16,32,64,128,256
    sysctltmp     0     0K       -      105  16,32,64,256,1024
      tidhash     1   256K       -        1  
      callout    49 13448K       -       49  
         umtx  2012   252K       -     2012  128
     p1003.1b     1     1K       -        1  16
         SWAP     2  1097K       -        2  32
          bus  2246   284K       -    48881  16,32,64,128,256,512,1024
       bus-sc   269  2024K       -    40020  16,32,128,256,512,1024,2048,4096,8192,16384,32768,65536
      devstat    44    89K       -       44  32,4096
 eventhandler   109     9K       -      109  64,128
   gtaskqueue   294    53K       -      294  16,32,256,8192
         kobj   341  1364K       -     1437  4096
      Per-cpu     1     1K       -        1  32
         rman   337    39K       -      807  32,128
         sbuf     0     0K       -    12260  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
    toponodes   124    16K       -      124  128
    taskqueue   189    24K       -      204  16,32,64,128,256,512
     terminal    11     3K       -       11  256
       Unitno    23     2K       -      533  32,64
         vmem     2   640K       -        6  16384,65536
     ioctlops     0     0K       -      119  256,512,1024
       select    37     5K       -       37  128
          iov     0     0K       -   273381  16,32,64,128,256,512
          msg     4    30K       -        4  2048,4096,8192,16384
          sem     4   106K       -        4  2048,4096
          shm     1    32K       -        1  32768
          tty    15    15K       -       15  1024
          pts     1     1K       -        1  256
     mbuf_tag     0     0K       -       27  32
        shmfd     1     8K       -        1  8192
       soname     4     1K       -    62160  16,32,128
          pcb    20 16535K       -       99  16,32,128,1024,2048,8192
       kbdmux     8    22K       -        8  16,512,1024,2048,16384
          acl     0     0K       -       46  4096
     vfscache     4 17025K       -        4  512
     vfs_hash     1  8192K       -        1  
       vnodes     1     1K       -        1  256
        mount   220     8K       -      506  16,32,64,128,256
       statfs     0     0K       -      273  512
          LED    56     4K       -       56  16,128
  vnodemarker     0     0K       -     1263  512
   arc4random     1    32K       -        1  32768
          BPF    11    18K       -       11  128,512,4096
        ifnet     4     7K       -        4  128,2048
       ifaddr    45    13K       -       47  16,32,64,128,256,512,2048,4096
  ether_multi    17     1K       -       22  16,32,64
        clone     7     1K       -        7  128
        ipsec     1     1K       -        1  256
      lltable    14     4K       -       14  256,512
     routetbl    25     4K       -       30  32,64,128,512
         igmp     3     1K       -        3  128
   in_mfilter     1     1K       -        1  1024
     in_multi     2     1K       -        3  256
  ip_moptions     2     1K       -        2  64,256
encap_export_host     2     2K       -        2  1024
    sctp_a_it     0     0K       -        2  16
     sctp_vrf     1     1K       -        1  64
     sctp_ifa     4     1K       -        4  128
     sctp_ifn     2     1K       -        2  128
    sctp_iter     0     0K       -        2  256
    hostcache     1    32K       -        1  32768
          LRO     8     8K       -        8  1024
      tcpfunc     1     1K       -        1  32
     syncache     1    68K       -        1  
  in6_mfilter     1     1K       -        1  1024
    in6_multi    15     2K       -       15  32,256
 ip6_moptions     2     1K       -        2  32,256
       ip6opt     3     1K       -        3  256
          mld     3     1K       -        3  128
       ip6ndp     4     1K       -        4  64,256
  inpcbpolicy    13     1K       -      213  32
     secasvar     1     1K       -        1  1024
       sahead     1     1K       -        1  1024
  ipsecpolicy     2     2K       -        6  256,1024
    ipsec-saq     2     2K       -        2  1024
       crypto     1     1K       -        1  512
          rpc     2    16K       -        2  8192
audit_evclass   189     6K       -      234  32
    ufs_quota     1  8192K       -        1  
    vm_pgdata     1     1K       -        1  128
      UMAHash    47 263131K       -      182  512,1024,2048,4096,8192,16384,32768,65536
      memdesc     1     4K       -        1  4096
          mpr    19   559K       -       59  16,32,64,128,256,1024,4096,8192,65536
CAM dev queue    18     2K       -       18  64
      scsi_da     0     0K       -      798  32,64,256
          ixl     2     8K       -        2  4096
       sr_iov     2     1K       -        2  512
       apmdev     1     1K       -        1  128
   madt_table     0     0K       -        1  4096
      entropy     1     1K       -       11  32,4096
      io_apic     9    10K       -        9  1024,2048
          MCA    97    31K       -       97  128,512
          msi    30     4K       -       30  128
     nexusdev     4     1K       -        4  16
   kstat_data     8     1K       -        8  64
      solaris 2099399 780112K       - 76828357  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
    sfs_nodes    36    18K       -       36  512

------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               384,      0,     238,       2,     238,   0,   0
UMA Zones:             6784,      0,     255,       0,     255,   0,   0
UMA Slabs:               80,      0,22397947,       3,22420279,   0,   0
UMA Hash:               256,      0,      61,      59,     108,   0,   0
4 Bucket:                32,      0,    3237,   12013,   69203,   0,   0
6 Bucket:                48,      0,     920,   12028,   23096,   0,   0
8 Bucket:                64,      0,     299,   11915,   15149,  15,   0
12 Bucket:               96,      0,     473,   10884,   33544,   0,   0
16 Bucket:              128,      0,     574,    8664,   30440,   0,   0
32 Bucket:              256,      0,    1452,    6678,  194385, 139,   0
64 Bucket:              512,      0,     717,    1739,  307173, 138,   0
128 Bucket:            1024,      0,    1090,    1806,  447171,   0,   0
256 Bucket:            2048,      0,    1862,    1830,  200778,  16,   0
vmem btag:               56,      0,   59405,   10033,   64207, 490,   0
VM OBJECT:              240,      0,    1881,    2759,   18824,   0,   0
RADIX NODE:             144,      0,   76564,    3653,  184780,   0,   0
MAP:                    240,      0,       3,      61,       3,   0,   0
KMAP ENTRY:             128,      0,       6,     397,       6,   0,   0
MAP ENTRY:              128,      0,     956,    6670,   51520,   0,   0
VMSPACE:               2520,      0,      32,     196,    1575,   0,   0
fakepg:                 104,      0,       0,     152,       3,   0,   0
mt_zone:              16400,      0,     408,       0,     408,   0,   0
16:                      16,      0,    8889,    9936,  667136,   0,   0
32:                      32,      0,    9176,   28824, 5527996,   0,   0
64:                      64,      0, 1079474,   14516, 9720798,   0,   0
128:                    128,      0,  313193,   12090, 2808003,   0,   0
256:                    256,      0,  437062,   11588,56903630,   0,   0
512:                    512,      0,  349086,    3970, 2254339,   0,   0
1024:                  1024,      0,    2200,     988,  120013,   0,   0
2048:                  2048,      0,     285,    2161, 2767034,   0,   0
4096:                  4096,      0,    2123,     188,   11336,   0,   0
8192:                  8192,      0,     203,     147,   16498,   0,   0
16384:                16384,      0,      31,      69,    6467,   0,   0
32768:                32768,      0,      47,      41,    3360,   0,   0
65536:                65536,      0,      45,      33,    4034,   0,   0
SLEEPQUEUE:              80,      0,    1007,    4883,    1007,   0,   0
64 pcpu:                  8,      0,    3013,    2875,    3021,   0,   0
Files:                   80,      0,      82,    5668,   18291,   0,   0
filedesc0:             1104,      0,      58,     527,    1600,   0,   0
rl_entry:                40,      0,     153,   11747,     153,   0,   0
TURNSTILE:              136,      0,    1007,    2233,    1007,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
umtx_shm:                88,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1424,      0,      57,     367,    1599,   0,   0
THREAD:                1552,      0,     927,      79,    1130,   0,   0
cpuset:                  96,      0,     783,    5244,    1097,   0,   0
audit_record:          1248,      0,       0,       0,       0,   0,   0
mbuf_packet:            256, 38679675,    8184,    9034,   12580,   0,   0
mbuf:                   256, 38679675,       1,   11145,   13401,   0,   0
mbuf_cluster:          2048, 6043698,   17204,      62,   17204,   0,   0
mbuf_jumbo_page:       4096, 3021848,       0,       3,       7,   0,   0
mbuf_jumbo_9k:         9216, 2686086,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384, 2014564,       0,       0,       0,   0,   0
FPU_save_area:         2696,      0,       0,       0,       0,   0,   0
g_bio:                  376,      0,      36,    9704,11026368,   0,   0
DMAR_MAP_ENTRY:         120,      0,       0,       0,       0,   0,   0
ttyinq:                 160,      0,     135,     915,     435,   0,   0
ttyoutq:                256,      0,      72,     978,     232,   0,   0
nvme_request:           128,      0,       0,       0,       0,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       1,   14690,   40583,   0,   0
VNODE:                  472,      0,     742,     802,     835,   0,   0
VNODEPOLL:              120,      0,       1,     131,       1,   0,   0
BUF TRIE:               144,      0,       0,  105948,       0,   0,   0
S VFS Cache:            108,      0,     703,    5317,    2375,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,       0,       0,       0,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NAMEI:                 1024,      0,       0,     600,   55619,   0,   0
rentr:                   24,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,       0,       0,       0,   0,   0
NCLNODE:                576,      0,       0,       0,       0,   0,   0
AIO:                    208,      0,       0,       0,       0,   0,   0
AIOP:                    32,      0,       0,       0,       0,   0,   0
AIOCB:                  752,      0,       0,       0,       0,   0,   0
AIOLIO:                 280,      0,       0,       0,       0,   0,   0
Mountpoints:            816,      0,      19,     176,      19,   0,   0
range_seg_cache:         64,      0,     652,   11748,   13476,   0,   0
metaslab_alloc_trace_cache:     64,      0,       0,   12152,   38435,   0,   0
zio_cache:             1040,      0,    2194,   25166,11241661,   0,   0
zio_link_cache:          48,      0,    2178,   34508,13726325,   0,   0
zio_buf_512:            512,      0,    2124,    1356,  535701,   0,   0
zio_data_buf_512:       512,      0,      55,    1313,  258022,   0,   0
zio_buf_1024:          1024,      0,      56,     540,    5534,   0,   0
zio_data_buf_1024:     1024,      0,      42,     546,  101722,   0,   0
zio_buf_1536:          1536,      0,       9,     397,    5057,   0,   0
zio_data_buf_1536:     1536,      0,      15,     381,   62096,   0,   0
zio_buf_2048:          2048,      0,      45,     231,    4257,   0,   0
zio_data_buf_2048:     2048,      0,      12,     184,   43958,   0,   0
zio_buf_2560:          2560,      0,      12,     129,     547,   0,   0
zio_data_buf_2560:     2560,      0,       7,     119,   38734,   0,   0
zio_buf_3072:          3072,      0,       7,     131,     322,   0,   0
zio_data_buf_3072:     3072,      0,       3,     129,   39511,   0,   0
zio_buf_3584:          3584,      0,       2,      45,     257,   0,   0
zio_data_buf_3584:     3584,      0,       4,      55,   47700,   0,   0
zio_buf_4096:          4096,      0,      27,    3945, 3152314,   0,   0
zio_data_buf_4096:     4096,      0,       5,     167,   28855,   0,   0
zio_buf_5120:          5120,      0,       3,      44,     241,   0,   0
zio_data_buf_5120:     5120,      0,       3,      56,   32951,   0,   0
zio_buf_6144:          6144,      0,       5,      48,     177,   0,   0
zio_data_buf_6144:     6144,      0,       4,      53,   22396,   0,   0
zio_buf_7168:          7168,      0,       2,      41,     121,   0,   0
zio_data_buf_7168:     7168,      0,       6,      50,   18886,   0,   0
zio_buf_8192:          8192,      0,       3,    1959,  585154,   0,   0
zio_data_buf_8192:     8192,      0,       9,      52,   16274,   0,   0
zio_buf_10240:        10240,      0,       2,      40,     120,   0,   0
zio_data_buf_10240:   10240,      0,       7,      55,   24350,   0,   0
zio_buf_12288:        12288,      0,       1,    1733,  653471,   0,   0
zio_data_buf_12288:   12288,      0,       8,      52,   19934,   0,   0
zio_buf_14336:        14336,      0,       0,      29,      63,   0,   0
zio_data_buf_14336:   14336,      0,       7,      49,   15583,   0,   0
zio_buf_16384:        16384,      0,     526,    1791,  829422,   0,   0
zio_data_buf_16384:   16384,      0,       8,      54,   14839,   0,   0
zio_buf_20480:        20480,      0,      13,    3915,  663814,   0,   0
zio_data_buf_20480:   20480,      0,       7,      52,   28948,   0,   0
zio_buf_24576:        24576,      0,       4,    2777,  170929,   0,   0
zio_data_buf_24576:   24576,      0,       9,      56,   16217,   0,   0
zio_buf_28672:        28672,      0,       4,    1546,   65027,   0,   0
zio_data_buf_28672:   28672,      0,       3,      54,   11554,   0,   0
zio_buf_32768:        32768,      0,    1033,    4397,  857173,   0,   0
zio_data_buf_32768:   32768,      0,       7,      53,   16143,   0,   0
zio_buf_40960:        40960,      0,       0,      65,   81085,   0,   0
zio_data_buf_40960:   40960,      0,      10,      56,   12373,   0,   0
zio_buf_49152:        49152,      0,       2,     172,   94059,   0,   0
zio_data_buf_49152:   49152,      0,       5,      59,    8868,   0,   0
zio_buf_57344:        57344,      0,       3,     169,  117765,   0,   0
zio_data_buf_57344:   57344,      0,       3,      52,    6590,   0,   0
zio_buf_65536:        65536,      0,       1,     190,  103301,   0,   0
zio_data_buf_65536:   65536,      0,       4,      55,    5741,   0,   0
zio_buf_81920:        81920,      0,       0,      69,  195743,   0,   0
zio_data_buf_81920:   81920,      0,       3,      55,   13127,   0,   0
zio_buf_98304:        98304,      0,       0,      71,   70635,   0,   0
zio_data_buf_98304:   98304,      0,       3,      53,    6340,   0,   0
zio_buf_114688:      114688,      0,       0,     183,   52918,   0,   0
zio_data_buf_114688: 114688,      0,       5,      56,    6532,   0,   0
zio_buf_131072:      131072,      0,     249,     735,  585161,   0,   0
zio_data_buf_131072: 131072,      0,   20933,    1227, 1164901,   0,   0
zio_buf_163840:      163840,      0,       0,      20,      31,   0,   0
zio_data_buf_163840: 163840,      0,       0,       0,       0,   0,   0
zio_buf_196608:      196608,      0,       0,       0,       0,   0,   0
zio_data_buf_196608: 196608,      0,       0,       0,       0,   0,   0
zio_buf_229376:      229376,      0,       0,       0,       0,   0,   0
zio_data_buf_229376: 229376,      0,       0,       0,       0,   0,   0
zio_buf_262144:      262144,      0,       0,       0,       0,   0,   0
zio_data_buf_262144: 262144,      0,       0,       0,       0,   0,   0
zio_buf_327680:      327680,      0,       0,       0,       0,   0,   0
zio_data_buf_327680: 327680,      0,       0,       0,       0,   0,   0
zio_buf_393216:      393216,      0,       0,       0,       0,   0,   0
zio_data_buf_393216: 393216,      0,       0,       0,       0,   0,   0
zio_buf_458752:      458752,      0,       0,       0,       0,   0,   0
zio_data_buf_458752: 458752,      0,       0,       0,       0,   0,   0
zio_buf_524288:      524288,      0,       0,       0,       0,   0,   0
zio_data_buf_524288: 524288,      0,       0,       0,       0,   0,   0
zio_buf_655360:      655360,      0,       0,       0,       0,   0,   0
zio_data_buf_655360: 655360,      0,       0,       0,       0,   0,   0
zio_buf_786432:      786432,      0,       0,       0,       0,   0,   0
zio_data_buf_786432: 786432,      0,       0,       0,       0,   0,   0
zio_buf_917504:      917504,      0,       0,       0,       0,   0,   0
zio_data_buf_917504: 917504,      0,       0,       0,       0,   0,   0
zio_buf_1048576:     1048576,      0,       0,       0,       0,   0,   0
zio_data_buf_1048576: 1048576,      0,       0,       0,       0,   0,   0
zio_buf_1310720:     1310720,      0,       0,       0,       0,   0,   0
zio_data_buf_1310720: 1310720,      0,       0,       0,       0,   0,   0
zio_buf_1572864:     1572864,      0,       0,       0,       0,   0,   0
zio_data_buf_1572864: 1572864,      0,       0,       0,       0,   0,   0
zio_buf_1835008:     1835008,      0,       0,       0,       0,   0,   0
zio_data_buf_1835008: 1835008,      0,       0,       0,       0,   0,   0
zio_buf_2097152:     2097152,      0,       0,       0,       0,   0,   0
zio_data_buf_2097152: 2097152,      0,       0,       0,       0,   0,   0
zio_buf_2621440:     2621440,      0,       0,       0,       0,   0,   0
zio_data_buf_2621440: 2621440,      0,       0,       0,       0,   0,   0
zio_buf_3145728:     3145728,      0,       0,       0,       0,   0,   0
zio_data_buf_3145728: 3145728,      0,       0,       0,       0,   0,   0
zio_buf_3670016:     3670016,      0,       0,       0,       0,   0,   0
zio_data_buf_3670016: 3670016,      0,       0,       0,       0,   0,   0
zio_buf_4194304:     4194304,      0,       0,       0,       0,   0,   0
zio_data_buf_4194304: 4194304,      0,       0,       0,       0,   0,   0
zio_buf_5242880:     5242880,      0,       0,       0,       0,   0,   0
zio_data_buf_5242880: 5242880,      0,       0,       0,       0,   0,   0
zio_buf_6291456:     6291456,      0,       0,       0,       0,   0,   0
zio_data_buf_6291456: 6291456,      0,       0,       0,       0,   0,   0
zio_buf_7340032:     7340032,      0,       0,       0,       0,   0,   0
zio_data_buf_7340032: 7340032,      0,       0,       0,       0,   0,   0
zio_buf_8388608:     8388608,      0,       0,       0,       0,   0,   0
zio_data_buf_8388608: 8388608,      0,       0,       0,       0,   0,   0
zio_buf_10485760:    10485760,      0,       0,       0,       0,   0,   0
zio_data_buf_10485760: 10485760,      0,       0,       0,       0,   0,   0
zio_buf_12582912:    12582912,      0,       0,       0,       0,   0,   0
zio_data_buf_12582912: 12582912,      0,       0,       0,       0,   0,   0
zio_buf_14680064:    14680064,      0,       0,       0,       0,   0,   0
zio_data_buf_14680064: 14680064,      0,       0,       0,       0,   0,   0
zio_buf_16777216:    16777216,      0,       0,       0,       0,   0,   0
zio_data_buf_16777216: 16777216,      0,       0,       0,       0,   0,   0
lz4_ctx:              16384,      0,       0,      82,   34011,   0,   0
abd_chunk:             4096,      0,22283282,    2428,26549803,   0,   0
sa_cache:               144,      0,     687,    2364,     778,   0,   0
dnode_t:                728,      0,    3604,    1041,   11152,   0,   0
arc_buf_hdr_t_full:     224,      0, 2065753,    1549, 2273683,   0,   0
arc_buf_hdr_t_l2only:     88,      0,       0,       0,       0,   0,   0
arc_buf_t:               64,      0,   22140,   10534, 2425480,   0,   0
dmu_buf_impl_t:         232,      0,   24198,    3206,  130259,   0,   0
zil_lwb_cache:          320,      0,       3,     381,      38,   0,   0
zil_zcw_cache:           80,      0,       0,    2400,     156,   0,   0
zfs_znode_cache:        264,      0,     687,     858,     778,   0,   0
pipe:                   760,      0,       3,     582,     467,   0,   0
procdesc:               136,      0,       0,     261,       4,   0,   0
ksiginfo:               112,      0,     173,    6967,    1978,   0,   0
itimer:                 352,      0,       0,       0,       0,   0,   0
KNOTE:                  128,      0,       2,     153,       4,   0,   0
socket:                 864, 3100852,      26,     598,    9078,   0,   0
unpcb:                  240, 3100864,      11,    2917,    8845,   0,   0
ipq:                     56, 188931,       0,       0,       0,   0,   0
udp_inpcb:              464, 3100856,       8,     920,     197,   0,   0
udpcb:                   32, 3100875,       8,    8117,     197,   0,   0
tcp_inpcb:              464, 3100856,       4,     172,       9,   0,   0
tcpcb:                 1040, 3100854,       4,      50,       9,   0,   0
tcptw:                   88,  27810,       0,       0,       0,   0,   0
syncache:               168,  15364,       0,      69,       1,   0,   0
hostcache:              128,  15376,       0,       0,       0,   0,   0
sackhole:                32,      0,       0,       0,       0,   0,   0
tcpreass:                40, 377800,       0,       0,       0,   0,   0
sctp_ep:               1248, 3100854,       0,       0,       0,   0,   0
sctp_asoc:             2408,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80012,       0,     830,       3,   0,   0
sctp_raddr:             736,  80000,       0,       0,       0,   0,   0
sctp_chunk:             152, 400010,       0,       0,       0,   0,   0
sctp_readq:             152, 400010,       0,       0,       0,   0,   0
sctp_stream_msg_out:    112, 400015,       0,       0,       0,   0,   0
sctp_asconf:             40, 400000,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400060,       0,       0,       0,   0,   0
udplite_inpcb:          464, 3100856,       0,       0,       0,   0,   0
ripcb:                  464, 3100856,       1,     119,       7,   0,   0
IPsec SA lft_c:          16,      0,       0,       0,       0,   0,   0
rtentry:                208,      0,      11,     616,      15,   0,   0
selfd:                   64,      0,      72,   10096,   41719,   0,   0
swpctrie:               144, 12087414,       0,       0,       0,   0,   0
swblk:                  136, 12087403,       0,       0,       0,   0,   0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
cpu0:timer                         88671        632
cpu1:timer                         77721        554
cpu39:timer                        86753        618
cpu8:timer                         82602        589
cpu16:timer                       243037       1732
cpu5:timer                         76733        547
cpu9:timer                         81452        580
cpu40:timer                        95309        679
cpu4:timer                         78734        561
cpu20:timer                        89738        639
cpu42:timer                        92478        659
cpu38:timer                        89811        640
cpu11:timer                        78309        558
cpu17:timer                        89052        635
cpu15:timer                        82714        589
cpu25:timer                        80850        576
cpu3:timer                         73616        525
cpu33:timer                        86913        619
cpu43:timer                        90640        646
cpu41:timer                        91162        650
cpu21:timer                        86331        615
cpu7:timer                         79949        570
cpu46:timer                        90772        647
cpu36:timer                        91137        649
cpu34:timer                        87641        624
cpu10:timer                        82084        585
cpu14:timer                        83376        594
cpu22:timer                        89974        641
cpu27:timer                        83512        595
cpu23:timer                        90143        642
cpu37:timer                        87446        623
cpu35:timer                        84863        605
cpu30:timer                        87731        625
cpu32:timer                        91948        655
cpu24:timer                        82903        591
cpu6:timer                         81815        583
cpu28:timer                        88449        630
cpu31:timer                        83339        594
cpu44:timer                        96889        690
cpu2:timer                         79495        566
cpu26:timer                        82766        590
cpu29:timer                        81603        581
cpu47:timer                        89971        641
cpu18:timer                        87927        627
cpu13:timer                        86471        616
cpu19:timer                        85707        611
cpu12:timer                        85240        607
cpu45:timer                        94199        671
irq264: ahci0                        433          3
irq265: xhci0                       5164         37
irq267: ixl0:aq                        2          0
irq268: ixl0:q0                     2023         14
irq269: ixl0:q1                     1338         10
irq270: ixl0:q2                      477          3
irq271: ixl0:q3                       61          0
irq272: ixl0:q4                      130          1
irq273: ixl0:q5                     2429         17
irq274: ixl0:q6                       69          0
irq275: ixl0:q7                       79          1
irq276: ixl1:aq                        2          0
irq285: mpr0                     2323266      16554
Total                            6615449      47137

------------------------------------------------------------------------
pstat -T

 82/3100849 files
0M/0M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity

------------------------------------------------------------------------
iostat

       tty            ada0             ada1              da0             cpu
 tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
   1    60 24.63   0  0.00  23.87   0  0.00  37.67 249  9.16   0  0  1  0 99

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME   

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME   

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME   


------------------------------------------------------------------------
ipcs -T

msginfo:
	msgmax:        16384	(max characters in a message)
	msgmni:           40	(# of message queues)
	msgmnb:         2048	(max characters in a message queue)
	msgtql:           40	(max # of messages in system)
	msgssz:            8	(size of a message segment)
	msgseg:         2048	(# of message segments in system)

shminfo:
	shmmax:    536870912	(max shared memory segment size)
	shmmin:            1	(min shared memory segment size)
	shmmni:          192	(max number of shared memory identifiers)
	shmseg:          128	(max shared memory segments per process)
	shmall:       131072	(max amount of shared memory in pages)

seminfo:
	semmni:           50	(# of semaphore identifiers)
	semmns:          340	(# of semaphores in system)
	semmnu:          150	(# of undo structures in system)
	semmsl:          340	(max # of semaphores per id)
	semopm:          100	(max # of operations per semop call)
	semume:           50	(max # of undo entries per process)
	semusz:          632	(size in bytes of undo structure)
	semvmx:        32767	(semaphore maximum value)
	semaem:        16384	(adjust on exit max value)


------------------------------------------------------------------------
nfsstat

Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0         0
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
        0         0         0         0         0         0         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
        0         0         0         0         0         0         0         0

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Server Ret-Failed
                0
Server Faults
            0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
        0         0         0         0
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
        0         0         0

------------------------------------------------------------------------
netstat -s

tcp:
	1201 packets sent
		1133 data packets (75258 bytes)
		0 data packets (0 bytes) retransmitted
		0 data packets unnecessarily retransmitted
		0 resends initiated by MTU discovery
		68 ack-only packets (58 delayed)
		0 URG only packets
		0 window probe packets
		0 window update packets
		0 control packets
	1780 packets received
		1043 acks (for 75259 bytes)
		0 duplicate acks
		0 acks for unsent data
		1039 packets (40242 bytes) received in-sequence
		0 completely duplicate packets (0 bytes)
		0 old duplicate packets
		0 packets with some dup. data (0 bytes duped)
		0 out-of-order packets (0 bytes)
		0 packets (0 bytes) of data after window
		0 window probes
		0 window update packets
		0 packets received after close
		0 discarded for bad checksums
		0 discarded for bad header offset fields
		0 discarded because packet too short
		0 discarded due to memory problems
	0 connection requests
	1 connection accept
	0 bad connection attempts
	0 listen queue overflows
	0 ignored RSTs in the windows
	1 connection established (including accepts)
		0 times used RTT from hostcache
		0 times used RTT variance from hostcache
		0 times used slow-start threshold from hostcache
	5 connections closed (including 0 drops)
		0 connections updated cached RTT on close
		0 connections updated cached RTT variance on close
		0 connections updated cached ssthresh on close
	0 embryonic connections dropped
	1043 segments updated rtt (of 1025 attempts)
	0 retransmit timeouts
		0 connections dropped by rexmit timeout
	0 persist timeouts
		0 connections dropped by persist timeout
	0 Connections (fin_wait_2) dropped because of timeout
	0 keepalive timeouts
		0 keepalive probes sent
		0 connections dropped by keepalive
	624 correct ACK header predictions
	697 correct data packet header predictions
	1 syncache entry added
		0 retransmitted
		0 dupsyn
		0 dropped
		1 completed
		0 bucket overflow
		0 cache overflow
		0 reset
		0 stale
		0 aborted
		0 badack
		0 unreach
		0 zone failures
	1 cookie sent
	0 cookies received
	0 hostcache entries added
		0 bucket overflow
	0 SACK recovery episodes
	0 segment rexmits in SACK recovery episodes
	0 byte rexmits in SACK recovery episodes
	0 SACK options (SACK blocks) received
	0 SACK options (SACK blocks) sent
	0 SACK scoreboard overflow
	0 packets with ECN CE bit set
	0 packets with ECN ECT(0) bit set
	0 packets with ECN ECT(1) bit set
	0 successful ECN handshakes
	0 times ECN reduced the congestion window
	0 packets with matching signature received
	0 packets with bad signature received
	0 times failed to make signature due to no SA
	0 times unexpected signature received
	0 times no signature provided by segment
TCP connection count by state:
	0 connections in CLOSED state
	3 connections in LISTEN state
	0 connections in SYN_SENT state
	0 connections in SYN_RCVD state
	1 connection  in ESTABLISHED state
	0 connections in CLOSE_WAIT state
	0 connections in FIN_WAIT_1 state
	0 connections in CLOSING state
	0 connections in LAST_ACK state
	0 connections in FIN_WAIT_2 state
	0 connections in TIME_WAIT state
udp:
	182 datagrams received
	0 with incomplete header
	0 with bad data length field
	0 with bad checksum
	0 with no checksum
	5 dropped due to no socket
	0 broadcast/multicast datagrams undelivered
	0 dropped due to full socket buffers
	0 not for hashed pcb
	177 delivered
	3809 datagrams output
	0 times multicast source filter matched
ip:
	1986 total packets received
	0 bad header checksums
	0 with size smaller than minimum
	0 with data size < data length
	0 with ip length > max ip packet size
	0 with header length < data size
	0 with data length < header length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 packets reassembled ok
	1984 packets for this host
	0 packets for unknown/unsupported protocol
	0 packets forwarded (0 packets fast forwarded)
	2 packets not forwardable
	0 packets received for unknown multicast group
	0 redirects sent
	5024 packets sent from this host
	14 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	0 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 tunneling packets that can't find gif
	0 datagrams with bad address in header
icmp:
	5 calls to icmp_error
	0 errors not generated in response to an icmp message
	Output histogram:
		echo reply: 3
		destination unreachable: 5
	0 messages with bad code fields
	0 messages less than the minimum length
	0 messages with bad checksum
	0 messages with bad length
	0 multicast echo requests ignored
	0 multicast timestamp requests ignored
	Input histogram:
		echo reply: 5
		destination unreachable: 6
		echo: 3
		time exceeded: 8
	3 message responses generated
	0 invalid return addresses
	0 no return routes
ipsec:
	0 inbound packets violated process security policy
	0 inbound packets failed due to insufficient memory
	0 invalid inbound packets
	0 outbound packets violated process security policy
	0 outbound packets with no SA available
	0 outbound packets failed due to insufficient memory
	0 outbound packets with no route available
	0 invalid outbound packets
	0 outbound packets with bundled SAs
	0 mbufs coalesced during clone
	0 clusters coalesced during clone
	0 clusters copied during clone
	0 mbufs inserted during makespace
ah:
	0 packets shorter than header shows
	0 packets dropped; protocol family not supported
	0 packets dropped; no TDB
	0 packets dropped; bad KCR
	0 packets dropped; queue full
	0 packets dropped; no transform
	0 replay counter wraps
	0 packets dropped; bad authentication detected
	0 packets dropped; bad authentication length
	0 possible replay packets detected
	0 packets in
	0 packets out
	0 packets dropped; invalid TDB
	0 bytes in
	0 bytes out
	0 packets dropped; larger than IP_MAXPACKET
	0 packets blocked due to policy
	0 crypto processing failures
	0 tunnel sanity check failures
esp:
	0 packets shorter than header shows
	0 packets dropped; protocol family not supported
	0 packets dropped; no TDB
	0 packets dropped; bad KCR
	0 packets dropped; queue full
	0 packets dropped; no transform
	0 packets dropped; bad ilen
	0 replay counter wraps
	0 packets dropped; bad encryption detected
	0 packets dropped; bad authentication detected
	0 possible replay packets detected
	0 packets in
	0 packets out
	0 packets dropped; invalid TDB
	0 bytes in
	0 bytes out
	0 packets dropped; larger than IP_MAXPACKET
	0 packets blocked due to policy
	0 crypto processing failures
	0 tunnel sanity check failures
ipcomp:
	0 packets shorter than header shows
	0 packets dropped; protocol family not supported
	0 packets dropped; no TDB
	0 packets dropped; bad KCR
	0 packets dropped; queue full
	0 packets dropped; no transform
	0 replay counter wraps
	0 packets in
	0 packets out
	0 packets dropped; invalid TDB
	0 bytes in
	0 bytes out
	0 packets dropped; larger than IP_MAXPACKET
	0 packets blocked due to policy
	0 crypto processing failures
	0 packets sent uncompressed; size < compr. algo. threshold
	0 packets sent uncompressed; compression was useless
arp:
	2 ARP requests sent
	1 ARP reply sent
	5 ARP requests received
	1 ARP reply received
	9 ARP packets received
	0 total packets dropped due to no ARP entry
	0 ARP entrys timed out
	0 Duplicate IPs seen
ip6:
	0 total packets received
	0 with size smaller than minimum
	0 with data size < data length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 fragments that exceeded limit
	0 packets reassembled ok
	0 packets for this host
	0 packets forwarded
	0 packets not forwardable
	0 redirects sent
	0 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	0 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 packets that violated scope rules
	0 multicast packets which we don't join
	Mbuf statistics:
		0 one mbuf
		0 one ext mbuf
		0 two or more ext mbuf
	0 packets whose headers are not contiguous
	0 tunneling packets that can't find gif
	0 packets discarded because of too many headers
	1 failure of source address selection
	source addresses on a non-outgoing I/F
		1 addresses scope=0xf
	Source addresses selection rule applied:
		1 same address
icmp6:
	0 calls to icmp6_error
	0 errors not generated in response to an icmp6 message
	0 errors not generated because of rate limitation
	0 messages with bad code fields
	0 messages < minimum length
	0 bad checksums
	0 messages with bad length
	Histogram of error messages to be generated:
		0 no route
		0 administratively prohibited
		0 beyond scope
		0 address unreachable
		0 port unreachable
		0 packet too big
		0 time exceed transit
		0 time exceed reassembly
		0 erroneous header field
		0 unrecognized next header
		0 unrecognized option
		0 redirect
		0 unknown
	0 message responses generated
	0 messages with too many ND options
	0 messages with bad ND options
	0 bad neighbor solicitation messages
	0 bad neighbor advertisement messages
	0 bad router solicitation messages
	0 bad router advertisement messages
	0 bad redirect messages
	0 path MTU changes
ipsec6:
	0 inbound packets violated process security policy
	0 inbound packets failed due to insufficient memory
	0 invalid inbound packets
	0 outbound packets violated process security policy
	0 outbound packets with no SA available
	0 outbound packets failed due to insufficient memory
	0 outbound packets with no route available
	0 invalid outbound packets
	0 outbound packets with bundled SAs
	0 mbufs coalesced during clone
	0 clusters coalesced during clone
	0 clusters copied during clone
	0 mbufs inserted during makespace
rip6:
	0 messages received
	0 checksum calculations on inbound
	0 messages with bad checksum
	0 messages dropped due to no socket
	0 multicast messages dropped due to no socket
	0 messages dropped due to full socket buffers
	0 delivered
	0 datagrams output
pfkey:
	0 requests sent from userland
	0 bytes sent from userland
	0 messages with invalid length field
	0 messages with invalid version field
	0 messages with invalid message type field
	0 messages too short
	0 messages with memory allocation failure
	0 messages with duplicate extension
	0 messages with invalid extension type
	0 messages with invalid sa type
	0 messages with invalid address extension
	0 requests sent to userland
	0 bytes sent to userland
	0 messages toward single socket
	0 messages toward all sockets
	0 messages toward registered sockets
	0 messages with memory allocation failure

------------------------------------------------------------------------
netstat -m

8185/20179/28364 mbufs in use (current/cache/total)
8170/9096/17266/6043698 mbuf clusters in use (current/cache/total/max)
8184/9034 mbuf+clusters out of packet secondary zone in use (current/cache)
0/3/3/3021848 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/2686086 9k jumbo clusters in use (current/cache/total/max)
0/0/0/2014564 16k jumbo clusters in use (current/cache/total/max)
18386K/23248K/41635K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 sendfile syscalls
0 sendfile syscalls completed without I/O request
0 requests for I/O initiated by sendfile
0 pages read by sendfile as part of a request
0 pages were valid at time of a sendfile request
0 pages were requested for read ahead by applications
0 pages were read ahead by sendfile
0 times sendfile encountered an already busy page
0 requests for sfbufs denied
0 requests for sfbufs delayed

------------------------------------------------------------------------
netstat -anA

Active Internet connections (including servers)
Tcpcb            Proto Recv-Q Send-Q Local Address      Foreign Address    (state)
fffff803117ab820 tcp4       0      0 192.168.1.13.59    192.168.1.1.20800  ESTABLISHED
fffff802d5a41820 tcp4       0      0 127.0.0.1.25       *.*                LISTEN
fffff802d5e6e410 tcp4       0      0 *.59               *.*                LISTEN
fffff802d5e6e820 tcp6       0      0 *.59               *.*                LISTEN
fffff8031113a570 udp4       0      0 127.0.0.1.123      *.*                
fffff8031113a740 udp6       0      0 fe80::1%lo0.123    *.*                
fffff8031113a910 udp6       0      0 ::1.123            *.*                
fffff8031113aae0 udp4       0      0 192.168.1.13.123   *.*                
fffff8031113acb0 udp4       0      0 *.123              *.*                
fffff8031113b000 udp6       0      0 *.123              *.*                
fffff801dae0bae0 udp4       0      0 *.514              *.*                
fffff801dae0bcb0 udp6       0      0 *.514              *.*                
Active UNIX domain sockets
Address          Type   Recv-Q Send-Q            Inode             Conn             Refs          Nextref Addr
fffff8018c7e52d0 stream      0      0 fffff80180926760                0                0                0 /var/run/devd.pipe
fffff8018c513780 dgram       0      0                0 fffff801dae115a0                0 fffff8018c4de960
fffff8018c4de960 dgram       0      0                0 fffff801dae115a0                0 fffff8018c7ed000
fffff8018c7ed000 dgram       0      0                0 fffff801dae115a0                0 fffff801e105c5a0
fffff801e105c5a0 dgram       0      0                0 fffff801dae115a0                0 fffff8018c7c3870
fffff801e17c4d20 dgram       0      0                0 fffff801dae11690                0                0
fffff8018c7c3870 dgram       0      0                0 fffff801dae115a0                0 fffff8018c513870
fffff8018c513870 dgram       0      0                0 fffff801dae115a0                0                0
fffff801dae115a0 dgram       0      0 fffff8018c4e91d8                0 fffff8018c513780                0 /var/run/logpriv
fffff801dae11690 dgram       0      0 fffff8018c221588                0 fffff801e17c4d20                0 /var/run/log
fffff8018c7e51e0 seqpac      0      0 fffff80180926588                0                0                0 /var/run/devd.seqpacket.pipe

------------------------------------------------------------------------
netstat -aL

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen                           Local Address         
tcp4  0/0/10                           localhost.smtp         
tcp4  0/0/128                          *.59                   
tcp6  0/0/128                          *.59                   
unix  0/0/4                            /var/run/devd.pipe
unix  0/0/4                            /var/run/devd.seqpacket.pipe

------------------------------------------------------------------------
fstat

fstat: can't read file 1 at 0x20000000008800d
fstat: can't read file 2 at 0x4000000000000a0
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20000000008800d
fstat: can't read file 2 at 0x4000000000000a0
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x2200000000
fstat: can't read file 7 at 0x20007ffffffffff
fstat: can't read file 8 at 0x4000000001fffff
fstat: can't read file 10 at 0x780000ffff
fstat: can't read file 11 at 0x200000000
fstat: can't read file 13 at 0x20007ffffffffff
fstat: can't read file 14 at 0x4000000001fffff
fstat: can't read file 16 at 0x780000ffff
fstat: can't read file 17 at 0x200000000
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x2200000000
fstat: can't read file 7 at 0x20007ffffffffff
fstat: can't read file 8 at 0x4000000001fffff
fstat: can't read file 10 at 0x780000ffff
fstat: can't read file 11 at 0x200000000
fstat: can't read file 13 at 0x20007ffffffffff
fstat: can't read file 14 at 0x4000000001fffff
fstat: can't read file 16 at 0x780000ffff
fstat: can't read file 17 at 0x200000000
fstat: can't read file 23 at 0xe400000001
fstat: can't read file 29 at 0x1c00000000
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x2200000000
fstat: can't read file 7 at 0x20007ffffffffff
fstat: can't read file 8 at 0x4000000001fffff
fstat: can't read file 10 at 0x780000ffff
fstat: can't read file 11 at 0x200000000
fstat: can't read file 13 at 0x20007ffffffffff
fstat: can't read file 14 at 0x4000000001fffff
fstat: can't read file 16 at 0x780000ffff
fstat: can't read file 17 at 0x200000000
fstat: can't read file 23 at 0xe000000000
fstat: can't read file 29 at 0x1c00000000
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x2400000000
fstat: can't read file 7 at 0x20007ffffffffff
fstat: can't read file 8 at 0x4000000001fffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x2200000000
fstat: can't read file 7 at 0x20007ffffffffff
fstat: can't read file 8 at 0x4000000001fffff
fstat: can't read file 10 at 0x780000ffff
fstat: can't read file 11 at 0x200000000
fstat: can't read file 13 at 0x20007ffffffffff
fstat: can't read file 14 at 0x4000000001fffff
fstat: can't read file 16 at 0x780000ffff
fstat: can't read file 17 at 0x200000000
fstat: can't read file 23 at 0x9000000000
fstat: can't read file 29 at 0x6800000000
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x2200000000
fstat: can't read file 7 at 0x20007ffffffffff
fstat: can't read file 8 at 0x4000000001fffff
fstat: can't read file 10 at 0x780000ffff
fstat: can't read file 11 at 0x200000000
fstat: can't read file 13 at 0x20007ffffffffff
fstat: can't read file 14 at 0x4000000001fffff
fstat: can't read file 16 at 0x780000ffff
fstat: can't read file 17 at 0x200000000
fstat: can't read file 23 at 0xf400000001
fstat: can't read file 29 at 0x1c00000000
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0xa00000000
fstat: can't read file 7 at 0x20007ffffffffff
fstat: can't read file 8 at 0x4000000001fffff
fstat: can't read file 10 at 0x780000ffff
fstat: can't read file 11 at 0x3e00000000
fstat: can't read file 13 at 0x20007ffffffffff
fstat: can't read file 14 at 0x4000000001fffff
fstat: can't read file 16 at 0x780000ffff
fstat: can't read file 17 at 0x800000000
fstat: can't read file 23 at 0x16c00000001
fstat: can't read file 29 at 0x9800000000
fstat: can't read file 35 at 0x1600000001
fstat: can't read file 41 at 0x4400000001
fstat: can't read file 47 at 0x1800000001
fstat: can't read file 53 at 0x3000000000
fstat: can't read file 59 at 0x3400000000
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x800000000
fstat: can't read file 7 at 0x20007ffffffffff
fstat: can't read file 8 at 0x4000000001fffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x600000000
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x600000000
fstat: can't read file 7 at 0x20007ffffffffff
fstat: can't read file 8 at 0x4000000001fffff
fstat: can't read file 10 at 0x780000ffff
fstat: can't read file 11 at 0x1a00000000
fstat: can't read file 13 at 0x20007ffffffffff
fstat: can't read file 14 at 0x4000000001fffff
fstat: can't read file 16 at 0x780000ffff
fstat: can't read file 17 at 0x600000000
fstat: can't read file 19 at 0x20007ffffffffff
fstat: can't read file 20 at 0x4000000001fffff
fstat: can't read file 22 at 0x780000ffff
fstat: can't read file 23 at 0x4200000001
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x400000000
fstat: can't read file 7 at 0x20007ffffffffff
fstat: can't read file 8 at 0x4000000001fffff
fstat: can't read file 10 at 0x780000ffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x800000000
fstat: can't read file 7 at 0x20007ffffffffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x800000000
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x200000000000000
fstat: can't read file 2 at 0x400000000000000
fstat: can't read file 5 at 0x800000000
fstat: can't read file 7 at 0x200000000000002
fstat: can't read file 8 at 0x400000000000000
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x800000000
fstat: can't read file 7 at 0x20007ffffffffff
fstat: can't read file 8 at 0x4000000001fffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
root     sleep       1599 root -         -       error    -
root     sleep       1599   wd -         -       error    -
root     sleep       1599 text -         -       error    -
root     sleep       1599 ctty /dev        261 crw--w----   ttyv1 rw
root     sleep       1599    0 /dev        261 crw--w----   ttyv1 rw
root     sleep       1596 root -         -       error    -
root     sleep       1596   wd -         -       error    -
root     sleep       1596 text -         -       error    -
root     sleep       1596 ctty /dev        260 crw--w----   ttyv0 rw
root     sleep       1596    0 /dev        260 crw--w----   ttyv0 rw
root     tail        1590 root -         -       error    -
root     tail        1590   wd -         -       error    -
root     tail        1590 text -         -       error    -
root     tail        1590 ctty /dev        260 crw--w----   ttyv0 rw
root     tail        1590    0 /dev        260 crw--w----   ttyv0 rw
root     bash        1570 root -         -       error    -
root     bash        1570   wd -         -       error    -
root     bash        1570 text -         -       error    -
root     bash        1570 ctty /dev        261 crw--w----   ttyv1 rw
root     bash        1570    0 /dev        261 crw--w----   ttyv1 rw
root     bash        1570    6 /dev        261 crw--w----   ttyv1 rw
root     bash        1570   12 /dev        261 crw--w----   ttyv1 rw
root     bash        1506 root -         -       error    -
root     bash        1506   wd -         -       error    -
root     bash        1506 text -         -       error    -
root     bash        1506 ctty /dev        261 crw--w----   ttyv1 rw
root     bash        1506    0 /dev        261 crw--w----   ttyv1 rw
root     bash        1506    6 /dev        261 crw--w----   ttyv1 rw
root     bash        1506   12 /dev        261 crw--w----   ttyv1 rw
root     systat      1499 root -         -       error    -
root     systat      1499   wd -         -       error    -
root     systat      1499 text -         -       error    -
root     systat      1499 ctty /dev        264 crw--w----   ttyv4 rw
root     systat      1499    0 /dev        264 crw--w----   ttyv4 rw
root     bash        1477 root -         -       error    -
root     bash        1477   wd -         -       error    -
root     bash        1477 text -         -       error    -
root     bash        1477 ctty /dev        264 crw--w----   ttyv4 rw
root     bash        1477    0 /dev        264 crw--w----   ttyv4 rw
root     bash        1477    6 /dev        264 crw--w----   ttyv4 rw
root     bash        1477   12 /dev        264 crw--w----   ttyv4 rw
root     zfs         1450 root -         -       error    -
root     zfs         1450   wd -         -       error    -
root     zfs         1450 text -         -       error    -
root     zfs         1450 ctty /dev        260 crw--w----   ttyv0 rw
root     zfs         1450    0 /dev         26 crw-rw-rw-    null  r
root     zfs         1450    6 /dev         26 crw-rw-rw-    null  w
root     bash        1444 root -         -       error    -
root     bash        1444   wd -         -       error    -
root     bash        1444 text -         -       error    -
root     bash        1444 ctty /dev        260 crw--w----   ttyv0 rw
root     bash        1444    0 /dev        260 crw--w----   ttyv0 rw
root     bash        1444    6 /dev        260 crw--w----   ttyv0 rw
root     bash        1444   12 /dev        260 crw--w----   ttyv0 rw
root     bash        1411 root -         -       error    -
root     bash        1411   wd -         -       error    -
root     bash        1411 text -         -       error    -
root     bash        1411 ctty /dev        260 crw--w----   ttyv0 rw
root     bash        1411    0 /dev        260 crw--w----   ttyv0 rw
root     bash        1411    6 /dev        260 crw--w----   ttyv0 rw
root     bash        1411   12 /dev        260 crw--w----   ttyv0 rw
root     bash        1293 root -         -       error    -
root     bash        1293   wd -         -       error    -
root     bash        1293 text -         -       error    -
root     bash        1293 ctty /dev        168 crw--w----   pts/0 rw
root     bash        1293    0 /dev        168 crw--w----   pts/0 rw
root     bash        1293    6 /dev        168 crw--w----   pts/0 rw
root     bash        1293   12 /dev        168 crw--w----   pts/0 rw
root     sshd        1291 root -         -       error    -
root     sshd        1291   wd -         -       error    -
root     sshd        1291 text -         -       error    -
root     sshd        1291    0 /dev         26 crw-rw-rw-    null rw
root     sshd        1291    6 /dev         26 crw-rw-rw-    null rw
root     getty       1290 root -         -       error    -
root     getty       1290   wd -         -       error    -
root     getty       1290 text -         -       error    -
root     getty       1290 ctty /dev        267 crw-------   ttyv7 rw
root     getty       1290    0 /dev        267 crw-------   ttyv7 rw
root     getty       1289 root -         -       error    -
root     getty       1289   wd -         -       error    -
root     getty       1289 text -         -       error    -
root     getty       1289 ctty /dev        266 crw-------   ttyv6 rw
root     getty       1289    0 /dev        266 crw-------   ttyv6 rw
root     getty       1288 root -         -       error    -
root     getty       1288   wd -         -       error    -
root     getty       1288 text -         -       error    -
root     getty       1288 ctty /dev        265 crw-------   ttyv5 rw
root     getty       1288    0 /dev        265 crw-------   ttyv5 rw
root     login       1287 root -         -       error    -
root     login       1287   wd -         -       error    -
root     login       1287 text -         -       error    -
root     login       1287 ctty /dev        264 crw--w----   ttyv4 rw
root     login       1287    0 /dev        264 crw--w----   ttyv4 rw
root     getty       1286 root -         -       error    -
root     getty       1286   wd -         -       error    -
root     getty       1286 text -         -       error    -
root     getty       1286 ctty /dev        263 crw-------   ttyv3 rw
root     getty       1286    0 /dev        263 crw-------   ttyv3 rw
root     getty       1285 root -         -       error    -
root     getty       1285   wd -         -       error    -
root     getty       1285 text -         -       error    -
root     getty       1285 ctty /dev        262 crw-------   ttyv2 rw
root     getty       1285    0 /dev        262 crw-------   ttyv2 rw
root     login       1284 root -         -       error    -
root     login       1284   wd -         -       error    -
root     login       1284 text -         -       error    -
root     login       1284 ctty /dev        261 crw--w----   ttyv1 rw
root     login       1284    0 /dev        261 crw--w----   ttyv1 rw
root     login       1283 root -         -       error    -
root     login       1283   wd -         -       error    -
root     login       1283 text -         -       error    -
root     login       1283 ctty /dev        260 crw--w----   ttyv0 rw
root     login       1283    0 /dev        260 crw--w----   ttyv0 rw
root     cron        1204 root -         -       error    -
root     cron        1204   wd -         -       error    -
root     cron        1204 text -         -       error    -
root     cron        1204    0 /dev         26 crw-rw-rw-    null rw
smmsp    sendmail    1196 root -         -       error    -
smmsp    sendmail    1196   wd -         -       error    -
smmsp    sendmail    1196 text -         -       error    -
smmsp    sendmail    1196    0 /dev         26 crw-rw-rw-    null  r
root     sendmail    1185 root -         -       error    -
root     sendmail    1185   wd -         -       error    -
root     sendmail    1185 text -         -       error    -
root     sendmail    1185    0 /dev         26 crw-rw-rw-    null  r
root     sshd        1171 root -         -       error    -
root     sshd        1171   wd -         -       error    -
root     sshd        1171 text -         -       error    -
root     sshd        1171    0 /dev         26 crw-rw-rw-    null rw
root     ntpd        1089 root -         -       error    -
root     ntpd        1089   wd -         -       error    -
root     ntpd        1089 text -         -       error    -
root     ntpd        1089    0 /dev         26 crw-rw-rw-    null  r
root     ntpd        1089    6 /dev         26 crw-rw-rw-    null  r
root     ntpd        1089   12 /dev         26 crw-rw-rw-    null  r
root     ntpd        1089   18* local dgram fffff8018c513870 <-> fffff801dae115a0
root     syslogd      882 root -         -       error    -
root     syslogd      882   wd -         -       error    -
root     syslogd      882 text -         -       error    -
root     syslogd      882    0 /dev         26 crw-rw-rw-    null rw
root     syslogd      882    6 /dev         26 crw-rw-rw-    null rw
root     devd         620 root -         -       error    -
root     devd         620   wd -         -       error    -
root     devd         620 text -         -       error    -
root     devd         620    0 /dev         26 crw-rw-rw-    null rw
root     devd         620    6 /dev         26 crw-rw-rw-    null rw
root     moused       606 root -         -       error    -
root     moused       606   wd -         -       error    -
root     moused       606 text -         -       error    -
root     moused       606    0 /dev         26 crw-rw-rw-    null rw
_dhcp    dhclient     553 root -         -       error    -
_dhcp    dhclient     553   wd -         -       error    -
_dhcp    dhclient     553 jail -         -       error    -
_dhcp    dhclient     553 text -         -       error    -
_dhcp    dhclient     553    0 /dev         26 crw-rw-rw-    null rw
_dhcp    dhclient     553    6 /dev         26 crw-rw-rw-    null rw
root     dhclient     491 root -         -       error    -
root     dhclient     491   wd -         -       error    -
root     dhclient     491 text -         -       error    -
root     dhclient     491    0 /dev         26 crw-rw-rw-    null rw
root     dhclient     491    6 /dev         26 crw-rw-rw-    null rw
root     adjkerntz    125 root -         -       error    -
root     adjkerntz    125   wd -         -       error    -
root     adjkerntz    125 text -         -       error    -
root     adjkerntz    125    0 /dev         26 crw-rw-rw-    null rw
root     zfskern        9 root -         -       error    -
root     zfskern        9   wd -         -       error    -
root     init           1 root -         -       error    -
root     init           1   wd -         -       error    -
root     init           1 text -         -       error    -
root     kernel         0 root -         -       error    -
root     kernel         0   wd -         -       error    -

------------------------------------------------------------------------
dmesg

Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.2-STABLE #0 r335674: Tue Jun 26 13:20:24 PDT 2018
    root@jimsdesk:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
VT(efifb): resolution 1680x1050
CPU: Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz (2300.07-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x50654  Family=0x6  Model=0x55  Stepping=4
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0xd39ffffb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,MPX,PQE,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PROCTRACE,AVX512CD,AVX512BW,AVX512VL>
  Structured Extended Features2=0x8<PKU>
  Structured Extended Features3=0xc000000<IBPB,STIBP>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 103079215104 (98304 MB)
avail memory = 98749128704 (94174 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <SUPERM SMCI--MB>
FreeBSD/SMP: Multiprocessor System Detected: 48 CPUs
FreeBSD/SMP: 2 package(s) x 12 core(s) x 2 hardware threads
ioapic6: Changing APIC ID to 16
ioapic7: Changing APIC ID to 17
ioapic8: Changing APIC ID to 18
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-31 on motherboard
ioapic2 <Version 2.0> irqs 32-39 on motherboard
ioapic3 <Version 2.0> irqs 40-47 on motherboard
ioapic4 <Version 2.0> irqs 48-55 on motherboard
ioapic5 <Version 2.0> irqs 72-79 on motherboard
ioapic6 <Version 2.0> irqs 80-87 on motherboard
ioapic7 <Version 2.0> irqs 88-95 on motherboard
ioapic8 <Version 2.0> irqs 96-103 on motherboard
SMP: AP CPU #1 Launched!
SMP: AP CPU #39 Launched!
SMP: AP CPU #8 Launched!
SMP: AP CPU #16 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #40 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #20 Launched!
SMP: AP CPU #42 Launched!
SMP: AP CPU #38 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #17 Launched!
SMP: AP CPU #15 Launched!
SMP: AP CPU #25 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #33 Launched!
SMP: AP CPU #43 Launched!
SMP: AP CPU #41 Launched!
SMP: AP CPU #21 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #46 Launched!
SMP: AP CPU #36 Launched!
SMP: AP CPU #34 Launched!
SMP: AP CPU #10 Launched!
SMP: AP CPU #14 Launched!
SMP: AP CPU #22 Launched!
SMP: AP CPU #27 Launched!
SMP: AP CPU #23 Launched!
SMP: AP CPU #37 Launched!
SMP: AP CPU #35 Launched!
SMP: AP CPU #30 Launched!
SMP: AP CPU #32 Launched!
SMP: AP CPU #24 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #28 Launched!
SMP: AP CPU #31 Launched!
SMP: AP CPU #44 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #26 Launched!
SMP: AP CPU #29 Launched!
SMP: AP CPU #47 Launched!
SMP: AP CPU #18 Launched!
SMP: AP CPU #13 Launched!
SMP: AP CPU #19 Launched!
SMP: AP CPU #12 Launched!
SMP: AP CPU #45 Launched!
Timecounter "TSC-low" frequency 1150036534 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff80ff4590, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
nexus0
cryptosoft0: <software crypto> on motherboard
acpi0: <SUPERM SUPERM> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> numa-domain 0 on acpi0
cpu1: <ACPI CPU> numa-domain 0 on acpi0
cpu2: <ACPI CPU> numa-domain 0 on acpi0
cpu3: <ACPI CPU> numa-domain 0 on acpi0
cpu4: <ACPI CPU> numa-domain 0 on acpi0
cpu5: <ACPI CPU> numa-domain 0 on acpi0
cpu6: <ACPI CPU> numa-domain 0 on acpi0
cpu7: <ACPI CPU> numa-domain 0 on acpi0
cpu8: <ACPI CPU> numa-domain 0 on acpi0
cpu9: <ACPI CPU> numa-domain 0 on acpi0
cpu10: <ACPI CPU> numa-domain 0 on acpi0
cpu11: <ACPI CPU> numa-domain 0 on acpi0
cpu12: <ACPI CPU> numa-domain 0 on acpi0
cpu13: <ACPI CPU> numa-domain 0 on acpi0
cpu14: <ACPI CPU> numa-domain 0 on acpi0
cpu15: <ACPI CPU> numa-domain 0 on acpi0
cpu16: <ACPI CPU> numa-domain 0 on acpi0
cpu17: <ACPI CPU> numa-domain 0 on acpi0
cpu18: <ACPI CPU> numa-domain 0 on acpi0
cpu19: <ACPI CPU> numa-domain 0 on acpi0
cpu20: <ACPI CPU> numa-domain 0 on acpi0
cpu21: <ACPI CPU> numa-domain 0 on acpi0
cpu22: <ACPI CPU> numa-domain 0 on acpi0
cpu23: <ACPI CPU> numa-domain 0 on acpi0
cpu24: <ACPI CPU> numa-domain 1 on acpi0
cpu25: <ACPI CPU> numa-domain 1 on acpi0
cpu26: <ACPI CPU> numa-domain 1 on acpi0
cpu27: <ACPI CPU> numa-domain 1 on acpi0
cpu28: <ACPI CPU> numa-domain 1 on acpi0
cpu29: <ACPI CPU> numa-domain 1 on acpi0
cpu30: <ACPI CPU> numa-domain 1 on acpi0
cpu31: <ACPI CPU> numa-domain 1 on acpi0
cpu32: <ACPI CPU> numa-domain 1 on acpi0
cpu33: <ACPI CPU> numa-domain 1 on acpi0
cpu34: <ACPI CPU> numa-domain 1 on acpi0
cpu35: <ACPI CPU> numa-domain 1 on acpi0
cpu36: <ACPI CPU> numa-domain 1 on acpi0
cpu37: <ACPI CPU> numa-domain 1 on acpi0
cpu38: <ACPI CPU> numa-domain 1 on acpi0
cpu39: <ACPI CPU> numa-domain 1 on acpi0
cpu40: <ACPI CPU> numa-domain 1 on acpi0
cpu41: <ACPI CPU> numa-domain 1 on acpi0
cpu42: <ACPI CPU> numa-domain 1 on acpi0
cpu43: <ACPI CPU> numa-domain 1 on acpi0
cpu44: <ACPI CPU> numa-domain 1 on acpi0
cpu45: <ACPI CPU> numa-domain 1 on acpi0
cpu46: <ACPI CPU> numa-domain 1 on acpi0
cpu47: <ACPI CPU> numa-domain 1 on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71,0x74-0x77 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 350
Event timer "HPET1" frequency 24000000 Hz quality 340
Event timer "HPET2" frequency 24000000 Hz quality 340
Event timer "HPET3" frequency 24000000 Hz quality 340
Event timer "HPET4" frequency 24000000 Hz quality 340
Event timer "HPET5" frequency 24000000 Hz quality 340
Event timer "HPET6" frequency 24000000 Hz quality 340
Event timer "HPET7" frequency 24000000 Hz quality 340
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x508-0x50b on acpi0
acpi_syscontainer0: <System Container> on acpi0
acpi_syscontainer1: <System Container> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff numa-domain 0 on acpi0
pcib0: _OSC returned error 0x10
pci0: <ACPI PCI bus> numa-domain 0 on pcib0
pci0: <dasp, performance counters> at device 8.1 (no driver attached)
pci0: <unknown> at device 17.0 (no driver attached)
pci0: <unknown> at device 17.1 (no driver attached)
ahci0: <Intel Lewisburg AHCI SATA controller> port 0x3070-0x3077,0x3060-0x3063,0x3020-0x303f mem 0x9d206000-0x9d207fff,0x9d209000-0x9d2090ff,0x9d180000-0x9d1fffff irq 16 at device 17.5 numa-domain 0 on pci0
ahci0: AHCI v1.31 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
xhci0: <Intel Lewisburg USB 3.0 controller> mem 0x383ffff00000-0x383ffff0ffff irq 16 at device 20.0 numa-domain 0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0: waiting for BIOS to give up control
xhci_interrupt: host controller halted
usbus0 numa-domain 0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <simple comms> at device 22.0 (no driver attached)
pci0: <simple comms> at device 22.1 (no driver attached)
pci0: <simple comms> at device 22.4 (no driver attached)
ahci1: <Intel Lewisburg AHCI SATA controller> port 0x3050-0x3057,0x3040-0x3043,0x3000-0x301f mem 0x9d204000-0x9d205fff,0x9d208000-0x9d2080ff,0x9d100000-0x9d17ffff irq 16 at device 23.0 numa-domain 0 on pci0
ahci1: AHCI v1.31 with 8 6Gbps ports, Port Multiplier not supported
ahcich6: <AHCI channel> at channel 0 on ahci1
ahcich7: <AHCI channel> at channel 1 on ahci1
ahcich8: <AHCI channel> at channel 2 on ahci1
ahcich9: <AHCI channel> at channel 3 on ahci1
ahcich10: <AHCI channel> at channel 4 on ahci1
ahcich11: <AHCI channel> at channel 5 on ahci1
ahcich12: <AHCI channel> at channel 6 on ahci1
ahcich13: <AHCI channel> at channel 7 on ahci1
ahciem1: <AHCI enclosure management bridge> on ahci1
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 numa-domain 0 on pci0
pci1: <ACPI PCI bus> numa-domain 0 on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.5 numa-domain 0 on pci0
pci2: <ACPI PCI bus> numa-domain 0 on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 17 at device 0.0 numa-domain 0 on pci2
pci3: <ACPI PCI bus> numa-domain 0 on pcib3
vgapci0: <VGA-compatible display> port 0x2000-0x207f mem 0x9c000000-0x9cffffff,0x9d000000-0x9d01ffff irq 17 at device 0.0 numa-domain 0 on pci3
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 numa-domain 0 on pci0
isa0: <ISA bus> numa-domain 0 on isab0
pci0: <memory> at device 31.2 (no driver attached)
pci0: <serial bus> at device 31.5 (no driver attached)
pcib4: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pcib4: _OSC returned error 0x10
pci4: <ACPI PCI bus> numa-domain 0 on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 39 at device 0.0 numa-domain 0 on pci4
pci5: <ACPI PCI bus> numa-domain 0 on pcib5
pcib6: <ACPI PCI-PCI bridge> mem 0xaae00000-0xaae1ffff irq 32 at device 0.0 numa-domain 0 on pci5
pci6: <ACPI PCI bus> numa-domain 0 on pcib6
pcib7: <PCI-PCI bridge> irq 38 at device 3.0 numa-domain 0 on pci6
pci7: <PCI bus> numa-domain 0 on pcib7
ixl0: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.9.9-k> mem 0xa9000000-0xa9ffffff,0xaa008000-0xaa00ffff irq 38 at device 0.0 numa-domain 0 on pci7
ixl0: using 1024 tx descriptors and 1024 rx descriptors
ixl0: fw 3.1.52349 api 1.5 nvm 3.25 etid 800009e7 oem 1.262.0
ixl0: PF-ID[0]: VFs 32, MSIX 129, VF MSIX 5, QPs 768, MDIO shared
ixl0: Using MSIX interrupts with 9 vectors
ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active
ixl0: Ethernet address: ac:1f:6b:61:54:06
ixl0: SR-IOV ready
queues is 0xfffffe0001d13000
ixl0: netmap queues/slots: TX 8/1024, RX 8/1024
ixl1: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.9.9-k> mem 0xa8000000-0xa8ffffff,0xaa000000-0xaa007fff irq 38 at device 0.1 numa-domain 0 on pci7
ixl1: using 1024 tx descriptors and 1024 rx descriptors
ixl1: fw 3.1.52349 api 1.5 nvm 3.25 etid 800009e7 oem 1.262.0
ixl1: PF-ID[1]: VFs 32, MSIX 129, VF MSIX 5, QPs 768, MDIO shared
ixl1: Using MSIX interrupts with 9 vectors
ixl1: Allocating 8 queues for PF LAN VSI; 8 queues active
ixl1: Ethernet address: ac:1f:6b:61:54:07
ixl1: SR-IOV ready
queues is 0xfffffe0001e69000
ixl1: netmap queues/slots: TX 8/1024, RX 8/1024
pcib8: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pcib8: _OSC returned error 0x10
pci8: <ACPI PCI bus> numa-domain 0 on pcib8
pcib9: <ACPI PCI-PCI bridge> irq 47 at device 2.0 numa-domain 0 on pci8
pci9: <ACPI PCI bus> numa-domain 0 on pcib9
mpr0: <Avago Technologies (LSI) SAS3008> port 0x7000-0x70ff mem 0xb8640000-0xb864ffff,0xb8600000-0xb863ffff irq 42 at device 0.0 numa-domain 0 on pci9
mpr0: Firmware: 15.00.03.00, Driver: 18.03.00.00-fbsd
mpr0: IOCCapabilities: 7a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc,FastPath,RDPQArray>
pcib10: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pcib10: _OSC returned error 0x10
pci10: <ACPI PCI bus> numa-domain 0 on pcib10
pci10: <dasp, performance counters> at device 14.0 (no driver attached)
pci10: <dasp, performance counters> at device 15.0 (no driver attached)
pci10: <dasp, performance counters> at device 18.0 (no driver attached)
pci10: <dasp, performance counters> at device 18.1 (no driver attached)
pcib11: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
ixl0: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: True, Flow Control: None
ixl0: link state changed to UP
ixl1: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: True, Flow Control: Full
ixl1: link state changed to UP
pcib11: _OSC returned error 0x10
pci11: <ACPI PCI bus> numa-domain 1 on pcib11
pci11: <dasp, performance counters> at device 8.1 (no driver attached)
pcib12: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
pcib12: _OSC returned error 0x10
pci12: <ACPI PCI bus> numa-domain 1 on pcib12
pcib13: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
pcib13: _OSC returned error 0x10
pci13: <ACPI PCI bus> numa-domain 1 on pcib13
pcib14: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
pcib14: _OSC returned error 0x10
pci14: <ACPI PCI bus> numa-domain 1 on pcib14
pci14: <dasp, performance counters> at device 14.0 (no driver attached)
pci14: <dasp, performance counters> at device 15.0 (no driver attached)
pci14: <dasp, performance counters> at device 18.0 (no driver attached)
pci14: <dasp, performance counters> at device 18.1 (no driver attached)
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
ppc0: cannot reserve I/O port range
est0: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu0
est1: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu1
est2: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu2
est3: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu3
est4: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu4
est5: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu5
est6: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu6
est7: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu7
est8: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu8
est9: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu9
est10: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu10
est11: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu11
est12: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu12
est13: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu13
est14: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu14
est15: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu15
est16: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu16
est17: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu17
est18: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu18
est19: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu19
est20: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu20
est21: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu21
est22: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu22
est23: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu23
est24: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu24
est25: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu25
est26: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu26
est27: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu27
est28: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu28
est29: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu29
est30: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu30
est31: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu31
est32: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu32
est33: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu33
est34: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu34
est35: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu35
est36: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu36
est37: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu37
est38: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu38
est39: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu39
est40: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu40
est41: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu41
est42: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu42
est43: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu43
est44: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu44
est45: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu45
est46: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu46
est47: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu47
random: unblocking device.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
mpr0: SAS Address from SAS device page0 = 5000c500955fb571
ugen0.1: <0x8086 XHCI root HUB> at usbus0
mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000a> enclosureHandle<0x0002> slot 0
mpr0: At enclosure level 0 and connector name (    )
mpr0: SAS Address from SAS device page0 = 5000c5009550e4f1
mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 1
mpr0: At enclosure level 0 and connector name (    )
mpr0: SAS Address from SAS device page0 = 5000c50095518c95
mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 2
mpr0: At enclosure level 0 and connector name (    )
mpr0: uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
SAS Address from SAS device page0 = 5000c500955fb3c5
mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000d> enclosureHandle<0x0002> slot 3
mpr0: At enclosure level 0 and connector name (    )
mpr0: SAS Address from SAS device page0 = 5000c500955fb021
mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000e> enclosureHandle<0x0002> slot 4
mpr0: At enclosure level 0 and connector name (    )
mpr0: SAS Address from SAS device page0 = 5000c5009550e891
mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000f> enclosureHandle<0x0002> slot 5
mpr0: At enclosure level 0 and connector name (    )
mpr0: SAS Address from SAS device page0 = 500304801eb5dbfd
mpr0: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x0010> enclosureHandle<0x0002> slot 12
mpr0: At enclosure level 0 and connector name (    )
ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ses1 at ahciem1 bus 0 scbus15 target 0 lun 0
ses1: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
ses1: SEMB SES Device
ses2 at mpr0 bus 0 scbus16 target 20 lun 0
ses2: <LSI SAS3x28 0601> Fixed Enclosure Services SPC-3 SCSI device
ses2: 1200.000MB/s transfers
ses2: Command Queueing enabled
ses2: SCSI-3 ENC Device
ada0 at ahcich4 bus 0 scbus4 target 0 lun 0
ada0: <INTEL SSDSC2KB240G7 SCV10121> ACS-3 ATA SATA 3.x device
ada0: Serial Number BTYS81150CWH240AGN
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 228936MB (468862128 512 byte sectors)
ada1 at ahcich5 bus 0 scbus5 target 0 lun 0
ada1: <INTEL SSDSC2KB240G7 SCV10121> ACS-3 ATA SATA 3.x device
ada1: Serial Number BTYS814205JS240AGN
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada1: Command Queueing enabled
ada1: 228936MB (468862128 512 byte sectors)
ses2: pass4,da0: Element descriptor: 'Slot00'
da4 at mpr0 bus 0 scbus16 target 12 lun 0
da0 at mpr0 bus 0 scbus16 target 8 lun 0
da1 at mpr0 bus 0 scbus16 target 9 lun 0
da2 at mpr0 bus 0 scbus16 target 10 lun 0
da5 at mpr0 bus 0 scbus16 target 13 lun 0
ses2: pass4,da0: SAS Device Slot Element: 1 Phys at Slot 0
ses2:  phy 0: SAS device type 1 id 0
ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
ses2:  phy 0: parent 500304801eb5dbff addr 5000c500955fb571
da1: <SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number ZC15V7DV0000C8258M38
da1: 1200.000MB/s transfers
uhub0: da1: Command Queueing enabled
da1: 3815447MB (976754646 4096 byte sectors)
da4: <SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number ZC1606RR0000R833G8YR
da4: 1200.000MB/s transfers
da4: Command Queueing enabled
da4: 3815447MB (976754646 4096 byte sectors)
da5: <SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
da2: <SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number ZC15T3GB0000C832EM0C
da2: 1200.000MB/s transfersda0: <SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number ZC1606NJ0000R833G9JK
da0: 1200.000MB/s transfers
da0: Command Queueing enabled
da0: 3815447MB (976754646 4096 byte sectors)
26 ports with 26 removable, self powered
da5: Serial Number ZC15KB320000R832RAR7
da5: 1200.000MB/s transfers
da5: Command Queueing enabled
da5: 3815447MB (976754646 4096 byte sectors)

da2: Command Queueing enabled
da2: 3815447MB (976754646 4096 byte sectors)
ses2: pass5,da1: Element descriptor: 'Slot01'
da3 at mpr0 bus 0 scbus16 target 11 lun 0
ses2: pass5,da1: SAS Device Slot Element: 1 Phys at Slot 1
ses2:  phy 0: SAS device type 1 id 0
ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
ses2:  phy 0: parent 500304801eb5dbff addr 5000c5009550e4f1
da3: <SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number ZC1606Q80000R833WKE8
da3: 1200.000MB/s transfers
da3: Command Queueing enabled
da3: 3815447MB (976754646 4096 byte sectors)
ses2: pass6,da2: Element descriptor: 'Slot02'
ses2: pass6,da2: SAS Device Slot Element: 1 Phys at Slot 2
ses2:  phy 0: SAS device type 1 id 0
ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
ses2:  phy 0: parent 500304801eb5dbff addr 5000c50095518c95
ugen0.2: <Dell Dell USB Entry Keyboard> at usbus0
ses2: pass7,da3: Element descriptor: 'Slot03'
ses2: pass7,da3: SAS Device Slot Element: 1 Phys at Slot 3
ses2:  phy 0: SAS device type 1 id 0
ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
ses2:  phy 0: parent 500304801eb5dbff addr 5000c500955fb3c5
ses2: pass8,da4: Element descriptor: 'Slot04'
ses2: pass8,da4: SAS Device Slot Element: 1 Phys at Slot 4
ses2:  phy 0: SAS device type 1 id 0
ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
ses2:  phy 0: parent 500304801eb5dbff addr 5000c500955fb021
ses2: pass9,da5: Element descriptor: 'Slot05'
ses2: pass9,da5: SAS Device Slot Element: 1 Phys at Slot 5
ses2:  phy 0: SAS device type 1 id 0
ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
ses2:  phy 0: parent 500304801eb5dbff addr 5000c5009550e891
ukbd0 numa-domain 0 on uhub0
ukbd0: <Dell Dell USB Entry Keyboard, class 0/0, rev 1.10/1.15, addr 1> on usbus0
Trying to mount root from zfs:mendeleev/default/root []...
Root mount waiting for: usbus0
kbd2 at ukbd0
ugen0.3: <vendor 0x0557 product 0x7000> at usbus0
uhub1 numa-domain 0 on uhub0
uhub1: <vendor 0x0557 product 0x7000, class 9/0, rev 2.00/0.00, addr 2> on usbus0
uhub1: 4 ports with 3 removable, self powered
Root mount waiting for: usbus0
ugen0.4: <vendor 0x0557 product 0x2419> at usbus0
ukbd1 numa-domain 0 on uhub1
ukbd1: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 3> on usbus0
kbd3 at ukbd1
/etc/rc: DEBUG: run_rc_command: doit: sysctl_start 
/etc/rc: DEBUG: checkyesno: hostid_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: doit: hostid_start 
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
Setting hostuuid: f6be925a-5a1e-11e8-bbbe-fe161f21af02.
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
Setting hostid: 0x2fca5b4d.
/etc/rc: DEBUG: checkyesno: zfs_enable is set to YES.
/etc/rc: DEBUG: load_kld: zfs kernel module already loaded.
/etc/rc: DEBUG: run_rc_command: doit: zvol_start 
/etc/rc: DEBUG: run_rc_command: doit: dumpon_start 
/etc/rc: DEBUG: checkyesno: ddb_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: start_precmd: [ -n "$(geli_make_list)" ] 
/etc/rc: DEBUG: run_rc_command: start_precmd: find_gbde_devices start 
/etc/rc: DEBUG: run_rc_command: doit: gbde_start 
/etc/rc: DEBUG: run_rc_command: doit: ccd_start 
/etc/rc: DEBUG: run_rc_command: doit: /sbin/swapon -aq 
/etc/rc: DEBUG: run_rc_command: doit: fsck_start 
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
Starting file system checks:
/etc/rc: DEBUG: checkyesno: background_fsck is set to YES.
/etc/rc: DEBUG: run_rc_command: doit: root_start 
/etc/rc: DEBUG: run_rc_command: start_precmd: [ -n "${_mdconfig_list}" ] 
/etc/rc: DEBUG: checkyesno: hostid_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: doit: hostid_save 
/etc/rc: DEBUG: run_rc_command: doit: mountcritlocal_start 
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
Mounting local filesystems:/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
.
/etc/rc: DEBUG: checkyesno: zfs_enable is set to YES.
/etc/rc: DEBUG: load_kld: zfs kernel module already loaded.
/etc/rc: DEBUG: run_rc_command: doit: be_start 
/etc/rc: DEBUG: checkyesno: zfs_enable is set to YES.
/etc/rc: DEBUG: load_kld: zfs kernel module already loaded.
/etc/rc: DEBUG: run_rc_command: doit: zfs_start 
/etc/rc: DEBUG: checkyesno: cleanvar_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: start_precmd: cleanvar_prestart 
/etc/rc: DEBUG: run_rc_command: doit: cleanvar_start 
/etc/rc: DEBUG: run_rc_command: doit: ldconfig_start 
/etc/rc: DEBUG: checkyesno: ldconfig_insecure is set to NO.
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/perl5/5.26/mach/CORE
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
32-bit compatibility ldconfig path: /usr/lib32
/etc/rc: DEBUG: checkyesno: kldxref_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: doit: kld_start 
/etc/rc: DEBUG: set_rcvar_obsolete: $swapfile(old) -> $(new) is defined
/etc/rc: DEBUG: set_rcvar_obsolete: $geli_swap_flags(old) -> $(new) is defined
/etc/rc: DEBUG: run_rc_command: doit: : 
/etc/rc: DEBUG: run_rc_command: doit: adjkerntz -i 
/etc/rc: DEBUG: checkyesno: atm_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: doit: hostname_start 
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
Setting hostname: mendeleev3/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
.
/etc/rc: DEBUG: set_rcvar_obsolete: $ipv6_enable(old) -> $ipv6_activate_all_interfaces(new) is defined
/etc/rc: DEBUG: set_rcvar_obsolete: $ipv6_prefer(old) -> $ip6addrctl_policy(new) is defined
/etc/rc: DEBUG: checkyesno: ip6addrctl_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: doit: ip6addrctl_start 
/etc/rc: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO.
/etc/rc: DEBUG: checkyesno: ip6addrctl_verbose is set to NO.
/etc/rc: DEBUG: run_rc_command: doit: netoptions_start 
/etc/rc: DEBUG: checkyesno: tcp_extensions is set to YES.
/etc/rc: DEBUG: checkyesno: tcp_keepalive is set to YES.
/etc/rc: DEBUG: checkyesno: tcp_drop_synfin is set to NO.
/etc/rc: DEBUG: checkyesno: ipv6_ipv4mapping is set to NO.
/etc/rc: DEBUG: checkyesno: ipv6_privacy is set to NO.
/etc/rc: DEBUG: run_rc_command: doit: random_start 
Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy: /etc/rc: DEBUG: entropy read from /var/db/entropy/saved-entropy.1
/etc/rc: DEBUG: entropy read from /var/db/entropy/saved-entropy.2
/etc/rc: DEBUG: entropy read from /var/db/entropy/saved-entropy.3
/etc/rc: DEBUG: entropy read from /var/db/entropy/saved-entropy.4
/etc/rc: DEBUG: entropy read from /entropy
/etc/rc: DEBUG: saving entropy to /entropy
/etc/rc: DEBUG: saving entropy to /boot/entropy
.
/etc/rc: DEBUG: run_rc_command: doit: sppp_start 
/etc/rc: DEBUG: checkyesno: ipfilter_enable is set to NO.
/etc/rc: DEBUG: checkyesno: ipnat_enable is set to NO.
/etc/rc: DEBUG: checkyesno: ipfs_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: doit: iovctl_start 
/etc/rc: DEBUG: set_rcvar_obsolete: $ipv6_enable(old) -> $ipv6_activate_all_interfaces(new) is defined
/etc/rc: DEBUG: set_rcvar_obsolete: $ipv6_prefer(old) -> $(new) is defined
/etc/rc: DEBUG: checkyesno: netif_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: doit: netif_start 
/etc/rc: DEBUG: Created wlan(4)s: 
/etc/rc: DEBUG: Cloned: 
/etc/rc: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO.
lo0: link state changed to UP
/etc/rc: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO.
/etc/rc: DEBUG: checkyesno: synchronous_dhclient is set to NO.
/etc/rc: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO.
Starting Network: lo0 ixl0 ixl1.
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo 
ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=6407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether ac:1f:6b:61:54:06
	hwaddr ac:1f:6b:61:54:06
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
ixl1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=6407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether ac:1f:6b:61:54:07
	hwaddr ac:1f:6b:61:54:07
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
	status: active
/etc/rc: DEBUG: The following interfaces were not configured: 
/etc/rc.d/ipfilter: DEBUG: checkyesno: ipfilter_enable is set to NO.
/etc/rc: DEBUG: checkyesno: devd_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: start_precmd: devd_prestart 
/etc/rc: DEBUG: checkyesno: devd_enable is set to YES.
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
Starting devd.
/etc/rc: DEBUG: run_rc_command: doit:  limits -C daemon /sbin/devd  
/etc/pccard_ether: DEBUG: run_rc_command: start_precmd: checkauto 
/etc/pccard_ether: DEBUG: run_rc_command: doit: pccard_ether_start 
/etc/pccard_ether: DEBUG: run_rc_command: start_precmd: checkauto 
/etc/pccard_ether: DEBUG: run_rc_command: doit: pccard_ether_start 
/etc/rc.d/netif: DEBUG: checkyesno: netif_enable is set to YES.
/etc/rc.d/netif: DEBUG: run_rc_command: doit: netif_start ixl1
/etc/rc.d/netif: DEBUG: Created wlan(4)s: 
/etc/rc.d/netif: DEBUG: Cloned: 
/etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO.
Starting Network: ixl1.
/etc/rc.d/netif: DEBUG: checkyesno: rc_startmsgs is set to YES.
ixl1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=6407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether ac:1f:6b:61:54:07
	hwaddr ac:1f:6b:61:54:07
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
	status: active
/etc/rc.d/netif: DEBUG: The following interfaces were not configured: 
/etc/rc.d/ipfilter: DEBUG: checkyesno: ipfilter_enable is set to NO.
/etc/rc.d/bridge: DEBUG: run_rc_command: doit: bridge_start 
/etc/rc.d/routing: DEBUG: run_rc_command: doit: routing_start static any ixl1
/etc/rc.d/routing: DEBUG: checkyesno: ipv6_gateway_enable is set to NO.
/etc/rc.d/dhclient: DEBUG: pid file (/var/run/dhclient.ixl0.pid): not readable.
/etc/rc.d/dhclient: DEBUG: run_rc_command: start_precmd: dhclient_prestart 
/etc/rc.d/dhclient: DEBUG: checkyesno: background_dhclient is set to NO.
/etc/rc.d/dhclient: DEBUG: checkyesno: rc_startmsgs is set to YES.
Starting dhclient.
/etc/rc.d/dhclient: DEBUG: run_rc_command: doit:  limits -C daemon /sbin/dhclient  ixl0 
DHCPDISCOVER on ixl0 to 255.255.255.255 port 67 interval 5
DHCPOFFER from 192.168.1.1
DHCPREQUEST on ixl0 to 255.255.255.255 port 67
DHCPACK from 192.168.1.1
bound to 192.168.1.13 -- renewal in 300 seconds.
/etc/rc.d/dhclient: DEBUG: pid file (/var/run/dhclient.ixl1.pid): not readable.
/etc/rc.d/dhclient: DEBUG: run_rc_command: start_precmd: dhclient_prestart 
/etc/rc.d/dhclient: DEBUG: 'ixl1' is not a DHCP-enabled interface
/etc/pccard_ether: DEBUG: run_rc_command: start_precmd: checkauto 
/etc/pccard_ether: DEBUG: run_rc_command: doit: pccard_ether_start 
/etc/rc.d/syscons: DEBUG: run_rc_command: doit: syscons_setkeyboard /dev/ukbd0
ums0 numa-domain 0 on uhub1
ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 3> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
/etc/rc.d/moused: DEBUG: pid file (/var/run/moused.ums0.pid): not readable.
/etc/rc.d/moused: DEBUG: checkyesno: moused_ums0_enable is set to YES.
/etc/rc.d/moused: DEBUG: run_rc_command: doit: moused_start ums0
/etc/rc.d/moused: DEBUG: checkyesno: rc_startmsgs is set to YES.
Starting ums0 moused/etc/rc.d/moused: DEBUG: checkyesno: rc_startmsgs is set to YES.
.
/etc/rc: DEBUG: checkyesno: ipsec_enable is set to NO.
/etc/rc: DEBUG: checkyesno: atm_enable is set to NO.
/etc/rc: DEBUG: checkyesno: pfsync_enable is set to NO.
/etc/rc: DEBUG: checkyesno: pflog_enable is set to NO.
/etc/rc: DEBUG: checkyesno: pf_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: doit: stf_up 
/etc/rc: DEBUG: checkyesno: ppp_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: doit: routing_start doall 
add host 127.0.0.1: gateway lo0 fib 0: route already in table
/etc/rc: DEBUG: checkyesno: icmp_bmcastecho is set to NO.
/etc/rc: DEBUG: checkyesno: icmp_drop_redirect is set to NO.
/etc/rc: DEBUG: checkyesno: icmp_log_redirect is set to NO.
/etc/rc: DEBUG: checkyesno: gateway_enable is set to NO.
/etc/rc: DEBUG: checkyesno: forward_sourceroute is set to NO.
/etc/rc: DEBUG: checkyesno: accept_sourceroute is set to NO.
/etc/rc: DEBUG: checkyesno: arpproxy_all is set to NO.
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
/etc/rc: DEBUG: checkyesno: ipv6_gateway_enable is set to NO.
/etc/rc: DEBUG: checkyesno: ipv6_gateway_enable is set to NO.
/etc/rc: DEBUG: set_rcvar_obsolete: $ipv6_firewall_enable(old) -> $(new) is defined
/etc/rc: DEBUG: checkyesno: firewall_enable is set to NO.
/etc/rc: DEBUG: checkyesno: netwait_enable is set to NO.
/etc/rc: DEBUG: checkyesno: local_unbound_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: doit: nsswitch_start 
/etc/rc: DEBUG: set_rcvar_obsolete: $router_enable(old) -> $routed_enable(new) is defined
/etc/rc: DEBUG: set_rcvar_obsolete: $router(old) -> $routed_program(new) is defined
/etc/rc: DEBUG: set_rcvar_obsolete: $router_flags(old) -> $routed_flags(new) is defined
/etc/rc: DEBUG: checkyesno: routed_enable is set to NO.
/etc/rc: DEBUG: checkyesno: rtsold_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: doit: static_ndp_start 
/etc/rc: DEBUG: run_rc_command: doit: static_arp_start 
/etc/rc: DEBUG: run_rc_command: doit: bridge_start 
/etc/rc: DEBUG: set_rcvar_obsolete: $ipv6_router_enable(old) -> $route6d_enable(new) is defined
/etc/rc: DEBUG: set_rcvar_obsolete: $ipv6_router(old) -> $route6d_program(new) is defined
/etc/rc: DEBUG: set_rcvar_obsolete: $ipv6_router_flags(old) -> $route6d_flags(new) is defined
/etc/rc: DEBUG: checkyesno: route6d_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: doit: defaultroute_start 
/etc/rc: DEBUG: run_rc_command: start_precmd: mountcritremote_precmd 
/etc/rc: DEBUG: run_rc_command: doit: mountcritremote_start 
/etc/rc: DEBUG: checkyesno: dmesg_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: doit: do_dmesg 
/etc/rc: DEBUG: run_rc_command: doit: devfs_start 
/etc/rc: DEBUG: checkyesno: devfs_load_rulesets is set to YES.
/etc/rc: DEBUG: reading rulesets from file (/etc/defaults/devfs.rules)
/etc/rc: DEBUG: found ruleset: devfsrules_hide_all=1
/etc/rc: DEBUG: adding rule (add hide)
/etc/rc: DEBUG: found ruleset: devfsrules_unhide_basic=2
/etc/rc: DEBUG: adding rule (add path log unhide)
/etc/rc: DEBUG: adding rule (add path null unhide)
/etc/rc: DEBUG: adding rule (add path zero unhide)
/etc/rc: DEBUG: adding rule (add path crypto unhide)
/etc/rc: DEBUG: adding rule (add path random unhide)
/etc/rc: DEBUG: adding rule (add path urandom unhide)
/etc/rc: DEBUG: found ruleset: devfsrules_unhide_login=3
/etc/rc: DEBUG: adding rule (add path 'ptyp*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ptyq*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ptyr*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ptys*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ptyP*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ptyQ*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ptyR*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ptyS*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ptyl*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ptym*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ptyn*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ptyo*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ptyL*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ptyM*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ptyN*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ptyO*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ttyp*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ttyq*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ttyr*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ttys*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ttyP*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ttyQ*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ttyR*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ttyS*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ttyl*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ttym*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ttyn*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ttyo*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ttyL*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ttyM*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ttyN*' unhide)
/etc/rc: DEBUG: adding rule (add path 'ttyO*' unhide)
/etc/rc: DEBUG: adding rule (add path ptmx unhide)
/etc/rc: DEBUG: adding rule (add path pts unhide)
/etc/rc: DEBUG: adding rule (add path 'pts/*' unhide)
/etc/rc: DEBUG: adding rule (add path fd unhide)
/etc/rc: DEBUG: adding rule (add path 'fd/*' unhide)
/etc/rc: DEBUG: adding rule (add path stdin unhide)
/etc/rc: DEBUG: adding rule (add path stdout unhide)
/etc/rc: DEBUG: adding rule (add path stderr unhide)
/etc/rc: DEBUG: found ruleset: devfsrules_jail=4
/etc/rc: DEBUG: adding rule (add include $devfsrules_hide_all)
/etc/rc: DEBUG: adding rule (add include $devfsrules_unhide_basic)
/etc/rc: DEBUG: adding rule (add include $devfsrules_unhide_login)
/etc/rc: DEBUG: adding rule (add path zfs unhide)
/etc/rc: DEBUG: devfs_rulesets_from_file: no such file (/etc/devfs.rules)
/etc/rc: DEBUG: devfs_init_rulesets: devfs rulesets initialized
/etc/rc: DEBUG: checkyesno: ipmon_enable is set to NO.
/etc/rc: DEBUG: set_rcvar_obsolete: $kerberos5_server_enable(old) -> $kdc_enable(new) is defined
/etc/rc: DEBUG: set_rcvar_obsolete: $kerberos5_server(old) -> $kdc_program(new) is defined
/etc/rc: DEBUG: set_rcvar_obsolete: $kerberos5_server_flags(old) -> $kdc_flags(new) is defined
/etc/rc: DEBUG: checkyesno: kdc_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: start_precmd: [ -n "${_mdconfig2_list}" ] 
/etc/rc: DEBUG: checkyesno: newsyslog_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: doit: newsyslog_start 
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
Creating and/or trimming log files/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
.
/etc/rc: DEBUG: checkyesno: syslogd_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: start_precmd: syslogd_precmd 
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
Starting syslogd.
/etc/rc: DEBUG: run_rc_command: doit:  limits -C daemon /usr/sbin/syslogd -s 
/etc/rc: DEBUG: checkyesno: watchdogd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: savecore_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: start_precmd: savecore_prestart 
/etc/rc: DEBUG: run_rc_command: doit: savecore_start 
savecore: reboot after panic: kdb_sysctl_panic
savecore: writing core to /var/crash/vmcore.2
/etc/rc: DEBUG: checkyesno: crashinfo_enable is set to YES.
Writing crash summary to /var/crash/core.txt.2.
/etc/rc: DEBUG: run_rc_command: doit: archdep_start 
/etc/rc: DEBUG: run_rc_command: doit: abi_start 
/etc/rc: DEBUG: checkyesno: sysvipc_enable is set to NO.
/etc/rc: DEBUG: checkyesno: linux_enable is set to NO.
/etc/rc: DEBUG: checkyesno: svr4_enable is set to NO.
/etc/rc: DEBUG: checkyesno: sysvipc_enable is set to NO.
/etc/rc: DEBUG: checkyesno: linux_enable is set to NO.
/etc/rc: DEBUG: checkyesno: svr4_enable is set to NO.
/etc/rc: DEBUG: checkyesno: vm_enable is set to NO.
/etc/rc: DEBUG: checkyesno: accounting_enable is set to NO.
/etc/rc: DEBUG: checkyesno: ntpdate_enable is set to NO.
/etc/rc: DEBUG: checkyesno: rpcbind_enable is set to NO.
/etc/rc: DEBUG: checkyesno: nfs_client_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: doit: nisdomain_start 
/etc/rc: DEBUG: checkyesno: nis_server_enable is set to NO.
/etc/rc: DEBUG: checkyesno: nis_client_enable is set to NO.
/etc/rc: DEBUG: checkyesno: nis_ypset_enable is set to NO.
/etc/rc: DEBUG: checkyesno: amd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: atm_enable is set to NO.
/etc/rc: DEBUG: checkyesno: auditd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: auditdistd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: autofs_enable is set to NO.
/etc/rc: DEBUG: checkyesno: autofs_enable is set to NO.
/etc/rc: DEBUG: checkyesno: autofs_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: doit: cleartmp_start 
/etc/rc: DEBUG: checkyesno: clear_tmp_enable is set to NO.
/etc/rc: DEBUG: checkyesno: clear_tmp_X is set to YES.
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
Clearing /tmp (X related).
/etc/rc: DEBUG: checkyesno: clear_tmp_X is set to YES.
/etc/rc: DEBUG: checkyesno: ctld_enable is set to NO.
/etc/rc: DEBUG: checkyesno: hastd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: iscsid_enable is set to NO.
/etc/rc: DEBUG: checkyesno: iscsictl_enable is set to NO.
/etc/rc: DEBUG: checkyesno: keyserv_enable is set to NO.
/etc/rc: DEBUG: checkyesno: nfsuserd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: gssd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: quota_enable is set to NO.
/etc/rc: DEBUG: checkyesno: mountd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: nfs_server_enable is set to NO.
/etc/rc: DEBUG: checkyesno: rpc_statd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: rpc_lockd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: pppoed_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: doit: pwcheck_start 
/etc/rc: DEBUG: checkyesno: virecover_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: doit: virecover_start 
Recovering vi editor sessions:.
/etc/rc: DEBUG: checkyesno: nis_ypldap_enable is set to NO.
/etc/rc: DEBUG: checkyesno: apm_enable is set to NO.
/etc/rc: DEBUG: checkyesno: apmd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: bootparamd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: hcsecd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: bthidd_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: doit: local_start 
/etc/rc: DEBUG: checkyesno: lpd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: update_motd is set to YES.
/etc/rc: DEBUG: run_rc_command: doit: motd_start 
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
Updating motd:/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
.
/etc/rc: DEBUG: run_rc_command: doit: mountlate_start 
Mounting late filesystems:.
/etc/rc: DEBUG: checkyesno: nscd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: ntpd_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: start_precmd: ntpd_precmd 
/etc/rc: DEBUG: checkyesno: ntpd_sync_on_start is set to NO.
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
Starting ntpd.
/etc/rc: DEBUG: run_rc_command: doit:  limits -C daemon /usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift 
/etc/rc: DEBUG: checkyesno: powerd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: rarpd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: rctl_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: doit: rctl_start 
/etc/rc: DEBUG: checkyesno: sdpd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: rfcomm_pppd_server_enable is set to NO.
/etc/rc: DEBUG: checkyesno: rtadvd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: rwhod_enable is set to NO.
/etc/rc: DEBUG: checkyesno: timed_enable is set to NO.
/etc/rc: DEBUG: checkyesno: ugidfw_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: doit: utx boot 
/etc/rc: DEBUG: checkyesno: nis_yppasswdd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: smartd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: rsyncd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: healthd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: git_daemon_enable is set to NO.
/etc/rc: DEBUG: checkyesno: zfsd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: nis_ypxfrd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: rpc_ypupdated_enable is set to NO.
/etc/rc: DEBUG: checkyesno: ubthidhci_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: start_precmd: syscons_precmd 
/etc/rc: DEBUG: run_rc_command: doit: syscons_start 
Configuring vt: blanktime.
/etc/rc: DEBUG: run_rc_command: doit: /sbin/swapon -aLq 
/etc/rc: DEBUG: checkyesno: sshd_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: start_precmd: sshd_precmd 
/etc/rc: DEBUG: checkyesno: sshd_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: doit: sshd_keygen 
/etc/rc: DEBUG: checkyesno: sshd_rsa1_enable is set to no.
/etc/rc: DEBUG: checkyesno: sshd_rsa_enable is set to yes.
/etc/rc: DEBUG: checkyesno: sshd_dsa_enable is set to no.
/etc/rc: DEBUG: checkyesno: sshd_ecdsa_enable is set to yes.
/etc/rc: DEBUG: checkyesno: sshd_ed25519_enable is set to yes.
/etc/rc: DEBUG: checkyesno: sshd_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: doit: sshd_configtest 
Performing sanity check on sshd configuration.
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
Starting sshd.
/etc/rc: DEBUG: run_rc_command: doit:  limits -C daemon /usr/sbin/sshd  
/etc/rc: DEBUG: checkyesno: sendmail_enable is set to NO.
/etc/rc: DEBUG: checkyesno: sendmail_submit_enable is set to YES.
/etc/rc: DEBUG: checkyesno: sendmail_enable is set to NO.
/etc/rc: DEBUG: checkyesno: sendmail_submit_enable is set to YES.
/etc/rc: DEBUG: checkyesno: sendmail_submit_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: start_precmd: sendmail_precmd 
/etc/rc: DEBUG: checkyesno: sendmail_submit_enable is set to YES.
/etc/rc: DEBUG: checkyesno: sendmail_rebuild_aliases is set to NO.
/etc/rc: DEBUG: checkyesno: sendmail_cert_create is set to YES.
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
Starting sendmail_submit.
/etc/rc: DEBUG: run_rc_command: doit:  limits -C daemon /usr/sbin/sendmail -L sm-mta -bd -q30m -ODaemonPortOptions=Addr=localhost 
/etc/rc: DEBUG: checkyesno: sendmail_outbound_enable is set to NO.
/etc/rc: DEBUG: checkyesno: sendmail_msp_queue_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: start_precmd: sendmail_precmd 
/etc/rc: DEBUG: checkyesno: sendmail_msp_queue_enable is set to YES.
/etc/rc: DEBUG: checkyesno: sendmail_rebuild_aliases is set to NO.
/etc/rc: DEBUG: checkyesno: sendmail_cert_create is set to YES.
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
Starting sendmail_msp_queue.
/etc/rc: DEBUG: run_rc_command: doit:  limits -C daemon /usr/sbin/sendmail -L sm-msp-queue -Ac -q30m 
/etc/rc: DEBUG: checkyesno: cron_dst is set to YES.
/etc/rc: DEBUG: checkyesno: cron_enable is set to YES.
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
Starting cron.
/etc/rc: DEBUG: run_rc_command: doit:  limits -C daemon /usr/sbin/cron  -s 
/etc/rc: DEBUG: checkyesno: jail_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: doit: pkg_start 
/etc/rc.d/sysctl: DEBUG: run_rc_command: doit: sysctl_start last 
/etc/rc: DEBUG: checkyesno: kern_securelevel_enable is set to NO.
/etc/rc: DEBUG: checkyesno: nfscbd_enable is set to NO.
/etc/rc: DEBUG: run_rc_command: doit: msgs_start 
/etc/rc: DEBUG: checkyesno: moused_enable is set to NO.
/etc/rc: DEBUG: checkyesno: mixer_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: doit: mixer_start 
/etc/rc: DEBUG: set_rcvar_obsolete: $kpasswdd_server_enable(old) -> $kpasswdd_enable(new) is defined
/etc/rc: DEBUG: set_rcvar_obsolete: $kpasswdd_server(old) -> $kpasswdd_program(new) is defined
/etc/rc: DEBUG: set_rcvar_obsolete: $kerberos5_server_enable(old) -> $kdc_enable(new) is defined
/etc/rc: DEBUG: checkyesno: kpasswdd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: kfd_enable is set to NO.
/etc/rc: DEBUG: set_rcvar_obsolete: $kadmind5_server_enable(old) -> $kadmind_enable(new) is defined
/etc/rc: DEBUG: set_rcvar_obsolete: $kadmind5_server(old) -> $kadmind_program(new) is defined
/etc/rc: DEBUG: set_rcvar_obsolete: $kerberos5_server_enable(old) -> $kdc_enable(new) is defined
/etc/rc: DEBUG: checkyesno: kadmind_enable is set to NO.
/etc/rc: DEBUG: checkyesno: ipropd_slave_enable is set to NO.
/etc/rc: DEBUG: checkyesno: ipropd_master_enable is set to NO.
/etc/rc: DEBUG: checkyesno: ipfw_netflow_enable is set to NO.
/etc/rc: DEBUG: checkyesno: inetd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: hostapd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: gptboot_enable is set to YES.
/etc/rc: DEBUG: run_rc_command: doit: gptboot_report 
/etc/rc: DEBUG: run_rc_command: doit: geli2_start 
/etc/rc: DEBUG: checkyesno: ftpd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: ftpproxy_enable is set to NO.
/etc/rc: DEBUG: checkyesno: bsnmpd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: blacklistd_enable is set to NO.
/etc/rc: DEBUG: checkyesno: background_fsck is set to YES.
/etc/rc: DEBUG: run_rc_command: start_precmd: bgfsck_start_precmd 
/etc/rc: DEBUG: run_rc_command: doit: bgfsck_start 
/etc/rc: DEBUG: checkyesno: rc_startmsgs is set to YES.
Starting background file system checks in 60 seconds.

Wed Jul 11 12:48:26 PDT 2018


Fatal trap 12: page fault while in kernel mode
cpuid = 38; apic id = 32
fault virtual address	= 0x5a
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80dff30d
stack pointer	        = 0x28:0xfffffe17eaaa8440
frame pointer	        = 0x28:0xfffffe17eaaa8480
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 0 (system_taskq_33)
trap number		= 12
panic: page fault
cpuid = 38
KDB: stack backtrace:
#0 0xffffffff80b3d907 at kdb_backtrace+0x67
#1 0xffffffff80af6e77 at vpanic+0x177
#2 0xffffffff80af6cf3 at panic+0x43
#3 0xffffffff80f77fef at trap_fatal+0x35f
#4 0xffffffff80f78049 at trap_pfault+0x49
#5 0xffffffff80f77824 at trap+0x2d4
#6 0xffffffff80f5786c at calltrap+0x8
#7 0xffffffff80f7938b at uma_small_alloc+0x5b
#8 0xffffffff80de5b72 at keg_alloc_slab+0xe2
#9 0xffffffff80de857e at keg_fetch_slab+0x14e
#10 0xffffffff80de7db4 at zone_fetch_slab+0x64
#11 0xffffffff80de7e8f at zone_import+0x3f
#12 0xffffffff80de4599 at uma_zalloc_arg+0x3d9
#13 0xffffffff8225b692 at abd_alloc+0x112
#14 0xffffffff8226064e at arc_hdr_alloc_pabd+0x8e
#15 0xffffffff8225dbcf at arc_hdr_alloc+0x11f
#16 0xffffffff8225f84f at arc_read+0x24f
#17 0xffffffff8228342d at traverse_prefetcher+0x18d
Uptime: 29m53s
Dumping 7011 out of 96901 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

------------------------------------------------------------------------
kernel config

options	CONFIG_AUTOGENERATED
ident	GENERIC
machine	amd64
cpu	HAMMER
makeoptions	WITH_CTF=1
makeoptions	DEBUG=-g
options	XENHVM
options	USB_DEBUG
options	ATH_ENABLE_11N
options	AH_AR5416_INTERRUPT_MITIGATION
options	AH_SUPPORT_AR5416
options	IEEE80211_SUPPORT_MESH
options	IEEE80211_AMPDU_AGE
options	IEEE80211_DEBUG
options	SC_PIXEL_MODE
options	VESA
options	AHD_REG_PRETTY_PRINT
options	AHC_REG_PRETTY_PRINT
options	PCI_IOV
options	PCI_HP
options	ACPI_DMAR
options	EARLY_AP_STARTUP
options	DEVICE_NUMA
options	SMP
options	KDB_TRACE
options	KDB
options	RCTL
options	RACCT_DEFAULT_TO_DISABLED
options	RACCT
options	INCLUDE_CONFIG_FILE
options	DDB_CTF
options	KDTRACE_HOOKS
options	KDTRACE_FRAME
options	MAC
options	CAPABILITIES
options	CAPABILITY_MODE
options	AUDIT
options	HWPMC_HOOKS
options	KBD_INSTALL_CDEV
options	PRINTF_BUFR_SIZE=128
options	_KPOSIX_PRIORITY_SCHEDULING
options	SYSVSEM
options	SYSVMSG
options	SYSVSHM
options	STACK
options	KTRACE
options	SCSI_DELAY=5000
options	COMPAT_FREEBSD10
options	COMPAT_FREEBSD9
options	COMPAT_FREEBSD7
options	COMPAT_FREEBSD6
options	COMPAT_FREEBSD5
options	COMPAT_FREEBSD4
options	COMPAT_FREEBSD32
options	GEOM_LABEL
options	GEOM_RAID
options	GEOM_PART_GPT
options	PSEUDOFS
options	PROCFS
options	CD9660
options	MSDOSFS
options	NFS_ROOT
options	NFSLOCKD
options	NFSD
options	NFSCL
options	MD_ROOT
options	QUOTA
options	UFS_GJOURNAL
options	UFS_DIRHASH
options	UFS_ACL
options	SOFTUPDATES
options	FFS
options	SCTP
options	TCP_OFFLOAD
options	IPSEC_SUPPORT
options	IPSEC
options	INET6
options	INET
options	PREEMPTION
options	SCHED_ULE
options	NEW_PCIB
options	GEOM_PART_MBR
options	GEOM_PART_EBR_COMPAT
options	GEOM_PART_EBR
options	GEOM_PART_BSD
device	isa
device	mem
device	io
device	uart_ns8250
device	cpufreq
device	acpi
device	pci
device	fdc
device	ahci
device	ata
device	mvs
device	siis
device	ahc
device	ahd
device	esp
device	hptiop
device	isp
device	mpt
device	mps
device	mpr
device	sym
device	trm
device	adv
device	adw
device	aic
device	bt
device	isci
device	ocs_fc
device	scbus
device	ch
device	da
device	sa
device	cd
device	pass
device	ses
device	amr
device	arcmsr
device	ciss
device	dpt
device	hptmv
device	hptnr
device	hptrr
device	hpt27xx
device	iir
device	ips
device	mly
device	twa
device	smartpqi
device	tws
device	aac
device	aacp
device	aacraid
device	ida
device	mfi
device	mlx
device	mrsas
device	pmspcv
device	twe
device	nvme
device	nvd
device	atkbdc
device	atkbd
device	psm
device	kbdmux
device	vga
device	splash
device	sc
device	vt
device	vt_vga
device	vt_efifb
device	agp
device	cbb
device	pccard
device	cardbus
device	uart
device	ppc
device	ppbus
device	lpt
device	ppi
device	puc
device	bxe
device	de
device	em
device	igb
device	ix
device	ixv
device	ixl
device	ixlv
device	le
device	ti
device	txp
device	vx
device	miibus
device	ae
device	age
device	alc
device	ale
device	bce
device	bfe
device	bge
device	cas
device	dc
device	et
device	fxp
device	gem
device	hme
device	jme
device	lge
device	msk
device	nfe
device	nge
device	pcn
device	re
device	rl
device	sf
device	sge
device	sis
device	sk
device	ste
device	stge
device	tl
device	tx
device	vge
device	vr
device	wb
device	xl
device	wlan
device	wlan_wep
device	wlan_ccmp
device	wlan_tkip
device	wlan_amrr
device	an
device	ath
device	ath_pci
device	ath_hal
device	ath_rate_sample
device	ipw
device	iwi
device	iwn
device	malo
device	mwl
device	ral
device	wi
device	wpi
device	loop
device	random
device	padlock_rng
device	rdrand_rng
device	ether
device	vlan
device	tun
device	md
device	gif
device	firmware
device	bpf
device	uhci
device	ohci
device	ehci
device	xhci
device	usb
device	ukbd
device	umass
device	sound
device	snd_cmi
device	snd_csa
device	snd_emu10kx
device	snd_es137x
device	snd_hda
device	snd_ich
device	snd_via8233
device	mmc
device	mmcsd
device	sdhci
device	virtio
device	virtio_pci
device	vtnet
device	virtio_blk
device	virtio_scsi
device	virtio_balloon
device	hyperv
device	xenpci
device	vmx
device	netmap
device	crypto

------------------------------------------------------------------------
ddb capture buffer

ddb: ddb_capture: kvm_nlist

--k1lZvvs/B4yU6o8G
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="zfs-stats.out"


------------------------------------------------------------------------
ZFS Subsystem Report				Thu Jul 12 11:31:07 2018
------------------------------------------------------------------------

System Information:

	Kernel Version:				1102501 (osreldate)
	Hardware Platform:			amd64
	Processor Architecture:			amd64

	ZFS Storage pool Version:		5000
	ZFS Filesystem Version:			5

FreeBSD 11.2-STABLE #0 r335674: Tue Jun 26 13:20:24 PDT 2018 root
11:31AM  up 19 mins, 1 user, load averages: 0.00, 0.00, 0.01

------------------------------------------------------------------------

System Memory:

	0.02%	20.51	MiB Active,	0.01%	13.54	MiB Inact
	9.05%	8.34	GiB Wired,	0.00%	0 Cache
	90.92%	83.84	GiB Free,	-0.00%	-204800	Bytes Gap

	Real Installed:				96.00	GiB
	Real Available:			98.57%	94.63	GiB
	Real Managed:			97.45%	92.22	GiB

	Logical Total:				96.00	GiB
	Logical Used:			12.65%	12.14	GiB
	Logical Free:			87.35%	83.86	GiB

Kernel Memory:					611.25	MiB
	Data:				94.07%	575.01	MiB
	Text:				5.93%	36.24	MiB

Kernel Memory Map:				92.22	GiB
	Size:				0.84%	795.25	MiB
	Free:				99.16%	91.44	GiB

------------------------------------------------------------------------

ARC Summary: (HEALTHY)
	Memory Throttle Count:			0

ARC Misc:
	Deleted:				0
	Recycle Misses:				0
	Mutex Misses:				0
	Evict Skips:				0

ARC Size:				7.09%	6.47	GiB
	Target Size: (Adaptive)		100.00%	91.22	GiB
	Min Size (Hard Limit):		12.50%	11.40	GiB
	Max Size (High Water):		8:1	91.22	GiB

ARC Size Breakdown:
	Recently Used Cache Size:	50.00%	45.61	GiB
	Frequently Used Cache Size:	50.00%	45.61	GiB

ARC Hash Breakdown:
	Elements Max:				486.60k
	Elements Current:		100.00%	486.60k
	Collisions:				6.94k
	Chain Max:				2
	Chains:					6.84k

------------------------------------------------------------------------

ARC Efficiency:					1.44m
	Cache Hit Ratio:		61.85%	891.12k
	Cache Miss Ratio:		38.15%	549.71k
	Actual Hit Ratio:		59.43%	856.26k

	Data Demand Efficiency:		96.28%	475.56k
	Data Prefetch Efficiency:	0.00%	449.51k

	CACHE HITS BY CACHE LIST:
	  Anonymously Used:		3.91%	34.85k
	  Most Recently Used:		59.27%	528.21k
	  Most Frequently Used:		36.81%	328.06k
	  Most Recently Used Ghost:	0.00%	0
	  Most Frequently Used Ghost:	0.00%	0

	CACHE HITS BY DATA TYPE:
	  Demand Data:			51.38%	457.85k
	  Prefetch Data:		0.00%	2
	  Demand Metadata:		44.71%	398.41k
	  Prefetch Metadata:		3.91%	34.86k

	CACHE MISSES BY DATA TYPE:
	  Demand Data:			3.22%	17.71k
	  Prefetch Data:		81.77%	449.51k
	  Demand Metadata:		8.57%	47.14k
	  Prefetch Metadata:		6.43%	35.35k

------------------------------------------------------------------------

L2ARC is disabled

------------------------------------------------------------------------

File-Level Prefetch: (HEALTHY)

DMU Efficiency:					115.87k
	Hit Ratio:			0.41%	476
	Miss Ratio:			99.59%	115.40k

	Colinear:				0
	  Hit Ratio:			100.00%	0
	  Miss Ratio:			100.00%	0

	Stride:					0
	  Hit Ratio:			100.00%	0
	  Miss Ratio:			100.00%	0

DMU Misc:
	Reclaim:				0
	  Successes:			100.00%	0
	  Failures:			100.00%	0

	Streams:				0
	  +Resets:			100.00%	0
	  -Resets:			100.00%	0
	  Bogus:				0

------------------------------------------------------------------------

VDEV cache is disabled

------------------------------------------------------------------------

ZFS Tunables (sysctl):
	kern.maxusers                           6392
	vm.kmem_size                            99019939840
	vm.kmem_size_scale                      1
	vm.kmem_size_min                        0
	vm.kmem_size_max                        1319413950874
	vfs.zfs.trim.max_interval               1
	vfs.zfs.trim.timeout                    30
	vfs.zfs.trim.txg_delay                  32
	vfs.zfs.trim.enabled                    1
	vfs.zfs.vol.immediate_write_sz          32768
	vfs.zfs.vol.unmap_sync_enabled          0
	vfs.zfs.vol.unmap_enabled               1
	vfs.zfs.vol.recursive                   0
	vfs.zfs.vol.mode                        1
	vfs.zfs.version.zpl                     5
	vfs.zfs.version.spa                     5000
	vfs.zfs.version.acl                     1
	vfs.zfs.version.ioctl                   7
	vfs.zfs.debug                           0
	vfs.zfs.super_owner                     0
	vfs.zfs.immediate_write_sz              32768
	vfs.zfs.sync_pass_rewrite               2
	vfs.zfs.sync_pass_dont_compress         5
	vfs.zfs.sync_pass_deferred_free         2
	vfs.zfs.zio.dva_throttle_enabled        1
	vfs.zfs.zio.exclude_metadata            0
	vfs.zfs.zio.use_uma                     1
	vfs.zfs.zil_slog_bulk                   786432
	vfs.zfs.cache_flush_disable             0
	vfs.zfs.zil_replay_disable              0
	vfs.zfs.standard_sm_blksz               131072
	vfs.zfs.dtl_sm_blksz                    4096
	vfs.zfs.min_auto_ashift                 12
	vfs.zfs.max_auto_ashift                 13
	vfs.zfs.vdev.trim_max_pending           10000
	vfs.zfs.vdev.bio_delete_disable         0
	vfs.zfs.vdev.bio_flush_disable          0
	vfs.zfs.vdev.queue_depth_pct            1000
	vfs.zfs.vdev.write_gap_limit            4096
	vfs.zfs.vdev.read_gap_limit             32768
	vfs.zfs.vdev.aggregation_limit          131072
	vfs.zfs.vdev.trim_max_active            64
	vfs.zfs.vdev.trim_min_active            1
	vfs.zfs.vdev.scrub_max_active           2
	vfs.zfs.vdev.scrub_min_active           1
	vfs.zfs.vdev.async_write_max_active     10
	vfs.zfs.vdev.async_write_min_active     1
	vfs.zfs.vdev.async_read_max_active      3
	vfs.zfs.vdev.async_read_min_active      1
	vfs.zfs.vdev.sync_write_max_active      10
	vfs.zfs.vdev.sync_write_min_active      10
	vfs.zfs.vdev.sync_read_max_active       10
	vfs.zfs.vdev.sync_read_min_active       10
	vfs.zfs.vdev.max_active                 1000
	vfs.zfs.vdev.async_write_active_max_dirty_percent60
	vfs.zfs.vdev.async_write_active_min_dirty_percent30
	vfs.zfs.vdev.mirror.non_rotating_seek_inc1
	vfs.zfs.vdev.mirror.non_rotating_inc    0
	vfs.zfs.vdev.mirror.rotating_seek_offset1048576
	vfs.zfs.vdev.mirror.rotating_seek_inc   5
	vfs.zfs.vdev.mirror.rotating_inc        0
	vfs.zfs.vdev.trim_on_init               1
	vfs.zfs.vdev.cache.bshift               16
	vfs.zfs.vdev.cache.size                 0
	vfs.zfs.vdev.cache.max                  16384
	vfs.zfs.vdev.default_ms_shift           29
	vfs.zfs.vdev.min_ms_count               16
	vfs.zfs.vdev.max_ms_count               200
	vfs.zfs.txg.timeout                     5
	vfs.zfs.spa_min_slop                    134217728
	vfs.zfs.spa_slop_shift                  5
	vfs.zfs.spa_asize_inflation             24
	vfs.zfs.deadman_enabled                 0
	vfs.zfs.deadman_checktime_ms            5000
	vfs.zfs.deadman_synctime_ms             1000000
	vfs.zfs.debug_flags                     0
	vfs.zfs.debugflags                      0
	vfs.zfs.recover                         0
	vfs.zfs.spa_load_verify_data            1
	vfs.zfs.spa_load_verify_metadata        1
	vfs.zfs.spa_load_verify_maxinflight     10000
	vfs.zfs.max_missing_tvds_scan           0
	vfs.zfs.max_missing_tvds_cachefile      2
	vfs.zfs.max_missing_tvds                0
	vfs.zfs.spa_load_print_vdev_tree        0
	vfs.zfs.ccw_retry_interval              300
	vfs.zfs.check_hostid                    1
	vfs.zfs.mg_fragmentation_threshold      85
	vfs.zfs.mg_noalloc_threshold            0
	vfs.zfs.condense_pct                    200
	vfs.zfs.metaslab_sm_blksz               4096
	vfs.zfs.metaslab.bias_enabled           1
	vfs.zfs.metaslab.lba_weighting_enabled  1
	vfs.zfs.metaslab.fragmentation_factor_enabled1
	vfs.zfs.metaslab.preload_enabled        1
	vfs.zfs.metaslab.preload_limit          3
	vfs.zfs.metaslab.unload_delay           8
	vfs.zfs.metaslab.load_pct               50
	vfs.zfs.metaslab.min_alloc_size         33554432
	vfs.zfs.metaslab.df_free_pct            4
	vfs.zfs.metaslab.df_alloc_threshold     131072
	vfs.zfs.metaslab.debug_unload           0
	vfs.zfs.metaslab.debug_load             0
	vfs.zfs.metaslab.fragmentation_threshold70
	vfs.zfs.metaslab.force_ganging          16777217
	vfs.zfs.free_bpobj_enabled              1
	vfs.zfs.free_max_blocks                 -1
	vfs.zfs.no_scrub_prefetch               0
	vfs.zfs.no_scrub_io                     0
	vfs.zfs.resilver_min_time_ms            3000
	vfs.zfs.free_min_time_ms                1000
	vfs.zfs.scan_min_time_ms                1000
	vfs.zfs.scan_idle                       50
	vfs.zfs.scrub_delay                     4
	vfs.zfs.resilver_delay                  2
	vfs.zfs.top_maxinflight                 32
	vfs.zfs.zfetch.array_rd_sz              1048576
	vfs.zfs.zfetch.max_idistance            67108864
	vfs.zfs.zfetch.max_distance             8388608
	vfs.zfs.zfetch.min_sec_reap             2
	vfs.zfs.zfetch.max_streams              8
	vfs.zfs.prefetch_disable                0
	vfs.zfs.delay_scale                     500000
	vfs.zfs.delay_min_dirty_percent         60
	vfs.zfs.dirty_data_sync                 67108864
	vfs.zfs.dirty_data_max_percent          10
	vfs.zfs.dirty_data_max_max              4294967296
	vfs.zfs.dirty_data_max                  4294967296
	vfs.zfs.max_recordsize                  1048576
	vfs.zfs.default_ibs                     17
	vfs.zfs.default_bs                      9
	vfs.zfs.send_holes_without_birth_time   1
	vfs.zfs.mdcomp_disable                  0
	vfs.zfs.per_txg_dirty_frees_percent     30
	vfs.zfs.nopwrite_enabled                1
	vfs.zfs.dedup.prefetch                  1
	vfs.zfs.dbuf_cache_lowater_pct          10
	vfs.zfs.dbuf_cache_hiwater_pct          10
	vfs.zfs.dbuf_cache_shift                5
	vfs.zfs.dbuf_cache_max_bytes            3060818688
	vfs.zfs.l2c_only_size                   0
	vfs.zfs.mfu_ghost_data_esize            0
	vfs.zfs.mfu_ghost_metadata_esize        0
	vfs.zfs.mfu_ghost_size                  0
	vfs.zfs.mfu_data_esize                  1380833792
	vfs.zfs.mfu_metadata_esize              24497152
	vfs.zfs.mfu_size                        1440531456
	vfs.zfs.mru_ghost_data_esize            0
	vfs.zfs.mru_ghost_metadata_esize        0
	vfs.zfs.mru_ghost_size                  0
	vfs.zfs.mru_data_esize                  5244487680
	vfs.zfs.mru_metadata_esize              109481472
	vfs.zfs.mru_size                        5392520192
	vfs.zfs.anon_data_esize                 0
	vfs.zfs.anon_metadata_esize             0
	vfs.zfs.anon_size                       1081344
	vfs.zfs.l2arc_norw                      1
	vfs.zfs.l2arc_feed_again                1
	vfs.zfs.l2arc_noprefetch                1
	vfs.zfs.l2arc_feed_min_ms               200
	vfs.zfs.l2arc_feed_secs                 1
	vfs.zfs.l2arc_headroom                  2
	vfs.zfs.l2arc_write_boost               8388608
	vfs.zfs.l2arc_write_max                 8388608
	vfs.zfs.arc_meta_limit                  24486549504
	vfs.zfs.arc_free_target                 167651
	vfs.zfs.compressed_arc_enabled          1
	vfs.zfs.arc_grow_retry                  60
	vfs.zfs.arc_shrink_shift                7
	vfs.zfs.arc_average_blocksize           8192
	vfs.zfs.arc_no_grow_shift               5
	vfs.zfs.arc_min                         12243274752
	vfs.zfs.arc_max                         97946198016
	vfs.zfs.abd_chunk_size                  4096

------------------------------------------------------------------------


--k1lZvvs/B4yU6o8G--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20180712183512.GA75020>