From owner-freebsd-current@freebsd.org Fri Mar 19 01:09:17 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A87135B2CD8 for ; Fri, 19 Mar 2021 01:09:17 +0000 (UTC) (envelope-from david.r.bergstein@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4F1m4s31qnz3HD9 for ; Fri, 19 Mar 2021 01:09:17 +0000 (UTC) (envelope-from david.r.bergstein@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 67AE75B2CD7; Fri, 19 Mar 2021 01:09:17 +0000 (UTC) Delivered-To: current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 675455B2CD6 for ; Fri, 19 Mar 2021 01:09:17 +0000 (UTC) (envelope-from david.r.bergstein@gmail.com) Received: from mail-qk1-x72e.google.com (mail-qk1-x72e.google.com [IPv6:2607:f8b0:4864:20::72e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F1m4r2Zxjz3H2G for ; Fri, 19 Mar 2021 01:09:16 +0000 (UTC) (envelope-from david.r.bergstein@gmail.com) Received: by mail-qk1-x72e.google.com with SMTP id q3so1321113qkq.12 for ; Thu, 18 Mar 2021 18:09:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=to:from:subject:message-id:date:user-agent:mime-version :content-transfer-encoding:content-language; bh=39T9qv03xbFU0CU+KF1rbaYiWmmZ85fHWHifTbezIcY=; b=kyIZaeX2z+A7mi9WhRtyK5Xt2qQApVZNdjfCbqyZZ3PCBJECprT5D3qmfxsjCloP/V nLkud2L14bi5GU1/Cz2PlmweS9EBSESqU9saTnXzju+jVXa3z4JHWF1W/Al+cvp3JGLj dvTecBs8+A3jekD60V4AaMH5ry0T51PZDthVCCsQj3rNTviUYpEWG26MgTkB4noEAPAe gOb6TRiGZL2acXYdWdFSew5lqEjCb//opCGRlRaCWqrISmga09e1pngUMdsbWDqpIUkB aJkFhZRhcEcYH1miuhq7ihLXDbQT2DasilDFmtrzxfD/beTYwzetogKrATEH2QT+NK/8 jtnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-transfer-encoding:content-language; bh=39T9qv03xbFU0CU+KF1rbaYiWmmZ85fHWHifTbezIcY=; b=A0IRwdNe6kOIg5jA0zlTY+A7ZSMqLx4vA9EQgHwcMCvufZJ3HCOTztHB9p8Qe5q1iS CtiwCNAORUpHGb343ojKTDqYufNjLL/4V9Dr5d8RHbJiC3pn/yo5J2gTMpOsAzWCMKy/ AwbLBc5Hw+0tieztsKC8Z33bFd6dOuHFJW3dcFVJBkPxwJUVp4DHf4f+Quw9VgyG4dSV IKDdJUGLLIAjJl33tPpg4lEIAC9WPJ3bDqS4dWYuJW8ncf8l+WugSdAgI2g56cbsC2oF K5MfI77zQVxWskPLSR5fhJ3edrTSfmZnJsJq6vQw5U8GGkWzOXEQOFsKmHgULurMGpGR MJqQ== X-Gm-Message-State: AOAM532BjR9SZUBjSK/iEG/7nVeYGRhLxM6Lzw03/0ZYom7ntKiHLkWi i218schrCDNLfM4Xv+h3PfU5kwBop5U= X-Google-Smtp-Source: ABdhPJzpidSmBEKwHux/k0DoqDiYMkRMP/iT1ODko59EjXFMnJiwmJHLNr1jGZcFi0KBxoJLRyILBw== X-Received: by 2002:a37:6888:: with SMTP id d130mr7108654qkc.368.1616116154104; Thu, 18 Mar 2021 18:09:14 -0700 (PDT) Received: from [192.168.1.6] (pool-96-241-46-90.washdc.fios.verizon.net. [96.241.46.90]) by smtp.googlemail.com with ESMTPSA id p8sm2747344qtu.8.2021.03.18.18.09.13 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 18 Mar 2021 18:09:13 -0700 (PDT) To: current@freebsd.org From: "David R. Bergstein" Subject: Boot time crash - Sleeping thread owns a non-sleepable lock Message-ID: <4e9908f5-3bbc-f7b2-e2bc-45376d524215@gmail.com> Date: Thu, 18 Mar 2021 21:09:12 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.8.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4F1m4r2Zxjz3H2G X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=kyIZaeX2; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of davidrbergstein@gmail.com designates 2607:f8b0:4864:20::72e as permitted sender) smtp.mailfrom=davidrbergstein@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RECEIVED_SPAMHAUS_PBL(0.00)[96.241.46.90:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::72e:from]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[current@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::72e:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::72e:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[current] X-Mailman-Approved-At: Fri, 19 Mar 2021 06:21:06 +0000 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Mar 2021 01:09:17 -0000 I have been seeing this boot time crash on an intermittent basis on a virtual machine; FreeBSD-13.0-RC2.   Below is a copy of the kernel crash dump file in case it is helpful to anyone: -------------------------------------------------- drb-freebsd-13.localdomain dumped core - see /var/crash/vmcore.4 Thu Mar 18 20:58:47 EDT 2021 FreeBSD drb-freebsd-13.localdomain 13.0-RC2 FreeBSD 13.0-RC2 #0 releng/13.0-n244684-13c22f74953: Fri Mar 12 04:05:19 UTC 2021 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 panic: sleeping thread GNU gdb (GDB) 10.1 [GDB v10.1 for FreeBSD] Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd13.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at:     . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /boot/kernel/kernel... Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug... Unread portion of the kernel message buffer: ---<>--- Copyright (c) 1992-2021 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994     The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-RC2 #0 releng/13.0-n244684-13c22f74953: Fri Mar 12 04:05:19 UTC 2021 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe) VT(vga): text 80x25 CPU: Intel(R) Core(TM) i7 CPU         950  @ 3.07GHz (3103.68-MHz K8-class CPU)   Origin="GenuineIntel"  Id=0x106a5  Family=0x6  Model=0x1a Stepping=5 Features=0x1783fbff   Features2=0x180201   AMD Features=0x28100800   AMD Features2=0x1   Structured Extended Features3=0x10000000   TSC: P-state invariant real memory  = 9126805504 (8704 MB) avail memory = 8285257728 (7901 MB) Event timer "LAPIC" quality 100 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 cache groups x 1 core(s) random: unblocking device. ioapic0: MADT APIC ID 4 != hw id 0 ioapic0 irqs 0-23 Launching APs: 2 1 3 Timecounter "TSC-low" frequency 1551839460 Hz quality 1000 KTLS: Initialized 4 threads random: entropy device external interface [ath_hal] loaded WARNING: Device "kbd" is Giant locked and may be deleted before FreeBSD 14.0. kbd1 at kbdmux0 000.000058 [4350] netmap_init               netmap: loaded module mlx5en: Mellanox Ethernet driver 3.6.0 (December 2020) nexus0 vtvga0: cryptosoft0: aesni0: No AES or SHA support. acpi0: acpi0: Power Button (fixed) acpi0: Sleep Button (fixed) cpu0: on acpi0 attimer0: port 0x40-0x43,0x50-0x53 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 isab0: at device 1.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xd000-0xd00f at device 1.1 on pci0 ata0: at channel 0 on atapci0 ata1: at channel 1 on atapci0 vgapci0: mem 0xe0000000-0xefffffff irq 18 at device 2.0 on pci0 vgapci0: Boot video device em0: port 0xd010-0xd017 mem 0xf0000000-0xf001ffff irq 19 at device 3.0 on pci0 em0: Using 1024 TX descriptors and 1024 RX descriptors <6>em0: Ethernet address: 08:00:27:dd:d4:fc <6>em0: netmap queues/slots: TX 1/1024, RX 1/1024 pcm0: port 0xd100-0xd1ff,0xd200-0xd23f irq 21 at device 5.0 on pci0 pcm0: ohci0: mem 0xf0804000-0xf0804fff irq 22 at device 6.0 on pci0 usbus0 on ohci0 usbus0: 12Mbps Full Speed USB v1.0 pci0: at device 7.0 (no driver attached) ehci0: mem 0xf0805000-0xf0805fff irq 19 at device 11.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci0 usbus1: 480Mbps High Speed USB v2.0 acpi_acad0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 14.0. psm0: model IntelliMouse Explorer, device ID 4 orm0: at iomem 0xc0000-0xc7fff,0xe2000-0xe2fff pnpid ORM0000 on isa0 vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff pnpid PNP0900 on isa0 atrtc0: at port 0x70 irq 8 on isa0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 atrtc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 14. Timecounters tick every 10.000 msec ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Trying to mount root from zfs:zroot/ROOT/default []... Root mount waiting for: CAM usbus0 usbus1 ugen0.1: at usbus0 uhub0 on usbus0 ugen1.1: at usbus1 uhub0: on usbus0 uhub1 on usbus1 uhub1: on usbus1 ada0 at ata0 bus 0 scbus0 target 0 lun 0 ada0: ATA8-ACS device ada0: Serial Number VBb8450cb8-277eed0b ada0: 33.300MB/s transfers (UDMA2, PIO 65536bytes) ada0: 61440MB (125829120 512 byte sectors) cd0 at ata1 bus 0 scbus1 target 0 lun 0 cd0: Removable CD-ROM SCSI device cd0: Serial Number VB2-01700376 cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present uhub0: 12 ports with 12 removable, self powered Root mount waiting for: usbus1 Root mount waiting for: usbus1 Root mount waiting for: usbus1 Root mount waiting for: usbus1 uhub1: 12 ports with 12 removable, self powered <118>Setting hostuuid: 56767ad9-d311-7b42-9f7f-db26ca8d9b82. <118>Setting hostid: 0x10458f66. <118>Starting file system checks: <118>no pools available to import <118>Mounting local filesystems:. <118>Setting hostname: drb-freebsd-13.localdomain. <118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/gcc10 /usr/local/lib/gcc9 /usr/local/lib/mysql /usr/local/lib/mysql/plugin /usr/local/lib/perl5/5.32/mach/CORE /usr/local/lib/qt5 /usr/local/lib/samba4 /usr/local/lib/signon /usr/local/lib/signon/extensions /usr/local/llvm10/lib <118>32-bit compatibility ldconfig path: /usr/lib32 <118>Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED <118>Feeding entropy: . vboxguest0 port 0xd020-0xd03f mem 0xf0400000-0xf07fffff,0xf0800000-0xf0803fff irq 20 at device 4.0 on pci0 Sleeping thread (tid 100521, pid 174) owns a non-sleepable lock KDB: stack backtrace of thread 100521: #0 0xffffffff80c16291 at mi_switch+0xc1 #1 0xffffffff8293cf37 at rtR0SemEventMultiBsdWait+0x297 #2 0xffffffff8291d36a at vgdrvHgcmAsyncWaitCallbackWorker+0x14a #3 0xffffffff8291e49b at VbglR0HGCMInternalConnect+0x11b #4 0xffffffff8291ad33 at VGDrvCommonIoCtl+0xb53 #5 0xffffffff82919af6 at VGDrvCommonProcessOptionsFromHost+0x146 #6 0xffffffff8291d9f8 at vgdrvFreeBSDAttach+0x1d8 #7 0xffffffff80c462ad at device_attach+0x3dd #8 0xffffffff80c45eb1 at device_probe_and_attach+0x41 #9 0xffffffff8085b966 at pci_driver_added+0xf6 #10 0xffffffff80c43d29 at devclass_driver_added+0x39 #11 0xffffffff80c43cad at devclass_add_driver+0x13d #12 0xffffffff80be6964 at module_register_init+0xa4 #13 0xffffffff80bd8667 at linker_load_module+0xbc7 #14 0xffffffff80bd9e4b at kern_kldload+0xbb #15 0xffffffff80bd9f7b at sys_kldload+0x5b #16 0xffffffff8108aa8c at amd64_syscall+0x10c #17 0xffffffff810618be at fast_syscall_common+0xf8 panic: sleeping thread cpuid = 0 time = 1616100801 KDB: stack backtrace: #0 0xffffffff80c57105 at kdb_backtrace+0x65 #1 0xffffffff80c09cd1 at vpanic+0x181 #2 0xffffffff80c09b43 at panic+0x43 #3 0xffffffff80c6e526 at propagate_priority+0x296 #4 0xffffffff80c6f0c3 at turnstile_wait+0x313 #5 0xffffffff80be7b9c at __mtx_lock_sleep+0x17c #6 0xffffffff80c256bf at softclock_call_cc+0xff #7 0xffffffff80c25b89 at softclock+0x79 #8 0xffffffff80bcae0d at ithread_loop+0x24d #9 0xffffffff80bc7c0e at fork_exit+0x7e #10 0xffffffff8106201e at fork_trampoline+0xe Uptime: 8s Dumping 364 out of 8152 MB:..5%..14%..22%..31%..44%..53%..62%..71%..84%..93% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55        __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu, (kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1  doadump (textdump=) at /usr/src/sys/kern/kern_shutdown.c:399 #2  0xffffffff80c098c6 in kern_reboot (howto=260)     at /usr/src/sys/kern/kern_shutdown.c:486 #3  0xffffffff80c09d40 in vpanic (fmt=, ap=)     at /usr/src/sys/kern/kern_shutdown.c:919 #4  0xffffffff80c09b43 in panic (fmt=)     at /usr/src/sys/kern/kern_shutdown.c:843 #5  0xffffffff80c6e526 in propagate_priority (td=0xfffffe00c2c0be00)     at /usr/src/sys/kern/subr_turnstile.c:251 #6  0xffffffff80c6f0c3 in turnstile_wait (ts=ts@entry=0xfffff800035b9540,     owner=, owner@entry=0xfffffe00c2c0be00, queue=queue@entry=0)     at /usr/src/sys/kern/subr_turnstile.c:808 #7  0xffffffff80be7b9c in __mtx_lock_sleep (c=0xffffffff81802718 ,     v=) at /usr/src/sys/kern/kern_mutex.c:664 #8  0xffffffff80c256bf in softclock_call_cc (     c=0xffffffff81d1b700 ,     cc=cc@entry=0xffffffff81ca8200 , direct=direct@entry=0)     at /usr/src/sys/kern/kern_timeout.c:658 #9  0xffffffff80c25b89 in softclock (arg=0xffffffff81ca8200 )     at /usr/src/sys/kern/kern_timeout.c:816 #10 0xffffffff80bcae0d in intr_event_execute_handlers (p=,     ie=0xfffff8000364a700) at /usr/src/sys/kern/kern_intr.c:1168 #11 ithread_execute_handlers (p=, ie=0xfffff8000364a700)     at /usr/src/sys/kern/kern_intr.c:1181 #12 ithread_loop (arg=arg@entry=0xfffff80003657d20)     at /usr/src/sys/kern/kern_intr.c:1269 #13 0xffffffff80bc7c0e in fork_exit (     callout=0xffffffff80bcabc0 , arg=0xfffff80003657d20,     frame=0xfffffe00835e8d40) at /usr/src/sys/kern/kern_fork.c:1069 #14 (kgdb) ------------------------------------------------------------------------ ps -axlww UID PID PPID  C PRI NI   VSZ  RSS MWCHAN   STAT TT     TIME COMMAND   0   0    0  3 -16  0     0    0 swapin   DLs   -  0:00.00 [kernel]   0   1    0  1  21  0 11824  968 wait     DLs   -  0:00.00 [init]   0   2    0  3 -16  0     0    0 -        DL    -  0:00.00 [KTLS]   0   3    0  3 -16  0     0    0 crypto_w DL    -  0:00.00 [crypto]   0   4    0  3 -16  0     0    0 crypto_r DL    -  0:00.00 [crypto returns 0]   0   5    0  0 -16  0     0    0 crypto_r DL    -  0:00.00 [crypto returns 1]   0   6    0  3 -16  0     0    0 crypto_r DL    -  0:00.00 [crypto returns 2]   0   7    0  0 -16  0     0    0 crypto_r DL    -  0:00.00 [crypto returns 3]   0   8    0  3 -16  0     0    0 -        RL    -  0:00.00 [cam]   0   9    0  2 -16  0     0    0 -        DL    -  0:00.00 [soaiod1]   0  10    0  3 -16  0     0    0 audit_wo DL    -  0:00.00 [audit]   0  11    0  0 155  0     0    0 -        RL    -  0:00.00 [idle]   0  12    0 -1 -52  0     0    0 -        WL    -  0:00.00 [intr]   0  13    0  0  -8  0     0    0 -        DL    -  0:00.00 [geom]   0  14    0  0 -16  0     0    0 seqstate DL    -  0:00.00 [sequencer 00]   0  15    0  0 -68  0     0    0 -        DL    -  0:00.00 [usb]   0  16    0  2 -16  0     0    0 -        DL    -  0:00.00 [soaiod2]   0  17    0  0 -16  0     0    0 -        DL    -  0:00.00 [soaiod3]   0  18    0  2 -16  0     0    0 -        DL    -  0:00.00 [soaiod4]   0  19    0  0  -8  0     0    0 t->zthr_ DL    -  0:00.03 [zfskern]   0  20    0  0 -16  0     0    0 -        DL    -  0:00.00 [rand_harvestq]   0  21    0  1 -16  0     0    0 psleep   DL    -  0:00.00 [pagedaemon]   0  22    0  3 -16  0     0    0 psleep   DL    -  0:00.00 [vmdaemon]   0  23    0  2 -16  0     0    0 qsleep   DL    -  0:00.00 [bufdaemon]   0  24    0  2 -16  0     0    0 vlruwt   DL    -  0:00.00 [vnlru]   0  25    0  0  16  0     0    0 syncer   DL    -  0:00.00 [syncer]   0  26    1  1  52  0 13624 2820 wait     Ds+   -  0:00.00 [sh]   0 172   26  2  52  0 13624 2820 wait     D+    -  0:00.00 [sh]   0 174  172  3  52  0 12768 2100 VBoxIS   D+    -  0:00.00 [kldload]   0 175    0  2 -20  0     0    0 VBoxIS   DL    -  0:00.00 [Timer] ------------------------------------------------------------------------ vmstat -s     19751 cpu context switches      1534 device interrupts       112 software interrupts     16622 traps     11362 system calls        26 kernel threads created       103  fork() calls        46 vfork() calls         0 rfork() calls         0 swap pager pageins         0 swap pager pages paged in         0 swap pager pageouts         0 swap pager pages paged out       222 vnode pager pageins      1540 vnode pager pages paged in         0 vnode pager pageouts         0 vnode pager pages paged out         0 page daemon wakeups         0 pages examined by the page daemon         0 clean page reclamation shortfalls         0 pages reactivated by the page daemon      5335 copy-on-write faults        26 copy-on-write optimized faults      8321 zero fill pages zeroed         0 zero fill pages prezeroed         0 intransit blocking page faults     15666 total VM faults taken       163 page faults requiring I/O         0 pages affected by kernel thread creation      4173 pages affected by  fork()      1610 pages affected by vfork()         0 pages affected by rfork()     52437 pages freed         0 pages freed by daemon         0 pages freed by exiting processes         0 pages active         0 pages inactive         0 pages in the laundry queue         0 pages wired down         0 virtual user pages wired down         0 pages free         0 bytes per page         0 total name lookups           cache hits (0% pos + 0% neg) system 0% per-directory           deletions 0%, falsehits 0%, toolong 0% ------------------------------------------------------------------------ vmstat -m vmstat: memstat_kvm_malloc:          Type InUse MemUse Requests  Size(s) ------------------------------------------------------------------------ vmstat -z ITEM                   SIZE  LIMIT     USED     FREE      REQ FAILSLEEP XDOMAIN UMA Kegs:               384,      0,     169,       0,     169, 0,   0,   0 UMA Zones:             1024,      0,     180,       0,     180, 0,   0,   0 UMA Slabs 0:             80,      0,    4234,       0,    4234, 0,   0,   0 UMA Slabs 1:            112,      0,       6,       0,       6, 0,   0,   0 UMA Hash:               256,      0,       0,       0,       0, 0,   0,   0 2 Bucket:                32,      0,     171,     333,    4435, 0,   0,   0 4 Bucket:                48,      0,      20,     316,      39, 0,   0,   0 8 Bucket:                80,      0,      71,     229,     855, 0,   0,   0 16 Bucket:              144,      0,      31,     137,    1716, 2,   0,   0 32 Bucket:              256,      0,      75,      30,     532, 0,   0,   0 64 Bucket:              512,      0,      44,      28,     403, 15,   0,   0 128 Bucket:            1024,      0,      64,      11, 637,1902,   0,   0 256 Bucket:            2048,      0,      75,       3,     493, 122,   0,   0 SMR SHARED:              24,      0,       2,      61,       2, 0,   0,   0 SMR CPU:                 32,      0,       2,      61,       2, 0,   0,   0 vmem:                  1856,      0,       1,       1,       1, 0,   0,   0 vmem btag:               56,      0,    3281,     230,    3281, 91,   0,   0 VM OBJECT:              264,      0,     359,      91,    2350, 0,   0,   0 RADIX NODE:             144,      0,    1848,     306,    8632, 0,   0,   0 KMAP ENTRY:              96,      0,       6,       0,       8, 0,   0,   0 MAP ENTRY:               96,      0,      85,     629,    6531, 0,   0,   0 VMSPACE:               2544,      0,       4,      23,     151, 0,   0,   0 fakepg:                 104,      0,       0,       0,       0, 0,   0,   0 pcpu-4:                   4,      0,       0,       0,       0, 0,   0,   0 pcpu-8:                   8,      0,    2985,     595,    2985, 0,   0,   0 pcpu-16:                 16,      0,      21,     745,      21, 0,   0,   0 pcpu-32:                 32,      0,       0,       0,       0, 0,   0,   0 pcpu-64:                 64,      0,     453,     123,     453, 0,   0,   0 malloc-16:               16,      0,   12459,     645,   40358, 0,   0,   0 malloc-32:               32,      0,    5906,     772,   16645, 0,   0,   0 malloc-64:               64,      0,   10648,     503,   50876, 0,   0,   0 malloc-128:             128,      0,   13450,     376,   73945, 0,   0,   0 malloc-256:             256,      0,    1155,     210,    3239, 0,   0,   0 malloc-384:             384,      0,     978,      22,   56179, 0,   0,   0 malloc-512:             512,      0,      26,     102,     586, 0,   0,   0 malloc-1024:           1024,      0,    1728,      52,    8627, 0,   0,   0 malloc-2048:           2048,      0,     168,      26,    2628, 0,   0,   0 malloc-4096:           4096,      0,     596,       1,    1553, 0,   0,   0 malloc-8192:           8192,      0,      23,       8,     301, 0,   0,   0 malloc-16384:         16384,      0,      14,       4,     432, 0,   0,   0 malloc-32768:         32768,      0,       7,       3,     172, 0,   0,   0 malloc-65536:         65536,      0,       8,       2,     172, 0,   0,   0 SLEEPQUEUE:              80,      0,     523,      85,     523, 0,   0,   0 kenv:                   258,      0,      17,      43,     781, 0,   0,   0 Files:                   80,      0,       4,     196,    1256, 0,   0,   0 filedesc0:             1072,      0,      30,      40,     176, 0,   0,   0 PWD:                     32,      0,       3,     501,      75, 0,   0,   0 rl_entry:                40,      0,      11,     292,      11, 0,   0,   0 TURNSTILE:              136,      0,     523,      44,     523, 0,   0,   0 rangeset pctrie nodes:    144,      0,       0,       0, 0,   0,   0,   0 umtx pi:                 96,      0,       0,       0,       0, 0,   0,   0 umtx_shm:                88,      0,       0,       0,       0, 0,   0,   0 swrbuf:                2368,    128,       0,       0,       0, 0,   0,   0 swwbuf:                2368,     64,       0,       0,       0, 0,   0,   0 nfspbuf:               2368,    128,       0,       0,       0, 0,   0,   0 mdpbuf:                2368,     25,       0,       0,       0, 0,   0,   0 clpbuf:                2368,    128,       0,       0,       0, 0,   0,   0 vnpbuf:                2368,    512,       0,       0,       0, 0,   0,   0 pbuf:                  2368,     16,       0,       0,       0, 0,   0,   0 MAC labels:              40,      0,       0,       0,       0, 0,   0,   0 PROC:                  1320,      0,      29,      10,     175, 0,   0,   0 PGRP:                    88,      0,       2,      90,       2, 0,   0,   0 THREAD:                1792,      0,     498,      23,     522, 0,   0,   0 cpuset:                 104,      0,      11,     113,      11, 0,   0,   0 domainset:               40,      0,       0,       0,       0, 0,   0,   0 audit_record:          1280,      0,       0,       0,       0, 0,   0,   0 mbuf_packet:            256,      0,       0,       0,       0, 0,   0,   0 mbuf:                   256, 3247531,       1,      14,       1, 0,   0,   0 mbuf_cluster:          2048, 507426,       0,       0,       0, 0,   0,   0 mbuf_jumbo_page:       4096, 253713,       0,       0,       0, 0,   0,   0 mbuf_jumbo_9k:         9216,  75174,       0,       0,       0, 0,   0,   0 mbuf_jumbo_16k:       16384,  42285,       0,       0,       0, 0,   0,   0 epoch_record pcpu:      256,      0,       4,      12,       4, 0,   0,   0 ktls_session:           192,      0,       0,       0,       0, 0,   0,   0 ttyinq:                 160,      0,      15,      10,      15, 0,   0,   0 ttyoutq:                256,      0,       8,       7,       8, 0,   0,   0 FPU_save_area:          512,      0,       1,       7,       1, 0,   0,   0 DEVCTL:                1024,   1000,     142,       0,     142, 0,   0,   0 IOMMU_MAP_ENTRY:        120,      0,       0,       0,       0, 0,   0,   0 g_bio:                  376,      0,       0,     180,    4596, 0,   0,   0 linux_dma_pctrie:       144,      0,       0,       0,       0, 0,   0,   0 linux_dma_object:        24,      0,       0,       0,       0, 0,   0,   0 cryptop:                280,      0,       0,       0,       0, 0,   0,   0 vtnet_tx_hdr:            24,      0,       0,       0,       0, 0,   0,   0 taskq_zone:             192,      0,       0,     189,     339, 0,   0,   0 VNODE:                  488,      0,     419,      61,     469, 0,   0,   0 BUF TRIE:               144,      0,       0,       0,       0, 0,   0,   0 NAMEI:                 1024,      0,       0,      16,    2162, 0,   0,   0 S VFS Cache:            104,      0,     475,     110,     496, 0,   0,   0 STS VFS Cache:          144,      0,       0,       0,       0, 0,   0,   0 L VFS Cache:            320,      0,       0,      12,       1, 0,   0,   0 LTS VFS Cache:          360,      0,       0,       0,       0, 0,   0,   0 TMPFS node:             224,      0,       0,       0,       0, 0,   0,   0 DIRHASH:               1024,      0,       0,       0,       0, 0,   0,   0 NCLNODE:                592,      0,       0,       0,       0, 0,   0,   0 pipe:                   744,      0,       0,      10,      37, 0,   0,   0 Mountpoints:           2752,      0,      12,       6,      12, 0,   0,   0 AIO:                    208,      0,       0,       0,       0, 0,   0,   0 AIOP:                    32,      0,       0,       0,       0, 0,   0,   0 AIOCB:                  552,      0,       0,       0,       0, 0,   0,   0 AIOLIO:                 272,      0,       0,       0,       0, 0,   0,   0 zfs_btree_leaf_cache:   4096,      0,     104,      21,     292, 0,   0,   0 metaslab_alloc_trace_cache:     72,      0,       0, 0,       0,   0,   0,   0 ddt_cache:            24840,      0,      13,       0,      13, 0,   0,   0 ddt_entry_cache:        392,      0,       0,       0,       0, 0,   0,   0 zio_cache:             1232,      0,       8,     178,   24000, 0,   0,   0 zio_link_cache:          48,      0,       0,    1008,   12945, 0,   0,   0 zio_buf_comb_512:       512,      0,      69,      59,   18526, 0,   0,   0 zio_buf_comb_1024:     1024,      0,      61,       7,      70, 0,   0,   0 zio_buf_comb_1536:     1536,      0,      40,      10,      53, 0,   0,   0 zio_buf_comb_2048:     2048,      0,      24,       4,      32, 0,   0,   0 zio_buf_comb_2560:     2560,      0,      21,       3,      29, 0,   0,   0 zio_buf_comb_3072:     3072,      0,      13,      11,      20, 0,   0,   0 zio_buf_comb_3584:     3584,      0,       4,       2,       4, 0,   0,   0 zio_buf_comb_4096:     4096,      0,     446,       7,    1248, 0,   0,   0 zio_buf_comb_5120:     5120,      0,       6,       0,       6, 0,   0,   0 zio_buf_comb_6144:     6144,      0,       3,       0,       3, 0,   0,   0 zio_buf_comb_7168:     7168,      0,       6,       0,       6, 0,   0,   0 zio_buf_comb_8192:     8192,      0,       9,       9,     150, 0,   0,   0 zio_buf_comb_10240:   10240,      0,       9,       0,       9, 0,   0,   0 zio_buf_comb_12288:   12288,      0,       3,       9,      64, 0,   0,   0 zio_buf_comb_14336:   14336,      0,       8,       0,       9, 0,   0,   0 zio_buf_comb_16384:   16384,      0,     202,      11,     422, 0,   0,   0 zio_buf_comb_20480:   20480,      0,       3,       7,      32, 0,   0,   0 zio_buf_comb_24576:   24576,      0,       7,       4,      27, 0,   0,   0 zio_buf_comb_28672:   28672,      0,       3,       5,      19, 0,   0,   0 zio_buf_comb_32768:   32768,      0,       2,       2,      19, 0,   0,   0 zio_buf_comb_40960:   40960,      0,       5,       3,      25, 0,   0,   0 zio_buf_comb_49152:   49152,      0,       2,       9,      47, 0,   0,   0 zio_buf_comb_57344:   57344,      0,       3,       6,      20, 0,   0,   0 zio_buf_comb_65536:   65536,      0,       1,       3,      11, 0,   0,   0 zio_buf_comb_81920:   81920,      0,       2,       8,      43, 0,   0,   0 zio_buf_comb_98304:   98304,      0,       1,      12,      58, 0,   0,   0 zio_buf_comb_114688: 114688,      0,       2,       6,      16, 0,   0,   0 zio_buf_comb_131072: 131072,      0,     119,      10,     535, 0,   0,   0 zio_buf_comb_163840: 163840,      0,       0,       0,       0, 0,   0,   0 zio_buf_comb_196608: 196608,      0,       0,       0,       0, 0,   0,   0 zio_buf_comb_229376: 229376,      0,       0,       0,       0, 0,   0,   0 zio_buf_comb_262144: 262144,      0,       0,       0,       0, 0,   0,   0 zio_buf_comb_327680: 327680,      0,       0,       0,       0, 0,   0,   0 zio_buf_comb_393216: 393216,      0,       0,       0,       0, 0,   0,   0 zio_buf_comb_458752: 458752,      0,       0,       0,       0, 0,   0,   0 zio_buf_comb_524288: 524288,      0,       0,       0,       0, 0,   0,   0 zio_buf_comb_655360: 655360,      0,       0,       0,       0, 0,   0,   0 zio_buf_comb_786432: 786432,      0,       0,       0,       0, 0,   0,   0 zio_buf_comb_917504: 917504,      0,       0,       0,       0, 0,   0,   0 zio_buf_comb_1048576: 1048576,      0,       0,       0, 0,   0,   0,   0 zio_buf_comb_1310720: 1310720,      0,       0,       0, 0,   0,   0,   0 zio_buf_comb_1572864: 1572864,      0,       0,       0, 0,   0,   0,   0 zio_buf_comb_1835008: 1835008,      0,       0,       0, 0,   0,   0,   0 zio_buf_comb_2097152: 2097152,      0,       0,       0, 0,   0,   0,   0 zio_buf_comb_2621440: 2621440,      0,       0,       0, 0,   0,   0,   0 zio_buf_comb_3145728: 3145728,      0,       0,       0, 0,   0,   0,   0 zio_buf_comb_3670016: 3670016,      0,       0,       0, 0,   0,   0,   0 zio_buf_comb_4194304: 4194304,      0,       0,       0, 0,   0,   0,   0 zio_buf_comb_5242880: 5242880,      0,       0,       0, 0,   0,   0,   0 zio_buf_comb_6291456: 6291456,      0,       0,       0, 0,   0,   0,   0 zio_buf_comb_7340032: 7340032,      0,       0,       0, 0,   0,   0,   0 zio_buf_comb_8388608: 8388608,      0,       0,       0, 0,   0,   0,   0 zio_buf_comb_10485760: 10485760,      0,       0,       0, 0,   0,   0,   0 zio_buf_comb_12582912: 12582912,      0,       0,       0, 0,   0,   0,   0 zio_buf_comb_14680064: 14680064,      0,       0,       0, 0,   0,   0,   0 zio_buf_comb_16777216: 16777216,      0,       0,       0, 0,   0,   0,   0 abd_chunk:             4096,      0,    1579,       1,    2466, 0,   0,   0 sa_cache:               288,      0,     394,      26,     404, 0,   0,   0 dnode_t:                816,      0,     910,      15,    1604, 0,   0,   0 arc_buf_hdr_t_full:     248,      0,     814,      34,   10055, 0,   0,   0 arc_buf_hdr_t_full_crypt:    312,      0,       0,       0, 0,   0,   0,   0 arc_buf_hdr_t_l2only:     96,      0,       0,       0,       0, 0,   0,   0 arc_buf_t:               64,      0,     634,     122,   10465, 0,   0,   0 dmu_buf_impl_t:         296,      0,    1333,      45,   10741, 0,   0,   0 zil_lwb_cache:          360,      0,       4,      18,       4, 0,   0,   0 zil_zcw_cache:           80,      0,       0,      50,       4, 0,   0,   0 sio_cache_0:            136,      0,       0,       0,       0, 0,   0,   0 sio_cache_1:            152,      0,       0,       0,       0, 0,   0,   0 sio_cache_2:            168,      0,       0,       0,       0, 0,   0,   0 zfs_znode_cache:        472,      0,     394,      30,     404, 0,   0,   0 ksiginfo:               112,      0,      17,     487,      17, 0,   0,   0 itimer:                 352,      0,       0,       0,       0, 0,   0,   0 KNOTE:                  160,      0,       0,       0,       0, 0,   0,   0 socket:                 944, 260881,       0,       8,       2, 0,   0,   0 ipq:                     56,  15857,       0,       0,       0, 0,   0,   0 udp_inpcb:              488, 260881,       0,      16,       2, 0,   0,   0 udpcb:                   32, 260881,       0,     252,       2, 0,   0,   0 tcp_inpcb:              488, 260881,       0,       0,       0, 0,   0,   0 tcpcb:                 1048, 260881,       0,       0,       0, 0,   0,   0 tcptw:                   88,  27767,       0,       0,       0, 0,   0,   0 syncache:               168,  15360,       0,       0,       0, 0,   0,   0 hostcache:               96,  15360,       0,       0,       0, 0,   0,   0 sackhole:                32,      0,       0,       0,       0, 0,   0,   0 tfo:                      4,      0,       0,       0,       0, 0,   0,   0 tfo_ccache_entries:      80,      0,       0,       0,       0, 0,   0,   0 tcpreass:                48,  31714,       0,       0,       0, 0,   0,   0 tcp_log:                408, 5000000,       0,       0,       0, 0,   0,   0 tcp_log_bucket:         176,      0,       0,       0,       0, 0,   0,   0 tcp_log_node:           120,      0,       0,       0,       0, 0,   0,   0 udplite_inpcb:          488, 260881,       0,       0,       0, 0,   0,   0 ripcb:                  488, 260881,       0,       0,       0, 0,   0,   0 IPsec SA lft_c:          16,      0,       0,       0,       0, 0,   0,   0 unpcb:                  256, 260881,       0,       0,       0, 0,   0,   0 routing nhops:          256,      0,       0,       0,       0, 0,   0,   0 rtentry:                176,      0,       0,       0,       0, 0,   0,   0 swpctrie:               144, 1014860,       0,       0,       0, 0,   0,   0 swblk:                  136, 1014855,       0,       0,       0, 0,   0,   0 ------------------------------------------------------------------------ vmstat -i interrupt                          total       rate irq1: atkbd0                           2          0 irq14: ata0                         1492          5 irq15: ata1                           39          0 irq20: vboxguest0                      1          0 cpu0:timer                           861          3 cpu1:timer                           722          2 cpu2:timer                           642          2 cpu3:timer                           544          2 Total                               4303         14 ------------------------------------------------------------------------ pstat -T   4/260881 files 0M/0M swap space ------------------------------------------------------------------------ pstat -s Device          512-blocks     Used    Avail Capacity ------------------------------------------------------------------------ iostat        tty            ada0              cd0 pass0             cpu  tin  tout KB/t  tps  MB/s  KB/t  tps  MB/s  KB/t  tps  MB/s  us ni sy in id    0    95 13.5  186   2.5   0.0    0   0.0   0.0    0   0.0   0 0 13  3 84 ------------------------------------------------------------------------ ipcs -a Message Queues: T           ID          KEY MODE        OWNER    GROUP    CREATOR CGROUP                 CBYTES                 QNUM QBYTES        LSPID        LRPID STIME    RTIME    CTIME Shared Memory: T           ID          KEY MODE        OWNER    GROUP    CREATOR CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME Semaphores: T           ID          KEY MODE        OWNER    GROUP    CREATOR CGROUP          NSEMS OTIME    CTIME ------------------------------------------------------------------------ ipcs -T msginfo:     msgmax:        16384    (max characters in a message)     msgmni:           40    (# of message queues)     msgmnb:         2048    (max characters in a message queue)     msgtql:           40    (max # of messages in system)     msgssz:            8    (size of a message segment)     msgseg:         2048    (# of message segments in system) shminfo:     shmmax:    536870912    (max shared memory segment size)     shmmin:            1    (min shared memory segment size)     shmmni:          192    (max number of shared memory identifiers)     shmseg:          128    (max shared memory segments per process)     shmall:       131072    (max amount of shared memory in pages) seminfo:     semmni:           50    (# of semaphore identifiers)     semmns:          340    (# of semaphores in system)     semmnu:          150    (# of undo structures in system)     semmsl:          340    (max # of semaphores per id)     semopm:          100    (max # of operations per semop call)     semume:           50    (max # of undo entries per process)     semusz:          632    (size in bytes of undo structure)     semvmx:        32767    (semaphore maximum value)     semaem:        16384    (adjust on exit max value) ------------------------------------------------------------------------ netstat -s tcp:     0 packets sent         0 data packets (0 bytes)         0 data packets (0 bytes) retransmitted         0 data packets unnecessarily retransmitted         0 resends initiated by MTU discovery         0 ack-only packets (0 delayed)         0 URG only packets         0 window probe packets         0 window update packets         0 control packets     0 packets received         0 acks (for 0 bytes)         0 duplicate acks         0 acks for unsent data         0 packets (0 bytes) received in-sequence         0 completely duplicate packets (0 bytes)         0 old duplicate packets         0 packets with some dup. data (0 bytes duped)         0 out-of-order packets (0 bytes)         0 packets (0 bytes) of data after window         0 window probes         0 window update packets         0 packets received after close         0 discarded for bad checksums         0 discarded for bad header offset fields         0 discarded because packet too short         0 discarded due to full reassembly queue     0 connection requests     0 connection accepts     0 bad connection attempts     0 listen queue overflows     0 ignored RSTs in the windows     0 connections established (including accepts)         0 times used RTT from hostcache         0 times used RTT variance from hostcache         0 times used slow-start threshold from hostcache     0 connections closed (including 0 drops)         0 connections updated cached RTT on close         0 connections updated cached RTT variance on close         0 connections updated cached ssthresh on close     0 embryonic connections dropped     0 segments updated rtt (of 0 attempts)     0 retransmit timeouts         0 connections dropped by rexmit timeout     0 persist timeouts         0 connections dropped by persist timeout     0 Connections (fin_wait_2) dropped because of timeout     0 keepalive timeouts         0 keepalive probes sent         0 connections dropped by keepalive     0 correct ACK header predictions     0 correct data packet header predictions     0 syncache entries added         0 retransmitted         0 dupsyn         0 dropped         0 completed         0 bucket overflow         0 cache overflow         0 reset         0 stale         0 aborted         0 badack         0 unreach         0 zone failures     0 cookies sent     0 cookies received     0 hostcache entries added         0 bucket overflow     0 SACK recovery episodes     0 segment rexmits in SACK recovery episodes     0 byte rexmits in SACK recovery episodes     0 SACK options (SACK blocks) received     0 SACK options (SACK blocks) sent     0 SACK scoreboard overflow     0 packets with ECN CE bit set     0 packets with ECN ECT(0) bit set     0 packets with ECN ECT(1) bit set     0 successful ECN handshakes     0 times ECN reduced the congestion window     0 packets with matching signature received     0 packets with bad signature received     0 times failed to make signature due to no SA     0 times unexpected signature received     0 times no signature provided by segment     0 Path MTU discovery black hole detection activations     0 Path MTU discovery black hole detection min MSS activations     0 Path MTU discovery black hole detection failures TCP connection count by state:     0 connections in CLOSED state     0 connections in LISTEN state     0 connections in SYN_SENT state     0 connections in SYN_RCVD state     0 connections in ESTABLISHED state     0 connections in CLOSE_WAIT state     0 connections in FIN_WAIT_1 state     0 connections in CLOSING state     0 connections in LAST_ACK state     0 connections in FIN_WAIT_2 state     0 connections in TIME_WAIT state udp:     0 datagrams received     0 with incomplete header     0 with bad data length field     0 with bad checksum     0 with no checksum     0 dropped due to no socket     0 broadcast/multicast datagrams undelivered     0 dropped due to full socket buffers     0 not for hashed pcb     0 delivered     0 datagrams output     0 times multicast source filter matched ip:     0 total packets received     0 bad header checksums     0 with size smaller than minimum     0 with data size < data length     0 with ip length > max ip packet size     0 with header length < data size     0 with data length < header length     0 with bad options     0 with incorrect version number     0 fragments received     0 fragments dropped (dup or out of space)     0 fragments dropped after timeout     0 packets reassembled ok     0 packets for this host     0 packets for unknown/unsupported protocol     0 packets forwarded (0 packets fast forwarded)     0 packets not forwardable     0 packets received for unknown multicast group     0 redirects sent     0 packets sent from this host     0 packets sent with fabricated ip header     0 output packets dropped due to no bufs, etc.     0 output packets discarded due to no route     0 output datagrams fragmented     0 fragments created     0 datagrams that can't be fragmented     0 tunneling packets that can't find gif     0 datagrams with bad address in header icmp:     0 calls to icmp_error     0 errors not generated in response to an icmp message     0 messages with bad code fields     0 messages less than the minimum length     0 messages with bad checksum     0 messages with bad length     0 multicast echo requests ignored     0 multicast timestamp requests ignored     0 message responses generated     0 invalid return addresses     0 no return routes arp:     0 ARP requests sent     0 ARP requests failed to sent     0 ARP replies sent     0 ARP requests received     0 ARP replies received     0 ARP packets received     0 total packets dropped due to no ARP entry     0 ARP entrys timed out     0 Duplicate IPs seen ip6:     0 total packets received     0 with size smaller than minimum     0 with data size < data length     0 with bad options     0 with incorrect version number     0 fragments received     0 fragments dropped (dup or out of space)     0 fragments dropped after timeout     0 fragments that exceeded limit     0 atomic fragments     0 packets reassembled ok     0 packets for this host     0 packets forwarded     0 packets not forwardable     0 redirects sent     0 packets sent from this host     0 packets sent with fabricated ip header     0 output packets dropped due to no bufs, etc.     0 output packets discarded due to no route     0 output datagrams fragmented     0 fragments created     0 datagrams that can't be fragmented     0 packets that violated scope rules     0 multicast packets which we don't join     Mbuf statistics:         0 one mbuf         0 one ext mbuf         0 two or more ext mbuf     0 packets whose headers are not contiguous     0 tunneling packets that can't find gif     0 packets discarded because of too many headers     0 failures of source address selection icmp6:     0 calls to icmp6_error     0 errors not generated in response to an icmp6 message     0 errors not generated because of rate limitation     0 messages with bad code fields     0 messages < minimum length     0 bad checksums     0 messages with bad length     Histogram of error messages to be generated:         0 no route         0 administratively prohibited         0 beyond scope         0 address unreachable         0 port unreachable         0 packet too big         0 time exceed transit         0 time exceed reassembly         0 erroneous header field         0 unrecognized next header         0 unrecognized option         0 redirect         0 unknown     0 message responses generated     0 messages with too many ND options     0 messages with bad ND options     0 bad neighbor solicitation messages     0 bad neighbor advertisement messages     0 bad router solicitation messages     0 bad router advertisement messages     0 bad redirect messages     0 default routers overflows     0 prefix overflows     0 neighbour entries overflows     0 redirect overflows     0 messages with invalid hop limit     0 path MTU changes rip6:     0 messages received     0 checksum calculations on inbound     0 messages with bad checksum     0 messages dropped due to no socket     0 multicast messages dropped due to no socket     0 messages dropped due to full socket buffers     0 delivered     0 datagrams output pfkey:     0 requests sent from userland     0 bytes sent from userland     0 messages with invalid length field     0 messages with invalid version field     0 messages with invalid message type field     0 messages too short     0 messages with memory allocation failure     0 messages with duplicate extension     0 messages with invalid extension type     0 messages with invalid sa type     0 messages with invalid address extension     0 requests sent to userland     0 bytes sent to userland     0 messages toward single socket     0 messages toward all sockets     0 messages toward registered sockets     0 messages with memory allocation failure ------------------------------------------------------------------------ netstat -m netstat: memstat_kvm_all: ------------------------------------------------------------------------ netstat -anA Active Internet connections (including servers) Tcpcb            Proto Recv-Q Send-Q Local Address      Foreign Address    (state) fffffe00db199418 tcp4       0      0 192.168.1.160.4551 172.217.7.163.80   ESTABLISHED fffffe00db4588f0 tcp4       0      0 192.168.1.160.4551 172.217.13.72.443  ESTABLISHED fffff800cc04daa8 tcp4       0      0 192.168.1.160.4551 96.47.72.84.443    TIME_WAIT fffff800cc04db00 tcp4       0      0 192.168.1.160.4551 96.47.72.84.443    TIME_WAIT fffff800cc04db58 tcp4       0      0 192.168.1.160.4551 96.47.72.84.443    TIME_WAIT fffff800cc04da50 tcp4       0      0 192.168.1.160.4551 96.47.72.84.443    TIME_WAIT fffff800cc04d9f8 tcp4       0      0 192.168.1.160.4551 96.47.72.84.443    TIME_WAIT fffff800cc04dbb0 tcp4       0      0 192.168.1.160.4550 96.47.72.84.443    TIME_WAIT fffffe00db1f1000 tcp4       0      0 192.168.1.160.4550 96.47.72.84.443    ESTABLISHED fffffe00db1f1418 tcp4       0      0 192.168.1.160.4550 96.47.72.84.443    ESTABLISHED fffffe00db1f1830 tcp4       0      0 192.168.1.160.4550 96.47.72.84.443    ESTABLISHED fffffe00db187c48 tcp4       0      0 192.168.1.160.4550 96.47.72.84.443    ESTABLISHED fffffe00db1f1c48 tcp4       0      0 192.168.1.160.4550 72.21.91.29.80     ESTABLISHED fffffe00db188060 tcp4       0      0 192.168.1.160.4550 34.98.75.36.443    ESTABLISHED fffffe00db199c48 tcp4       0      0 192.168.1.160.4550 13.249.39.62.443   ESTABLISHED fffffe00db195ca8 tcp4       0      0 192.168.1.160.4547 172.217.15.78.443  ESTABLISHED fffffe00db188ca8 tcp4       0      0 192.168.1.160.4547 192.0.73.2.443     ESTABLISHED fffff800cc04d840 tcp4       0      0 192.168.1.160.4546 172.217.2.98.443   TIME_WAIT fffff800cc04d8f0 tcp4       0      0 192.168.1.160.4546 142.250.73.227.443 TIME_WAIT fffff800cc04d898 tcp4       0      0 192.168.1.160.4545 172.217.7.226.443  TIME_WAIT fffff800cc04d948 tcp4       0      0 192.168.1.160.4545 172.217.2.110.443  TIME_WAIT fffff800cc04d9a0 tcp4       0      0 192.168.1.160.4545 142.250.73.227.443 TIME_WAIT fffff800cc04dc08 tcp4       0      0 192.168.1.160.4545 142.250.73.195.443 TIME_WAIT fffff800cc04d7e8 tcp4       0      0 192.168.1.160.4545 142.250.31.105.443 TIME_WAIT fffffe00d8b90890 tcp4       0      0 192.168.1.160.4545 172.217.7.227.443  ESTABLISHED fffff800cc04dc60 tcp4       0      0 192.168.1.160.4544 172.217.9.195.443  TIME_WAIT fffffe00d8bdb830 tcp4       0      0 192.168.1.160.4544 172.217.3.99.80    ESTABLISHED fffffe00db1894d8 tcp4       0      0 192.168.1.160.4544 142.250.31.104.443 ESTABLISHED fffffe00d8b3f478 tcp4       0      0 192.168.1.160.4544 142.250.31.99.443  CLOSE_WAIT fffffe00d8b3f890 tcp4       0      0 192.168.1.160.4544 172.217.3.99.80    ESTABLISHED fffffe00d8b8f000 tcp4       0      0 192.168.1.160.4544 72.21.91.29.80     CLOSED fffffe00d8b90060 tcp4       0      0 192.168.1.160.4544 185.199.108.153.44 ESTABLISHED fffffe00d8b8f418 tcp4       0      0 192.168.1.160.4544 185.199.110.153.44 ESTABLISHED fffffe00d8cfd478 tcp4       0      0 192.168.1.160.4543 72.21.91.29.80     CLOSE_WAIT fffffe00d8b404d8 tcp4       0      0 192.168.1.160.4543 72.21.91.29.80     CLOSED fffffe00d8b8f830 tcp4       0      0 192.168.1.160.4543 13.249.39.105.443  CLOSED fffffe00d8b8fc48 tcp4       0      0 192.168.1.160.4543 13.249.39.105.443  CLOSED fffffe00d8b90478 tcp4       0      0 192.168.1.160.4543 34.120.237.76.443  CLOSE_WAIT fffffe00d8bdbc48 tcp4       0      0 192.168.1.160.4543 172.217.3.99.80    ESTABLISHED fffffe00d8b3fca8 tcp4       0      0 192.168.1.160.4543 13.249.39.105.443  CLOSED fffffe00d8b400c0 tcp4       0      0 192.168.1.160.4543 52.33.6.204.443    ESTABLISHED fffffe00d8cfd890 tcp4       0      0 192.168.1.160.4542 99.84.176.92.443   CLOSED fffffe00d8b90ca8 tcp4       0      0 192.168.1.160.4542 72.21.91.29.80     CLOSED fffffe00d8bdc060 tcp4       0      0 192.168.1.160.4542 99.84.176.92.443   CLOSED fffffe00d8cfdca8 tcp4       0      0 192.168.1.160.4542 34.120.5.221.443   CLOSE_WAIT fffffe00d8cfe4d8 tcp4       0      0 192.168.1.160.4542 13.32.196.214.80   ESTABLISHED fffffe00d8b910c0 tcp4       0      0 192.168.1.160.3787 52.85.149.37.443   ESTABLISHED fffffe00d8b914d8 tcp4       0      0 192.168.1.160.2102 18.208.93.32.443   CLOSE_WAIT fffffe00d8cfe0c0 tcp4       0      0 192.168.1.160.6160 34.107.221.82.80   ESTABLISHED fffffe00d8bdc478 tcp4       0      0 192.168.1.160.4903 34.107.221.82.80   ESTABLISHED fffffe00d8bdc890 tcp4       0      0 127.0.0.1.139 *.*                LISTEN fffffe00d8bdcca8 tcp4       0      0 127.0.0.1.445 *.*                LISTEN fffffe00d8bdd0c0 tcp4       0      0 192.168.1.160.139 *.*                LISTEN fffffe00d8bdd4d8 tcp4       0      0 192.168.1.160.445 *.*                LISTEN fffffe00d8cfe8f0 tcp4       0      0 192.168.1.160.5357 *.*                LISTEN fffffe00d8bdd8f0 tcp4       0      0 *.22 *.*                LISTEN fffffe00d8b408f0 tcp6       0      0 *.22 *.*                LISTEN fffffe00d8b918f0 tcp4       0      0 127.0.0.1.25 *.*                LISTEN fffff8000d7d0b70 udp4       0      0 192.168.1.160.3702 *.* fffff8000dc88d58 udp4       0      0 *.11821 *.* fffff800870cb5b8 udp4       0      0 239.255.255.250.37 *.* fffff800870cbb70 udp4       0      0 192.168.1.255.138 *.* fffff800870cbd58 udp4       0      0 192.168.1.160.138 *.* fffff8000d870000 udp4       0      0 192.168.1.255.137 *.* fffff8000d8701e8 udp4       0      0 192.168.1.160.137 *.* fffff8000d870988 udp4       0      0 *.138 *.* fffff8000d870d58 udp4       0      0 *.137 *.* fffff8000d5723d0 udp6       0      0 *.23305 *.* fffff8000d572b70 udp4       0      0 *.58262 *.* fffff8000d5725b8 udp6       0      0 *.5353 *.* fffff8000d572988 udp4       0      0 *.5353 *.* fffff8000d572d58 udp4       0      0 *.514 *.* fffff8000d5727a0 udp6       0      0 *.514 *.* ------------------------------------------------------------------------ netstat -aL Current listen queue sizes (qlen/incqlen/maxqlen) Proto Listen                           Local Address tcp4  0/0/128                          localhost.netbios-ssn tcp4  0/0/128                          localhost.microsoft-ds tcp4  0/0/128                          drb-freebsd-13.f.netbi tcp4  0/0/128                          drb-freebsd-13.f.micro tcp4  0/0/5                            drb-freebsd-13.f.5357 tcp4  0/0/128                          *.ssh tcp6  0/0/128                          *.ssh tcp4  0/0/10                           localhost.smtp ------------------------------------------------------------------------ fstat USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W root     Timer        175 root /            34 drwxr-xr-x      26 r root     Timer        175   wd /            34 drwxr-xr-x      26 r root     kldload      174 root /            34 drwxr-xr-x      26 r root     kldload      174   wd /            34 drwxr-xr-x      26 r root     kldload      174 text /           603 -r-xr-xr-x    8720 r root     kldload      174 ctty /dev          6 crw-------  console rw root     kldload      174    0 /dev          6 crw-------  console rw root     kldload      174    1 /dev         13 crw-rw-rw-    null w root     kldload      174    2 /dev         13 crw-rw-rw-    null w root     sh           172 root /            34 drwxr-xr-x      26 r root     sh           172   wd /            34 drwxr-xr-x      26 r root     sh           172 text /        361232 -r-xr-xr-x  162616 r root     sh           172 ctty /dev          6 crw-------  console rw root     sh           172    0 /dev          6 crw-------  console rw root     sh           172    1 /dev          6 crw-------  console rw root     sh           172    2 /dev          6 crw-------  console rw root     sh           172   10 /        428033 -rwxr-xr-x    1178 r root     sh            26 root /            34 drwxr-xr-x      26 r root     sh            26   wd /            34 drwxr-xr-x      26 r root     sh            26 text /        361232 -r-xr-xr-x  162616 r root     sh            26 ctty /dev          6 crw-------  console rw root     sh            26    0 /dev          6 crw-------  console rw root     sh            26    1 /dev          6 crw-------  console rw root     sh            26    2 /dev          6 crw-------  console rw root     sh            26   10 /           880 -rw-r--r--    5042 r root     init           1 root /            34 drwxr-xr-x      26 r root     init           1   wd /            34 drwxr-xr-x      26 r root     init           1 text /        361300 -r-xr-xr-x 1253152  r root     kernel         0 root /            34 drwxr-xr-x      26 r root     kernel         0   wd /            34 drwxr-xr-x      26 r ------------------------------------------------------------------------ dmesg ---<>--- Copyright (c) 1992-2021 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994     The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-RC2 #0 releng/13.0-n244684-13c22f74953: Fri Mar 12 04:05:19 UTC 2021 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe) VT(vga): text 80x25 CPU: Intel(R) Core(TM) i7 CPU         950  @ 3.07GHz (3103.68-MHz K8-class CPU)   Origin="GenuineIntel"  Id=0x106a5  Family=0x6  Model=0x1a Stepping=5 Features=0x1783fbff   Features2=0x180201   AMD Features=0x28100800   AMD Features2=0x1   Structured Extended Features3=0x10000000   TSC: P-state invariant real memory  = 9126805504 (8704 MB) avail memory = 8285257728 (7901 MB) Event timer "LAPIC" quality 100 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 cache groups x 1 core(s) random: unblocking device. ioapic0: MADT APIC ID 4 != hw id 0 ioapic0 irqs 0-23 Launching APs: 2 1 3 Timecounter "TSC-low" frequency 1551839460 Hz quality 1000 KTLS: Initialized 4 threads random: entropy device external interface [ath_hal] loaded WARNING: Device "kbd" is Giant locked and may be deleted before FreeBSD 14.0. kbd1 at kbdmux0 000.000058 [4350] netmap_init               netmap: loaded module mlx5en: Mellanox Ethernet driver 3.6.0 (December 2020) nexus0 vtvga0: cryptosoft0: aesni0: No AES or SHA support. acpi0: acpi0: Power Button (fixed) acpi0: Sleep Button (fixed) cpu0: on acpi0 attimer0: port 0x40-0x43,0x50-0x53 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 isab0: at device 1.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xd000-0xd00f at device 1.1 on pci0 ata0: at channel 0 on atapci0 ata1: at channel 1 on atapci0 vgapci0: mem 0xe0000000-0xefffffff irq 18 at device 2.0 on pci0 vgapci0: Boot video device em0: port 0xd010-0xd017 mem 0xf0000000-0xf001ffff irq 19 at device 3.0 on pci0 em0: Using 1024 TX descriptors and 1024 RX descriptors em0: Ethernet address: 08:00:27:dd:d4:fc em0: netmap queues/slots: TX 1/1024, RX 1/1024 pcm0: port 0xd100-0xd1ff,0xd200-0xd23f irq 21 at device 5.0 on pci0 pcm0: ohci0: mem 0xf0804000-0xf0804fff irq 22 at device 6.0 on pci0 usbus0 on ohci0 usbus0: 12Mbps Full Speed USB v1.0 pci0: at device 7.0 (no driver attached) ehci0: mem 0xf0805000-0xf0805fff irq 19 at device 11.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci0 usbus1: 480Mbps High Speed USB v2.0 acpi_acad0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 14.0. psm0: model IntelliMouse Explorer, device ID 4 orm0: at iomem 0xc0000-0xc7fff,0xe2000-0xe2fff pnpid ORM0000 on isa0 vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff pnpid PNP0900 on isa0 atrtc0: at port 0x70 irq 8 on isa0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 atrtc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 14. Timecounters tick every 10.000 msec ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Trying to mount root from zfs:zroot/ROOT/default []... Root mount waiting for: CAM usbus0 usbus1 ugen0.1: at usbus0 uhub0 on usbus0 ugen1.1: at usbus1 uhub0: on usbus0 uhub1 on usbus1 uhub1: on usbus1 ada0 at ata0 bus 0 scbus0 target 0 lun 0 ada0: ATA8-ACS device ada0: Serial Number VBb8450cb8-277eed0b ada0: 33.300MB/s transfers (UDMA2, PIO 65536bytes) ada0: 61440MB (125829120 512 byte sectors) cd0 at ata1 bus 0 scbus1 target 0 lun 0 cd0: Removable CD-ROM SCSI device cd0: Serial Number VB2-01700376 cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present uhub0: 12 ports with 12 removable, self powered Root mount waiting for: usbus1 Root mount waiting for: usbus1 Root mount waiting for: usbus1 Root mount waiting for: usbus1 uhub1: 12 ports with 12 removable, self powered Setting hostuuid: 56767ad9-d311-7b42-9f7f-db26ca8d9b82. Setting hostid: 0x10458f66. Starting file system checks: no pools available to import Mounting local filesystems:. Setting hostname: drb-freebsd-13.localdomain. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/gcc10 /usr/local/lib/gcc9 /usr/local/lib/mysql /usr/local/lib/mysql/plugin /usr/local/lib/perl5/5.32/mach/CORE /usr/local/lib/qt5 /usr/local/lib/samba4 /usr/local/lib/signon /usr/local/lib/signon/extensions /usr/local/llvm10/lib 32-bit compatibility ldconfig path: /usr/lib32 Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: . vboxguest0 port 0xd020-0xd03f mem 0xf0400000-0xf07fffff,0xf0800000-0xf0803fff irq 20 at device 4.0 on pci0 Sleeping thread (tid 100521, pid 174) owns a non-sleepable lock KDB: stack backtrace of thread 100521: #0 0xffffffff80c16291 at mi_switch+0xc1 #1 0xffffffff8293cf37 at rtR0SemEventMultiBsdWait+0x297 #2 0xffffffff8291d36a at vgdrvHgcmAsyncWaitCallbackWorker+0x14a #3 0xffffffff8291e49b at VbglR0HGCMInternalConnect+0x11b #4 0xffffffff8291ad33 at VGDrvCommonIoCtl+0xb53 #5 0xffffffff82919af6 at VGDrvCommonProcessOptionsFromHost+0x146 #6 0xffffffff8291d9f8 at vgdrvFreeBSDAttach+0x1d8 #7 0xffffffff80c462ad at device_attach+0x3dd #8 0xffffffff80c45eb1 at device_probe_and_attach+0x41 #9 0xffffffff8085b966 at pci_driver_added+0xf6 #10 0xffffffff80c43d29 at devclass_driver_added+0x39 #11 0xffffffff80c43cad at devclass_add_driver+0x13d #12 0xffffffff80be6964 at module_register_init+0xa4 #13 0xffffffff80bd8667 at linker_load_module+0xbc7 #14 0xffffffff80bd9e4b at kern_kldload+0xbb #15 0xffffffff80bd9f7b at sys_kldload+0x5b #16 0xffffffff8108aa8c at amd64_syscall+0x10c #17 0xffffffff810618be at fast_syscall_common+0xf8 panic: sleeping thread cpuid = 0 time = 1616100801 KDB: stack backtrace: #0 0xffffffff80c57105 at kdb_backtrace+0x65 #1 0xffffffff80c09cd1 at vpanic+0x181 #2 0xffffffff80c09b43 at panic+0x43 #3 0xffffffff80c6e526 at propagate_priority+0x296 #4 0xffffffff80c6f0c3 at turnstile_wait+0x313 #5 0xffffffff80be7b9c at __mtx_lock_sleep+0x17c #6 0xffffffff80c256bf at softclock_call_cc+0xff #7 0xffffffff80c25b89 at softclock+0x79 #8 0xffffffff80bcae0d at ithread_loop+0x24d #9 0xffffffff80bc7c0e at fork_exit+0x7e #10 0xffffffff8106201e at fork_trampoline+0xe Uptime: 8s Dumping 364 out of 8152 MB:..5%..14%..22%..31%..44%..53%..62%..71%..84%..93% ------------------------------------------------------------------------ kernel config options    CONFIG_AUTOGENERATED ident    GENERIC machine    amd64 cpu    HAMMER makeoptions    WITH_CTF=1 makeoptions    DEBUG=-g options    IICHID_SAMPLING options    HID_DEBUG options    EVDEV_SUPPORT options    XENHVM options    USB_DEBUG options    ATH_ENABLE_11N options    AH_AR5416_INTERRUPT_MITIGATION options    IEEE80211_SUPPORT_MESH options    IEEE80211_DEBUG options    SC_PIXEL_MODE options    VESA options    COMPAT_LINUXKPI options    PCI_IOV options    PCI_HP options    IOMMU options    EARLY_AP_STARTUP options    SMP options    NETGDB options    NETDUMP options    DEBUGNET options    ZSTDIO options    GZIO options    EKCD options    KDB_TRACE options    KDB options    RCTL options    RACCT_DEFAULT_TO_DISABLED options    RACCT options    INCLUDE_CONFIG_FILE options    DDB_CTF options    KDTRACE_HOOKS options    KDTRACE_FRAME options    MAC options    CAPABILITIES options    CAPABILITY_MODE options    AUDIT options    HWPMC_HOOKS options    KBD_INSTALL_CDEV options    PRINTF_BUFR_SIZE=128 options    _KPOSIX_PRIORITY_SCHEDULING options    SYSVSEM options    SYSVMSG options    SYSVSHM options    STACK options    KTRACE options    SCSI_DELAY=5000 options    COMPAT_FREEBSD12 options    COMPAT_FREEBSD11 options    COMPAT_FREEBSD10 options    COMPAT_FREEBSD9 options    COMPAT_FREEBSD7 options    COMPAT_FREEBSD6 options    COMPAT_FREEBSD5 options    COMPAT_FREEBSD4 options    COMPAT_FREEBSD32 options    EFIRT options    GEOM_LABEL options    GEOM_RAID options    TMPFS options    PSEUDOFS options    PROCFS options    CD9660 options    MSDOSFS options    NFS_ROOT options    NFSLOCKD options    NFSD options    NFSCL options    MD_ROOT options    QUOTA options    UFS_GJOURNAL options    UFS_DIRHASH options    UFS_ACL options    SOFTUPDATES options    FFS options    KERN_TLS options    SCTP_SUPPORT options    TCP_RFC7413 options    TCP_HHOOK options    TCP_BLACKBOX options    TCP_OFFLOAD options    ROUTE_MPATH options    IPSEC_SUPPORT options    INET6 options    INET options    VIMAGE options    PREEMPTION options    NUMA options    SCHED_ULE options    NEW_PCIB options    GEOM_PART_GPT options    GEOM_PART_MBR options    GEOM_PART_EBR options    GEOM_PART_BSD device    isa device    mem device    io device    uart_ns8250 device    cpufreq device    acpi device    pci device    fdc device    ahci device    ata device    mvs device    siis device    ahc device    ahd device    esp device    hptiop device    isp device    mpt device    mps device    mpr device    sym device    isci device    ocs_fc device    pvscsi device    scbus device    ch device    da device    sa device    cd device    pass device    ses device    amr device    arcmsr device    ciss device    iir device    ips device    mly device    twa device    smartpqi device    tws device    aac device    aacp device    aacraid device    ida device    mfi device    mlx device    mrsas device    pmspcv device    twe device    nvme device    nvd device    vmd device    vmd_bus device    atkbdc device    atkbd device    psm device    kbdmux device    vga device    splash device    sc device    vt device    vt_vga device    vt_efifb device    vt_vbefb device    agp device    cbb device    pccard device    cardbus device    uart device    ppc device    ppbus device    lpt device    ppi device    puc device    iflib device    em device    ix device    ixv device    ixl device    iavf device    ice device    vmx device    axp device    bxe device    le device    ti device    mlx5 device    mlxfw device    mlx5en device    miibus device    ae device    age device    alc device    ale device    bce device    bfe device    bge device    cas device    dc device    et device    fxp device    gem device    jme device    lge device    msk device    nfe device    nge device    re device    rl device    sge device    sis device    sk device    ste device    stge device    vge device    vr device    xl device    wlan device    wlan_wep device    wlan_ccmp device    wlan_tkip device    wlan_amrr device    an device    ath device    ath_pci device    ath_hal device    ath_rate_sample device    ipw device    iwi device    iwn device    malo device    mwl device    ral device    wpi device    crypto device    aesni device    loop device    padlock_rng device    rdrand_rng device    ether device    vlan device    tuntap device    md device    gif device    firmware device    xz device    bpf device    uhci device    ohci device    ehci device    xhci device    usb device    ukbd device    umass device    sound device    snd_cmi device    snd_csa device    snd_emu10kx device    snd_es137x device    snd_hda device    snd_ich device    snd_via8233 device    mmc device    mmcsd device    sdhci device    rtsx device    virtio device    virtio_pci device    vtnet device    virtio_blk device    virtio_scsi device    virtio_balloon device    hyperv device    xenpci device    netmap device    evdev device    uinput device    hid ------------------------------------------------------------------------ ddb capture buffer ddb: ddb_capture: kvm_nlist