From owner-freebsd-stable@FreeBSD.ORG Sun Feb 2 19:46:28 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BD661E57 for ; Sun, 2 Feb 2014 19:46:28 +0000 (UTC) Received: from mail.modirum.com (mail.modirum.com [31.185.27.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id DF20E17F8 for ; Sun, 2 Feb 2014 19:46:27 +0000 (UTC) Received: from [77.87.241.103] (helo=unknown) by mail.modirum.com with esmtpsa (TLSv1:DHE-RSA-AES128-SHA:128) (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1WA2zk-000EJX-Lh for freebsd-stable@freebsd.org; Sun, 02 Feb 2014 19:46:22 +0000 Date: Sun, 2 Feb 2014 20:46:23 +0100 From: Matthew Rezny To: freebsd-stable@freebsd.org Subject: Tuning kern.maxswzone is minor compared to hangs in "kmem a" state Message-ID: <20140202204623.00003fe5@unknown> In-Reply-To: <20140201221612.00001897@unknown> References: <20140201070912.00007971@unknown> <20140201221612.00001897@unknown> Organization: RezTek, s.r.o. X-Mailer: Claws Mail 3.9.2-55-g74b05b (GTK+ 2.16.6; i586-pc-mingw32msvc) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-SA-Authenticated: Yes X-SA-Exim-Connect-IP: 77.87.241.103 X-SA-Exim-Mail-From: matthew@reztek.cz X-SA-Exim-Scanned: No (on mail.modirum.com); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Feb 2014 19:46:28 -0000 On Sat, 1 Feb 2014 22:16:12 +0100 Matthew Rezny wrote: > On Sat, 01 Feb 2014 13:03:15 -0600 > Richard Todd wrote: > > > Matthew Rezny writes: > > > > > So, as best I can tell, the actual effective number used for > > > kern.maxswzone is indeed approximately 8x available RAM. If there > > > is some need to turn it down (using substantially less swap) then > > > that is possible, but turning it up (as suggested by the warning > > > message) is not possible. Setting any value higher does not appear > > > to actually > > > > Yeah, IIRC I ran into that when configing some VMs with > > really big swap space for benefit of tmpfs. This is the quick hack > > I used to get around that, you might give it a try. > > > > # HG changeset patch > > # Parent e4dd7df011139e2b224835aa6e330c90afcf9a55 > > patch swap_pager to unconditionally use maxswzone tunable if it is > > set -- helpful for our tbvm VMs with large swap space for tmpfs > > > > diff -r e4dd7df01113 sys/vm/swap_pager.c > > --- a/sys/vm/swap_pager.c Wed Feb 20 00:15:49 2013 -0600 > > +++ b/sys/vm/swap_pager.c Sat Feb 23 13:08:54 2013 -0600 > > @@ -546,7 +546,7 @@ > > * is typically limited to around 32MB by default. > > */ > > n = cnt.v_page_count / 2; > > - if (maxswzone && n > maxswzone / sizeof(struct swblock)) > > + if (maxswzone) > > n = maxswzone / sizeof(struct swblock); > > n2 = n; > > swap_zone = uma_zcreate("SWAPMETA", sizeof(struct swblock), > > NULL, NULL, > > > Thank you for pointing me in the right direction. Now that I'm > looking at the right file, I don't know how I failed to find it > myself with grep. The logic here is rather obviously flawed, the > maxswzone value is only used if it is less than the calculated > default. If there is some reason to not allow adjusting this up, then > the warning message is incorrect. Either way, something should change > and I guess I should file a PR on this one. At least I can see the > warning is taking the doubling for safety into account, so for my > particular case with an overrun under 10% it should probably be ok to > put fixing this on the back burner. > > > Also, > > > > > With /usr/src cleared (and after running fsck) I booted back into > > > 10-PRERELEASE to try to fetch the 10-STABLE sources again. I > > > started svnlite co and find it hung shortly thereafter. I tried a > > > few times but each time I see it does a couple hundred files at > > > best and just stops. When it stops, I can't login to another > > > terminal. If I have a spare console logged in, I can't run > > > anything. After a few tries, I manged to catch it where I had top > > > running in one VT, started the checkout, and then switched back > > > just in time. I never could even get top up with rm running (it > > > probably blows over some limit faster). When the checkout hangs, > > > the state of svnlite is "kmem a" according to top. I can only > > > guess that is short for kmem alloc, I guess some syscall is > > > waiting on an allocation that will never happen because something > > > already is using or has leaked everything that could satisfy that > > > allocation. It looks like activity on too many files within a > > > short period runs something out. > > > > No, it's just a new bit of debugging code that causes the system > > to spend lots of CPU time verifying integrity of some of its > > internal data structures, especially on wimpy hardware (e.g., my > > dual PII/400 box, which is where I noticed this recently.) You'll > > find if you're patient that it isn't a complete hang, it will > > actually get work done in between the debug passes. Set sysctl > > debug.vmem_check=0 to disable the check. This is I think > > completely independent from the maxswzone stuff, it's just you were > > seeing it for the first time since the debug code in question was > > only recently added to 10-STABLE. > > > If only it were that simple. I'm not yet on 10-STABLE, I'm struggling > to get the sources to reach that point. These C3 boxes are > 10-PRERELEASE so I don't yet have this debug.vmem_check to tweak. > sysctl says unknown oid when I query it. I tried setting it from > loader prompt but still says unknown oid and I see no change in > behavior. > > Also, I'm not seeing anything using lots of CPU time. If I start off > top on another VT before I start svnlite, then I have a decent chance > of seeing what goes on until the situation becomes dire. svnlite > starts off moving quick and using lots of CPU time (>50%) for the > first hundred files or so, then it halts in the "kmem a" state and > CPU is completely idle. It sits there for a while, and eventually does > something more. Each time the stop is longer and less work is done in > the interval between stops. Eventually the process appears to hang > completely in that I can leave it for half a day and no more progress > is made. The rm process could go longer in this state with still some > visible progress since it's operation is sufficiently simple to > actually observe it managed to do something, whereas svnlite might do > something occasionally but it's not enough to get to the next file in > the list. > > With each burst and wait cycle, I see a spray in dmesg saying > calcru: runtime went backwards [lots] for {all processes}. If I hit > ctrl-C and wait, it'll interrupt svnlite the next time it would do > something other than wait. It can easily take 10 min for that to > happen, meaning the wait period is long enough to consider the process > as not effectively running. If I let it go long enough, it gets to a > state where it never exits on ctrl-c (or at least it'd take more hours > than I've been willing to wait). In this last attempt, I let it go for > 5 min, pushed ctrl-c, waited another 10 min, then tried ctrl-alt-del > and that just beeps so the attempt to interrupt the process resulted > in total I/O lockup to the point key presses are not handled. That > last one was enough to finally require manual fsck on reboot (which > should be a testament to the resiliency of UFS as I've pushed the > reset button a hundred times in the past day and a half). > > > Richard > > > > > > Thank you for taking the time to respond. It's now clear the maxswzone > is just a red herring, the real issue are the apparent hangs which I'm > seeing on several more boxes. The mystery is why the one box with > slightly more RAM seems ok, but a couple boxes with far more RAM are > not ok. That will probably be answered when I figure out what the > cause is. Reply to self for status update. The situation with these C3 boxes and 10.0 is just miserable. I tried using tar, both streaming with netcat and from a file, to populate /usr/src on the smaller box, but tar would hang eventually too. Tar got further probably because it's only handling the source files and not constantly updating a database file as svn does. I decided to try running svnlite through truss to see if there was a particular syscall causing trouble. In two runs I saw that it got further than ever before (likely truss output was limiting overall execution speed and thus reducing peak instantaneous disk I/O load) but stopped at different points. The first attempt hung on a stat() call on wc.db and I saw that a call involving the same file just a few lines before had failed with file not found, but that file must exist and indeed was present (but corrupt) following a reboot and fsck. The second run through stopped on a blank line as if the next syscall was setup but the name didn't get printed before everything hung hard. Following the pattern of 5 gettimeofday() between each lstat(), the next call should have been an lstat() but on which file is unknown because each lstat() prior was a different file. It looks like the hangs might be connected to errors opening files that do exist. That bit of info ties this in another way to the P4 boxes mentioned in my other thread, which simply failed out of builworld with claims source files didn't exist, but completed buildworld on the next attempt. After deciding there was just no way to have significant disk I/O on the box with only 256MB RAM, I moved it's harddrive to the box with 384MB and extracted the tar there, That succeeded on the first attempt, which verifies the UFS volume hasn't been horribly trashed in some way that manifests as file access related hangs. Booting the drive in the other system also verifies it is not any minor accidental configuration difference as now all that changed was the available memory and suddenly the same OS instance could extract the tar of both /usr/src and /usr/ports no problem. I put the harddrives back in their respective boxes and booted them both up. On each I started off top on one VT and then went to another VT and ran svnlite status to verify that svn regards the source tree as complete. On the box with 384MB RAM, that command returned the expected result in under a minute. On the box with 256MB RAM, I saw a quick burst of disk I/O and then nothing. Top confirmed svnlite was sitting in "kmem a" state. I hit ctrl-c and let it sit. An hour later it was sitting in the same place (so zero progress) and I went to sleep. When I awoke some 6 hours later, it was in the same state with the same 9sec CPU time, the process definitely isn't even creeping along with periodic progress like rm sometimes manages. Looking at top revealed that ntpd has joined svnlite in the "kmem a" state. I had noticed yesterday the clock was off by several hours. Now I know why, if I leave it hung too long ntpd gets hung and the clock grinds to a halt. How could a process hanging stop the RTC on the mobo? Only thing I can think of is that might happen if ntpd were to hang in the midst of an update to the RTC. I wonder if all the calcru: runtime went backwards errors are related to periodic clock stops. I rebooted the hung box and fsck showed nothing significant. I started top on both (to ensure it's available) and then started a make buildworld on each. On the box with 256MB, that hung in the initial rm -rf /usr/obj/usr/src/tmp. Yep, that should be no surprise, the old /usr/obj is still there and it's too big to rm. Using mfsbsd I was able to clear that directory with just a few tries (same as before, rm causes pressure fast enough to blow out sh, getty and everything short of init). Booted back into the installed OS, I started the buildworld again and it had been making reasonable progress for a few minutes. During compilation, file I/O will be bursts with long waits between as the CPU churns. My amd64 boxes are fast enough to be bottlenecked on disk I/O unless I build to a tmpfs. These C3 boxes are so slow the compiler could never hope to saturate the disk, but a big copy or an mtree might blow it up the same as the initial rm. Just as I was about to hit send, the box panicked. Finally a panic and not just a hang so there's a backtrace! That should be the first hint as to where in the kernel I'm hitting trouble. The panic points the finger at UFS. The filesystem has seen some thrashing but it did just fsck without significant error before that build was started. So, I'm thinking it's not on-disk corruption but an in-memory problem. It blew up writing a softdep, which starts me wondering if the buffers used for softupdates bookkeeping are overflowing in the kernel. That would make some sense, rapidly creating (svn/tar) or deleting (rm) many small files will create a huge amount of activity for softupdates to track, defer and reorder. The rm case is probably the one than can trigger it the fastest because deletes are focusing the beating on the metadata without wasting time between on file data I/O. Here's the full core.txt in hopes the vmstat numbers add some useful information beyond the backtrace. Sun Feb 2 20:07:30 CET 2014 FreeBSD music2.reztek 10.0-PRERELEASE FreeBSD 10.0-PRERELEASE #1: Sat Jan 11 09:00:08 CET 2014 root@music2.reztek:/usr/obj/usr/src/sys/MUSIC i386 panic: handle_written_inodeblock: Invalid link count 65529 for inodedep 0xc7648b00 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... Unread portion of the kernel message buffer: panic: handle_written_inodeblock: Invalid link count 65529 for inodedep 0xc7648b00 KDB: stack backtrace: #0 0xc0667f6f at kdb_backtrace+0x52 #1 0xc063dc7d at panic+0x85 #2 0xc084483f at softdep_disk_write_complete+0x1294 #3 0xc06b0656 at bufdone_finish+0x23 #4 0xc06b04f5 at bufdone+0x39 #5 0xc05e0d13 at g_vfs_done+0x289 #6 0xc05dd8ea at g_io_deliver+0x28f #7 0xc05e0648 at g_std_done+0x4e #8 0xc05dd8ea at g_io_deliver+0x28f #9 0xc05e0648 at g_std_done+0x4e #10 0xc05dd8ea at g_io_deliver+0x28f #11 0xc05e0648 at g_std_done+0x4e #12 0xc05dd8ea at g_io_deliver+0x28f #13 0xc05db68f at g_disk_done_single+0xfa #14 0xc0487fb7 at adadone+0x482 #15 0xc046de75 at xpt_done_process+0x435 #16 0xc0470a11 at xpt_done_td+0x15a #17 0xc0615573 at fork_exit+0xa3 Uptime: 6m58s Physical memory: 239 MB Dumping 102 MB: 87 71 55 39 23 7 No symbol "stopped_cpus" in current context. #0 doadump (textdump=-1030183936) at pcpu.h:233 233 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=-1030183936) at pcpu.h:233 #1 0xc063d96b in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:452 #2 0xc063dce6 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:759 #3 0xc084483f in softdep_disk_write_complete (bp=0x0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:11270 #4 0xc06b0656 in bufdone_finish (bp=0xc1ecfbe0) at buf.h:420 #5 0xc06b04f5 in bufdone (bp=) at /usr/src/sys/kern/vfs_bio.c:3754 #6 0xc05e0d13 in g_vfs_done (bip=) at /usr/src/sys/geom/geom_vfs.c:157 #7 0xc05dd8ea in g_io_deliver (bp=0xc724f390, error=) at /usr/src/sys/geom/geom_io.c:669 #8 0xc05e0648 in g_std_done (bp=) at /usr/src/sys/geom/geom_subr.c:1018 #9 0xc05dd8ea in g_io_deliver (bp=0xc58f5000, error=) at /usr/src/sys/geom/geom_io.c:669 #10 0xc05e0648 in g_std_done (bp=) at /usr/src/sys/geom/geom_subr.c:1018 #11 0xc05dd8ea in g_io_deliver (bp=0xc724f980, error=) at /usr/src/sys/geom/geom_io.c:669 #12 0xc05e0648 in g_std_done (bp=) at /usr/src/sys/geom/geom_subr.c:1018 #13 0xc05dd8ea in g_io_deliver (bp=0xc724fb48, error=) at /usr/src/sys/geom/geom_io.c:669 #14 0xc05db68f in g_disk_done_single (bp=0xc724fb48) at /usr/src/sys/geom/geom_disk.c:275 #15 0xc0487fb7 in adadone (periph=, done_ccb=) at /usr/src/sys/cam/ata/ata_da.c:1759 #16 0xc046de75 in xpt_done_process (ccb_h=0xc564c800) at /usr/src/sys/cam/cam_xpt.c:5247 #17 0xc0470a11 in xpt_done_td (arg=0xc09c3e00) at /usr/src/sys/cam/cam_xpt.c:5278 #18 0xc0615573 in fork_exit (callout=0xc04708b7 ) at /usr/src/sys/kern/kern_fork.c:995 #19 0xc08bd154 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:279 Current language: auto; currently minimal (kgdb) ------------------------------------------------------------------------ ps -axlww UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 0 0 0 -8 0 0 0 - DLs - 0:00.01 [kernel] 0 1 0 0 8 0 9240 3154304 wait DLs - 0:00.03 [init] 0 2 0 0 8 0 0 0 crypto_w DL - 0:00.00 [crypto] 0 3 0 0 8 0 0 0 crypto_r DL - 0:00.00 [crypto returns] 0 4 0 0 -8 0 0 0 ccb_scan DL - 0:03.55 [cam] 0 5 0 0 -8 0 0 0 ctl_work DL - 0:00.00 [ctl_thrd] 0 6 0 0 -16 0 0 0 waiting_ DL - 0:00.00 [sctp_iterator] 0 7 0 0 -16 0 0 0 psleep DL - 0:00.83 [pagedaemon] 0 8 0 0 16 0 0 0 psleep DL - 0:00.08 [vmdaemon] 0 9 0 0 155 0 0 0 pgzero DL - 0:00.00 [pagezero] 0 10 0 0 -16 0 0 0 audit_wo DL - 0:00.00 [audit] 0 11 0 0 155 0 0 0 - RL - 1:48.78 [idle] 0 12 0 0 -84 0 0 0 - WL - 0:11.94 [intr] 0 13 0 0 -8 0 0 0 - DL - 0:00.05 [geom] 0 14 0 0 -16 0 0 0 - DL - 0:00.65 [rand_harvestq] 0 15 0 0 -68 0 0 0 - DL - 0:00.01 [usb] 0 16 0 0 -16 0 0 0 psleep DL - 0:00.36 [bufdaemon] 0 17 0 0 16 0 0 0 syncer DL - 0:00.18 [syncer] 0 18 0 0 -4 0 0 0 vlruwt DL - 0:00.01 [vnlru] 0 19 0 0 -16 0 0 0 sdflush DL - 0:00.30 [softdepflush] 0 20 0 0 -16 0 0 0 - DL - 0:00.06 [schedcpu] 0 1090 1 0 60 0 10048 3151424 select Ds - 0:00.01 [dhclient] 65 1126 1 0 55 0 10048 3155264 select Ds - 0:00.00 [dhclient] 0 1127 1 0 40 0 9224 3152384 select Ds - 0:00.00 [devd] 0 1274 1 0 40 0 9944 3153344 select Ds - 0:00.39 [syslogd] 0 1369 1 0 40 0 12032 3148544 select Ds - 0:00.15 [ntpd] 0 1398 1 0 59 0 14312 3149504 select Ds - 0:00.00 [sshd] 0 1402 1 0 8 0 9984 3146624 nanslp Ds - 0:00.12 [cron] 0 1425 1 0 59 0 10068 3158144 select Ds - 0:00.00 [moused] 0 1468 1 0 58 0 9936 3176448 ttyin Ds+ - 0:00.01 [getty] 0 1469 1 0 8 0 10416 3174528 wait DWs - 0:00.00 [login] 0 1470 1 0 8 0 10416 3177408 wait DWs - 0:00.00 [login] 0 1471 1 0 58 0 9936 3150464 ttyin Ds+ - 0:00.01 [getty] 0 1472 1 0 58 0 9936 3172608 ttyin Ds+ - 0:00.01 [getty] 0 1473 1 0 58 0 9936 3173568 ttyin Ds+ - 0:00.01 [getty] 0 1474 1 0 58 0 9936 3171648 ttyin Ds+ - 0:00.01 [getty] 0 1475 1 0 58 0 9936 3170688 ttyin Ds+ - 0:00.01 [getty] 0 1476 1470 0 16 0 10656 3169728 pause DW - 0:00.00 [csh] 0 1478 1476 0 40 0 11056 3168768 select D+ - 0:02.19 [top] 0 1479 1469 0 16 0 10656 3167808 pause DW - 0:00.00 [csh] 0 1483 1479 0 8 0 8944 3166848 wait DW+ - 0:00.00 [make] 0 1488 1483 0 8 0 8944 3165888 wait DW+ - 0:00.00 [make] 0 1619 1488 0 8 0 8944 3164928 wait DW+ - 0:00.00 [make] 0 1623 1619 0 8 0 10272 3163968 wait DW+ - 0:00.00 (sh) 0 1725 1623 0 8 0 8944 3147584 wait D+ - 0:00.76 [make] 0 1793 1725 0 8 0 27316 3156224 wait D+ - 0:00.04 [c++] 0 1794 1793 0 -16 0 37300 4128768 vnread DL+ - 0:06.44 [c++] ------------------------------------------------------------------------ vmstat -s 201483 cpu context switches 867407 device interrupts 21239 software interrupts 229915 traps 396065 system calls 20 kernel threads created 1419 fork() calls 355 vfork() calls 0 rfork() calls 9 swap pager pageins 33 swap pager pages paged in 52 swap pager pageouts 153 swap pager pages paged out 1558 vnode pager pageins 10442 vnode pager pages paged in 0 vnode pager pageouts 0 vnode pager pages paged out 108 page daemon wakeups 0 pages examined by the page daemon 116 pages reactivated 37654 copy-on-write faults 248 copy-on-write optimized faults 144312 zero fill pages zeroed 3353 zero fill pages prezeroed 9 intransit blocking page faults 201812 total VM faults taken 1537 page faults requiring I/O 0 pages affected by kernel thread creation 54059 pages affected by fork() 16118 pages affected by vfork() 0 pages affected by rfork() 0 pages cached 238064 pages freed 0 pages freed by daemon 0 pages freed by exiting processes 10588 pages active 265 pages inactive 8 pages in VM cache 20998 pages wired down 28211 pages free 4096 bytes per page 504005 total name lookups cache hits (89% pos + 1% neg) system 0% per-directory deletions 0%, falsehits 0%, toolong 0% ------------------------------------------------------------------------ vmstat -m Type InUse MemUse HighUse Requests Size(s) ctlmem 9179 18345K - 9179 64,2048 cdev 10 2K - 10 128 CAM XPT 25 2K - 85 16,32,128,256,512,1024,2048 ctlblk 200 800K - 200 4096 filedesc 49 52K - 1797 16,1024,4096 sigio 1 1K - 1 32 filecaps 1 1K - 3 16 kdtrace 125 23K - 1873 64,256 kenv 82 7K - 93 16,32,64,128,4096 kqueue 0 0K - 12 128,1024 proc-args 24 2K - 610 16,32,64,128,256 ramdisk 1 4096K - 1 hhook 2 1K - 2 128 ithread 59 5K - 59 16,64,128 mixer 1 4K - 1 4096 KTRACE 100 13K - 100 128 kbdmux 6 18K - 6 16,256,1024,2048 linker 106 4K - 108 16,32,256 lockf 17 1K - 49 32,64 loginclass 3 1K - 16 64 cache 1 1K - 1 16 devbuf 810 45406K - 1186 16,32,64,128,256,512,1024,2048,4096 temp 18 77K - 1696 16,32,64,128,256,512,1024,2048,4096 ip6ndp 5 1K - 5 64,128 module 220 14K - 220 64 mtx_pool 2 8K - 2 4096 pmchooks 1 1K - 1 64 UART 12 7K - 12 16,512,1024 CAM DEV 6 12K - 12 2048 pgrp 21 2K - 26 64 session 17 3K - 22 128 proc 2 2K - 2 1024 subproc 114 205K - 1862 256,4096 cred 46 5K - 6934 64,128 plimit 12 3K - 209 256 uidinfo 3 1K - 6 64,256 CAM CCB 0 0K - 32459 2048 CAM path 8 1K - 50 16 ctlpool 532 138K - 532 16,512 USB 6 3K - 6 16,64,256,2048 USBdev 5 1K - 5 32,64,128 sysctl 0 0K - 2612 16,32,64 sysctloid 2164 66K - 2215 16,32,64,128 sysctltmp 0 0K - 210 16,32,64,128 CAM periph 6 1K - 21 16,32,64,128 tidhash 1 2K - 1 2048 callout 2 480K - 2 umtx 180 17K - 180 64,128 p1003.1b 1 1K - 1 16 SWAP 2 277K - 2 64 bus 457 21K - 1529 16,32,64,128,256,1024 bus-sc 44 60K - 564 16,32,64,128,256,512,1024,2048,4096 CAM queue 10 2K - 30 16,256 devstat 6 13K - 6 16,4096 eventhandler 82 4K - 82 32,64,128 DEVFS3 134 17K - 147 128 kobj 117 234K - 154 2048 DEVFS1 111 28K - 117 256 Per-cpu 1 1K - 1 16 DEVFS 17 1K - 18 16,64 DEVFSP 2 1K - 2 32 rman 100 6K - 523 16,32,64 sbuf 0 0K - 646 16,32,64,128,256,512,1024,2048,4096 taskqueue 13 2K - 13 16,128 Unitno 22 2K - 320 16,64 vmem 4 38K - 11 128,512,1024,2048,4096 ioctlops 0 0K - 1297 16,32,64,128,256,512,1024 select 10 1K - 10 64 iov 0 0K - 79818 16,64,128,256 msg 4 25K - 4 1024,4096 sem 4 101K - 4 1024,4096 shm 1 12K - 1 tty 21 11K - 21 512 mbuf_tag 0 0K - 18 32 ksem 1 4K - 1 4096 shmfd 1 4K - 1 4096 soname 3 1K - 9384 16,32,128 pcb 19 91K - 54 16,64,512,1024,4096 vfscache 1 128K - 1 cl_savebuf 0 0K - 7 32 vfs_hash 1 64K - 1 vnodes 1 1K - 1 128 mount 34 2K - 143 16,32,64,128,256 vnodemarker 0 0K - 42 512 GSS-API 1 1K - 1 32 BPF 12 18K - 12 64,128,512,4096 ifnet 5 5K - 5 64,1024 ifaddr 53 10K - 54 16,32,64,128,256,512,1024,2048 ether_multi 17 1K - 24 16,32,64 clone 8 1K - 8 128 arpcom 1 1K - 1 16 lltable 12 3K - 12 256 routetbl 33 4K - 171 16,32,64,128,256 igmp 4 1K - 4 128 in_multi 2 1K - 3 128 encap_export_host 2 2K - 2 1024 sctp_a_it 0 0K - 3 16 sctp_vrf 1 1K - 1 64 sctp_ifa 4 1K - 4 128 sctp_ifn 2 1K - 2 128 sctp_iter 0 0K - 3 256 hostcache 1 16K - 1 syncache 1 44K - 1 in6_multi 15 2K - 15 16,256 mld 4 1K - 4 128 inpcbpolicy 11 1K - 179 16 ipsecpolicy 22 3K - 358 128 IpFw/IpAcct 17 31K - 28 16,32,64,128,512 NFS FHA 1 1K - 1 1024 crypto 1 1K - 1 512 rpc 2 1K - 2 128 audit_evclass 187 3K - 228 16 pagedep 2 9K - 1649 128 inodedep 25 70K - 4100 256 bmsafemap 5 5K - 1837 128,4096 newblk 24 131K - 3278 128 indirdep 0 0K - 124 64 freefrag 2 1K - 48 64 freeblks 4 1K - 717 128 freefile 6 1K - 1687 32 diradd 16 1K - 1742 64 mkdir 0 0K - 3274 64 dirrem 0 0K - 79 64 newdirblk 0 0K - 1639 32 freework 18 3K - 834 128,256 sbdep 0 0K - 16 32 savedino 8 2K - 1223 256 softdep 1 1K - 1 256 ufs_dirhash 29 10K - 89 16,32,64,512 ufs_mount 4 13K - 8 64,256,1024,4096 vm_pgdata 2 17K - 2 64 nullfs_hash 1 64K - 1 atkbddev 2 1K - 2 32 pfs_nodes 21 3K - 21 128 pfs_vncache 1 1K - 1 32 GEOM 113 21K - 505 16,32,64,128,256,512,1024 ppbusdev 3 1K - 3 128 memdesc 1 4K - 1 4096 entropy 1026 65K - 1028 16,64,4096 CAM dev queue 4 1K - 4 64 $PIR 6 1K - 6 32 envy24ht 15 194K - 15 32,64,1024 spicds 7 1K - 7 64 CAM SIM 4 1K - 4 128 agp 2 65K - 2 16 isadev 21 2K - 21 64 legacydrv 4 1K - 4 16 nexusdev 4 1K - 4 16 feeder 16 1K - 18 16,64 ------------------------------------------------------------------------ vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 256, 0, 106, 14, 106, 0, 0 UMA Zones: 408, 0, 106, 2, 106, 0, 0 UMA Slabs: 56, 0, 5122, 62, 5573, 0, 0 UMA RCntSlabs: 64, 0, 319, 59, 374, 0, 0 UMA Hash: 128, 0, 3, 28, 3, 0, 0 4 Bucket: 16, 0, 6, 246, 381, 0, 0 6 Bucket: 24, 0, 0, 0, 0, 0, 0 8 Bucket: 32, 0, 2, 124, 2021, 0, 0 12 Bucket: 48, 0, 0, 0, 0, 0, 0 16 Bucket: 64, 0, 8, 118, 48, 0, 0 32 Bucket: 128, 0, 9, 146, 2054, 0, 0 64 Bucket: 256, 0, 14, 61, 215, 0, 0 128 Bucket: 512, 0, 62, 2, 795,3846, 0 vmem btag: 28, 0, 7567, 497, 8975, 110, 0 VM OBJECT: 156, 0, 1515, 60, 18171, 0, 0 RADIX NODE: 44, 59605, 5330, 4953, 122443, 0, 0 MAP: 144, 0, 3, 81, 3, 0, 0 KMAP ENTRY: 76, 0, 5, 154, 5, 0, 0 MAP ENTRY: 76, 0, 478, 158, 40973, 0, 0 VMSPACE: 240, 0, 27, 69, 1776, 0, 0 fakepg: 68, 0, 0, 0, 0, 0, 0 mt_zone: 76, 0, 299, 72, 299, 0, 0 16: 16, 0, 2152, 368, 8788, 0, 0 32: 32, 0, 944, 442, 7929, 0, 0 64: 64, 0, 2449, 638, 87186, 0, 0 128: 128, 0, 1115, 466, 26670, 0, 0 256: 256, 0, 396, 144, 6359, 0, 0 512: 512, 0, 340, 36, 1128, 0, 0 1024: 1024, 0, 84, 32, 2220, 0, 0 2048: 2048, 0, 9321, 9, 41863, 0, 0 4096: 4096, 0, 273, 4, 2055, 0, 0 uint64 pcpu: 8, 0, 2301, 219, 2301, 0, 0 SLEEPQUEUE: 44, 0, 91, 161, 91, 0, 0 Files: 56, 0, 61, 155, 53221, 0, 0 rl_entry: 28, 0, 34, 254, 34, 0, 0 TURNSTILE: 72, 0, 91, 126, 91, 0, 0 umtx pi: 52, 0, 0, 0, 0, 0, 0 MAC labels: 20, 0, 0, 0, 0, 0, 0 PROC: 748, 0, 46, 19, 1794, 0, 0 THREAD: 760, 0, 77, 13, 77, 0, 0 cpuset: 40, 0, 45, 157, 45, 0, 0 audit_record: 1112, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 32085, 512, 137, 567, 0, 0 mbuf: 256, 32085, 1, 262, 6795, 0, 0 mbuf_cluster: 2048, 5014, 637, 1, 750, 0, 0 mbuf_jumbo_page: 4096, 2506, 0, 0, 0, 0, 0 mbuf_jumbo_9k: 9216, 2226, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 1668, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 ttyoutq: 256, 0, 64, 41, 256, 0, 0 g_bio: 152, 0, 27, 103, 129555, 0, 0 icl_conn: 88, 0, 0, 0, 0, 0, 0 icl_pdu: 48, 0, 0, 0, 0, 0, 0 ttyinq: 152, 0, 120, 10, 480, 0, 0 ata_request: 220, 0, 1, 17, 34069, 0, 0 cryptop: 60, 0, 0, 0, 0, 0, 0 cryptodesc: 56, 0, 0, 0, 0, 0, 0 cfiscsi_data_wait: 32, 0, 0, 0, 0, 0, 0 VNODE: 284, 0, 2752, 20, 4442, 0, 0 VNODEPOLL: 60, 0, 0, 0, 0, 0, 0 BUF TRIE: 44, 0, 33, 2242, 8894, 0, 0 NAMEI: 1024, 0, 0, 16, 87795, 0, 0 S VFS Cache: 72, 0, 2886, 138, 45560, 0, 0 STS VFS Cache: 92, 0, 0, 0, 0, 0, 0 L VFS Cache: 292, 0, 0, 0, 0, 0, 0 LTS VFS Cache: 312, 0, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 16, 16, 116, 0, 0 NCLNODE: 356, 0, 0, 0, 0, 0, 0 mqnode: 344, 0, 3, 8, 3, 0, 0 mqueue: 132, 0, 0, 0, 0, 0, 0 mvdata: 32, 0, 0, 0, 0, 0, 0 mqnotifier: 136, 0, 0, 0, 0, 0, 0 procdesc: 72, 0, 0, 0, 0, 0, 0 Mountpoints: 672, 0, 3, 15, 3, 0, 0 pipe: 408, 0, 1, 35, 1091, 0, 0 ksiginfo: 80, 0, 42, 158, 119, 0, 0 itimer: 232, 0, 1, 16, 1, 0, 0 KNOTE: 72, 0, 0, 0, 12, 0, 0 socket: 424, 7668, 19, 35, 1202, 0, 0 ipq: 32, 252, 0, 0, 0, 0, 0 udp_inpcb: 252, 7680, 8, 72, 160, 0, 0 udpcb: 8, 7812, 8, 244, 160, 0, 0 tcp_inpcb: 252, 7680, 2, 62, 6, 0, 0 tcpcb: 752, 7670, 2, 13, 6, 0, 0 tcptw: 60, 1541, 0, 0, 0, 0, 0 syncache: 124, 15360, 0, 0, 0, 0, 0 hostcache: 76, 15370, 0, 0, 0, 0, 0 tcpreass: 20, 404, 0, 0, 0, 0, 0 sackhole: 20, 0, 0, 0, 0, 0, 0 sctp_ep: 976, 7668, 0, 0, 0, 0, 0 sctp_asoc: 1592, 40000, 0, 0, 0, 0, 0 sctp_laddr: 24, 80136, 0, 0, 3, 0, 0 sctp_raddr: 532, 80003, 0, 0, 0, 0, 0 sctp_chunk: 96, 400008, 0, 0, 0, 0, 0 sctp_readq: 76, 400044, 0, 0, 0, 0, 0 sctp_stream_msg_out: 68, 400020, 0, 0, 0, 0, 0 sctp_asconf: 24, 400008, 0, 0, 0, 0, 0 sctp_asconf_ack: 24, 400008, 0, 0, 0, 0, 0 ripcb: 252, 7680, 1, 63, 13, 0, 0 unpcb: 172, 7682, 6, 63, 1013, 0, 0 rtentry: 108, 0, 13, 24, 14, 0, 0 IPFW dynamic rule: 108, 4107, 0, 0, 0, 0, 0 divcb: 252, 7680, 0, 0, 0, 0, 0 selfd: 28, 0, 31, 113, 13555, 0, 0 SWAPMETA: 276, 30086, 31, 39, 36, 0, 0 FFS inode: 116, 0, 2711, 111, 4398, 0, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 2711, 49, 4398, 0, 0 ------------------------------------------------------------------------ vmstat -i interrupt total rate irq0: attimer0 834262 10695 irq1: atkbd0 275 3 irq7: ppc0 1 0 irq11: pcm0 bge0 92 1 irq14: ata0 32354 414 irq15: ata1 423 5 Total 867407 11120 ------------------------------------------------------------------------ pstat -T 61/7663 files 0M/2047M swap space ------------------------------------------------------------------------ pstat -s Device 512-blocks Used Avail Capacity /dev/label/Swap 4194040 1016 4193024 0% ------------------------------------------------------------------------ iostat iostat: kvm_read(_tk_nin): invalid address (0x0) iostat: disabling TTY statistics ada0 ada1 pass0 cpu KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 29.37 407 11.69 3.50 1 0.00 0.00 0 0.00 57 2 13 0 27 ------------------------------------------------------------------------ ipcs -a Message Queues: T ID KEY MODE OWNER GROUP CREATOR CGROUP CBYTES QNUM QBYTES LSPID LRPID STIME RTIME CTIME Shared Memory: T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME Semaphores: T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME CTIME ------------------------------------------------------------------------ ipcs -T msginfo: msgmax: 16384 (max characters in a message) msgmni: 40 (# of message queues) msgmnb: 2048 (max characters in a message queue) msgtql: 40 (max # of messages in system) msgssz: 8 (size of a message segment) msgseg: 2048 (# of message segments in system) shminfo: shmmax: 536870912 (max shared memory segment size) shmmin: 1 (min shared memory segment size) shmmni: 192 (max number of shared memory identifiers) shmseg: 128 (max shared memory segments per process) shmall: 131072 (max amount of shared memory in pages) seminfo: semmni: 50 (# of semaphore identifiers) semmns: 340 (# of semaphores in system) semmnu: 150 (# of undo structures in system) semmsl: 340 (max # of semaphores per id) semopm: 100 (max # of operations per semop call) semume: 50 (max # of undo entries per process) semusz: 616 (size in bytes of undo structure) semvmx: 32767 (semaphore maximum value) semaem: 16384 (adjust on exit max value) ------------------------------------------------------------------------ nfsstat Client Info: Rpc Counts: Getattr Setattr Lookup Readlink Read Write Create Remove 0 0 0 0 0 0 0 0 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 0 0 0 0 0 Mknod Fsstat Fsinfo PathConf Commit 0 0 0 0 0 Rpc Info: TimedOut Invalid X Replies Retries Requests 0 0 0 0 0 Cache Info: Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses 0 0 0 0 0 0 0 0 BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits Misses 0 0 0 0 0 0 0 0 Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 0 0 0 0 0 0 0 0 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 0 0 0 0 0 Mknod Fsstat Fsinfo PathConf Commit 0 0 0 0 0 Server Ret-Failed 0 Server Faults 0 Server Cache Stats: Inprog Idem Non-idem Misses 0 0 0 0 Server Write Gathering: WriteOps WriteRPC Opsaved 0 0 0 ------------------------------------------------------------------------ netstat -s tcp: 0 packets sent 0 data packets (0 bytes) 0 data packets (0 bytes) retransmitted 0 data packets unnecessarily retransmitted 90916420708437821 resends initiated by MTU discovery 0 ack-only packets (0 delayed) 0 URG only packets 0 window probe packets 0 window update packets 0 control packets 0 packets received 0 acks (for 0 bytes) 0 duplicate acks 0 acks for unsent data 0 packets (0 bytes) received in-sequence 0 completely duplicate packets (0 bytes) 0 old duplicate packets 0 packets with some dup. data (0 bytes duped) 0 out-of-order packets (0 bytes) 0 packets (0 bytes) of data after window 0 window probes 0 window update packets 0 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded due to memory problems 0 connection requests 0 connection accepts 0 bad connection attempts 0 listen queue overflows 7525346126759879536 ignored RSTs in the windows 0 connections established (including accepts) 0 connections closed (including 4 drops) 0 connections updated cached RTT on close 0 connections updated cached RTT variance on close 14223757787205729232 connections updated cached ssthresh on close 0 embryonic connections dropped 0 segments updated rtt (of 0 attempts) 0 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 0 keepalive timeouts 0 keepalive probes sent 0 connections dropped by keepalive 0 correct ACK header predictions 0 correct data packet header predictions 0 syncache entries added 0 retransmitted 0 dupsyn 0 dropped 0 completed 0 bucket overflow 3231735672 cache overflow 13878490090660978560 reset 14024559434085171200 stale 0 aborted 0 badack 0 unreach 0 zone failures 0 cookies sent 0 cookies received 0 hostcache entries added 0 bucket overflow 0 SACK recovery episodes 0 segment rexmits in SACK recovery episodes 0 byte rexmits in SACK recovery episodes 8315177820320982133 SACK options (SACK blocks) received 0 SACK options (SACK blocks) sent 0 SACK scoreboard overflow 0 packets with ECN CE bit set 0 packets with ECN ECT(0) bit set 0 packets with ECN ECT(1) bit set 0 successful ECN handshakes 0 times ECN reduced the congestion window udp: 39 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 2 with no checksum 38 dropped due to no socket 0 broadcast/multicast datagrams undelivered 14202747855163872744 dropped due to full socket buffers 0 not for hashed pcb 4243996218545678873 delivered 13876054981724591340 datagrams output 90916420708441584 times multicast source filter matched ip: 40 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 13880185520411137644 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 39 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 0 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 0 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 13880189073412325376 packets sent from this host 0 packets sent with fabricated ip header 13880189107772063744 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header icmp: 0 calls to icmp_error 0 errors not generated in response to an icmp message 0 messages with bad code fields 0 messages less than the minimum length 0 messages with bad checksum 0 messages with bad length 0 multicast echo requests ignored 0 multicast timestamp requests ignored Input histogram: destination unreachable: 3311909552 source quench: 3311909568 0 message responses generated 0 invalid return addresses 0 no return routes igmp: 0 messages received 0 messages received with too few bytes 0 messages received with wrong TTL 0 messages received with bad checksum 0 V1/V2 membership queries received 0 V3 membership queries received 0 membership queries received with invalid field(s) 0 general queries received 0 group queries received 0 group-source queries received 0 group-source queries dropped 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 0 V3 reports received without Router Alert 0 membership reports sent ipsec: 0 inbound packets violated process security policy 0 inbound packets failed due to insufficient memory 0 invalid inbound packets 0 outbound packets violated process security policy 0 outbound packets with no SA available 0 outbound packets failed due to insufficient memory 0 outbound packets with no route available 0 invalid outbound packets 0 outbound packets with bundled SAs 8029953746877509737 mbufs coalesced during clone 0 clusters coalesced during clone 0 clusters copied during clone 0 mbufs inserted during makespace ah: 0 packets shorter than header shows 0 packets dropped; protocol family not supported 0 packets dropped; no TDB 0 packets dropped; bad KCR 0 packets dropped; queue full 0 packets dropped; no transform 0 replay counter wraps 0 packets dropped; bad authentication detected 0 packets dropped; bad authentication length 0 possible replay packets detected 0 packets in 0 packets out 0 packets dropped; invalid TDB 0 bytes in 0 bytes out 0 packets dropped; larger than IP_MAXPACKET 0 packets blocked due to policy 0 crypto processing failures 0 tunnel sanity check failures esp: 0 packets shorter than header shows 0 packets dropped; protocol family not supported 0 packets dropped; no TDB 0 packets dropped; bad KCR 0 packets dropped; queue full 0 packets dropped; no transform 0 packets dropped; bad ilen 0 replay counter wraps 0 packets dropped; bad encryption detected 0 packets dropped; bad authentication detected 0 possible replay packets detected 0 packets in 0 packets out 0 packets dropped; invalid TDB 0 bytes in 0 bytes out 0 packets dropped; larger than IP_MAXPACKET 0 packets blocked due to policy 0 crypto processing failures 0 tunnel sanity check failures ESP output histogram: #139: 14027524820258542848 #144: 2326202804928401481 #148: 17042560347969784149 #149: 615594868859046229 #152: 13878550752779075852 #153: 17042560347969784149 #154: 7235444513182665806 ipcomp: 0 packets shorter than header shows 0 packets dropped; protocol family not supported 0 packets dropped; no TDB 0 packets dropped; bad KCR 0 packets dropped; queue full 0 packets dropped; no transform 0 replay counter wraps 0 packets in 0 packets out 0 packets dropped; invalid TDB 0 bytes in 0 bytes out 0 packets dropped; larger than IP_MAXPACKET 0 packets blocked due to policy 0 crypto processing failures COMP output histogram: #6: 14027524820258542848 0 packets sent uncompressed; size < compr. algo. threshold 0 packets sent uncompressed; compression was useless arp: 2 ARP requests sent 4 ARP replies sent 5 ARP requests received 0 ARP replies received 0 ARP packets received 7598817722882813545 total packets dropped due to no ARP entry 0 ARP entrys timed out 0 Duplicate IPs seen ip6: 0 total packets received 0 with size smaller than minimum 0 with data size < data length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 fragments that exceeded limit 0 packets reassembled ok 0 packets for this host 0 packets forwarded 0 packets not forwardable 4 redirects sent 0 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 packets that violated scope rules 0 multicast packets which we don't join Input histogram: #184: 13880216990699749376 #185: 13880217025059487744 #187: 219269010088473984 #194: 14222497300200060400 #195: 7233175079288729193 #198: 14202724899545415680 #233: 13879875595571065172 #236: 17042560347969784149 #239: 3263241216 #240: 13880218897665228800 Mbuf statistics: 0 one mbuf two or more mbuf: (null)= 8029953746877509737 0 one ext mbuf 0 two or more ext mbuf 0 packets whose headers are not contiguous 0 tunneling packets that can't find gif 0 packets discarded because of too many headers 0 failures of source address selection Source addresses selection rule applied: icmp6: 0 calls to icmp6_error 0 errors not generated in response to an icmp6 message 0 errors not generated because of rate limitation 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Input histogram: #11: 75153822012557581 #15: 3231738164 #16: 13878518420265265468 #17: 13878517389473114436 #18: 3231738188 #19: 13878523986542881108 #227: 13880216990699749376 #228: 13880217025059487744 #237: 14222505134140714960 #240: 14202724645154207384 Histogram of error messages to be generated: 0 no route 0 administratively prohibited 0 beyond scope 0 address unreachable 0 port unreachable 0 packet too big 0 time exceed transit 0 time exceed reassembly 0 erroneous header field 0 unrecognized next header 0 unrecognized option 0 redirect 0 unknown 0 message responses generated 0 messages with too many ND options 0 messages with bad ND options 0 bad neighbor solicitation messages 0 bad neighbor advertisement messages 13878529449741283660 bad router solicitation messages 0 bad router advertisement messages 0 bad redirect messages 0 path MTU changes ipsec6: 0 inbound packets violated process security policy 0 inbound packets failed due to insufficient memory 0 invalid inbound packets 0 outbound packets violated process security policy 0 outbound packets with no SA available 0 outbound packets failed due to insufficient memory 0 outbound packets with no route available 0 invalid outbound packets 0 outbound packets with bundled SAs 0 mbufs coalesced during clone 0 clusters coalesced during clone 0 clusters copied during clone 0 mbufs inserted during makespace rip6: 0 messages received 0 checksum calculations on inbound 0 messages with bad checksum 0 messages dropped due to no socket 0 multicast messages dropped due to no socket 0 messages dropped due to full socket buffers 0 delivered 0 datagrams output pfkey: 0 requests sent from userland 0 bytes sent from userland 0 messages with invalid length field 0 messages with invalid version field 0 messages with invalid message type field 0 messages too short 0 messages with memory allocation failure 0 messages with duplicate extension 0 messages with invalid extension type 0 messages with invalid sa type 0 messages with invalid address extension 0 requests sent to userland 0 bytes sent to userland histogram by message type: #175: 7022364301717819465 #182: 17042560347969784149 #185: 13878548759914250504 #186: 17042560347969784149 0 messages toward single socket 0 messages toward all sockets 0 messages toward registered sockets 0 messages with memory allocation failure ------------------------------------------------------------------------ netstat -m 513/399/912 mbufs in use (current/cache/total) 500/138/638/5014 mbuf clusters in use (current/cache/total/max) 512/137 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/2506 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/2226 9k jumbo clusters in use (current/cache/total/max) 0/0/0/1668 16k jumbo clusters in use (current/cache/total/max) 1128K/375K/1504K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile ------------------------------------------------------------------------ netstat -idW Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop bge0 1500 00:10:18:0a:dd:27 45 0 0 41 0 0 0 bge0 1500 192.168.1.0 192.168.1.184 39 - - 38 - - - plip0* 1500 0 0 0 0 0 0 0 lo0 16384 0 0 0 0 0 0 0 lo0 16384 localhost ::1 0 - - 0 - - - lo0 16384 fe80::1%lo0 fe80::1 0 - - 0 - - - lo0 16384 your-net localhost 0 - - 0 - - - enc0* 1536 0 0 0 0 0 0 0 ------------------------------------------------------------------------ netstat -anr Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 192.168.1.1 UGS 0 32 bge0 127.0.0.1 link#3 UH 0 0 lo0 192.168.1.0/24 link#1 U 0 6 bge0 192.168.1.184 link#1 UHS 0 0 lo0 Internet6: Destination Gateway Flags Netif Expire ::/96 ::1 UGRS lo0 ::1 link#3 UH lo0 ::ffff:0.0.0.0/96 ::1 UGRS lo0 fe80::/10 ::1 UGRS lo0 fe80::%lo0/64 link#3 U lo0 fe80::1%lo0 link#3 UHS lo0 ff01::%lo0/32 ::1 U lo0 ff02::/16 ::1 UGRS lo0 ff02::%lo0/32 ::1 U lo0 ------------------------------------------------------------------------ netstat -anA Active Internet connections (including servers) Tcpcb Proto Recv-Q Send-Q Local Address Foreign Address (state) c59822f0 tcp4 0 0 *.22 *.* LISTEN c59825e0 tcp6 0 0 *.22 *.* LISTEN c57bd4ec udp4 0 0 127.0.0.1.123 *.* c57bd5e8 udp6 0 0 fe80::1%lo0.123 *.* c57bd6e4 udp6 0 0 ::1.123 *.* c57bd7e0 udp4 0 0 192.168.1.184.123 *.* c57bd8dc udp6 0 0 *.123 *.* c57bd9d8 udp4 0 0 *.123 *.* c57bdad4 udp4 0 0 *.514 *.* c57bdbd0 udp6 0 0 *.514 *.* Active UNIX domain sockets Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr c5940968 stream 0 0 c5941c34 0 0 0 /var/run/devd.pipe c594060c dgram 0 0 0 c5940810 0 c59406b8 c59406b8 dgram 0 0 0 c5940810 0 c5940764 c5940764 dgram 0 0 0 c5940810 0 0 c5940810 dgram 0 0 c5968d50 0 c594060c 0 /var/run/logpriv c59408bc dgram 0 0 c5968e6c 0 0 0 /var/run/log ------------------------------------------------------------------------ netstat -aL Current listen queue sizes (qlen/incqlen/maxqlen) Proto Listen Local Address tcp4 0/0/128 *.ssh tcp6 0/0/128 *.ssh unix 0/0/4 /var/run/devd.pipe ------------------------------------------------------------------------ fstat fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 3 at 0x1fffff fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 3 at 0x1fffff fstat: can't read file 4 at 0x4000000 fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 3 at 0x1fffff fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 3 at 0x1fffff fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 3 at 0x1fffff fstat: can't read file 4 at 0x4000000 fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 3 at 0x1fffff fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 3 at 0x1fffff fstat: can't read file 4 at 0x4000000 fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 3 at 0x1fffff fstat: can't read file 4 at 0x4000000 fstat: can't read file 6 at 0xffff fstat: can't read file 7 at 0x78 fstat: can't read file 10 at 0xffffffff fstat: can't read file 11 at 0x200007f fstat: can't read file 12 at 0x1fffff fstat: can't read file 13 at 0x4000000 fstat: can't read file 15 at 0xffff fstat: can't read file 16 at 0x78 fstat: can't read file 19 at 0xffffffff fstat: can't read file 20 at 0x200007f fstat: can't read file 21 at 0x1fffff fstat: can't read file 22 at 0x4000000 fstat: can't read file 24 at 0xffff fstat: can't read file 25 at 0x78 fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 3 at 0x1fffff fstat: can't read file 4 at 0x4000000 fstat: can't read file 6 at 0xffff fstat: can't read file 7 at 0x78 fstat: can't read file 10 at 0xffffffff fstat: can't read file 11 at 0x200007f fstat: can't read file 12 at 0x1fffff fstat: can't read file 13 at 0x4000000 fstat: can't read file 15 at 0xffff fstat: can't read file 16 at 0x78 fstat: can't read file 19 at 0xffffffff fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 3 at 0x1fffff fstat: can't read file 4 at 0x4000000 fstat: can't read file 2 at 0x2000000 fstat: can't read file 4 at 0x4000000 fstat: can't read file 1 at 0xffffffff fstat: can't read file 2 at 0x200007f fstat: can't read file 3 at 0x1fffff fstat: can't read file 4 at 0x4000000 fstat: can't read file 6 at 0xffff fstat: can't read file 7 at 0x78 USER CMD PID FD MOUNT INUM MODE SZ|DV R/W root c++ 1794 root / 2 drwxr-xr-x 512 r root c++ 1794 wd / 1532195 drwxr-xr-x 1024 r root c++ 1794 text / 323107 -r-xr-xr-x 19301616 r root c++ 1794 ctty /dev 70 crw------- ttyv1 rw root c++ 1794 0 /dev 70 crw------- ttyv1 rw root c++ 1793 root / 2 drwxr-xr-x 512 r root c++ 1793 wd / 1532195 drwxr-xr-x 1024 r root c++ 1793 text / 323107 -r-xr-xr-x 19301616 r root c++ 1793 ctty /dev 70 crw------- ttyv1 rw root c++ 1793 0 /dev 70 crw------- ttyv1 rw root make 1725 root / 2 drwxr-xr-x 512 r root make 1725 wd / 1532195 drwxr-xr-x 1024 r root make 1725 text / 321734 -r-xr-xr-x 532776 r root make 1725 ctty /dev 70 crw------- ttyv1 rw root make 1725 0 /dev 70 crw------- ttyv1 rw root sh 1623 root / 2 drwxr-xr-x 512 r root sh 1623 wd / 809467 drwxr-xr-x 512 r root sh 1623 text / 3290533 -r-xr-xr-x 103076 r root sh 1623 ctty /dev 70 crw------- ttyv1 rw root sh 1623 0 /dev 70 crw------- ttyv1 rw root make 1619 root / 2 drwxr-xr-x 512 r root make 1619 wd / 1450151 drwxr-xr-x 512 r root make 1619 text / 321734 -r-xr-xr-x 532776 r root make 1619 ctty /dev 70 crw------- ttyv1 rw root make 1619 0 /dev 70 crw------- ttyv1 rw root make 1488 root / 2 drwxr-xr-x 512 r root make 1488 wd / 321026 drwxr-xr-x 1024 r root make 1488 text / 321734 -r-xr-xr-x 532776 r root make 1488 ctty /dev 70 crw------- ttyv1 rw root make 1488 0 /dev 70 crw------- ttyv1 rw root make 1483 root / 2 drwxr-xr-x 512 r root make 1483 wd / 321026 drwxr-xr-x 1024 r root make 1483 text / 321734 -r-xr-xr-x 532776 r root make 1483 ctty /dev 70 crw------- ttyv1 rw root make 1483 0 /dev 70 crw------- ttyv1 rw root csh 1479 root / 2 drwxr-xr-x 512 r root csh 1479 wd / 321026 drwxr-xr-x 1024 r root csh 1479 text / 3290501 -r-xr-xr-x 295816 r root csh 1479 ctty /dev 70 crw------- ttyv1 rw root top 1478 root / 2 drwxr-xr-x 512 r root top 1478 wd / 642048 drwxr-xr-x 512 r root top 1478 text / 321074 -r-xr-xr-x 44520 r root top 1478 ctty /dev 71 crw------- ttyv2 rw root top 1478 0 /dev 71 crw------- ttyv2 rw root csh 1476 root / 2 drwxr-xr-x 512 r root csh 1476 wd / 642048 drwxr-xr-x 512 r root csh 1476 text / 3290501 -r-xr-xr-x 295816 r root csh 1476 ctty /dev 71 crw------- ttyv2 rw root getty 1475 root / 2 drwxr-xr-x 512 r root getty 1475 wd / 2 drwxr-xr-x 512 r root getty 1475 text / 326373 -r-xr-xr-x 20816 r root getty 1475 ctty /dev 76 crw------- ttyv7 rw root getty 1475 0 /dev 76 crw------- ttyv7 rw root getty 1474 root / 2 drwxr-xr-x 512 r root getty 1474 wd / 2 drwxr-xr-x 512 r root getty 1474 text / 326373 -r-xr-xr-x 20816 r root getty 1474 ctty /dev 75 crw------- ttyv6 rw root getty 1474 0 /dev 75 crw------- ttyv6 rw root getty 1473 root / 2 drwxr-xr-x 512 r root getty 1473 wd / 2 drwxr-xr-x 512 r root getty 1473 text / 326373 -r-xr-xr-x 20816 r root getty 1473 ctty /dev 74 crw------- ttyv5 rw root getty 1473 0 /dev 74 crw------- ttyv5 rw root getty 1472 root / 2 drwxr-xr-x 512 r root getty 1472 wd / 2 drwxr-xr-x 512 r root getty 1472 text / 326373 -r-xr-xr-x 20816 r root getty 1472 ctty /dev 73 crw------- ttyv4 rw root getty 1472 0 /dev 73 crw------- ttyv4 rw root getty 1471 root / 2 drwxr-xr-x 512 r root getty 1471 wd / 2 drwxr-xr-x 512 r root getty 1471 text / 326373 -r-xr-xr-x 20816 r root getty 1471 ctty /dev 72 crw------- ttyv3 rw root getty 1471 0 /dev 72 crw------- ttyv3 rw root login 1470 root / 2 drwxr-xr-x 512 r root login 1470 wd / 642048 drwxr-xr-x 512 r root login 1470 text / 324375 -r-sr-xr-x 20220 r root login 1470 ctty /dev 71 crw------- ttyv2 rw root login 1470 0 /dev 71 crw------- ttyv2 rw root login 1469 root / 2 drwxr-xr-x 512 r root login 1469 wd / 642048 drwxr-xr-x 512 r root login 1469 text / 324375 -r-sr-xr-x 20220 r root login 1469 ctty /dev 70 crw------- ttyv1 rw root login 1469 0 /dev 70 crw------- ttyv1 rw root getty 1468 root / 2 drwxr-xr-x 512 r root getty 1468 wd / 2 drwxr-xr-x 512 r root getty 1468 text / 326373 -r-xr-xr-x 20816 r root getty 1468 ctty /dev 69 crw------- ttyv0 rw root getty 1468 0 /dev 69 crw------- ttyv0 rw root moused 1425 root / 2 drwxr-xr-x 512 r root moused 1425 wd / 2 drwxr-xr-x 512 r root moused 1425 text / 331879 -r-xr-xr-x 33784 r root moused 1425 0 /dev 25 crw-rw-rw- null rw root cron 1402 root / 2 drwxr-xr-x 512 r root cron 1402 wd / 2327438 drwxr-x--- 512 r root cron 1402 text / 329872 -r-xr-xr-x 32236 r root cron 1402 0 /dev 25 crw-rw-rw- null rw root sshd 1398 root / 2 drwxr-xr-x 512 r root sshd 1398 wd / 2 drwxr-xr-x 512 r root sshd 1398 text / 325266 -r-xr-xr-x 239160 r root sshd 1398 0 /dev 25 crw-rw-rw- null rw root ntpd 1369 root / 2 drwxr-xr-x 512 r root ntpd 1369 wd / 2 drwxr-xr-x 512 r root ntpd 1369 text / 322091 -r-xr-xr-x 324060 r root ntpd 1369 0 /dev 25 crw-rw-rw- null rw root ntpd 1369 9 /dev 25 crw-rw-rw- null rw root ntpd 1369 18 /dev 25 crw-rw-rw- null rw root syslogd 1274 root / 2 drwxr-xr-x 512 r root syslogd 1274 wd / 2 drwxr-xr-x 512 r root syslogd 1274 text / 327708 -r-xr-xr-x 33220 r root syslogd 1274 0 /dev 25 crw-rw-rw- null rw root syslogd 1274 9 /dev 25 crw-rw-rw- null rw root syslogd 1274 18 /dev 25 crw-rw-rw- null rw root devd 1127 root / 2 drwxr-xr-x 512 r root devd 1127 wd / 2 drwxr-xr-x 512 r root devd 1127 text / 3211443 -r-xr-xr-x 813828 r root devd 1127 0 /dev 25 crw-rw-rw- null rw _dhcp dhclient 1126 root / 2327440 dr-xr-xr-x 512 r _dhcp dhclient 1126 wd / 2327440 dr-xr-xr-x 512 r _dhcp dhclient 1126 jail / 2327440 dr-xr-xr-x 512 r _dhcp dhclient 1126 text / 3211441 -r-xr-xr-x 73992 r _dhcp dhclient 1126 0 /dev 25 crw-rw-rw- null rw _dhcp dhclient 1126 9 /dev 25 crw-rw-rw- null rw root dhclient 1090 root / 2 drwxr-xr-x 512 r root dhclient 1090 wd / 2 drwxr-xr-x 512 r root dhclient 1090 text / 3211441 -r-xr-xr-x 73992 r root dhclient 1090 0 /dev 25 crw-rw-rw- null rw root init 1 root / 2 drwxr-xr-x 512 r root init 1 wd / 2 drwxr-xr-x 512 r root init 1 text / 3211467 -r-xr-xr-x 757256 r root kernel 0 root / 2 drwxr-xr-x 512 r root kernel 0 wd / 2 drwxr-xr-x 512 r ------------------------------------------------------------------------ dmesg Copyright (c) 1992-2014 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-PRERELEASE #1: Sat Jan 11 09:00:08 CET 2014 root@music2.reztek:/usr/obj/usr/src/sys/MUSIC i386 FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610 CPU: VIA Samuel 2 (797.98-MHz 686-class CPU) Origin = "CentaurHauls" Id = 0x673 Family = 0x6 Model = 0x7 Stepping = 3 Features=0x803035 AMD Features=0x80000000<3DNow!> real memory = 268435456 (256 MB) avail memory = 244457472 (233 MB) random: initialized kbd1 at kbdmux0 cryptosoft0: on motherboard apm0: on motherboard apm0: found APM BIOS v1.2, connected at v1.2 pcib0 pcibus 0 on motherboard pir0: on motherboard pci0: on pcib0 agp0: on hostb0 agp0: aperture size is 64M pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: mem 0xdf000000-0xdf7fffff,0xdfae0000-0xdfafffff,0xde800000-0xdeffffff at device 0.0 on pci1 vgapci0: Boot video device isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 7.1 on pci0 ata0: at channel 0 on atapci0 ata1: at channel 1 on atapci0 uhci0: port 0xec00-0xec1f irq 10 at device 7.3 on pci0 usbus0 on uhci0 viapropm0: port 0x400-0x40f at device 7.4 on pci0 smbus0: on viapropm0 smb0: on smbus0 bge0: mem 0xdfff0000-0xdfffffff irq 11 at device 11.0 on pci0 bge0: CHIP ID 0x00003001; ASIC REV 0x03; CHIP REV 0x30; PCI 33 MHz; 32bit miibus0: on bge0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bge0: Ethernet address: 00:10:18:0a:dd:27 pcm0: port 0xe800-0xe81f,0xe400-0xe47f irq 11 at device 12.0 on pci0 pcm0: system configuration SubVendorID: 0x1412, SubDeviceID: 0x2403 XIN2 Clock Source: 24.576MHz(96kHz*256) MPU-401 UART(s) #: not implemented ADC #: 1 and SPDIF receiver connected DAC #: 4 Multi-track converter type: AC'97(SDATA_OUT:packed) S/PDIF(IN/OUT): 1/1 ID# 0x00 GPIO(mask/dir/state): 0xff/0xff/0xff cpu0 on motherboard pmtimer0 on isa0 unknown: can't assign resources (port) unknown: can't assign resources (memory) attimer0: at port 0x40-0x43 irq 0 pnpid PNP0100 on isa0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: at port 0x70-0x71 irq 8 pnpid PNP0b00 on isa0 atkbdc0: at port 0x60,0x64 irq 1 pnpid PNP0303 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 4 flags 0x10 pnpid PNP0501 on isa0 uart1: <16550 or compatible> at port 0x2f8-0x2ff irq 3 pnpid PNP0501 on isa0 ppc0: at port 0x378-0x37f,0x778-0x77a irq 7 drq 3 pnpid PNP0401 on isa0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/8 bytes threshold ppbus0: on ppc0 plip0: on ppbus0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 uart2: <16550 or compatible> at port 0x3e8-0x3ef irq 5 pnpid PNP0501 on isa0 uart3: <16550 or compatible> at port 0x2e8-0x2ef irq 9 pnpid PNP0501 on isa0 psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse Explorer, device ID 4 orm0: at iomem 0xc0000-0xcbfff,0xcc000-0xcd7ff,0xe0000-0xe0fff pnpid ORM0000 on isa0 sc0: at flags 0x180 on isa0 sc0: VGA <16 virtual consoles, flags=0x380> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 unknown: can't assign resources (port) unknown: can't assign resources (memory) driver bug: Unable to set devclass (class: fdc devname: (unknown)) Timecounters tick every 1.000 msec IPsec: Initialized Security Association Processing. ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to deny, logging disabled random: unblocking device. usbus0: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 ada0 at ata0 bus 0 scbus0 target 0 lun 0 ada0: ATA-5 device ada0: Serial Number M0000000 ada0: 66.700MB/s transfers (UDMA4, PIO 8192bytes) ada0: 34837MB (71346240 512 byte sectors: 16H 255S/T 16383C) ada0: Previously was known as ad0 ada1 at ata1 bus 0 scbus1 target 0 lun 0 ada1: ATA-0 device ada1: Serial Number SSSI128M04Z13A22431A ada1: 16.700MB/s transfers (PIO4, PIO 512bytes) ada1: 122MB (250368 512 byte sectors: 8H 32S/T 978C) ada1: Previously was known as ad2 uhub0: 2 ports with 2 removable, self powered Trying to mount root from ufs:/dev/label/Root [rw,noatime]... WARNING: /rw/mnt was not properly dismounted Setting hostuuid: 3ca7a290-710d-11e3-a608-00a0c94b3124. Setting hostid: 0xfac708b2. Entropy harvesting: interrupts ethernet point_to_point swi. warning: total configured swap (524287 pages) exceeds maximum recommended amount (481376 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/label/Root: DEFER FOR BACKGROUND CHECKING WARNING: / was not properly dismounted Mounting local file systems:. Writing entropy file:. Setting hostname: music2.reztek. Starting Network: lo0 bge0 plip0 enc0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 bge0: flags=8843 metric 0 mtu 1500 options=8009b ether 00:10:18:0a:dd:27 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active plip0: flags=8810 metric 0 mtu 1500 nd6 options=29 enc0: flags=0<> metric 0 mtu 1536 nd6 options=29 Starting devd. Starting Network: plip0. plip0: flags=8810 metric 0 mtu 1500 nd6 options=29 Starting Network: enc0. enc0: flags=0<> metric 0 mtu 1536 nd6 options=29 Starting dhclient. DHCPREQUEST on bge0 to 255.255.255.255 port 67 DHCPACK from 192.168.1.1 bound to 192.168.1.184 -- renewal in 3600 seconds. add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Flushed all rules. 00100 allow ip from any to any via lo0 00200 deny ip from any to 127.0.0.0/8 00300 deny ip from 127.0.0.0/8 to any 00400 deny ip from any to ::1 00500 deny ip from ::1 to any 00600 allow ipv6-icmp from :: to ff02::/16 00700 allow ipv6-icmp from fe80::/10 to fe80::/10 00800 allow ipv6-icmp from fe80::/10 to ff02::/16 00900 allow ipv6-icmp from any to any ip6 icmp6types 1 01000 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136 65000 allow ip from any to any Firewall rules loaded. Creating and/or trimming log files. Starting syslogd. No core dumps found. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Clearing /tmp (X related). Updating motd:. Mounting late file systems:. Starting ntpd. Configuring syscons: keyrate blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting cron. Starting default moused. mixer: WRITE_MIXER: Device not configured mixer: WRITE_MIXER: Device not configured mixer: WRITE_MIXER: Device not configured mixer: WRITE_MIXER: Device not configured mixer: WRITE_MIXER: Device not configured mixer: WRITE_MIXER: Device not configured Starting background file system checks in 60 seconds. Sun Feb 2 19:58:36 CET 2014 Feb 2 19:58:48 music2 login: ROOT LOGIN (root) ON ttyv2 Feb 2 19:59:02 music2 login: ROOT LOGIN (root) ON ttyv1 calcru: runtime went backwards from 102407 usec to 100464 usec for pid 1479 (csh) calcru: runtime went backwards from 3221 usec to 3160 usec for pid 1479 (csh) calcru: runtime went backwards from 123173 usec to 120836 usec for pid 1476 (csh) calcru: runtime went backwards from 3236 usec to 3175 usec for pid 1476 (csh) calcru: runtime went backwards from 13832 usec to 13569 usec for pid 1475 (getty) calcru: runtime went backwards from 14486 usec to 14211 usec for pid 1474 (getty) calcru: runtime went backwards from 13986 usec to 13721 usec for pid 1473 (getty) calcru: runtime went backwards from 13541 usec to 13284 usec for pid 1472 (getty) calcru: runtime went backwards from 13637 usec to 13378 usec for pid 1471 (getty) calcru: runtime went backwards from 439925 usec to 431578 usec for pid 1470 (login) calcru: runtime went backwards from 430411 usec to 422245 usec for pid 1469 (login) calcru: runtime went backwards from 14666 usec to 14387 usec for pid 1468 (getty) calcru: runtime went backwards from 6080 usec to 5965 usec for pid 1467 (sleep) calcru: runtime went backwards from 8127 usec to 7973 usec for pid 1465 (logger) calcru: runtime went backwards from 1916 usec to 1879 usec for pid 1464 (sh) calcru: runtime went backwards from 960 usec to 941 usec for pid 1425 (moused) calcru: runtime went backwards from 35678 usec to 35001 usec for pid 1402 (cron) calcru: runtime went backwards from 129531 usec to 127073 usec for pid 1402 (cron) calcru: runtime went backwards from 4575 usec to 4488 usec for pid 1398 (sshd) calcru: runtime went backwards from 82237 usec to 81131 usec for pid 1369 (ntpd) calcru: runtime went backwards from 50080 usec to 49244 usec for pid 1274 (syslogd) calcru: runtime went backwards from 559 usec to 548 usec for pid 1127 (devd) calcru: runtime went backwards from 2008 usec to 1970 usec for pid 1126 (dhclient) calcru: runtime went backwards from 5698 usec to 5590 usec for pid 1090 (dhclient) calcru: runtime went backwards from 261424 usec to 256463 usec for pid 1090 (dhclient) calcru: runtime went backwards from 2864 usec to 2810 usec for pid 15 (usb) calcru: runtime went backwards from 53852 usec to 53166 usec for pid 4 (cam) calcru: runtime went backwards from 48080 usec to 47168 usec for pid 13 (geom) calcru: runtime went backwards from 4733641 usec to 4644716 usec for pid 12 (intr) calcru: runtime went backwards from 31166 usec to 30575 usec for pid 1 (init) calcru: runtime went backwards from 14544175 usec to 14268208 usec for pid 1 (init) calcru: runtime went backwards from 6105 usec to 5989 usec for pid 0 (kernel) calcru: runtime went backwards from 1412020 usec to 1397733 usec for pid 1642 (cc) calcru: runtime went backwards from 28811 usec to 28307 usec for pid 1641 (sh) calcru: runtime went backwards from 165082 usec to 162195 usec for pid 1628 (make) calcru: runtime went backwards from 745974 usec to 732927 usec for pid 1628 (make) calcru: runtime went backwards from 12279 usec to 12065 usec for pid 1623 (sh) calcru: runtime went backwards from 173321 usec to 170290 usec for pid 1623 (sh) calcru: runtime went backwards from 117057 usec to 115010 usec for pid 1619 (make) calcru: runtime went backwards from 74623 usec to 73318 usec for pid 1619 (make) calcru: runtime went backwards from 158886 usec to 156107 usec for pid 1488 (make) calcru: runtime went backwards from 4271263 usec to 4196562 usec for pid 1488 (make) calcru: runtime went backwards from 33800 usec to 33052 usec for pid 1483 (make) calcru: runtime went backwards from 75088 usec to 73425 usec for pid 1483 (make) calcru: runtime went backwards from 110913 usec to 108457 usec for pid 1479 (csh) calcru: runtime went backwards from 3160 usec to 3090 usec for pid 1479 (csh) calcru: runtime went backwards from 120836 usec to 118161 usec for pid 1476 (csh) calcru: runtime went backwards from 3175 usec to 3105 usec for pid 1476 (csh) calcru: runtime went backwards from 13569 usec to 13269 usec for pid 1475 (getty) calcru: runtime went backwards from 14211 usec to 13896 usec for pid 1474 (getty) calcru: runtime went backwards from 13721 usec to 13417 usec for pid 1473 (getty) calcru: runtime went backwards from 13284 usec to 12990 usec for pid 1472 (getty) calcru: runtime went backwards from 13378 usec to 13082 usec for pid 1471 (getty) calcru: runtime went backwards from 431578 usec to 422023 usec for pid 1470 (login) calcru: runtime went backwards from 422245 usec to 412896 usec for pid 1469 (login) calcru: runtime went backwards from 14387 usec to 14069 usec for pid 1468 (getty) calcru: runtime went backwards from 7973 usec to 7797 usec for pid 1465 (logger) calcru: runtime went backwards from 18177 usec to 17775 usec for pid 1464 (fsck) calcru: runtime went backwards from 18473 usec to 18064 usec for pid 1464 (fsck) calcru: runtime went backwards from 941 usec to 920 usec for pid 1425 (moused) calcru: runtime went backwards from 53539 usec to 52603 usec for pid 1402 (cron) calcru: runtime went backwards from 127073 usec to 124260 usec for pid 1402 (cron) calcru: runtime went backwards from 4488 usec to 4388 usec for pid 1398 (sshd) calcru: runtime went backwards from 94122 usec to 92825 usec for pid 1369 (ntpd) calcru: runtime went backwards from 53106 usec to 51930 usec for pid 1274 (syslogd) calcru: runtime went backwards from 598 usec to 585 usec for pid 1127 (devd) calcru: runtime went backwards from 1970 usec to 1926 usec for pid 1126 (dhclient) calcru: runtime went backwards from 5590 usec to 5466 usec for pid 1090 (dhclient) calcru: runtime went backwards from 256463 usec to 250785 usec for pid 1090 (dhclient) calcru: runtime went backwards from 7718 usec to 7639 usec for pid 19 (softdepflush) calcru: runtime went backwards from 38789 usec to 38221 usec for pid 17 (syncer) calcru: runtime went backwards from 358656 usec to 352431 usec for pid 16 (bufdaemon) calcru: runtime went backwards from 4154 usec to 4081 usec for pid 15 (usb) calcru: runtime went backwards from 47168 usec to 46123 usec for pid 13 (geom) calcru: runtime went backwards from 5887433 usec to 5826978 usec for pid 12 (intr) calcru: runtime went backwards from 92330108 usec to 91081619 usec for pid 11 (idle) calcru: runtime went backwards from 30575 usec to 29898 usec for pid 1 (init) calcru: runtime went backwards from 14268208 usec to 13952316 usec for pid 1 (init) calcru: runtime went backwards from 7442 usec to 7312 usec for pid 0 (kernel) panic: handle_written_inodeblock: Invalid link count 65529 for inodedep 0xc7648b00 KDB: stack backtrace: #0 0xc0667f6f at kdb_backtrace+0x52 #1 0xc063dc7d at panic+0x85 #2 0xc084483f at softdep_disk_write_complete+0x1294 #3 0xc06b0656 at bufdone_finish+0x23 #4 0xc06b04f5 at bufdone+0x39 #5 0xc05e0d13 at g_vfs_done+0x289 #6 0xc05dd8ea at g_io_deliver+0x28f #7 0xc05e0648 at g_std_done+0x4e #8 0xc05dd8ea at g_io_deliver+0x28f #9 0xc05e0648 at g_std_done+0x4e #10 0xc05dd8ea at g_io_deliver+0x28f #11 0xc05e0648 at g_std_done+0x4e #12 0xc05dd8ea at g_io_deliver+0x28f #13 0xc05db68f at g_disk_done_single+0xfa #14 0xc0487fb7 at adadone+0x482 #15 0xc046de75 at xpt_done_process+0x435 #16 0xc0470a11 at xpt_done_td+0x15a #17 0xc0615573 at fork_exit+0xa3 Uptime: 6m58s Physical memory: 239 MB Dumping 102 MB: 87 71 55 39 23 7 ------------------------------------------------------------------------ kernel config options CONFIG_AUTOGENERATED ident MUSIC machine i386 cpu I686_CPU cpu I586_CPU makeoptions WITH_CTF=1 makeoptions DEBUG=-g options SC_TWOBUTTON_MOUSE options SC_HISTORY_SIZE=5000 options SC_PIXEL_MODE options X86BIOS options VESA options ATA_STATIC_ID options DDB_CTF options KDB_TRACE options KDB options INCLUDE_CONFIG_FILE options KDTRACE_HOOKS options MAC options PROCDESC options CAPABILITIES options CAPABILITY_MODE options AUDIT options HWPMC_HOOKS options KBD_INSTALL_CDEV options PRINTF_BUFR_SIZE=128 options P1003_1B_MQUEUE options P1003_1B_SEMAPHORES options _KPOSIX_PRIORITY_SCHEDULING options SYSVSEM options SYSVMSG options SYSVSHM options STACK options KTRACE options SCSI_DELAY=5000 options COMPAT_FREEBSD7 options COMPAT_FREEBSD6 options COMPAT_FREEBSD5 options COMPAT_FREEBSD4 options GEOM_LABEL options GEOM_PART_GPT options TMPFS options PSEUDOFS options PROCFS options NULLFS options FDESCFS options CD9660 options MSDOSFS options NFS_ROOT options NFSLOCKD options NFSD options NFSCL options KGSSAPI options MD_ROOT options UFS_GJOURNAL options UFS_DIRHASH options UFS_ACL options SOFTUPDATES options FFS options TCP_SIGNATURE options IPSEC_NAT_T options IPSEC options LIBALIAS options IPDIVERT options IPFIREWALL_NAT options IPFIREWALL options SCTP options TCP_OFFLOAD options INET6 options INET options PREEMPTION options SCHED_4BSD options NEW_PCIB options NATIVE options GEOM_PART_MBR options GEOM_PART_EBR_COMPAT options GEOM_PART_EBR options GEOM_PART_BSD options ISAPNP device isa device npx device mem device io device uart_ns8250 device atpic device pci device agp device fdc device ata device scbus device da device sa device cd device pass device ses device ctl device atkbdc device atkbd device kbdmux device psm device vga device splash device sc device apm device pmtimer device smbus device viapm device smb device iicbus device iicbb device iicsmb device uart device ppc device ppbus device lpt device plip device ppi device mii device brgphy device bge device loop device random device ether device vlan device enc device tun device gif device faith device crypto device pty device md device firmware device bpf device uhci device usb device uhid device ukbd device umass device ums device sound device snd_envy24ht device snd_spicds ------------------------------------------------------------------------ ddb capture buffer ddb: ddb_capture: kvm_nlist