From owner-freebsd-stable@FreeBSD.ORG Thu Feb 9 04:28:02 2012 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4BC411065672 for ; Thu, 9 Feb 2012 04:28:02 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norman-vivat.ru [89.250.210.68]) by mx1.freebsd.org (Postfix) with ESMTP id 578D38FC0C for ; Thu, 9 Feb 2012 04:28:00 +0000 (UTC) Received: from bsdrookie.norma.com. ([IPv6:fd00::7fc]) by elf.hq.norma.perm.ru (8.14.4/8.14.4) with ESMTP id q194RXUM031298 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Thu, 9 Feb 2012 09:27:33 +0500 (YEKT) (envelope-from emz@norma.perm.ru) Message-ID: <4F334B35.5040708@norma.perm.ru> Date: Thu, 09 Feb 2012 10:27:33 +0600 From: "Eugene M. Zheganin" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:7.0) Gecko/20111001 Thunderbird/7.0 MIME-Version: 1.0 To: freebsd-stable References: <4F30E284.8080905@norma.perm.ru> <4F310115.3070507@FreeBSD.org> <4F310C5A.6070400@norma.perm.ru> <4F310E75.7090301@FreeBSD.org> <4F3144A9.2000505@norma.perm.ru> <4F314892.50806@FreeBSD.org> <4F314B5B.100@norma.perm.ru> <4F3186C6.8000904@FreeBSD.org> <4F324F10.2060508@norma.perm.ru> <4F32DB30.6020600@FreeBSD.org> In-Reply-To: <4F32DB30.6020600@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (elf.hq.norma.perm.ru [IPv6:fd00::30a]); Thu, 09 Feb 2012 09:27:33 +0500 (YEKT) X-Spam-Status: No hits=-98.5 bayes=0.5 testhits RDNS_NONE=0.793, SPF_SOFTFAIL=0.665,USER_IN_WHITELIST=-100 autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on elf.hq.norma.perm.ru Cc: Subject: Re: zfs arc and amount of wired memory X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 04:28:02 -0000 Hi. On 09.02.2012 02:29, Andriy Gapon wrote: > on 08/02/2012 12:31 Eugene M. Zheganin said the following: >> Hi. >> >> On 08.02.2012 02:17, Andriy Gapon wrote: >>> [output snipped] >>> >>> Thank you. I don't see anything suspicious/unusual there. >>> Just case, do you have ZFS dedup enabled by a chance? >>> >>> I think that examination of vmstat -m and vmstat -z outputs may provide some >>> clues as to what got all that memory wired. >>> >> Nope, I don't have deduplication feature enabled. > OK. So, did you have a chance to inspect vmstat -m and vmstat -z? I did. I didn't understand it, but kinda 'felt the atmosphere'. It was pretty much similar to the output I supplied below. Most of the sizes were used by 'solaris' and numerous 'zio' caches. > > It could be very well possible that swap on zvol doesn't work well when the > kernel itself is starved on memory. > >> So I want to ask - how to report it and what should I include in such pr ? > I am leaving swap-on-zvol issue aside. Your original problem doesn't seem to be > ZFS-related. I suspect that you might be running into some kernel memory leak. > If you manage to reproduce the high wired value again, then vmstat -m and > vmstat -z may provide some useful information. > > In this vein, do you use any out-of-tree kernel modules? > Also, can you try to monitor your system to see when wired count grows? > Nope, I don't have any 3rd party kernel modules. Yes, I can monitor it, but I have no idea what should I exactly monitor. This system is running squid with a dozens of authentication helpers, freeradius + postgresql, sendmail and a perl squid log parser, which uses postgresql too. net/isc-dhcp, quagga, net/mpd5, a bunch of sendmail milters, net/samba35, bind. So it's some kind of a corporate production zoo. As I write this letter, the wired amount of memory increases by 70 megs. Excuse me, 80 megs now. The output I promised (if it's MORE acceptable in the form of a link to a paste site, just say it): [emz@taiga:etc/snmp]# vmstat -m Type InUse MemUse HighUse Requests Size(s) hhook 2 1K - 2 128 ithread 85 14K - 85 32,128,256 KTRACE 100 13K - 100 128 linker 280 226K - 384 16,32,64,128,256,512,1024,2048,4096 lockf 94 10K - 20264872 64,128 loginclass 3 1K - 367 64 pci_link 13 2K - 13 16,128 ip6ndp 55 5K - 78 64,128 ip6opt 23 6K - 142134 32,256 temp 146 20K - 114199 16,32,64,128,256,512,1024,2048,4096 devbuf 28285 56235K - 29225 16,32,64,128,256,512,1024,2048,4096 module 291 37K - 291 128 USBdev 39 10K - 39 64,128,512,1024 mtx_pool 2 16K - 2 USB 55 166K - 58 16,32,64,128,256,512,2048,4096 osd 22 1K - 10870 16,64 ddb_capture 1 48K - 1 subproc 831 1312K - 56233 512,4096 proc 2 16K - 2 session 66 9K - 16431 128 pgrp 73 10K - 16581 128 cred 650 102K - 818736 64,256 uidinfo 15 4K - 5420 128,2048 plimit 25 7K - 4948 256 kbdmux 8 18K - 8 16,512,1024,2048 sysctltmp 0 0K - 9741241 16,32,64,128,4096 sysctloid 4837 243K - 4950 16,32,64,128 sysctl 0 0K - 50230 16,32,64 tidhash 1 16K - 1 callout 3 1536K - 3 umtx 2712 339K - 2766 128 p1003.1b 1 1K - 1 16 SWAP 2 1097K - 2 64 bus-sc 84 686K - 2193 16,32,64,128,256,512,1024,2048,4096 bus 861 78K - 4641 16,32,64,128,256,512,1024 devstat 4 9K - 4 32,4096 eventhandler 83 7K - 83 64,128 kobj 194 776K - 231 4096 Per-cpu 1 1K - 1 32 aacbuf 241 72K - 273 64,128,512 rman 219 23K - 449 16,32,128 acpiintr 1 1K - 1 64 sbuf 1 1K - 967 16,32,64,128,256,512,1024,2048,4096 acpica 1641 174K - 50289 16,32,64,128,256,512,1024,2048,4096 DEVFS1 106 53K - 111 512 DEVFS3 261 66K - 269 256 stack 0 0K - 2 256 taskqueue 85 8K - 121 16,32,64,128,1024 Unitno 21 1K - 208557 32,64 DEVFS2 106 2K - 108 16 DEVFS_RULE 54 26K - 54 64,512 DEVFS 39 1K - 40 16,128 Witness 1 128K - 1 iov 0 0K - 12708587 16,32,64,128,256,512 select 1081 136K - 1108 128 ioctlops 0 0K - 5846280 16,32,64,128,256,512,1024,2048,4096 msg 4 30K - 4 2048,4096 sem 4 106K - 4 2048,4096 shm 11 40K - 13055 2048 tty 23 23K - 29 1024,2048 pts 5 2K - 9 256 mbuf_tag 133 13K - 425972406 32,64,128 shmfd 1 8K - 1 DEVFSP 8 1K - 8 64 pcb 618 176K - 1042950 16,32,64,128,1024,2048,4096 soname 221 27K - 70492663 16,32,128 acl 0 0K - 1575 4096 vfscache 1 2048K - 1 vfs_hash 1 1024K - 1 vnodes 6 1K - 17 64,256 vnodemarker 0 0K - 72943 512 mount 256 16K - 1498 16,32,64,128,256,512 BPF 65 74K - 66 128,512,4096 ether_multi 564 32K - 786 16,32,64 ifaddr 330 93K - 339 16,32,64,128,256,512,2048,4096 ifnet 34 67K - 37 128,256,512,2048 clone 8 32K - 8 4096 arpcom 23 1K - 23 16 gif 1 1K - 1 256 lltable 767 209K - 15188 256,512 vlan 103 8K - 465 64,128 routetbl 840 45K - 13455 32,64,128,256,512 igmp 33 9K - 34 256 CARP 32 13K - 32 64,256,1024 ipid 2 24K - 2 ip_moptions 2 1K - 2 64,256 in_multi 23 6K - 24 256 in_mfilter 1 1K - 1 1024 encap_export_host 1 1K - 1 1024 sctp_iter 0 0K - 29 256 sctp_ifn 16 2K - 16 128 sctp_ifa 44 6K - 44 128 sctp_vrf 1 1K - 1 64 sctp_a_it 0 0K - 29 16 hostcache 1 28K - 1 syncache 1 96K - 1 fragment 0 0K - 18 64,128 ip6_moptions 10 2K - 10 32,256 in6_multi 188 28K - 196 32,256 in6_mfilter 5 5K - 5 1024 mld 33 5K - 34 128 NLM 0 0K - 1 32 rpc 64 8K - 398 16,32,64,128,256,512,1024,2048 audit_evclass 179 6K - 218 32 freework 1 1K - 1 64 newblk 1 128K - 1 bmsafemap 1 8K - 1 inodedep 1 1024K - 1 pagedep 1 128K - 1 vm_pgdata 2 129K - 2 128 UMAHash 2 5K - 6 512,1024,2048,4096 NFSD srvcache 0 0K - 91 128 pfs_nodes 21 6K - 21 256 acpitask 1 2K - 1 2048 memdesc 1 4K - 1 4096 GEOM 61 12K - 297 16,32,64,128,256,512,1024,2048 atkbddev 2 1K - 2 64 ata_pci 1 1K - 1 64 CAM dev queue 3 1K - 3 128 CAM queue 11 1K - 56 16,32 acpisem 16 2K - 16 128 CAM SIM 3 1K - 3 256 scsi_cd 0 0K - 4 16 CAM periph 4 1K - 18 16,32,64,128,256 isadev 7 1K - 7 128 entropy 1024 64K - 1024 64 apmdev 1 1K - 1 128 CAM XPT 29 14K - 80 32,64,128,1024,2048 UART 3 2K - 3 16,512,1024 cdev 8 2K - 8 256 io_apic 1 2K - 1 2048 sigio 1 1K - 3 64 filedesc 544 566K - 60333 16,32,64,128,256,512,1024,2048,4096 msi 2 1K - 2 128 nexusdev 3 1K - 3 16 kenv 104 12K - 108 16,32,64,128 kqueue 24 39K - 405980 256,2048,4096 acpidev 27 2K - 27 64 proc-args 219 24K - 449753 16,32,64,128,256 solaris 427482 2357264K - 228638905 16,32,64,128,256,512,1024,2048,4096 kstat_data 4 1K - 4 64 netgraph_msg 0 0K - 17743 64,128,256,512,1024 netgraph_node 11 3K - 26 256 netgraph_hook 14 2K - 46 128 netgraph 3 1K - 9 64,256,512,1024 netgraph_sock 5 1K - 10 128 netgraph_path 0 0K - 8912 16,32 netgraph_mppc 0 0K - 3 1024 netgraph_l2tp 2 3K - 2 128,2048 netgraph_ksock 1 1K - 2 128 netgraph_iface 1 1K - 2 128 netgraph_ppp 1 12K - 2 [emz@taiga:etc/snmp]# vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 208, 0, 199, 5, 199, 0, 0 UMA Zones: 896, 0, 199, 1, 199, 0, 0 UMA Slabs: 568, 0, 38730, 827, 2045963, 0, 0 UMA RCntSlabs: 568, 0, 2417, 152, 10161, 0, 0 UMA Hash: 256, 0, 79, 11, 81, 0, 0 16 Bucket: 152, 0, 18, 132, 156, 0, 0 32 Bucket: 280, 0, 35, 119, 299, 0, 0 64 Bucket: 536, 0, 54, 72, 644, 57, 0 128 Bucket: 1048, 0, 467, 262, 112841,14481, 0 VM OBJECT: 216, 0, 39763, 2267, 1794570, 0, 0 MAP: 232, 0, 7, 25, 7, 0, 0 KMAP ENTRY: 120, 150412, 3666, 2131, 4810417, 0, 0 MAP ENTRY: 120, 0, 16746, 8457, 4147673, 0, 0 fakepg: 120, 0, 0, 0, 0, 0, 0 mt_zone: 4112, 0, 283, 77, 283, 0, 0 16: 16, 0, 65795, 565,103575464, 0, 0 32: 32, 0, 14388, 964,32512329, 0, 0 64: 64, 0, 164377, 14879,486891319, 0, 0 128: 128, 0, 17439, 2571,112770466, 0, 0 256: 256, 0, 12848, 15832,32654808, 0, 0 512: 512, 0, 149161, 4727, 6242680, 0, 0 1024: 1024, 0, 1970, 314, 151802, 0, 0 2048: 2048, 0, 1717, 349, 465184, 0, 0 4096: 4096, 0, 7621, 669, 375642, 0, 0 Files: 80, 0, 2210, 805,47573314, 0, 0 TURNSTILE: 136, 0, 1357, 83, 1384, 0, 0 umtx pi: 96, 0, 0, 0, 0, 0, 0 MAC labels: 40, 0, 0, 0, 0, 0, 0 PROC: 1160, 0, 259, 314, 55680, 0, 0 THREAD: 1112, 0, 864, 492, 53330, 0, 0 SLEEPQUEUE: 88, 0, 1357, 122, 1384, 0, 0 VMSPACE: 392, 0, 239, 681, 55662, 0, 0 cpuset: 72, 0, 2, 98, 2, 0, 0 audit_record: 960, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 0, 2107, 709,137988001, 0, 0 mbuf: 256, 0, 58204, 1065,523117709, 0, 0 mbuf_cluster: 2048, 25600, 2816, 914, 7273137, 0, 0 mbuf_jumbo_page: 4096, 12800, 9, 543, 7683953, 0, 0 mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 672, 29126, 0, 0 g_bio: 232, 0, 0, 752, 5733840, 0, 0 ttyinq: 160, 0, 240, 120, 555, 0, 0 ttyoutq: 256, 0, 127, 83, 293, 0, 0 ata_request: 328, 0, 0, 36, 14, 0, 0 ata_composite: 336, 0, 0, 0, 0, 0, 0 taskq_zone: 48, 0, 0, 792, 69760, 0, 0 VNODE: 480, 0, 37315, 2269, 429291, 0, 0 VNODEPOLL: 112, 0, 0, 0, 0, 0, 0 S VFS Cache: 108, 0, 16761, 28350, 349225, 0, 0 L VFS Cache: 328, 0, 18, 1086, 13342, 0, 0 NAMEI: 1024, 0, 0, 384,29340521, 0, 0 NCLNODE: 560, 0, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 0, 0, 0, 0, 0 zio_cache: 880, 0, 1, 1135,40366913, 0, 0 zio_link_cache: 48, 0, 0, 1440,11333105, 0, 0 zio_buf_512: 512, 0, 0, 0, 0, 0, 0 zio_data_buf_512: 512, 0, 0, 0, 0, 0, 0 zio_buf_1024: 1024, 0, 0, 0, 0, 0, 0 zio_data_buf_1024: 1024, 0, 0, 0, 0, 0, 0 zio_buf_1536: 1536, 0, 0, 0, 0, 0, 0 zio_data_buf_1536: 1536, 0, 0, 0, 0, 0, 0 zio_buf_2048: 2048, 0, 0, 0, 0, 0, 0 zio_data_buf_2048: 2048, 0, 0, 0, 0, 0, 0 zio_buf_2560: 2560, 0, 0, 0, 0, 0, 0 zio_data_buf_2560: 2560, 0, 0, 0, 0, 0, 0 zio_buf_3072: 3072, 0, 0, 0, 0, 0, 0 zio_data_buf_3072: 3072, 0, 0, 0, 0, 0, 0 zio_buf_3584: 3584, 0, 0, 0, 0, 0, 0 zio_data_buf_3584: 3584, 0, 0, 0, 0, 0, 0 zio_buf_4096: 4096, 0, 0, 0, 0, 0, 0 zio_data_buf_4096: 4096, 0, 0, 0, 0, 0, 0 zio_buf_5120: 5120, 0, 0, 0, 0, 0, 0 zio_data_buf_5120: 5120, 0, 0, 0, 0, 0, 0 zio_buf_6144: 6144, 0, 0, 0, 0, 0, 0 zio_data_buf_6144: 6144, 0, 0, 0, 0, 0, 0 zio_buf_7168: 7168, 0, 0, 0, 0, 0, 0 zio_data_buf_7168: 7168, 0, 0, 0, 0, 0, 0 zio_buf_8192: 8192, 0, 0, 0, 0, 0, 0 zio_data_buf_8192: 8192, 0, 0, 0, 0, 0, 0 zio_buf_10240: 10240, 0, 0, 0, 0, 0, 0 zio_data_buf_10240: 10240, 0, 0, 0, 0, 0, 0 zio_buf_12288: 12288, 0, 0, 0, 0, 0, 0 zio_data_buf_12288: 12288, 0, 0, 0, 0, 0, 0 zio_buf_14336: 14336, 0, 0, 0, 0, 0, 0 zio_data_buf_14336: 14336, 0, 0, 0, 0, 0, 0 zio_buf_16384: 16384, 0, 0, 0, 0, 0, 0 zio_data_buf_16384: 16384, 0, 0, 0, 0, 0, 0 zio_buf_20480: 20480, 0, 0, 0, 0, 0, 0 zio_data_buf_20480: 20480, 0, 0, 0, 0, 0, 0 zio_buf_24576: 24576, 0, 0, 0, 0, 0, 0 zio_data_buf_24576: 24576, 0, 0, 0, 0, 0, 0 zio_buf_28672: 28672, 0, 0, 0, 0, 0, 0 zio_data_buf_28672: 28672, 0, 0, 0, 0, 0, 0 zio_buf_32768: 32768, 0, 0, 0, 0, 0, 0 zio_data_buf_32768: 32768, 0, 0, 0, 0, 0, 0 zio_buf_36864: 36864, 0, 0, 0, 0, 0, 0 zio_data_buf_36864: 36864, 0, 0, 0, 0, 0, 0 zio_buf_40960: 40960, 0, 0, 0, 0, 0, 0 zio_data_buf_40960: 40960, 0, 0, 0, 0, 0, 0 zio_buf_45056: 45056, 0, 0, 0, 0, 0, 0 zio_data_buf_45056: 45056, 0, 0, 0, 0, 0, 0 zio_buf_49152: 49152, 0, 0, 0, 0, 0, 0 zio_data_buf_49152: 49152, 0, 0, 0, 0, 0, 0 zio_buf_53248: 53248, 0, 0, 0, 0, 0, 0 zio_data_buf_53248: 53248, 0, 0, 0, 0, 0, 0 zio_buf_57344: 57344, 0, 0, 0, 0, 0, 0 zio_data_buf_57344: 57344, 0, 0, 0, 0, 0, 0 zio_buf_61440: 61440, 0, 0, 0, 0, 0, 0 zio_data_buf_61440: 61440, 0, 0, 0, 0, 0, 0 zio_buf_65536: 65536, 0, 0, 0, 0, 0, 0 zio_data_buf_65536: 65536, 0, 0, 0, 0, 0, 0 zio_buf_69632: 69632, 0, 0, 0, 0, 0, 0 zio_data_buf_69632: 69632, 0, 0, 0, 0, 0, 0 zio_buf_73728: 73728, 0, 0, 0, 0, 0, 0 zio_data_buf_73728: 73728, 0, 0, 0, 0, 0, 0 zio_buf_77824: 77824, 0, 0, 0, 0, 0, 0 zio_data_buf_77824: 77824, 0, 0, 0, 0, 0, 0 zio_buf_81920: 81920, 0, 0, 0, 0, 0, 0 zio_data_buf_81920: 81920, 0, 0, 0, 0, 0, 0 zio_buf_86016: 86016, 0, 0, 0, 0, 0, 0 zio_data_buf_86016: 86016, 0, 0, 0, 0, 0, 0 zio_buf_90112: 90112, 0, 0, 0, 0, 0, 0 zio_data_buf_90112: 90112, 0, 0, 0, 0, 0, 0 zio_buf_94208: 94208, 0, 0, 0, 0, 0, 0 zio_data_buf_94208: 94208, 0, 0, 0, 0, 0, 0 zio_buf_98304: 98304, 0, 0, 0, 0, 0, 0 zio_data_buf_98304: 98304, 0, 0, 0, 0, 0, 0 zio_buf_102400: 102400, 0, 0, 0, 0, 0, 0 zio_data_buf_102400: 102400, 0, 0, 0, 0, 0, 0 zio_buf_106496: 106496, 0, 0, 0, 0, 0, 0 zio_data_buf_106496: 106496, 0, 0, 0, 0, 0, 0 zio_buf_110592: 110592, 0, 0, 0, 0, 0, 0 zio_data_buf_110592: 110592, 0, 0, 0, 0, 0, 0 zio_buf_114688: 114688, 0, 0, 0, 0, 0, 0 zio_data_buf_114688: 114688, 0, 0, 0, 0, 0, 0 zio_buf_118784: 118784, 0, 0, 0, 0, 0, 0 zio_data_buf_118784: 118784, 0, 0, 0, 0, 0, 0 zio_buf_122880: 122880, 0, 0, 0, 0, 0, 0 zio_data_buf_122880: 122880, 0, 0, 0, 0, 0, 0 zio_buf_126976: 126976, 0, 0, 0, 0, 0, 0 zio_data_buf_126976: 126976, 0, 0, 0, 0, 0, 0 zio_buf_131072: 131072, 0, 0, 0, 0, 0, 0 zio_data_buf_131072: 131072, 0, 0, 0, 0, 0, 0 sa_cache: 80, 0, 37248, 2262, 429178, 0, 0 dnode_t: 856, 0, 151801, 1919, 249658, 0, 0 dmu_buf_impl_t: 224, 0, 177337, 2404, 600337, 0, 0 arc_buf_hdr_t: 216, 0, 105487, 1253, 519289, 0, 0 arc_buf_t: 104, 0, 38391, 5205, 559830, 0, 0 zil_lwb_cache: 192, 0, 9, 611, 80899, 0, 0 zfs_znode_cache: 400, 0, 37248, 2325, 429178, 0, 0 pipe: 728, 0, 42, 328, 39330, 0, 0 Mountpoints: 768, 0, 20, 20, 20, 0, 0 ksiginfo: 112, 0, 564, 657, 29202, 0, 0 itimer: 344, 0, 1, 21, 1, 0, 0 KNOTE: 128, 0, 877, 776,90925709, 0, 0 pfsrctrpl: 152, 10000, 0, 0, 0, 0, 0 pfrulepl: 936, 0, 325, 11, 325, 0, 0 pfstatepl: 288, 10010, 3360, 2568, 1170450, 0, 0 pfstatekeypl: 288, 0, 3983, 2621, 1916481, 0, 0 pfstateitempl: 288, 0, 3983, 2439, 1439637, 0, 0 pfaltqpl: 240, 0, 0, 0, 0, 0, 0 pfpooladdrpl: 88, 0, 71, 55, 71, 0, 0 pfrktable: 1296, 1002, 6, 9, 11, 0, 0 pfrkentry: 160, 200016, 21, 27, 21, 0, 0 pffrent: 32, 5050, 0, 0, 0, 0, 0 pffrag: 80, 0, 0, 0, 0, 0, 0 pffrcache: 80, 10035, 0, 0, 0, 0, 0 pffrcent: 24, 50022, 0, 0, 0, 0, 0 pfstatescrub: 40, 0, 0, 0, 0, 0, 0 pfiaddrpl: 120, 0, 0, 0, 0, 0, 0 pfospfen: 112, 0, 700, 26, 700, 0, 0 pfosfp: 40, 0, 410, 94, 410, 0, 0 pfsync: 88, 0, 0, 0, 0, 0, 0 socket: 680, 25602, 1726, 596,25837811, 0, 0 ipq: 56, 819, 0, 126, 396, 0, 0 udp_inpcb: 392, 25600, 152, 728,20355207, 0, 0 udpcb: 16, 25704, 152, 856,20355207, 0, 0 tcp_inpcb: 392, 25600, 1782, 4968, 5284799, 0, 0 tcpcb: 976, 25600, 982, 762, 5284799, 0, 0 tcptw: 72, 5150, 800, 4350, 306578,97901, 0 syncache: 152, 15375, 0, 325, 5052872, 0, 0 hostcache: 136, 15372, 2053, 467, 7364, 0, 0 tcpreass: 40, 1680, 3, 585, 216370, 0, 0 sackhole: 32, 0, 0, 909, 55622, 0, 0 sctp_ep: 1368, 25600, 0, 0, 0, 0, 0 sctp_asoc: 2280, 40000, 0, 0, 0, 0, 0 sctp_laddr: 48, 80064, 0, 288, 43, 0, 0 sctp_raddr: 704, 80000, 0, 0, 0, 0, 0 sctp_chunk: 136, 400008, 0, 0, 0, 0, 0 sctp_readq: 104, 400032, 0, 0, 0, 0, 0 sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0 sctp_asconf: 40, 400008, 0, 0, 0, 0, 0 sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0 ripcb: 392, 25600, 6, 34, 8, 0, 0 unpcb: 240, 25600, 570, 550, 197771, 0, 0 rtentry: 200, 0, 756, 137, 1395, 0, 0 selfd: 56, 0, 1504, 701,705385407, 0, 0 SWAPMETA: 288, 116519, 4, 113, 866, 0, 0 NetGraph items: 72, 4118, 0, 812, 115049, 0, 0 NetGraph data items: 72, 522, 0, 493, 189271, 0, 0 P.S. after I pasted this, the wired memory gone down by 70 megs, the final output (I guess enough is enough can be seen here: http://dpaste.org/iA5mP/ ). Thanks. Eugene.