From owner-freebsd-stable@FreeBSD.ORG Sat Apr 13 23:50:34 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 135DE5DE for ; Sat, 13 Apr 2013 23:50:34 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from qmta15.emeryville.ca.mail.comcast.net (qmta15.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:44:76:96:27:228]) by mx1.freebsd.org (Postfix) with ESMTP id E99E1180D for ; Sat, 13 Apr 2013 23:50:33 +0000 (UTC) Received: from omta10.emeryville.ca.mail.comcast.net ([76.96.30.28]) by qmta15.emeryville.ca.mail.comcast.net with comcast id PbDd1l0010cQ2SLAFbqY7E; Sat, 13 Apr 2013 23:50:32 +0000 Received: from koitsu.strangled.net ([67.180.84.87]) by omta10.emeryville.ca.mail.comcast.net with comcast id PbqX1l00c1t3BNj8WbqX3N; Sat, 13 Apr 2013 23:50:32 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 8C64273A33; Sat, 13 Apr 2013 16:50:31 -0700 (PDT) Date: Sat, 13 Apr 2013 16:50:31 -0700 From: Jeremy Chadwick To: Chris Forgeron Subject: Re: kern/165903: mbuf leak Message-ID: <20130413235031.GA8212@icarus.home.lan> References: <46D80686C389884BB0C047851038EC456D8BCEBC@AA-EX0.acsi.ca> <20130410235347.GA38492@icarus.home.lan> <20130411000818.GA38803@icarus.home.lan> <46D80686C389884BB0C047851038EC456D8C0EF0@AA-EX0.acsi.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <46D80686C389884BB0C047851038EC456D8C0EF0@AA-EX0.acsi.ca> User-Agent: Mutt/1.5.21 (2010-09-15) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20121106; t=1365897032; bh=6W4AaPhYpMDKa12/SHn2zr++AuWbqHtf4gVia/Q24nw=; h=Received:Received:Received:Date:From:To:Subject:Message-ID: MIME-Version:Content-Type; b=LCx7ka+iA3PobdBJiNvnaNlh7bK3Jntb27VvUHgWNnXip6WwgzNiebkJwFP+bAPbg oi5oxFtfRm5vITrVPAn6gCocPqL7/IaYtn4y/IX0AM4shXwOHvZYhTPIkutcu85EQK E0JL85otVZ2e0SNWymX3SR84xXZE+PhUr7vmnSsui0ygwQ+sdpalqRdr2pLaHEmpf4 su8OTjiWiAsZMTT1EZe9DSEl4VbZXiUi9x9GxnIWZkEpEsuKvEB/ehjG4W5iYlrTQO wx+jpI3ld8Sk9vndZIH/GcSdxboPoCB6m9D3r8b5sEy3Ihu9ShtWzAvT0m+eS7SNwm iDHTA45PHSWCw== Cc: "freebsd-stable@freebsd.org" , Jack Vogel , John Baldwin X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Apr 2013 23:50:34 -0000 On Sat, Apr 13, 2013 at 11:02:17PM +0000, Chris Forgeron wrote: > To follow-up: > > I installed 9.1-RELEASE last night, then went and cvsup'ed to the latest that RELENG_9 would give me. > > Everything is completely stock, I have not modified any config files other than filling out the setup questions (hostname, em0 set to DHCP, added a base user so I can ssh to it) > > Mu current uname -a is: FreeBSD test 9.1-STABLE FreeBSD 9.1-STABLE #0: Sat Apr 13 00:29:24 ADT 2013 root@test:/usr/obj/usr/src/sys/GENERIC amd64 > > I am still having the same mbuf drain problem, that doesn't exist on 9.0-STABLE of July 2012. > > It's only been 2 hours since my last reboot, and you can already see the drain starting. I expect this box will be unresponsive on the network sometime tomorrow. > > All that has been running is a ssh connection that wasn't moving any traffic. > > > Here is a dump of the requested logs. Please note that I give the stats at boot, then ~2 hours later. > > Please let me know how I can help this move forward. > > ------------------------------------------------- > After Boot > > > Sat Apr 13 17:41:04 ADT 2013 > root@test:/usr/home/aatech # vmstat -z > ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP > > UMA Kegs: 208, 0, 89, 13, 89, 0, 0 > UMA Zones: 640, 0, 89, 1, 89, 0, 0 > UMA Slabs: 568, 0, 3817, 5, 6208, 0, 0 > UMA RCntSlabs: 568, 0, 273, 0, 273, 0, 0 > UMA Hash: 256, 0, 3, 12, 4, 0, 0 > 16 Bucket: 152, 0, 64, 11, 64, 0, 0 > 32 Bucket: 280, 0, 51, 5, 51, 0, 0 > 64 Bucket: 536, 0, 34, 1, 34, 62, 0 > 128 Bucket: 1048, 0, 41, 1, 41, 676, 0 > VM OBJECT: 232, 0, 977, 31, 10100, 0, 0 > MAP: 232, 0, 7, 25, 7, 0, 0 > KMAP ENTRY: 120, 150257, 33, 246, 8450, 0, 0 > MAP ENTRY: 120, 0, 753, 115, 23051, 0, 0 > fakepg: 120, 0, 0, 0, 0, 0, 0 > mt_zone: 4112, 0, 323, 4, 323, 0, 0 > 16: 16, 0, 2486, 202, 38914, 0, 0 > 32: 32, 0, 2990, 444, 16915, 0, 0 > 64: 64, 0, 7243, 261, 37658, 0, 0 > 128: 128, 0, 8311, 186, 12364, 0, 0 > 256: 256, 0, 465, 75, 3012, 0, 0 > 512: 512, 0, 613, 66, 3743, 0, 0 > 1024: 1024, 0, 34, 162, 4785, 0, 0 > 2048: 2048, 0, 5129, 141, 5692, 0, 0 > 4096: 4096, 0, 421, 34, 6083, 0, 0 > Files: 80, 0, 78, 102, 3712, 0, 0 > rl_entry: 40, 0, 29, 223, 29, 0, 0 > TURNSTILE: 136, 0, 115, 45, 115, 0, 0 > umtx pi: 96, 0, 0, 0, 0, 0, 0 > MAC labels: 40, 0, 0, 0, 0, 0, 0 > PROC: 1184, 0, 44, 28, 893, 0, 0 > THREAD: 1160, 0, 98, 16, 98, 0, 0 > SLEEPQUEUE: 80, 0, 115, 30, 115, 0, 0 > VMSPACE: 392, 0, 26, 24, 876, 0, 0 > cpuset: 72, 0, 51, 49, 51, 0, 0 > audit_record: 960, 0, 0, 0, 0, 0, 0 > mbuf_packet: 256, 0, 288, 224, 1462, 0, 0 > mbuf: 256, 0, 2, 266, 380, 0, 0 > mbuf_cluster: 2048, 25600, 512, 16, 512, 0, 0 > mbuf_jumbo_page: 4096, 12800, 0, 9, 6, 0, 0 > mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0 > mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0 > mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 > g_bio: 232, 0, 0, 144, 3498, 0, 0 > ttyinq: 160, 0, 180, 60, 765, 0, 0 > ttyoutq: 256, 0, 95, 25, 407, 0, 0 > ata_request: 328, 0, 0, 48, 86, 0, 0 > ata_composite: 336, 0, 0, 0, 0, 0, 0 > vtnet_tx_hdr: 24, 0, 0, 0, 0, 0, 0 > FPU_save_area: 832, 0, 0, 0, 0, 0, 0 > VNODE: 504, 0, 490, 30, 516, 0, 0 > VNODEPOLL: 112, 0, 0, 0, 0, 0, 0 > NAMEI: 1024, 0, 0, 36, 7095, 0, 0 > S VFS Cache: 108, 0, 475, 53, 1000, 0, 0 > STS VFS Cache: 148, 0, 0, 0, 0, 0, 0 > L VFS Cache: 328, 0, 0, 0, 0, 0, 0 > LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0 > NCLNODE: 568, 0, 0, 0, 0, 0, 0 > DIRHASH: 1024, 0, 23, 13, 23, 0, 0 > pipe: 728, 0, 2, 28, 460, 0, 0 > Mountpoints: 824, 0, 2, 6, 2, 0, 0 > ksiginfo: 112, 0, 57, 999, 63, 0, 0 > itimer: 344, 0, 1, 21, 1, 0, 0 > KNOTE: 128, 0, 0, 87, 40, 0, 0 > socket: 680, 25602, 23, 13, 283, 0, 0 > ipq: 56, 819, 0, 0, 0, 0, 0 > udp_inpcb: 392, 25600, 8, 22, 222, 0, 0 > udpcb: 16, 25704, 8, 328, 222, 0, 0 > tcp_inpcb: 392, 25600, 4, 26, 10, 0, 0 > tcpcb: 976, 25600, 4, 8, 10, 0, 0 > tcptw: 72, 5150, 0, 0, 0, 0, 0 > syncache: 152, 15375, 0, 50, 1, 0, 0 > hostcache: 136, 15372, 0, 0, 0, 0, 0 > tcpreass: 40, 1680, 0, 0, 0, 0, 0 > sackhole: 32, 0, 0, 0, 0, 0, 0 > sctp_ep: 1376, 25600, 0, 0, 0, 0, 0 > sctp_asoc: 2288, 40000, 0, 0, 0, 0, 0 > sctp_laddr: 48, 80064, 0, 216, 3, 0, 0 > sctp_raddr: 704, 80000, 0, 0, 0, 0, 0 > sctp_chunk: 136, 400008, 0, 0, 0, 0, 0 > sctp_readq: 104, 400032, 0, 0, 0, 0, 0 > sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0 > sctp_asconf: 40, 400008, 0, 0, 0, 0, 0 > sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0 > ripcb: 392, 25600, 1, 19, 1, 0, 0 > unpcb: 240, 25600, 8, 40, 40, 0, 0 > rtentry: 200, 0, 13, 44, 14, 0, 0 > selfd: 56, 0, 38, 151, 2435, 0, 0 > SWAPMETA: 288, 502372, 0, 0, 0, 0, 0 > FFS inode: 168, 0, 453, 53, 477, 0, 0 > FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 > FFS2 dinode: 256, 0, 453, 27, 477, 0, 0 > > root@test:/usr/home/aatech # netstat -Q > Configuration: > Setting Current Limit > Thread count 1 1 > Default queue limit 256 10240 > Dispatch policy direct n/a > Threads bound to CPUs disabled n/a > > Protocols: > Name Proto QLimit Policy Dispatch Flags > ip 1 256 flow default --- > igmp 2 256 source default --- > rtsock 3 256 source default --- > arp 7 256 source default --- > ether 9 256 source direct --- > ip6 10 256 flow default --- > > Workstreams: > WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled > 0 0 ip 0 0 6 0 0 0 6 > 0 0 igmp 0 0 0 0 0 0 0 > 0 0 rtsock 0 2 0 0 0 16 16 > 0 0 arp 0 0 1 0 0 0 1 > 0 0 ether 0 0 7 0 0 0 7 > 0 0 ip6 0 0 0 0 0 0 0 > root@test:/usr/home/aatech # netstat -n -x > Active Internet connections > Proto Recv-Q Send-Q Local Address Foreign Address R-MBUF S-MBUF R-CLUS S-CLUS R-HIWA S-HIWA R-LOWA S-LOWA > > R-BCNT S-BCNT R-BMAX S-BMAX rexmt persist keep 2msl delack rcvtime > tcp4 0 52 192.168.9.224.22 192.168.9.108.61563 0 1 0 0 65700 33580 1 2048 > > 0 256 525600 268640 0.37 0.00 7199.90 0.00 0.00 0.10 > udp4 0 0 127.0.0.1.123 *.* 0 0 0 0 42080 9216 1 2048 > > 0 0 336640 73728 > udp6 0 0 fe80:4::1.123 *.* 0 0 0 0 42080 9216 1 2048 > > 0 0 336640 73728 > udp6 0 0 ::1.123 *.* 0 0 0 0 42080 9216 1 2048 > > 0 0 336640 73728 > udp4 0 0 192.168.9.224.123 *.* 0 0 0 0 42080 9216 1 2048 > > 0 0 336640 73728 > Active UNIX domain sockets > Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr > fffffe00064ab690 stream 0 0 0 fffffe00064ab5a0 0 0 > fffffe00064ab5a0 stream 0 0 0 fffffe00064ab690 0 0 > fffffe00064a00f0 stream 0 0 fffffe00063e8bd0 0 0 0 /var/run/devd.pipe > fffffe00064aa5a0 dgram 0 0 0 fffffe00064aa960 0 0 > fffffe00064ab0f0 dgram 0 0 0 fffffe00064aa870 0 fffffe00064ab1e0 > fffffe00064ab1e0 dgram 0 0 0 fffffe00064aa870 0 0 > fffffe00064aa870 dgram 0 0 fffffe0006531bd0 0 fffffe00064ab0f0 0 /var/run/logpriv > fffffe00064aa960 dgram 0 0 fffffe0006531dc8 0 fffffe00064aa5a0 0 /var/run/log > root@test:/usr/home/aatech # netstat -m > 298/737/1035 mbufs in use (current/cache/total) > 296/360/656/25600 mbuf clusters in use (current/cache/total/max) > 296/344 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/9/9/12800 4k (page size) jumbo clusters in use (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 666K/940K/1606K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > > ---------------------------------------------------- > After ~2 hours > > root@test:/usr/home/aatech # date > Sat Apr 13 19:32:34 ADT 2013 > root@test:/usr/home/aatech # netstat -m > 1616/694/2310 mbufs in use (current/cache/total) > 1614/322/1936/25600 mbuf clusters in use (current/cache/total/max) > 1614/306 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/9/9/12800 4k (page size) jumbo clusters in use (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 3632K/853K/4485K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > root@test:/usr/home/aatech # vmstat -z > ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP > > UMA Kegs: 208, 0, 89, 13, 89, 0, 0 > UMA Zones: 640, 0, 89, 1, 89, 0, 0 > UMA Slabs: 568, 0, 3851, 6, 6244, 0, 0 > UMA RCntSlabs: 568, 0, 977, 3, 977, 0, 0 > UMA Hash: 256, 0, 3, 12, 4, 0, 0 > 16 Bucket: 152, 0, 65, 10, 65, 0, 0 > 32 Bucket: 280, 0, 51, 5, 51, 0, 0 > 64 Bucket: 536, 0, 40, 2, 40, 62, 0 > 128 Bucket: 1048, 0, 43, 2, 43, 676, 0 > VM OBJECT: 232, 0, 1001, 87, 13050, 0, 0 > MAP: 232, 0, 7, 25, 7, 0, 0 > KMAP ENTRY: 120, 150257, 33, 246, 8454, 0, 0 > MAP ENTRY: 120, 0, 753, 146, 29632, 0, 0 > fakepg: 120, 0, 0, 0, 0, 0, 0 > mt_zone: 4112, 0, 323, 4, 323, 0, 0 > 16: 16, 0, 2486, 202, 40587, 0, 0 > 32: 32, 0, 2990, 444, 17187, 0, 0 > 64: 64, 0, 7260, 300, 41779, 0, 0 > 128: 128, 0, 8367, 275, 13151, 0, 0 > 256: 256, 0, 469, 71, 4864, 0, 0 > 512: 512, 0, 637, 91, 4628, 0, 0 > 1024: 1024, 0, 34, 162, 5307, 0, 0 > 2048: 2048, 0, 5129, 141, 5694, 0, 0 > 4096: 4096, 0, 421, 66, 6369, 0, 0 > Files: 80, 0, 78, 147, 5507, 0, 0 > rl_entry: 40, 0, 43, 209, 43, 0, 0 > TURNSTILE: 136, 0, 133, 27, 133, 0, 0 > umtx pi: 96, 0, 0, 0, 0, 0, 0 > MAC labels: 40, 0, 0, 0, 0, 0, 0 > PROC: 1184, 0, 44, 43, 1158, 0, 0 > THREAD: 1160, 0, 116, 16, 116, 0, 0 > SLEEPQUEUE: 80, 0, 133, 41, 133, 0, 0 > VMSPACE: 392, 0, 26, 24, 1076, 0, 0 > cpuset: 72, 0, 51, 49, 51, 0, 0 > audit_record: 960, 0, 0, 0, 0, 0, 0 > mbuf_packet: 256, 0, 1622, 298, 31168, 0, 0 > mbuf: 256, 0, 2, 388, 5267, 0, 0 > mbuf_cluster: 2048, 25600, 1920, 16, 1920, 0, 0 > mbuf_jumbo_page: 4096, 12800, 0, 9, 6, 0, 0 > mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0 > mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0 > mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 > g_bio: 232, 0, 0, 144, 5175, 0, 0 > ttyinq: 160, 0, 180, 60, 765, 0, 0 > ttyoutq: 256, 0, 95, 25, 407, 0, 0 > ata_request: 328, 0, 0, 48, 2315, 0, 0 > ata_composite: 336, 0, 0, 0, 0, 0, 0 > vtnet_tx_hdr: 24, 0, 0, 0, 0, 0, 0 > FPU_save_area: 832, 0, 0, 0, 0, 0, 0 > VNODE: 504, 0, 497, 23, 537, 0, 0 > VNODEPOLL: 112, 0, 0, 0, 0, 0, 0 > NAMEI: 1024, 0, 0, 36, 13140, 0, 0 > S VFS Cache: 108, 0, 471, 123, 1525, 0, 0 > STS VFS Cache: 148, 0, 0, 0, 0, 0, 0 > L VFS Cache: 328, 0, 0, 0, 0, 0, 0 > LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0 > NCLNODE: 568, 0, 0, 0, 0, 0, 0 > DIRHASH: 1024, 0, 23, 13, 23, 0, 0 > pipe: 728, 0, 2, 28, 534, 0, 0 > Mountpoints: 824, 0, 2, 6, 2, 0, 0 > ksiginfo: 112, 0, 75, 981, 311, 0, 0 > itimer: 344, 0, 1, 21, 1, 0, 0 > KNOTE: 128, 0, 0, 87, 40, 0, 0 > socket: 680, 25602, 23, 25, 443, 0, 0 > ipq: 56, 819, 0, 0, 0, 0, 0 > udp_inpcb: 392, 25600, 8, 22, 336, 0, 0 > udpcb: 16, 25704, 8, 328, 336, 0, 0 > tcp_inpcb: 392, 25600, 4, 26, 14, 0, 0 > tcpcb: 976, 25600, 4, 8, 14, 0, 0 > tcptw: 72, 5150, 0, 100, 2, 0, 0 > syncache: 152, 15375, 0, 50, 3, 0, 0 > hostcache: 136, 15372, 0, 0, 0, 0, 0 > tcpreass: 40, 1680, 0, 0, 0, 0, 0 > sackhole: 32, 0, 0, 0, 0, 0, 0 > sctp_ep: 1376, 25600, 0, 0, 0, 0, 0 > sctp_asoc: 2288, 40000, 0, 0, 0, 0, 0 > sctp_laddr: 48, 80064, 0, 216, 3, 0, 0 > sctp_raddr: 704, 80000, 0, 0, 0, 0, 0 > sctp_chunk: 136, 400008, 0, 0, 0, 0, 0 > sctp_readq: 104, 400032, 0, 0, 0, 0, 0 > sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0 > sctp_asconf: 40, 400008, 0, 0, 0, 0, 0 > sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0 > ripcb: 392, 25600, 1, 19, 1, 0, 0 > unpcb: 240, 25600, 8, 40, 82, 0, 0 > rtentry: 200, 0, 13, 44, 14, 0, 0 > selfd: 56, 0, 42, 147, 64963, 0, 0 > SWAPMETA: 288, 502372, 0, 0, 0, 0, 0 > FFS inode: 168, 0, 459, 47, 496, 0, 0 > FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 > FFS2 dinode: 256, 0, 459, 21, 496, 0, 0 > > root@test:/usr/home/aatech # > root@test:/usr/home/aatech # netstat -Q > Configuration: > Setting Current Limit > Thread count 1 1 > Default queue limit 256 10240 > Dispatch policy direct n/a > Threads bound to CPUs disabled n/a > > Protocols: > Name Proto QLimit Policy Dispatch Flags > ip 1 256 flow default --- > igmp 2 256 source default --- > rtsock 3 256 source default --- > arp 7 256 source default --- > ether 9 256 source direct --- > ip6 10 256 flow default --- > > Workstreams: > WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled > 0 0 ip 0 0 166 0 0 0 166 > 0 0 igmp 0 0 0 0 0 0 0 > 0 0 rtsock 0 2 0 0 0 16 16 > 0 0 arp 0 0 63 0 0 0 63 > 0 0 ether 0 0 229 0 0 0 229 > 0 0 ip6 0 0 0 0 0 0 0 > root@test:/usr/home/aatech # > > root@test:/usr/home/aatech # netstat -n -x > Active Internet connections > Proto Recv-Q Send-Q Local Address Foreign Address R-MBUF S-MBUF R-CLUS S-CLUS R-HIWA S-HIWA R-LOWA S-LOWA > > R-BCNT S-BCNT R-BMAX S-BMAX rexmt persist keep 2msl delack rcvtime > tcp4 0 52 192.168.9.224.22 192.168.9.108.61563 0 1 0 0 65700 33580 1 2048 > > 0 256 525600 268640 0.41 0.00 7199.92 0.00 0.00 0.08 > udp4 0 0 127.0.0.1.123 *.* 0 0 0 0 42080 9216 1 2048 > > 0 0 336640 73728 > udp6 0 0 fe80:4::1.123 *.* 0 0 0 0 42080 9216 1 2048 > > 0 0 336640 73728 > udp6 0 0 ::1.123 *.* 0 0 0 0 42080 9216 1 2048 > > 0 0 336640 73728 > udp4 0 0 192.168.9.224.123 *.* 0 0 0 0 42080 9216 1 2048 > > 0 0 336640 73728 > Active UNIX domain sockets > Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr > fffffe00064ab690 stream 0 0 0 fffffe00064ab5a0 0 0 > fffffe00064ab5a0 stream 0 0 0 fffffe00064ab690 0 0 > fffffe00064a00f0 stream 0 0 fffffe00063e8bd0 0 0 0 /var/run/devd.pipe > fffffe00064aa5a0 dgram 0 0 0 fffffe00064aa960 0 0 > fffffe00064ab0f0 dgram 0 0 0 fffffe00064aa870 0 fffffe00064ab1e0 > fffffe00064ab1e0 dgram 0 0 0 fffffe00064aa870 0 0 > fffffe00064aa870 dgram 0 0 fffffe0006531bd0 0 fffffe00064ab0f0 0 /var/run/logpriv > fffffe00064aa960 dgram 0 0 fffffe0006531dc8 0 fffffe00064aa5a0 0 /var/run/log > root@test:/usr/home/aatech # Thanks for the info Chris, greatly appreciated. It looks like you're still using csup/cvsup to pull down your src bits, rather than svn. I was under the impression the CVS stuff had been turned off, or at least deprecated to the point where SVN->CVS backporting had been disabled for quite some time (since November) -- so you may still be running off of "older" drivers. Driver commits are here: http://svnweb.freebsd.org/base/stable/9/sys/dev/e1000/ Could you also provide output from these commands (in the state of having high mbuf counts): * dmesg * pciconf -lvbc (only need the lines relevant to em0) * sysctl -a dev.em.0 I've CC'd Jack Vogel of Intel (driver maintainer) and John Baldwin for possible assistance in tracking this down further. Finally, just for comparison, here are numbers from two systems I have access to -- one is bare metal, the other is a VPS running under QEMU. The former has an uptime of 8 days, the latter 2 days (I was doing world updates at that time). Bare metal box uses an 82573E (card=0x108c15d9 chip=0x108c8086 rev=0x03): 1024/1796/2820 mbufs in use (current/cache/total) 1023/1383/2406/25600 mbuf clusters in use (current/cache/total/max) 1023/769 mbuf+clusters out of packet secondary zone in use (current/cache) 0/789/789/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 2302K/6371K/8673K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP mbuf_packet: 256, 0, 1023, 769,18120075, 0, 0 mbuf: 256, 0, 1, 1027,63685407, 0, 0 mbuf_cluster: 2048, 25600, 1792, 614, 9344, 0, 0 mbuf_jumbo_page: 4096, 12800, 0, 789,12882701, 0, 0 mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 QEMU system emulates an 82540EM (card=0x11001af4 chip=0x100e8086 rev=0x03): ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP mbuf_packet: 256, 0, 256, 128, 2804208, 0, 0 mbuf: 256, 0, 2, 334,10103829, 0, 0 mbuf_cluster: 2048, 25600, 384, 6, 384, 0, 0 mbuf_jumbo_page: 4096, 12800, 0, 24, 4632, 0, 0 mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 168, 11688, 0, 0 258/462/720 mbufs in use (current/cache/total) 256/134/390/25600 mbuf clusters in use (current/cache/total/max) 256/128 mbuf+clusters out of packet secondary zone in use (current/cache) 0/24/24/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 576K/479K/1056K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 197 requests for I/O initiated by sendfile 0 calls to protocol drain routines -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |