Date: Sat, 13 Apr 2013 23:02:17 +0000 From: Chris Forgeron <cforgeron@acsi.ca> To: Jeremy Chadwick <jdc@koitsu.org> Cc: "freebsd-stable@freebsd.org" <freebsd-stable@freebsd.org> Subject: RE: kern/165903: mbuf leak Message-ID: <46D80686C389884BB0C047851038EC456D8C0EF0@AA-EX0.acsi.ca> In-Reply-To: <20130411000818.GA38803@icarus.home.lan> References: <46D80686C389884BB0C047851038EC456D8BCEBC@AA-EX0.acsi.ca> <20130410235347.GA38492@icarus.home.lan> <20130411000818.GA38803@icarus.home.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
To follow-up: I installed 9.1-RELEASE last night, then went and cvsup'ed to the latest th= at RELENG_9 would give me. Everything is completely stock, I have not modified any config files other = than filling out the setup questions (hostname, em0 set to DHCP, added a ba= se user so I can ssh to it) Mu current uname -a is: FreeBSD test 9.1-STABLE FreeBSD 9.1-STABLE #0: Sat = Apr 13 00:29:24 ADT 2013 root@test:/usr/obj/usr/src/sys/GENERIC amd64 I am still having the same mbuf drain problem, that doesn't exist on 9.0-ST= ABLE of July 2012. It's only been 2 hours since my last reboot, and you can already see the dr= ain starting. I expect this box will be unresponsive on the network sometim= e tomorrow. All that has been running is a ssh connection that wasn't moving any traffi= c. Here is a dump of the requested logs. Please note that I give the stats at = boot, then ~2 hours later. Please let me know how I can help this move forward.=20 ------------------------------------------------- After Boot Sat Apr 13 17:41:04 ADT 2013 root@test:/usr/home/aatech # vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 208, 0, 89, 13, 89, 0, 0 UMA Zones: 640, 0, 89, 1, 89, 0, 0 UMA Slabs: 568, 0, 3817, 5, 6208, 0, 0 UMA RCntSlabs: 568, 0, 273, 0, 273, 0, 0 UMA Hash: 256, 0, 3, 12, 4, 0, 0 16 Bucket: 152, 0, 64, 11, 64, 0, 0 32 Bucket: 280, 0, 51, 5, 51, 0, 0 64 Bucket: 536, 0, 34, 1, 34, 62, 0 128 Bucket: 1048, 0, 41, 1, 41, 676, 0 VM OBJECT: 232, 0, 977, 31, 10100, 0, 0 MAP: 232, 0, 7, 25, 7, 0, 0 KMAP ENTRY: 120, 150257, 33, 246, 8450, 0, 0 MAP ENTRY: 120, 0, 753, 115, 23051, 0, 0 fakepg: 120, 0, 0, 0, 0, 0, 0 mt_zone: 4112, 0, 323, 4, 323, 0, 0 16: 16, 0, 2486, 202, 38914, 0, 0 32: 32, 0, 2990, 444, 16915, 0, 0 64: 64, 0, 7243, 261, 37658, 0, 0 128: 128, 0, 8311, 186, 12364, 0, 0 256: 256, 0, 465, 75, 3012, 0, 0 512: 512, 0, 613, 66, 3743, 0, 0 1024: 1024, 0, 34, 162, 4785, 0, 0 2048: 2048, 0, 5129, 141, 5692, 0, 0 4096: 4096, 0, 421, 34, 6083, 0, 0 Files: 80, 0, 78, 102, 3712, 0, 0 rl_entry: 40, 0, 29, 223, 29, 0, 0 TURNSTILE: 136, 0, 115, 45, 115, 0, 0 umtx pi: 96, 0, 0, 0, 0, 0, 0 MAC labels: 40, 0, 0, 0, 0, 0, 0 PROC: 1184, 0, 44, 28, 893, 0, 0 THREAD: 1160, 0, 98, 16, 98, 0, 0 SLEEPQUEUE: 80, 0, 115, 30, 115, 0, 0 VMSPACE: 392, 0, 26, 24, 876, 0, 0 cpuset: 72, 0, 51, 49, 51, 0, 0 audit_record: 960, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 0, 288, 224, 1462, 0, 0 mbuf: 256, 0, 2, 266, 380, 0, 0 mbuf_cluster: 2048, 25600, 512, 16, 512, 0, 0 mbuf_jumbo_page: 4096, 12800, 0, 9, 6, 0, 0 mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 g_bio: 232, 0, 0, 144, 3498, 0, 0 ttyinq: 160, 0, 180, 60, 765, 0, 0 ttyoutq: 256, 0, 95, 25, 407, 0, 0 ata_request: 328, 0, 0, 48, 86, 0, 0 ata_composite: 336, 0, 0, 0, 0, 0, 0 vtnet_tx_hdr: 24, 0, 0, 0, 0, 0, 0 FPU_save_area: 832, 0, 0, 0, 0, 0, 0 VNODE: 504, 0, 490, 30, 516, 0, 0 VNODEPOLL: 112, 0, 0, 0, 0, 0, 0 NAMEI: 1024, 0, 0, 36, 7095, 0, 0 S VFS Cache: 108, 0, 475, 53, 1000, 0, 0 STS VFS Cache: 148, 0, 0, 0, 0, 0, 0 L VFS Cache: 328, 0, 0, 0, 0, 0, 0 LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0 NCLNODE: 568, 0, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 23, 13, 23, 0, 0 pipe: 728, 0, 2, 28, 460, 0, 0 Mountpoints: 824, 0, 2, 6, 2, 0, 0 ksiginfo: 112, 0, 57, 999, 63, 0, 0 itimer: 344, 0, 1, 21, 1, 0, 0 KNOTE: 128, 0, 0, 87, 40, 0, 0 socket: 680, 25602, 23, 13, 283, 0, 0 ipq: 56, 819, 0, 0, 0, 0, 0 udp_inpcb: 392, 25600, 8, 22, 222, 0, 0 udpcb: 16, 25704, 8, 328, 222, 0, 0 tcp_inpcb: 392, 25600, 4, 26, 10, 0, 0 tcpcb: 976, 25600, 4, 8, 10, 0, 0 tcptw: 72, 5150, 0, 0, 0, 0, 0 syncache: 152, 15375, 0, 50, 1, 0, 0 hostcache: 136, 15372, 0, 0, 0, 0, 0 tcpreass: 40, 1680, 0, 0, 0, 0, 0 sackhole: 32, 0, 0, 0, 0, 0, 0 sctp_ep: 1376, 25600, 0, 0, 0, 0, 0 sctp_asoc: 2288, 40000, 0, 0, 0, 0, 0 sctp_laddr: 48, 80064, 0, 216, 3, 0, 0 sctp_raddr: 704, 80000, 0, 0, 0, 0, 0 sctp_chunk: 136, 400008, 0, 0, 0, 0, 0 sctp_readq: 104, 400032, 0, 0, 0, 0, 0 sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0 sctp_asconf: 40, 400008, 0, 0, 0, 0, 0 sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0 ripcb: 392, 25600, 1, 19, 1, 0, 0 unpcb: 240, 25600, 8, 40, 40, 0, 0 rtentry: 200, 0, 13, 44, 14, 0, 0 selfd: 56, 0, 38, 151, 2435, 0, 0 SWAPMETA: 288, 502372, 0, 0, 0, 0, 0 FFS inode: 168, 0, 453, 53, 477, 0, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 453, 27, 477, 0, 0 root@test:/usr/home/aatech # netstat -Q Configuration: Setting Current Limit Thread count 1 1 Default queue limit 256 10240 Dispatch policy direct n/a Threads bound to CPUs disabled n/a Protocols: Name Proto QLimit Policy Dispatch Flags ip 1 256 flow default --- igmp 2 256 source default --- rtsock 3 256 source default --- arp 7 256 source default --- ether 9 256 source direct --- ip6 10 256 flow default --- Workstreams: WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled 0 0 ip 0 0 6 0 0 0 6 0 0 igmp 0 0 0 0 0 0 0 0 0 rtsock 0 2 0 0 0 16 16 0 0 arp 0 0 1 0 0 0 1 0 0 ether 0 0 7 0 0 0 7 0 0 ip6 0 0 0 0 0 0 0 root@test:/usr/home/aatech # netstat -n -x Active Internet connections Proto Recv-Q Send-Q Local Address Foreign Address R-MBUF S-= MBUF R-CLUS S-CLUS R-HIWA S-HIWA R-LOWA S-LOWA=20 R-BCNT S-BCNT R-BMAX S-BMAX rexmt persist keep 2msl delack rcvtime tcp4 0 52 192.168.9.224.22 192.168.9.108.61563 0 = 1 0 0 65700 33580 1 2048=20 0 256 525600 268640 0.37 0.00 7199.90 0.00 0.00 0.10 udp4 0 0 127.0.0.1.123 *.* 0 = 0 0 0 42080 9216 1 2048=20 0 0 336640 73728 udp6 0 0 fe80:4::1.123 *.* 0 = 0 0 0 42080 9216 1 2048=20 0 0 336640 73728 udp6 0 0 ::1.123 *.* 0 = 0 0 0 42080 9216 1 2048=20 0 0 336640 73728 udp4 0 0 192.168.9.224.123 *.* 0 = 0 0 0 42080 9216 1 2048=20 0 0 336640 73728 Active UNIX domain sockets Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr fffffe00064ab690 stream 0 0 0 fffffe00064ab5a0 0 = 0 fffffe00064ab5a0 stream 0 0 0 fffffe00064ab690 0 = 0 fffffe00064a00f0 stream 0 0 fffffe00063e8bd0 0 0 = 0 /var/run/devd.pipe fffffe00064aa5a0 dgram 0 0 0 fffffe00064aa960 0 = 0 fffffe00064ab0f0 dgram 0 0 0 fffffe00064aa870 0 ff= fffe00064ab1e0 fffffe00064ab1e0 dgram 0 0 0 fffffe00064aa870 0 = 0 fffffe00064aa870 dgram 0 0 fffffe0006531bd0 0 fffffe00064= ab0f0 0 /var/run/logpriv fffffe00064aa960 dgram 0 0 fffffe0006531dc8 0 fffffe00064= aa5a0 0 /var/run/log root@test:/usr/home/aatech # netstat -m 298/737/1035 mbufs in use (current/cache/total) 296/360/656/25600 mbuf clusters in use (current/cache/total/max) 296/344 mbuf+clusters out of packet secondary zone in use (current/cache) 0/9/9/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 666K/940K/1606K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines ---------------------------------------------------- After ~2 hours root@test:/usr/home/aatech # date Sat Apr 13 19:32:34 ADT 2013 root@test:/usr/home/aatech # netstat -m 1616/694/2310 mbufs in use (current/cache/total) 1614/322/1936/25600 mbuf clusters in use (current/cache/total/max) 1614/306 mbuf+clusters out of packet secondary zone in use (current/cache) 0/9/9/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 3632K/853K/4485K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines root@test:/usr/home/aatech # vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 208, 0, 89, 13, 89, 0, 0 UMA Zones: 640, 0, 89, 1, 89, 0, 0 UMA Slabs: 568, 0, 3851, 6, 6244, 0, 0 UMA RCntSlabs: 568, 0, 977, 3, 977, 0, 0 UMA Hash: 256, 0, 3, 12, 4, 0, 0 16 Bucket: 152, 0, 65, 10, 65, 0, 0 32 Bucket: 280, 0, 51, 5, 51, 0, 0 64 Bucket: 536, 0, 40, 2, 40, 62, 0 128 Bucket: 1048, 0, 43, 2, 43, 676, 0 VM OBJECT: 232, 0, 1001, 87, 13050, 0, 0 MAP: 232, 0, 7, 25, 7, 0, 0 KMAP ENTRY: 120, 150257, 33, 246, 8454, 0, 0 MAP ENTRY: 120, 0, 753, 146, 29632, 0, 0 fakepg: 120, 0, 0, 0, 0, 0, 0 mt_zone: 4112, 0, 323, 4, 323, 0, 0 16: 16, 0, 2486, 202, 40587, 0, 0 32: 32, 0, 2990, 444, 17187, 0, 0 64: 64, 0, 7260, 300, 41779, 0, 0 128: 128, 0, 8367, 275, 13151, 0, 0 256: 256, 0, 469, 71, 4864, 0, 0 512: 512, 0, 637, 91, 4628, 0, 0 1024: 1024, 0, 34, 162, 5307, 0, 0 2048: 2048, 0, 5129, 141, 5694, 0, 0 4096: 4096, 0, 421, 66, 6369, 0, 0 Files: 80, 0, 78, 147, 5507, 0, 0 rl_entry: 40, 0, 43, 209, 43, 0, 0 TURNSTILE: 136, 0, 133, 27, 133, 0, 0 umtx pi: 96, 0, 0, 0, 0, 0, 0 MAC labels: 40, 0, 0, 0, 0, 0, 0 PROC: 1184, 0, 44, 43, 1158, 0, 0 THREAD: 1160, 0, 116, 16, 116, 0, 0 SLEEPQUEUE: 80, 0, 133, 41, 133, 0, 0 VMSPACE: 392, 0, 26, 24, 1076, 0, 0 cpuset: 72, 0, 51, 49, 51, 0, 0 audit_record: 960, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 0, 1622, 298, 31168, 0, 0 mbuf: 256, 0, 2, 388, 5267, 0, 0 mbuf_cluster: 2048, 25600, 1920, 16, 1920, 0, 0 mbuf_jumbo_page: 4096, 12800, 0, 9, 6, 0, 0 mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 g_bio: 232, 0, 0, 144, 5175, 0, 0 ttyinq: 160, 0, 180, 60, 765, 0, 0 ttyoutq: 256, 0, 95, 25, 407, 0, 0 ata_request: 328, 0, 0, 48, 2315, 0, 0 ata_composite: 336, 0, 0, 0, 0, 0, 0 vtnet_tx_hdr: 24, 0, 0, 0, 0, 0, 0 FPU_save_area: 832, 0, 0, 0, 0, 0, 0 VNODE: 504, 0, 497, 23, 537, 0, 0 VNODEPOLL: 112, 0, 0, 0, 0, 0, 0 NAMEI: 1024, 0, 0, 36, 13140, 0, 0 S VFS Cache: 108, 0, 471, 123, 1525, 0, 0 STS VFS Cache: 148, 0, 0, 0, 0, 0, 0 L VFS Cache: 328, 0, 0, 0, 0, 0, 0 LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0 NCLNODE: 568, 0, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 23, 13, 23, 0, 0 pipe: 728, 0, 2, 28, 534, 0, 0 Mountpoints: 824, 0, 2, 6, 2, 0, 0 ksiginfo: 112, 0, 75, 981, 311, 0, 0 itimer: 344, 0, 1, 21, 1, 0, 0 KNOTE: 128, 0, 0, 87, 40, 0, 0 socket: 680, 25602, 23, 25, 443, 0, 0 ipq: 56, 819, 0, 0, 0, 0, 0 udp_inpcb: 392, 25600, 8, 22, 336, 0, 0 udpcb: 16, 25704, 8, 328, 336, 0, 0 tcp_inpcb: 392, 25600, 4, 26, 14, 0, 0 tcpcb: 976, 25600, 4, 8, 14, 0, 0 tcptw: 72, 5150, 0, 100, 2, 0, 0 syncache: 152, 15375, 0, 50, 3, 0, 0 hostcache: 136, 15372, 0, 0, 0, 0, 0 tcpreass: 40, 1680, 0, 0, 0, 0, 0 sackhole: 32, 0, 0, 0, 0, 0, 0 sctp_ep: 1376, 25600, 0, 0, 0, 0, 0 sctp_asoc: 2288, 40000, 0, 0, 0, 0, 0 sctp_laddr: 48, 80064, 0, 216, 3, 0, 0 sctp_raddr: 704, 80000, 0, 0, 0, 0, 0 sctp_chunk: 136, 400008, 0, 0, 0, 0, 0 sctp_readq: 104, 400032, 0, 0, 0, 0, 0 sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0 sctp_asconf: 40, 400008, 0, 0, 0, 0, 0 sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0 ripcb: 392, 25600, 1, 19, 1, 0, 0 unpcb: 240, 25600, 8, 40, 82, 0, 0 rtentry: 200, 0, 13, 44, 14, 0, 0 selfd: 56, 0, 42, 147, 64963, 0, 0 SWAPMETA: 288, 502372, 0, 0, 0, 0, 0 FFS inode: 168, 0, 459, 47, 496, 0, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 459, 21, 496, 0, 0 root@test:/usr/home/aatech #=20 root@test:/usr/home/aatech # netstat -Q Configuration: Setting Current Limit Thread count 1 1 Default queue limit 256 10240 Dispatch policy direct n/a Threads bound to CPUs disabled n/a Protocols: Name Proto QLimit Policy Dispatch Flags ip 1 256 flow default --- igmp 2 256 source default --- rtsock 3 256 source default --- arp 7 256 source default --- ether 9 256 source direct --- ip6 10 256 flow default --- Workstreams: WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled 0 0 ip 0 0 166 0 0 0 166 0 0 igmp 0 0 0 0 0 0 0 0 0 rtsock 0 2 0 0 0 16 16 0 0 arp 0 0 63 0 0 0 63 0 0 ether 0 0 229 0 0 0 229 0 0 ip6 0 0 0 0 0 0 0 root@test:/usr/home/aatech #=20 root@test:/usr/home/aatech # netstat -n -x Active Internet connections Proto Recv-Q Send-Q Local Address Foreign Address R-MBUF S-= MBUF R-CLUS S-CLUS R-HIWA S-HIWA R-LOWA S-LOWA=20 R-BCNT S-BCNT R-BMAX S-BMAX rexmt persist keep 2msl delack rcvtime tcp4 0 52 192.168.9.224.22 192.168.9.108.61563 0 = 1 0 0 65700 33580 1 2048=20 0 256 525600 268640 0.41 0.00 7199.92 0.00 0.00 0.08 udp4 0 0 127.0.0.1.123 *.* 0 = 0 0 0 42080 9216 1 2048=20 0 0 336640 73728 udp6 0 0 fe80:4::1.123 *.* 0 = 0 0 0 42080 9216 1 2048=20 0 0 336640 73728 udp6 0 0 ::1.123 *.* 0 = 0 0 0 42080 9216 1 2048=20 0 0 336640 73728 udp4 0 0 192.168.9.224.123 *.* 0 = 0 0 0 42080 9216 1 2048=20 0 0 336640 73728 Active UNIX domain sockets Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr fffffe00064ab690 stream 0 0 0 fffffe00064ab5a0 0 = 0 fffffe00064ab5a0 stream 0 0 0 fffffe00064ab690 0 = 0 fffffe00064a00f0 stream 0 0 fffffe00063e8bd0 0 0 = 0 /var/run/devd.pipe fffffe00064aa5a0 dgram 0 0 0 fffffe00064aa960 0 = 0 fffffe00064ab0f0 dgram 0 0 0 fffffe00064aa870 0 ff= fffe00064ab1e0 fffffe00064ab1e0 dgram 0 0 0 fffffe00064aa870 0 = 0 fffffe00064aa870 dgram 0 0 fffffe0006531bd0 0 fffffe00064= ab0f0 0 /var/run/logpriv fffffe00064aa960 dgram 0 0 fffffe0006531dc8 0 fffffe00064= aa5a0 0 /var/run/log root@test:/usr/home/aatech #
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?46D80686C389884BB0C047851038EC456D8C0EF0>