From owner-freebsd-stable@FreeBSD.ORG Fri Mar 30 02:37:01 2007 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 65CA716A401 for ; Fri, 30 Mar 2007 02:37:01 +0000 (UTC) (envelope-from freebsd@hub.org) Received: from hub.org (hub.org [200.46.204.220]) by mx1.freebsd.org (Postfix) with ESMTP id CE5C313C44B for ; Fri, 30 Mar 2007 02:37:00 +0000 (UTC) (envelope-from freebsd@hub.org) Received: from localhost (unknown [200.46.204.187]) by hub.org (Postfix) with ESMTP id 36A6285C8F4 for ; Thu, 29 Mar 2007 23:36:54 -0300 (ADT) Received: from hub.org ([200.46.204.220]) by localhost (mx1.hub.org [200.46.204.187]) (amavisd-maia, port 10024) with ESMTP id 29186-02 for ; Thu, 29 Mar 2007 23:36:53 -0300 (ADT) Received: from ganymede.hub.org (blk-89-241-126.eastlink.ca [24.89.241.126]) by hub.org (Postfix) with ESMTP id 5714A85C8E8 for ; Thu, 29 Mar 2007 23:36:53 -0300 (ADT) Received: from localhost (localhost [127.0.0.1]) by ganymede.hub.org (Postfix) with ESMTP id ECA1F3869E for ; Thu, 29 Mar 2007 23:37:00 -0300 (ADT) Date: Thu, 29 Mar 2007 23:37:00 -0300 From: "Marc G. Fournier" To: freebsd-stable@freebsd.org Message-ID: <12F4628FC7F68F8CBC37B104@ganymede.hub.org> X-Mailer: Mulberry/4.0.7 (Linux/x86) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: More on: No buffer space available X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2007 02:37:01 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 'k, here's a bit more data, see if something jumps out at anyone ... first, a bit of a timeline ... Tonight, after ~75hrs uptime, I got an SMS, which is what generally indicates the problem has begun ... the SMS is a result of a cron job running on the machine that pings another server on the same network ... if it can't ping it, it sends me an SMS (ie. the other server is down) ... the neat thing is that I'm also sent one when it comes back up again ... normally, I get the 'down' before I get the 'up' ... when this problem starts to manifest itself, I get the 'up' message, followed a bit later by the 'down' ... the reason: since the network is down (it can't ping), it also can't send out the 'down' message, so it queues it up ... as soon as it comes back up again, mail works again, so the 'up' goes out before queue is re-processed ... That said ... When I got my first SMS, I logged in and ran the two commands Robert suggested: 142/968/1110 mbufs in use (current/cache/total) 136/552/688/25600 mbuf clusters in use (current/cache/total/max) 136/388 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 307K/1346K/1653K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/38/6656 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 396 requests for I/O initiated by sendfile 713 calls to protocol drain routines Now, just over 4 hours later (I'm just about to reboot): 421/914/1335 mbufs in use (current/cache/total) 233/617/850/25600 mbuf clusters in use (current/cache/total/max) 233/361 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 571K/1462K/2033K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/38/6656 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 406 requests for I/O initiated by sendfile 733 calls to protocol drain routines Now, the vmstat -z is a wee bit longer: ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 140, 0, 64, 8, 64, 0 UMA Zones: 480, 0, 64, 0, 64, 0 UMA Slabs: 64, 0, 3069, 1238, 128266, 0 UMA RCntSlabs: 104, 0, 344, 100, 14486, 0 UMA Hash: 128, 0, 4, 26, 7, 0 16 Bucket: 76, 0, 15, 35, 70, 0 32 Bucket: 140, 0, 22, 34, 87, 0 64 Bucket: 268, 0, 25, 59, 220, 9 128 Bucket: 524, 0, 2975, 147, 80119, 1572874 VM OBJECT: 132, 0, 74837, 29795, 94095430, 0 MAP: 192, 0, 7, 13, 7, 0 KMAP ENTRY: 68, 57344, 747, 877, 623962, 0 MAP ENTRY: 68, 0, 89827, 13941, 192935836, 0 PV ENTRY: 24, 2245615, 1255065, 247860, 3181072639, 0 DP fakepg: 72, 0, 0, 0, 0, 0 mt_zone: 1024, 0, 134, 126, 134, 0 16: 16, 0, 8483, 449, 19117061, 0 32: 32, 0, 4673, 3915, 8407030, 0 64: 64, 0, 8889, 3029, 24622501, 0 128: 128, 0, 19660, 3620, 88038870, 0 256: 256, 0, 3891, 2319, 9037915, 0 512: 512, 0, 847, 993, 1874043, 0 1024: 1024, 0, 631, 357, 2814842, 0 2048: 2048, 0, 293, 381, 812570, 0 4096: 4096, 0, 1130, 439, 4342201, 0 Files: 72, 0, 11119, 3191, 72134326, 0 PROC: 536, 0, 1073, 971, 2510920, 0 THREAD: 376, 0, 2188, 372, 352225, 0 KSEGRP: 88, 0, 2137, 383, 32084, 0 UPCALL: 44, 0, 119, 349, 30104, 0 VMSPACE: 296, 0, 1034, 812, 2510845, 0 mbuf_packet: 256, 0, 142, 382, 33140922, 0 mbuf: 256, 0, 44, 542, 65939839, 0 mbuf_cluster: 2048, 25600, 524, 164, 345305, 0 mbuf_jumbo_pagesize: 4096, 0, 0, 0, 0, 0 mbuf_jumbo_9k: 9216, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 0, 0, 0, 0, 0 ACL UMA zone: 388, 0, 0, 0, 0, 0 g_bio: 132, 0, 0, 4205, 87153652, 0 VNODE: 272, 0, 71264, 22158, 1241560352, 0 VNODEPOLL: 76, 0, 0, 100, 3, 0 S VFS Cache: 68, 0, 73121, 29135, 1248334482, 0 L VFS Cache: 291, 0, 124, 1085, 682683, 0 NAMEI: 1024, 0, 0, 304, 1434961352, 0 DIRHASH: 1024, 0, 1810, 258, 18000204, 0 PIPE: 408, 0, 1981, 602, 1091976, 0 KNOTE: 68, 0, 32, 360, 3972127, 0 socket: 356, 12331, 12271, 60, 8439626, 1141 unpcb: 144, 12339, 11561, 373, 5337418, 0 ipq: 32, 904, 0, 226, 2, 0 udpcb: 180, 12342, 74, 146, 2173707, 0 inpcb: 180, 12342, 678, 1478, 927361, 0 tcpcb: 464, 12328, 619, 717, 927361, 0 tcptw: 48, 2496, 59, 1501, 256613, 0 syncache: 100, 15366, 0, 195, 676224, 0 hostcache: 76, 15400, 512, 688, 34850, 0 tcpreass: 20, 1690, 0, 507, 53830, 0 sackhole: 20, 0, 0, 507, 20912, 0 ripcb: 180, 12342, 0, 88, 1127, 0 rtentry: 132, 0, 203, 319, 6656, 0 g_stripe_zone: 131072, 100, 0, 0, 0, 0 SWAPMETA: 276, 121576, 957, 429, 17641, 0 Mountpoints: 664, 0, 197, 19, 200, 0 FFS inode: 132, 0, 70901, 17491, 1239034732, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 70901, 7924, 1239034732, 0 If the '4 hour later' version is of any use, please ask, I did save a copy before rebooting ... Does this provide anything? Is there something else I should do/try? Thanks ... - ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email . scrappy@hub.org MSN . scrappy@hub.org Yahoo . yscrappy Skype: hub.org ICQ . 7615664 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (FreeBSD) iD8DBQFGDHfM4QvfyHIvDvMRAhZlAJ4sR9Xe3fuC5egjtt9o9dX8Ek+opACcCu3H euSZyKGB9/HVcuwilQicfMM= =bQo7 -----END PGP SIGNATURE-----