From owner-freebsd-stable@FreeBSD.ORG Thu Dec 16 13:11:27 2004 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2221C16A4CE for ; Thu, 16 Dec 2004 13:11:27 +0000 (GMT) Received: from brugere.aub.dk (fw.aub.dk [195.24.1.195]) by mx1.FreeBSD.org (Postfix) with ESMTP id A248543D39 for ; Thu, 16 Dec 2004 13:11:21 +0000 (GMT) (envelope-from techlists@motrix.dk) Received: by brugere.aub.dk (Postfix, from userid 1693) id BBEBFC447; Thu, 16 Dec 2004 14:11:19 +0100 (CET) Received: from glitter (jmp.aub.dk [10.1.4.50]) by brugere.aub.dk (Postfix) with ESMTP id A0DB5C2C2; Thu, 16 Dec 2004 14:11:19 +0100 (CET) From: "J. Martin Petersen" To: "'Doug White'" Date: Thu, 16 Dec 2004 14:11:19 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook, Build 11.0.6353 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 Thread-Index: AcThQGfe5eN3XEf7SqybObVkMGwJUwCLxXhA In-Reply-To: <20041213101844.S92964@carver.gumbysoft.com> Message-Id: <20041216131119.A0DB5C2C2@brugere.aub.dk> cc: freebsd-stable@freebsd.org Subject: RE: netstat fails with memory allocation error and error in kvm_read X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Dec 2004 13:11:27 -0000 > You might also log 'sysctl vm.kvm_free' and 'sysctl vm.zone'. I just got another "netstat: kvm_read: Bad address". vm.kvm_free was 675278848 two seconds earlier and a couple of seconds later (and also currently). The output from vm.zone a couple of seconds after: vm.zone: ITEM SIZE LIMIT USED FREE REQUESTS pfosfp: 28, 0, 155, 226, 1085 pfospfen: 108, 0, 293, 31, 2051 pfiaddrpl: 92, 0, 5, 79, 28 pfstatescrub: 8, 0, 26676, 28540, 37487180 pffrcent: 12, 50141, 0, 0, 0 pffrcache: 48, 10062, 0, 0, 0 pffrag: 48, 0, 0, 234, 378044 pffrent: 16, 5075, 0, 203, 934804 pfrkentry: 152, 0, 4, 48, 8 pfrktable: 248, 0, 4, 28, 12 pfpooladdrpl: 68, 0, 21, 91, 126 pfaltqpl: 128, 0, 19, 71, 133 pfstatepl: 256, 65010, 14396, 23014, 29835551 pfrulepl: 560, 0, 248, 221, 1693 pfsrctrpl: 80, 0, 0, 0, 0 FFS2 dinode: 256, 0, 32828, 427, 1154768 FFS1 dinode: 128, 0, 0, 0, 0 FFS inode: 140, 0, 32828, 352, 1154768 SWAPMETA: 276, 63672, 9, 47, 40 rtentry: 132, 0, 219, 187, 6666 unpcb: 140, 17220, 9, 47, 5217 ripcb: 180, 8140, 0, 44, 15 sackhole: 16, 0, 0, 203, 3107 tcpreass: 20, 1183, 0, 169, 134 hostcache: 88, 15400, 3, 85, 39 syncache: 108, 15372, 0, 72, 374 tcptw: 56, 1675, 2, 132, 48 tcpcb: 448, 8136, 4, 23, 489 inpcb: 180, 8140, 6, 38, 489 udpcb: 180, 8140, 11, 33, 6297398 socket: 324, 8136, 24, 36, 6303121 KNOTE: 68, 0, 0, 112, 12538429 PIPE: 384, 0, 2, 78, 251051 DIRHASH: 1024, 0, 1621, 11, 4318 NAMEI: 1024, 0, 3, 17, 131064562 L VFS Cache: 291, 0, 38, 456, 8482 S VFS Cache: 68, 0, 29954, 5550, 1214821 VNODEPOLL: 64, 0, 0, 118, 2 VNODE: 264, 0, 33205, 20, 33205 ata_request: 200, 0, 0, 38, 2322599 g_bio: 132, 0, 0, 2204, 4301499 MbufClust: 2048, 17216, 906, 124, 12121216 Mbuf: 256, 0, 907, 188, 2520398988 Packet: 256, 0, 740, 355, 2728311990 VMSPACE: 300, 0, 23, 81, 726569 UPCALL: 44, 0, 0, 0, 0 KSEGRP: 104, 0, 144, 31, 144 TID: 140, 0, 1, 53, 1 THREAD: 388, 0, 144, 6, 144 PROC: 452, 0, 73, 71, 726627 Files: 68, 0, 71, 153, 122894112 4096: 4096, 0, 129, 60, 746477 2048: 2048, 0, 130, 30, 24039 1024: 1024, 0, 23, 61, 14341894 512: 512, 0, 244, 92, 656520 256: 256, 0, 256, 1094, 7153814 128: 128, 0, 1842, 318, 15428344 64: 64, 0, 4481, 1183, 8923514 32: 32, 0, 2595, 682, 4033338850 16: 16, 0, 1930, 303, 22115093 DP fakepg: 72, 0, 0, 0, 0 PV ENTRY: 24, 944530, 5540, 17080, 315728548 MAP ENTRY: 68, 0, 439, 1241, 32683625 KMAP ENTRY: 68, 32200, 40, 464, 5444457 MAP: 192, 0, 7, 33, 5 VM OBJECT: 132, 0, 32560, 1428, 14688659 128 Bucket: 524, 0, 966, 0, 0 64 Bucket: 268, 0, 30, 40, 0 32 Bucket: 140, 0, 30, 26, 0 16 Bucket: 76, 0, 15, 35, 0 UMA Hash: 128, 0, 2, 28, 0 UMA RCntSlab: 104, 0, 515, 3, 0 UMA Slabs: 64, 0, 793, 33, 0 UMA Zones: 88, 0, 73, 7, 0 UMA Kegs: 136, 0, 73, 23, 0 vm.zone a couple of seconds before: ITEM SIZE LIMIT USED FREE REQUESTS pfosfp: 28, 0, 155, 226, 1085 pfospfen: 108, 0, 293, 31, 2051 pfiaddrpl: 92, 0, 5, 79, 28 pfstatescrub: 8, 0, 27146, 28070, 37485813 pffrcent: 12, 50141, 0, 0, 0 pffrcache: 48, 10062, 0, 0, 0 pffrag: 48, 0, 0, 234, 378044 pffrent: 16, 5075, 0, 203, 934804 pfrkentry: 152, 0, 4, 48, 8 pfrktable: 248, 0, 4, 28, 12 pfpooladdrpl: 68, 0, 21, 91, 126 pfaltqpl: 128, 0, 19, 71, 133 pfstatepl: 256, 65010, 14858, 22552, 29834555 pfrulepl: 560, 0, 248, 221, 1693 pfsrctrpl: 80, 0, 0, 0, 0 FFS2 dinode: 256, 0, 32820, 435, 1154765 FFS1 dinode: 128, 0, 0, 0, 0 FFS inode: 140, 0, 32820, 360, 1154765 SWAPMETA: 276, 63672, 9, 47, 40 rtentry: 132, 0, 222, 184, 6666 unpcb: 140, 17220, 9, 47, 5217 ripcb: 180, 8140, 0, 44, 15 sackhole: 16, 0, 0, 203, 3107 tcpreass: 20, 1183, 0, 169, 134 hostcache: 88, 15400, 1, 87, 39 syncache: 108, 15372, 0, 72, 374 tcptw: 56, 1675, 0, 134, 48 tcpcb: 448, 8136, 4, 23, 489 inpcb: 180, 8140, 4, 40, 489 udpcb: 180, 8140, 11, 33, 6296980 socket: 324, 8136, 24, 36, 6302703 KNOTE: 68, 0, 0, 112, 12537593 PIPE: 384, 0, 2, 78, 251017 DIRHASH: 1024, 0, 1621, 11, 4318 NAMEI: 1024, 0, 3, 17, 131046202 L VFS Cache: 291, 0, 38, 456, 8482 S VFS Cache: 68, 0, 29953, 5551, 1214811 VNODEPOLL: 64, 0, 0, 118, 2 VNODE: 264, 0, 33205, 20, 33205 ata_request: 200, 0, 0, 38, 2322417 g_bio: 132, 0, 0, 2204, 4301228 MbufClust: 2048, 17216, 906, 124, 12121216 Mbuf: 256, 0, 907, 188, 2520288299 Packet: 256, 0, 674, 421, 2728222296 VMSPACE: 300, 0, 23, 81, 726488 UPCALL: 44, 0, 0, 0, 0 KSEGRP: 104, 0, 144, 31, 144 TID: 140, 0, 1, 53, 1 THREAD: 388, 0, 144, 6, 144 PROC: 452, 0, 73, 71, 726546 Files: 68, 0, 71, 153, 122876814 4096: 4096, 0, 129, 60, 746392 2048: 2048, 0, 130, 30, 24033 1024: 1024, 0, 23, 61, 14340393 512: 512, 0, 244, 92, 656448 256: 256, 0, 239, 1111, 7152982 128: 128, 0, 1785, 375, 15426920 64: 64, 0, 4243, 1421, 8922809 32: 32, 0, 2519, 758, 4033195752 16: 16, 0, 1928, 305, 22113261 DP fakepg: 72, 0, 0, 0, 0 PV ENTRY: 24, 944530, 5540, 17080, 315699586 MAP ENTRY: 68, 0, 439, 1241, 32680232 KMAP ENTRY: 68, 32200, 40, 464, 5443857 MAP: 192, 0, 7, 33, 5 VM OBJECT: 132, 0, 32552, 1436, 14687110 128 Bucket: 524, 0, 966, 0, 0 64 Bucket: 268, 0, 30, 40, 0 32 Bucket: 140, 0, 30, 26, 0 16 Bucket: 76, 0, 15, 35, 0 UMA Hash: 128, 0, 2, 28, 0 UMA RCntSlab: 104, 0, 515, 3, 0 UMA Slabs: 64, 0, 793, 33, 0 UMA Zones: 88, 0, 73, 7, 0 UMA Kegs: 136, 0, 73, 23, 0 There doesn't seem to be anything really changed between then and now (i.e. in one hour), but netstat works again. Can anyone recommend a suitable course of actions? I'm somewhat at a loss as to what I should look for now. It seems that when the machine has been up for a few days this starts happening once or twice a day, until the machine finally deadlocks (it usually does this after running for a week). I tried to enable WITNESS, but as fas as I can gather from the dmesg, it doesn't work as there is not enough buffer space or something (the error is "witness_get: witness exhausted"). -Martin