From owner-freebsd-stable@FreeBSD.ORG Mon Dec 13 05:54:17 2004 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6CC3D16A4CE for ; Mon, 13 Dec 2004 05:54:17 +0000 (GMT) Received: from carver.gumbysoft.com (carver.gumbysoft.com [66.220.23.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 49FDB43D53 for ; Mon, 13 Dec 2004 05:54:17 +0000 (GMT) (envelope-from dwhite@gumbysoft.com) Received: by carver.gumbysoft.com (Postfix, from userid 1000) id 3B3E272DD4; Sun, 12 Dec 2004 21:54:17 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by carver.gumbysoft.com (Postfix) with ESMTP id 35E2B72DCB; Sun, 12 Dec 2004 21:54:17 -0800 (PST) Date: Sun, 12 Dec 2004 21:54:17 -0800 (PST) From: Doug White To: "J. Martin Petersen" In-Reply-To: <1102716189.41ba1d1ddfd03@mail.aub.dk> Message-ID: <20041212215318.S83257@carver.gumbysoft.com> References: <1102716189.41ba1d1ddfd03@mail.aub.dk> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-stable@freebsd.org Subject: Re: netstat fails with memory allocation error and error in kvm_read X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Dec 2004 05:54:17 -0000 On Fri, 10 Dec 2004, J. Martin Petersen wrote: > We are trying to gather some debug information for a problem that is difficult > to diagnose, as the machine always ends up hard frozen (does not do anything, > can not break to debugger, does not respond to keyboard, etc.), so we are > dumping output from netstat, vmstat, iostat etc. quite often. > > The cron jobs fail ever so often with error messages I do not quite understand, > and I can not find anything relevant in the archives. Can anyone shed some light > on this? > > Command: netstat -r > Error message: netstat: kvm_read: Bad address > Debug before: http://www.aub.dk/~jmp/fw/debug-2004.12.09-21.34.41.gz > Debug after: http://www.aub.dk/~jmp/fw/debug-2004.12.09-21.35.06.gz > # errors: 7 > > Command: netstat -an > Error message: netstat: sysctl: net.inet.udp.pcblist: Cannot allocate memory > Debug before: http://www.aub.dk/~jmp/fw/debug-2004.12.09-07.38.48.gz > Debug after: http://www.aub.dk/~jmp/fw/debug-2004.12.09-07.39.04.gz > # errors: 3 You appear to be running out of kernel memory. Since you're capturing the output of vmstat -m, you should check that for any bins that are growing at a high rate of speed. Seems possible that its in pf :) > > Command: netstat -an > Error message: netstat: sysctl: net.inet.raw.pcblist: Cannot allocate memory > Debug before: http://www.aub.dk/~jmp/fw/debug-2004.12.10-22.19.50.gz > Debug after: http://www.aub.dk/~jmp/fw/debug-2004.12.10-22.20.10.gz > # errors: 6 > > The number of errors are in the last five days. Kernelconfig (FIREWALL), dmesg, > pf.conf etc. can be found at http://www.aub.dk/~jmp/fw/ > > The commands we are running are: > netstat -i, netstat -m, netstat -s, netstat -an, netstat -r > vmstat, vmstat -i, vmstat -f, vmstat -m, vmstat -z > iostat > ps waux > every 15 seconds. We are running the same commands (plus pfctl -vvsa) from > root's cron every minute. > > The hard lockups seems to be connected to our use of pf and altq, that is at > least the problem we're trying to gather debug information for. > > Thanks in advance > Martin > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > -- Doug White | FreeBSD: The Power to Serve dwhite@gumbysoft.com | www.FreeBSD.org