From owner-freebsd-stable@FreeBSD.ORG Mon Dec 27 12:35:27 2004 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B16ED16A4CE; Mon, 27 Dec 2004 12:35:27 +0000 (GMT) Received: from brugere.aub.dk (fw.aub.dk [195.24.1.195]) by mx1.FreeBSD.org (Postfix) with ESMTP id A376E43D2F; Mon, 27 Dec 2004 12:35:22 +0000 (GMT) (envelope-from techlists@motrix.dk) Received: by brugere.aub.dk (Postfix, from userid 1693) id C153BC331; Mon, 27 Dec 2004 13:35:20 +0100 (CET) Received: from glitter (jmp.aub.dk [10.1.4.50]) by brugere.aub.dk (Postfix) with ESMTP id 9F3FCC2EF; Mon, 27 Dec 2004 13:35:20 +0100 (CET) From: To: "'Doug White'" , Date: Mon, 27 Dec 2004 13:35:21 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook, Build 11.0.6353 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 In-Reply-To: <20041213101844.S92964@carver.gumbysoft.com> thread-index: AcThQGfe5eN3XEf7SqybObVkMGwJUwKzWlGA Message-Id: <20041227123520.9F3FCC2EF@brugere.aub.dk> cc: mlaier@freebsd.org Subject: RE: netstat fails with memory allocation error and error in kvm_read X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Dec 2004 12:35:27 -0000 > > > You appear to be running out of kernel memory. Since you're > > > capturing the output of vmstat -m, you should check that for any > > > bins that are growing at a high rate of speed. > > > > > > Seems possible that its in pf :) > > > > I've checked the numbers from just before the freeze (it's > within 15 > > secs) with two sets of data: From a fresh boot and five minutes > > minutes before the freeze. > > You might also log 'sysctl vm.kvm_free' and 'sysctl vm.zone'. sysctl vm.zone is identical to vmstat -z (according to man vmstat). I've graphed the output from iostat (idle/user/...), vmstat -i (interrupt rate), vmstat -m (in use), vmstat -z (used), sysctl vm.kvm_free (which is constant) and the number of pfstates. The graphs are at . The newest data are from just after a deadlock. Are there something else I should graph? IRQ 20 is the NIC on our internal network (800+ machines), IRQ 18 and IRQ21 are NICs connected to the internet. There are a lot of changes on the vmstat -m graphs just before midnight last night that seems to correspond with the increase in interrupts on IRQ 18. The only graphs I can see changing up to the deadlock are: irq20 (internal NIC), irq21 (primary external NIC), the buckets (vmstat -z) all grow (I suppose this is normal?) the Mbufs seems to grow, but nothing extreme pffrag, pffrent (but not to levels they haven't been at before) Most notably most of the pf graphs doesn't change. Where can I see memory used by pf/altq? If it is pfaltqpl (in vmstat -z), it doesn't change at all. I'm in the process of setting up a serial console in the hope that I can break to the debugger with that. I'm also trying to provoke the deadlock so it will happen more frequently. /Martin