From owner-freebsd-stable@FreeBSD.ORG Sat Nov 2 19:54:31 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id B27444E3 for ; Sat, 2 Nov 2013 19:54:31 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8362F25A9 for ; Sat, 2 Nov 2013 19:54:31 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id rA2JsPRV095076 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 2 Nov 2013 12:54:25 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id rA2JsPGh095075; Sat, 2 Nov 2013 12:54:25 -0700 (PDT) (envelope-from jmg) Date: Sat, 2 Nov 2013 12:54:25 -0700 From: John-Mark Gurney To: Shawn Wallbridge Subject: Re: 9.2-RELEASE Kernel panic, mbuf underflow Message-ID: <20131102195425.GI73243@funkthat.com> Mail-Followup-To: Shawn Wallbridge , freebsd-stable@freebsd.org References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Sat, 02 Nov 2013 12:54:25 -0700 (PDT) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Nov 2013 19:54:31 -0000 Shawn Wallbridge wrote this message on Tue, Oct 29, 2013 at 21:37 -0700: > I have a file server that keeps panic?ing with a mbuf cluster in the 17 Quadrillion range (2^64 - 2). I am pretty sure it?s a buffer underflow. Ok, after some tracking stuff down, I do not think it has anything to do w/ mbufs, as the stats appear to be correct... The problem is that mbuf clusters takes into the fact that some clusters might be still associated w/ packets (from usr.bin/netstat/mbuf.c): printf("%ju/%ju/%ju/%ju mbuf clusters in use " "(current/cache/total/max)\n", cluster_count - packet_free, cluster_free + packet_free, cluster_count + cluster_free, cluster_limit); notice how current is cluster_count - packet_free instead of something like cluster_count - cluster_free... And I just printed your values from vmcore.6, and apparently packet_count is 0, while packet_free is 5215... cluster_count is 2049, cluster_free is 1997.. And because packet is a secondary zone of mbufs, things apparently get confused... So I wouldn't go down this road anymore... This looks like a simple race/accounting error in the status... > I have opened a PR, but I haven?t had any movement on it. This happened while I was running 9.1-RELEASE as well. > > Here is the PR.. > > http://www.freebsd.org/cgi/query-pr.cgi?pr=183424 > > And I have uploaded the crash dumps here.. > > http://www.wallbridge.net/crash/ > > If anyone has any ideas, I would be grateful as this is a production box and it?s really impacting us. Have you done a full fsck on the fs to make sure that there isn't any corruption on the disk that keeps popping up? I do realize that it will take a LONG time to fsck... Sadly, you're last three cores (all on 9.2-R) are for different inodes... Could you tell me the path and filename of inodes: 3226539015, 3224134148 and 3343904256? It could help us track down which app is causing this and being able to reproduce this... To find the inode on the fs use find -inum , so: find -inum 3226539015 -or -inum 3224134148 -or -inum 3343904256 will do it in one pass so it won't take so long... Thanks. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."