From owner-freebsd-current@FreeBSD.ORG Wed May 1 18:08:58 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 929FE5A2; Wed, 1 May 2013 18:08:58 +0000 (UTC) (envelope-from rwatson@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [204.107.128.56]) by mx1.freebsd.org (Postfix) with ESMTP id 6F32B1D42; Wed, 1 May 2013 18:08:58 +0000 (UTC) Received: from fell28n62.joh.private.cam.ac.uk (global-1-82.nat.csx.cam.ac.uk [131.111.184.82]) by cyrus.watson.org (Postfix) with ESMTPSA id B240846B35; Wed, 1 May 2013 14:08:57 -0400 (EDT) Subject: Re: panic: in_pcblookup_local (?) Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: "Robert N. M. Watson" In-Reply-To: <20130501180321.GA44525@glenbarber.us> Date: Wed, 1 May 2013 19:08:57 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <49916D2B-496A-40EA-971F-62951FF6B584@freebsd.org> References: <201304301653.13845.jhb@freebsd.org> <20130430211908.GB1621@glenbarber.us> <201305011156.03974.jhb@freebsd.org> <20130501180321.GA44525@glenbarber.us> To: Glen Barber X-Mailer: Apple Mail (2.1283) Cc: Ian FREISLICH , freebsd-current@freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 May 2013 18:08:58 -0000 On 1 May 2013, at 19:03, Glen Barber wrote: >> I'll need to catch up on this thread later, but a few questions: >>=20 >> Do we know if the application in question is multithreaded, and >> if so, might it be attempting concurrent operations on this socket? >=20 > I do not know if zabbix-agent is multithreaded, but cf-agent is. If in DDB, it would be useful to do a "ps" so we can identify threads in = the process, and in particular, whether they might be in the kernel = around the moment of the panic. > I will follow up with this information as soon as possible. Thanks. Do keep around as much information as you can from DDB, = crashdumps, etc. A useful set of things to keep from DDB includes the = initial panic information and trap frame, "show pcpu", "show allpcpu", = "trace", "alltrace", "ps", and if WITNESS is compiled in, "show locks" = and "show alllocks". On busy systems, all the backtraces add up to a lot = of space, so you might hold onto that rather than e-mail it, but contain = useful information. Often, debugging this sort of race condition = involves looking at what other network-centred threads are doing -- = e.g., device-driver ithreads, netisr, other involved user threads. You = may be able to extract much of that information using ps on the = crashdump (not sure if procstat is there yet for crashdumps) -- if so, = be sure to use -H (or whatever the argument is to print thread, not just = process, information). Off to a formal dinner, but back later! Robert=