From owner-freebsd-current@FreeBSD.ORG Thu May 2 10:42:24 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 0F1A4817; Thu, 2 May 2013 10:42:24 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from onyx.glenbarber.us (onyx.glenbarber.us [199.48.134.227]) by mx1.freebsd.org (Postfix) with ESMTP id E0E451362; Thu, 2 May 2013 10:42:23 +0000 (UTC) Received: from glenbarber.us (kaos.glenbarber.us [71.224.221.174]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) (Authenticated sender: gjb) by onyx.glenbarber.us (Postfix) with ESMTPSA id B250823F804; Thu, 2 May 2013 06:42:21 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.8.2 onyx.glenbarber.us B250823F804 Authentication-Results: onyx.glenbarber.us; dkim=none reason="no signature"; dkim-adsp=none Date: Thu, 2 May 2013 06:42:19 -0400 From: Glen Barber To: "Robert N. M. Watson" Subject: Re: panic: in_pcblookup_local (?) Message-ID: <20130502104219.GA1586@glenbarber.us> References: <20130501180321.GA44525@glenbarber.us> <49916D2B-496A-40EA-971F-62951FF6B584@freebsd.org> <201305011430.37106.jhb@freebsd.org> <20130502005704.GB1623@glenbarber.us> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="lrZ03NoBR/3+SXJZ" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 10.0-CURRENT amd64 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Ian FREISLICH , freebsd-current@freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 May 2013 10:42:24 -0000 --lrZ03NoBR/3+SXJZ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, May 02, 2013 at 10:27:39AM +0100, Robert N. M. Watson wrote: >=20 > On 2 May 2013, at 01:57, Glen Barber wrote: >=20 > > So, I am admittedly not too familiar with DDB. In fact, I just now > > realize the kernel is built without DDB... >=20 > DDB is a very powerful tool in that it's been custom-developed > to help debug common kernel panics. It lacks some of the flexibility, > and especially the data-type awareness of GDB, but GDB is a less > well-suited tool when investigating common crash patterns. I'll > usually start out debugging in DDB, and find that 90% of my > in-development panics can be debugged with it, resorting to GDB for > post-mortem analyses in production or particularly hard debugging > cases (usually where DDB's pretty printers for data types fall > short). I've wanted, for a long time, to teach DDB how to pretty-print > arbitrary types using DTrace's CTF meta-data, which would address > the most significant major case where I turn to GDB. Mind you, the > limitations I see in GDB are made up for in most part by John's GDB > scripts :-). >=20 Hmm. Perhaps it would be worthwhile for me to rebuild the current kernel with DDB support. It looks like the machine has panicked a few times over the last two weeks or so, but based on the timestamps of the crash dumps and nagios complaints, happened during the middle of the night when I would not have really noticed, or otherwise would have just blamed my ISP. Two of the panics are ath(4) related. One looks similar to the one referenced in this thread, similarly triggered by a CFEngine process. In that case, the backtrace looks like: #4 0xffffffff808cdbb3 at calltrap+0x8 #5 0xffffffff807371d8 at in_pcb_lport+0x128 #6 0xffffffff8073745a at in_pcbbind_setup+0x16a #7 0xffffffff80737d8e at in_pcbconnect_setup+0x71e #8 0xffffffff80737df9 at in_pcbconnect_mbuf+0x59 #9 0xffffffff807bf29f at udp_connect+0x11f #10 0xffffffff80680615 at kern_connectat+0x275 Regarding DDB though, it would be rather difficult to access the machine if it drops to a DDB debugger session, since the machine acts as my firewall. > >> Put those in a dir and do 'source gdb6'. You can then run 'ps' to get= a good=20 > >> ps listing that includes threads. You can also use 'thread apply all = bt' to=20 > >> get stacktraces of all threads in kgdb. I believe there is an 'allpcp= u'=20 > >> command that is similar to 'show allpcpu' in DDB. > >=20 > > I have the outputs of 'ps', 'allpcpu', and 'thread apply all bt' saved > > to separate script(1) files. Is there anything in particular I can look > > for before uploading the files somewhere public? At quick-ish look > > though, I did not see anything cf-agent (the current process at time of > > panic) related. >=20 > To be honest, it's probably easiest if I just take a look at it > and see what I see. In as much as I find interesting things, I'll > follow up explaining what they are. We may find we can't track this > problem down from the data we have -- but it's worth a try. >=20 Sure. The files are available here: https://www.glenbarber.us/stuff/in_pcblookup_local/vmcore.4.ps.txt https://www.glenbarber.us/stuff/in_pcblookup_local/vmcore.4.allpcpu.txt https://www.glenbarber.us/stuff/in_pcblookup_local/vmcore.4.thread_appl= y_all_bt.txt Thanks to both of you for looking into this. Glen --lrZ03NoBR/3+SXJZ Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQEcBAEBCAAGBQJRgkMLAAoJEFJPDDeguUajBQYIAKR8BP8hK9zycL9J8JAlYqJh nO+hhLS5FWHsK5gy7cxb10TTKTu6bw8lcNV3mBuANBxXh0tVf+UYuk7WgTBXvWcb WUUFWQcY1Mhh8rdoEs2hax0Kuvp3wVHhIo0mWoK36S+jh9FmcGQWGqHn5oDW4Izo /Bbyqs8ir/Q73RQwWmuijvGGMR2tBUEPU50cSWS4IZobz1zJUa1wuf8TaNb9wJ0j FSpCk93U7K98h+u9gAegNOrfvWpaZa5MVH496D5iDfta4UN1MhS3LGSrR5uOElKa H82RXAKxz0wijOru4+D1Ne1gxNFrwuojGic0b2/3EDOLmz6Q5j09zgzv4RQjc6M= =mMpJ -----END PGP SIGNATURE----- --lrZ03NoBR/3+SXJZ--