From owner-freebsd-net@freebsd.org Fri Sep 18 14:29:06 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 85C689CFAA8 for ; Fri, 18 Sep 2015 14:29:06 +0000 (UTC) (envelope-from girgen@FreeBSD.org) Received: from mail.pingpong.net (mail.pingpong.net [79.136.116.202]) by mx1.freebsd.org (Postfix) with ESMTP id 45ACA18DA; Fri, 18 Sep 2015 14:29:05 +0000 (UTC) (envelope-from girgen@FreeBSD.org) Received: from [10.0.0.106] (citron2.pingpong.net [195.178.173.68]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.pingpong.net (Postfix) with ESMTPSA id 46967D246; Fri, 18 Sep 2015 16:28:58 +0200 (CEST) Subject: Re: Kernel panics in tcp_twclose using jails + VIMAGE Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Content-Type: multipart/signed; boundary="Apple-Mail=_0D107C28-45CE-4957-9CD0-8DCB8DFC6727"; protocol="application/pgp-signature"; micalg=pgp-sha256 X-Pgp-Agent: GPGMail 2.5.1 From: Palle Girgensohn In-Reply-To: <55FC1809.3070903@freebsd.org> Date: Fri, 18 Sep 2015 16:28:57 +0200 Cc: freebsd-net@freebsd.org Message-Id: References: <26B0FF93-8AE3-4514-BDA1-B966230AAB65@FreeBSD.org> <55FC1809.3070903@freebsd.org> To: Julien Charbon X-Mailer: Apple Mail (2.2104) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Sep 2015 14:29:06 -0000 --Apple-Mail=_0D107C28-45CE-4957-9CD0-8DCB8DFC6727 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 > 18 sep 2015 kl. 15:56 skrev Julien Charbon : >=20 > Hi Palle, >=20 > On 18/09/15 11:12, Palle Girgensohn wrote: >> We see daily panics on our production systems (web server, apache >> running MPM event, openjdk8. Kernel with VIMAGE. Jails using netgraph >> interfaces [not epair]). >>=20 >> The problem started after the summer. Normal port upgrades seems to >> be the only difference. The problem occurs with 10.2-p2 kernel as >> well as 10.1-p4 and 10.1-p15. >>=20 >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203175 >>=20 >> Any ideas? >=20 > Thanks for you detailed report. I am not aware of any tcp_twclose() > related issues (without VIMAGE) since FreeBSD 10.0 (does not mean = there > are none). Few interesting facts (at least for me): >=20 > - Your crash happens when unlocking a inp exclusive lock with = INP_WUNLOCK() >=20 > - Something is already wrong before calling turnstile_broadcast() as = it > is called with ts =3D NULL: >=20 > turnstile_broadcast (ts=3D0x0, queue=3D1) at > /usr/src/sys/kern/subr_turnstile.c:838 > __rw_wunlock_hard () at /usr/src/sys/kern/kern_rwlock.c:988 > tcp_twclose () at /usr/src/sys/netinet/tcp_timewait.c:540 > tcp_tw_2msl_scan () at /usr/src/sys/netinet/tcp_timewait.c:748 > tcp_slowtimo () at /usr/src/sys/netinet/tcp_timer.c:198 >=20 > I won't go to far here as I am not expert enough in VIMAGE, but one > question anyway: >=20 > - Can you correlate this kernel panic to a particular event? Like for > example a VIMAGE/VNET jail destruction. >=20 > I will test that on my side on a 10.2 machine. >=20 > -- > Julien >=20 Hi, thank for your reply. It is not related to jail destruction. It = *might* be related to apache httpd (MPM event) forking during normal = operation, but we have not found any specific event that triggers the = panic. The system crash during normal operation, no excessive load (but = load is at least partly responsible, a more loaded server is more likely = to crash). Note that we use netgraph, not epair, although I don't believe it makes = a difference. Palle --Apple-Mail=_0D107C28-45CE-4957-9CD0-8DCB8DFC6727 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBCAAGBQJV/B+pAAoJEIhV+7FrxBJDBb8H/RXUyYSm2xIRAT0/gIGLbQVh rLyEOPQcbQ4ST319Gtf/Us99qy2zF973m3FMlmeeuN5hmqB9I0KHPxskD7HZKd00 5kzXAvbsot8f96629sc7Vpp62XWXpd5kvO4uNijbyuGUSbI1j3GSurKIxgq1Jc86 MOYNY2h0DDdzkbUjCYUo/4bQ3YQ+DaGrT407tU3bdYbsxSHqrfbhkiLJhJCidiYS 4t5EpVqtu1FypJaMdJCdxbmPMnk3y1HcyKai651zRSePsAXNUb4GzGJ2+RqHF+N6 rqj3mZFg0G36Xx4OQ4qe6t+1+YCMsBRiC6K0I5NKXXJEZlNvFcNguQgJafHUhIU= =C6TS -----END PGP SIGNATURE----- --Apple-Mail=_0D107C28-45CE-4957-9CD0-8DCB8DFC6727--