From owner-freebsd-net@freebsd.org Mon Sep 21 08:28:56 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9F25A052BE for ; Mon, 21 Sep 2015 08:28:56 +0000 (UTC) (envelope-from julien@jch.io) Received: from mail-wi0-f176.google.com (mail-wi0-f176.google.com [209.85.212.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 73BF218B0 for ; Mon, 21 Sep 2015 08:28:56 +0000 (UTC) (envelope-from julien@jch.io) Received: by wiclk2 with SMTP id lk2so134869459wic.0 for ; Mon, 21 Sep 2015 01:28:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type; bh=seQgeVL3VMoMKmYpURnkFaJnqjT6Ywc1uMGP8ASJJJU=; b=I9aPnGi7zC0oix52+E4O3Glq7TsybK0Uh7+toQWv0JMn1UquCHarcvsfXTHvw1PNBX kQsFfevO8/d5v2ZFttsj4Gpx10USagzJyRlsSgqPDQplTDeqTzwkmJBEZxDBuCigFoDo VEdSeCQwlauSRhx+JHcuAeSz4yFkoOwGolRAUi5Ja//rMm6+oQCOmt6YXG2jsAKHlpHB TYqddcweh2jndi/ehtiNfUHVokjxZvnU++oMinOxm3TuWPmf4TtuF2veUyfQtHKvcuxI MhqyVr2iWF3gO6KX21uIjLNJ3yNmCzDFi++tEPVl2Oi80Jv56WjugRZ4WFYfIiEkZWKC 0kyQ== X-Gm-Message-State: ALoCoQltiWviK7jPPBSGQfFz34kq48c73yipajiTQdYEs29t0XszVfqjJmzyGjbklg+Q33XLpnY4 X-Received: by 10.194.23.167 with SMTP id n7mr20833361wjf.112.1442824129029; Mon, 21 Sep 2015 01:28:49 -0700 (PDT) Received: from res1momishra-l1.vcorp.ad.vrsn.com ([217.30.88.7]) by smtp.googlemail.com with ESMTPSA id kb5sm22864757wjc.17.2015.09.21.01.28.47 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Sep 2015 01:28:48 -0700 (PDT) Subject: Re: Kernel panics in tcp_twclose To: Palle Girgensohn , Konstantin Belousov , Adrian Chadd References: <26B0FF93-8AE3-4514-BDA1-B966230AAB65@FreeBSD.org> <55FC1809.3070903@freebsd.org> <20150918160605.GN67105@kib.kiev.ua> <9A234106-62EC-49C9-954A-2DA8315E9B4A@pingpong.net> Cc: Palle Girgensohn , "freebsd-net@freebsd.org" From: Julien Charbon X-Enigmail-Draft-Status: N1110 Message-ID: <55FFBFBC.30905@freebsd.org> Date: Mon, 21 Sep 2015 10:28:44 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <9A234106-62EC-49C9-954A-2DA8315E9B4A@pingpong.net> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="bVkmOmAognLcpdvLOM6QEA6rrTAiq6GbA" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Sep 2015 08:28:56 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --bVkmOmAognLcpdvLOM6QEA6rrTAiq6GbA Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi Palle, On 18/09/15 22:42, Palle Girgensohn wrote: >> 18 sep 2015 kl. 18:06 skrev Konstantin Belousov >> : >>=20 >>> On Fri, Sep 18, 2015 at 03:56:25PM +0200, Julien Charbon wrote:=20 >>> Hi Palle, >>>=20 >>>> On 18/09/15 11:12, Palle Girgensohn wrote: We see daily panics >>>> on our production systems (web server, apache running MPM >>>> event, openjdk8. Kernel with VIMAGE. Jails using netgraph=20 >>>> interfaces [not epair]). >>>>=20 >>>> The problem started after the summer. Normal port upgrades >>>> seems to be the only difference. The problem occurs with >>>> 10.2-p2 kernel as well as 10.1-p4 and 10.1-p15. >>>>=20 >>>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203175 >>>>=20 >>>> Any ideas? >>>=20 >>> Thanks for you detailed report. I am not aware of any >>> tcp_twclose() related issues (without VIMAGE) since FreeBSD 10.0 >>> (does not mean there are none). Few interesting facts (at least >>> for me): >>>=20 >>> - Your crash happens when unlocking a inp exclusive lock with >>> INP_WUNLOCK() >>>=20 >>> - Something is already wrong before calling turnstile_broadcast() >>> as it is called with ts =3D NULL: >> In the kernel without witness this is a 99%-sure indication of >> attempt to unlock not owned lock. >>=20 >>>=20 >>> turnstile_broadcast (ts=3D0x0, queue=3D1) at=20 >>> /usr/src/sys/kern/subr_turnstile.c:838 __rw_wunlock_hard () at >>> /usr/src/sys/kern/kern_rwlock.c:988 tcp_twclose () at >>> /usr/src/sys/netinet/tcp_timewait.c:540 tcp_tw_2msl_scan () at >>> /usr/src/sys/netinet/tcp_timewait.c:748 tcp_slowtimo () at >>> /usr/src/sys/netinet/tcp_timer.c:198 >>>=20 >>> I won't go to far here as I am not expert enough in VIMAGE, but >>> one question anyway: >>>=20 >>> - Can you correlate this kernel panic to a particular event? >>> Like for example a VIMAGE/VNET jail destruction. >>>=20 >>> I will test that on my side on a 10.2 machine. >=20 > I just got a response from adrian@ where he seems to remember that it > has all been fixed in head. >=20 > I would really prefer not to run a head kernel in production unless I > have to, so the question is if it is possible to pin down the > specific fixes for this problem? Any suggestions? >=20 > Thanks for all the help so far! On my side, all issues we have found in TCP stack are currently both fixed in 10.2 and HEAD. The remaining differences are only performance improvements that are solely in HEAD. adrian@ might have more details on fixes he has in mind. -- Julien --bVkmOmAognLcpdvLOM6QEA6rrTAiq6GbA Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJV/7/CAAoJEKVlQ5Je6dhxNb0H/i+UyIVET1W7Qcv+zmCj3G5j WGkXE6VltZy52Hb8dl/vbkUuWaeu6jxxiQDY9uAp73twxYChrIWLIvRQ0yhOxZHo lNuwNaK//ahKwMVn2Q7ALJaMli7j318DoVAeS0XgPLa9m9xN9/mGURZJIeF/vNM8 s85GPa6rbnTNJMsOlZfCIlh384jtzIL31XlQTnBV+hRUypuSohjJTfZsGa8ISxmJ XDxmxxwe8yPFJ0ch5PpSqITth1SEs61L/UkY/TIxNMF1zpdeT+9xSXn7YMAeObzA WaJXDFbgWibhsg+wo96eUrFc6vkfU2Y68xczHuTUy+22gmcLSMNVLYVfFsFFXV8= =JGZA -----END PGP SIGNATURE----- --bVkmOmAognLcpdvLOM6QEA6rrTAiq6GbA--