From owner-freebsd-net@FreeBSD.ORG Wed Mar 25 17:59:07 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 006E0606 for ; Wed, 25 Mar 2015 17:59:06 +0000 (UTC) Received: from mail-wg0-f43.google.com (mail-wg0-f43.google.com [74.125.82.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A1BD1B76 for ; Wed, 25 Mar 2015 17:59:06 +0000 (UTC) Received: by wgra20 with SMTP id a20so36598963wgr.3 for ; Wed, 25 Mar 2015 10:58:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type; bh=KGXllimYQdnxoAN6y4r18/IpUAl973PRMTUvFM67Rgw=; b=PvRSKqaMx4fD6cwhTPT83d4NbKND5D9TUXeHabAI0KgOgoLCDUQSq2PjRfWib4ZW7L KqBie7453vLWA2KG8FI8HaCZo7KIF+pR5NWK78x+3Hq5kcgM/YtO3alPG6uWSLGggqWb Fut8YN0Mg71G8aB2a7F4Lwkb4r8sz9tsvdPk8ZrMWCI1nEXjdJBbolFfUItV2SDtCBbn Aeh/UN4I/7rvHdRxJcDhyrOAGrpCGqd/py24CbYspHmc/g12FFdBV/RL++/7CQDMzCGw dVYVyh2a5tTMX9SS2dcuFUvC1XdePVi59aYpT4iOZNM4OdyNaerdY/Q//kx+ckWaMAvC jTfA== X-Gm-Message-State: ALoCoQnYhmPtnphf4p05taf2XBaGKZQzwakveOs4hefHEn2Lo1QEM9dyM4E4TtSSDRfZN9RumYNJ X-Received: by 10.194.157.39 with SMTP id wj7mr20381152wjb.57.1427306339573; Wed, 25 Mar 2015 10:58:59 -0700 (PDT) Received: from FRI2JCHARBON-M1.local ([217.30.88.7]) by mx.google.com with ESMTPSA id ax10sm4735508wjc.26.2015.03.25.10.58.57 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Mar 2015 10:58:58 -0700 (PDT) Message-ID: <5512F75A.1020607@freebsd.org> Date: Wed, 25 Mar 2015 18:58:50 +0100 From: Julien Charbon User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: "Robert N. M. Watson" , Adrian Chadd , FreeBSD Net Subject: Re: VIMAGE UDP memory leak fix References: <20141121002937.4f82daea@x23> <9300CB5F-6140-4C49-B026-EB69B0E8B37E@FreeBSD.org> <20141121120201.6c77ea5b@x23> <20141121162042.449b22dc@x23> <072B7B0F-4DE3-4D37-BC94-1DEA38CF3B12@FreeBSD.org> <85C7A32E-121D-495F-93C7-9D2B2F134FF6@FreeBSD.org> In-Reply-To: <85C7A32E-121D-495F-93C7-9D2B2F134FF6@FreeBSD.org> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="DTQGpVj3gP2fbOT70eogpa2tDFbHKKuaI" Cc: Craig Rodrigues , "Bjoern A. Zeeb" , Marko Zec X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2015 17:59:07 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --DTQGpVj3gP2fbOT70eogpa2tDFbHKKuaI Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi, On 22/11/14 18:09, Robert N. M. Watson wrote: > On 21 Nov 2014, at 17:40, Adrian Chadd wrote: >>>> Skimming through a bunch of hosts with moderately loaded hosts >>>> with reasonably high uptime I couldn't find one where >>>> net.inet.tcp.timer_race was not zero. A ny suggestions how to >>>> best reproduce the race(s) in tcp_timer.c? >>>=20 >>> They would likely occur only on very highly loaded hosts, as they >>> require race conditions to arise between TCP timers and TCP >>> close. I think I did manage to reproduce it at one stage, and >>> left the counter in to see if we could spot it in production, and >>> I have had (multiple) reports of it in deployed systems. I'm not >>> sure it's worth trying to reproduce them, given that knowledge -- >>> we should simply fix them. >>=20 >> Wasn't this just fixed by Julien @ Verisign? >=20 > I don't believe so, although it's the kind of thing Julien is very > good at fixing! >=20 > The issue here is that we can't call callout_drain() from contexts > where we finalise TCP connection close and attempt to free the inpcb. > The 'easy' fix is to create a taskqueue thread to do the > callout_drain() in the event that we discover that callout_stop() > isn't able to guarantee that pending callouts are neither in > execution nor scheduled. We'd then defer the very tail of TCP > teardown to that asynchronous context rather than trying to do it to > completion in the current (and rather more sensitive) one. This would > happen only very in frequently so have little overhead in practice, > although one would want to carefully look at the sync behaviour to > make sure it wasn't frequently enough that a backlog might build up. Ironically, I was not aware of this discussion before we: - (re)discovered this TCP timers callout race condition - followed Robert's XXXRW comments and advices in source code - proposed a patch for fixing it: Fix TCP timers use-after-free old race conditions https://reviews.freebsd.org/D2079 If the proposed patch follows most of Robert's advices, instead of using a taskqueue thread calling callout_drain(), we re-use directly the callout API to schedule tcpcb's uma_zfree() at appropriate time. Register to the review if your are interested in this fix proposition. But still, nice irony to find this thread _after_ proposing a fix... :) -- Julien --DTQGpVj3gP2fbOT70eogpa2tDFbHKKuaI Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJVEvdgAAoJEKVlQ5Je6dhx/kIH/1/FN8F8Ixq5LbxwJN9WbvGK BxQketKppg18yTN1DOXpj4IGnRiAURaNGUicRriRRYxyOB4q4U0mSJlAB+dPNlxf RJEFXExekNyi5+jh0ktSUxbwfcKlsuFGcJkgHmyfxFJDXvWFx3Euu5Quc4DgNTQO FgOdNNxwCJut1lauPvfNEoHmNhUgiG+LD13gkubkOU77k/3BK5La8NrembUqvtGD gpCKOD5yMsEj4asl90VYqGh6TKgRaByO/MdWe+NAmq+9eNTqVfF415uCfHJuz64H Y0B5SMKcfvDP2pqw8H/rtSkbJSyRAFPWN+n2j5L7w7gQKxen8ECcPhGncsyogYE= =FZn6 -----END PGP SIGNATURE----- --DTQGpVj3gP2fbOT70eogpa2tDFbHKKuaI--