From owner-freebsd-net@FreeBSD.ORG Tue Jun 3 10:42:55 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 06ED5DD2 for ; Tue, 3 Jun 2014 10:42:55 +0000 (UTC) Received: from plane.gmane.org (plane.gmane.org [80.91.229.3]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B5E7F298A for ; Tue, 3 Jun 2014 10:42:54 +0000 (UTC) Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1WrmB2-0006DF-QQ for freebsd-net@freebsd.org; Tue, 03 Jun 2014 12:42:44 +0200 Received: from 217.30.88.7 ([217.30.88.7]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 03 Jun 2014 12:42:44 +0200 Received: from jcharbon by 217.30.88.7 with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 03 Jun 2014 12:42:44 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-net@freebsd.org From: Julien Charbon Subject: Re: [rfc] TCP timewait and credential handling - why would we get a TW with no credential? Date: Tue, 03 Jun 2014 12:42:46 +0200 Lines: 64 Message-ID: <538DA6A6.6040201@verisign.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 217.30.88.7 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 In-Reply-To: X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jun 2014 10:42:55 -0000 Hi Adrian, On 01/06/14 17:21, Adrian Chadd wrote: > I've been seeing this panic under high very short-term connection TCP > throughput tests (30,000 connections a second) with SO_REUSEPORT: > > Current language: auto; currently minimal > > (kgdb) frame 11 > > #11 0xffffffff80a6bdf1 in tcp_twclose (tw=0xfffff801348b5780, reuse=0) > at /usr/home/adrian/git/github/erikarn/freebsd/sys/netinet/tcp_timewait.c:644 > > 644 crfree(tw->tw_cred); > > (kgdb) frame 11 > > #11 0xffffffff80a6bdf1 in tcp_twclose (tw=0xfffff801348b5780, reuse=0) > at /usr/home/adrian/git/github/erikarn/freebsd/sys/netinet/tcp_timewait.c:644 > > 644 crfree(tw->tw_cred); > > (kgdb) print tw > > $1 = (struct tcptw *) 0xfffff801348b5780 > > (kgdb) print tw->tw_cred > > $2 = (struct ucred *) 0x0 > [...] > > ... there's only one path to deleting the credentials and that's via > tcp_twclose() -> tcp_tw_2msl_stop(). > > Has anyone seen this kind of problem before? Interesting question: I already got this issue also under short-lived high connection rate (~120k connections per second) and using SO_REUSEPORT _however_ it was with our TCP scaling patches applied (thus it might be a red herring for you) and I fixed it with a: tcp_tw_2msl_stop(): ... if (tw->tw_cred != NULL) crfree(tw->tw_cred); https://github.com/verisign/freebsd/commit/918fe94ad3f3ccd1f6bf577e7e79a138378262a0#diff-4f0c0d9222bfb69f2f358113d424cf89R610 At that time I thought you could create a tw with its tw->tw_cred field set to NULL and then tcp_tw_2msl_stop() simply missed a sanity check before calling crfree(). Now I re-checked how tw->tw_cred is set, its seems impossible to create a tw with tw->tw_cred = NULL (aha!). As this tw_cred field seems well protected by both INP_INFO_WLOCK and INP_WLOCK from race conditions, I am currently checking if something could inadvertently overwrite tw_cred in struct tcptw. My 2 cents. -- Julien