From owner-freebsd-net@FreeBSD.ORG Fri Feb 2 14:15:56 2007 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D88BF16A401 for ; Fri, 2 Feb 2007 14:15:56 +0000 (UTC) (envelope-from david.baukus@us.fujitsu.com) Received: from fujitsu0.fujitsu.com (fujitsu0.fujitsu.com [192.240.0.5]) by mx1.freebsd.org (Postfix) with ESMTP id B13F413C481 for ; Fri, 2 Feb 2007 14:15:56 +0000 (UTC) (envelope-from david.baukus@us.fujitsu.com) Received: from fujitsu0.fujitsu.com (localhost [127.0.0.1]) by fujitsu0.fujitsu.com (8.13.7/8.13.7) with ESMTP id l12EFtWh007807 for ; Fri, 2 Feb 2007 06:15:55 -0800 (PST) Received: from fujitsu7i.fnanic.fujitsu.com ([133.164.253.7]) by fujitsu0.fujitsu.com (8.13.7/8.13.7) with ESMTP id l12EFsL4007804 for ; Fri, 2 Feb 2007 06:15:55 -0800 (PST) Received: from fncnmp02.fnc.fujitsu.com (localhost [127.0.0.1]) by fujitsu7i.fnanic.fujitsu.com (8.13.7/8.13.7) with ESMTP id l12EFsxL003456 for ; Fri, 2 Feb 2007 06:15:54 -0800 (PST) Received: from unknown (HELO nova.tx.fnc.fujitsu.com) ([167.254.252.121]) by fncnmp02.fnc.fujitsu.com with ESMTP; 02 Feb 2007 08:12:43 -0600 X-IronPort-AV: i="4.13,273,1167631200"; d="scan'208"; a="59645204:sNHT19537812" Received: from miranda.tx.fnc.fujitsu.com (localhost [127.0.0.1]) by nova.tx.fnc.fujitsu.com (8.13.7+Sun/8.13.7) with ESMTP id l12ECirc020952; Fri, 2 Feb 2007 08:12:44 -0600 (CST) Received: from tdd2172.tddeng00.fnts.com (tdd2172.tddeng00.fnts.com [167.254.250.128]) by miranda.tx.fnc.fujitsu.com (8.13.7+Sun/8.13.7) with ESMTP id l12ECfir004304; Fri, 2 Feb 2007 08:12:41 -0600 (CST) Received: from [127.0.0.1] (localhost [127.0.0.1]) by tdd2172.tddeng00.fnts.com (8.11.7p3+Sun/8.11.7) with ESMTP id l12ECfG13320; Fri, 2 Feb 2007 08:12:41 -0600 (CST) Message-ID: <45C346D7.4090305@us.fujitsu.com> Date: Fri, 02 Feb 2007 08:12:39 -0600 From: Dave Baukus User-Agent: Mozilla Thunderbird 1.0 (X11/20041208) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-net@freebsd.org References: <45C2765C.7010708@us.fujitsu.com> In-Reply-To: <45C2765C.7010708@us.fujitsu.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Dave Baukus Subject: Re: ETIMEDOUT bug X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Feb 2007 14:15:57 -0000 I realized, late last night, that I was wrong on a few details concerning this bug: 1.) The retransmit timer does not keep popping on without being restarted. 2.) ip_output() must return ENOBUFS (TCP_MAXRXTSHIFT + 1) times to the same, non-transmitting TCP. 3.) Given a TCP as described below, when tcp_output() uses ENOBUFS to blindly start the retransmit timer then tp->t_rxtshift will be falsely incremented and never cleared. Thus the bug manifests itself because it appears for a TCP that never transmits nobody ever clears clears tp->t_rxtshift; this allows tp->t_rxtshift to slowly count up to TCP_MAXRXTSHIFT; once TCP_MAXRXTSHIFT is exceeded tcp_timer_rexmt() will kill the poor innocent TCP. On 02/01/07 17:23, Dave Baukus wrote: > There is a bug tcp_output() for at least freeBSD6.1 > that causes a perfectly good TCP to be dropped by its > retransmit timer; the application receives ETIMEDOUT. > > Consider a TCP that never transmits (the receive end of the ttcp > utility is an example), while the TCP is established > snd_max == snd_una == snd_nxt == (isr + 1) and the retransmit > timer should never be started. If the retransmit timer is started > then it is never stopped by tcp_input/tcp_out because > snd_max == snd_una == snd_nxt (always). Once started the > timer continues its count up till tp->t_rxtshift == 12 and > the connection that never transmitted gets falsely killed. > > The bug is to blindly rely on the return value of ip_output(). > If ip_output() returns ENOBUFS then the retransmit timer is > activated: > > From the end of tcp_output(): > out: > SOCKBUF_UNLOCK_ASSERT(&so->so_snd); /* Check gotos. */ > if (error == ENOBUFS) { > if (!callout_active(tp->tt_rexmt) && > !callout_active(tp->tt_persist)) > callout_reset(tp->tt_rexmt, tp->t_rxtcur, > tcp_timer_rexmt, tp); > tp->snd_cwnd = tp->t_maxseg; > return (0); > } > > My simple minded fix would be not to start the retransmit timer; > if tcp_output() wanted to time this transmit it would have started > the timer up above. > > This ETIMEDOUT problem is easily recreated on any old machine > using a single slow ethernet device and the ttcp test utility. > First, fire up a couple ttcp receivers. Second, flood the same > interface with enough ttcp transmitters to cause the driver's transmit > ring and interface queue to back up. Eventually, one of the ttcp > receives will get ENOBUFS from ip_output() and the retransmit > timer will be wrongly activated for a pure ACK segment. > > I was able to do it w/ the following on freeBSD6.1: > > box1: > ttcp -s -l 16384 -p 9444 -v -b 128000 -r > ttcp -s -l 16384 -p 9445 -v -b 128000 -r > ttcp -s -n 6553600 -l 4096 -p 9446 -v -b 128000 -t 192.168.222.13 > ttcp -s -n 9999999 -l 333 -p 9447 -v -b 128000 -t 192.168.222.13 > ttcp -s -n 9999999 -l 8192 -p 9448 -v -b 128000 -t 192.168.222.13 > ttcp -s -n 9999999 -l 333 -p 9449 -v -b 128000 -t 192.168.222.13 > ttcp -s -n 9999999 -l 8192 -p 9450 -v -b 128000 -t 192.168.222.13 > > box2: > ttcp -s -n 6553600 -l 8192 -p 9444 -v -b 128000 -t 192.168.222.222 > ttcp -s -n 9999999 -l 128 -p 9445 -v -b 128000 -t 192.168.222.222 > ttcp -s -l 16384 -p 9446 -v -b 128000 -r > ttcp -s -l 16384 -p 9447 -v -b 128000 -r > ttcp -s -l 16384 -p 9448 -v -b 128000 -r > ttcp -s -l 16384 -p 9449 -v -b 128000 -r > ttcp -s -l 16384 -p 9450 -v -b 128000 -r > -- Dave Baukus david.baukus@us.fujitsu.com 972-479-2491 Fujitsu Network Communications Richardson, Texas USA