From owner-freebsd-net@FreeBSD.ORG Tue Apr 22 23:47:13 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 06AA5106564A for ; Tue, 22 Apr 2008 23:47:13 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 7ADAE8FC0A for ; Tue, 22 Apr 2008 23:47:12 +0000 (UTC) (envelope-from andre@freebsd.org) Received: (qmail 46825 invoked from network); 22 Apr 2008 22:51:45 -0000 Received: from localhost (HELO [127.0.0.1]) ([127.0.0.1]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 22 Apr 2008 22:51:45 -0000 Message-ID: <480E7901.5000804@freebsd.org> Date: Wed, 23 Apr 2008 01:47:13 +0200 From: Andre Oppermann User-Agent: Thunderbird 1.5.0.14 (Windows/20071210) MIME-Version: 1.0 To: Mark Hills References: <20080420025010.GJ73016@server.vk2pj.dyndns.org> <480BBD7E.8010700@freebsd.org> <480C9AC6.8090802@freebsd.org> In-Reply-To: <480C9AC6.8090802@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Peter Jeremy , freebsd-net@freebsd.org Subject: Re: read() returns ETIMEDOUT on steady TCP connection X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Apr 2008 23:47:13 -0000 Andre Oppermann wrote: > Mark Hills wrote: >> On Mon, 21 Apr 2008, Andre Oppermann wrote: >> >>> Mark Hills wrote: >>>> On Sun, 20 Apr 2008, Peter Jeremy wrote: >> >>>>> I can't explain the problem but it definitely looks like a resource >>>>> starvation issue within the kernel. >>>> >>>> I've traced the source of the ETIMEDOUT within the kernel to >>>> tcp_timer_rexmt() in tcp_timer.c: >>>> >>>> if (++tp->t_rxtshift > TCP_MAXRXTSHIFT) { >>>> tp->t_rxtshift = TCP_MAXRXTSHIFT; >>>> tcpstat.tcps_timeoutdrop++; >>>> tp = tcp_drop(tp, tp->t_softerror ? >>>> tp->t_softerror : ETIMEDOUT); >>>> goto out; >>>> } >>> >>> Yes, this is related to either lack of mbufs to create a segment >>> or a problem in sending it. That may be full interface queue, a >>> bandwidth manager (dummynet) or some firewall internally rejecting >>> the segment (ipfw, pf). Do you run any firewall in stateful mode? >> >> There's no firewall running. >> >>>> I'm new to FreeBSD, but it seems to implies that it's reaching a >>>> limit of a number of retransmits of sending ACKs on the TCP >>>> connection receiving the inbound data? But I checked this using >>>> tcpdump on the server and could see no retransmissions. >>> >>> When you have internal problems the segment never makes it to the >>> wire and thus you wont see it in tcpdump. >>> >>> Please report the output of 'netstat -s -p tcp' and 'netstat -m'. >> >> Posted below. You can see it it in there: "131 connections dropped by >> rexmit timeout" >> >>>> As a test, I ran a simulation with the necessary changes to increase >>>> TCP_MAXRXTSHIFT (including adding appropriate entries to >>>> tcp_sync_backoff[] and tcp_backoff[]) and it appeared I was able to >>>> reduce the frequency of the problem occurring, but not to a usable >>>> level. >>> >>> Possible causes are timers that fire too early. Resource starvation >>> (you are doing a lot of traffic). Or of course some bug in the code. >> >> As I said in my original email, the data transfer doesn't stop or >> splutter, it's simply cut mid-flow. Sounds like something happening >> prematurely. >> >> Thanks for the help, > > The output doesn't show any obvious problems. I have to write some > debug code to run on your system. I'll do that later today if time > permits. Otherwise tomorrow. http://people.freebsd.org/~andre/tcp_output-error-log.diff Please apply this patch and enable the sysctl net.inet.tcp.log_debug=1 and report any output. You likely get some (normal) noise from syncache. What we are looking for is reports from tcp_output. -- Andre