From owner-freebsd-net@FreeBSD.ORG Wed May 7 22:54:26 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4CEAB106568B for ; Wed, 7 May 2008 22:54:26 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id AF93A8FC15 for ; Wed, 7 May 2008 22:54:25 +0000 (UTC) (envelope-from andre@freebsd.org) Received: (qmail 54307 invoked from network); 7 May 2008 21:56:11 -0000 Received: from localhost (HELO [127.0.0.1]) ([127.0.0.1]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 7 May 2008 21:56:10 -0000 Message-ID: <48223324.6070203@freebsd.org> Date: Thu, 08 May 2008 00:54:28 +0200 From: Andre Oppermann User-Agent: Thunderbird 1.5.0.14 (Windows/20071210) MIME-Version: 1.0 To: Deng XueFeng , Mark Hills References: <20080506133208.C2EC.B627C632@gmail.com> In-Reply-To: <20080506133208.C2EC.B627C632@gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: read() returns ETIMEDOUT on steady TCP connection X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 May 2008 22:54:26 -0000 I've looked at the code paths again. There are two possibilities: a) the mbuf allocator has some anomaly where it rejects memory requests but doesn't update the statistics (the code is there however). b) the error doesn't come from the mbuf allocation but from ip_output() and further down the chain. To differentiate please try this updated patch and report the log output again (don't forget to set net.inet.tcp.log_debug=1): http://people.freebsd.org/~andre/tcp_output-error-log.diff -- Andre Deng XueFeng wrote: > hi > I'am also meet this problem in my mss server(missey streaming server). > one encoder push stream to mss, then run 100 client player playing the > sream, as the client number increase, mss will occur this error sooner or later > like this: > I'am using kqueue, and will got a event with EV_EOF and fflags = > ETIMEDOUT, > if i ignore EV_EOF flag, then ETIMEDOUT will be return by read(2), > > and the tcpdump also show that server will send RST packet to encoder. > > >> Hello, >> >> I'm are having a trouble with TCP connections being dropped with "read: >> Operation timed out". What is unusual is that this is happening right in >> the middle of sending a steady stream of data with no network congestion. >> >> The system is FreeBSD 7 and a bespoke streaming server with 1Gbit >> connection. The server receives a 192kbps inbound stream over TCP, and >> broadcasts it over a large number of TCP streams. >> >> With no visible or obvious pattern, the inbound read() fails with >> ETIMEDOUT. The likelihood of this happening seems to increase as the >> number of audience connections increases. It's happens every few minutes >> even with a small audience (eg. 300 outbound connections and about >> 60mbit). >> >> It doesn't cough and splutter -- steady data is coming in, then it just >> drops the connection. >> >> systat doesn't show problems inbound; all packets received are delivered >> to the upper layer. But on outbound, there is consistent 'output drops': >> >> IP Output >> 7028 total packets sent >> 7028 - generated locally >> 314 - output drops >> >> As the number of outbound connections increases, the 'output drops' >> increases to around 10% of the total packets sent and maintains that >> ratio. There's no problems with network capacity. >> >> I've tried different servers, different network interfaces (bge, em), >> different kernel (7-RELEASE, 7-STABLE). Have also checked dev.bge.0.stats >> and dev.em.0.stats for CRC errors etc. which show no problems. 'netstat >> -m' doesn't show any reaching of mbuf and sbuf limits. The problem is seen >> in a dedicated, uncontended test environment. >> >> Can anyone explain why the packets are being dropped outbound, and how >> this could affect inbound TCP data in such an abrupt way? What can I do to >> solve this? >> >> Thanks, >> >> Mark >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >