From owner-freebsd-net@FreeBSD.ORG Sat Apr 19 14:27:30 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5761D106564A for ; Sat, 19 Apr 2008 14:27:30 +0000 (UTC) (envelope-from mark@pogo.org.uk) Received: from metheny.ijneb.com (unknown [IPv6:2001:ba8:0:1ba:214:22ff:feb1:2693]) by mx1.freebsd.org (Postfix) with ESMTP id EDF4C8FC0C for ; Sat, 19 Apr 2008 14:27:29 +0000 (UTC) (envelope-from mark@pogo.org.uk) Received: from localhost ([127.0.0.1] ident=mark) by metheny.ijneb.com with esmtp (Exim 4.69) (envelope-from ) id 1JnE2K-00036g-WD for freebsd-net@freebsd.org; Sat, 19 Apr 2008 15:27:29 +0100 Date: Sat, 19 Apr 2008 15:27:28 +0100 (BST) From: Mark Hills To: freebsd-net@freebsd.org Message-ID: User-Agent: Alpine 1.10 (BSO 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII X-SA-Exim-Connect-IP: 127.0.0.1 X-SA-Exim-Mail-From: mark@pogo.org.uk Subject: read() returns ETIMEDOUT on steady TCP connection X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Apr 2008 14:27:30 -0000 Hello, I'm are having a trouble with TCP connections being dropped with "read: Operation timed out". What is unusual is that this is happening right in the middle of sending a steady stream of data with no network congestion. The system is FreeBSD 7 and a bespoke streaming server with 1Gbit connection. The server receives a 192kbps inbound stream over TCP, and broadcasts it over a large number of TCP streams. With no visible or obvious pattern, the inbound read() fails with ETIMEDOUT. The likelihood of this happening seems to increase as the number of audience connections increases. It's happens every few minutes even with a small audience (eg. 300 outbound connections and about 60mbit). It doesn't cough and splutter -- steady data is coming in, then it just drops the connection. systat doesn't show problems inbound; all packets received are delivered to the upper layer. But on outbound, there is consistent 'output drops': IP Output 7028 total packets sent 7028 - generated locally 314 - output drops As the number of outbound connections increases, the 'output drops' increases to around 10% of the total packets sent and maintains that ratio. There's no problems with network capacity. I've tried different servers, different network interfaces (bge, em), different kernel (7-RELEASE, 7-STABLE). Have also checked dev.bge.0.stats and dev.em.0.stats for CRC errors etc. which show no problems. 'netstat -m' doesn't show any reaching of mbuf and sbuf limits. The problem is seen in a dedicated, uncontended test environment. Can anyone explain why the packets are being dropped outbound, and how this could affect inbound TCP data in such an abrupt way? What can I do to solve this? Thanks, Mark