From owner-freebsd-net@FreeBSD.ORG Sat May 10 09:08:44 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5711A106566C; Sat, 10 May 2008 09:08:44 +0000 (UTC) (envelope-from tim@gebbettco.com) Received: from outmail137080.authsmtp.com (outmail137080.authsmtp.com [62.13.137.80]) by mx1.freebsd.org (Postfix) with ESMTP id 056488FC18; Sat, 10 May 2008 09:08:43 +0000 (UTC) (envelope-from tim@gebbettco.com) Received: from mail-c188.authsmtp.com (mail-c188.authsmtp.com [62.13.128.25]) by punt4.authsmtp.com (8.14.1/8.14.1/Kp) with ESMTP id m4A8ojW1084652; Sat, 10 May 2008 09:50:45 +0100 (BST) Received: from [10.111.0.10] (host217-34-40-180.in-addr.btopenworld.com [217.34.40.180]) (authenticated bits=0) by mail.authsmtp.com (8.14.2/8.14.2/Kp) with ESMTP id m4A8ohpf073413; Sat, 10 May 2008 09:50:44 +0100 (BST) Message-ID: <482561F3.6080701@gebbettco.com> Date: Sat, 10 May 2008 09:50:59 +0100 From: Tim Gebbett User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Andre Oppermann References: <4822BABB.4020407@freebsd.org> <4824211C.9090105@freebsd.org> In-Reply-To: <4824211C.9090105@freebsd.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Server-Quench: 2d2e8dc9-1e6e-11dd-aecc-001871e930f4 X-AuthRoute: OCdxaQwcClZJTQEy BS4OBiJcVQ4iYBZL BAkGMA9GIUINWEQN c1ADdh12OUxbHwkB C3YLUl5RUFd1XS13 ag1PbgFDZkpQXg1q T0pMQFdNFEs1BW0J WUZeBBl1dAFHezBz YkRlEHIPXhFyIRV0 X08BEzgbZGY0PH1O BRMLagNUcQtNf0pM aVApUD1vNG8XJC8x E09ydyo8IShFLj8d bz0sCH8+Zns3Vjk6 DwseEDwjDAU5bB17 JBsgLFMXAEcWNC05 X-Authentic-SMTP: 61633239343939.cat.dmpriest.net.uk:953/Kp X-Report-SPAM: If SPAM / abuse - report it at: http://www.authsmtp.com/abuse Cc: freebsd-net@freebsd.org, Deng XueFeng , Mark Hills Subject: Re: read() returns ETIMEDOUT on steady TCP connection X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 May 2008 09:08:44 -0000 Hi Andre, did some careful testing yesterday and last night. I seem to be still hitting an unknown buffer although the probem is much alleviated. The system achieved a 7hour run at 500mbit where ETIMEDOUT occured. I was feeding 11 other streams to the server whos counters show an uninterrupted eleven hours. The feeder streams are from the same source, so it is unlikely that the one feeding the test could of had a problem without affecting the counters of the others. sysctls are: (loader.conf) hw.em.txd=4096 net.inet.tcp.sendspace=78840 net.inet.tcp.recvspace=78840 kern.ipc.nmbjumbop=51200 kern.ipc.nmbclusters=78840 kern.maxfiles=50000 IP stats are miraculously improved, going from a 10% packet loss within stack (output drops) to a consistent zero at peaks of 80000 pps. I believe the problem is now being shunted to the NIC from the following output: dev.em.0.debug=1 < em0: Adapter hardware address = 0xc520b224 < em0: CTRL = 0x48f00249 RCTL = 0x8002 < em0: Packet buffer = Tx=16k Rx=48k < em0: Flow control watermarks high = 47104 low = 45604 < em0: tx_int_delay = 66, tx_abs_int_delay = 66 < em0: rx_int_delay = 0, rx_abs_int_delay = 66 < em0: fifo workaround = 0, fifo_reset_count = 0 < em0: hw tdh = 3285, hw tdt = 3285 < em0: hw rdh = 201, hw rdt = 200 < em0: Num Tx descriptors avail = 4096 < em0: Tx Descriptors not avail1 = 4591225 < em0: Tx Descriptors not avail2 = 0 < em0: Std mbuf failed = 0 < em0: Std mbuf cluster failed = 0 < em0: Driver dropped packets = 0 < em0: Driver tx dma failure in encap = 0 dev.em.0.stats=1 < em0: Excessive collisions = 0 < em0: Sequence errors = 0 < em0: Defer count = 0 < em0: Missed Packets = 16581181 < em0: Receive No Buffers = 74605555 < em0: Receive Length Errors = 0 < em0: Receive errors = 0 < em0: Crc errors = 0 < em0: Alignment errors = 0 < em0: Collision/Carrier extension errors = 0 < em0: RX overruns = 289717 < em0: watchdog timeouts = 0 < em0: XON Rcvd = 0 < em0: XON Xmtd = 0 < em0: XOFF Rcvd = 0 < em0: XOFF Xmtd = 0 < em0: Good Packets Rcvd = 848158221 < em0: Good Packets Xmtd = 1080368640 < em0: TSO Contexts Xmtd = 0 < em0: TSO Contexts Failed = 0 Does the counter 'Tx Descriptors not avail1' indicate lack of decriptors at the time not available, and would this be symptomatic of something Mark suggested: "(the stack) needs to handle local buffer fills not as a failed attempt on transmission that increments the retry counter, a possible better strategy required for backoff when the hardware buffer is full?" Thanks for your continued time and effort - Tim Andre Oppermann wrote: > Tim Gebbett wrote: >> Hi all, >> >> applied the patch, >> >> Well before a ETIMEDOUT error occurred (around 60secs), the tcp debug >> started venting massive >> quantities of tcp_output error 55 while sending with syncache noise: > > The error seems to be coming from the interface send queue which hits > the limit. > If you are using em(4) network interface please add this line to > loader.conf(5): > > hw.em.txd=1024 > > Or even more if problems persist. The maximum is 4096. >