From owner-freebsd-net@FreeBSD.ORG Fri May 9 02:48:28 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA6D41065679 for ; Fri, 9 May 2008 02:48:28 +0000 (UTC) (envelope-from dengxf@gmail.com) Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.174]) by mx1.freebsd.org (Postfix) with ESMTP id BBAB58FC16 for ; Fri, 9 May 2008 02:48:28 +0000 (UTC) (envelope-from dengxf@gmail.com) Received: by wf-out-1314.google.com with SMTP id 28so921229wfa.7 for ; Thu, 08 May 2008 19:48:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:date:from:to:subject:cc:in-reply-to:references:message-id:mime-version:content-type:content-transfer-encoding:x-mailer; bh=HSWNnPi/gm25S3XrkTTX49///QL8VqXO5aZitHeiiL8=; b=HqCxcg4WrVvgcXjHQNfuTcFEXZcARaMtZJVCqnaCSqftPfEcXMjBdQFQ5wbTejb4d8ntaMacO1QxOIvMdB6n6sGN/1MgbxC7b+9hu/XK7DEQSQWuXRnpIlmZxKZ1EIj61Dgu0wy/xSKKiZ5yaitQto0sY5uTA5GxCizmvKWO9fg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:subject:cc:in-reply-to:references:message-id:mime-version:content-type:content-transfer-encoding:x-mailer; b=fQhVfUFAaJd6M7PLJlH5HoUpT9taa9QQPbo+FwyevOmpcIB0BAER7uMykPT9jG6Rl81NBXzTVzNVJqShBV8EFaZNIDDEEdxKxXhm6XNfp91drMpM7LNPCfiYp8BLH0BvZj2jD2/TwaPt/sjB9Z0NRnl7R1RUMCSJm2n96HTc7Tw= Received: by 10.142.216.9 with SMTP id o9mr1713310wfg.172.1210301308436; Thu, 08 May 2008 19:48:28 -0700 (PDT) Received: from ?192.168.0.160? ( [218.94.128.114]) by mx.google.com with ESMTPS id 24sm9249226wff.12.2008.05.08.19.48.22 (version=SSLv3 cipher=RC4-MD5); Thu, 08 May 2008 19:48:25 -0700 (PDT) Date: Fri, 09 May 2008 10:48:37 +0800 From: Deng XueFeng To: Andre Oppermann In-Reply-To: <482324F9.7030802@freebsd.org> References: <482324F9.7030802@freebsd.org> Message-Id: <20080509104700.A502.B627C632@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Mailer: Becky! ver. 2.45.02 [CN] Cc: Tim@gebbettco.com, Mark Hills , freebsd-net@freebsd.org Subject: Re: read() returns ETIMEDOUT on steady TCP connection X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 May 2008 02:48:29 -0000 hi, applied the patch for 6.2, rebuild & install kernel, but nothing changed. ETIMEDOUT still occur. > Hi Tim, > > looking at the ip_output() path there are some places that can > return ENOBUFS: > > a) interface queue length check > > b) packet filter > > c) destination address rewrite through NAT > > d) if_output() call > > e) IP fragmentation if DF was not set > > The first one of those is the most likely to be the source of the > error. The output interface queue length in read unlocked and may > be a stale value on an SMP machine. Further down in ether_output() > there are some further possibilities for ENOBUFS errors. But lets > concentrate on a) first. > > For testing purposes please apply the following patch to ip_output(): > > ------------------------------------------------------------------- > cvs diff -up ip_output.c > Index: ip_output.c > =================================================================== > RCS file: /home/ncvs/src/sys/netinet/ip_output.c,v > retrieving revision 1.276.2.1 > diff -u -p -r1.276.2.1 ip_output.c > --- ip_output.c 9 Mar 2008 21:04:54 -0000 1.276.2.1 > +++ ip_output.c 8 May 2008 16:02:32 -0000 > @@ -370,7 +370,7 @@ again: > ip->ip_src = IA_SIN(ia)->sin_addr; > } > } > - > +#if 0 > /* > * Verify that we have any chance at all of being able to queue the > * packet or packet fragments, unless ALTQ is enabled on the given > @@ -390,7 +390,7 @@ again: > ifp->if_snd.ifq_drops += (ip->ip_len / ifp->if_mtu + 1); > goto bad; > } > - > +#endif > /* > * Look for broadcast address and > * verify user is allowed to send > ------------------------------------------------------------------- > > If there is a real interface output queue full event the IFQ_HANDOFF() > and IFQ_ENQUEUE() macros will report it too. Then we can focus on the > interface queues. > > -- > Andre > > Tim Gebbett wrote: > > Hi all, > > > > applied the patch, > > > > Well before a ETIMEDOUT error occurred (around 60secs), the tcp debug started venting massive quantities of tcp_output error 55 while sending with syncache noise: > > > > > > y 8 12:14:26 timtest kernel: :63859 to [192.168.5.40]:80; tcp_output: error 55 whilTeC Ps:e n[d1i9n2g. 1(6i8p._5o.u4t3p]u:t64 0371 )t > > May 8 12:14:26 timtest kernel: o > > May 8 12:14:26 timtest kernel: [192.168.5.40]:80; tcp_output: error 55 while sendingT > > May 8 12:14:26 timtest kernel: C > > May 8 12:14:26 timtest kernel: P: [192.168.5.43]:63859 to [192.168.5.40]:80; tcp_output: error 55 while sending > > May 8 12:14:26 timtest kernel: TCP: [192.168.5.43]:64037 to [192.168.5.40]:80; tcp_output: error 55 while sending (ip_output 1) > > May 8 12:14:26 timtest kernel: TCP: [192.168.5.43]:64037 to [192.168.5.40]:80; tcp_output: error 55 while sending > > May 8 12:14:26 timtest kernel: TCP: [192.168.5.43]:63857 to [192.168.5.40]:80; tcp_output: error 55 while sending (ip_output 1) > > May 8 12:14:26 timtest kernel: TCP: [192.168.5.42]:56421 toT C[P1:9 2[.119628..51.6480.]5:.8403;] :6t3c8p57_ otuot p[u1t9:2 .e1r68r.o5r. 40]:8505; whticlpe_ osuetnpduitn:g e(rirpo_ro utpu5t5 w1h)i > > > > interspersed with clean blocks of 20 entries or so of: > > > > May 8 12:14:26 timtest kernel: TCP: [192.168.5.43]:64037 to [192.168.5.40]:80; tcp_output: error 55 while sending (ip_output 1) > > May 8 12:14:26 timtest kernel: TCP: [192.168.5.43]:64037 to [192.168.5.40]:80; tcp_output: error 55 while sending > > May 8 12:14:26 timtest kernel: TCP: [192.168.5.43]:63857 to [192.168.5.40]:80; tcp_output: error 55 while sending (ip_output 1) > > > > > > The output did not look appreciably different when the ETIMEDOUT occurred. > > > > On stopping the client test program: > > > > May 8 12:14:46 timtest kernel: TCP: [192.168.5.43]:63978 to [192.168.5.40]:80 tcpflags 0x4; syncache_chkrst: Spurious RST without matching syncache entry (possibly syncookie only), segment ignored > > May 8 12:14:46 timtest kernel: TCP: [192.168.5.43]:63978 to [192.168.5.40]:80 tcpflags 0x4; syncache_chkrst: Spurious RST without matching syncache entry (possibly syncookie only), segment ignored > > May 8 12:14:46 timtest kernel: TCP: [192.168.5.43]:63978 to [192.168.5.40]:80 tcpflags 0x4; syncache_chkrst: Spurious RST without matching syncache entry (possibly syncookie only), segment ignored > > > > netstat -m > > > > 258/11007/11265 mbufs in use (current/cache/total) > > 256/1596/1852/25600 mbuf clusters in use (current/cache/total/max) > > 256/1536 mbuf+clusters out of packet secondary zone in use (current/cache) > > 0/7585/7585/51200 4k (page size) jumbo clusters in use (current/cache/total/max) > > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > > 576K/36283K/36860K bytes allocated to network (current/cache/total) > > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > > 0/4/6656 sfbufs in use (current/peak/max) > > 0 requests for sfbufs denied > > 0 requests for sfbufs delayed > > 0 requests for I/O initiated by sendfile > > 0 calls to protocol drain routines > > > > Thanks again for your help - Tim > > > > > > > > > > > > > > > > > > -- Deng XueFeng