From owner-freebsd-net@FreeBSD.ORG Mon Jan 17 00:56:21 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F3B6106566C for ; Mon, 17 Jan 2011 00:56:21 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 1BF6C8FC14 for ; Mon, 17 Jan 2011 00:56:20 +0000 (UTC) Received: by pwi10 with SMTP id 10so875574pwi.13 for ; Sun, 16 Jan 2011 16:56:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:date:to:cc:subject:message-id:reply-to :references:mime-version:content-type:content-disposition :in-reply-to:user-agent; bh=hmgm6vgULfGDISEe0kyFA6a2pEEa0cilVmqnOLa0a+c=; b=s2H+pENqh4AANeg6LG+rfDEXMs9c8T2/Mrshp0Ok2WxwZDl3cw42Ia/FQDOzQEsSB4 IDyFozWBv0bETH/RePre20Py+uY2z/Qszby+gPUoG1dsWqVPCKdxYc6A9JCXOC+OBhHd 0LoqpucA7HDMG4hEmA3xStR5YqASCJSi+YmLU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=oj4IyKKm8tOqkpSIiGFJRXzM+8+EiifINZ7YxglPpcFgGD92Oqc8Th+tJ4Em/O0x2J amfi7Z2gYUX+3As2Iz7eNHPthQh1N0Ti7EQ8Mcs8XKJLR+g/Yf3aBnObfN92bFc9ebc5 JwSszgePw4FEq1w1Fa+EUx/X/Xnjr7G/8KpuQ= Received: by 10.142.194.5 with SMTP id r5mr3002411wff.157.1295225778785; Sun, 16 Jan 2011 16:56:18 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id w42sm5781917wfh.3.2011.01.16.16.56.16 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sun, 16 Jan 2011 16:56:17 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Sun, 16 Jan 2011 16:55:24 -0800 From: Pyun YongHyeon Date: Sun, 16 Jan 2011 16:55:24 -0800 To: Rick Macklem Message-ID: <20110117005524.GA1305@michelle.cdnetworks.com> References: <708793006.292748.1295186099577.JavaMail.root@erie.cs.uoguelph.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <708793006.292748.1295186099577.JavaMail.root@erie.cs.uoguelph.ca> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, Ronald Klop Subject: Re: bogus 0 len IP packet, was: Hang in VOP_LOCK1_APV on 8-STABLE with NFS. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Jan 2011 00:56:21 -0000 On Sun, Jan 16, 2011 at 08:54:59AM -0500, Rick Macklem wrote: > Ronald has reported having a problem with the FreeBSD NFS client using > 8.2-prerelease. I've redirected it here, since it looks like there is > a TCP/IP issue that is causing it. > > > > > >> > > >> These are the links to the dumps: > > >> http://klop.ws/~ronald/nfs-problem/procstat.nolockd > > >> http://klop.ws/~ronald/nfs-problem/ps.nolockd > > >> http://klop.ws/~ronald/nfs-problem/linux5.nfs.nolockd.dump.gz > > >> http://klop.ws/~ronald/nfs-problem/linux5.nfs.with_rpc_patch.dump.gz > > >> > > I looked at the last of these via wireshark and it seems that the FreeBSD > client is sending bogus TCP/IP packets with a IP length == 0. (If you look > at the above dump (...with_rpc_patch.dump.gz), the first one is at packet #46, > then #3024, then persistently starting at #3234.) Basically the packet looks > like: > frame len: 4446 > MAC: dst 00:0d:56:70:b7:6c src b8:ac:6f:47:73:6e type: 08 00 (IP) > IP > version: 4 > header length: 20 > differentiated services field: 0x00 > total length: 0 > - followed by what looks like a legitimate TCP/IP packet > > Here's the first bytes of the raw packet data: > 00 0d 56 70 b7 6c b8 ac 6f 47 73 6e 08 00 > 45 00 00 00 ... > > After this packet is sent to the Linux server, it replies with a TCP ack, > which gets ACK'd from FreeBSD as well. For the persistent case, it just > keeps doing this (bogus 0 length packet from FreeBSD -> Linux server, > followed by the two TCP ack packets) over and over and over again, to > the end of the dump. > > So, does anyone have an idea why the IP length field would be set to 0 > for these TCP/IP packets? > > Here's some info from Ronald w.r.t. his hardware. (All I can think of is > that he could try disabling TSO, etc?) > > Thanks in advance for any help with this, rick > It seems that issue came from TSO. Driver will set ip_len and ip_sum field to 0 before passing the TCP segment to controller. The failed length were 4446, 5858, 3034 and 4310 and the total number of such frames are more than 35k within 90 seconds. Since failed length 4310 is continuously repeated I guess there is edge case where em(4) didn't free failed TCP segment for TSO. I remember there was commit to HEAD(r217295) which could be related with this issue. > > > > I just looked at the last dump and there seems to be a network > > > issue. > > > (It first shows up at packet #46, then again at #3025, then > > > persistently > > > starting at #3234.) > > > > > > I'd like to post on freebsd-net@ to see if anyone more conversant > > > with > > > TCP/IP can look, but first I'd like to get a little more info on > > > your > > > hardware/software config. > > > > > > In particular, what network hardware does the FreeBSD client use? > > > > > > And I assume the server is some variant of Linux? > > > > > > Thanks for creating the tcpdumps, rick > > > ps: If you look on wireshark, the problem seems to start with a > > > badly formed IP datagram that then causes acks in both > > > directions. > > > > We are getting off list now. I don't know how good that is. > > > I've redirected it to freebsd-net@ in the hopes that networking folks > can help. > > > But here is some info. I also noticed the bad packets. And my computer > > is > > resending the same info at quite a high rate (MB/s). > > The server is an up-to-date Linux Debian 5 with a 2.6.26-1-686-bigmem > > kernel. Colleagues using Linux clients don't have these problems. > > > > dmesg | grep em0 > > em0: port 0xece0-0xecff > > mem > > 0xf7fe0000-0xf7ffffff,0xf7fd9000-0xf7fd9fff irq 21 at device 25.0 on > > pci0 > > em0: Using an MSI interrupt > > em0: [FILTER] > > em0: Ethernet address: b8:ac:6f:47:73:6e > > > > pciconf -lv > > em0@pci0:0:25:0: class=0x020000 card=0x02761028 chip=0x10de8086 > > rev=0x02 hdr=0x00 > > vendor = 'Intel Corporation' > > device = 'Intel Gigabit network connection (82567LM-3 )' > > class = network > > subclass = ethernet > > > > > > [root@ronald ~]# ifconfig em0 > > em0: flags=8843 metric 0 mtu > > 1500 > > options=219b > > ether b8:ac:6f:47:73:6e > > inet 10.1.20.49 netmask 0xffffff00 broadcast 10.1.20.255 > > media: Ethernet autoselect (100baseTX ) > > status: active > > > > Thanks for looking into this. If you need more just let me know. I can > > also reproduce the problem and send nfsstat or netstat output or .... > > > > Thanks, > > Ronald.