From owner-freebsd-fs@FreeBSD.ORG Thu Jun 10 23:32:41 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EEB4B1065673; Thu, 10 Jun 2010 23:32:40 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 82ED58FC17; Thu, 10 Jun 2010 23:32:40 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEABsTEUyDaFvK/2dsb2JhbACeeHG/CIUYBA X-IronPort-AV: E=Sophos;i="4.53,400,1272859200"; d="scan'208";a="80271424" Received: from fraser.cs.uoguelph.ca ([131.104.91.202]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 10 Jun 2010 19:32:37 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by fraser.cs.uoguelph.ca (Postfix) with ESMTP id 8A6A2109C350; Thu, 10 Jun 2010 19:32:39 -0400 (EDT) X-Virus-Scanned: amavisd-new at fraser.cs.uoguelph.ca Received: from fraser.cs.uoguelph.ca ([127.0.0.1]) by localhost (fraser.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bmtIrr2oPbuO; Thu, 10 Jun 2010 19:32:39 -0400 (EDT) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by fraser.cs.uoguelph.ca (Postfix) with ESMTP id 00592109C34B; Thu, 10 Jun 2010 19:32:38 -0400 (EDT) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o5ANmnW07793; Thu, 10 Jun 2010 19:48:49 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Thu, 10 Jun 2010 19:48:49 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Jeremy Chadwick In-Reply-To: <20100610133859.GA74094@icarus.home.lan> Message-ID: References: <20100608083649.GA77452@fupp.net> <20100609122517.GA16231@fupp.net> <20100610081710.GA64350@server.vk2pj.dyndns.org> <20100610110609.GA87243@fupp.net> <20100610114831.GB71432@icarus.home.lan> <20100610130307.GA33285@fupp.net> <20100610133859.GA74094@icarus.home.lan> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@FreeBSD.org, Peter Jeremy , Anders Nordby Subject: Re: Odd network issues on ZFS based NFS server X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jun 2010 23:32:41 -0000 On Thu, 10 Jun 2010, Jeremy Chadwick wrote: > > The interrupt rate for bge1 (irq26) is very high during the problem, > while otherwise is only ~6/sec. Shot in the dark, but this is probably > the cause of the packet loss you see. Oddly, your uhci2 interface (used > for USB) is also firing at a very high rate. I don't know if this is > the sign of a NIC problem, driver problem, or interrupt (think APIC?) > routing problem. > > Debugging this is beyond my capability, but folks like John Baldwin may > have some ideas on where to go from here. > > Also, have you used "netstat -ibn -I bge1" (to look at byte counters) or > "tcpdump -l -n -s 0 -i bge1" to watch network traffic live when this is > happening? The reason I ask is to determine if there's any chance this > box starts seeing problems due to DoS attacks or excessive LAN traffic > which is unexpected. Basically, be sure that all the network I/O going > on across bge1 is expected. > Yes, I think Jeremy is on the right track. I'd second the recommendation to look at traffic when it is happening. I might choose: tcpdump -s 0 -w -i bge1 and then load "" into wireshark, since wireshark is much better at making sense of NFS traffic. (Since the nfsd is at the top of the process list, it hints that there may be heavy nfs traffic being received by bge1.) If you do this tcpdump for a short period of time and then email "" to me as an attachment, I can take a look at it. (If the traffic isn't NFS, then there's not much point in doing this.) We might have a case where a client is retrying the same RPC (or RPC sequence) over and over and over again, my friend (sorry I couldn't resist:-). Given that you stated FreeBSD8.1-Prerelease I think you should have the patch, but please make sure that your sys/nfsserver/nfs_srvsubs.c is at least r206406. Let me know how it goes, rick