From owner-freebsd-net@freebsd.org Thu Nov 5 00:06:11 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B019AA26BDB for ; Thu, 5 Nov 2015 00:06:11 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 5501A1CB8 for ; Thu, 5 Nov 2015 00:06:10 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:O4BAJxbhODJB4wtkPiRQ9Yb/LSx+4OfEezUN459isYplN5qZpcm8bnLW6fgltlLVR4KTs6sC0LqL9fu5EjVRqb+681k8M7V0HycfjssXmwFySOWkMmbcaMDQUiohAc5ZX0Vk9XzoeWJcGcL5ekGA6ibqtW1aJBzzOEJPK/jvHcaK1oLsh730oMSYOlQArQH+SI0xBS3+lR/WuMgSjNkqAYcK4TyNnEF1ff9Lz3hjP1OZkkW0zM6x+Jl+73YY4Kp5pIYTGZn9Kos1V6ZZEHwFp2AzrJnkuAPZTBfJ5WYRUmM+mxdJRQ3d41f2U8GinDH9s79H2SKZdej/RrMwVDHqu71uQRTrjCoCHyM+/3zajtRwyqlS9kHy7ydjypLZNdnGfMF1ebnQKIsX X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DWAwDJnDpW/61jaINehA4sAUIGvW0BDYFeFwqFJ0oCgXYUAQEBAQEBAQGBCYIuggcBAQEDAQEBASAEJyALEAIBCBgCAg0ZAgInAQkmAgQIBwQBHASIBQgNsRWRAgEBAQEBAQEDAQEBAQEBAQEXBIEBhVOEfoQ7AQEFgzOBQwWOEYg3hR2FH4RBh2SKMIhRAh8BAUKEIiA0B4NsOoEHAQEB X-IronPort-AV: E=Sophos;i="5.20,245,1444708800"; d="scan'208";a="249300968" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 04 Nov 2015 19:05:01 -0500 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 095E215F55D; Wed, 4 Nov 2015 19:05:00 -0500 (EST) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 4Oyubks-akNW; Wed, 4 Nov 2015 19:05:00 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 0334015F565; Wed, 4 Nov 2015 19:04:59 -0500 (EST) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id zp73noTHgB9Y; Wed, 4 Nov 2015 19:04:59 -0500 (EST) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 8BE0B15F55D; Wed, 4 Nov 2015 19:04:59 -0500 (EST) Date: Wed, 4 Nov 2015 19:04:59 -0500 (EST) From: Rick Macklem To: Gerrit =?utf-8?B?S8O8aG4=?= Cc: freebsd-net@freebsd.org Message-ID: <1569514731.73706862.1446681899411.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <20151104144354.acccf3c7315a6009264d755d@aei.mpg.de> References: <20151104144354.acccf3c7315a6009264d755d@aei.mpg.de> Subject: Re: strange nfs/rsync stalls MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.10] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: strange nfs/rsync stalls Thread-Index: cL59sYuzEHXL83gixW2iLMRLk2zwGA== X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2015 00:06:11 -0000 Gerritt Kuhn wrote: > Hi all, > > I see some weird (I think) NFS issue here: > I have a NFS server running 10.2-RELEASE exporting a directory to a NFS > client running 10.2-STABLE (as of yesterday, tried that update from 10.1 > to see if it would fix this issue, but it stayed with me). I use rsync on > the client to copy data from local disks to nfs. > This appears to work for a few minutes, then the rsync process gets stuck > in state "NEWNFS". Messages "newnfs server servername:/data/systems: not > responding" start appearing and get repeated over and over. > > What I think is strange: I cannot see any issue apart from these rsync > jobs getting stuck. I can still access the nfs-mounted data on the client > just fine in a different shell. I can read, write, do whatever I like, but > the rsync job never moves on. > I have a couple of other machines that use the same server in the same way, > none of them ever showed this behaviour. Are there any suggestions how to > debug/fix this? > You didn't mention what kind of net interface you are using. 1 - disable TSO if you haven't already done so. 2 - try capturing packets when it is stuck and nothing else is being done on the mount point and see if any traffic is being sent to the server. # tcpdump -s 0 -w .pcap host - run on the client for a little while should do it. --> Look at .pcap in wireshark, since it knows NFS, unlike tcpdump. 3 - Do "nfsstat -e -c" on the client a few times when it is stuck and see what RPC counts are increasing. (Again when nothing else is being done on the mount.) Hopefully, the above will give you some idea of what is going on. (Btw, all processes stuck in "newnfs" tells you is that they are waiting for an NFS client vnode lock held by some other thread.) Good luck with it, rick > > cu > Gerrit > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >