From owner-freebsd-stable@FreeBSD.ORG Fri Feb 22 20:32:48 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E88781FA for ; Fri, 22 Feb 2013 20:32:48 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id A120B9E3 for ; Fri, 22 Feb 2013 20:32:48 +0000 (UTC) Received: from pampa.cs.huji.ac.il ([132.65.80.32] ident=danny) by kabab.cs.huji.ac.il with esmtp id 1U8zIT-000Kfk-M6; Fri, 22 Feb 2013 22:32:45 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.3 To: Rick Macklem Subject: Re: zfs/nfs/proftpd problem In-reply-to: <773493110.3210623.1361547227044.JavaMail.root@erie.cs.uoguelph.ca> References: <773493110.3210623.1361547227044.JavaMail.root@erie.cs.uoguelph.ca> Comments: In-reply-to Rick Macklem message dated "Fri, 22 Feb 2013 10:33:47 -0500." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Fri, 22 Feb 2013 22:32:45 +0200 From: Daniel Braniss Message-ID: Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Feb 2013 20:32:49 -0000 > Daniel Braniss wrote: > > after upgrading the 'ftp storage' from 8.3 to 9.1-stable, our ftp > > server is stuck. > > > > the old, (ProFTPD Version 1.3.2) and working till before the upgrade > > is stuck > > in nlmrcv: > > ... > > 10000 1213 992 0 44 0 7340 3692 nlmrcv D ?? 0:08.07 proftpd: > > ftp - > > crawl-66-249-73-193.googlebot.com: anonymous/googlebot@google.com: > > RETR 00690145.JPG (proftpd) > > ... > > > I suspect you know that this is waiting for a reply from some rpc.lockd. > > > so we upgraded the ftp server too, to 9.1/ProFTPD Version 1.3.4b and > > this one > > is stuck in rpccwnd: > > 10000 1197 984 0 20 0 32292 4792 rpccwnd D ?? 0:00.01 proftpd: ftp > > - > > mbpro.cs.huji.ac.il: anonymous/mozilla@example.com: LIST (proftpd) > > > This one is stuck in the client side of UDP for the krpc, in the > primitive congestion control stuff that is there. may be it's too primitive? > > > > > any wise suggestions :-) > > > Well, maybe not wise, but you may already be aware that NFS etc over > UDP and the NLM are two of my favourite things (especially the NLM). > > Basically, it appears to be having difficulties doing RPCs over UDP, > at least for the NLM (rpc.lockd), suggesting some transport related > issue. > > First, make sure rpc.statd and rpc.lockd are running on the NFS server > and all clients (or disable use of it via the "nolockd" mount option). all are ruuning bot rpc.statd and rpc.lockd > > You can also do a "netstat -s" and see if there is a non-zero count > for "fragments dropped due to timeout" in the IP section. (This happens > when your network fabric can't handle the burst of IP fragments > generated by a large RPC message over UDP.) > there are none on the cliet (the ftp server) > Things you could try: > - If you are using a udp mount for NFS... > - reduce your rsize and wsize (especially if "fragments dropped due > to timeout" is non-zero) > or > - switch to TCP > > If you are not using udp mounts, then the NLM (rpc.lockd) is using > UDP anyhow. If you don't need multiple NFS clients to see the file > locks, add "nolockd" to your mount(s). > > Beyond that, you'll need to capture packets and look at them in > wireshark, to see what is going on. > the mount is tcp. I have been staring at the tcpdump and nothing sticks out, but it's been a while since I looked at rpc traffic. some facts: it happens every time, with any ftp command, it gets stuck on either nlmrcv or rpccwnd, mostly the latter. I will try to disable the lock stuff, but isn't it avoiding the issue? > Good luck with it, rick thanks, danny