From owner-freebsd-stable@FreeBSD.ORG Fri Feb 22 23:28:05 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 4C29E9D5 for ; Fri, 22 Feb 2013 23:28:05 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 11F431B9 for ; Fri, 22 Feb 2013 23:28:04 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqEEALz9J1GDaFvO/2dsb2JhbABEDoZAuBWCWoEic4IfAQEEASNWBRYOCgICDRkCWQaIHwasfpIkgSOMMAeBADQHgi2BEwOIaI1SkGOCSF2BTAEeHg X-IronPort-AV: E=Sophos;i="4.84,717,1355115600"; d="scan'208";a="17877487" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 22 Feb 2013 18:27:57 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id D5619B4033; Fri, 22 Feb 2013 18:27:57 -0500 (EST) Date: Fri, 22 Feb 2013 18:27:57 -0500 (EST) From: Rick Macklem To: Daniel Braniss Message-ID: <749270079.3229291.1361575677819.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: Subject: Re: zfs/nfs/proftpd problem MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Feb 2013 23:28:05 -0000 Daniel Braniss wrote: > > Daniel Braniss wrote: > > > after upgrading the 'ftp storage' from 8.3 to 9.1-stable, our ftp > > > server is stuck. > > > > > > the old, (ProFTPD Version 1.3.2) and working till before the > > > upgrade > > > is stuck > > > in nlmrcv: > > > ... > > > 10000 1213 992 0 44 0 7340 3692 nlmrcv D ?? 0:08.07 proftpd: > > > ftp - > > > crawl-66-249-73-193.googlebot.com: anonymous/googlebot@google.com: > > > RETR 00690145.JPG (proftpd) > > > ... > > > > > I suspect you know that this is waiting for a reply from some > > rpc.lockd. > > > > > so we upgraded the ftp server too, to 9.1/ProFTPD Version 1.3.4b > > > and > > > this one > > > is stuck in rpccwnd: > > > 10000 1197 984 0 20 0 32292 4792 rpccwnd D ?? 0:00.01 proftpd: ftp > > > - > > > mbpro.cs.huji.ac.il: anonymous/mozilla@example.com: LIST (proftpd) > > > > > This one is stuck in the client side of UDP for the krpc, in the > > primitive congestion control stuff that is there. > may be it's too primitive? > Yes, but the only alternative is no congestion avoidance at all. The RPC RTT includes round trip time for the messages plus the delay for processing the RPC at the server. The latter is highly variable and depends greatly on what the RPC is and how heavily loaded the server is. (The pre-krpc NFS client could do a little better, since it "knew" what the RPC was and could assume "writes" would take a lot longer than a Getattr. A generic krpc implementation can't know anything about what the RPC does.) If you're network fabric needs congestion control to achieve low loss of packets, then TCP is the way to go. Remember that, if any packet in a request/response is lost, the entire RPC must be retried when running over UDP. > > > > > > > > any wise suggestions :-) > > > > > Well, maybe not wise, but you may already be aware that NFS etc over > > UDP and the NLM are two of my favourite things (especially the NLM). > > > > Basically, it appears to be having difficulties doing RPCs over UDP, > > at least for the NLM (rpc.lockd), suggesting some transport related > > issue. > > > > First, make sure rpc.statd and rpc.lockd are running on the NFS > > server > > and all clients (or disable use of it via the "nolockd" mount > > option). > all are ruuning bot rpc.statd and rpc.lockd > > > > You can also do a "netstat -s" and see if there is a non-zero count > > for "fragments dropped due to timeout" in the IP section. (This > > happens > > when your network fabric can't handle the burst of IP fragments > > generated by a large RPC message over UDP.) > > > > there are none on the cliet (the ftp server) > > > Things you could try: > > - If you are using a udp mount for NFS... > > - reduce your rsize and wsize (especially if "fragments dropped > > due > > to timeout" is non-zero) > > or > > - switch to TCP > > > > If you are not using udp mounts, then the NLM (rpc.lockd) is using > > UDP anyhow. If you don't need multiple NFS clients to see the file > > locks, add "nolockd" to your mount(s). > > > > Beyond that, you'll need to capture packets and look at them in > > wireshark, to see what is going on. > > > the mount is tcp. > I have been staring at the tcpdump and nothing sticks out, but it's > been a > while > since I looked at rpc traffic. > I'll assume you are looking at it using wireshark (tcpdump doesn't understand these protocols). You would be looking for repeated RPC request messages without a corresponding RPC reply from the other end. They would be NLM or NSM protocol RPCs. (Since you mentioned that your NFS mount was TCP, it must be the NLM and/or associated NSM stuff that is using UDP, I think.) > some facts: > it happens every time, with any ftp command, it gets stuck on either > nlmrcv > or rpccwnd, mostly the latter. > I will try to disable the lock stuff, but isn't it avoiding the issue? > If you mean "avoiding the use of a fundamentally flawed protocol designed in the 1980s for a handful of locally connected machines that always remain up with the same fixed hostname/ip address" then, yes, you are trying to avoid the issue. Further to that, the protocol was never well published, so implementations "guessed" w.r.t. the semantics for things like "how many times should the NSM try to ping another machine before assuming the other machine has crashed and lost the file lock. One of the main parts of NFSv4 was an effort to fix file locking. Although it isn't widely adopted yet, it is now a 10year old protocol. (RFC3530 is dated April 2003, if I recall correctly.) Since rpc.lockd and rpc.statd haven't changed much in a long time and are essentially the same in 8.3 as stable/9, I'd suspect that something else has broken this (assuming it worked fine for 8.3). I would suspect the network device driver for your hardware and I'd suggest trying things like disabling checksum offload options, TSO and anything else you can try via ifconfig. If you happen to have a different kind of network hardware port, I'd try switching to that as well. rick > > Good luck with it, rick > thanks, > danny