From owner-freebsd-fs@freebsd.org Fri Aug 12 00:35:54 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BEFFFBB5F76 for ; Fri, 12 Aug 2016 00:35:54 +0000 (UTC) (envelope-from Marc.Goroff@Quorum.net) Received: from mail.quorumlabs.com (mail.quorum.net [64.74.133.216]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (Client CN "mail.quorumlabs.com", Issuer "Go Daddy Secure Certification Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id AC7BB1D7D for ; Fri, 12 Aug 2016 00:35:53 +0000 (UTC) (envelope-from Marc.Goroff@Quorum.net) Received: from Marc-Goroffs-MacBook-Air-4.local (10.20.7.96) by QLEXC01.Quorum.local (10.30.0.22) with Microsoft SMTP Server (TLS) id 14.2.318.1; Thu, 11 Aug 2016 17:34:46 -0700 Subject: Re: Hanging/stalling mountd on heavily loaded NFS server To: Rick Macklem , "freebsd-fs@freebsd.org" References: <98b4db11-8b41-608c-c714-f704a78914b7@quorum.net> From: Marc Goroff Message-ID: Date: Thu, 11 Aug 2016 17:34:45 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: X-Originating-IP: [10.20.7.96] Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2016 00:35:54 -0000 Just to followup on this issue, the patch referenced below seems to have fixed the problem. Thanks! Marc On 7/27/16 6:41 PM, Rick Macklem wrote: > > Marc Goroff wrote: > > *> From:* owner-freebsd-fs@freebsd.org > on behalf of Marc Goroff > *> Sent:* Wednesday, July 27, 2016 7:04 PM > *> To:* freebsd-fs@freebsd.org > *> Subject:* Hanging/stalling mountd on heavily loaded NFS server > > > > We have a large and busy production NFS server running 10.2 that is > > serving approximately 200 ZFS file systems to production VMs. The > system > > has been very stable up until last night when we attempted to mount new > > ZFS filesystems on NFS clients. The mountd process hung and client > mount > > requests timed out. The NFS server continued to serve traffic to > > existing clients during this time. The mountd was hung in state > nfsv4lck: > > > > [root@zfs-west1 ~]# ps -axgl|grep mount > 0 38043 1 0 20 0 63672 17644 nfsv4lck Ds - 0:00.30 > /usr/sbin/mountd -r -S /etc/exports /etc/zfs/exports > > > > It remains in this state for an indeterminate amount of time. I once > saw > > it continue on after several minutes, but most of the time it seems to > > stay in this state for 15+ minutes. During this time, it does not > > respond to kill -9 but it will eventually exit after many minutes. > > Restarting mountd will allow the existing NFS clients to continue (they > > hang when mountd exits), but any attempt to perform additional NFS > > mounts will push mountd back into the bad state. > > > > This problem seems to be related to the number of NFS mounts off the > > server. If we unmount some of the clients, we can successfully perform > > the NFS mounts of the new ZFS filesystems. However, when we attempt to > > mount all of the production NFS mounts, mountd will hang as above. > > > Stuff snipped for brevity... > > > > Any suggestion on how to resolve this issue? Since this is a production > > server, my options for intrusive debugging are very limited. > > > I think you should try the patch that is r300254 in stable/10. It is a > simple > patch you can apply to your kernel without other changes. > > http://svnweb.freebsd.org/base/stable/10/sys/fs/nfsserver/nfs_nfsdkrpc.c?r1=291869&r2=300254 > > It reverses the lock acquisition priority so that mountd doesn't wait > until the > nfsd threads are idle before updating exports. > > rick > > > Thanks. > > > > Marc > > > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"