Date: Thu, 11 Aug 2016 17:34:45 -0700 From: Marc Goroff <marc.goroff@quorum.net> To: Rick Macklem <rmacklem@uoguelph.ca>, "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: Hanging/stalling mountd on heavily loaded NFS server Message-ID: <bd618cab-f46b-e4d6-cd9a-ea18b19e39b2@quorum.net> In-Reply-To: <YTXPR01MB04952015505F90E89CEEDA0EDD000@YTXPR01MB0495.CANPRD01.PROD.OUTLOOK.COM> References: <98b4db11-8b41-608c-c714-f704a78914b7@quorum.net> <YTXPR01MB04952015505F90E89CEEDA0EDD000@YTXPR01MB0495.CANPRD01.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
Just to followup on this issue, the patch referenced below seems to have fixed the problem. Thanks! Marc On 7/27/16 6:41 PM, Rick Macklem wrote: > > Marc Goroff wrote: > > *> From:* owner-freebsd-fs@freebsd.org <owner-freebsd-fs@freebsd.org> > on behalf of Marc Goroff <marc.goroff@quorum.net > > *> Sent:* Wednesday, July 27, 2016 7:04 PM > *> To:* freebsd-fs@freebsd.org > *> Subject:* Hanging/stalling mountd on heavily loaded NFS server > > > > We have a large and busy production NFS server running 10.2 that is > > serving approximately 200 ZFS file systems to production VMs. The > system > > has been very stable up until last night when we attempted to mount new > > ZFS filesystems on NFS clients. The mountd process hung and client > mount > > requests timed out. The NFS server continued to serve traffic to > > existing clients during this time. The mountd was hung in state > nfsv4lck: > > > > [root@zfs-west1 ~]# ps -axgl|grep mount > 0 38043 1 0 20 0 63672 17644 nfsv4lck Ds - 0:00.30 > /usr/sbin/mountd -r -S /etc/exports /etc/zfs/exports > > > > It remains in this state for an indeterminate amount of time. I once > saw > > it continue on after several minutes, but most of the time it seems to > > stay in this state for 15+ minutes. During this time, it does not > > respond to kill -9 but it will eventually exit after many minutes. > > Restarting mountd will allow the existing NFS clients to continue (they > > hang when mountd exits), but any attempt to perform additional NFS > > mounts will push mountd back into the bad state. > > > > This problem seems to be related to the number of NFS mounts off the > > server. If we unmount some of the clients, we can successfully perform > > the NFS mounts of the new ZFS filesystems. However, when we attempt to > > mount all of the production NFS mounts, mountd will hang as above. > > > Stuff snipped for brevity... > > > > Any suggestion on how to resolve this issue? Since this is a production > > server, my options for intrusive debugging are very limited. > > > I think you should try the patch that is r300254 in stable/10. It is a > simple > patch you can apply to your kernel without other changes. > > http://svnweb.freebsd.org/base/stable/10/sys/fs/nfsserver/nfs_nfsdkrpc.c?r1=291869&r2=300254 > > It reverses the lock acquisition priority so that mountd doesn't wait > until the > nfsd threads are idle before updating exports. > > rick > > > Thanks. > > > > Marc > > > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>; > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bd618cab-f46b-e4d6-cd9a-ea18b19e39b2>