Date: Thu, 28 Jul 2016 01:41:03 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Marc Goroff <marc.goroff@quorum.net>, "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: Hanging/stalling mountd on heavily loaded NFS server Message-ID: <YTXPR01MB04952015505F90E89CEEDA0EDD000@YTXPR01MB0495.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <98b4db11-8b41-608c-c714-f704a78914b7@quorum.net> References: <98b4db11-8b41-608c-c714-f704a78914b7@quorum.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Marc Goroff wrote: > From: owner-freebsd-fs@freebsd.org <owner-freebsd-fs@freebsd.org> on beha= lf of Marc Goroff <marc.goroff@quorum.net > Sent: Wednesday, July 27, 2016 7:04 PM > To: freebsd-fs@freebsd.org > Subject: Hanging/stalling mountd on heavily loaded NFS server > > We have a large and busy production NFS server running 10.2 that is > serving approximately 200 ZFS file systems to production VMs. The system > has been very stable up until last night when we attempted to mount new > ZFS filesystems on NFS clients. The mountd process hung and client mount > requests timed out. The NFS server continued to serve traffic to > existing clients during this time. The mountd was hung in state nfsv4lck: > > [root@zfs-west1 ~]# ps -axgl|grep mount 0 38043 1 0 20 0 63672 17644 nfsv4lck Ds - 0:00.30 /usr/sbin/mountd -r -S /etc/exports /etc/zfs/exports > > It remains in this state for an indeterminate amount of time. I once saw > it continue on after several minutes, but most of the time it seems to > stay in this state for 15+ minutes. During this time, it does not > respond to kill -9 but it will eventually exit after many minutes. > Restarting mountd will allow the existing NFS clients to continue (they > hang when mountd exits), but any attempt to perform additional NFS > mounts will push mountd back into the bad state. > > This problem seems to be related to the number of NFS mounts off the > server. If we unmount some of the clients, we can successfully perform > the NFS mounts of the new ZFS filesystems. However, when we attempt to > mount all of the production NFS mounts, mountd will hang as above. > Stuff snipped for brevity... > > Any suggestion on how to resolve this issue? Since this is a production > server, my options for intrusive debugging are very limited. > I think you should try the patch that is r300254 in stable/10. It is a simp= le patch you can apply to your kernel without other changes. http://svnweb.freebsd.org/base/stable/10/sys/fs/nfsserver/nfs_nfsdkrpc.c?r1= =3D291869&r2=3D300254 It reverses the lock acquisition priority so that mountd doesn't wait until= the nfsd threads are idle before updating exports. rick > Thanks. > > Marc > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs<https://lists.freeb= sd.org/mailman/listinfo/freebsd-fs> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YTXPR01MB04952015505F90E89CEEDA0EDD000>