Date: Thu, 7 Mar 2024 22:59:02 -0500 From: Garrett Wollman <wollman@bimajority.org> To: Rick Macklem <rick.macklem@gmail.com> Cc: stable@freebsd.org Subject: Re: 13-stable NFS server hang Message-ID: <26090.36102.379531.160926@hergotha.csail.mit.edu> In-Reply-To: <26083.64612.717082.366639@hergotha.csail.mit.edu> References: <26078.50375.679881.64018@hergotha.csail.mit.edu> <CAM5tNy7ZZ2bVLmYnOCWzrS9wq6yudoV5JKG5ObRU0=wLt1ofZw@mail.gmail.com> <26083.64612.717082.366639@hergotha.csail.mit.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
<<On Sat, 2 Mar 2024 23:28:20 -0500, I wrote: > I believe this explains why vn_copy_file_range sometimes takes much > longer than a second: our servers often have lots of data waiting to > be written to disk, and if the file being copied was recently modified > (and so is dirty), this might take several seconds. I've set > vfs.zfs.dmu_offset_next_sync=0 on the server that was hurting the most > and am watching to see if we have more freezes. > If this does the trick, then I can delay deploying a new kernel until > April, after my upcoming vacation. Since zeroing dmu_offset_next_sync, I've seen about 8000 copy operations on the problematic server and no NFS work stoppages due to the copy. I have observed a few others in a similar posture, where one client wants to ExchangeID and is waiting for other requests to drain, but nothing long enough to cause a service problem.[1] I think in general this choice to prefer "accurate" but very slow hole detection is a poor choice on the part of the OpenZFS developers, but so long as we can disable it, I don't think we need to change anything in the NFS server itself. It would be a good idea longer term to figure out a lock-free or synchronization-free way of handling these client session accept/teardown operations, because it is still a performance degradation, just not disruptive enough for users to notice. -GAWollman [1] Saw one with a slow nfsrv_readdirplus and another with a bunch of threads blocked on an upcall to nfsuserd.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?26090.36102.379531.160926>