From owner-freebsd-current@freebsd.org Thu Apr 15 18:35:30 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0E7755D19B1 for ; Thu, 15 Apr 2021 18:35:30 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from tor1-11.mx.scaleengine.net (tor1-11.mx.scaleengine.net [IPv6:2001:470:1:474::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4FLp1Y4CW6z4hXZ for ; Thu, 15 Apr 2021 18:35:29 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.3] (senat1-01.HML3.ScaleEngine.net [209.51.186.5]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by tor1-11.mx.scaleengine.net (Postfix) with ESMTPSA id ADDC017BC2 for ; Thu, 15 Apr 2021 18:35:23 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 tor1-11.mx.scaleengine.net ADDC017BC2 Subject: Re: NFS issues since upgrading to 13-RELEASE To: freebsd-current@freebsd.org References: <902a3c81-2ce8-49c0-b163-5ffa4b90afe5@www.fastmail.com> From: Allan Jude Message-ID: Date: Thu, 15 Apr 2021 14:35:22 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.9.0 MIME-Version: 1.0 In-Reply-To: <902a3c81-2ce8-49c0-b163-5ffa4b90afe5@www.fastmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4FLp1Y4CW6z4hXZ X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[freebsd.org]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Apr 2021 18:35:30 -0000 On 4/15/2021 9:22 AM, Chris Roose wrote: > I posted this in -questions and someone suggested I post here as well. > > I'm having NFS availability issues between my Proxmox client and FreeBSD server (10G link) since upgrading to 13-RELEASE. And unfortunately I upgraded my ZFS pool to v2.0.0 before I noticed the issue, so I'm kind of stuck. > > Periodically, the NFS server (I've tried both v3 and v4.2 clients) will go unresponsive for several minutes. I never had this problem on 12.2, and as far as I can tell it's not a disk or network I/O issue. I'll get several "nfs: server not responding, still trying" messages on the client and a few minutes later it usually recovers. It's not clear to me yet what's causing the block. Restarting nfsd on the server will resolve the issue if it doesn't clear itself. > > Any pointers for troubleshooting this? I've been looking through vmstat, gstat, top, etc. when the problem occurs, but I haven't been able to pinpoint the issue. I can get pcap, but it would be from the hosts, because I don't have a 10G tap or managed switch. > run `nfsstat -d 1` and try to capture a few lines from before, during, and after the stall, and that may provide some insight. Specifically, does the queue length grow, suggesting it is waiting on the I/O subsystem, or does it just stop getting traffic all together. -- Allan Jude