From owner-freebsd-bugs@freebsd.org Sun Oct 4 23:34:19 2020 Return-Path: Delivered-To: freebsd-bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 095A7436F08 for ; Sun, 4 Oct 2020 23:34:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4C4KnQ6Zrcz4PTw for ; Sun, 4 Oct 2020 23:34:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id E00D7436F90; Sun, 4 Oct 2020 23:34:18 +0000 (UTC) Delivered-To: bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DEC69437112 for ; Sun, 4 Oct 2020 23:34:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C4KnQ5HzCz4Pch for ; Sun, 4 Oct 2020 23:34:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 953D0C963 for ; Sun, 4 Oct 2020 23:34:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 094NYI5i053005 for ; Sun, 4 Oct 2020 23:34:18 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 094NYIbS053004 for bugs@FreeBSD.org; Sun, 4 Oct 2020 23:34:18 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 249871] NFSv4 faulty directory listings under heavy load Date: Sun, 04 Oct 2020 23:34:18 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Oct 2020 23:34:19 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D249871 --- Comment #6 from Rick Macklem --- One additional thing you could do is: # vmstat -z | fgrep mbuf_cluster on the server when it is under heavy load, to see if it is running out of mbuf clusters. (Theoretically the NFS server should keep working when mbuf clusters are exhausted, but the result would be a Readdir reply made up of a long list of regular mbufs. That could impact things like TSO, if the net interface on the server has that enabled.) Basically, other than possible mbuf exhaustion, I can't think of any way heavy load would affect the NFS server code (except slower response). Since the name cache doesn't seem to be the culprit, that leaves all the caching that goes on inside ZFS. --> If the readdir contents is somehow reordered by ZFS when the directory is under heavy readdir load or the directory offset cookies somehow change, that would explain the problem. Yet one more thing that you could try is having client mounts done with "nfsv3,rdirplus". You mentioned that NFSv3 worked ok. NFSv3 + ridrplus works more closely to NFSv4 in the server, and whether or not that fails might be useful information. --=20 You are receiving this mail because: You are the assignee for the bug.=