From owner-freebsd-fs@freebsd.org Mon Dec 14 03:52:02 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 840A94AD2A3 for ; Mon, 14 Dec 2020 03:52:02 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-lf1-x12e.google.com (mail-lf1-x12e.google.com [IPv6:2a00:1450:4864:20::12e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CvSBT4nqmz3sZ3 for ; Mon, 14 Dec 2020 03:52:01 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: by mail-lf1-x12e.google.com with SMTP id u18so26650926lfd.9 for ; Sun, 13 Dec 2020 19:52:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=pTnZFSVYWHSaPXUZ3mtbqMS1N8mqIZ1DM9QDI0fBLyA=; b=KaTTjlIZyqJ4XZdHGyE3XMvdqPmEYUGQGh1PohQEdqu4CjO4DHh4Ovt1g8VKMHHDnD wW1B8szPLtgqTnjnRbx9AZVaBMoKdNWBkMonv938Qle8BWP1qB7SGQGebD+fgAmiJKzA 7gn5sKZ29zJlr4QQTu11EhzuqVUrYCSw7uuKZHmiGHnALf9SNJNQvi7iZJ4f5Ydegs8x RzT5OyToCtLQ3+rIvyiaC0CASai06x3PvDiaqBHr6A/pJIunIJqNyN3vqcjZF9Rn4RZL PjNrhTJZLAPsvYWvG9TJ422iWNdUcWp7iF+PczvC+x+ZI5avzqyJeWuYoBZw2bupiV1S 3oXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=pTnZFSVYWHSaPXUZ3mtbqMS1N8mqIZ1DM9QDI0fBLyA=; b=pTa9FXJfFlNxth+UJybK+u3AH2aLO6MC3qw0Ws52N+EANmx3+/HchsJb+K7YIl+bEf 0YQARpToHTgR8Jo42I8Un2FaNbvjNmS/DnKWOch12A61eHolNax0SSs10OQJnbiu50SN MYDvchovyefJloZ85VpGzz9aIsRQFJ+ElBY2ZVdaprf/dEZ1zfTXYfH2ZH/6rLA8uZQc V9VE2UZdi08BgSt/74xkve6fA7O6Tqr/eth2MlOUhVGpNWQYUWXQANUCZPwR06AXsDVB MWpryZvciWXx9dkTrwMjRGg20ExOG1hyT8+WRRC4fLcGbatxMIas66X4YzDe61L8lZhd 4G6g== X-Gm-Message-State: AOAM530nehjvrHXMBza6qjhEsCmZN16P3DQVVskXygUOk2X6RMbJaMbo 3EiSvqhYrKiAR9hh4Yh0poQbaSLWHm0XH9ikg0A= X-Google-Smtp-Source: ABdhPJyl3sDEB4DNsGsUiS0p8DZZjAcyY/hScMYeKqv8A+aIXQ5lbpzTylQc1qY3yC0mK7nXMYVE6g5JU84pZsQIu9E= X-Received: by 2002:a2e:b80c:: with SMTP id u12mr9858960ljo.490.1607917919528; Sun, 13 Dec 2020 19:51:59 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: J David Date: Sun, 13 Dec 2020 22:51:48 -0500 Message-ID: Subject: Re: Major issues with nfsv4 To: Konstantin Belousov Cc: Rick Macklem , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4CvSBT4nqmz3sZ3 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=KaTTjlIZ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2a00:1450:4864:20::12e as permitted sender) smtp.mailfrom=jdavidlists@gmail.com X-Spamd-Result: default: False [-3.98 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.98)[-0.979]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::12e:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::12e:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::12e:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Dec 2020 03:52:02 -0000 On Sun, Dec 13, 2020 at 4:25 PM Konstantin Belousov wrote: > Nullfs with -o nocache (default for NFS mounts) should not cache vnodes. > So it is more likely a local load that has 130k files open. Of course, > it is the OP who can answer the question. This I can rule out; there is no visible correlation between "Opens" and the number of files open on the system. Just finishing a test right now, and: $ sudo nfsstat -E -c | fgrep -A1 OpenOwner OpenOwner Opens LockOwner Locks Delegs LocalOwn 4678 36245 15 6 0 0 $ sudo fstat | wc -l 2698 $ ps Haxlww | wc -l 1012 The value of Opens increases consistently over time. Killing the processes causing this behavior *did not* reduce the number of OpenOwner or Opens. Unmounting the nullfs mounts (after the processes were gone) *did*: $ sudo nfsstat -E -c | fgrep -A1 OpenOwner OpenOwner Opens LockOwner Locks Delegs LocalOwn 130 41 0 0 0 0 Mutex contention was observed this time, but once it was apparent that "Opens" was increasing over time, I didn't let the test get to the point of disrupting activities. This test ended at Opens = 36589, which is well short of the previous 130,000+. It is possible that mutex contention becomes an issue once system CPU resources are exhausted. More about the results of the latest test after the data is analyzed. After that's done, I'll attempt Rick's patch. In the long run, we would definitely like to get delegation to work. Baby steps! Thanks!