From owner-freebsd-fs@freebsd.org Tue Jan 12 22:51:07 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B21304E8842 for ; Tue, 12 Jan 2021 22:51:07 +0000 (UTC) (envelope-from freebsd-fs@m.gmane-mx.org) Received: from ciao.gmane.io (ciao.gmane.io [116.202.254.214]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DFm5Q5JzVz4n66 for ; Tue, 12 Jan 2021 22:51:06 +0000 (UTC) (envelope-from freebsd-fs@m.gmane-mx.org) Received: from list by ciao.gmane.io with local (Exim 4.92) (envelope-from ) id 1kzSVC-0008eC-RF for freebsd-fs@freebsd.org; Tue, 12 Jan 2021 23:51:02 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Lapo Luchini Subject: `zfs send` immediately stops with "Assertion failed: (progress)" Date: Tue, 12 Jan 2021 23:50:56 +0100 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0 SeaMonkey/2.53.5.1 X-Mozilla-News-Host: news://news.gmane.org:119 X-Rspamd-Queue-Id: 4DFm5Q5JzVz4n66 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=lapo.it (policy=none); spf=pass (mx1.freebsd.org: domain of freebsd-fs@m.gmane-mx.org designates 116.202.254.214 as permitted sender) smtp.mailfrom=freebsd-fs@m.gmane-mx.org X-Spamd-Result: default: False [0.10 / 15.00]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; FORGED_MUA_SEAMONKEY_MSGID_UNKNOWN(2.50)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[116.202.254.214:from]; MV_CASE(0.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[116.202.254.214:from:127.0.2.255]; DMARC_POLICY_SOFTFAIL(0.10)[lapo.it : SPF not aligned (relaxed), No valid DKIM,none]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[lapo@lapo.it,freebsd-fs@m.gmane-mx.org]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:116.202.0.0/16, country:DE]; FROM_NEQ_ENVFROM(0.00)[lapo@lapo.it,freebsd-fs@m.gmane-mx.org]; MAILMAN_DEST(0.00)[freebsd-fs]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2021 22:51:07 -0000 I migrated this server a few times already on new hardware over the years, by using `zfs send`. Even right now I'm using `zrepl` to periodically have a remote copy to a distant site (for disaster recovery). But trying to migrate on a new server in the same server farm recently I'm seeing this error: $ sudo zfs send -Rv z@recentSnapshot >/dev/null Assertion failed: (progress), file /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c, line 1511. This is happening in a freshly updated FreeBSD 12.2-RELEASE-p2. I don't know how to reproduce it (or debug it) and the server is serving production data, so I can only do non-intrusive tests. I guess it has something to do with clones, given the comments at line 1480 and line 1492 so maybe I can print something with `zdb`? Any idea how to debug (and solve)? PS: it's a bit out of topic, but since the code in latest OpenZFS seems to be exactly the same, I opened a bug there too: https://github.com/openzfs/zfs/issues/11457 -- Lapo Luchini - http://lapo.it/ From owner-freebsd-fs@freebsd.org Thu Jan 14 13:50:48 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 933054E333E for ; Thu, 14 Jan 2021 13:50:48 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-lj1-x22e.google.com (mail-lj1-x22e.google.com [IPv6:2a00:1450:4864:20::22e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DGm136F4Vz3tCt for ; Thu, 14 Jan 2021 13:50:47 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: by mail-lj1-x22e.google.com with SMTP id x23so6481809lji.7 for ; Thu, 14 Jan 2021 05:50:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=IJiXZSi624SmVVLKfy5FIC4f9Y1AyUPWTmfxnJl5r5k=; b=kTVf0JmJoUnV3nxzckMCPNJVISLkjSSsgObhvyi/AWgIHauVG00DHqiQCN2r355fym a3PkCBdhyYueVCtfeMXqBfLSvwOQeDBA4WFPki+6HCZufbPIwQ359+hvKrEEeIOrv+j6 qE8E8GgGevFDYHk+uOXZhS2b4mSXuYpQe6eeSRv3LBJ2GtoRpsaKRQHGCnVjc1b06eZz LK9EFwvawtveM37xU6krEjU1YI99ZLwD2qi49TblMxUgegGQRnn+UF3Uogd9cbmaxyJD fHSoa+ToqBB8O5BrbrKUDYlR0KEuRGVXu2CeqGCAfrHq/2Mo31itqlq3B7tbzU6ZFWJG x6WQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=IJiXZSi624SmVVLKfy5FIC4f9Y1AyUPWTmfxnJl5r5k=; b=e2G1napg3/+d77XHHPdGISa30fgEbV47S1ERRK3L1REQpTuvZLBiaitDHpZL9VVNOm 7kZjiodOMP1Eu96Zcp8ipPkXq9mG05fKZf7tpOKC+OVHr85qV5MlqMibLjGC6TZDLHtd w9S6BTVgYORRUa2Q2dSRJyTOI+05sXmZ1YFoiqOtr5At84EClnbL4DLc9uRze6bZ1Ust pZSohXqPBCX5VjLRQJIcNIyDzNts74C3OClC0RWUdn6112rQwuf13hH5VEAoKQIV+wnl APRFZUo8+86hRMRBkXjoF+N6OzyUYFfbbCqrAQkelMlSUaEzAgJf0yhsem9R1/yH2Aa9 pFkA== X-Gm-Message-State: AOAM531SxeLKCvUU98tR8c4lgj7JQhtsgra0LFSv8S3Z3Zje6y0pinwH 2XdcMUjTHndU/9Xe+Ge+95rZ4fFkOHTU8a72trw= X-Google-Smtp-Source: ABdhPJwgwr5oLUYPdDUzLg18IIawav0PL35whOK5XOGdLSMx6jBPztBjcuTC1j8Gg91o/7EX+Tl1ru/T+n5bkowR+3I= X-Received: by 2002:a2e:586:: with SMTP id 128mr3254128ljf.273.1610632245762; Thu, 14 Jan 2021 05:50:45 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: J David Date: Thu, 14 Jan 2021 08:50:34 -0500 Message-ID: Subject: Re: Major issues with nfsv4 To: Rick Macklem Cc: Konstantin Belousov , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4DGm136F4Vz3tCt X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=kTVf0JmJ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2a00:1450:4864:20::22e as permitted sender) smtp.mailfrom=jdavidlists@gmail.com X-Spamd-Result: default: False [-1.97 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.97)[-0.970]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::22e:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::22e:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22e:from]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jan 2021 13:50:48 -0000 On Wed, Dec 16, 2020 at 11:25 PM Rick Macklem wrote: > If you can do so when the "Opens" count has gone fairly high, > please "sysctl vfs.deferred_inact" and let us know what that > returns. $ sysctl vfs.deferred_inact sysctl: unknown oid 'vfs.deferred_inact' $ sysctl -a vfs | fgrep defer $ Sorry for the delay in responding to this. I got my knuckles rapped for allowing this to happen so much. It happened just now because some of the "use NFSv4.1" config leaked out to a production machine, but not all of it. As a result, only the read-only "job binary" filesystems were mounted with nullfs+nfsv4.1. So it is unlikely to be related to creating files. Hopefully, that narrows things down. $ sudo nfsstat -E -c [...] OpenOwner Opens LockOwner Locks Delegs LocalOwn 37473 303469 0 0 1 0 [...] "nfscl: never fnd open" continues to appear regularly on console/dmesg, even at the end of the reboot: Waiting (max 60 seconds) for system thread `bufspacedaemon-2' to stop... done Waiting (max 60 seconds) for system thread `bufspacedaemon-5' to stop... done Waiting (max 60 seconds) for system thread `bufspacedaemon-1' to stop... done Waiting (max 60 seconds) for system thread `bufspacedaemon-6' to stop... done All buffers synced. nfscl: never fnd open nfscl: never fnd open nfscl: never fnd open nfscl: never fnd open nfscl: never fnd open nfscl: never fnd open Uptime: 4d13h59m27s Rebooting... cpu_reset: Stopping other CPUs ---<>--- It did not appear 300,000 times, though. More like a few times a day. Also, I set up an idle system with the NFSv4.1+nullfs config, as requested. It has been up for 32 days and appears not to have leaked anything. But it does also have a fistful of those "nfscl: never fnd open" messages. There is also a third system in a test environment with the nullfs+nfsv4.1 config. That system is up 34 days, has no exhibited problems, and shows this: OpenOwner Opens LockOwner Locks Delegs LocalOwn 342 15098 2 0 0 0 That machine shows one "nfscl: never fnd open" in the dmesg. A fourth system has the NFSv4.1-no-nullfs config in production with net.inet.ip.portrange.lowlast tweaked and a limit on simultaneous jobs. That system had issues requiring a restart 18 days ago. It also occasionally gets "nfscl: never fnd open" in the dmesg and has relatively large Open numbers: As of right now: OpenOwner Opens LockOwner Locks Delegs LocalOwn 23214 46304 0 0 0 0 The "OpenOwner" value on that system seems to swing dramatically, ranging between 45,000 to 10,000 in just a few minutes. It appears to correlate well to simultaneous jobs. The "Opens" value goes up and down a bit, but trends upward over time. However, when I found and killed one long-running job and unmounted its filesystems, "Opens" dropped 90% to around 4600. Note there are *no* nullfs mounts on that system. So nullfs may not be a necessary component of the problem. As a next step, I will try to create a fake job that opens a ton of files. Then I'll test it on the binary read-only nullfs+nfsv4.1 mounts and on the system that runs nfsv4.1 directly. Thanks! From owner-freebsd-fs@freebsd.org Thu Jan 14 22:30:50 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 492414D0228 for ; Thu, 14 Jan 2021 22:30:50 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660087.outbound.protection.outlook.com [40.107.66.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DGzY50mHGz3LBx for ; Thu, 14 Jan 2021 22:30:48 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BwsQylWb8ubkdP5CS7PP/iDi9qU6oFUWqKhV5zeGvym4Onx0x+6qjfoPVliw6begu9q0FA0j+rIYmIcqql5w8//G/44NM2LQ2HtyaFd2jbYgvmMz6roRXNjB/Ej47NH/k/64ko3tVIrhKwx8yS8Nk+L42nRWhNjNuifBAh+lPEDF8sP73kTtkXTzJE2BYRg0Nwo/c1pNYRmtppxTY8nYGRM5Gd3syM+f7VQ2/2fDw/n63ZN4VakolSqfxVpcziTdzqHlfxG5gqWvp4RHfC7rZYhBJFv+lFkhoemKTgBEp6cEd9NqyxzPamTe5zXALqSce0nwV+ey9CIe/8GN5ou/UA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QlCff2JooXpJTfI4GMgLN+UKlKl7UldM//evEkyA2DM=; b=Fm61xpEvGsPWr2iSXouHtQ1ogVVmklrYQA/e0eAMHLP+L8JStampgOfBRcqYVVL9tihHHbA54jd0YMiOpFzfuE28DaRveBteqEyXl/aMhEZnZGzew97GefLLWmmGo7QoNe5N2D+TGbc409twPhJWwaDLYjvSvYgBgRsMSWzvZ3G4LRyT08TZLchmSEwQjNtuSQt0y9TfywUdynKeVhMU8vduTRsh4vk5vw/tzAniqdg7BCzGfpgyaYuRAFWDGfVK8106WGsjzOt3sc+Cfh41xfR89/9KPKt/Zqdxuj1WLHKo1v3Kp++KQbiv/ZpNBFxS4YmuBxXiQvJuOOfN6aIvSg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QlCff2JooXpJTfI4GMgLN+UKlKl7UldM//evEkyA2DM=; b=YRqd1FcCrxmAQn8uJokQuwQSNWIwqey/d7GIjSX8T6MYhxM41iefSicZH4NZSf264A0hTp2EwdVXgAGQDDntK8OMVCjGWSrCFDpKl0w4reTKpFVkBFx6kzgDvDxwP55NxQjO6RzULr7hXYBADnzTYIWsLS8/xVixmBuSkxzjQGNrRKWU//n1vuYjGXRFOG0oe0UfsH59+IVP8IspiG01qe0OBIdYhSi5nIprj6ugucgEW4AScNPBnnmtHcdDaPwr7Ps12eMAqK4Xm37QJjL6uwueNF+yLqbdLfRvuxDcKBBaSur3wUi8ahw5JukO+xi3m+fZaIMIMTSJKAJ4urvdmQ== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQXPR01MB2664.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:44::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.9; Thu, 14 Jan 2021 22:30:47 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::3d86:c7f9:bc4c:40c0]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::3d86:c7f9:bc4c:40c0%6]) with mapi id 15.20.3763.010; Thu, 14 Jan 2021 22:30:47 +0000 From: Rick Macklem To: J David CC: Konstantin Belousov , "freebsd-fs@freebsd.org" Subject: Re: Major issues with nfsv4 Thread-Topic: Major issues with nfsv4 Thread-Index: AQHWzw/HDat+dHoH9kKG5K3Xpd53kqnxDteQgAFi0QCAABTa84AALLCAgAAVvcmAAAu0AIAA4wiAgAAI5gCAAO/IAIAAiDrmgABWOgCAAGv4AIAEv9nSgCyfuACAAIn3Cw== Date: Thu, 14 Jan 2021 22:30:47 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 9f142443-1b04-413c-4b30-08d8b8dc0a09 x-ms-traffictypediagnostic: YQXPR01MB2664: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 5m5tOy1TSYrvP/eU/q33zXIYhYSMMIcDkOqYkzthSq7PgunLIGG/DrRGchUfXihqw83NwcrnyNnZjWgN3bfzRV1fq31iamHPx53wnBj5mj377GNv5tVypdDrFJFu/yOEP5IiW8vHzXXJihByinskYLXaFRkEwQ5FM38FwuVByuhyEQ8ryO8B7mrNJH4by74DxDt/Wk+bJveOugnnHywvfJDDkBx/KaxtlkH4/u0WjWnRPgL6jxFSlyPHOZ2KxLJtS183K65NtO5wTOTnd1ezuehHDvgx0NY5ZXcKEdJzAAOTHsSju6xeqBugN1AVuDmYFcecmjVBHk2jZD8ZrQpBAX9wyzofp/epNgVXuAWz9aMWodOK1Fh2JXINIe/UKCZp1Z1f0/mQ2SjPw5VRIg0I4w== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(376002)(39860400002)(136003)(366004)(396003)(346002)(5660300002)(66476007)(4326008)(54906003)(7696005)(186003)(91956017)(66446008)(64756008)(66946007)(6506007)(86362001)(76116006)(33656002)(66556008)(478600001)(83380400001)(8936002)(8676002)(9686003)(52536014)(2906002)(316002)(55016002)(786003)(71200400001)(6916009); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?//PSsuSV24CzNSvFQJf3pZQ41WXCzt3rk4NDPL+p6WGn9Md0n6iaBKqRKA?= =?iso-8859-1?Q?xErbeVyGFt/hqATaPqjfCHlcsytg5yW6XBjPt+bu24dtY8/d+6k6mXtnpI?= =?iso-8859-1?Q?4Bm6H9ZGrFcZLAhwJwZw6lQZB6Mgo8gxWnLEPQO1kkULZzSVmb5z7Bg76Q?= =?iso-8859-1?Q?f4xeUtIxfPvwuPwF1BxlOg3ZLX7Yci7VyaVMGQu/92rVPgOt/nnDuHMthw?= =?iso-8859-1?Q?C4Ya7PjSlChDuIASsAJEhnMkKHfv0nv5XUtU+Jrpa7WJ2/O5r+n8RSyJdp?= =?iso-8859-1?Q?iWBPc1V5hxkY+kkfprT31GpcEJs2yYkWe3rgYdGiOtdyDLND/7SkEzdjSC?= =?iso-8859-1?Q?eAVWUVehRG/05J3404Cg1jmyytIA+5RDNLBzlIQKL0PscNoTCX1cEx5fb/?= =?iso-8859-1?Q?KqdiRp/KWNLodD3m2Iz/mw0JyXT1Rb1RBgtsJMrPjeUujNvN8QDUmKhSMU?= =?iso-8859-1?Q?E4zk9H9SMeoy3yE1uSqKrnaIX+8wSavMR6dR2E/2Ojh4zXG9kshXOV5X9K?= =?iso-8859-1?Q?N3xm/TckOlCVG7waKHZ4n43DoAH6ck/6tcLx3P0sH8V+wtD5Jh6R67Rqb7?= =?iso-8859-1?Q?ikqeA1s7Dem7JE/v1qxBtw4L0AZktrP8w3/r6PepTOa3eh4FIOtSAB7WQ0?= =?iso-8859-1?Q?BFWOF5VD6bB0RjHhp7MD1S0Q3M3SB/ZRUZehnxc0elUHEz2a9+JTZi2aBD?= =?iso-8859-1?Q?tr4Z7nkKVZC0W14zw4XOkQw8++Rw/QdyWnsFOGoy4+c81kd7UEUaUf14ca?= =?iso-8859-1?Q?DZMixWq+UDpdzBEpmyQYDXzWQj45OmoP6JHPnIvRqkYrpHyJ27Imk7O8HY?= =?iso-8859-1?Q?JO9U2Ht7R3Bcq/Ai/pektzKB4GDmeeQFDD9DKkjGHRoBRrTWhdxJ1Kjr1e?= =?iso-8859-1?Q?V5jaPcM80R4VDjwlhEyd6gx9iS4V8BQB+OugmsZfDbHY/QFtlNqr0QM0Ts?= =?iso-8859-1?Q?2VijlmBfp4n19CDDMeg6BtGm9/R9/mJW+RPn74K8TUpsCkREfACv1lt5KP?= =?iso-8859-1?Q?PxvniE3a3FlUNgCzygUBfnxGr6FkGfiAGie+ZOjo6FiG8cmV/fmJ4wa9qk?= =?iso-8859-1?Q?bt+oVtKZoFFEnBi6kscA4cU=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 9f142443-1b04-413c-4b30-08d8b8dc0a09 X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Jan 2021 22:30:47.3222 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ASYpXV7PoohejODQXdzrxm+J+BfKQxy9FK8Q8B5BaSr9sxNspYNKLs7YktowisvN6/8CCfhM6mFypszhPaz1Eg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB2664 X-Rspamd-Queue-Id: 4DGzY50mHGz3LBx X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=YRqd1FcC; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.87 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.66.87:from]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[40.107.66.87:from:127.0.2.255]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[40.107.66.87:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.66.87:from]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jan 2021 22:30:50 -0000 J David wrote:=0A= >On Wed, Dec 16, 2020 at 11:25 PM Rick Macklem wrote= :=0A= >> If you can do so when the "Opens" count has gone fairly high,=0A= >> please "sysctl vfs.deferred_inact" and let us know what that=0A= >> returns.=0A= >=0A= >$ sysctl vfs.deferred_inact=0A= >sysctl: unknown oid 'vfs.deferred_inact'=0A= >$ sysctl -a vfs | fgrep defer=0A= >$=0A= Yes. I did not realize how different FreeBSD12 is when compared with FreeBS= D13/head=0A= in this area.=0A= At a quick glance, I do not see where the syncer tries to vinactive() vnode= s where=0A= the VOP_INACTIVE() has been deferred.=0A= =0A= --> It is possible that this problem is fixed in FreeBSD13/head.=0A= Any chance you can test a FreeBSD13/head system?=0A= =0A= Kostik, does FreeBSD12 try to vinactive() deferred VOP_INACTIVE() vnodes vi= a the=0A= syncer?=0A= =0A= >Sorry for the delay in responding to this. I got my knuckles rapped=0A= >for allowing this to happen so much.=0A= >=0A= >It happened just now because some of the "use NFSv4.1" config leaked=0A= >out to a production machine, but not all of it. As a result, only the=0A= >read-only "job binary" filesystems were mounted with nullfs+nfsv4.1.=0A= >So it is unlikely to be related to creating files. Hopefully, that=0A= >narrows things down.=0A= >=0A= >$ sudo nfsstat -E -c=0A= >[...]=0A= > OpenOwner Opens LockOwner Locks Delegs LocalOwn=0A= > 37473 303469 0 0 1 0=0A= >[...]=0A= >=0A= >"nfscl: never fnd open" continues to appear regularly on=0A= >console/dmesg, even at the end of the reboot:=0A= Not sure what this implies. The message means that it cannot find=0A= a NFSv4 Open to Close.=0A= It may indicate something is broken in the client, but is not by itself, se= rious.=0A= =0A= >Waiting (max 60 seconds) for system thread `bufspacedaemon-2' to stop... d= one=0A= >Waiting (max 60 seconds) for system thread `bufspacedaemon-5' to stop... d= one=0A= >Waiting (max 60 seconds) for system thread `bufspacedaemon-1' to stop... d= one=0A= >Waiting (max 60 seconds) for system thread `bufspacedaemon-6' to stop... d= one=0A= >All buffers synced.=0A= >nfscl: never fnd open=0A= >nfscl: never fnd open=0A= >nfscl: never fnd open=0A= >nfscl: never fnd open=0A= >nfscl: never fnd open=0A= >nfscl: never fnd open=0A= >Uptime: 4d13h59m27s=0A= >Rebooting...=0A= >cpu_reset: Stopping other CPUs=0A= >---<>---=0A= >=0A= >It did not appear 300,000 times, though. More like a few times a day.=0A= >=0A= >Also, I set up an idle system with the NFSv4.1+nullfs config, as=0A= >requested. It has been up for 32 days and appears not to have leaked=0A= >anything. But it does also have a fistful of those "nfscl: never fnd=0A= >open" messages.=0A= >=0A= >There is also a third system in a test environment with the=0A= >nullfs+nfsv4.1 config. That system is up 34 days, has no exhibited=0A= >problems, and shows this:=0A= >=0A= > OpenOwner Opens LockOwner Locks Delegs LocalOwn=0A= > 342 15098 2 0 0 0=0A= >=0A= >That machine shows one "nfscl: never fnd open" in the dmesg.=0A= >=0A= >A fourth system has the NFSv4.1-no-nullfs config in production with=0A= >net.inet.ip.portrange.lowlast tweaked and a limit on simultaneous=0A= >jobs. That system had issues requiring a restart 18 days ago. It also=0A= >occasionally gets "nfscl: never fnd open" in the dmesg and has=0A= >relatively large Open numbers:=0A= >=0A= >As of right now:=0A= > OpenOwner Opens LockOwner Locks Delegs LocalOwn=0A= > 23214 46304 0 0 0 0=0A= >=0A= >The "OpenOwner" value on that system seems to swing dramatically,=0A= >ranging between 45,000 to 10,000 in just a few minutes. It appears to=0A= >correlate well to simultaneous jobs.=0A= This sounds normal, since an OpenOwner refers to a process on the client=0A= doing a file Open.=0A= =0A= > The "Opens" value goes up and=0A= >down a bit, but trends upward over time. However, when I found and=0A= >killed one long-running job and unmounted its filesystems, "Opens"=0A= >dropped 90% to around 4600. Note there are *no* nullfs mounts on that=0A= >system. So nullfs may not be a necessary component of the problem.=0A= This also sounds reasonable. The NFSv4 Opens can only be closed once=0A= the process doing the Open plus all chidren processes have closed the=0A= file.=0A= --> If a program is "lazy" and doesn't do closes, they won't=0A= happen until the process exits. And then children processes=0A= will also need to exit before it leaves zombie state.=0A= =0A= One thing to try (other than a FreeBSD13/head system, if possible)=0A= is the "oneopenown" mount option.=0A= --> It can only be used on NFSv4.1 mounts (not NFSv4.0) and=0A= makes the mount only use one OpenOwner for all Opens=0A= instead of a different one for each process doing an Open.=0A= --> This would reduce the number of Opens for the case=0A= where multiple processes open the same file.=0A= --> It also simplifies the search for an Open, since there=0A= is only one for each file.=0A= =0A= rick=0A= =0A= As a next step, I will try to create a fake job that opens a ton of=0A= files. Then I'll test it on the binary read-only nullfs+nfsv4.1=0A= mounts and on the system that runs nfsv4.1 directly.=0A= =0A= Thanks!=0A= From owner-freebsd-fs@freebsd.org Sat Jan 16 05:23:51 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 50F2A4D38DA for ; Sat, 16 Jan 2021 05:23:51 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-lf1-x129.google.com (mail-lf1-x129.google.com [IPv6:2a00:1450:4864:20::129]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DHmgB5BMZz4rHp for ; Sat, 16 Jan 2021 05:23:50 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: by mail-lf1-x129.google.com with SMTP id m25so16328712lfc.11 for ; Fri, 15 Jan 2021 21:23:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=t25NAXjITrZ6RjcEhuy478lZwWQZQG0Ex2FBQgBxuf0=; b=k1SsFHms61L+KKMqp898wuFrj5kSKfbqhk/jfZMdKou9ZtTqQrd9mX42dMAP60ARhA Xh9zUiyFJaC6uqb5Nxk309KOtonJ+PVC+GpUbZdSzpiCV9a9/xAmtFC9U+l+LFw9+Qc8 4SgtTXk/b7LFb1n3o3jTzz9uhc8mVkdUpe3PGWMI6UTXs7FnZyXrp6rLFKoHzoHVkFSV BRHl/I5TJLi9dzl+L+kc1/LXBF5vayQbRV99OmAx25K8lo+x8Cl1xpNjGJCQpuVTegNZ EN2CuiKx2amcxMXpmwPZuEIrMkMwQftzfZZJeJYBX9RvpYIgDC6NFuk0l5P+3rfRayfP 20yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=t25NAXjITrZ6RjcEhuy478lZwWQZQG0Ex2FBQgBxuf0=; b=UVYqdiFgr7I3LebFKUEetVmxvctt3rtS9q5J/4TEBaKZR4VQKZnX2Byn9HHqhj9MnJ 2b/U8IkzJpZjShCj7qkAXihevSvfztz+9FPybiGrbMEDUCqYxdxEGuerFDEaQgtgG9QD GtxmKdZI/1AEygyYIua0bjOLnwQSOF31ei3qIy1Pg1M13sVU9jDK1+YxPvwIn1IBTVXx JMTzj/klHmFnHMjmvmaIKJWhEXtpbpEz6oqjFfd0f3D06MI643OKin9jO7z6ilQtT3rt rHgeR1LyURRe44Nvq0di5kQM0ivtorBnbFf9ADAnDeOT3+1JIEt+vwlRwUjHYFHjzn3E N+bQ== X-Gm-Message-State: AOAM533srAa80cF9eDTfobhJB1loUInTwwPduhjtfWFBnXeyHs2Q8Zdg tuitVZZTmpuz8Z4eeLooLmKRza/0zsj1EBR4wiargbSqWBo= X-Google-Smtp-Source: ABdhPJw54c74+aBj1WfwaABhS3MnD945KCHqA4aV6VE+5fOSK6+CXXIRy6Md+K6iUjs4Bl2YuJa3OUbifOVIT439SQs= X-Received: by 2002:a19:23d8:: with SMTP id j207mr7655211lfj.144.1610774628998; Fri, 15 Jan 2021 21:23:48 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: J David Date: Sat, 16 Jan 2021 00:23:37 -0500 Message-ID: Subject: Re: Major issues with nfsv4 To: Rick Macklem Cc: Konstantin Belousov , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4DHmgB5BMZz4rHp X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=k1SsFHms; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2a00:1450:4864:20::129 as permitted sender) smtp.mailfrom=jdavidlists@gmail.com X-Spamd-Result: default: False [-3.19 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.19)[-0.193]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::129:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::129:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::129:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2021 05:23:51 -0000 On Thu, Jan 14, 2021 at 5:30 PM Rick Macklem wrote: > One thing to try (other than a FreeBSD13/head system, if possible) > is the "oneopenown" mount option. The odds of being able to run an unreleased version of FreeBSD on production servers are slim to none. While trying to develop a reproduction, I think I have narrowed down what the problem is. There are no jails or nullfs involved here, just NFSv4.1. Window 1: (to track OpenOwner/Opens) while true; do date; nfsstat -E -c | fgrep -A1 OpenOwner; sleep 1; done Window 2: mount -o ro,nfsv4,minorversion=1,nosuid fileserver:/path/to/freesbd/root /mnt chroot /mnt (OpenOwner is now 1 and Opens is now 9.) Window 3: chroot /mnt (OpenOwner is now 2 and Opens is now 18.) ls (OpenOwner is now 3 and Opens is now 21.) ls (OpenOwner is now 4 and Opens is now 24.) ls (OpenOwner is now 5 and Opens is now 27.) ls (OpenOwner is now 6 and Opens is now 30.) ls (OpenOwner is now 7 and Opens is now 33.) bash while true; do ls | true; done (Allow about a minute to pass, hit CTRL-C. OpenOwner is now 4647 and Opens is now 13957) exit exit (OpenOwner is now 4647 and Opens is now 13952.) Back in Window 2: exit (wait about 30 seconds, OpenOwner is now 0 and Opens is now 0.) So it looks like the NFSv4 code can't let go of *any* Opens on a file/directory until *all* references to that file/directory are closed. If chroot is too much, "vi /mnt/etc/motd" in Window 2 and "cat /mnt/etc/motd" in Window 3 have the same effect, leaking one Open per cat instead of 3. You probably don't even need a FreeBSD install on the NFS mount; just hold a single file open in one window and open/close it repeatedly in another. Then I re-tested this with "-o ro,nfsv4,minorversion=1,nosuid,oneopenown." At least for this simple case, the problem did not occur with oneopenown set. Are there downsides to the oneopenown flag other than breaking delegations? Thanks! From owner-freebsd-fs@freebsd.org Sat Jan 16 05:43:37 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 60CB04D4944 for ; Sat, 16 Jan 2021 05:43:37 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x72d.google.com (mail-qk1-x72d.google.com [IPv6:2607:f8b0:4864:20::72d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DHn601rsnz4sdx for ; Sat, 16 Jan 2021 05:43:35 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x72d.google.com with SMTP id b64so13940573qkc.12 for ; Fri, 15 Jan 2021 21:43:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=XccDeVbR7GfrXtvOalK59/obA8VoHXoQcTlBa8mpH1s=; b=pUiVGAveJh6Il4Mcw7Qp+EDT/OB2XAsd8OSd2U6GnH4NwqA9YvmaRD/dQvdMayRovE vnF1bTkGmTaro4SuWMPZF60CGrKvajMHZOWrRYpMGVbYnIYt7q843Vf/asBK014BmGic 5Ca5bkz3fUQiXEcc2m91tHXavaoH1fqOIQY6hNRxOBXRXXnGB8jrc0bp0hVIP3i377N7 jkWPrjC0QMva8UyX0KR0ZYoFAAMU6anGCtKVWwFluTuSF+M9wqiQeR7Vd8Vy7QrEtNP8 JUBeOHiWvVcI6Rbq8kdXORgubka02GSEHJKWangq6Mzu4yGAb3GTFjXKcLHIjWcCtGea sABQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=XccDeVbR7GfrXtvOalK59/obA8VoHXoQcTlBa8mpH1s=; b=ubXuF9p09A2eP49QgTQbmv0cgBuzGkggsG3spsva/OXgo+7Rvk4MTGVoSvbsGzONg3 SfxQbvTsOlDgFLDvmibFnCnnUx2Vg2LODou7fVi3jN1iZQVUL90HcItnSdKNm4wlIjRg MWspABcKdq2NZxUY33yP/6KxJqo9/nxOYgS1nDPiuICFUOlTjcol3fm30mBorccNQRZO ESRtfCam90BsfsCqIToE8B4bQAIDIwJtbCEUn59sBdC1U4b3wy1Nyr35ADVQkMObSmx0 QVSj0u8pYTKyxFPqYZeX2MNkY682WHwZyvpDLpz/apPibnqfL2O3gAwgy7hBpVnB6VaN 4/HA== X-Gm-Message-State: AOAM533T323jPf2+unrq0t1D1zQh1FG/56CtJJXJrPMF86GUfjeg/6tc rh4AhJui7HytAYYyWz7Ee+RC0MQnyG39v9WneYSEcw== X-Google-Smtp-Source: ABdhPJz5DRV3FsSqRGLSh/a+h3WcgoEYSBuhDdDtiXmHc2MMSSpL8C2UmUxD8V5uGCjWSVwiREQWzdz8RV8PUTreLFo= X-Received: by 2002:ae9:ebd5:: with SMTP id b204mr15633763qkg.195.1610775815195; Fri, 15 Jan 2021 21:43:35 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Fri, 15 Jan 2021 22:43:23 -0700 Message-ID: Subject: Re: Major issues with nfsv4 To: J David Cc: Rick Macklem , FreeBSD FS X-Rspamd-Queue-Id: 4DHn601rsnz4sdx X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=pUiVGAve; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::72d) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-1.61 / 15.00]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; FREEMAIL_TO(0.00)[gmail.com]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::72d:from]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_SPAM_SHORT(0.39)[0.391]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::72d:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::72d:from]; R_SPF_NA(0.00)[no SPF record]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2021 05:43:37 -0000 On Fri, Jan 15, 2021, 10:24 PM J David wrote: > On Thu, Jan 14, 2021 at 5:30 PM Rick Macklem wrote: > > One thing to try (other than a FreeBSD13/head system, if possible) > > is the "oneopenown" mount option. > > The odds of being able to run an unreleased version of FreeBSD on > production servers are slim to none. > We run FreeBSD current on all our production systems at Netflix. We update monthly and run most of the updates... to be fair, though, we don't use NFS.. Warner While trying to develop a reproduction, I think I have narrowed down > what the problem is. There are no jails or nullfs involved here, just > NFSv4.1. > > Window 1: (to track OpenOwner/Opens) > > while true; do date; nfsstat -E -c | fgrep -A1 OpenOwner; sleep 1; done > > Window 2: > > mount -o ro,nfsv4,minorversion=1,nosuid fileserver:/path/to/freesbd/root > /mnt > chroot /mnt > > (OpenOwner is now 1 and Opens is now 9.) > > Window 3: > > chroot /mnt > (OpenOwner is now 2 and Opens is now 18.) > ls > (OpenOwner is now 3 and Opens is now 21.) > ls > (OpenOwner is now 4 and Opens is now 24.) > ls > (OpenOwner is now 5 and Opens is now 27.) > ls > (OpenOwner is now 6 and Opens is now 30.) > ls > (OpenOwner is now 7 and Opens is now 33.) > bash > while true; do ls | true; done > (Allow about a minute to pass, hit CTRL-C. OpenOwner is now 4647 and > Opens is now 13957) > exit > exit > (OpenOwner is now 4647 and Opens is now 13952.) > > Back in Window 2: > > exit > (wait about 30 seconds, OpenOwner is now 0 and Opens is now 0.) > > So it looks like the NFSv4 code can't let go of *any* Opens on a > file/directory until *all* references to that file/directory are > closed. > > If chroot is too much, "vi /mnt/etc/motd" in Window 2 and "cat > /mnt/etc/motd" in Window 3 have the same effect, leaking one Open per > cat instead of 3. You probably don't even need a FreeBSD install on > the NFS mount; just hold a single file open in one window and > open/close it repeatedly in another. > > Then I re-tested this with "-o > ro,nfsv4,minorversion=1,nosuid,oneopenown." At least for this simple > case, the problem did not occur with oneopenown set. > > Are there downsides to the oneopenown flag other than breaking delegations? > > Thanks! > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Sat Jan 16 05:53:50 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6A28F4D4BE5 for ; Sat, 16 Jan 2021 05:53:50 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-lj1-x232.google.com (mail-lj1-x232.google.com [IPv6:2a00:1450:4864:20::232]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DHnKn2Nc4z4tN6 for ; Sat, 16 Jan 2021 05:53:48 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: by mail-lj1-x232.google.com with SMTP id n11so12707633lji.5 for ; Fri, 15 Jan 2021 21:53:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=tanuLQvtvPsD+E3O0FFCdFSRvhPU92lrHELu5nkWUtc=; b=VnjNmb60vJhYSVnKGMmo6uTn4WFSQ1rA+TnDlKHv+p0bCgJnQrj+IP9hDA098sEUnq MXG8vXgsK7I3CU5XcAt4AjIgjjcKDJUpRFTxerHWnsJ1WUH4YFiw6SUhtcLoss8Fb4sX AOvWlMzH6ZK4GeB7QJiRaS6ggL3YkbH08cIiU13OKMSyb1zVh63uilIU06x6/SdgKTh7 FdXfHIYyHqSxLy5qtibbVFEZxZ6VXtd/RkzvGP3fo4gV7fCREZotLq/I1Q4GA7sPpH8c dTsc8sDc96QCjlb0vwZgMrIXwGE8H3st0xsCQ7rESDCC35Sfb5cs1o3IfYvi7D79Dnyg 2S2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=tanuLQvtvPsD+E3O0FFCdFSRvhPU92lrHELu5nkWUtc=; b=pXj57dQ9UVLrchZgCB85tYe3mVSXmPQr/0ossns7lweqt96AXfqOW+u8via7acjjOD k24QNtGJkUT+NzITLZYhgvVlho2m8uP4O9VoqvyZGdY5xo7FFuBAthATRxlCQAKseNYH 3GRjlc+afHbWl2ghYQ346Alp1lQ/oDGu2+TPJpeWl+e3bBI3h2HpVooTgdElv6exoloX SvQZBaDT+98Oqbz1PX7pW4PzaLmUyBsEuu6bZNwTvsW6SccqL3XJv6wN3mdP/CqtM/xm v0DnJkuwMmay+639joZaQMRauOWWd6F28zxgpZFaDeoxs5KeJQzI85UT8TOshEzjmeX7 Gq8w== X-Gm-Message-State: AOAM530O6imznHykr96yQe8nTb6BIORNH48t5YOu/1gNaDJqzHJcPq8S hRR8muUe6GRmCkHGOG8PzKCP7cTeVvnELgJC9D5UySs9 X-Google-Smtp-Source: ABdhPJwL5SFeKGZjay1OFBfmUbiA08C/g52wge+67VWhgxmu4RaB3afk7YA2EJok8YRjKGTe3wanQI4pLXJ5rZ365J8= X-Received: by 2002:a2e:9641:: with SMTP id z1mr6489708ljh.171.1610776426707; Fri, 15 Jan 2021 21:53:46 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: J David Date: Sat, 16 Jan 2021 00:53:35 -0500 Message-ID: Subject: Re: Major issues with nfsv4 To: Warner Losh Cc: Rick Macklem , FreeBSD FS Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4DHnKn2Nc4z4tN6 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=VnjNmb60; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2a00:1450:4864:20::232 as permitted sender) smtp.mailfrom=jdavidlists@gmail.com X-Spamd-Result: default: False [-3.17 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.17)[-0.166]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::232:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::232:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::232:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2021 05:53:50 -0000 On Sat, Jan 16, 2021 at 12:43 AM Warner Losh wrote: > We run FreeBSD current on all our production systems at Netflix. We update monthly and run most of the updates... to be fair, though, we don't use NFS.. Netflix has billions of dollars and a huge engineering staff intimately familiar with FreeBSD internals. You get to do a lot of things that aren't realistic in the average use case. From owner-freebsd-fs@freebsd.org Sat Jan 16 22:57:40 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CC3A04EE34A for ; Sat, 16 Jan 2021 22:57:40 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660051.outbound.protection.outlook.com [40.107.66.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DJD375z0Nz3FSn for ; Sat, 16 Jan 2021 22:57:39 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DGNPfeauODxYQ+jJs1a/Loo909UJ9RX53WfUuC+NQ4Fb53AvT0gMU1szeGOXq43EpBEouD262mg6aeXQh7VwmBzpElgTIXR14HYx3hU1ACjtzCk6wKDRjolR3uLxTbWbXt6vkdm0F+GA274D1J6xyHycGHVqHRqkJOn4hYFE5q6v7tMnHoVQPaXg3kewXRsYP8Naon84yBSvaPXK9o+cAGTReN7QfbDBo8WibVaVwvi17H6a5NR3ixvNigAHAu1sz5gepaFzDsU4RWM6FSIrbHRl0ZsCvLa3LqZFFCJwTP25k6uFYuCA6/SRTEtcdDzUDW5ZEmNq+x9mwUPKf9jocA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DUPBAOwc1YQwEtodr25hR6mDbReloLeoaN/KmiWw9eQ=; b=i+9K2nf9nYDkvUKoPRJqB8zho+ajFSWuaNmII4HAsQ0GYFzJRMvo3cx2E/T7W6LbTWs32jFraKrNNN9hlRjMmzSgqKl6GL6/Bk1GSm+lorNx13ThwwXaFkjtIFAyLKhoYS1dBHyKyacnKxqD5WCJLHpIzhT6nWHpPHmZnEutbzRqo49+VF3Ee6rlWX31b04ShmL9raNhstIaqgxorz/WmJoZOLrpHg8ETCjejc/X2nw2iktyRu3P8jGBuCtHjlGPMjhamp318AyDffKQg1fH9VsK4bMxptWFRBYsqOqQxrA23W34ycdu5oxcSMdgcCJPPdiQfUt357uF+3dOmuyKGA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DUPBAOwc1YQwEtodr25hR6mDbReloLeoaN/KmiWw9eQ=; b=LaglVyMizeQ5PtCErlHeG3c4+2DS6rmSfQhQ4CJvLROkja233TU1sny2mUmOVSeV7/FMjJMeVToQ+jhVHBnqpcuOsZu84bCCCNkDWZYoHo13jDOzOLmm3tPf6dyy+HhK45UTzmvTHhddjaxI88MDghb5bFWDRuAifmLe+xVQbiz6FNYTSO5p8/sHoV2cbwOv33Zxsus0DjHMt3HX3uu1lTHn7Cb2UUVHmNFamhc5YY3rsY3W2gsjhZiQm6t11SG3j+mAcQ5vnYntZFS9Wd9MN9cXQBrHi9ZedOCX2Emfi5MPp32iCKAjGxw5lFlZrKqI029+9/7NKgJ07A2ampBdaA== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQBPR0101MB1761.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.10; Sat, 16 Jan 2021 22:57:37 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::3d86:c7f9:bc4c:40c0]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::3d86:c7f9:bc4c:40c0%6]) with mapi id 15.20.3763.011; Sat, 16 Jan 2021 22:57:37 +0000 From: Rick Macklem To: J David CC: Konstantin Belousov , "freebsd-fs@freebsd.org" Subject: Re: Major issues with nfsv4 Thread-Topic: Major issues with nfsv4 Thread-Index: AQHWzw/HDat+dHoH9kKG5K3Xpd53kqnxDteQgAFi0QCAABTa84AALLCAgAAVvcmAAAu0AIAA4wiAgAAI5gCAAO/IAIAAiDrmgABWOgCAAGv4AIAEv9nSgCyfuACAAIn3C4ACDQ+AgAESerU= Date: Sat, 16 Jan 2021 22:57:37 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 18151869-107f-4eea-57aa-08d8ba721eb6 x-ms-traffictypediagnostic: YQBPR0101MB1761: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: V8cMPJawYCAy3RnGp23Z14vy4v7XPe8D9DEFLbq1SdxrF0rbQKqFY0MgokkfkaBOPwpSJgl7A6KUF8fosTOV84hdFdhpsttjEIpjpRDJDGzmRraiyrHGddyvCgCPQ/G7tY57Rm7CBEFUp0QBOSFKFLGMwE4eMnXHwZ9HIaWkVNbBcUOqSiY7JuMuro0fWUoTg2yCGorZqZMyOeBeFmCyn6YScWHi6B5WXVI55ByGLx45K69MguXOIctC+AbkB5aCsQ3SEXgfGg2yb6Zz+K+H0few0pbWVT3sIIVqaBuTmbVeW60G2GjX9TK/rHimel4FkuqHMQKNYOkyvGkZ68Ub0C4EgbnkjWTx2IiD6cs0GBOt7J4adLy++4XYXe0kbYvcdMeMzmROEcZfoRMCMPgdTQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(366004)(346002)(39860400002)(396003)(136003)(376002)(2906002)(186003)(4326008)(6506007)(83380400001)(33656002)(478600001)(8936002)(86362001)(786003)(316002)(71200400001)(55016002)(9686003)(6916009)(8676002)(66446008)(64756008)(66556008)(66476007)(66946007)(52536014)(76116006)(91956017)(54906003)(5660300002)(7696005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?DiHy3wdkE++nFDKIjqWwpSZF8Rod8yxhgACJNoAQ5sUlF+n5GGD5I9FmC2?= =?iso-8859-1?Q?sIpdUCrby9I/WrNlLDDxKM527F6EAFtWY0aZS/WXwI6G69cbIq+6zSkwmG?= =?iso-8859-1?Q?RIRGjqRh4oMWb8Mp/LyxQJM5KQNq3VvpLP7agzky/Gc9ZlAf0/cDcy5FQC?= =?iso-8859-1?Q?QZiND7z7C5YNaEtaBn9sB2kMjGDBbEwrITXHuQRI7kIcNxCa+sKvPHLwv6?= =?iso-8859-1?Q?IF/q/in07Ft9LPiNUo4VKHaeI2ZBd5TRLhVZI3KFA3r5PLCfm2hkHC28jd?= =?iso-8859-1?Q?LlPbC1hUZij+ov2XX/xM861BYQ9N7GoFXHteNEfQyEjF1mV6NF5HQ+/Sey?= =?iso-8859-1?Q?hmTBeD41kLV83X2JUpzjbY5oGOy91+hsTTWpCcluzBzw2K9TRnXxVdNEUD?= =?iso-8859-1?Q?bA4lRLwV47tIRd+Z43nb0Y4M5n/gkw88DrYOxpB6YfkcciZDdNKmcmYDxn?= =?iso-8859-1?Q?fAidLaiaxJFkOhx0oI9Pf84BLDVkS02BonLvHuUTfGRJICZOWJDvyN+au6?= =?iso-8859-1?Q?QmCreccaYyQkuqvNCDjKmffU0E8JmVM8PVaeO1p/P8oWSzSERQZC3fz/WN?= =?iso-8859-1?Q?ni1eLi8Kn9VZm/4p7l1w4IfceehPs8084QJvSmfegoFV/HIbtLxpNiNAf/?= =?iso-8859-1?Q?5Trjsx/zjLGpT+41p7Gx4WJ+/5BtGl6Y1jE6cjGdn7kiDs6+DjZ9mRvPO2?= =?iso-8859-1?Q?Knt1qnC0nb7xaCPSd2SnUmWf8GLAenb5D+1hkrbNKWhYCdw7k+xqRrClQj?= =?iso-8859-1?Q?fiCsQZ997Jtz4Czgd8qaEsqP1JNBNhp1BT7Nm5OAdLg8zsyo+jw+9vxeul?= =?iso-8859-1?Q?n82ZT5zyYcLR60Db+L/N17dLzZ2INdR5OFqWWKW06KU5l5g6o1npAxu8zK?= =?iso-8859-1?Q?v8WXrHtrpyjeB/fRlqYv8qzD4loIDLMeOT5CpZvy0yXio9B8Vxl5ARFMJB?= =?iso-8859-1?Q?2CP2p8gSaoDEePZatAvQLcWpcgrk0ZPZFKh2IK9u+xrth2PDi2H+KsIL8w?= =?iso-8859-1?Q?dw0uUT+1D6iavgehUnF15DCkkS7ErrDi7RFYG4I1mhukh82z9pHpnzQY2s?= =?iso-8859-1?Q?Q3SmDwb94s/PfYj1Urnq6TU=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 18151869-107f-4eea-57aa-08d8ba721eb6 X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Jan 2021 22:57:37.6789 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: NlQEly9//iylfo0XsEoURdTvxJ3JQ+lRbW2rUlIKztgeSd1NmkrusYIpj2r7f9S98p6PXS6Pk9/wb5pzaUS0YQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1761 X-Rspamd-Queue-Id: 4DJD375z0Nz3FSn X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=LaglVyMi; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.51 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.66.51:from]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_SPAM_SHORT(1.00)[1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[40.107.66.51:from:127.0.2.255]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[40.107.66.51:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.66.51:from]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2021 22:57:40 -0000 J David wrote:=0A= >On Thu, Jan 14, 2021 at 5:30 PM Rick Macklem wrote:= =0A= >> One thing to try (other than a FreeBSD13/head system, if possible)=0A= >> is the "oneopenown" mount option.=0A= >=0A= >The odds of being able to run an unreleased version of FreeBSD on=0A= >production servers are slim to none.=0A= Well, the current release schedule has FreeBSD13 being released at the=0A= end of March. If you set up a test system now/soon, you might be able=0A= to determine if an upgrade to FreeBSD13 is justified when it is released?= =0A= =0A= >While trying to develop a reproduction, I think I have narrowed down=0A= >what the problem is. There are no jails or nullfs involved here, just=0A= >NFSv4.1.=0A= >=0A= >Window 1: (to track OpenOwner/Opens)=0A= >=0A= >while true; do date; nfsstat -E -c | fgrep -A1 OpenOwner; sleep 1; done=0A= >=0A= >Window 2:=0A= >=0A= >mount -o ro,nfsv4,minorversion=3D1,nosuid fileserver:/path/to/freesbd/root= /mnt=0A= >chroot /mnt=0A= >=0A= >(OpenOwner is now 1 and Opens is now 9.)=0A= >=0A= >Window 3:=0A= >=0A= >chroot /mnt=0A= >(OpenOwner is now 2 and Opens is now 18.)=0A= >ls=0A= >(OpenOwner is now 3 and Opens is now 21.)=0A= >ls=0A= >(OpenOwner is now 4 and Opens is now 24.)=0A= >ls=0A= >(OpenOwner is now 5 and Opens is now 27.)=0A= >ls=0A= >(OpenOwner is now 6 and Opens is now 30.)=0A= >ls=0A= >(OpenOwner is now 7 and Opens is now 33.)=0A= >bash=0A= >while true; do ls | true; done=0A= >(Allow about a minute to pass, hit CTRL-C. OpenOwner is now 4647 and=0A= >Opens is now 13957)=0A= >exit=0A= >exit=0A= >(OpenOwner is now 4647 and Opens is now 13952.)=0A= Hmm. Not sure what files would get opened each time. NFSv4 Opens=0A= only apply to regular files, so "ls" shouldn't result in Opens.=0A= =0A= I may try this and see what the 3 files being Open'd are.=0A= (Obviously something related to the chroot. But what?)=0A= =0A= >Back in Window 2:=0A= >=0A= >exit=0A= >(wait about 30 seconds, OpenOwner is now 0 and Opens is now 0.)=0A= Yes, it could take a while to close all those opens.=0A= =0A= >So it looks like the NFSv4 code can't let go of *any* Opens on a=0A= >file/directory until *all* references to that file/directory are=0A= >closed.=0A= True for regular files, but not directories.=0A= =0A= >If chroot is too much, "vi /mnt/etc/motd" in Window 2 and "cat=0A= >/mnt/etc/motd" in Window 3 have the same effect, leaking one Open per=0A= >cat instead of 3. You probably don't even need a FreeBSD install on=0A= >the NFS mount; just hold a single file open in one window and=0A= >open/close it repeatedly in another.=0A= An NFSv4 Open is unique for open_owner/file, so the same file opened=0A= by different processes (an openowner represents a process unless you=0A= use "oneopenown") results in separate NFSv4 Opens.=0A= However, since the NFSv4 client cannot know which Open a VOP_CLOSE()=0A= is associated with, due to file descriptor inheritance, none of the NFSv4= =0A= Opens can be closed until all FreeBSD open file descriptors for the file=0A= are closed.=0A= --> Just the way it is. It is not an unintended leak. They go away once=0A= all file descriptors get closed, so long as the VOP_INACTIVE() gets= =0A= called for the NFSv4 vnode.=0A= --> It is the handling of deferred VOP_INACTIVE() calls that has=0A= changed for FreeBSD13.=0A= However, none of the above seems unexpected, except maybe for why=0A= "ls" in the chroot opens 3 regular files each time. I don't know what=0A= chroot actually does for something like "ls"? I'll look.=0A= =0A= >Then I re-tested this with "-o=0A= >ro,nfsv4,minorversion=3D1,nosuid,oneopenown." At least for this simple=0A= >case, the problem did not occur with oneopenown set.=0A= Yes. For the oneopenown case, there will only be one NFSv4 Open for=0A= each file opened.=0A= =0A= >Are there downsides to the oneopenown flag other than breaking delegations= ?=0A= Here's an example:=0A= - One process running as J David opens a file for reading, which works sinc= e=0A= J David has read permissions on the file.=0A= - Another process running as Warner opens the same file for writing, which= =0A= works, since Warner has write access for the file.=0A= =0A= Now, network partition the client from the server until the lease expires..= .=0A= The client now gets a NFSERR_EXPIRED error, which forces it to retry the=0A= open(s).=0A= =0A= Without "oneopenown", the above FreeBSD opens resulted in 2 NFSv4=0A= Opens, both of which probably reopen successfully (unless a chmod/chown/=0A= setfacl on the file makes the reopen fail).=0A= =0A= With "oneopenown", the above FreeBSD opens results in one NFSv4 Open=0A= for reading/writing. A retry of this one Open might succeed, depending on= =0A= what the file permissions are. (I think the code uses the credentials for= =0A= Warner in this case, assuming that the credentials that opened it for=0A= writing is more likely to succeed, but there are no guarantees.=0A= =0A= --> For normal operation, it should be fine. A network partition that=0A= results in NFSERR_EXPIRED is a worst case scenario, where all=0A= byte range locks will be lost and Opens may be lost.=0A= =0A= For delegations, the story is similar, but happens routinely when=0A= delegations are recalled by the server.=0A= --> For delegation recall, the reopen is done using a special variant of Op= en called=0A= claim_delegate_current. A server should allow claim_delegate_current= =0A= irrespective of what the file permissions are, but the original RFC3= 530=0A= did not make this clear.=0A= --> Is allowed by a FreeBSD NFSv4 server.=0A= As such, the warning in the man page is mainly there for NFSv4 servers=0A= where the reopen done at delegation recall time can fail, due to permission= =0A= checking.=0A= =0A= There is also the fact that the case of "oneopenown+delegations" is not wel= l=0A= tested and the current FreeBSD client code will use separate open_owners=0A= (violating the "one open owner" principal) when delegations are recalled.= =0A= =0A= Are delegations useful?=0A= - Short answer, often not.=0A= =0A= Delegations allow the client to do 2 things:=0A= 1 - More extensive file data caching, since the delegation guarantees that= =0A= other clients will not be modifying the file.=0A= This is not exploited by current clients, as far as I know. I had som= ething=0A= called Packrat, but it never made it into FreeBSD. More on Packrat be= low,=0A= for anyone interested.=0A= 2 - Do NFSv4 Opens locally in the client, avoiding the Open/Close RPCs.=0A= Since delegations are per file, this only helps if the same files get = opened=0A= over and over and over again. Doesn't happen for many loads, from what= =0A= I've seen.=0A= --> With delegations turned on, you can compare the counts for Opens= =0A= vs LocalOpens in the "nfsstat -E -c" stats, to see how many Open= /Close=0A= RPCs get saved.=0A= =0A= Packrats: Some now bitrotted code in the subversion projects area that did= =0A= whole file caching of small files in non-volatile storage on = the client=0A= when a delegation was acquired for the file.=0A= This was intended for devices like laptops, with slow/flakey = network=0A= connectivity.=0A= --> Recent FreeBSD changes might inspire me to resurrect this= .=0A= - FreeBSD has recently become more laptop friendly.=0A= - nfs-over-tls provides an easy way to make NFSv4 mount= s=0A= from anywhere (as a laptop might) relatively secure.= =0A= =0A= rick=0A= =0A= Thanks!=0A=