From owner-freebsd-fs@freebsd.org Sat Dec 26 23:10:05 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CA4024C5460 for ; Sat, 26 Dec 2020 23:10:05 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670063.outbound.protection.outlook.com [40.107.67.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D3KK81ZS0z4pxK for ; Sat, 26 Dec 2020 23:10:03 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RW2sPa0qH24kClc+XOFellrXTR2ZK5SZgIq24QKzQKHgT52kV4OWDl2Xg8UOy3arC9ie7D3QJ0Oxw3ogvPsIhikzSkPT929jTNAh5unLvD+JNonjdshzM3vz4fWauM7cghftsMi1lWHfWGs/02PHOJIkqitu15ircFDVEldDChb+Sim6QWk5Ig5sAmPZ3ddV4HCfagJPBlp+3XGHeHQsIq1BB/MSGMnN9x0rZl8jcFtDpLOy9ddyDaj2xKj+C2avHs6y4Zq2F6sNyfav9HpI0+3eUPwePpyx4huk2LhmYOHphOGJ0LLmEhoaHfGPCI3eB8KfPQjifWBoixUC3NBUSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CtBUoyJxLg+raTMH1Vdc+i2WLSo+eqsLTyUK990+cCU=; b=OTqmP0LDWqHMNqQYKISZGFuaA6soCIpHlmeTwqYYI2SX6WfQnYoJD/8OivntImZShPEeeKvVlxzfXf2PaDAqhraPpHadON8nTfZ8e9tpvlDOlxjy9VJE2uxvGbdEQI2jrr43Q2IMmQI9OFti43+Sc1JKa36njqvuHl1MOu+TpPtITPeENnvu8tsdw1KZ692RMqcbQta0Crv66M8c9j0/PWOZwQ3cJYu60JTu6uthVS2+cXj7d1QqT/SSOn5xlAfjO8xY1s6Lfi+IbKu+/LNKQANDQoHVrl/X/TdC+7JSHci5u8dG7UPMa54jaIKe7aHexR53V3E/qPHxzBR8TRevnw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CtBUoyJxLg+raTMH1Vdc+i2WLSo+eqsLTyUK990+cCU=; b=JS2ajxGnkG3XwKqH3Oy/Qux2l/8ys5CBlE4x/t5ODdBzuLyWFs1nF/njM0Nu2fwrABR/SFzTH+m9D4MdvOnzHYiUmmbw1YUTTmuQ25H0pZ+LysZXrZ5/0ATYIpxL8ULqkd6zdpR9Qy0PW+lIEaabz+mxPbdi6EAGMt7eKojw+LrfThS8HU4IjyMPKZQ4ZXXGw2uIvF+6ODCXr02LXWYYk0vhA4vKvKTDEGii8Ccve57zU10aNPbkR6XO4zDmAwYd5Kc8PW3hGWFod1fLmcBmLUwIQZKkUQzEvHfhph12Z/XCPe0dQMk5uGSHRJ6KpIrpRAdcl4aIBz0Bx/Sl3zZWGQ== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by QB1PR01MB2977.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:39::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3700.28; Sat, 26 Dec 2020 23:10:01 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::7d6b:aa68:78f4:5d94]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::7d6b:aa68:78f4:5d94%7]) with mapi id 15.20.3676.035; Sat, 26 Dec 2020 23:10:01 +0000 From: Rick Macklem To: J David , Konstantin Belousov CC: "freebsd-fs@freebsd.org" Subject: Re: Major issues with nfsv4 Thread-Topic: Major issues with nfsv4 Thread-Index: AQHWzw/HDat+dHoH9kKG5K3Xpd53kqnxDteQgAFi0QCAABTa84AALLCAgAAVvcmAAAu0AIAA4wiAgAAI5gCAAO/IAIAAiDrmgABWOgCAAGv4AIAEv9nSgA9Z2rs= Date: Sat, 26 Dec 2020 23:10:01 +0000 Message-ID: References: , , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 816dbe06-a9a0-4285-cf9b-08d8a9f35f90 x-ms-traffictypediagnostic: QB1PR01MB2977: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 55i8cWvzwB6GneR06i1yQfzX6pHz7queY/hzM5wgCTDm4uvsjmZmmBzxe3rV2S/zsfI7eCcCuAC+7xVqjUbKWVPTLNP/rFGMFaiYrQ/S0d7s+/bd4ns7zkEx7Sk9Z+eANIOhBrSjmf2iFNc6f1Q2Q/MRid0Cbq6Gj2kbM0RE4rKMWhvKBICiIl/dzHL0CTTUN3aOUrZ7awKqblvtAeBhmAGUwBgLVbwtO2auSDwq8rNKfCnanDmerGWpaiNhSFXAwJhh8x52H1gZGxTxDS+iBBBvATNlwNV/cfHYZzYjvpxjEBiZS/ReWGQ+jdWbHsQFtEUPnrlprmWD8nrWFFf1gYEt//OH+Bu0ScWqGQvzmtmbkKUuS47YHIzhl3gqpdZHMdH4q5y89L0Jr/xF6hA2F7VyOmmFSFVqiOUfod2Wx9Ole+AWp54aY9+266wzHnEpDUbR9BvItYKso5NEoROnkQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(376002)(396003)(136003)(39850400004)(346002)(366004)(8676002)(186003)(83380400001)(8936002)(2906002)(7696005)(478600001)(4326008)(5660300002)(66946007)(91956017)(86362001)(64756008)(66446008)(966005)(76116006)(55016002)(9686003)(71200400001)(66476007)(66556008)(786003)(52536014)(110136005)(33656002)(6506007)(53546011)(316002)(26005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?OswftiqXxHJ/KCQwYop5t+MyKQ21B/WN1XspkFAtVZES9Pq5olMAIH606A4l?= =?us-ascii?Q?HRw5G91t53LB3EYWfkykqKdySq8FPGYRDLbg5VJJ3hVE1J0EU5GX0NIQznK8?= =?us-ascii?Q?nI9n47+lNOw1mgJU4pm3yGmV5XWx/6qlMM+14ckwAdA5flW996KxjnXeYL/7?= =?us-ascii?Q?d8z5AIQonslX6cqwVn/76egfA2oWn5kRzzUMBVgw2fVgvrxVHrsMQkueIBMX?= =?us-ascii?Q?MR2wXO71VHvRX+qY1+1MxBwY19WEJCYQhUIGR8dlDfMQk39FjZyBGeYna1Kf?= =?us-ascii?Q?LniMJl5fz7SzemB+ELB8xjWxEAXcrRqI0H9ujhP2P4gx8Smoq/Jl+qQSVt2T?= =?us-ascii?Q?DALJ9CBEM0M60kthY7EiBTUXRpZx38vvtidICueU7B8MPKS12d1XUflYc0Ye?= =?us-ascii?Q?uXNNhy0IvtREvI5AsiYbYZUfId7jy+RLGmCn5AHygQnY6jV9KXvhXgKr54U8?= =?us-ascii?Q?zXaSSR8dRIDaIuQR4obvUpI8fK6V6o68VF6usXUf6nReq73IIsCITUjb1rsC?= =?us-ascii?Q?EcNoiaCDrNroKQN0SYaUWPCTZbtiPV+kINa8BAAsC7PzyjOuMjQlkw/kH7MH?= =?us-ascii?Q?FvXZAu0i0DDzkkcSRSnuUwQAr4mx37fKlRPrDPl6RAGt6UKxWhVJUWSv6CSU?= =?us-ascii?Q?Bp7A7iW4CJYm5Qu5M+vtqhr4JXNpZDo8LGkxMW5LiaHuGcn0xaZSQzAH7Ply?= =?us-ascii?Q?nwUTc31JjLHx6hJqgfkd9Cvv6V4hCKD7wCBo1vcjm6dWOrn0vu7qfT0HC0MZ?= =?us-ascii?Q?NR2cpBd3pApi9Rc1SunFloo7gey4z6EBBpC4nIgxYAS6lbCSux8xJCsrTFmU?= =?us-ascii?Q?KRyZ8OzJhDk8U8Ks+Qx7ZSkUV6HJcv9gbMblie3IY1kErMVRfr09jsHdO3AN?= =?us-ascii?Q?88PplUoUluV9Xz5IffkwxctOftsKhCDsCfo453ievPvScVIZiDQI1F/d3AWf?= =?us-ascii?Q?NrUJMq1+NkHnzAshJ3hxLLjeeUL9KVmGE84yIjDt35Q=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 816dbe06-a9a0-4285-cf9b-08d8a9f35f90 X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Dec 2020 23:10:01.6998 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: eq6D5pZidMOVWMhi9Zqs6uyQ4vLxgmT6RcFK+Wxmt/fQvws2MBTNS/msBOADmn2t/2uqG6pYAbVUjD0qpFCqfg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: QB1PR01MB2977 X-Rspamd-Queue-Id: 4D3KK81ZS0z4pxK X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=JS2ajxGn; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.63 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.10 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_IN_DNSWL_LOW(-0.10)[40.107.67.63:from]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.67.63:from]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FROM_EQ_ENVFROM(0.00)[]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; SPAMHAUS_ZRD(0.00)[40.107.67.63:from:127.0.2.255]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.63:from]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Dec 2020 23:10:05 -0000 Although you have not posted the value for vfs.deferred_inact, if that value has become relatively large when the problem occurs, it might support this theory w.r.t. how this could happen. Two processes in different jails do "stat()" or similar on the same file in the NFS file system at basically the same time. --> They both get shared locked nullfs vnodes, both of which hold shared locks on the same lowervp (the NFS client one). --> They both do vput() on these nullfs vnodes concurrently. If both call vput_final() concurrently, I think both could have the VOP_LOCK(vp, LK_UPGRADE | LK_INTERLOCK | LK_NOWAIT) at line #3147 fail, since this will call null_lock() for both nullfs vnodes and then both null_lock() calls will do VOP_LOCK(lvp, flags); at line #705. --> The call fails for both processes, since the other one still holds the shared lock on the NFS client vnode. If I have this right, then both processes end up calling vdefer_inactive() for the upper nullfs vnodes. If this is what is happening, then when does the VOP_INACTIVE() get called for the lowervp? I see vfs_deferred_inactive() in sys/kern/vfs_subr.c, but I do not know when/how it gets called? Hopefully Kostik can evaluate/correct this theory? rick ________________________________________ From: owner-freebsd-fs@freebsd.org on behalf= of Rick Macklem Sent: Wednesday, December 16, 2020 11:25 PM To: J David; Konstantin Belousov Cc: freebsd-fs@freebsd.org Subject: Re: Major issues with nfsv4 If you can do so when the "Opens" count has gone fairly high, please "sysctl vfs.deferred_inact" and let us know what that returns. rick ________________________________________ From: J David Sent: Sunday, December 13, 2020 10:51 PM To: Konstantin Belousov Cc: Rick Macklem; freebsd-fs@freebsd.org Subject: Re: Major issues with nfsv4 CAUTION: This email originated from outside of the University of Guelph. Do= not click links or open attachments unless you recognize the sender and kn= ow the content is safe. If in doubt, forward suspicious emails to IThelp@uo= guelph.ca On Sun, Dec 13, 2020 at 4:25 PM Konstantin Belousov w= rote: > Nullfs with -o nocache (default for NFS mounts) should not cache vnodes. > So it is more likely a local load that has 130k files open. Of course, > it is the OP who can answer the question. This I can rule out; there is no visible correlation between "Opens" and the number of files open on the system. Just finishing a test right now, and: $ sudo nfsstat -E -c | fgrep -A1 OpenOwner OpenOwner Opens LockOwner Locks Delegs Local= Own 4678 36245 15 6 0 = 0 $ sudo fstat | wc -l 2698 $ ps Haxlww | wc -l 1012 The value of Opens increases consistently over time. Killing the processes causing this behavior *did not* reduce the number of OpenOwner or Opens. Unmounting the nullfs mounts (after the processes were gone) *did*: $ sudo nfsstat -E -c | fgrep -A1 OpenOwner OpenOwner Opens LockOwner Locks Delegs Local= Own 130 41 0 0 0 = 0 Mutex contention was observed this time, but once it was apparent that "Opens" was increasing over time, I didn't let the test get to the point of disrupting activities. This test ended at Opens =3D 36589, which is well short of the previous 130,000+. It is possible that mutex contention becomes an issue once system CPU resources are exhausted. More about the results of the latest test after the data is analyzed. After that's done, I'll attempt Rick's patch. In the long run, we would definitely like to get delegation to work. Baby steps! Thanks! _______________________________________________ freebsd-fs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"