From owner-freebsd-net@freebsd.org Tue Jun 2 04:30:26 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 67348329D4A for ; Tue, 2 Jun 2020 04:30:26 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660081.outbound.protection.outlook.com [40.107.66.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49bfGm61y3z3VZC; Tue, 2 Jun 2020 04:30:24 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XaeMehFgUda16Uwl/qiHK2C6+15NFfA/42TyiCj3O5fwujbln9azgIx9QephDtcVqPE8GPgBVVQJy+xxELwU/dPHC67L+5MnS7jWLBx/ntyCgxKZwlBNNrYX9zqgtMAh4eXtXf2CvKWhrXsdGEyw5pORv9VKGJ2BIt1ZEtP7SUwhli6FR1DLEhE0sNyftjy1Br4GHXeEGo+FJHP2GL0CDOYxHvLrkmQ21tIXt7LC2EhL/1SLRHriqy450zE/i0zSqhsVeIE04cFDtqTS/gabCiLQwkZ8A/1V3CXH4ZaEgrB6F68WY4EhtETQTM+1fvw0+x8RPHH+lrK5ji/4WotZ+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7E+piRq/F/tIUYpz4tcx2OhBvaG/rs36gt0zZ39GaJY=; b=RDD0jcSEnIgJDvnO2U8wGV4ewZjQJEzi41CjCuyg1Em8e5TLOpd5xU5DzcuklL4TQexVRcTiWmVLqUuQD/eKq2qTFbC/GaiYq6CKDiYXExL9Ca2ZfgH3NMUGEgOEaAh/PO4gIw/LE6rCtMILnMyP9B73WUc7A9efywndipV0DnerdYMAJtxuSnMG7IFIAfgkqWEQYubsAVzVZ8Jp4UAG6B6GNYwZpoJo5FUJP/1adpg2z2t/B5J00OvkeU2c4Kx2neHq2ce+suF2SH6IUG/BGiYyHSyjdtUPJnqxukgS+rmKxdnZh7lKjB+C7U7vceWNFWPaSSJ9I4uq+r3/i8ybgg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7E+piRq/F/tIUYpz4tcx2OhBvaG/rs36gt0zZ39GaJY=; b=U7boCcIGeyjKAM/w6Bv4cv850pcroH2cYEma0ra8J0DsZtXh31GOcaOUSdqzz8o/5bfCl3zMRQaSrJi6Tnt0qP8nlMcn6m7PkeNGxyrexp6UNSAhmy3uJsf+eqFVjeIAlXsubeyLKwsYFE8Hf8M6nFhHOSbSpmfZj1lx0asJVBJmqT52OH1h5q8Tov6EzySCp41JYauoD2QNDsd14lR0dJ+qAYrX9bDdgx559Cfndm+ZtgFO3bLYm4ZGipACDbnwzUH525Fr+2tREEYREsFj5FG5GDOVswjIR5ZGgY5L63Z1P/J4ZmIxSHsVfOPC7FEnxmDjbDmpiGpagI9fdlxWpw== Received: from YTBPR01MB3664.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:25::19) by YTBPR01MB3935.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:1e::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3045.19; Tue, 2 Jun 2020 04:30:23 +0000 Received: from YTBPR01MB3664.CANPRD01.PROD.OUTLOOK.COM ([fe80::692b:69ad:7a4c:a465]) by YTBPR01MB3664.CANPRD01.PROD.OUTLOOK.COM ([fe80::692b:69ad:7a4c:a465%5]) with mapi id 15.20.3045.024; Tue, 2 Jun 2020 04:30:23 +0000 From: Rick Macklem To: "Rodney W. Grimes" CC: "freebsd-net@freebsd.org" , Mark Johnston , "patrykkotlowski@gmail.com" Subject: Re: how to fix an interesting issue with mountd? Thread-Topic: how to fix an interesting issue with mountd? Thread-Index: AQHWOJM1VwWbfER5x0eYgpEDvZCC2Q== Date: Tue, 2 Jun 2020 04:30:23 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: e880e8c9-5be3-4686-f305-08d806adaa68 x-ms-traffictypediagnostic: YTBPR01MB3935: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:7219; x-forefront-prvs: 0422860ED4 x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: JO6AN0UxaYvGzwnIUGeI3RQZkJz0vNMG4Mj3rWmGrcVV1H+wQvtECJcQ76V4sBUDLu0MN0uIEvUN77nGge3dIXiTICoxNOynTdrn1ZxxXbx2+XZUCzOG2JT51gxM/T38qj+7kcStNOu8XuLOEV+cF5Fx/k4m02uPi1j0ahNTxkjAgmjdaAPVPYSCM4ddLCFE4dfkFTc2ey0GQFT7s6QAccrm0xP/tns6vCkpyhP4ZBvaCA3YYiniNrPAlVirDRp++B0+UYb1fcCSJ4fBx/158dIHpp584w36NIB6FXuhC3Gyf8CuJqjx0LKsIC0utnSEgR2Sm94ADaM9MqBwsXhkqw== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YTBPR01MB3664.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFTY:; SFS:(136003)(396003)(346002)(376002)(366004)(39860400002)(86362001)(71200400001)(6916009)(316002)(9686003)(55016002)(4326008)(5660300002)(2906002)(786003)(66476007)(66446008)(54906003)(66556008)(64756008)(478600001)(186003)(7696005)(66946007)(76116006)(6506007)(83380400001)(8936002)(33656002)(52536014)(8676002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: V2nKhAYhdIpWF4bGcaPDTohGrLuxDCfVrwt6G4GhVbfVzuEGXVghzX6sakeAdfSJbLJnMUJLa1uyJwUx0BZBVu9QgZGqbnUIIFMBCVcT78hwjjTMJQ5rfdgCvkVCB0X3YraUwLxpelgP5V2jx3WPe8p6bypZURwByDFOxknQYskvlbyVYCWpvwnx0tQNxBUREFiQalICM6n2b1WZWhCPBBKto2UapsmSD0h0GONsceXsTQ5ah3KneoXmMrGVbKmcdPoeEiIEZp3lXov4Arq+oZqqZxg9w8NIgcLg8vGFkuqxpnTbzBNy2pOUBJwk/fdwZPpubDCPoPtR76SShqGDYJkW/7MKGpJ10p05x48GpI5aU40KgF7lE+ttw2/omBbpLX/lngtju6ZnUF6KOzw4DfMymntjjy+c+Hc3SFYzbfVyXsO2xTKa+tPbY+mQhLniUWkJYcWB2MDZoPk3ZM/KpPBmAd2MMuM7Aq8WUaFVA/hU7RWzM8Uo7Yw9IWK6amqUETFjkvm9jVRpwKhy7NuMRxom0+jtMGCT7LnceCHglDo0IV25UxjKT5qzAFFxd6pV x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: e880e8c9-5be3-4686-f305-08d806adaa68 X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Jun 2020 04:30:23.0509 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: beEH+OT+gMX6Ez5fcWQxLwqOI2BzGW+Lw+w+lKESBt90eN8yc71ez2OQtQyLGBhbPe4/zMW0dsGT2F/Tc0BMzw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTBPR01MB3935 X-Rspamd-Queue-Id: 49bfGm61y3z3VZC X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=U7boCcIG; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.81 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-3.54 / 15.00]; FAKE_REPLY(1.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; NEURAL_HAM_MEDIUM(-0.96)[-0.957]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[uoguelph.ca]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; NEURAL_HAM_SHORT(-1.06)[-1.059]; RCVD_IN_DNSWL_NONE(0.00)[40.107.66.81:from]; NEURAL_HAM_LONG(-1.02)[-1.019]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; FREEMAIL_CC(0.00)[freebsd.org,FreeBSD.org,gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.66.81:from] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2020 04:30:26 -0000 Rodney Grimes wrote:=0A= >> Hi,=0A= >> =0A= >> I'm posting this one to freebsd-net@ since it seems vaguely similar=0A= >> to a network congestion problem and thought that network types=0A= >> might have some ideas w.r.t. fixing it?=0A= >> =0A= >> PR#246597 - Reports a problem (which if I understand it is) where a sigh= up=0A= >> is posted to mountd and then another sighup is posted to mountd while= =0A= >> it is reloading exports and the exports are not reloaded again.=0A= >> --> The simple patch in the PR fixes the above problem, but I think w= ill=0A= >> aggravate another one.=0A= >> For some NFS servers, it can take minutes to reload the exports file(s).= =0A= >> (I believe Peter Erriksonn has a server with 80000+ file systems exporte= d.)=0A= >> r348590 reduced the time taken, but it is still minutes, if I recall cor= rectly.=0A= Actually, my recollection w.r.t. the times was way off.=0A= I just looked at the old PR#237860 and, without r348590, it was 16seconds= =0A= (aka seconds, not minutes) and with r348590 that went down to a fraction=0A= of a second (there was no exact number in the PR, but I noted milliseconds = in=0A= the commit log entry.=0A= =0A= I still think there is a risk of doing the reloads repeatedly.=0A= =0A= >> --> If you apply the patch in the PR and sighups are posted to mountd as= =0A= >> often as it takes to reload the exports file(s), it will simply r= eload the=0A= >> exports file(s) over and over and over again, instead of processi= ng=0A= >> Mount RPC requests.=0A= >> =0A= >> So, finally to the interesting part...=0A= >> - It seems that the code needs to be changed so that it won't "forget"= =0A= >> sighup(s) posted to it, but it should not reload the exports file(s) t= oo=0A= >> frequently.=0A= >> --> My thoughts are something like:=0A= >> - Note that sighup(s) were posted while reloading the exports file(s) = and=0A= >> do the reload again, after some minimum delay.=0A= >> --> The minimum delay might only need to be 1second to allow some=0A= >> RPCs to be processed before reload happens again.=0A= >> Or=0A= >> --> The minimum delay could be some fraction of how long a reload ta= kes.=0A= >> (The code could time the reload and use that to calculate how = long to=0A= >> delay before doing the reload again.)=0A= >> =0A= >> Any ideas or suggestions? rick=0A= >> ps: I've actually known about this for some time, but since I didn't hav= e a good=0A= >> solution...=0A= >=0A= >Build a system that allows adding and removing entries from the=0A= >in mountd exports data so that you do not have to do a full=0A= >reload every time one is added or removed?=0A= >=0A= >Build a system that used 2 exports tables, the active one, and the=0A= >one that was being loaded, so that you can process RPC's and reloads=0A= >at the same time.=0A= Well, r348590 modified mountd so that it built a new set of linked list=0A= structures from the modified exports file(s) and then compared them with=0A= the old ones, only doing updates to the kernel exports for changes.=0A= =0A= It still processes the entire exports file each time, to produce the in mou= ntd=0A= memory linked lists (using hash tables and a binary tree).=0A= =0A= Peter did send me a patch to use a db frontend, but he felt the only=0A= performance improvements would be related to ZFS.=0A= Since ZFS is something I avoid like the plague I never pursued it.=0A= (If anyone willing to ZFS stuff wants to pursue this,=0A= just email and I can send you the patch.)=0A= Here's a snippet of what he said about it.=0A= > It looks like a very simple patch to create and even though it wouldn=92= t really > improve the speed for the work that mountd does it would= make possible really > drastic speed improvements in the zfs commands. Th= ey (zfs commands) currently > reads the thru text-based exports file multi= ple times when you do work with zfs > filesystems (mounting/sharing/chang= ing share options etc). Using a db based =0A= > exports file for the zfs exports (b-tree based probably) would allow the= zfs code > to be much faster.=0A= =0A= At this point, I am just interested in fixing the problem in the PR, rick= =0A= =0A=