From owner-freebsd-current@freebsd.org Thu Dec 31 22:56:14 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 57A364D39C0 for ; Thu, 31 Dec 2020 22:56:14 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660068.outbound.protection.outlook.com [40.107.66.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D6Nmt1HfLz4VV3; Thu, 31 Dec 2020 22:56:13 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iPcAmLyKLtbiGpm42VxEyknkj97r9YPK0yMLHpUJR+lCRbgl8UTT05IgB8Zf89uDau/sLhJLg15sa9Us0REI090aZStw+uZNiSpQ9dFWC+daeuYutpxvbixzF2jSSmrXkU+PAijlveC/QgHNOxKv5GO5SWb/gEKrd2OrWehv90ZaS/GVM5hUPCalgPYxjK7PEKmYP7hdUtDVw8aIgDVvAsDny0L0LGwfLC1ielEoLvqHBRvnrVYazj+4CPkjYdT8oIwdne7JpSq/J7a7zcD2sgjjVWbmYSaNDlJUwP9hmUxksk50L3YTy1ISIODSZlBLZweh9BrqbqEY4ub4hrRo6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fV9LYHICxaZKp1X7mEPgZR+mz/Gd1S6ZfKQK4TmGG5U=; b=X8w0AvWFZCUjRFR/b/3aiTlPXpBZnSuctVvmJZt7NZGOjEs1rbjiN5xoYl6FkXbQ89qUQK+D7CIzotB8/UiYzRDnKa3pNbbTvJCtPjN5VE0On/M4ouS59UwStCa640GRIJcrh1w/ytl0Dw+WgJl2lkgULnEiLnWAjY/z6mjXyp4qBrQERxBU7XasC9/5D55Xd75dcsCQpXXI5dOeY4tLsmn99LkAaR1HhdqOQ621qvZMQK6FPQbe/LGp6d4JLxI+OC3EiCvA9BR8DwnPxaOOx4SL81YAEytgdr7RkDBrSIhQwpLEY2vLq2CiL6xN7tN7t9GJDvznrw6jw5enyuOYLA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQXPR01MB2583.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:4f::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3721.19; Thu, 31 Dec 2020 22:56:12 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::3d86:c7f9:bc4c:40c0]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::3d86:c7f9:bc4c:40c0%6]) with mapi id 15.20.3721.020; Thu, 31 Dec 2020 22:56:05 +0000 From: Rick Macklem To: Konstantin Belousov CC: "freebsd-current@freebsd.org" , Alan Somers , Kirk McKusick , Mark Johnston Subject: Re: r367672 broke the NFS server Thread-Topic: r367672 broke the NFS server Thread-Index: AQHW3k3ynlL3lGhXRkC37zjbBa1PUKoPmNaAgAA7NVaAABFYAIAAwg6JgABvToCAALma1A== Date: Thu, 31 Dec 2020 22:56:04 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 34e3ab03-b2a2-4ad3-32ec-08d8addf40d8 x-ms-traffictypediagnostic: YQXPR01MB2583: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: wCxShn6Cqj+ZWzg8uQTdJwhQXxvs/Nl4hn0san9smsLZyody6ybUkpVJqmIUJqV6LoIOzCOV9svL7ZGoTDOnCAuvoH5SpsQlrZDrD6QEjQwfmyVH6FSWGCuSGO+/K0I5MZlaw552efHGgFxPvKp4LCjbFRc5Er2f7ep3XLuBqE8Hf1hhhA8/n83ej6iN7WU4Qwt02kWTiiok9xa3WvEe781r/T1YMrBLsgz6BM3t1oB2Mfc6XfDXjxBgmzyuDI4HmL0FMzgNfqDCnZGY8Us8W0nlgH1knkAONEUfe/j/H+no7T+mlA0s6bxnfw1OoL10zmHFKGqv9UtSmGw6fAB2wwo+Ob46g7q2U+pSNPBlVGMA04FmWmlsU1cDr8rqfyzL0wgHX7AXfkEFXtWY6DpuR/9vj0My+Aq+8Pw8wnPVJENHc/ABrmge2BWqOMfpUawdPR1hTOs0vONfMWdouWSCWA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(396003)(346002)(366004)(39850400004)(136003)(376002)(4326008)(478600001)(966005)(33656002)(8676002)(9686003)(55016002)(8936002)(83380400001)(71200400001)(86362001)(5660300002)(186003)(54906003)(52536014)(64756008)(66446008)(316002)(2906002)(6916009)(786003)(26005)(66556008)(53546011)(66476007)(6506007)(91956017)(76116006)(66946007)(7696005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?8S3KdPf8LMCdhyQMR44F0xuBm8rPXt0CXfOsHPxMzsbgzLxPQGYXAN7M6V?= =?iso-8859-1?Q?hmmitKbzpY4sQecfvmYOP7MAfnuo6UUGDavjIgTiQUPmMhdxGCoZUPRYFH?= =?iso-8859-1?Q?YLxu9CvFLFec2W5VuHS2gT3du3y3AR4+ROfKbHhRp9aB02ewYuao89YbSr?= =?iso-8859-1?Q?rbB4QP4DYk1N/VSqKYOsBjnQ0lmvmRpv0cJt+lYJzZ9za6+ah1t7tg0kWB?= =?iso-8859-1?Q?UhXhPnDP3dCKW27nGNOZ11uscxKwOd1IPMbKLcog5n7oXzw6Fcq3b6K1Xh?= =?iso-8859-1?Q?/biNN+MZqA6rSASwb8/gICDwI09JqaR4ge/0whe65dsGqcUbg4lkT9Pn/G?= =?iso-8859-1?Q?J8z8htfDqde8V1XorS0HgHgWAoPsVm/IH2es8roaflbrBGF1sC8rVG4pAP?= =?iso-8859-1?Q?OvHN4kmveq2CKCS9Jf90K3nTO/HTcGmGaj8L2ukkjL03hCnvWtxUpkQaQO?= =?iso-8859-1?Q?sNxpLWrrdsr89PRt+xRSgUm1O+OeG6sGZcc1wm70k0PkcNi19ZEEQtXcIz?= =?iso-8859-1?Q?ZisogdPLRCof0SB8rz6jN1fC82fIWWvvqrRHQXtd0FPu8Fze3alquyd+A7?= =?iso-8859-1?Q?PQrmMToRZsdBVvh67J1Xt3Ie63QkftO1hf4Qi276dPMsVzu38VXZrS3o8a?= =?iso-8859-1?Q?Ic6m4WFOihTFUeYjmBBWH1BoJs/LLnua3ECtRvU5eNnmBM17UGsXlm9HKu?= =?iso-8859-1?Q?yNG6cOaY8EH8vmEulMyGcO/Qxclg7yUUU6tWSYXkcmz3+oympNp5p2s1rk?= =?iso-8859-1?Q?ziXbhYAqxg/ngtGWEyHEE3bdFus9pqFICacUfAGO78iGiAQv5T/XC1RGvK?= =?iso-8859-1?Q?XbG4+rY3xgvBk1RaLaaU/PgP0H/LbHGrDvsaItGa3L1lRMYPpm1sf5rNqv?= =?iso-8859-1?Q?M77mbRQleZb9Wu2QAgucaYwH0CBYk3UW6kuLkBa3g1rDi4pECzLX3Y9qt2?= =?iso-8859-1?Q?NMbxjj++JaKZQ76N2Tpf0eh4RhsVd9oHcU7vd9SzsyNsjesBbaEQFvLZUt?= =?iso-8859-1?Q?x8T6XN2KuFvUArorg=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 34e3ab03-b2a2-4ad3-32ec-08d8addf40d8 X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Dec 2020 22:56:04.8978 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: k1EO04jqO7v2un1VPYlmL5PM3ADCpX5zljL+juH+YjZksrY8Syz8RoJ5Nq7MkoEW3v1tUhA3lMPEILv3WZLEHQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB2583 X-Rspamd-Queue-Id: 4D6Nmt1HfLz4VV3 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Dec 2020 22:56:14 -0000 Just fyi, I have put a patch up on phabricator as D27875 that seems=0A= to fix the problem for all NFS client mounts except NFSv4.0.=0A= NFSv4.0 will require an additional fix so that the "seqid" is=0A= properly maintained during redos of the Open caused by=0A= the ERELOOKUP redo.=0A= =0A= If anyone is running a recent head kernel on a system that=0A= NFS exports UFS file systems, please test this patch.=0A= =0A= Peter, can you test this?=0A= =0A= If acquiring the patch from phabricator is awkward,=0A= just email and I will send you a copy of the patch.=0A= =0A= Thanks, rick=0A= ps: If possible, I'd like to commit this patch in a=0A= couple of days, given the FreeBSD release schedule.=0A= =0A= ________________________________________=0A= From: owner-freebsd-current@freebsd.org = on behalf of Konstantin Belousov =0A= Sent: Thursday, December 31, 2020 6:40 AM=0A= To: Rick Macklem=0A= Cc: freebsd-current@freebsd.org; Alan Somers; Kirk McKusick; Mark Johnston= =0A= Subject: Re: r367672 broke the NFS server=0A= =0A= CAUTION: This email originated from outside of the University of Guelph. Do= not click links or open attachments unless you recognize the sender and kn= ow the content is safe. If in doubt, forward suspicious emails to IThelp@uo= guelph.ca=0A= =0A= =0A= On Thu, Dec 31, 2020 at 05:16:27AM +0000, Rick Macklem wrote:=0A= > Rick Macklem wrote:=0A= > >Kostik wrote:=0A= > > >=0A= > > >Idea of the change is to restart the syscall at top level. So for NFS= =0A= > > >server the right approach is to not send a response and also to not=0A= > > >free the request mbuf chain, but to restart processing.=0A= > > Yes. I took a look and I think restarting the operation by rolling the= =0A= > > working position in the mbuf lists back and redoing the operation=0A= > > is feasible and easier than fixing the individual operations.=0A= > >=0A= > > For NFSv4, you cannot redo the entire compound, since non-idempotent=0A= > > operations like exclusive open may have already been completed.=0A= > > However, rolling back to the beginning of the operation should be=0A= > > doable.=0A= > Turned out to be quite easy. I'll stick a patch up on phabricator=0A= > tomorrow, after I do a little more testing.=0A= > NFSv4.0 is still broken, because it screws up the seqid, but I can=0A= > fix that separately.=0A= >=0A= > I do see the code looping about 2-3 times before it gets a successful=0A= > ufs_create(). Does that sound reasonable?=0A= In the simple case, it could be described as is: ERELOOKUP is returned=0A= if the parent directory cannot be locked sleep-less, and we have to drop=0A= the lock for opened vnode to sleep on it. More elaborate (but still=0A= not precise) description is that parent directory might also need to=0A= be synced, in which case its parent might need to be locked, and so on=0A= recursively.=0A= =0A= Slightly reformulating, I expect that ERELOOKUPs come out in case several= =0A= threads create files in the same directory.=0A= =0A= > Here's some debug printfs for the test run of 4 concurrent compiles.=0A= > (proc=3D8 is create and proc=3D12 is remove. Each line is a ERELOOKUP=0A= > retry. This is for the 4 threads, but I had the thread tid in another pr= intf=0A= > and it showed 2-3 attempts for the same thread. They should be serialize= d=0A= > by the exclusive lock on the directory vnode.)=0A= I cannot make any conclusion from the output and its description.=0A= Are there opens that do not result in ERELOOKUP, i.e. does the op=0A= eventually succeed ? What is the ratio of ERELOOKUP vs. success ?=0A= =0A= Also note that any VOP that modify the volume' metadata might result=0A= in ERELOOKUP.=0A= =0A= > tryag3 stat=3D0 proc=3D8=0A= > tryag3 stat=3D0 proc=3D8=0A= _______________________________________________=0A= freebsd-current@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-current=0A= To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"= =0A= =0A=