From owner-freebsd-current@freebsd.org Thu Dec 31 05:16:30 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 042254DCAB1 for ; Thu, 31 Dec 2020 05:16:30 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660046.outbound.protection.outlook.com [40.107.66.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D5xG55xKnz3m4N; Thu, 31 Dec 2020 05:16:29 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PjGz8iZdCrNlfIw+JpfX+/Z2IHBi9Zps8Ci7wgMoERQvbDzwvsR+XG6q//Rk5n3Azh1JxO8NXqhMN1vZGgu8UCKX1o730XLM4q0En2FOq8SVCdrott03R6C4SCF6wKFRTvI737SOYhsl57oiKECWjRtE42A0c3y61VkfGgbtDKu4dOW4qKy/rDGoU8aR+5EucnRHvYcx8Ju6BmTbcaRpYN++N/jv6haFrh87Tl54yIvw0mGxDzyIjujNno7yAsRArNgINPQxuLq7m6JzQnrTB1S8imh8yYaG5dW5IYAY/xvtox5AFOzcYLz95Sr0I6tXKNIpF5dwZ4vH13nQunyBFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IMAvQKotJvvvZX0BXqL4guhVp1arJASSuHvkPJIvPFY=; b=bdCgxMKc19fWRB3kTr4bFYopVwdgbCrvK9KcL3clNUPD+gawnnSid3tWGaakaDL1YCFr8TTPPZeth11QOsr3SQsy536S+q4cXDUyuFp1Du3JKz4c4DO64YqxyAV48SePD7lw6d4WQppsGxOhM8uIjO5d8ILAw5KKQI0+/lcWQmLCalD5FajQAsPrsEDSwZXSTc6txGgNPPzPGTSPPsS4K7Or0mP6ApwF4AwBdeGRnBmFYYGMgluVKLRUBYCxKAev2YzKZ8o5wj4E3en/ifBvTWg/hSk/o5HnuBM71Kr8c7Gw54JVY7sVGFko5QVVUICAxNUJLotwx5cGxh7Wmn7JGA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by QB1PR01MB2867.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:3d::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3721.20; Thu, 31 Dec 2020 05:16:27 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::3d86:c7f9:bc4c:40c0]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::3d86:c7f9:bc4c:40c0%6]) with mapi id 15.20.3721.020; Thu, 31 Dec 2020 05:16:27 +0000 From: Rick Macklem To: Konstantin Belousov CC: "freebsd-current@freebsd.org" , Alan Somers , Kirk McKusick , Mark Johnston Subject: Re: r367672 broke the NFS server Thread-Topic: r367672 broke the NFS server Thread-Index: AQHW3k3ynlL3lGhXRkC37zjbBa1PUKoPmNaAgAA7NVaAABFYAIAAwg6J Date: Thu, 31 Dec 2020 05:16:27 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 9455ca48-988e-497d-b925-08d8ad4b39e8 x-ms-traffictypediagnostic: QB1PR01MB2867: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: +iCteJfXi67+vexwT/hbfDJsrHqHfRDGe3y81MmPsvlrXLr+NFGj72nqONlZGfnFFILfG2rh7o6rfBHsToQDuLk1CGzG4QrwXX7CjfyvvN4FsJ42eEiMmSdsLUdGiSW9veLDIZN/Znk+A+vZjpVNafCHYMmz8QO1fqDJISNF3dJIV3rXj1dRa00F6NNiGq0ADY9jZT6WsIabShULhdgYw53AEOgq9pAI5FBivG/M2El4Xv/FfHXl7lOuyfF0oDENFOTXiY4FA9OnlPfDs79fjyCi9P1LIhzxyJD71SCW6lMjayOBjTZKxalJOtlmu6zw1jO50xRTt/udFawLFo7DcfHBX85yqSA9/x1L/qdpc5vzkSoSIXmk74ypIQlBjxd4zWoz7BsmMImyVF79wexbrK39A2eyhaEtGC0FOC5ZXzM+Uhews4baQfx+GM8IW5/h5XeV1GYScL49FoCYLAg2YQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(366004)(346002)(396003)(39860400002)(136003)(376002)(8936002)(26005)(2906002)(6506007)(8676002)(52536014)(76116006)(316002)(55016002)(91956017)(786003)(71200400001)(4326008)(186003)(54906003)(9686003)(66476007)(66446008)(66556008)(64756008)(966005)(86362001)(478600001)(7696005)(5660300002)(6916009)(33656002)(66946007)(83380400001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?5TOXO8Uo30AZ9OggBjAofgys3aA2jK3MGAfDzciQ5pGZHoAZIj/PybSBJe?= =?iso-8859-1?Q?ENrVW78/r553atEUpmvuGkVA6w5gAbnn07r54Qz8nKFVsDsYYxKaxdw3su?= =?iso-8859-1?Q?SU98K2n6dFj9RHikHUrhJc23eH1K/KIVsC4/zvhGiYWQDOkaOj5fZufoV9?= =?iso-8859-1?Q?dWYhGg6XVGou226kZGtrxE27y7/C+ZfTFqvpKIqk5S2rkUkFhNY54oMp1/?= =?iso-8859-1?Q?lHX9wE8yc2/aPK8gGUL7lnySm9drfURc7aNkg+ITs/Ny7DoCL+rBHHVxqC?= =?iso-8859-1?Q?aFldLHuZSQxjjIDoyNGUALcRbNyO6nQPNX1MpgC7SQ3Y6zN/QMIPK334fu?= =?iso-8859-1?Q?yfRpk2POTiQo6SeI1DGwxkQVvoC1WwkEuIbZOJGu/KTC2wjLGq4eSclZSe?= =?iso-8859-1?Q?AMuFjff/uylew5DXyuDAeGz0EhvQfNGKlwmFnU8SGWFOluvYeVrOCwK1Vs?= =?iso-8859-1?Q?4OQAogifxy9NUL1b43LXoNmuP6HgVC+Go6hlVyBybY/OrU0siVVKypwGuy?= =?iso-8859-1?Q?z9dz6geVsqE4YX8cJnmKqYD0+pG00W+3QW2VBmjqm4KcKIravT1FpRNjOP?= =?iso-8859-1?Q?0LN8aMA1lHtncWWLZZq0Tz58SAb0e1lhjGmSXJHnsIJXR0sSlx/XoMxEXX?= =?iso-8859-1?Q?8l8pUYtmZg1ijtfraMFnBm4ZVeo8wAdZDBiXP3P64IRu5ASXxygZZbl7YZ?= =?iso-8859-1?Q?Kz63sAGVDckTTInDsZSxUdEsWaa8OyR1ofvvzFs0HSRCvPuZmg8f78KMg7?= =?iso-8859-1?Q?xzPgl+NAlustY/vIcKPqsU/ds0rKn/UK7xMtOG9ZL9UVPRD8+v5Xs3JAhl?= =?iso-8859-1?Q?wF62D6csipo2WOB2sbiNeBhGqh9k9WKu+6yYu96GOgUyYbzOQL1rDqdOx8?= =?iso-8859-1?Q?cQCfEDj3yOzgtB6NslT2iHdLNuYGk6aGbF9llHFtzwIVgGLLe7khyMaxSC?= =?iso-8859-1?Q?OCYqR8wPO7on/bJz6fG6PIYLEpK/wM+157wetE2cPufuH0oKY7Pq1H6wK0?= =?iso-8859-1?Q?L2/AbLCHiL+2WVF24=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 9455ca48-988e-497d-b925-08d8ad4b39e8 X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Dec 2020 05:16:27.7881 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Mc5aQM2y4EsQD5zKd+U14YGb19dpdQmIS9meIg0zv9htWhKE7OnfyAUb+uX1sED+Od9RV3KuSLPNb/RnZYrFEQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: QB1PR01MB2867 X-Rspamd-Queue-Id: 4D5xG55xKnz3m4N X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Dec 2020 05:16:30 -0000 Rick Macklem wrote:=0A= >Kostik wrote:=0A= > >=0A= > >Idea of the change is to restart the syscall at top level. So for NFS= =0A= > >server the right approach is to not send a response and also to not=0A= > >free the request mbuf chain, but to restart processing.=0A= > Yes. I took a look and I think restarting the operation by rolling the=0A= > working position in the mbuf lists back and redoing the operation=0A= > is feasible and easier than fixing the individual operations.=0A= >=0A= > For NFSv4, you cannot redo the entire compound, since non-idempotent=0A= > operations like exclusive open may have already been completed.=0A= > However, rolling back to the beginning of the operation should be=0A= > doable.=0A= Turned out to be quite easy. I'll stick a patch up on phabricator=0A= tomorrow, after I do a little more testing.=0A= NFSv4.0 is still broken, because it screws up the seqid, but I can=0A= fix that separately.=0A= =0A= I do see the code looping about 2-3 times before it gets a successful=0A= ufs_create(). Does that sound reasonable?=0A= Here's some debug printfs for the test run of 4 concurrent compiles.=0A= (proc=3D8 is create and proc=3D12 is remove. Each line is a ERELOOKUP=0A= retry. This is for the 4 threads, but I had the thread tid in another prin= tf=0A= and it showed 2-3 attempts for the same thread. They should be serialized= =0A= by the exclusive lock on the directory vnode.)=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D12=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D12=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D12=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D12=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D12=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D12=0A= tryag3 stat=3D0 proc=3D8=0A= tryag3 stat=3D0 proc=3D8=0A= =0A= Thanks for the suggestion, Kostik.=0A= =0A= rick=0A= =0A= > --> It will serve as a good test, in that it may expose bugs in the=0A= > RPC/operation code where failure (ERELOOKUP) doesn't clean=0A= > things up correctly.=0A= > --> In NFSv4, there is the open/lock state that cannot be updated= =0A= > for this error case. (The seqid stuff in NFSv4.0 Open can be = fun.=0A= > Its used to serialize the operations and the number must be= =0A= > incremented for some errors, but not for others. The 10026=0A= > error occurs when you don't get this right.)=0A= Note that ERELOOKUP error can only show up from the VOPs that modify the vo= lume.=0A= Otherwise we simply do not call into SU. In particular, I believe that ope= ns=0A= in the sense of NFS are safe.=0A= =0A= Regardless of it, there should be either a catch-all check for ERELOOKUP,= =0A= or assert that ERELOOKUP did not leaked, as it is done for syscalls=0A= =0A= >=0A= > I'll start working on this to-day, but I have no idea how long it might= =0A= > take?=0A= >=0A= > >I am sorry I forgot about NFS server when designing this fix, the only= =0A= > >mild excuse I can provide is that the change was quite complicated as is= .=0A= > >I will start looking at the fix.=0A= > No problem. Sometimes I'd like to forget about NFS too;-).=0A= >=0A= > For the rollback/redo the RPC/operation case, it's probably easier for me= =0A= > to do it. As above, I'll start on it, but...=0A= >=0A= > My main concern is how long it will take, given the FreeBSD13 release=0A= > starts soon.=0A= For sure I will help you if needed, and I believe that we could ask for=0A= testing from Peter.=0A= _______________________________________________=0A= freebsd-current@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-current=0A= To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"= =0A= =0A=