From owner-freebsd-stable@freebsd.org Mon May 3 00:27:53 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 46DBB5FF86D for ; Mon, 3 May 2021 00:27:53 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670082.outbound.protection.outlook.com [40.107.67.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FYP2J16MFz3kst; Mon, 3 May 2021 00:27:51 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IJRnHnkv8YJkLJk7SdKLJ0+8QSp8WjhhIvkproPqbAtCy4Xk6vLzIzCP5mQWRiIOz8VTm6vQ8bWPyv5dlZ/35Ac72/fyRjzksr051w9Pf5/+coCSjkEzSdj4BwZbHgHUfgPYzjfduq0kvkfL1FWO5G8J1CiibFOTuVqEiGAblBbJ6iPSbRu81b7B9sRGcHlF+vlNTkk67gah5AEZhA3r8LNCQ4Q1w1X2w2MG9rlc9S66HJjDgmzBo4fFCFo5jWHMvvd1JbdulKmMzfx/moLfFuUqYSCiT3D2+GzyjBm65oFUvZNWQLea3H/uJqoFU/aDJcNs77XUpOAbBh0QT5quZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=d2cVHY1xu7/c2xCEjWRj6HtaOsAFhVE5HpeAE+rzT9g=; b=UnMW5X+beOWOANeBWPpuPKup4HcYqfkTlhxK0HxwZfpj+HM8DY4suXIMxP3Xl0ctYu/eEXG+ymNfa/Xt9PlCsbtivqYXZi0bE3Kf/SCpoxrkpDGNFJ4Y2LWgOOiJzGTCa0wTRVrM7Yzg2l66WkyPx3rx5R2UdnAgjfpqUAwDTwCaqGwnRqe/dNyrwzHcda9uqYOJmiMRLiZeOCBaQJ5i2taw2vvytRAnEXFrNZV/4P8meQoP/WjCE28/VNTN0m7zOKjgevVVbH1RYrCIEaQTmbHfilmqYWorQPZWyf+8sL1MbkJb0gD/78y7O+HkqqmKw9/XBcffDF1OtM/EE3g9IA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=d2cVHY1xu7/c2xCEjWRj6HtaOsAFhVE5HpeAE+rzT9g=; b=cuMEMja96IFz8E3G4tKSDLCwwTl7UW4l9RJUrBAbxuXOlzpYYIoF9lxpoln7s85FOIsFC3ec7Ocyxadvaes0ACnSF/R3IfbG0ANfcTP4s9zerQj0vSxKl91+3ve9eJP1LI4Y6t2CZgv7CKguv4f4hHr3/0IZL8CUItuDmRsuPVGCukbfo5LlPrYAQAzNcIh6dHpWcD9yRcaml2i1oGoG+F9T0a2ripvuwFojCopE7jfLn13hWs5zaWXf72p82iKeElFk9ZEbw1XpwTFz/GR9IdHgjQewZnB3omzhfYLgL51k8GI2BPLfjty2t1owGKq6RTs8FoCLeSvNt4i+K5IVtg== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQXPR0101MB1478.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:17::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4065.25; Mon, 3 May 2021 00:27:49 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::1c05:585a:132a:f08e]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::1c05:585a:132a:f08e%4]) with mapi id 15.20.4087.043; Mon, 3 May 2021 00:27:42 +0000 From: Rick Macklem To: freebsd-stable CC: Peter Eriksson , Ryan Moeller , Garrett Wollman , Alan Somers , Juraj Lutter Subject: Re: wanna solve the Linux NFSv4 client puzzle? Thread-Topic: wanna solve the Linux NFSv4 client puzzle? Thread-Index: AQHXPEHIsjwmd971wkuLdHVmW5Ho6KrQ2a+E Date: Mon, 3 May 2021 00:27:42 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 811809be-0fa2-4314-1f48-08d90dca4426 x-ms-traffictypediagnostic: YQXPR0101MB1478: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:538; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: eVsvDg0nt631mFI6CKUV4ZoWm9QDWoOfwayJ3ZfH2LbIiDRXfQVGKnlLtG8okPMYNMNpxnqlofFi9E/N0j+dXuUdJAX9aJ7Ov68pgmGSksNZrRFPkjHXuWZCdkV7R2WNDwaHziwJkSwU2j7l02iUXugNMnOljnFD5IW2Md++51m21CGgxkZunRWIVWzMFOdaQwZCNAMbZlC1Fq0mpy8fzoTTNPcGF0uM5MC+/Uk5zvfnIrkxVsXJUw0JGC5qkP4aufQjW7q2LX+tyi5arapny0VCH7W9gbT5aZZT13sdvARLtlGVxTVaPfdI/lxf0+iZqU04+j/5Lt/qdPitdBFup/wiYzOXc2C2iunAJeWBQMRrPe93japTu13RXjfACRmiJw6sCsZ9MhyAnu9FEn8EGI0m8oZJ5L6JnMxNqyEDay2pcJh77j4zULGwlluYBOchhyqfrbIkmiUYeD5o7LidEx8m8upXW2wD/8WUDCMBMXaJEYepSBhRyaR3C0CfDCwys1T4RHYMsCAD4Xa7mBlvJBWSxrZYqT4b3Ir4hUR7+/4/qazYacK7AFLY3Coo/frE/RzNekxAfZDOsRRTin2H7UiMS3Imu1xfC8/LvGs8hp57x8rgzObTO5iihWu67+JbrqstlgxTPURksvDnDxQESgoRjpJa9K5ksjgmDp3GusdkxtYN/ZSQYBGbtsiMtMfy x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(366004)(346002)(396003)(376002)(136003)(39850400004)(2906002)(7696005)(9686003)(66476007)(8676002)(5660300002)(8936002)(6506007)(186003)(316002)(54906003)(786003)(478600001)(55016002)(4326008)(52536014)(86362001)(6916009)(83380400001)(91956017)(66946007)(38100700002)(966005)(122000001)(66446008)(76116006)(33656002)(71200400001)(66556008)(64756008); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?kSDIGwOCSEB5bgLs0yHGmuTizjgIE00NL+LTjD8G51pQqz3hgH/VG3RMx5?= =?iso-8859-1?Q?qSsGWl6u64cOEsj6J/iSFzHx3D3bpM085CF9ArwNMdD1cfjnNnGUE69tOc?= =?iso-8859-1?Q?ftTKZ0MU/AEvi3fyR/M0sLAjtnS+W+UK2m5zud35B+3y3Tj6U0bx61wgXr?= =?iso-8859-1?Q?0GY+yX1gVG2VDwi4MThd6hSuwnsBbtXDrwGac4MOYNnQjrtaw8yfHc3bAS?= =?iso-8859-1?Q?lnBmUNNEx6SCzbQawE8gZW9EAvy1deW7MsMimTEvkcsEHMoip02hz25jNg?= =?iso-8859-1?Q?cvyqelbzrmI1oi27rONu3wHTV1l9F4JEMBZumh0fUo5HHtyzbVXNGobjhU?= =?iso-8859-1?Q?2AIxRFsfT4Eix6dpFlvriGU1yIl5jW139Ru9k8HldHG02mEQnzX1H1RBU6?= =?iso-8859-1?Q?qDCkqz8BK3B9z2m+PGyDFrQ/d18hpCc/I+tR67hEO4cQ5rIqp/L46mHMWo?= =?iso-8859-1?Q?Mla3vYKM6CWcxqRFq9aF68tZkb/dn5Fajc0WsXTiE9uvNIkwaRdQ3RTejF?= =?iso-8859-1?Q?+W1i12dJa4CS451s/SXpvzyplvJdwjGMsp+xpb1AsVsN++ebYUaCX48QbP?= =?iso-8859-1?Q?0rJziWbyercLzJHPBGlEa8kdSgvwvsBg0nXKPkq7++G3PGwXBsZwBwECyu?= =?iso-8859-1?Q?SoCTmg1jCne1ywhXvc/zpugpRjScv2MHocyPUh/EEDf3sZfJtg+zEZCp2L?= =?iso-8859-1?Q?62OEGayTCuP7+CPerR5VMTvKMgc4i7QsW1+tR93Yc1r4GdMxdUm6wYr3WY?= =?iso-8859-1?Q?UXO3Fpny1QnGPvJyiYWdElaEPu28Aas6PQABJCKrnGIpG3kCvDOA01U/7r?= =?iso-8859-1?Q?3SzH+4AAfIRanLQX65J+ps8DnAafyeB/920/jVXLmig+0bwj2iRYdBxz0B?= =?iso-8859-1?Q?ROMQVW5w7J2qlje7qfI9UPXPJ0EHyRMuFEpIt1aicwUjNN8jk2Ps9RKGOf?= =?iso-8859-1?Q?7SGIKjko47THJ5jITFvmfrBKuwXFS+DqXLrkRByYuB8hxZMWKAv4TeqHoc?= =?iso-8859-1?Q?yP0+nZlk7a4doE6EPKtvIvKIqb5uPfaP8OxGMgq2/DjHXHL82UCQQ+oRm3?= =?iso-8859-1?Q?bz4TDAEDhFpE8O2MnQOShvrNxEQ/zFw+ghaXySj6z8bO1ciPPwuoNEd0pd?= =?iso-8859-1?Q?0K88VQsfgphFp1yEDmlc28+7Go49HqBCi9NChyfHwMgEpuryYQUvE5r8mV?= =?iso-8859-1?Q?XHddYb4r3IKwoKJqtAgQ2KEMf1vQhV6u6RcyiLInq0khcJu0vi1JmsEOfv?= =?iso-8859-1?Q?XENngiP5NEt+d1vFZp3ebcF9/GoYxFpPnQtIpDuzrnrldcXR6+4smxnuEV?= =?iso-8859-1?Q?rWkU6wKqETm83UVKMb/JkLIXB6suxP0Ox+hzH0mu+3dQeNmEKmvTh+YNrG?= =?iso-8859-1?Q?c5t9CScMRne7E+ddZIE5L9n3sJJ41MFk+HBZOUnleq/2ZAtplg+ZWoljho?= =?iso-8859-1?Q?XAb+SXiQmW6J8oA9?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 811809be-0fa2-4314-1f48-08d90dca4426 X-MS-Exchange-CrossTenant-originalarrivaltime: 03 May 2021 00:27:42.6260 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: uPZ6Hzq4nCgDXdx4615BHBapgwtJm4n7kKzrjsbwcXSSIznpMpfqvc3FpjXHPBJ6WaB9vnFYyymzQQ5Mr48tvw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR0101MB1478 X-Rspamd-Queue-Id: 4FYP2J16MFz3kst X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=cuMEMja9; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.82 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-3.00 / 15.00]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; RCPT_COUNT_FIVE(0.00)[6]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; FROM_EQ_ENVFROM(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.67.82:from]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(1.00)[0.999]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[40.107.67.82:from:127.0.2.255]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[40.107.67.82:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.82:from]; MAILMAN_DEST(0.00)[freebsd-stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 May 2021 00:27:53 -0000 Rick Macklem wrote:=0A= >Hi,=0A= >=0A= >I posted recently that enabling delegations should be avoided at this time= ,=0A= >especially if your FreeBSD NFS server has Linux client mounts...=0A= >=0A= >I thought some of you might be curious why, and I thought it would be=0A= >more fun if you look for yourselves.=0A= >To play the game, you need to download a packet capture:=0A= >fetch https://people.freebsd.org/~rmacklem/twoclientdeleg.pcap=0A= >and then load it into wireshark.=0A= >=0A= >192.168.1.5 - FreeBSD server with all recent patches=0A= >192.168.1.6 - FedoraCore 30 (Linux 5.2 kernel) client=0A= >192.168.1.13 - FreeBSD client=0A= >=0A= >A few hints buried in RFC5661:=0A= >- A fore channel is used for normal client->server RPCs and a back channel= =0A= > is used for server->client callback RPCs.=0A= >- After a new TCP is created, neither the fore nor back channels=0A= > are bound to the connection.=0A= >- Bindings channel(s) to a connection is done by BindConnectionToSession.= =0A= > but an implicit binding for the fore channel is created when the first R= PC=0A= > request with a Sequence operation in it is sent on the new TCP connectio= n.=0A= >- A server->client callback cannot be done until the back channel is bound= =0A= > via BindConnectionToServer.=0A= >=0A= >Ok, so we are ready...=0A= >- Look at packet #s 3518->3605.=0A= > - What is going on here?=0A= Ok, so here's my solution...=0A= packet #3518, 3520 and 3521 are delegation recalls (CB_RECALL)=0A= for 3 different delegations on three different session slots.=0A= time: 137.5=0A= =0A= Expected response from the Linux client--> 3 replies to the CB_RECALLs.=0A= What does it actually do?=0A= --> Creates a new TCP connection using same port#. You can see it send=0A= a FIN (packet# 3523) and a SYN (packet# 3527).=0A= This means that the client is no longer obliged to reply to the CB_RE= CALLs=0A= and the FreeBSD server will probably need to retry them.=0A= --> It also means that no back channel is bound to the session, so th= e=0A= server cannot do callbacks (ie. cannot retry the CB_RECALLs ye= t).=0A= =0A= packet# 3530 is a Setattr RPC, which has a Sequence operation in it.=0A= --> This means the fore channel is implicitly bound to the new TCP=0A= connection, but no back channel, so the server cannot retry the CB_RE= CALLs.=0A= =0A= You will notice a bunch of Setattr RPCs getting NFS4ERR_DELAY replies.=0A= This tells the Linux client to "try again later".=0A= --> It happens because the FreeBSD server cannot perform the Setattr=0A= until the client returns a delegation.=0A= --> That requires a CB_RECALL.=0A= =0A= packet# 3582 is a Setattr RPC reply. If you look in the Sequence operation= =0A= reply, you will see the flag SEQ4_STATUS_CB_PATH_DOWN is set.=0A= --> This is the FreeBSD server telling the Linux client that the callback p= ath=0A= is down (the back channel is not bound to the new TCP connection).= =0A= Time: 137.6 (took about 0.1sec for the server to notice that the callback= =0A= path/back channel is not working).=0A= =0A= packet# 3604 Linux client does a BindConnectionToSession to bind the=0A= back channel.=0A= --> This is not permitted by RFC5661, since it is required to be done on=0A= the new TCP connection before the implicit binding of the fore=0A= channel only, already done by packet# 3530.=0A= packet# 3605 FreeBSD server violates RFC5661 and allows the binding=0A= to be done, so that CB_RECALLs can again be done.=0A= Time: 152.7=0A= =0A= - How long does this take?=0A= 152.7 - 137.5 =3D 15.2seconds=0A= =0A= >--> One more hint. Starting with #3605, things are working again.=0A= --> Things start working again because the FreeBSD server=0A= cheats and allows the BindConnectionToSession to be done.=0A= RFC5661 specifies a reply of NFS4ERR_INVAL for this.=0A= =0A= >There are actually 3 other examples of this in the pack capture.=0A= Every time multiple concurrent callbacks are attempted, the Linux=0A= client "bails out" by creating a new TCP connection.=0A= --> This is said to be fixed in Linux 5.3, but I haven't tested a newer=0A= kernel than 5.2 yet.=0A= =0A= >Btw, one of the weirdnesses is said to be fixed in Linux 5.3 and the other= =0A= >in Linux 5.7, although I have not yet upgraded my kernel and tested this.= =0A= The "do BindConnectionToSession after an implicit binding" is said to be fi= xed=0A= in Linux 5.7, however the fix is not exactly what I would have expected.=0A= --> I would have expected a BindConnectionToSession to be done right=0A= away when a new TCP connection is created.=0A= --> Linux 5.7 and newer is said to still wait (15sec?) to do the=0A= BindConnectionToSession, but fixes the bug by creating yet=0A= another new TCP connection just before doing the=0A= BindConnectionToSession RPC.=0A= --> A SEQ4_STATUS_CB_PATH_DOWN flag set in a Sequence operation=0A= reply is what triggers the BindConnectionToSession and that is = still=0A= required for 5.7 or newer, but I'll need to test to see how lon= g it takes=0A= for newer kernels?=0A= =0A= The old "cheat", which is still in the released server code (recently remov= ed=0A= by a patch in main, stable/12 and stable/13) implicitly bound both the fore= =0A= and back channels. Look for this comment in sys/fs/nfsserver/nfs_nfsdstate.= c=0A= in unpatched code...=0A= /*=0A= * If this session handles the backchannel, save the nd_xprt for this=0A= * RPC, since this is the one being used.=0A= * RFC-5661 specifies that the fore channel will be implicitly=0A= * bound by a Sequence operation. However, since some NFSv4.1 clients=0A= * erroneously assumed that the back channel would be implicitly=0A= * bound as well, do the implicit binding unless a=0A= * BindConnectiontoSession has already been done on the session.=0A= */=0A= =0A= --> This worked fine and avoided most of the above craziness, but...=0A= (A) It violated RFC5661.=0A= and=0A= (B) It broke the Linux client badly when the "nconnects" mount=0A= option (added fairly recently) was used.=0A= --> So I felt I had to get rid of it. (The non-conformance with=0A= RFC5661 was reported by redhat.)=0A= =0A= Bottom line...unless all your Linux clients are kernel version 5.3 or newer= ,=0A= avoid enabling delegations in the FreeBSD NFSv4.1/4.2 server.=0A= --> Even with a completely patched server, you will still get 15second paus= es=0A= every time the server attempts multiple concurrent callbacks.=0A= =0A= >Have fun with it, rick=0A= At least you can now see why I have "fun with it";-) rick=0A= =0A= _______________________________________________=0A= freebsd-stable@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"= =0A=