From owner-freebsd-current@freebsd.org Mon Jun 18 21:42:20 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C87410241E7 for ; Mon, 18 Jun 2018 21:42:20 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670086.outbound.protection.outlook.com [40.107.67.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT TLS CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8E7F17511F; Mon, 18 Jun 2018 21:42:19 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB0953.CANPRD01.PROD.OUTLOOK.COM (52.132.44.24) by YTOPR0101MB1532.CANPRD01.PROD.OUTLOOK.COM (52.132.49.148) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.863.19; Mon, 18 Jun 2018 21:42:18 +0000 Received: from YTOPR0101MB0953.CANPRD01.PROD.OUTLOOK.COM ([fe80::d0eb:3783:7c99:2802]) by YTOPR0101MB0953.CANPRD01.PROD.OUTLOOK.COM ([fe80::d0eb:3783:7c99:2802%3]) with mapi id 15.20.0863.016; Mon, 18 Jun 2018 21:42:18 +0000 From: Rick Macklem To: Steve Wills , "freebsd-current@freebsd.org" CC: "andreas.nagy@frequentis.com" Subject: Re: ESXi NFSv4.1 client id is nasty Thread-Topic: ESXi NFSv4.1 client id is nasty Thread-Index: AQHUBjX1G4uSBXqIhUeci7+GbxFK06RmiCEAgAAFxog= Date: Mon, 18 Jun 2018 21:42:18 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB1532; 7:3onYG45slYVh2Ja4EVgymnqm2bnR7cIF3sFNtaE8polOpiM4x2r8/XoZ3+NCUPK2IKUMi/SHkneQGCDsZ2vxA6D5gIwbjp9NwA06a0DExAL2dKDNbPwyscCkmPPH2IWqXXRr0eaL11PSnwuSpXjM2SEiUsheFQVPXaurs2ENk7ozCOpQDS7/zTADwfn18JDpr+AXNNKzmzdJ/mSMbQf+8MT83rhIITmG3vTMSkJyQR4cs8MJF76YDsj9C0PXxrf2 x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: da9cc615-6930-435d-6211-08d5d5645d59 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(8989080)(4534165)(4627221)(201703031133081)(201702281549075)(8990040)(5600026)(711020)(2017052603328)(7153060)(7193020); SRVR:YTOPR0101MB1532; x-ms-traffictypediagnostic: YTOPR0101MB1532: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(75325880899374); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(5005006)(8121501046)(93006095)(93001095)(3231254)(944501410)(52105095)(10201501046)(3002001)(149027)(150027)(6041310)(20161123564045)(20161123558120)(20161123560045)(20161123562045)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011)(7699016); SRVR:YTOPR0101MB1532; BCL:0; PCL:0; RULEID:; SRVR:YTOPR0101MB1532; x-forefront-prvs: 0707248B64 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(376002)(396003)(39860400002)(346002)(366004)(39380400002)(51874003)(199004)(189003)(76176011)(102836004)(2906002)(476003)(99286004)(316002)(786003)(6506007)(110136005)(59450400001)(53546011)(5250100002)(97736004)(478600001)(8936002)(6306002)(966005)(2501003)(9686003)(14454004)(186003)(486006)(11346002)(26005)(446003)(7696005)(55016002)(86362001)(6436002)(81156014)(81166006)(53936002)(6246003)(74316002)(74482002)(5660300001)(305945005)(3660700001)(25786009)(4326008)(105586002)(229853002)(68736007)(33656002)(2900100001)(8676002)(3280700002)(106356001)(12363001); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB1532; H:YTOPR0101MB0953.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: ctcdNfvrHCVKYW1KV1L9hu0KVE1+O8PXQYnTDvg0gzxFca9ZOtFrdFfJlGepRBz9x4ZsoNPMFCfZPbvAzxl+sCC8DE9dQGjmsShe1HeO+IXzC7/8FUJ4k+m8tVvyc+cI422wZ7tKoIlFX0Zj4nKlupFAGiNKUMzSXXenvRyQbqnjpGxtqS3F644UDBzLohu3cDnpvqb08ypjmdS1SwyGJlFf7vEaSHi7jrnjrLNQFIxZxdSUHRRPjZIvqU/hpro6S96Hg90AReJeSxQ2Qv0YZEcFD7zAMgv9TMDkXWzAZXSx/KYNe3UnLxNxkMiqdkfA+04cJqqiV2iRZFWOTS4eig== spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: da9cc615-6930-435d-6211-08d5d5645d59 X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Jun 2018 21:42:18.1909 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB1532 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jun 2018 21:42:20 -0000 Steve Wills wrote: >Would it be possible or reasonable to use the client ID to log a message >telling the admin to enable a sysctl to enable the hacks? Yes. However, this client implementation id is only seen by the server when the client makes a mount attempt. I suppose it could log the message and fail the mount, if the "hack" sysctl= isn't set? rick [stuff snipped] ________________________________________ From: Steve Wills Sent: Monday, June 18, 2018 5:21:10 PM To: Rick Macklem; freebsd-current@freebsd.org Cc: andreas.nagy@frequentis.com Subject: Re: ESXi NFSv4.1 client id is nasty Would it be possible or reasonable to use the client ID to log a message telling the admin to enable a sysctl to enable the hacks? Steve On 06/17/18 08:35, Rick Macklem wrote: > Hi, > > Andreas Nagy has been doing a lot of testing of the NFSv4.1 client in ESX= i 6.5u1 > (VMware) against the FreeBSD server. I have given him a bunch of hackish = patches > to try and some of them do help. However not all issues are resolved. > The problem is that these hacks pretty obviously violate the NFSv4.1 RFC = (5661). > (Details on these come later, for those interested in such things.) > > I can think of three ways to deal with this: > 1 - Just leave the server as is and point people to the issues that shoul= d be addressed > in the ESXi client. > 2 - Put the hacks in, but only enable them based on a sysctl not enabled = by default. > (The main problem with this is when the server also has non-ESXi mo= unts.) > 3 - Enable the hacks for ESXi client mounts only, using the implementatio= n ID > it presents at mount time in its ExchangeID arguments. > - This is my preferred solution, but the RFC says: > An example use for implementation identifiers would be diagnostic > software that extracts this information in an attempt to identify > interoperability problems, performance workload behaviors, or general > usage statistics. Since the intent of having access to this > information is for planning or general diagnosis only, the client and > server MUST NOT interpret this implementation identity information in > a way that affects interoperational behavior of the implementation. > The reason is that if clients and servers did such a thing, they > might use fewer capabilities of the protocol than the peer can > support, or the client and server might refuse to interoperate. > > Note the "MUST NOT" w.r.t. doing this. Of course, I could argue that, sin= ce the > hacks violate the RFC, then why not enable them in a way that violates th= e RFC. > > Anyhow, I would like to hear from others w.r.t. how they think this shoul= d be handled? > > Here's details on the breakage and workarounds for those interested, from= looking > at packet traces in wireshark: > Fairly benign ones: > - The client does a ReclaimComplete with one_fs =3D=3D false and then doe= s a > ReclaimComplete with one_fs =3D=3D true. The server returns > NFS4ERR_COMPLETE_ALREADY for the second one, which the ESXi client > doesn't like. > Woraround: Don't return an error for the one_fs =3D=3D true case and j= ust assume > that same as "one_fs =3D=3D false". > There is also a case where the client only does the > ReclaimComplete with one_fs =3D=3D true. Since FreeBSD exports a hiera= rchy of > file systems, this doesn't indicate to the server that all reclaims ar= e done. > (Other extant clients never do the "one_fs =3D=3D true" variant of > ReclaimComplete.) > This case of just doing the "one_fs =3D=3D true" variant is actually a= limitation > of the server which I don't know how to fix. However the same workarou= nd > as listed about gets around it. > > - The client puts random garbage in the delegate_type argument for > Open/ClaimPrevious. > Workaround: Since the client sets OPEN4_SHARE_ACCESS_WANT_NO_DELEG, it= doesn't > want a delegation, so assume OPEN_DELEGATE_NONE or OPEN_DELEGATE_N= ONE_EXT > instead of garbage. (Not sure which of the two values makes it hap= pier.) > > Serious ones: > - The client does a OpenDowngrade with arguments set to OPEN_SHARE_ACCESS= _BOTH > and OPEN_SHARE_DENY_BOTH. > Since OpenDowngrade is supposed to decrease share_access and share_den= y, > the server returns NFS4ERR_INVAL. OpenDowngrade is not supposed to eve= r > conflict with another Open. (A conflict happens when another Open has > set an OPEN_SHARE_DENY that denies the result of the OpenDowngrade.) > with NFS4ERR_SHARE_DENIED. > I believe this one is done by the client for something it calls a > "device lock" and really doesn't like this failing. > Workaround: All I can think of is ignore the check for new bits not be= ing set > and reply NFS_OK, when no conflicting Open exists. > When there is a conflicting Open, returning NFS4ERR_INVAL seems to= be the > only option, since NFS4ERR_SHARE_DENIED isn't listed for OpenDowng= rade. > > - When a server reboots, client does not serialize ExchangeID/CreateSessi= on. > When the server reboots, a client needs to do a serialized set of RPCs > with ExchangeID followed by CreateSession to confirm it. The reply to > ExchangeID has a sequence number (csr_sequence) in it and the > CreateSession needs to have the same value in its csa_sequence argumen= t > to confirm the clientid issued by the ExchangeID. > The client sends many ExchangeIDs and CreateSessions, so they end up f= ailing > many times due to the sequence number not matching the last ExchangeID= . > (This might only happen in the trunked case.) > Workaround: Nothing that I can think of. > > - ExchangeID sometimes sends eia_clientowner.co_verifier argument as all = zeros. > Sometimes the client bogusly fills in the eia_clientowner.co_verifier > argument to ExchangeID with all 0s instead of the correct value. > This indicates to the server that the client has rebooted (it has not) > and results in the server discarding any state for the client and > re-initializing the clientid. > Workaround: The server can ignore the verifier changing and make the r= ecovery > work better. This clearly violates RFC5661 and can only be done fo= r > ESXi clients, since ignoring this breaks a Linux client hard reboo= t. > > - The client doesn't seem to handle NFS4ERR_GRACE errors correctly. > These occur when any non-reclaim operations are done during the grace > period after a server boot. > (A client needs to delay a while and then retry the operation, repeati= ng > for as long as NFS4ERR_GRACE is received from the server. This client > does not do this.) > Workaround: Nothing that I can think of. > > Thanks in advance for any comments, rick > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org= " >