From owner-freebsd-stable@freebsd.org Thu Jan 9 03:24:12 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E2F5F225986 for ; Thu, 9 Jan 2020 03:24:12 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670077.outbound.protection.outlook.com [40.107.67.77]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47tWgH6jstz4WrJ for ; Thu, 9 Jan 2020 03:24:11 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TJK0gbQA1y2NeYLh0lYYAgBc7coaRY2LASgbGvFf/lwHPsN3H30gwAGHr0hMtIk2tAg0UsjSGLVVEncfXoF8X+D6DNhGhsvW/il0iT8kVj+Rf+5lcy/MA49BRP5BMbbsUfbiVLjLVCPycgvQKYO65mRDXSA+DanXRbKqz+j55eKx3ooaYOwDQgw8b4h6GswXboU8aACrJdWHclMlffLblx9pi29geqJ7NT0Mn+DoIZpyLD1HyyhDWmw4Jh17/YS4cunit7XhHsIsmHNHX62QbZj6oor8Cf4+L3JtJmlVJczeVrkQaPBzA8r6Ljt7eLCTQjeSx9DluwC+HGkpM/7Nlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qbXF4//RDq+mE40/zuJ4bckDege+77HD0SL/HMrij2M=; b=oaBdcz+9gaeuKHGG5C01A5JREtTT1uBKD+Hs1mYTaYGPGSudnH9ubJsp09vRjvRe8imhNeMUlG0/F5+w230mKBKY4Y7yoJXhkZzvc/4gJ0q/tBqz37mpKrXqKBqybDBmrNbSeJ4ld+/jnYIX7MyVlpYztTXFqwChto5h1sAvaBSwmty2vLy2afH4BwcQYATIXUZ6VFCnOWPvcjBXBbqOgcolXa25Y4wyz6wTAnSld95/FWqIeRrXAy7zbgHLv3R9HwySzLNGTER2dY96MyLTBshZ69MZ0CYE67/fYUxOqANDcFI0nRVZw/JSTgRaxHFpuG+xTRKdm5/b2hOTRYr+0g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM (52.132.69.153) by YQBPR0101MB1892.CANPRD01.PROD.OUTLOOK.COM (52.132.71.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2623.10; Thu, 9 Jan 2020 03:24:10 +0000 Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::7512:8580:8d82:6c94]) by YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::7512:8580:8d82:6c94%6]) with mapi id 15.20.2602.016; Thu, 9 Jan 2020 03:24:10 +0000 From: Rick Macklem To: Daniel Braniss CC: Richard P Mackerras , Adam McDougall , "freebsd-stable@freebsd.org" Subject: Re: nfs lockd errors after NetApp software upgrade. Thread-Topic: nfs lockd errors after NetApp software upgrade. Thread-Index: AQHVtawq+ga5QLcdVkqBDG/GW9zFg6e/+Am+gAARTACAAANHAIAAi7Y3gACf34CAAEVO6IAABk4AgADWGACAAO1eZYAA7uGAgACmPw2AANdsAIAAsCi6gAF3uACAAC25gIAAlUcYgBiCXYCAAKrkUA== Date: Thu, 9 Jan 2020 03:24:10 +0000 Message-ID: References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> <8A78F67B-C244-45CF-B9BF-D7062669B33B@cs.huji.ac.il> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 8a7e1f1b-9f52-45be-c803-08d794b3649b x-ms-traffictypediagnostic: YQBPR0101MB1892: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 02778BF158 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(136003)(366004)(396003)(376002)(346002)(54094003)(199004)(189003)(7696005)(54906003)(4326008)(478600001)(55016002)(966005)(9686003)(786003)(53546011)(6506007)(316002)(71200400001)(2906002)(6916009)(81156014)(33656002)(86362001)(186003)(52536014)(81166006)(8676002)(66476007)(66446008)(64756008)(5660300002)(66556008)(66946007)(8936002)(76116006)(66616009); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB1892; H:YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: DzmmeQkVn7d4Jj4rwibJ8nknDy32oFDlMjouFvRRrdkkcy/K2HxIRQlSIePTIUK8sTilmypeoqFmAKRJpuH6DYEDl9ORfEJr8u6W+udTPo9htdFQHD/xqdUnDSewvBuctQO5NWxZ8EUDl3ealH08UhE4Lx6CyU3/ZwVT0gOk9wlU4DPPV1T7+z5V0yXbaqTCgiEM8zSULO7sY7AlOZ05euyD0IN9TRfE5Lg0sIUBQCLIDYyfpI7fSwAdAoRPDN7MgooZJxO9jHlSn687a6p8VC0AEKnqD9TvmiURHKsO3GBaQzSQfZSRdWcJApFdv3Z5ku5XgioBDcxds6FvoDigpbTczGGOBLtzHxPhyVwRrryRyMXIxViOUdc2GrfJBlCjpFfQYEyI94tK0VVflHdjBotKfhMd7D9TNojx+VGpD46C4RaQ3HFQlRpgDoVjHVvn9SQkyzJGT/4L/c+icV6TgV7unACo/hBS9VHsPDQNjIKUiYukQYG2ZAzppsdq7IZSl0QxYUhgcdT3+hcqGQ8oqPRUn3s4D4KgJx6C7AVHB2BXoDzTswp22aII4IxLEmli x-ms-exchange-transport-forked: True Content-Type: multipart/mixed; boundary="_002_YQBPR0101MB1427FF31676F6C4C641CA933DD390YQBPR0101MB1427_" MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 8a7e1f1b-9f52-45be-c803-08d794b3649b X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Jan 2020 03:24:10.2912 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: DAiEle6Zhnpwlelj+vfgjxYduZGvtXwJDD9I9tooKcMqaIMy7Nyn9lD6n15gyyqd8inxIB52er9D2Hyb/4xNkA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1892 X-Rspamd-Queue-Id: 47tWgH6jstz4WrJ X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.77 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.68 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; HAS_ATTACHMENT(0.00)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; DMARC_NA(0.00)[uoguelph.ca]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[77.67.107.40.list.dnswl.org : 127.0.3.0]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[i=1]; IP_SCORE(-1.38)[ipnet: 40.64.0.0/10(-3.84), asn: 8075(-2.99), country: US(-0.05)]; FREEMAIL_CC(0.00)[gmail.com] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jan 2020 03:24:12 -0000 --_002_YQBPR0101MB1427FF31676F6C4C641CA933DD390YQBPR0101MB1427_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable The attached patch changes the xid to be a global for all "connections" for the krpc UDP client. You could try it if you'd like. It passed a trivial test, but I don't know = why there is that "misfeature" comment means, so I don't know if this breaks th= at. I can't think of why "xid" would have been per-connection (especially since= a connection is a questionable concept for UDP), except that this might have originated in a userland library and carried into the kernel during porting= . rick ________________________________________ From: Daniel Braniss Sent: Wednesday, January 8, 2020 12:08 PM To: Rick Macklem Cc: Richard P Mackerras; Adam McDougall; freebsd-stable@freebsd.org Subject: Re: nfs lockd errors after NetApp software upgrade. top posting NetAPP reply: =85 Here you can see transaction ID (0x5e15f77a) being used over port 886 and t= he NFS server successfully responds. 4480695 2020-01-08 12:20:54 132.65.116.111 132.65= .60.56 NLM 0x5e15f77a (1578497914) 886 = V4 UNLOCK Call (Reply In 4480696) FH:0x54b075a0 svid:13629 pos:0-0 4480696 2020-01-08 12:20:54 132.65.60.56 132.65= .116.111 NLM 0x5e15f77a (1578497914) 4045 = V4 UNLOCK Reply (Call In 4480695) Here you see that 2 minutes later the client uses the same transaction ID (= 0x5e15f77a) and the same port again, but the file handle is different, so t= he client is unlocking a different file. 4591136 2020-01-08 12:22:54 132.65.116.111 132.65= .60.56 NLM 0x5e15f77a (1578497914) 886 = [RPC retransmission of #4480695]V4 UNLOCK Call (Reply In 4480696) FH:0xb1= 4b75a8 svid:13629 pos:0-0 4592588 2020-01-08 12:22:57 132.65.116.111 132.65= .60.56 NLM 0x5e15f77a (1578497914) 886 = [RPC retransmission of #4480695]V4 UNLOCK Call (Reply In 4480696) FH:0xb1= 4b75a8 svid:13629 pos:0-0 4598862 2020-01-08 12:23:03 132.65.116.111 132.65= .60.56 NLM 0x5e15f77a (1578497914) 886 = [RPC retransmission of #4480695]V4 UNLOCK Call (Reply In 4480696) FH:0xb1= 4b75a8 svid:13629 pos:0-0 4608871 2020-01-08 12:23:21 132.65.116.111 132.65= .60.56 NLM 0x5e15f77a (1578497914) 886 = [RPC retransmission of #4480695]V4 UNLOCK Call (Reply In 4480696) FH:0xb1= 4b75a8 svid:13629 pos:0-0 4635984 2020-01-08 12:23:59 132.65.116.111 132.65= .60.56 NLM 0x5e15f77a (1578497914) 886 = [RPC retransmission of #4480695]V4 UNLOCK Call (Reply In 4480696) FH:0xb1= 4b75a8 svid:13629 pos:0-0 transaction ID reuse is also seen for a number of other transaction IDs sta= rting at the same time. Withing ONTAP 9.3 we have changed the way our Replay-Cache tracks requests = by including a checksum of the RPC request. Both in in this and earlier rel= eases ONTAP would cache the call in frame 4480695, but starintg in 9.3 we t= hen cache the checksum as part of that. When the client sends the request in frame 4591136 it uses the same transac= tion ID (0x5e15f77a) and same port again. Here the problem is that we alrea= dy hold a checksum in cache for the =93same transaction=94 =85 this seems to be happening after the client did not receive the response an= d re-transmits the request. danny On 24 Dec 2019, at 5:02, Rick Macklem > wrote: Richard P Mackerras wrote: Hi, We had some bully type workloads emerge when we moved a lot of block storage from old XIV to new all flash 3PAR. I wonder if your IMAP issue might have emerged just because suddenly there was the opportunity with all flash. QOS is good on 9.x ONTAP. If anyone says it=92s not then they last looked on 8.x. So I suggest you QOS the IMAP workload. Nobody should be using UDP with NFS unless they have a very specific set of circumstances. TCP was a real step forward. Well, I can't argue with this, considering I did the first working implemen= tation of NFS over TCP. It was actually Mike Karels that suggested I try doing so, There's a paper in a very old Usenix Conference Proceedings, but it is so o= ld that it isn't on the Usenix web page (around 1988 in Denver, if I recall). = I don't even have a copy myself, although I was the author. Now, having said that, I must note that the Network Lock Manager (NLM) and Network Status Monitor (NSM) were not NFS. They were separate stateful protocols (poorly designed imho) that Sun never published. NFS as Sun designed it (NFSv2 and NFSv3) were "stateless server" protocols, so that they could work reliably without server crash recovery. However, the NLM was inherently stateful, since it was dealing with file lo= cks. So, you can't really lump the NLM with NFS (and you should avoid use of the NLM over any transport imho). NFSv4 tackled the difficult problem of having a "stateful server" and crash= recovery, which resulted in a much more complex protocol (compare the size of RFC-181= 3 vs RFC-5661 to get some idea of this). rick Cheers Richard _______________________________________________ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" _______________________________________________ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" --_002_YQBPR0101MB1427FF31676F6C4C641CA933DD390YQBPR0101MB1427_ Content-Type: application/octet-stream; name="xid.patch" Content-Description: xid.patch Content-Disposition: attachment; filename="xid.patch"; size=1682; creation-date="Thu, 09 Jan 2020 03:24:00 GMT"; modification-date="Thu, 09 Jan 2020 03:24:00 GMT" Content-Transfer-Encoding: base64 LS0tIHJwYy9jbG50X2RnLmMuc2F2CTIwMjAtMDEtMDggMTQ6MjA6MzQuMTkzOTkzMDAwIC0wODAw CisrKyBycGMvY2xudF9kZy5jCTIwMjAtMDEtMDggMTQ6NDY6MDMuMjEzMzkzMDAwIC0wODAwCkBA IC05NCw2ICs5NCw4IEBAIHN0YXRpYyBzdHJ1Y3QgY2xudF9vcHMgY2xudF9kZ19vcHMgPSB7CiAJ LmNsX2NvbnRyb2wgPQljbG50X2RnX2NvbnRyb2wKIH07CiAKK3N0YXRpYyB2b2xhdGlsZSB1aW50 MzJfdCBycGNfeGlkID0gMDsKKwogLyoKICAqIEEgcGVuZGluZyBSUEMgcmVxdWVzdCB3aGljaCBh d2FpdHMgYSByZXBseS4gUmVxdWVzdHMgd2hpY2ggaGF2ZQogICogcmVjZWl2ZWQgdGhlaXIgcmVw bHkgd2lsbCBoYXZlIGNyX3hpZCBzZXQgdG8gemVybyBhbmQgY3JfbXJlcCB0bwpAQCAtMTkzLDYg KzE5NSw3IEBAIGNsbnRfZGdfY3JlYXRlKAogCXN0cnVjdCBfX3JwY19zb2NraW5mbyBzaTsKIAlY RFIgeGRyczsKIAlpbnQgZXJyb3I7CisJdWludDMyX3QgbmV3eGlkOwogCiAJaWYgKHN2Y2FkZHIg PT0gTlVMTCkgewogCQlycGNfY3JlYXRlZXJyLmNmX3N0YXQgPSBSUENfVU5LTk9XTkFERFI7CkBA IC0yNDUsOCArMjQ4LDkgQEAgY2xudF9kZ19jcmVhdGUoCiAJY3UtPmN1X3NlbnQgPSAwOwogCWN1 LT5jdV9jd25kX3dhaXQgPSBGQUxTRTsKIAkodm9pZCkgZ2V0bWljcm90aW1lKCZub3cpOwotCWN1 LT5jdV94aWQgPSBfX1JQQ19HRVRYSUQoJm5vdyk7Ci0JY2FsbF9tc2cucm1feGlkID0gY3UtPmN1 X3hpZDsKKwluZXd4aWQgPSBfX1JQQ19HRVRYSUQoJm5vdyk7CisJYXRvbWljX2NtcHNldF8zMigm cnBjX3hpZCwgMCwgbmV3eGlkKTsKKwljYWxsX21zZy5ybV94aWQgPSBhdG9taWNfZmV0Y2hhZGRf MzIoJnJwY194aWQsIDEpOwogCWNhbGxfbXNnLnJtX2NhbGwuY2JfcHJvZyA9IHByb2dyYW07CiAJ Y2FsbF9tc2cucm1fY2FsbC5jYl92ZXJzID0gdmVyc2lvbjsKIAl4ZHJtZW1fY3JlYXRlKCZ4ZHJz LCBjdS0+Y3VfbWNhbGxjLCBNQ0FMTF9NU0dfU0laRSwgWERSX0VOQ09ERSk7CkBAIC00MTgsOCAr NDIyLDcgQEAgY2xudF9kZ19jYWxsKAogY2FsbF9hZ2FpbjoKIAltdHhfYXNzZXJ0KCZjcy0+Y3Nf bG9jaywgTUFfT1dORUQpOwogCi0JY3UtPmN1X3hpZCsrOwotCXhpZCA9IGN1LT5jdV94aWQ7CisJ eGlkID0gYXRvbWljX2ZldGNoYWRkXzMyKCZycGNfeGlkLCAxKTsKIAogc2VuZF9hZ2FpbjoKIAlt dHhfdW5sb2NrKCZjcy0+Y3NfbG9jayk7CkBAIC04NjUsMTQgKzg2OCwxNiBAQCBjbG50X2RnX2Nv bnRyb2woQ0xJRU5UICpjbCwgdV9pbnQgcmVxdWVzdCwgdm9pZCAqaW5mbykKIAkJKHZvaWQpIG1l bWNweSgmY3UtPmN1X3JhZGRyLCBhZGRyLCBhZGRyLT5zYV9sZW4pOwogCQlicmVhazsKIAljYXNl IENMR0VUX1hJRDoKLQkJKih1aW50MzJfdCAqKWluZm8gPSBjdS0+Y3VfeGlkOworCQkqKHVpbnQz Ml90ICopaW5mbyA9IHJwY194aWQ7CiAJCWJyZWFrOwogCisjaWZkZWYgbm90bm93CiAJY2FzZSBD TFNFVF9YSUQ6CiAJCS8qIFRoaXMgd2lsbCBzZXQgdGhlIHhpZCBvZiB0aGUgTkVYVCBjYWxsICov CiAJCS8qIGRlY3JlbWVudCBieSAxIGFzIGNsbnRfZGdfY2FsbCgpIGluY3JlbWVudHMgb25jZSAq LwogCQljdS0+Y3VfeGlkID0gKih1aW50MzJfdCAqKWluZm8gLSAxOwogCQlicmVhazsKKyNlbmRp ZgogCiAJY2FzZSBDTEdFVF9WRVJTOgogCQkvKgo= --_002_YQBPR0101MB1427FF31676F6C4C641CA933DD390YQBPR0101MB1427_--