From owner-freebsd-stable@freebsd.org Wed Jan 8 17:08:14 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 633B21F4AB9 for ; Wed, 8 Jan 2020 17:08:14 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47tG0X63yBz3Lbv for ; Wed, 8 Jan 2020 17:08:12 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cs.huji.ac.il; s=57791128; h=References:To:Cc:In-Reply-To:Date:Subject:Mime-Version:Content-Type:Message-Id:From; bh=dMWGfbcSZ8L72qPMBroBWkDQhRU0Tz305bhKj1DyRA4=; b=Sz6KBctG1SGSVcofsZHYV/p5utbDIIdsAuIVkB21vOVITboxFirzIyl8/IiQiR1hYWAT8F/Swfm51plSX8aw0Yh8Jp/A3FTfrTckWY32qYLmV20JY+f5gXNohCvYHEZCzJ52/fbEV3W8/Zzdf32OaPtg8iXX6o+xi5JMZy48a6XvvDiZ67jT6YE3UOY3na5QpynlwJd1JTC493K96IrZfLXHwBy466xBk12T5qcaSj7TFuW0+Nktz67sxYpHK7usc7ERCVBaVv74MnnEYdd3X16/k1VvX/LTcsN0jMwBFQn0bKB8cUNAS67KE5O4CxEBTSyecgtLIBdU3TsOZNDB1Q==; Received: from macmini.bk.cs.huji.ac.il ([132.65.179.19]) by kabab.cs.huji.ac.il with esmtp id 1ipEoR-000Lj4-9B; Wed, 08 Jan 2020 19:08:07 +0200 From: Daniel Braniss Message-Id: Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.1\)) Subject: Re: nfs lockd errors after NetApp software upgrade. Date: Wed, 8 Jan 2020 19:08:07 +0200 In-Reply-To: Cc: Richard P Mackerras , Adam McDougall , "freebsd-stable@freebsd.org" To: Rick Macklem References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> <8A78F67B-C244-45CF-B9BF-D7062669B33B@cs.huji.ac.il> X-Mailer: Apple Mail (2.3608.60.0.2.1) X-Rspamd-Queue-Id: 47tG0X63yBz3Lbv X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cs.huji.ac.il header.s=57791128 header.b=Sz6KBctG; dmarc=pass (policy=none) header.from=huji.ac.il; spf=none (mx1.freebsd.org: domain of danny@cs.huji.ac.il has no SPF policy when checking 132.65.116.210) smtp.mailfrom=danny@cs.huji.ac.il X-Spamd-Result: default: False [-3.23 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[cs.huji.ac.il:s=57791128]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; IP_SCORE(-1.93)[ip: (-4.91), ipnet: 132.64.0.0/13(-2.67), asn: 378(-2.14), country: IL(0.05)]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; URI_COUNT_ODD(1.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[cs.huji.ac.il:+]; DMARC_POLICY_ALLOW(-0.50)[huji.ac.il,none]; RCVD_IN_DNSWL_NONE(0.00)[210.116.65.132.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:378, ipnet:132.64.0.0/13, country:IL]; FREEMAIL_CC(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jan 2020 17:08:14 -0000 top posting NetAPP reply: =E2=80=A6 Here you can see transaction ID (0x5e15f77a) being used over port 886 = and the NFS server successfully responds. =20 4480695 2020-01-08 12:20:54 132.65.116.111 = 132.65.60.56 NLM 0x5e15f77a (1578497914) 886 = V4 UNLOCK Call (Reply In 4480696) FH:0x54b075a0 svid:13629 = pos:0-0 4480696 2020-01-08 12:20:54 132.65.60.56 = 132.65.116.111 NLM 0x5e15f77a (1578497914) 4045 = V4 UNLOCK Reply (Call In 4480695) =20 Here you see that 2 minutes later the client uses the same transaction = ID (0x5e15f77a) and the same port again, but the file handle is = different, so the client is unlocking a different file. =20 4591136 2020-01-08 12:22:54 132.65.116.111 = 132.65.60.56 NLM 0x5e15f77a (1578497914) 886 = [RPC retransmission of #4480695]V4 UNLOCK Call (Reply In = 4480696) FH:0xb14b75a8 svid:13629 pos:0-0 4592588 2020-01-08 12:22:57 132.65.116.111 = 132.65.60.56 NLM 0x5e15f77a (1578497914) 886 = [RPC retransmission of #4480695]V4 UNLOCK Call (Reply In = 4480696) FH:0xb14b75a8 svid:13629 pos:0-0 4598862 2020-01-08 12:23:03 132.65.116.111 = 132.65.60.56 NLM 0x5e15f77a (1578497914) 886 = [RPC retransmission of #4480695]V4 UNLOCK Call (Reply In = 4480696) FH:0xb14b75a8 svid:13629 pos:0-0 4608871 2020-01-08 12:23:21 132.65.116.111 = 132.65.60.56 NLM 0x5e15f77a (1578497914) 886 = [RPC retransmission of #4480695]V4 UNLOCK Call (Reply In = 4480696) FH:0xb14b75a8 svid:13629 pos:0-0 4635984 2020-01-08 12:23:59 132.65.116.111 = 132.65.60.56 NLM 0x5e15f77a (1578497914) 886 = [RPC retransmission of #4480695]V4 UNLOCK Call (Reply In = 4480696) FH:0xb14b75a8 svid:13629 pos:0-0 =20 transaction ID reuse is also seen for a number of other transaction IDs = starting at the same time. =20 Withing ONTAP 9.3 we have changed the way our Replay-Cache tracks = requests by including a checksum of the RPC request. Both in in this and = earlier releases ONTAP would cache the call in frame 4480695, but = starintg in 9.3 we then cache the checksum as part of that. =20 When the client sends the request in frame 4591136 it uses the same = transaction ID (0x5e15f77a) and same port again. Here the problem is = that we already hold a checksum in cache for the =E2=80=9Csame = transaction=E2=80=9D =E2=80=A6 this seems to be happening after the client did not receive the response = and re-transmits the request. danny > On 24 Dec 2019, at 5:02, Rick Macklem wrote: >=20 > Richard P Mackerras wrote: >> Hi, >>=20 >> We had some bully type workloads emerge when we moved a lot of block >> storage from old XIV to new all flash 3PAR. I wonder if your IMAP = issue >> might have emerged just because suddenly there was the opportunity = with all >> flash. QOS is good on 9.x ONTAP. If anyone says it=E2=80=99s not then = they last >> looked on 8.x. So I suggest you QOS the IMAP workload. >>=20 >> Nobody should be using UDP with NFS unless they have a very specific = set >> of circumstances. TCP was a real step forward. > Well, I can't argue with this, considering I did the first working = implementation > of NFS over TCP. It was actually Mike Karels that suggested I try = doing so, > There's a paper in a very old Usenix Conference Proceedings, but it is = so old > that it isn't on the Usenix web page (around 1988 in Denver, if I = recall). I don't > even have a copy myself, although I was the author. >=20 > Now, having said that, I must note that the Network Lock Manager (NLM) = and > Network Status Monitor (NSM) were not NFS. They were separate stateful > protocols (poorly designed imho) that Sun never published. >=20 > NFS as Sun designed it (NFSv2 and NFSv3) were "stateless server" = protocols, > so that they could work reliably without server crash recovery. > However, the NLM was inherently stateful, since it was dealing with = file locks. >=20 > So, you can't really lump the NLM with NFS (and you should avoid use = of the > NLM over any transport imho). >=20 > NFSv4 tackled the difficult problem of having a "stateful server" and = crash recovery, > which resulted in a much more complex protocol (compare the size of = RFC-1813 > vs RFC-5661 to get some idea of this). >=20 > rick >=20 > Cheers >=20 > Richard > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org"