From owner-freebsd-stable@freebsd.org Mon Dec 23 15:13:26 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 001481CA1D0 for ; Mon, 23 Dec 2019 15:13:25 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from mail.egr.msu.edu (boomhauer.egr.msu.edu [35.9.37.164]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47hNCT17r8z3HTZ for ; Mon, 23 Dec 2019 15:13:24 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from boomhauer (localhost [127.0.0.1]) by mail.egr.msu.edu (Postfix) with ESMTP id 9FFA9EB053 for ; Mon, 23 Dec 2019 10:13:23 -0500 (EST) X-Virus-Scanned: amavisd-new at egr.msu.edu Received: from mail.egr.msu.edu ([127.0.0.1]) by boomhauer (boomhauer.egr.msu.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id W9TQjm-bDwTJ for ; Mon, 23 Dec 2019 10:13:23 -0500 (EST) Received: from EGR authenticated sender mcdouga9 Subject: Re: nfs lockd errors after NetApp software upgrade. To: freebsd-stable@freebsd.org References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> <8A78F67B-C244-45CF-B9BF-D7062669B33B@cs.huji.ac.il> From: Adam McDougall Message-ID: Date: Mon, 23 Dec 2019 10:13:22 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 47hNCT17r8z3HTZ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=msu.edu; spf=pass (mx1.freebsd.org: domain of mcdouga9@egr.msu.edu designates 35.9.37.164 as permitted sender) smtp.mailfrom=mcdouga9@egr.msu.edu X-Spamd-Result: default: False [-3.09 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:boomhauer.egr.msu.edu]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[164.37.9.35.list.dnswl.org : 127.0.11.2]; DMARC_POLICY_ALLOW(-0.50)[msu.edu,none]; IP_SCORE(-0.09)[asn: 237(-0.40), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:237, ipnet:35.0.0.0/10, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Dec 2019 15:13:26 -0000 On 12/22/19 12:01 PM, Rick Macklem wrote: > Well, I've noted the flawed protocol. Here's an example (from my limited understanding of these protocols, where there has never been a published spec) : > - The NLM supports a "blocking lock request" that goes something like this... > - client requests lock and is willing to wait for it > - if server has a conflicting lock on the file, it replies "I'll acquire the lock for > you when I can and let you know". > --> When the conflicting lock is released, the server acquires the lock and does > a callback (server->client RPC) to tell the client it now has the lock. > You don't have to think about this for long to realize that any network unreliability > or partitioning could result in trouble. > The kernel RPC layer may do some retries of the RPCs (this is controlled by the > parameters set for the RPC), but at some point the protocol asks the NSM > (rpc.statd) if the machine is "up" and then uses the NSM's answer to deal with it. > (The NSM basically pokes other systems and notes they are "up" if they get > replies to these pokes. It uses IP broadcast at some point.) > > Now, maybe switching to TCP will make the RPCs reliable enough that it will > work, or maybe it won't? (It certainly sounds like the Netapp upgrade is causing > some kind of network issue, and the NLM doesn't tolerate that well.) > > rick tl;dr I think netapp effectively nerfed UDP lockd performance in newer versions, maybe cluster mode. >From my very un-fun experience after migrating our volumes off an older netapp onto a new netapp with flash drives (plenty fast) running Ontap 9.x ("cluster mode"), our typical IO load from idle time IMAP connections was enough to overwhelm the new netapp and drive performance into the ground. The same IO that was perfectly fine on the old netapp. Going into a workday in this state was absolutely not possible. I opened a high priority ticket with netapp, didn't really get anywhere that very long day and settled on nolockd so I could go home and sleep. Both my hunch later and netapp support suggested switching lockd traffic to TCP even though I had no network problems (the old netapp was fine). I think I still run into occasional load issues but the newer netapp OS seemed way more capable of this load when using TCP lockd. Of course they also suggested switching to nfsv4 but I could not seriously entertain validating that type of change for production in less than a day.