From owner-freebsd-stable@freebsd.org Sun Dec 22 17:01:29 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 357B91D7766 for ; Sun, 22 Dec 2019 17:01:29 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660056.outbound.protection.outlook.com [40.107.66.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47gpfc35yHz4ZV8 for ; Sun, 22 Dec 2019 17:01:28 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IzTx503JEXNDmcababWz38EKJdD+/dn78iK6NTkEKz2Moj3tERnMw0uDSdS7l2W4sDLMNhhQtXx9wgmRlkrqiUBSHIok9rNNT0wLJvO2Jxo9b5N2CRsZSNQRs3ZQDj4E+IETUtsoR5g/lQ9CMoV5lh2bmWm/BdZA5KMtJhyAGdt/ZwSwMRlrShiKfvbqyci/6olzWBcGF5fG6XSYta52hV0bQJ3JhCdy2RK4031mAwNu1hu9u4kHBYEIck9DDauMDc+RtltSF8RRLaAI9QhtGTmnfcFe97/m/Ts/TVjwVZJXav8kXf6OAR1NITDfg6wmkUvAOYzdP9CccD5NnZuglw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FJz7uBOnfOW+W3b2XR+QP4mjtEfQFiU/bFewya/fK9Q=; b=ZA8MvEsWgzdXItQRFDHOrYGoQjoOZVvUOUnON1owCTFSxYg8NMlbeWbpd9ThEi2s1WgzAHTjioQvt1FuwRLssnMk+WJzNC09p6cRFgRnF82pYdJ/eIepN5PHmYkqmSB1+Ci7Pahs4HEegSyx7MqBJZFDIQz18kmGMEb31MGwHbnzONZjKCH0HKOtBlVX2VPDozpAwrWRNKwz13tmsJWSPf1Bv0wllcDqHeN2ckOAhs8RkonrORrQ343mWrOL6GbH5zadDAdP6YZkbyCuTPu/NBTu3jTSHCv/FpvJKKXF+S/1wJPIxCnttEOTyu8ZpBQVecDp38PUiH9lUL7j5l+u+A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM (52.132.69.153) by YQBPR0101MB1252.CANPRD01.PROD.OUTLOOK.COM (52.132.72.141) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2559.16; Sun, 22 Dec 2019 17:01:24 +0000 Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::9504:a50d:ee12:b75]) by YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::9504:a50d:ee12:b75%5]) with mapi id 15.20.2538.023; Sun, 22 Dec 2019 17:01:24 +0000 From: Rick Macklem To: Daniel Braniss CC: Adam McDougall , "freebsd-stable@freebsd.org" Subject: Re: nfs lockd errors after NetApp software upgrade. Thread-Topic: nfs lockd errors after NetApp software upgrade. Thread-Index: AQHVtawq+ga5QLcdVkqBDG/GW9zFg6e/+Am+gAARTACAAANHAIAAi7Y3gACf34CAAEVO6IAABk4AgADWGACAAO1eZYAA7uGAgACmPw2AANdsAIAAsCi6 Date: Sun, 22 Dec 2019 17:01:24 +0000 Message-ID: References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> <8A78F67B-C244-45CF-B9BF-D7062669B33B@cs.huji.ac.il> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 46953a5e-7daf-48c3-c256-08d7870093e9 x-ms-traffictypediagnostic: YQBPR0101MB1252: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:4125; x-forefront-prvs: 02596AB7DA x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(136003)(346002)(366004)(396003)(376002)(189003)(199004)(55016002)(5660300002)(66446008)(54906003)(53546011)(478600001)(66946007)(966005)(66556008)(64756008)(66476007)(6506007)(2906002)(52536014)(9686003)(4326008)(6916009)(7696005)(33656002)(316002)(786003)(71200400001)(76116006)(86362001)(81156014)(8676002)(81166006)(186003)(8936002); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB1252; H:YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: lmLn9FT5e3FOERjC+hUL4EBxxWzuPYSSvIrbqhHRFfkvcdIYTSutgIuAavuVvqIrREGadair0WbjTUaCZdhPaD9ZYqF6s+fx2fcZjr/CwaAUJp8lAOwKr7dXzSqiwTSXwra7W0rI9yP781nP4SGV18VPR2b5ncEpIeaYm2cOKUzfcw0InrsInnI4jWJ0AysPZ3Jh1LdcJ2vg3Ue0NTqekCYxbTOdrQBPNQdD48p8qeJksC8KWssTBpZuw9pD1cBmWuwGZX7YI1rFFGdZ7qyUhDih+f23rXitbKK1Ye6vSEGLyrRldzryoUUBg3kZ8AxrOIWGJXoSd+EueppSbZZlWHHKgOx/zsM9J1y0jKUoLDFtW7uHn5+nJZxqX3iXNG0Dg33UyzgoZAyMyiLp6VNvf32lfGL7s8RFlDw23lIKgP8uNGajxSrer7PXQULOs1XNKETbnpPUawmHnywYDJpKSrelouy5o5t3ESRp2aApbzs= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 46953a5e-7daf-48c3-c256-08d7870093e9 X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Dec 2019 17:01:24.5835 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: LVjDWtCObw32wSqgLMU91jc6LGOlfoB0aP0U+yNCw0756WwVK0wJ8uKi0kAeM736OJeItq1iCoSLXtJZ59i1BQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1252 X-Rspamd-Queue-Id: 47gpfc35yHz4ZV8 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.56 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.66 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[uoguelph.ca]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[56.66.107.40.list.dnswl.org : 127.0.3.0]; IP_SCORE(-1.36)[ipnet: 40.64.0.0/10(-3.84), asn: 8075(-2.93), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[i=1] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Dec 2019 17:01:29 -0000 Daniel Braniss wrote:=0A= >> On 21 Dec 2019, at 19:32, Rick Macklem wrote:=0A= >>=0A= >> Daniel Braniss wrote:=0A= >>>> On 20 Dec 2019, at 19:19, Rick Macklem >>>> wrote:=0A= >>>>=0A= >>>> Adam McDougall wrote:=0A= >>>>> Try changing bool_t do_tcp =3D FALSE; to TRUE in=0A= >>>>> /usr/src/sys/nlm/nlm_prot_impl.c, recompile the kernel and try again.= I=0A= >>>>> think this makes it match Linux client behavior. I suspect I ran into= =0A= >>>>> the same issue as you. I do think I used nolockd is a workaround=0A= >>>>> temporarily. I can provide some more details if it works.=0A= >>>> If this fixes the problem, please let me know.=0A= >>>>=0A= >>>> I'm not sure I'd want to change the default, since it might break thin= gs for=0A= >>>> others, but I can definitely make it a tunable, so that people don't n= eed to=0A= >>>> recompile a kernel to deal with it.=0A= >>>>=0A= >>>>=0A= >>> great! I was just about to see how it can be done(tunable) but need to = check if it can >be done=0A= >>> at any time, or just at boot time.=0A= >> I haven't looked at the code, but I suspect changing it on the fly could= cause problems,=0A= >> so I am inclined to make it a tunable (boot time only).=0A= my feelings too.=0A= >>=0A= >>> thanks.=0A= >>> btw, currently, from several hours of analysing the traffic, it seems t= hat nlm is UDP.=0A= >> I assume that means you haven't tried flipping it to TCP yet.=0A= >I will soon, but I have my doubts, the problem is caused my multiple event= s, i.e, it >happened once while=0A= >I was doing svn checkout, but i have done it several times since, and no i= ssues. So it >must be=0A= >an aggregation of factors. Other hosts are reporting locks times too.=0A= Well, I've noted the flawed protocol. Here's an example (from my limited un= derstanding of these protocols, where there has never been a published spec= ) :=0A= - The NLM supports a "blocking lock request" that goes something like this.= ..=0A= - client requests lock and is willing to wait for it=0A= - if server has a conflicting lock on the file, it replies "I'll acquire= the lock for=0A= you when I can and let you know".=0A= --> When the conflicting lock is released, the server acquires the loc= k and does=0A= a callback (server->client RPC) to tell the client it now has t= he lock.=0A= You don't have to think about this for long to realize that any network unr= eliability=0A= or partitioning could result in trouble.=0A= The kernel RPC layer may do some retries of the RPCs (this is controlled by= the=0A= parameters set for the RPC), but at some point the protocol asks the NSM=0A= (rpc.statd) if the machine is "up" and then uses the NSM's answer to deal w= ith it.=0A= (The NSM basically pokes other systems and notes they are "up" if they get= =0A= replies to these pokes. It uses IP broadcast at some point.)=0A= =0A= Now, maybe switching to TCP will make the RPCs reliable enough that it will= =0A= work, or maybe it won't? (It certainly sounds like the Netapp upgrade is ca= using=0A= some kind of network issue, and the NLM doesn't tolerate that well.)=0A= =0A= rick=0A= =0A= danny=0A= =0A= >=0A= > Please let us know how it goes, rick=0A= >=0A= > danny=0A= >=0A= >=0A= > rick=0A= >=0A= > On 12/19/19 9:21 AM, Daniel Braniss wrote:=0A= >=0A= >=0A= > On 19 Dec 2019, at 16:09, Rick Macklem > wrote:=0A= >=0A= > Daniel Braniss wrote:=0A= > [stuff snipped]=0A= > all mounts are nfsv3/tcp=0A= > This doesn't affect what the NLM code (rpc.lockd) uses. I honestly don't = know when=0A= > the NLM uses tcp vs udp. I think rpc.statd still uses IP broadcast at tim= es.=0A= > can the replay cache have any influence here? I tend to remember way back= issues=0A= > with it,=0A= >=0A= > To me, it looks like a network configuration issue.=0A= > that was/is my gut feelings too, but, as far as we can tell, nothing has = changed in the network infrastructure,=0A= > the problems appeared after the NetAPP=92s software was updated, it was w= orking fine till then.=0A= >=0A= > the problems are also happening on freebsd 12.1=0A= >=0A= > You could capture packets (maybe when a client first starts rpc.statd and= rpc.lockd)=0A= > and then look at them in wireshark. I'd disable statup of rpc.lockd and r= pc.statd=0A= > at boot for a test client and then run something like:=0A= > # tcpdump -s 0 -s out.pcap host =0A= > - and then start rpc.statd and rpc.lockd=0A= > Then I'd look at out.pcap in wireshark (much better at decoding this stuf= f than=0A= > tcpdump). I'd look for things like different reply IP addresses from the = Netapp,=0A= > which might confuse this tired old NLM protocol Sun devised in the mid-19= 80s.=0A= >=0A= > it=92s going to be an interesting week end :-(=0A= >=0A= > the error is also appearing on freebsd-11.2-stable, I=92m now checking if= it=92s also=0A= > happening on 12.1=0A= > btw, the NetApp version is 9.3P17=0A= > Yes. I wasn't the author of the NSM and NLM code (long ago I refused to e= ven=0A= > try to implement it, because I knew the protocol was badly broken) and I = avoid=0A= > fiddling with. As such, it won't have change much since around FreeBSD7.= =0A= > and we haven=92t had any issues with it for years, so you must have done = something good=0A= >=0A= > cheers,=0A= > danny=0A= >=0A= >=0A= > rick=0A= >=0A= > cheers,=0A= > danny=0A= >=0A= > rick=0A= >=0A= > Cheers=0A= >=0A= > Richard=0A= > (NetApp admin)=0A= >=0A= > On Wed, 18 Dec 2019 at 15:46, Daniel Braniss > wrote:=0A= >=0A= >=0A= > On 18 Dec 2019, at 16:55, Rick Macklem > wrote:=0A= >=0A= > Daniel Braniss wrote:=0A= >=0A= > Hi,=0A= > The server with the problems is running FreeBSD 11.1 stable, it was worki= ng fine for >several months,=0A= > but after a software upgrade of our NetAPP server it=92s reporting many l= ockd errors >and becomes catatonic,=0A= > ...=0A= > Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd not respo= nding=0A= > Dec 18 13:11:45 moo-09 last message repeated 7 times=0A= > Dec 18 13:12:55 moo-09 last message repeated 8 times=0A= > Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd is alive = again=0A= > Dec 18 13:13:10 moo-09 last message repeated 8 times=0A= > Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen = queue >overflow: 194 already in queue awaiting acceptance (1 occurrences)= =0A= > Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen = queue >overflow: 193 already in queue awaiting acceptance (3957 occurrences= )=0A= > Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen = queue >overflow: 193 already in queue awaiting acceptance =85=0A= > Seems like their software upgrade didn't improve handling of NLM RPCs?=0A= > Appears to be handling RPCs slowly and/or intermittently. Note that no on= e=0A= > tests it with IPv6, so at least make sure you are still using IPv4 for th= e mounts and=0A= > try and make sure IP broadcast works between client and Netapp. I think t= he NLM=0A= > and NSM (rpc.statd) still use IP broadcast sometimes.=0A= >=0A= > we are ipv4 - we have our own class c :-)=0A= > Maybe the network guys can suggest more w.r.t. why, but as I've stated be= fore,=0A= > the NLM is a fundamentally broken protocol which was never published by S= un,=0A= > so I suggest you avoid using it if at all possible.=0A= > well, at the moment the ball is on NetAPP court, and switching to NFSv4 a= t the moment is out of the question, it=92s=0A= > a production server used by several thousand students.=0A= >=0A= >=0A= > - If the locks don't need to be seen by other clients, you can just use t= he "nolockd"=0A= > mount option.=0A= > or=0A= > - If locks need to be seen by other clients, try NFSv4 mounts. Netapp fil= ers=0A= > should support NFSv4.1, which is a much better protocol that NFSv4.0.=0A= >=0A= > Good luck with it, rick=0A= > thanks=0A= > danny=0A= >=0A= > =85=0A= > any ideas?=0A= >=0A= > thanks,=0A= > danny=0A= >=0A= > _______________________________________________=0A= > freebsd-stable@freebsd.org mailing list=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org<= mailto:freebsd-stable-unsubscribe@freebsd.org>"=0A= >=0A= > _______________________________________________=0A= > freebsd-stable@freebsd.org mailing list=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org<= mailto:freebsd-stable-unsubscribe@freebsd.org>"=0A= >=0A= >=0A= > _______________________________________________=0A= > freebsd-stable@freebsd.org mailing lis= t=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"= =0A= >=0A= >=0A= > _______________________________________________=0A= > freebsd-stable@freebsd.org mailing lis= t=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org<= mailto:freebsd-stable-unsubscribe@freebsd.org>"=0A= > _______________________________________________=0A= > freebsd-stable@freebsd.org mailing lis= t=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org<= mailto:freebsd-stable-unsubscribe@freebsd.org>"=0A= >=0A= =0A= _______________________________________________=0A= freebsd-stable@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"= =0A=