From owner-freebsd-net@freebsd.org Wed Mar 17 17:17:57 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A5F5E5712D8 for ; Wed, 17 Mar 2021 17:17:57 +0000 (UTC) (envelope-from jbreitman@tildenparkcapital.com) Received: from us-smtp-delivery-145.mimecast.com (us-smtp-delivery-145.mimecast.com [216.205.24.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.mimecast.com", Issuer "DigiCert TLS RSA SHA256 2020 CA1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0xgS4NmYz4lD6 for ; Wed, 17 Mar 2021 17:17:56 +0000 (UTC) (envelope-from jbreitman@tildenparkcapital.com) Received: from zmcc-3-mta-2.zmailcloud.com (zmcc-3-mta-2.zmailcloud.com [35.238.170.66]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-465-sGD8Mj3cONu-kHD97twkig-1; Wed, 17 Mar 2021 13:17:53 -0400 X-MC-Unique: sGD8Mj3cONu-kHD97twkig-1 Received: from zmcc-3-mta-2.zmailcloud.com (localhost [127.0.0.1]) by zmcc-3-mta-2.zmailcloud.com (Postfix) with ESMTPS id 166CAE0E18 for ; Wed, 17 Mar 2021 12:17:53 -0500 (CDT) Received: from localhost (localhost [127.0.0.1]) by zmcc-3-mta-2.zmailcloud.com (Postfix) with ESMTP id 087CFE0E07 for ; Wed, 17 Mar 2021 12:17:53 -0500 (CDT) X-Virus-Scanned: amavisd-new at zmcc-3-mta-2.zmailcloud.com Received: from zmcc-3-mta-2.zmailcloud.com ([127.0.0.1]) by localhost (zmcc-3-mta-2.zmailcloud.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id kAvn3S2H6QBc for ; Wed, 17 Mar 2021 12:17:52 -0500 (CDT) Received: from jbreitman-mac.zxcvm.com (unknown [72.22.182.150]) by zmcc-3-mta-2.zmailcloud.com (Postfix) with ESMTPSA id CA246E0D63 for ; Wed, 17 Mar 2021 12:17:52 -0500 (CDT) From: Jason Breitman Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: NFS Mount Hangs Date: Wed, 17 Mar 2021 13:17:52 -0400 References: <3750001D-3F1C-4D9A-A9D9-98BCA6CA65A4@tildenparkcapital.com> To: freebsd-net@freebsd.org In-Reply-To: <3750001D-3F1C-4D9A-A9D9-98BCA6CA65A4@tildenparkcapital.com> Message-Id: <33693DE3-7FF8-4FAB-9A75-75576B88A566@tildenparkcapital.com> X-Mailer: Apple Mail (2.3608.120.23.2.4) X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: tildenparkcapital.com Content-Type: text/plain; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4F0xgS4NmYz4lD6 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of jbreitman@tildenparkcapital.com designates 216.205.24.145 as permitted sender) smtp.mailfrom=jbreitman@tildenparkcapital.com X-Spamd-Result: default: False [-2.90 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_VERYGOOD(0.00)[216.205.24.145:from]; RCVD_COUNT_FIVE(0.00)[6]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; MV_CASE(0.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; ARC_NA(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:216.205.24.0/24]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_NA(0.00)[tildenparkcapital.com: no valid DMARC record]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:30031, ipnet:216.205.24.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net]; RCVD_IN_DNSWL_LOW(-0.10)[216.205.24.145:from] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Mar 2021 17:17:57 -0000 Please review the details below and let me know if there is a setting that = I should apply to my FreeBSD NFS Server or if there is a bug fix that I can= apply to resolve my issue. I shared this information with the linux-nfs mailing list and they believe = the issue is on the server side. Issue NFSv4 mounts periodically hang on the NFS Client. During this time, it is possible to manually mount from another NFS Server = on the NFS Client having issues. Also, other NFS Clients are successfully mounting from the NFS Server in qu= estion. Rebooting the NFS Client appears to be the only solution. Environment NFS Server OS: FreeBSD 12.1-RELEASE-p5 NFS Client OS: =09=09Debian Buster 10.8 Kernel:=094.19.171-2 Protocol:=09NFSv4 with Kerberos Security Mount Options:=09nfs-server.domain.com:/data=09/mnt/data=09nfs4=09lookupcac= he=3Dpos,noresvport,sec=3Dkrb5,hard,rsize=3D1048576,wsize=3D1048576=0900 Output from the NFS Client when the issue occurs # netstat -an | grep NFS.Server.IP.X tcp 0 0 NFS.Client.IP.X:46896 NFS.Server.IP.X:2049 F= IN_WAIT2 # cat /sys/kernel/debug/sunrpc/rpc_xprt/*/info netid: tcp addr: NFS.Server.IP.X port: 2049 state: 0x51=20 syslog Mar 4 10:29:27 hostname kernel: [437414.131978] -pid- flgs status -client-= --rqstp- -timeout ---ops-- Mar 4 10:29:27 hostname kernel: [437414.133158] 57419 40a1 0 9b723c73= 143cfadf 30000 4ca953b5 nfsv4 OPEN_NOATTR a:call_connect_status [sunrpc= ] q:xprt_pending Mar 4 10:29:27 hostname kernel: [437414.135211] 57420 4081 -11 9b723c73= (null) 0 4ca953b5 nfsv4 OPEN_NOATTR a:call_reserveresult [sunrpc]= q:xprt_sending Mar 4 10:29:27 hostname kernel: [437414.137250] 57421 4081 -11 9b723c73= (null) 0 4ca953b5 nfsv4 OPEN_NOATTR a:call_reserveresult [sunrpc]= q:xprt_sending Mar 4 10:29:27 hostname kernel: [437414.139345] 57422 4081 -11 9b723c73= (null) 0 4ca953b5 nfsv4 OPEN_NOATTR a:call_reserveresult [sunrpc]= q:xprt_sending Mar 4 10:29:27 hostname kernel: [437414.141496] 57423 4081 -11 9b723c73= (null) 0 4ca953b5 nfsv4 OPEN_NOATTR a:call_reserveresult [sunrpc]= q:xprt_sending Mar 4 10:29:27 hostname kernel: [437414.143712] 57424 4081 -11 9b723c73= (null) 0 4ca953b5 nfsv4 OPEN_NOATTR a:call_reserveresult [sunrpc]= q:xprt_sending Mar 4 10:29:27 hostname kernel: [437414.145940] 57425 4081 -11 9b723c73= (null) 0 4ca953b5 nfsv4 OPEN_NOATTR a:call_reserveresult [sunrpc]= q:xprt_sending Mar 4 10:29:27 hostname kernel: [437414.148227] 57426 4081 -11 9b723c73= (null) 0 4ca953b5 nfsv4 OPEN_NOATTR a:call_reserveresult [sunrpc]= q:xprt_sending Mar 4 10:29:27 hostname kernel: [437414.150575] 57427 4081 -11 9b723c73= (null) 0 4ca953b5 nfsv4 OPEN_NOATTR a:call_reserveresult [sunrpc]= q:xprt_sending Mar 4 10:29:27 hostname kernel: [437414.152938] 57428 4080 -11 9b723c73= (null) 0 fb0400d nfsv4 LOOKUP a:call_reserveresult [sunrpc] q:xp= rt_sending Mar 4 10:29:27 hostname kernel: [437414.154478] 57433 4080 -11 27bf33c1= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.156023] 57434 4080 -11 27bf33c1= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.157549] 57435 4080 -11 27bf33c1= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.159073] 57436 4080 -11 27bf33c1= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.160587] 57437 4080 -11 27bf33c1= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.162094] 57438 4080 -11 27bf33c1= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.163597] 57431 4080 -11 3118865a= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.165100] 57432 4080 -11 3118865a= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.166598] 57439 4080 -11 3118865a= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.168088] 57440 4080 -11 3118865a= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.169573] 57441 4080 -11 3118865a= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.171058] 57442 4080 -11 3118865a= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.172532] 57443 4080 -11 3118865a= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.173991] 57444 4080 -11 3118865a= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.175452] 57445 4080 -11 3118865a= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.176906] 57446 4080 -11 3118865a= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.178349] 57447 4080 -11 3118865a= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.179792] 57448 4080 -11 3118865a= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.181227] 57449 4080 -11 3118865a= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.182655] 57450 4080 -11 3118865a= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.184081] 57451 4080 -11 3118865a= (null) 0 fb0400d nfsv4 GETATTR a:call_reserveresult [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.185494] 57418 4880 -11 d42d6144= ab8b1696 0 fb0400d nfsv4 STATFS a:call_connect_status [sunrpc] q:x= prt_sending Mar 4 10:29:27 hostname kernel: [437414.186905] 57430 4080 -11 d42d6144= (null) 0 fb0400d nfsv4 STATFS a:call_reserveresult [sunrpc] q:xp= rt_sending Mar 4 10:29:27 hostname kernel: [437414.188310] 57429 5281 -11 907fb25c= (null) 0 5fa6554c nfsv4 SEQUENCE a:call_reserveresult [sunrpc] q:= xprt_sending Mar 4 10:29:30 hostname kernel: [437417.110517] RPC: 57419 xprt_connect_st= atus: connect attempt timed out Mar 4 10:29:30 hostname kernel: [437417.112172] RPC: 57419 call_connect_st= atus (status -110) Mar 4 10:29:30 hostname kernel: [437417.113337] RPC: 57419 call_timeout (m= ajor) Mar 4 10:29:30 hostname kernel: [437417.114385] RPC: 57419 call_bind (stat= us 0) Mar 4 10:29:30 hostname kernel: [437417.115402] RPC: 57419 call_connect xp= rt 00000000e061831b is not connected Mar 4 10:29:30 hostname kernel: [437417.116547] RPC: 57419 xprt_connect xp= rt 00000000e061831b is not connected Mar 4 10:30:31 hostname kernel: [437478.551090] RPC: 57419 xprt_connect_st= atus: connect attempt timed out Mar 4 10:30:31 hostname kernel: [437478.552396] RPC: 57419 call_connect_st= atus (status -110) Mar 4 10:30:31 hostname kernel: [437478.553417] RPC: 57419 call_timeout (m= inor) Mar 4 10:30:31 hostname kernel: [437478.554327] RPC: 57419 call_bind (stat= us 0) Mar 4 10:30:31 hostname kernel: [437478.555220] RPC: 57419 call_connect xp= rt 00000000e061831b is not connected Mar 4 10:30:31 hostname kernel: [437478.556254] RPC: 57419 xprt_connect xp= rt 00000000e061831b is not connected Jason Breitman