From owner-freebsd-net@freebsd.org Wed Mar 17 21:37:24 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 38512579657 for ; Wed, 17 Mar 2021 21:37:24 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-qb1can01on0620.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe5c::620]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F13Qq31wBz3MXh for ; Wed, 17 Mar 2021 21:37:23 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fiasquyav1BqWNDuyqRerc6vGkKuAj1UPH4wv5rm7nin2nqEepyfYKm7Gw2JZV6MXclJqzDK4kIuD8sKrwEKVxCMMnA/kvvge9Id2rmUQATKVb3KxGKQBTkggODA7SCmUM4Nq40es2hHxgOSFi2/j/sR1jv24HWkzUwH59ZUE5NEqLzfAU+4xsKWt1YORX/w17D6pkcPpB/yVvUGldQibymRdpH+bJwzzARP/cgdVQHS+xdvCnQ4qrbbI/hVvELuLH7eLeEU60Q7Kk3aLtzQnuMYJ6iCWyuW63GP9fQ0IpgsQpHiYisITEDai03DRFA+VKjDPa2twrT2Xo48REUJ9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mJFShkCYwL5vmMg6nBSVfwkgucwpjzNXBju8+PQkaJQ=; b=BS8AhXXn4OrGCH5yKnGTpPZetlkUBZH/s2LPrYVQBA9GvDyiYjrbKOT5T5NmXeMAI25DtX6opMof8HoHs1HVI64AJaxJWMfNdRdLgrEwWe9UfTL6VJbkfgXtePLA76ljG/PLCrXXtmpEx0hwjOSSm37oRH/PKY1cbrdJuqUjRbFeaMjOTmUlYy1ymBiG78gFTPsffIkavrmzwp0izv/BbsnG6wT5UeNBGZdA6isNJoVRwehOj4OSn6MkYYiDgOKtVTUcsifrRhTk7gYEe6Qy+suWt9FvqiW09XlNhUGlsGbtq9wSl3t6Y6UdlojYe7hk9Tf/bsaQNHFH2DDcTygEEw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mJFShkCYwL5vmMg6nBSVfwkgucwpjzNXBju8+PQkaJQ=; b=KFVOdLm3XN0euuDkEjBJr1z2cwGHNLdsv3ulN03SSgmd0K/FMiI2uKlZqqjMs7sJKDYKvKL6MrVLTglwnxsOVAugEegqN3zr8CNEToYzMgNbpdiSAXmd50xI36k/d6QM7V6QZfT2xRm9Z5XDinHluWmqrjggaGBQPBgBFElZGFxbGOsC/ce3WmjX6qLxPUfpHUcYngEa3kuyVrNOp+ZtAO7t9Yx32IFReE5n7ODZzbrCPjgHOA/wUsy3fSilHt6h5FH5TXKQ62BqtO2VRXJ06zQXtFdx58Tm0jhL0RWtJhqRVtbRgZcrMxLUnXO4T3C8uTzgV1HcKO41owLG36GSKQ== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQXPR01MB4359.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:7::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3933.32; Wed, 17 Mar 2021 21:37:20 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a%7]) with mapi id 15.20.3933.032; Wed, 17 Mar 2021 21:37:20 +0000 From: Rick Macklem To: Jason Breitman , "freebsd-net@freebsd.org" Subject: Re: NFS Mount Hangs Thread-Topic: NFS Mount Hangs Thread-Index: AQHXG1G2D7AHBwtmAkS1jBAqNNo2I6qIqQ+S Date: Wed, 17 Mar 2021 21:37:20 +0000 Message-ID: References: <3750001D-3F1C-4D9A-A9D9-98BCA6CA65A4@tildenparkcapital.com>, <33693DE3-7FF8-4FAB-9A75-75576B88A566@tildenparkcapital.com> In-Reply-To: <33693DE3-7FF8-4FAB-9A75-75576B88A566@tildenparkcapital.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 7f1e19ba-84fe-403b-e364-08d8e98cd865 x-ms-traffictypediagnostic: YQXPR01MB4359: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: J5vjtmr6uHgcG3gzHj1kFBITVJ2JLYsn7uDHCc6QXEilLexc9BfMop6kZXfZN4vo/qeq8E6+6OI9Mfa+C85J5vfUs04NCBczJcd82hTod6DP8UjXryoHQM8kz8dFgG/RELsC+UsX++s5mGdF9svcS+sGQM1ZqCYhxS4io9QiI86MQjwcir+n4kkrItaIRVFDXq/5up9ghvR4l6BpkId2IrAju11LzjncdwOidNCUqqoSRlDKU1kTbwFvB4pfpFSECs84YLnQEIDLGr7BubhVCeH1pof7X4+ufODlBQMPe8rnrRCu5r+taz95YCdY2UJfdTl+1APxGUaHFXIMm8CoTcF8KHzyAZ37TH4+cj8ccZmXKxPfRR4rubxzRvMITBdJJzUwJss6lCuJ4b6Uk5z6ugXBTnDa9gt7kdGSMHEgJ8yD/iEjtH/Q64Sv3KItXNtAN0XVLgCdYyhfCX/WkhAJdrAWnRa0FCaqxUPigiuTKiKgBKXUjzzofkfr6zcYsmigfTN3XR1blA2TVOj4aGlWtRQtK/Nd9rSJAPXSK5c7Ul5uT7b+JVcUadcbxyXZic4UNuXOu5ivdlsZdxq2yhISCXB1yxpF6qgSIGZ8z/ZJYiI/hLWkwiWhp0ngwAwHIPYzP7W01xFAu7oHG+aNcmfpRg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(366004)(136003)(376002)(346002)(396003)(39860400002)(55016002)(64756008)(7116003)(316002)(9686003)(110136005)(186003)(478600001)(86362001)(33656002)(8676002)(3480700007)(966005)(66556008)(786003)(8936002)(71200400001)(91956017)(66476007)(5660300002)(66446008)(6506007)(83380400001)(2906002)(76116006)(52536014)(7696005)(66946007); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?o2J1RPtvzSjtnllnhbkUCsRsoyJzdEGm5EbdUwixBk25inR1qV6FCkxsy2?= =?iso-8859-1?Q?kKNjsI4t6tJ2tGhjjAz6R80jKp5gF4meB/L1X8GoIBF5JLSVc7m+Csfbhi?= =?iso-8859-1?Q?NxMcjkIgyLGW0lH0rrn4QiLaqLGKP4v4TgsMSuG2eiKu1qb5JswEpn+ldk?= =?iso-8859-1?Q?sp58oZCYbCOH+afbrqX+vAvDIurbLSPuvAIsXWr4eZxDPZDOYm4ZpNr4Tz?= =?iso-8859-1?Q?TPt6uJBad5itKVpSNq1/UNfNtMTyeU1fIehQOESGkGP8Lz9OQzxTy13o/m?= =?iso-8859-1?Q?A8+XpyohFA9cQrwXDjCY5uJy+IfmfCujACVORa8LTkjM86C4tC5IeUJdnc?= =?iso-8859-1?Q?a7OzUC9+zijIOi1Jvs1ysowxckUONAEq0oxOhrcRxGKVwu2orKaqxtpyne?= =?iso-8859-1?Q?1h9OpNbKwEmlWqAGoFLLyfMgf8X6qUIWmAlIG+ZReb3VUoSgzeeywE2jGp?= =?iso-8859-1?Q?ynpc2ycKlP58MMX6ala1LGoxEvIFh6zZzIoD1dXTGXCiJ7AkaMQEYMyylO?= =?iso-8859-1?Q?AkfFYimZI6TDA6OSvSvY5GwCXmiWlGEgdLQMFU4PXl782G7OiCHhp8t9rf?= =?iso-8859-1?Q?GywblaB2K14UJDjKa5ceR0xepPpnXjKP7UGiMNtpS12Xn9CqzsouCze1gK?= =?iso-8859-1?Q?h/rj3/fdItrG3WaB3HxADlHmeJoFI9OniXquTQ26XVbDIDM4Gxjz2f0oHe?= =?iso-8859-1?Q?s8Ott7xTzak+HAU+LgnkxSxvyaM+BKgmP1cUihI7HzMYsSahBLlmk0jY6D?= =?iso-8859-1?Q?ZG5en7Qud0mv1k2NNggUp1RdV27Es2klMxGGIDJhKnFdBMDud6kzYT+yMg?= =?iso-8859-1?Q?Cms7fzhlQnH51f5VrJDDN6fG22fWYq4MOIp/V+oqTf20CEW1h/xChuzXr8?= =?iso-8859-1?Q?j8pIJKTgWjqu7Y7f2uTfSSXo2hKij4UN5VNOjR1lbWT0C5EtxR+6Qv2fVH?= =?iso-8859-1?Q?CYpFaAmWXESVSJ+DBiglFcczvEIdWUagqg3YmZbMNKF5ladR2slIZe0bxn?= =?iso-8859-1?Q?sqO2GghCxOQhU1yErld+5vzsqy88FJo5k7Vz9EjIcgeoE6Xq//0KFpIfQQ?= =?iso-8859-1?Q?zYkUfG5HyJvU9eUggRCEqoc+9kIO85RwHrWEELe6qD3KMN7TN6lFEXzdZ5?= =?iso-8859-1?Q?FEHqNRNfXotZMG3ZuMZHYUS+LrK8+bpPQU7/fC8MbDa41amUFQoGVynMlh?= =?iso-8859-1?Q?DiqH8K5P1lVtWkPiBa8B4hnWYaiNxlO+6I5GCnT7B56HnYuwkR0iVxdNuM?= =?iso-8859-1?Q?rDccoEF3uTSou2BrEJ7xcPQjpBNOpRqiz5O9SsvQiu+cbbfbaQBRj7r4XR?= =?iso-8859-1?Q?9RRmhq1FrI4WjPAFUti0/NelTaTBbobsHS6xf4P9zTbvDjtzOngm+C1E17?= =?iso-8859-1?Q?8T1lnTtggu8Je9qzVVXy04PpmAONHfCeXFOljOFdyz5KLhXRikYHs=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 7f1e19ba-84fe-403b-e364-08d8e98cd865 X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Mar 2021 21:37:20.6235 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ps0AUfr+9bkJVYyiz8BpNDaYQLq6AobLGJRyE83zPWTJEA22KPvqApJE0nudKost/3W0KtDFjlxYR4f7soL0Fg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB4359 X-Rspamd-Queue-Id: 4F13Qq31wBz3MXh X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=KFVOdLm3; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 2a01:111:f400:fe5c::620 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a01:111:f400:fe5c::620:from]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f400::/48]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; SPAMHAUS_ZRD(0.00)[2a01:111:f400:fe5c::620:from:127.0.2.255]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-net] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Mar 2021 21:37:24 -0000 Jason Breitman wrote:=0A= >Please review the details below and let me know if there is a setting that= I should >apply to my FreeBSD NFS Server or if there is a bug fix that I c= an apply to resolve my >issue.=0A= >I shared this information with the linux-nfs mailing list and they believe= the issue is >on the server side.=0A= I actually lurk there and saw your post. I'll admit I smiled when Trond arg= ued=0A= that a hung Linux system is the result of a server failing to send a fin/ac= k for=0A= a closing TCP connection. But, here's a few comments..=0A= =0A= >Issue=0A= >NFSv4 mounts periodically hang on the NFS Client.=0A= >=0A= >During this time, it is possible to manually mount from another NFS Server= on the >NFS Client having issues.=0A= >Also, other NFS Clients are successfully mounting from the NFS Server in q= uestion.=0A= >Rebooting the NFS Client appears to be the only solution.=0A= >=0A= >Environment=0A= >NFS Server=0A= >OS: FreeBSD 12.1-RELEASE-p5=0A= >=0A= >NFS Client=0A= >OS: Debian Buster 10.8=0A= >Kernel: 4.19.171-2=0A= >Protocol: NFSv4 with Kerberos Security=0A= >Mount Options: nfs-server.domain.com:/data /mnt/data nfs4 >l= ookupcache=3Dpos,noresvport,sec=3Dkrb5,hard,rsize=3D1048576,wsize=3D1048576= 00=0A= The maximum I/O size supported by FreeBSD is 128K.=0A= The client should acquire the attributes that indicate that and set rsize/w= size=0A= to that. "# nfsstat -m" on the client should show you what the client=0A= is actually using. If it is larger than 128K, set both rsize and wsize to 1= 28K.=0A= =0A= >Output from the NFS Client when the issue occurs=0A= ># netstat -an | grep NFS.Server.IP.X=0A= >tcp 0 0 NFS.Client.IP.X:46896 NFS.Server.IP.X:2049 = FIN_WAIT2=0A= I'm no TCP guy. Hopefully others might know why the client would be=0A= stuck in FIN_WAIT2 (I vaguely recall this means it is waiting for a fin/ack= ,=0A= but could be wrong?)=0A= =0A= ># cat /sys/kernel/debug/sunrpc/rpc_xprt/*/info=0A= >netid: tcp=0A= >addr: NFS.Server.IP.X=0A= >port: 2049=0A= >state: 0x51=0A= >=0A= >syslog=0A= >Mar 4 10:29:27 hostname kernel: [437414.131978] -pid- flgs status -client= - --rqstp- ->timeout ---ops--=0A= >Mar 4 10:29:27 hostname kernel: [437414.133158] 57419 40a1 0 9b723c7= 3 >143cfadf 30000 4ca953b5 nfsv4 OPEN_NOATTR a:call_connect_status [sunr= pc] >q:xprt_pending=0A= I don't know what OPEN_NOATTR means, but I assume it is some variant=0A= of NFSv4 Open operation.=0A= [stuff snipped]=0A= >Mar 4 10:29:30 hostname kernel: [437417.110517] RPC: 57419 xprt_connect_s= tatus: >connect attempt timed out=0A= >Mar 4 10:29:30 hostname kernel: [437417.112172] RPC: 57419 call_connect_s= tatus =0A= >(status -110)=0A= I have no idea what status -110 means?=0A= >Mar 4 10:29:30 hostname kernel: [437417.113337] RPC: 57419 call_timeout (= major)=0A= >Mar 4 10:29:30 hostname kernel: [437417.114385] RPC: 57419 call_bind (sta= tus 0)=0A= >Mar 4 10:29:30 hostname kernel: [437417.115402] RPC: 57419 call_connect x= prt >00000000e061831b is not connected=0A= >Mar 4 10:29:30 hostname kernel: [437417.116547] RPC: 57419 xprt_connect x= prt >00000000e061831b is not connected=0A= >Mar 4 10:30:31 hostname kernel: [437478.551090] RPC: 57419 xprt_connect_s= tatus: >connect attempt timed out=0A= >Mar 4 10:30:31 hostname kernel: [437478.552396] RPC: 57419 call_connect_s= tatus >(status -110)=0A= >Mar 4 10:30:31 hostname kernel: [437478.553417] RPC: 57419 call_timeout (= minor)=0A= >Mar 4 10:30:31 hostname kernel: [437478.554327] RPC: 57419 call_bind (sta= tus 0)=0A= >Mar 4 10:30:31 hostname kernel: [437478.555220] RPC: 57419 call_connect x= prt >00000000e061831b is not connected=0A= >Mar 4 10:30:31 hostname kernel: [437478.556254] RPC: 57419 xprt_connect x= prt >00000000e061831b is not connected=0A= Is it possible that the client is trying to (re)connect using the same clie= nt port#?=0A= I would normally expect the client to create a new TCP connection using a= =0A= different client port# and then retry the outstanding RPCs.=0A= --> Capturing packets when this happens would show us what is going on.=0A= =0A= If there is a problem on the FreeBSD end, it is most likely a broken=0A= network device driver.=0A= --> Try disabling TSO , LRO.=0A= --> Try a different driver for the net hardware on the server.=0A= --> Try a different net chip on the server.=0A= If you can capture packets when (not after) the hang=0A= occurs, then you can look at them in wireshark and see=0A= what is actually happening. (Ideally on both client and=0A= server, to check that your network hasn't dropped anything.)=0A= --> I know, if the hangs aren't easily reproducible, this isn't=0A= easily done.=0A= --> Try a newer Linux kernel and see if the problem persists.=0A= The Linux folk will get more interested if you can reproduce=0A= the problem on 5.12. (Recent bakeathon testing of the 5.12=0A= kernel against the FreeBSD server did not find any issues.)=0A= =0A= Hopefully the network folk have some insight w.r.t. why=0A= the TCP connection is sitting in FIN_WAIT2.=0A= =0A= rick=0A= =0A= =0A= =0A= Jason Breitman=0A= =0A= =0A= =0A= =0A= =0A= =0A= _______________________________________________=0A= freebsd-net@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-net=0A= To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"=0A= =0A=