From owner-freebsd-net@freebsd.org Sun Apr 4 11:50:54 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 991575D4A3D for ; Sun, 4 Apr 2021 11:50:54 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2076.outbound.protection.outlook.com [40.107.94.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FCsYm6F8Hz4r2m; Sun, 4 Apr 2021 11:50:52 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lqs5OL0Po6blPKXEjwMPe0WLXeLtCbedRpIWPXFe21Y58kdaQH1Bhh3xeEUrYICVefS7+moHwIIQXuXRRQWUyZIJpBbay7YKiCD6qFLEawkx4UNXvTv2SGM7oQcgvz0woxz7SDoRQjAQOdwlMcm224iCJaNG3oXUWBcrlrz3eMSTS+CUzH0YgUMZiqi62lVORbFiB+EL8APXMKGdHJw+KWwygD6xwdcnGiZ3zkijG/ssKUZpjI/AR4gbZOilpWMMPV+JRXvIsxHkpU1XjQ324uc8mp6Ar5HR7/H1euDQn0RCHabrrtjnF3m3sy1BGif3K2btuCB2CqVA7oViVDr22Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WQ4lQq3MpjZWQdVaSWrbTGvFdB4K4yGOD8vrY/dYklY=; b=eR10rPVsMyulbgRWPucBtevwcsusyVXOOkG6nyrSM2egYQdxVyYkpneJA09tkS1IsYglpsbQsPj5vu5DIGLdDMgeUNzM0gAwaBr5xI3Vae5NVh38qfy1UcuDuJcKu10Xbk8jwpoiMzMuVbFyYg/6B4Md2Y7SOzKSE51l1gaPS12MTKfy+cB9GpPPnDnF2PTVq24RKxSGZOzQGgKE/jZ8RN2BPZIQbzj1uqqlUz7yHdiTOUBuBUmWeCeAadqIWIrvwbbYjXZZCBZnL+vAFw9mYbQMbPUWbv6awBX8dzAUUB3/MKTK+MfbJoz+O9TZTW4ffNE+OtdE8lSapc3upGYVNw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netapp.com; dmarc=pass action=none header.from=netapp.com; dkim=pass header.d=netapp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netapp.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WQ4lQq3MpjZWQdVaSWrbTGvFdB4K4yGOD8vrY/dYklY=; b=GvEt8WpZHnVHHgUo7SKMwNy+MN+nr+e2jVKTStumqp2l4hjpHEWiGm2zjZcNY0XACsF8M24GS6cw0lnxyknKAwSwI2wDnxhZjQsWdn8XEkhS78tUfoKPHL5Tye5+JX4My527teRJiyE6aMiwm80o0T10yhJXQeydV8KfUCwM2QSjGWBWrB1UL6nARTI4k6f8zO/29CVv5LHuGXcXX3vs1705fx0ed6EmqWBQFMaQ+YB9OYDh4iVGi3mHPNjTkvglKtk6hNQ2U5gRDIsrvIUIIRAJqbs5LuTKG4IscHnaCDBtDS9X3vj8DhLLtPZyTtQ/bbLAx6p33AdQW6PRs6qaRA== Received: from SN4PR0601MB3728.namprd06.prod.outlook.com (2603:10b6:803:51::24) by SA0PR06MB6810.namprd06.prod.outlook.com (2603:10b6:806:bc::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.28; Sun, 4 Apr 2021 11:50:48 +0000 Received: from SN4PR0601MB3728.namprd06.prod.outlook.com ([fe80::ccb:944d:e270:63ef]) by SN4PR0601MB3728.namprd06.prod.outlook.com ([fe80::ccb:944d:e270:63ef%6]) with mapi id 15.20.3999.032; Sun, 4 Apr 2021 11:50:48 +0000 From: "Scheffenegger, Richard" To: Rick Macklem , "tuexen@freebsd.org" CC: Youssef GHORBAL , "freebsd-net@freebsd.org" Subject: Re: NFS Mount Hangs Thread-Topic: NFS Mount Hangs Thread-Index: AQHXG1GB6agsoGWN0UqRoZFo/qoHTaqMDIkAgAL97ICACMXzgIAAsfOAgAfvbwCAAQ5PAIAAWDiAgAKBMZU= Date: Sun, 4 Apr 2021 11:50:47 +0000 Message-ID: References: <3750001D-3F1C-4D9A-A9D9-98BCA6CA65A4@tildenparkcapital.com> <33693DE3-7FF8-4FAB-9A75-75576B88A566@tildenparkcapital.com> <8E745920-1092-4312-B251-B49D11FE8028@pasteur.fr> , , In-Reply-To: Accept-Language: de-AT, en-US Content-Language: de-AT X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [2001:4bb8:10e:a0d5:dca1:b74c:eaaa:2a7d] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: a8985786-3aed-4083-50e2-08d8f75fe3ad x-ms-traffictypediagnostic: SA0PR06MB6810: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:397; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: eYJXnQlp/DgePNo1D5zBlDuwbnBay1JtSJu8sgR//qNJVVaZtN17ep66hDBp1XZFjr+LU2JnCpv6cAJU5efAdfVwaItc3UUDXsEW6xwUp7BX1L4KxlkNNT2+WNMSXibFG+TTUnDnHIM4GxI96dXJXtbwMpS/qR/aVELgnZS7ripWSZ4MIEc788wytnaFbGoEcK5Mi2fo0KrQfD+cTW+/S6nqo7gKJXNZ/KJrimqFyyrDf8jhPywB3R1u8fwLhog0YStoDR/EQ/9b5kDldRl0ypOjzHE1nt5HSYGiXx4Jz9zuFdOeknBI8FZ6daYMtQYpytRiPoOQvdjPueDMBABCETjFEDUGDZfxfaunMSe61K3FQyeCDSlYhmOQ+hhSlDH961nMqLuMs1VI6cvZQhl+227NqVtIRaghXr/ZBnV7sKZ1Jn/hVxfmlMOAYDsmkKoBnvjB+FjexXmUm0O1IhnulIf/fzt/oDkZNEYCIwgs9fSjzeOL4Aw4qXChrBZbZIuO+COYiArg3Vyyd1H3i8vvcTsXFkizdHwwdRaswoVU/LtSbqbOWJlOJIrgdq9dyejP/8o1y1X8k7818os4vrRQ/45c3A/eHDkazc6VEPu0UxuDOfSEER04usMzyrSBfTyxL7jq9Kpoccezha/hg9ZqKjD3pKAk30Y/zWVTyiOHtXsZG+G4dP69skgdhISkYhb57Lq6dL/kiufRpe5w5eBSTwENBpOQ1fs5HbajDSC7+XE= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN4PR0601MB3728.namprd06.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(376002)(396003)(366004)(136003)(346002)(39860400002)(53546011)(186003)(52536014)(9686003)(166002)(83380400001)(71200400001)(66574015)(7696005)(55016002)(478600001)(66946007)(316002)(8676002)(4326008)(966005)(64756008)(8936002)(66556008)(66446008)(66476007)(54906003)(3480700007)(110136005)(33656002)(91956017)(296002)(76116006)(30864003)(5660300002)(86362001)(2906002)(7116003)(6506007)(38100700001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?Windows-1252?Q?q9axFf6SESzv8DeqChlSghiveTxaQQLRXuwRMJCpmkxstDGuToUXOvUW?= =?Windows-1252?Q?DR7p3HyGbBcAiY8oe0A3aTarOjgK0qzddQMb295QXLWsfbQVcMyz4Ef4?= =?Windows-1252?Q?bI3K3QZBDQKiwJYu9RK3Yls1KmwH/YwtA7GOZEwMr0uwcYdyOJyYDgYW?= =?Windows-1252?Q?0Ni7120I4eypTmbHCVd6lABqnAMQ5kZCUIE2gnXKPLe7F9/xxfflYTY9?= =?Windows-1252?Q?2sdHCE2ApfaeVsvwTSenHlsVcg11SsS7hc0a1pYHmlPyX0lIeVwujtSY?= =?Windows-1252?Q?MYn+Lk1xr70C1eUz3gO8OyZAmjRawQT9yTJl0GgWlfwoXKcrrOFk4/hi?= =?Windows-1252?Q?tpCsf5Q9Y7Vs41472S3hkFdWEF4B0WG/JBi0YvX0n+Xp9zupolDiKbeP?= =?Windows-1252?Q?uBIZ6yyyElYKRQd0Iko2IQQD4M5o8XXc6MDi7BGZS1a45TkAKr/mZ7y2?= =?Windows-1252?Q?1bKccq2bci0Tj5IHd+YV3XPHa8SJcMJa6Fsu7v5XZw6FeOdRnjo+W4nB?= =?Windows-1252?Q?LFirWzGh6hPZZuP2676jNPzFlCyTpmV3lWJwlNlvrHDGfNJTexTdAJtL?= =?Windows-1252?Q?ZB6U66T4sdxPElLobhUI0dS51qWPAod69TbrUrp58cwLYyWGtE/RnifA?= =?Windows-1252?Q?0qeNMxrik+6turBW5746J1DSUT4dEEz9G80pCqfyA6r9ZSE2KDcLMVXl?= =?Windows-1252?Q?nw2B2cwzdlGWc52n5thGoXSkb+iESb0Qaickx+uUVWZ5Vmz/pKKkduAi?= =?Windows-1252?Q?ySSvT4SqOTbbWl2EFdXKosU63JdMn3GWEPXBuky3r8c9m8Ru7U289yS6?= =?Windows-1252?Q?aHSO1VnvTJi3416YcrilMzOjrwc+d0q/gYGa+iPUh9+5Wz1TVhCMpJl5?= =?Windows-1252?Q?6NBOZ0WfgCGpmZadY6WtThGks/QWLhKWxeI+8oMMKeOM2P88+QRoKb2o?= =?Windows-1252?Q?eijir3Yul1RBQ2MIpjYydsLmKchI6bYJmwJc0SaR7XWP0mA9AQB50pnG?= =?Windows-1252?Q?bUIV0LP2J5gcbjVx6D7f3t/wGTc++snqle+uJAqu7XGCpJacIuJ2ufd3?= =?Windows-1252?Q?Lp7BcjOQpTklor3t7+5GUkZF+ad38D7/0ikmO9jVYBkUFEC/MwoOyuIh?= =?Windows-1252?Q?pa9QQa+Hx1clJMrGqtU/jdX/DNiZ2JRYHmp0trl4PaA7Wd9FTkcs0T8C?= =?Windows-1252?Q?wefUMIq6OF0TczJAeFu5uyU+tNQMDI9UMGdx9368AiVZVNiEl/hMnPIt?= =?Windows-1252?Q?KqAozeAoSTstKgzklzRKqLEXgtsg/zpDJArqgRoQHPvkcQk9pC9T4ksb?= =?Windows-1252?Q?XZOfdkWLH6EZYyPWflyd2r9lziGtmrElu1ViNUaGdw9PPyGWJ4J/kZQw?= =?Windows-1252?Q?mdVv2S3otU5KDbkRkhLseQ5NVTMe+9qDv6Rp54VYQB2w0KvoNjWP0hCn?= =?Windows-1252?Q?fSIrsGQW+WDF0M5hFlmnVA9Qx12UYxIkDjrbkdaSYskToU34JPvpT7bm?= =?Windows-1252?Q?+h3Z/S5vLO4sZGXuIq4bJAwTdGMZvg=3D=3D?= x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: netapp.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN4PR0601MB3728.namprd06.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: a8985786-3aed-4083-50e2-08d8f75fe3ad X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Apr 2021 11:50:47.9879 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4b0911a0-929b-4715-944b-c03745165b3a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 3UMw0pmWPl+bmiSAm9/0VzVzrB7Eb6Oz+cRJUR+LSrDz0uDxDoyXWmZSD7fCvdJ0QvHJd+fKKI3xijBS8UZVkw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR06MB6810 X-Rspamd-Queue-Id: 4FCsYm6F8Hz4r2m X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=netapp.com header.s=selector1 header.b=GvEt8WpZ; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=netapp.com; spf=pass (mx1.freebsd.org: domain of Richard.Scheffenegger@netapp.com designates 40.107.94.76 as permitted sender) smtp.mailfrom=Richard.Scheffenegger@netapp.com X-Spamd-Result: default: False [-6.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; HAS_XOIP(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[netapp.com:+]; DMARC_POLICY_ALLOW(-0.50)[netapp.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.94.76:from]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[netapp.com:s=selector1]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; SPAMHAUS_ZRD(0.00)[40.107.94.76:from:127.0.2.255]; DWL_DNSWL_LOW(-1.00)[netapp.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[40.107.94.76:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.94.76:from]; MAILMAN_DEST(0.00)[freebsd-net] Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Apr 2021 11:50:54 -0000 For what it=91s worth, suse found two bugs in the linux nfconntrack (statef= ul firewall), and pfifo-fast scheduler, which could conspire to make tcp se= ssions hang forever. One is a missed updaten when the c=F6ient is not using the noresvport moint= option, which makes tje firewall think rsts are illegal (and drop them); The fast scheduler can run into an issue if only a single packet should be = forwarded (note that this is not the default scheduler, but often recommend= ed for perf, as it runs lockless and lower cpu cost that pfq (default). If = no other/additional packet pushes out that last packet of a flow, it can be= come stuck forever... I can try getting the relevant bug info next week... ________________________________ Von: owner-freebsd-net@freebsd.org im Auftr= ag von Rick Macklem Gesendet: Friday, April 2, 2021 11:31:01 PM An: tuexen@freebsd.org Cc: Youssef GHORBAL ; freebsd-net@freebsd.org <= freebsd-net@freebsd.org> Betreff: Re: NFS Mount Hangs NetApp Security WARNING: This is an external email. Do not click links or o= pen attachments unless you recognize the sender and know the content is saf= e. tuexen@freebsd.org wrote: >> On 2. Apr 2021, at 02:07, Rick Macklem wrote: >> >> I hope you don't mind a top post... >> I've been testing network partitioning between the only Linux client >> I have (5.2 kernel) and a FreeBSD server with the xprtdied.patch >> (does soshutdown(..SHUT_WR) when it knows the socket is broken) >> applied to it. >> >> I'm not enough of a TCP guy to know if this is useful, but here's what >> I see... >> >> While partitioned: >> On the FreeBSD server end, the socket either goes to CLOSED during >> the network partition or stays ESTABLISHED. >If it goes to CLOSED you called shutdown(, SHUT_WR) and the peer also >sent a FIN, but you never called close() on the socket. >If the socket stays in ESTABLISHED, there is no communication ongoing, >I guess, and therefore the server does not even detect that the peer >is not reachable. >> On the Linux end, the socket seems to remain ESTABLISHED for a >> little while, and then disappears. >So how does Linux detect the peer is not reachable? Well, here's what I see in a packet capture in the Linux client once I partition it (just unplug the net cable): - lots of retransmits of the same segment (with ACK) for 54sec - then only ARP queries Once I plug the net cable back in: - ARP works - one more retransmit of the same segement - receives RST from FreeBSD ** So, is this now a "new" TCP connection, despite using the same port#. --> It matters for NFS, since "new connection" implies "must retry all outstanding RPCs". - sends SYN - receives SYN, ACK from FreeBSD --> connection starts working again Always uses same port#. On the FreeBSD server end: - receives the last retransmit of the segment (with ACK) - sends RST - receives SYN - sends SYN, ACK I thought that there was no RST in the capture I looked at yesterday, so I'm not sure if FreeBSD always sends an RST, but the Linux client behaviour was the same. (Sent a SYN, etc). The socket disappears from the Linux "netstat -a" and I suspect that happens after about 54sec, but I am not sure about the timing. >> >> After unpartitioning: >> On the FreeBSD server end, you get another socket showing up at >> the same port# >> Active Internet connections (including servers) >> Proto Recv-Q Send-Q Local Address Foreign Address (state= ) >> tcp4 0 0 nfsv4-new3.nfsd nfsv4-linux.678 ESTABL= ISHED >> tcp4 0 0 nfsv4-new3.nfsd nfsv4-linux.678 CLOSED >> >> The Linux client shows the same connection ESTABLISHED. But disappears from "netstat -a" for a while during the partitioning. >> (The mount sometimes reports an error. I haven't looked at packet >> traces to see if it retries RPCs or why the errors occur.) I have now done so, as above. >> --> However I never get hangs. >> Sometimes it goes to SYN_SENT for a while and the FreeBSD server >> shows FIN_WAIT_1, but then both ends go to ESTABLISHED and the >> mount starts working again. >> >> The most obvious thing is that the Linux client always keeps using >> the same port#. (The FreeBSD client will use a different port# when >> it does a TCP reconnect after no response from the NFS server for >> a little while.) >> >> What do those TCP conversant think? >I guess you are you are never calling close() on the socket, for with >the connection state is CLOSED. Ok, that makes sense. For this case the Linux client has not done a BindConnectionToSession to re-assign the back channel. I'll have to bug them about this. However, I'll bet they'll answer that I have to tell them the back channel needs re-assignment or something like that. I am pretty certain they are broken, in that the client needs to retry all outstanding RPCs. For others, here's the long winded version of this that I just put on the phabricator review: In the server side kernel RPC, the socket (struct socket *) is in a structure called SVCXPRT (normally pointed to by "xprt"). These structures a ref counted and the soclose() is done when the ref. cnt goes to zero. My understanding is that "struct socket *" is free'd by soclose() so this cannot be done before the xprt ref. cnt goes to zero. For NFSv4.1/4.2 there is something called a back channel which means that a "xprt" is used for server->client RPCs, although the TCP connection is established by the client to the server. --> This back channel holds a ref cnt on "xprt" until the client re-assigns it to a different TCP connection via an operation called BindConnectionToSession and the Linux client is not doing this soon enough, it appears. So, the soclose() is delayed, which is why I think the TCP connection gets stuck in CLOSE_WAIT and that is why I've added the soshutdown(..SHUT_WR) calls, which can happen before the client gets around to re-assigning the back channel. Thanks for your help with this Michael, rick Best regards Michael > > rick > ps: I can capture packets while doing this, if anyone has a use > for them. > > > > > > > ________________________________________ > From: owner-freebsd-net@freebsd.org on be= half of Youssef GHORBAL > Sent: Saturday, March 27, 2021 6:57 PM > To: Jason Breitman > Cc: Rick Macklem; freebsd-net@freebsd.org > Subject: Re: NFS Mount Hangs > > CAUTION: This email originated from outside of the University of Guelph. = Do not click links or open attachments unless you recognize the sender and = know the content is safe. If in doubt, forward suspicious emails to IThelp@= uoguelph.ca > > > > > On 27 Mar 2021, at 13:20, Jason Breitman > wrote: > > The issue happened again so we can say that disabling TSO and LRO on the = NIC did not resolve this issue. > # ifconfig lagg0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso > # ifconfig lagg0 > lagg0: flags=3D8943 metri= c 0 mtu 1500 > options=3D8100b8 > > We can also say that the sysctl settings did not resolve this issue. > > # sysctl net.inet.tcp.fast_finwait2_recycle=3D1 > net.inet.tcp.fast_finwait2_recycle: 0 -> 1 > > # sysctl net.inet.tcp.finwait2_timeout=3D1000 > net.inet.tcp.finwait2_timeout: 60000 -> 1000 > > I don=92t think those will do anything in your case since the FIN_WAIT2 a= re on the client side and those sysctls are for BSD. > By the way it seems that Linux recycles automatically TCP sessions in FIN= _WAIT2 after 60 seconds (sysctl net.ipv4.tcp_fin_timeout) > > tcp_fin_timeout (integer; default: 60; since Linux 2.2) > This specifies how many seconds to wait for a final FIN > packet before the socket is forcibly closed. This is > strictly a violation of the TCP specification, but > required to prevent denial-of-service attacks. In Linux > 2.2, the default value was 180. > > So I don=92t get why it stucks in the FIN_WAIT2 state anyway. > > You really need to have a packet capture during the outage (client and se= rver side) so you=92ll get over the wire chat and start speculating from th= ere. > No need to capture the beginning of the outage for now. All you have to d= o, is run a tcpdump for 10 minutes or so when you notice a client stuck. > > * I have not rebooted the NFS Server nor have I restarted nfsd, but do no= t believe that is required as these settings are at the TCP level and I wou= ld expect new sessions to use the updated settings. > > The issue occurred after 5 days following a reboot of the client machines= . > I ran the capture information again to make use of the situation. > > #!/bin/sh > > while true > do > /bin/date >> /tmp/nfs-hang.log > /bin/ps axHl | grep nfsd | grep -v grep >> /tmp/nfs-hang.log > /usr/bin/procstat -kk 2947 >> /tmp/nfs-hang.log > /usr/bin/procstat -kk 2944 >> /tmp/nfs-hang.log > /bin/sleep 60 > done > > > On the NFS Server > Active Internet connections (including servers) > Proto Recv-Q Send-Q Local Address Foreign Address (state) > tcp4 0 0 NFS.Server.IP.X.2049 NFS.Client.IP.X.48286 C= LOSE_WAIT > > On the NFS Client > tcp 0 0 NFS.Client.IP.X:48286 NFS.Server.IP.X:2049 = FIN_WAIT2 > > > > You had also asked for the output below. > > # nfsstat -E -s > BackChannelCtBindConnToSes > 0 0 > > # sysctl vfs.nfsd.request_space_throttle_count > vfs.nfsd.request_space_throttle_count: 0 > > I see that you are testing a patch and I look forward to seeing the resul= ts. > > > Jason Breitman > > > On Mar 21, 2021, at 6:21 PM, Rick Macklem > wrote: > > Youssef GHORBAL > wrote: >> Hi Jason, >> >>> On 17 Mar 2021, at 18:17, Jason Breitman > wrote: >>> >>> Please review the details below and let me know if there is a setting t= hat I should apply to my FreeBSD NFS Server or if there is a bug fix that I= can apply to resolve my issue. >>> I shared this information with the linux-nfs mailing list and they beli= eve the issue is on the server side. >>> >>> Issue >>> NFSv4 mounts periodically hang on the NFS Client. >>> >>> During this time, it is possible to manually mount from another NFS Ser= ver on the NFS Client having issues. >>> Also, other NFS Clients are successfully mounting from the NFS Server i= n question. >>> Rebooting the NFS Client appears to be the only solution. >> >> I had experienced a similar weird situation with periodically stuck Linu= x NFS clients >mounting Isilon NFS servers (Isilon is FreeBSD based but the= y seem to have there >own nfsd) > Yes, my understanding is that Isilon uses a proprietary user space nfsd a= nd > not the kernel based RPC and nfsd in FreeBSD. > >> We=92ve had better luck and we did manage to have packet captures on bot= h sides >during the issue. The gist of it goes like follows: >> >> - Data flows correctly between SERVER and the CLIENT >> - At some point SERVER starts decreasing it's TCP Receive Window until i= t reachs 0 >> - The client (eager to send data) can only ack data sent by SERVER. >> - When SERVER was done sending data, the client starts sending TCP Windo= w >Probes hoping that the TCP Window opens again so he can flush its buffer= s. >> - SERVER responds with a TCP Zero Window to those probes. > Having the window size drop to zero is not necessarily incorrect. > If the server is overloaded (has a backlog of NFS requests), it can stop = doing > soreceive() on the socket (so the socket rcv buffer can fill up and the T= CP window > closes). This results in "backpressure" to stop the NFS client from flood= ing the > NFS server with requests. > --> However, once the backlog is handled, the nfsd should start to sorece= ive() > again and this shouls cause the window to open back up. > --> Maybe this is broken in the socket/TCP code. I quickly got lost in > tcp_output() when it decides what to do about the rcvwin. > >> - After 6 minutes (the NFS server default Idle timeout) SERVER racefully= closes the >TCP connection sending a FIN Packet (and still a TCP Window 0) > This probably does not happen for Jason's case, since the 6minute timeout > is disabled when the TCP connection is assigned as a backchannel (most li= kely > the case for NFSv4.1). > >> - CLIENT ACK that FIN. >> - SERVER goes in FIN_WAIT_2 state >> - CLIENT closes its half part part of the socket and goes in LAST_ACK st= ate. >> - FIN is never sent by the client since there still data in its SendQ an= d receiver TCP >Window is still 0. At this stage the client starts sending = TCP Window Probes again >and again hoping that the server opens its TCP Win= dow so it can flush it's buffers >and terminate its side of the socket. >> - SERVER keeps responding with a TCP Zero Window to those probes. >> =3D> The last two steps goes on and on for hours/days freezing the NFS m= ount bound >to that TCP session. >> >> If we had a situation where CLIENT was responsible for closing the TCP W= indow (and >initiating the TCP FIN first) and server wanting to send data w= e=92ll end up in the same >state as you I think. >> >> We=92ve never had the root cause of why the SERVER decided to close the = TCP >Window and no more acccept data, the fix on the Isilon part was to rec= ycle more >aggressively the FIN_WAIT_2 sockets (net.inet.tcp.fast_finwait2_= recycle=3D1 & >net.inet.tcp.finwait2_timeout=3D5000). Once the socket recyc= led and at the next >occurence of CLIENT TCP Window probe, SERVER sends a R= ST, triggering the >teardown of the session on the client side, a new TCP h= andchake, etc and traffic >flows again (NFS starts responding) >> >> To avoid rebooting the client (and before the aggressive FIN_WAIT_2 was = >implemented on the Isilon side) we=92ve added a check script on the client= that detects >LAST_ACK sockets on the client and through iptables rule enf= orces a TCP RST, >Something like: -A OUTPUT -p tcp -d $nfs_server_addr --sp= ort $local_port -j REJECT >--reject-with tcp-reset (the script removes this= iptables rule as soon as the LAST_ACK >disappears) >> >> The bottom line would be to have a packet capture during the outage (cli= ent and/or >server side), it will show you at least the shape of the TCP ex= change when NFS is >stuck. > Interesting story and good work w.r.t. sluething, Youssef, thanks. > > I looked at Jason's log and it shows everything is ok w.r.t the nfsd thre= ads. > (They're just waiting for RPC requests.) > However, I do now think I know why the soclose() does not happen. > When the TCP connection is assigned as a backchannel, that takes a refere= nce > cnt on the structure. This refcnt won't be released until the connection = is > replaced by a BindConnectiotoSession operation from the client. But that = won't > happen until the client creates a new TCP connection. > --> No refcnt release-->no refcnt of 0-->no soclose(). > > I've created the attached patch (completely different from the previous o= ne) > that adds soshutdown(SHUT_WR) calls in the three places where the TCP > connection is going away. This seems to get it past CLOSE_WAIT without a > soclose(). > --> I know you are not comfortable with patching your server, but I do th= ink > this change will get the socket shutdown to complete. > > There are a couple more things you can check on the server... > # nfsstat -E -s > --> Look for the count under "BindConnToSes". > --> If non-zero, backchannels have been assigned > # sysctl -a | fgrep request_space_throttle_count > --> If non-zero, the server has been overloaded at some point. > > I think the attached patch might work around the problem. > The code that should open up the receive window needs to be checked. > I am also looking at enabling the 6minute timeout when a backchannel is > assigned. > > rick > > Youssef > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://urldefense.com/v3/__https://lists.freebsd.org/mailman/listinfo/fr= eebsd-net__;!!JFdNOqOXpB6UZW0!_c2MFNbir59GXudWPVdE5bNBm-qqjXeBuJ2UEmFv5OZci= Lj4ObR_drJNv5yryaERfIbhKR2d$ > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"