From owner-freebsd-current@freebsd.org Thu Apr 15 21:14:36 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 04A115D894C for ; Thu, 15 Apr 2021 21:14:36 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) Received: from NAM04-SN1-obe.outbound.protection.outlook.com (mail-eopbgr700072.outbound.protection.outlook.com [40.107.70.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FLsY66RFmz4sks; Thu, 15 Apr 2021 21:14:34 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BHiT2D8TETF9pEWqT+kTSvzB2WvEb+WZMA46B/c7AMffsNtDA8GN5Ol9uUcUjBjDKUYvqIK3WWVH4WiblpRFy8+dfwWlSsSZ0A4SilB7UXKtfHny7bJpDLGk+xGO7s2MFGREuVT8Qi6dHPr03p/NytHU2VXEp3voP9ZQ9P33kESVNUySdADKVLJzKX+fumGfnpqzv6IIMUoD8xyTAbtllP9g8dPtlSgXeqv5UtAs1hZeKmxuPayVgvGwJ496XKB15Aq+yFqQclRLEHxssVMk8PCaZEhwpb8ZQVywLBuMDjkzIM3g+m07JO/AvP8IflMR/n94746l0Gypi6Xl5cg3lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gJyuKv75jY2kSBV2IUNhuyMOTVq6LUM7T41l1Oz5L9I=; b=QwODETedH99/mdCYaXVjEl2OtOQEs6pNqPqeD3C/HDOzYWGmUWR7oNX0S6kobVTxFIPzf9SGunD/vrWwtOHc85G9y/JY53uMEzInZl8LtN+LwBRQrM+b+D2Unz+s+cjthLEZrV0F+2+ka3PIzV1fXgGZBmotQATxPhdN7Nun+8hNY8G3BcEPmn13WAnCZAb2f3evpdxkDIS0d5t7NZXamapXRYO2HVW+q9QdoFh7T3CJeuZQImLK1gcLXIsjjethdjqDMm0OMnHS/k2ymyIzZyQZcQ5RdW0PHi9tQvJCJLsX45lGtTQwBYKTX206Ewx3OR0+nJlgoeTSoRbjkQn8ow== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netapp.com; dmarc=pass action=none header.from=netapp.com; dkim=pass header.d=netapp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netapp.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gJyuKv75jY2kSBV2IUNhuyMOTVq6LUM7T41l1Oz5L9I=; b=UnlRBoV8Jc95p+QkqVrzYP1tC37KDicyCn+bh2zu2yH7bN4zoOFCaBFDIkgOsnZyZnHw8gL/RFoaJL1zhomEnPYgEs9U458dshXo5mjVIZ9ofm4hUAMe9po1f34dsXGm9Rvt/m+CNB9VUWQl970cqif0gPLwW8sV5Xq6YNYzCYu2nbFk8XdwRUyOZuB7XDn10wbmfBUauyrw2b94SRMBgBe3qo+CUb5INOjQDz+eFHwyL9b3GGrKJUwD8Qbu72uwBy+0ULbwBTTQJdWmHgAxiGUp1KeynoAfbOPfgCdoxW6lDYOFPU6TtDW+5/XswWKk2e606uhB+n7I4P8hexd6kw== Received: from SN4PR0601MB3728.namprd06.prod.outlook.com (2603:10b6:803:51::24) by SA0PR06MB6906.namprd06.prod.outlook.com (2603:10b6:806:b2::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4020.18; Thu, 15 Apr 2021 21:14:33 +0000 Received: from SN4PR0601MB3728.namprd06.prod.outlook.com ([fe80::ccb:944d:e270:63ef]) by SN4PR0601MB3728.namprd06.prod.outlook.com ([fe80::ccb:944d:e270:63ef%6]) with mapi id 15.20.4042.018; Thu, 15 Apr 2021 21:14:33 +0000 From: "Scheffenegger, Richard" To: Rick Macklem , Allan Jude , "freebsd-current@freebsd.org" CC: Richard Scheffenegger , Juraj Lutter Subject: AW: NFS issues since upgrading to 13-RELEASE Thread-Topic: NFS issues since upgrading to 13-RELEASE Thread-Index: AQHXMhVsKLCb5bI16E6Ydj5qgkLmjKq16AQAgAAfNj2AAAnjjoAAAZxg Date: Thu, 15 Apr 2021 21:14:32 +0000 Message-ID: References: <902a3c81-2ce8-49c0-b163-5ffa4b90afe5@www.fastmail.com>, , In-Reply-To: Accept-Language: de-AT, en-US Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [185.236.167.136] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 1ebd77e1-5073-4312-ce9d-08d90053771b x-ms-traffictypediagnostic: SA0PR06MB6906: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6430; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: uSTdmzhnHp2YSbr9vwPQBbZuvsK6UF2x6qRd0kbT3Y9slE/PwAe7QNUl3fs9FZVCWVrJcecFid3cbRjDopAaJYU+h9isWYAKa6wEGDJ/2EQq0bPpZPi8ek9V93WoA34+zqgg5BvygSjrwho88N4cqJvvXc6LVafqCW7UG2HF+k6KdJnp++GYw0bnMgtg95GB9PU8F/kRSKculcm5VAe6XXiI8CaosjiwH8xtDcVCXin+5149qLzRtbhCp6ruMF0IpT8Scip1tSvUzTGZwwxQ6m5mccy0RqteMFE80n1CN+r3NFCu3cY2GeXo5Zn+U/yyB/x/g0D/abrpCVjDE7MtbrOVnyySUWhEhjh8aXe+mb0eMCjnGa1ugsKJMNC+plHXvNh4UHu1tPt6ihIZKRgJvonorRU8V8fD7KYTuEBDHSH/RjZDBwfBctwmQzPtNRXsyCjFeSkDLBmkKKYpX9IE2HcJJLgG4o03bUmMk5HC3j6gzXQy46ELZMTgLrmOp9IFf4DG/x+/Fx3rCPiwyiVo0ozSHvF6hf7Ec5A47f39DovXU8oOqLyy3PbVv4c9gR5KXJ8vJ38e1KkJ86+ao8fmjJr7D8nvKd5hmtf9ogGgEaK2eMKBNcdCQoVaH1qwmk9NHpPUtYWoNGYaTrpW9ztr3dVKEVZkN447UW4jzfcOKhirWvbfdAe6UvOs+NM6ltu9 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN4PR0601MB3728.namprd06.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(346002)(39860400002)(366004)(66476007)(5660300002)(316002)(71200400001)(76116006)(7696005)(110136005)(55016002)(2906002)(6506007)(66574015)(26005)(83380400001)(9686003)(66946007)(966005)(54906003)(64756008)(4326008)(66556008)(296002)(478600001)(66446008)(86362001)(122000001)(8936002)(8676002)(33656002)(186003)(38100700002)(52536014); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?bqXFg958LrXHh5RlDiYtW6i8Wg62B332eFt/lvO11+uenTmxIfvOFDNquG?= =?iso-8859-1?Q?uDNecrKBuertvAZzlSeGug8aWT2ls0XtbMAOTIQ1hdwFdBjdu6EsCn0vKn?= =?iso-8859-1?Q?tIpzMKtPDqQUgPlHBnuYOhJSEOfGJC18IlcO0+Saq89SghEFGrEHPIkGZ7?= =?iso-8859-1?Q?O6IUgzgk6e7Lz878O1WtmXnPXCfalVxxm4yZj3ShrrKx5CIcKn6Rzt1sYQ?= =?iso-8859-1?Q?O1xJ/5YAApnNP5CdlEAvJqVbkcmVl7Z7UcYnNmkDYp3+/1K7Nhl34V3RsH?= =?iso-8859-1?Q?YDU4A9IMIpv69svMlDM6UzQDxVGRG3nKTT9ebYv7Ffa4tFgPtrM14iCr6+?= =?iso-8859-1?Q?VI7pvYgAE6efv4w6D7x3NHxGF4ULkqKz6di5zMhzyHzUOEoTiyYuLo7hnG?= =?iso-8859-1?Q?O2pTHtgCDJtgwxYA3cZ8mJc2wu7CpETfyP8kOWyJu/evTlVMY7YRxC/CLH?= =?iso-8859-1?Q?09cGiDNUbI0QpsOb6erivkW3HCMH9kAOCaaVTSBJQRRAMRYFNn0FYf3HgZ?= =?iso-8859-1?Q?4DOeYQz4d4WQXC5WOZptQ8ZM55g2EAEp2lDdlo0sudKGAL0Sz9GYjnLOg7?= =?iso-8859-1?Q?GPmchdyt5OMx9pTbLbFTirrosbK5kvFRKdR5aL7Ef18wSefM6sthiaPrmI?= =?iso-8859-1?Q?yR7kctr74sH3mr9yDtdtjWkZUHWVDtQXZZ1N825WegRORZrJThv1ykEKii?= =?iso-8859-1?Q?bwtAgvUN8x/Gpr3pjVhssvzvvZaDemVNm3gEueC1x/l45/iMxJpCxadlUl?= =?iso-8859-1?Q?/V4WgAl35zUCc1YnH2lR8o+vcClkqv3pB1tiVGq7K6dwfrdm6xTAipMjzr?= =?iso-8859-1?Q?oHlRVXghj9czXXtQjMQDK9Krh3TRyGMYduGbzx/BZvZ+BLa7/mCY6aWW8G?= =?iso-8859-1?Q?wjjfVYIOBymo5D3KpTNxpGTi540OPOxniYWT6bNJ5mw5e/TPCHY2OpVs+x?= =?iso-8859-1?Q?79xaAXBIdWEK2avQcbH1AFHB7ebF72OnNbrW40OyisigO2RfYkWXtvUJFG?= =?iso-8859-1?Q?rEbsyUwo48gtT/UPtiGEQIqE7s7Vjo10ujKoHDk9nPeT29l33e1UCrd1JF?= =?iso-8859-1?Q?IGlaa3g+9kxyqKhuNgt9Jf7TSDv1rz3CLHrOgu/C3ktc0c6Yrw3eWrfOSx?= =?iso-8859-1?Q?oGYXcoRvoyk9DPbMOLqZQUW6sIkTl4xe2Hes9/9h0Mb9GIiGzEYMwICNeb?= =?iso-8859-1?Q?8L0XPBRPXWy3xDBTfFKxl4znO+rJEsQ1UUiKniCyE4YwZ9/7FtFTEOjSC5?= =?iso-8859-1?Q?73wUVYgbJQtbppBOfkmER2uj07oxlL1pUK/08do9gBeepYBL00RspXdf3H?= =?iso-8859-1?Q?Ih+xFTDfu/6n8qU3Dh4oqS9bDkmz3ZEq0FOQ3OuYRz/I35SXPE8XgGx+Lq?= =?iso-8859-1?Q?7eppwv6WIs?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: netapp.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN4PR0601MB3728.namprd06.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1ebd77e1-5073-4312-ce9d-08d90053771b X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Apr 2021 21:14:32.8623 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4b0911a0-929b-4715-944b-c03745165b3a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: YASw2fFQAJ+Vy1vCL0ajExsTRQVTM7aU5qMDrKGshmriBBL5wQp44LjyuLkg5Q9TIHS/7vca/nYTMMBxxB+CQQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR06MB6906 X-Rspamd-Queue-Id: 4FLsY66RFmz4sks X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=netapp.com header.s=selector1 header.b=UnlRBoV8; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=netapp.com; spf=pass (mx1.freebsd.org: domain of Richard.Scheffenegger@netapp.com designates 40.107.70.72 as permitted sender) smtp.mailfrom=Richard.Scheffenegger@netapp.com X-Spamd-Result: default: False [-6.10 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; HAS_XOIP(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[netapp.com:+]; DMARC_POLICY_ALLOW(-0.50)[netapp.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_IN_DNSWL_LOW(-0.10)[40.107.70.72:from]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.70.72:from]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[netapp.com:s=selector1]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[40.107.70.72:from:127.0.2.255]; DWL_DNSWL_LOW(-1.00)[netapp.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.70.72:from]; MAILMAN_DEST(0.00)[freebsd-current] X-Mailman-Approved-At: Fri, 16 Apr 2021 10:33:07 +0000 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Apr 2021 21:14:36 -0000 FWIW: r367492 fixes an issue around "premature" transmission of an ACK due to the= incoming segment only been partially processed at the time - related to in= -kernel TCP consumers which use socket upcalls. Rick mentioned, that the NFS server (one in-kernel TCP user) has stringent = requirements on the state of the socket during the upcall, thus D29690 is r= etaining the lock on the socket buffer until TCP processing is finalized an= d the upcall can be done without running any risk for transmitting outdated= information back to the other end. However, I have no proper way to verify/validate this interaction. My ask would be to test the behavior with D29690 first - but if similar han= gs keep reoccurring, then revert r367492 (which will also mean more severe = surgery on the TCP processing flow). Thanks. Richard Scheffenegger -----Urspr=FCngliche Nachricht----- Von: Rick Macklem =20 Gesendet: Donnerstag, 15. April 2021 23:05 An: Allan Jude ; freebsd-current@freebsd.org Cc: Richard Scheffenegger ; Juraj Lutter Betreff: Re: NFS issues since upgrading to 13-RELEASE NetApp Security WARNING: This is an external email. Do not click links or o= pen attachments unless you recognize the sender and know the content is saf= e. I wrote: [stuff snipped] >- Alternately you can try rscheff@'s alternate proposed patch that is=20 >at > https://reviews.freebsd.og/D29690. Oops, that's https:/reviews.freebsd.org/D29690 rick I have not yet had time to test this one, but since I cannot reproduce th= e hang, I can only do testing of it to see that it is "no worse" than reverting r367492= for my setup. Please let us know which you choose and whether or not it fixes your proble= m. >> Any pointers for troubleshooting this? I've been looking through vmstat,= gstat, top, etc. when the problem occurs, but I haven't been able to pinpo= int the issue. I can get pcap, but it would be from the hosts, because I do= n't have a 10G tap or managed switch. >> > >run `nfsstat -d 1` and try to capture a few lines from before, during,=20 >and after the stall, and that may provide some insight. > >Specifically, does the queue length grow, suggesting it is waiting on=20 >the I/O subsystem, or does it just stop getting traffic all together. If the revert of r367492 does not fix the problem, monitor the TCP connecti= on(s) via "netstat -a" and, if possible, capture packets via tcpdump -s 0 -= w hang.pcap host or similar, run on the server. Ideally the tcpdump would be started before the "hang" occurs, but running= one while the hang is occurring (until after it recovers) could also be us= eful. Thanks for reporting this, rick -- Allan Jude _______________________________________________ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/= listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"