From owner-freebsd-net@freebsd.org Sat Apr 10 14:40:28 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C179A5CE345 for ; Sat, 10 Apr 2021 14:40:28 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) Received: from NAM04-CO1-obe.outbound.protection.outlook.com (mail-co1nam04on0601.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe4d::601]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FHd2g3fF9z3Pty; Sat, 10 Apr 2021 14:40:27 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Hyqk5JSCh0cs5ZRQmoFgtGFTX693OddMeJxWkNqAXvbhdLWlWwCUZ+hGN6bt82uBecxF1HQAQmOsfYJRuB7Fd7Fy1FYCvt+D0aRXMjBKMZNVuwtoD6rLULdoGb6OB4DIacZzHfNyL5/IhVcUPbuWjshZjNEjcOl5Ey9e82zf68TkVu+iyCJsXMUFyUj9YW7+wmoMV6+yNmU8N8cgf+4K4OsIDbrcr6Be1qC+1eabR4zYrw72JqulmgfD5FNz14b1+VARoGA5DwzTKok1Rh54zGTZkT4EJxDuSX6Pdvk4Eyf+k1HTwO9/z0+ebICEjvc54Ey8PejOvMcLK2WzCxGivg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lZ0UcAxE56ydXfA0pu9KENj17Z8lLfD2F8rzkTgMF0Q=; b=d3wfJq+fQc6aW+xvDyUu0iDHuJmb0jVvqZIlOW4MMwlfNvAHZhv/lo+BD5j/kymllxJ9ib0E+1EvvYvDZt/Vv6k6b2y3TPK5PGkvD1omjLrKDG7WFbIaK9vH5d6+6vGRQRNMyxzGsV7Y6Oqv4P6U6pPkN+VcYgFv31XohW0u8yRxlo8mATWZZjCsgMg6reFcDxpoTHgzmh8lvmAcZWUZiTZLA4BZ1zA1so4OlAAds0PRpu/vk3ky8aYSU0lYRhKDHQ4Kjh7HyHtvbWbuqR3qkssZbYZ8HG18dCOgUMdEwpozGHKpJWgwoOSnO3MHUy+hOkep8ASE+J5bK4jVlEvXGw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netapp.com; dmarc=pass action=none header.from=netapp.com; dkim=pass header.d=netapp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netapp.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lZ0UcAxE56ydXfA0pu9KENj17Z8lLfD2F8rzkTgMF0Q=; b=RxDPGAwKx3eZHgAWKK+Lk24fSFD2t9o/OFJehFYtf6wFTHtJBnMxNVCLKxG4QHaNufuIyLqi22+MNjJOwnjSjHcGYQU32HoF49XRL+19UqtGeCftzvWOE/T4WYBq3mm/U0mAHxw6s3wPghqnk97V0i76xpuZNZWfZXK2jiMfOQ9VSrFR7ab0g4TUqN8sp8B4hgOCLqU+NtGf/UboUNoGr6b2wQ7gEuHVnzXpQYFbUGDGL2s/cpLdi91BUSnWZdvqzWnnf/qmC6iPC+Zp0PYXY+MtQJ3KC6arI9ITX7v3zWGZD4x6tdktHbgJxh1ZFjYn1i58FrPtqs4TVtqWCO+VYw== Received: from SN4PR0601MB3728.namprd06.prod.outlook.com (2603:10b6:803:51::24) by SN6PR06MB4621.namprd06.prod.outlook.com (2603:10b6:805:91::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.28; Sat, 10 Apr 2021 14:40:24 +0000 Received: from SN4PR0601MB3728.namprd06.prod.outlook.com ([fe80::ccb:944d:e270:63ef]) by SN4PR0601MB3728.namprd06.prod.outlook.com ([fe80::ccb:944d:e270:63ef%6]) with mapi id 15.20.4020.021; Sat, 10 Apr 2021 14:40:24 +0000 From: "Scheffenegger, Richard" To: "tuexen@freebsd.org" CC: Rick Macklem , Youssef GHORBAL , "freebsd-net@freebsd.org" Subject: Re: NFS Mount Hangs Thread-Topic: NFS Mount Hangs Thread-Index: AQHXG1GB6agsoGWN0UqRoZFo/qoHTaqMDIkAgAL97ICACMXzgIAAsfOAgAfvbwCAAQ5PAIAAWDiAgAKBMZWAAD3WgIAAFNIAgAA/e4CAABvaAIAAEe2AgAEE0ACAAJCpAIAAgu0AgAXcwwCAAH0H4IAARSaAgAAmg3g= Date: Sat, 10 Apr 2021 14:40:24 +0000 Message-ID: References: <3750001D-3F1C-4D9A-A9D9-98BCA6CA65A4@tildenparkcapital.com> <33693DE3-7FF8-4FAB-9A75-75576B88A566@tildenparkcapital.com> <8E745920-1092-4312-B251-B49D11FE8028@pasteur.fr> <765CE1CD-6AAB-4BEF-97C6-C2A1F0FF4AC5@freebsd.org> <2B189169-C0C9-4DE6-A01A-BE916F10BABA@freebsd.org> , <077ECE2B-A84C-440D-AAAB-00293C841F14@freebsd.org> In-Reply-To: <077ECE2B-A84C-440D-AAAB-00293C841F14@freebsd.org> Accept-Language: de-AT, en-US Content-Language: de-AT X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [2001:4bb8:11a:b40f:c82c:3a7f:c649:8f03] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 381f128e-46ad-49e7-189b-08d8fc2e935a x-ms-traffictypediagnostic: SN6PR06MB4621: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:4502; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: dM7QqOJle2mEsHw497MEFMQw/2UczI7JNYijuxagLR+2+9t9Uay9I6GX8W0erIkAkUjhYaSI/9uDzp0deLyHurgQtTJcEuw8p8IlEnGL73M9nbD/TQLnBTb73nJi74Hpy5aFhC/dGQYbfythj8vG8pnkV0s0KWfOH1WyGi4PAXCVprhKlgX4zVmWZnpN3v3lDxVni2dN3a/V6juV/d+jcRRmskjRqx0Uuj5RfyH179Ghvv6qBmInzYMB0B6jQuoFhWX1i4mmVmYXP2WcKk/AxI/ZIurrRM92dl6quI8P1lMsMGgynuqYvejsjXrPyFSpqEpkTwN8Xx9X4QFvXNSk57xTeFjVxl3hhtmtykzOslPTYv8dRX2XdQwP9zFujusqpmSfjwH41FLXtCWRGIm207SxxIrt0tnqih1vAJRSgAqboeFqS9auWGh0EJIi10Bz4PbwKxtrmxBfQxZTEqtREM4jaE+qHysJQ03/FAXjbPVvfN29PvdUiF27mKqXYekb1Oic5YADWxCJ01LmnXVWbJUeRjejasIh+OHe91mGfV+BsU3geLuwN1KJSddPrK0ovgvu+Eq9nk2N+1/5Q22CCv6U7HK5pSFGqXTf1m/F0MYNuGw5+nF7cJUWTTzt/J+hPBSHrGxr0gkn4KiAOhBFEyKF5nBdXpilMpSa0SQODbiQLWR5jasbvx9+lG7oLE21 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN4PR0601MB3728.namprd06.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(136003)(376002)(39860400002)(346002)(396003)(166002)(54906003)(64756008)(966005)(8936002)(71200400001)(55016002)(66476007)(66556008)(66946007)(66446008)(5660300002)(91956017)(478600001)(52536014)(2906002)(9686003)(6506007)(7696005)(76116006)(8676002)(186003)(6916009)(86362001)(33656002)(316002)(38100700002)(7116003)(3480700007)(4326008)(83380400001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?Windows-1252?Q?Riy+i3o4swjXEwb64jAbH99Al1Gs4svILld01UHc13l98mfoGTBFbBU5?= =?Windows-1252?Q?TVrvzkycp78g0/BzuEN5QeoQ022dvl9RX5m1CNZJKO695UpHEE0Vb+mw?= =?Windows-1252?Q?kYrUvJIxx6N531K5s+J9zzHcWPeaDAIMP/Yf070J1JbGmwVp6zAV3Jry?= =?Windows-1252?Q?P4rfpviWqAcbXgSNmRlR8RxxJwNhJLSG6hjeOU7CBx+SAIOYG5dXp3/9?= =?Windows-1252?Q?wr2kT+Pjkonsj3advNQ5w6WJh2VCPAZFwo7jP0wqKFwib5fTlTAnre04?= =?Windows-1252?Q?SFbiNOFPM+tdos8ufH9mxH767AP9eTDNJ08/jVFOFmHzopl1X9chYVWD?= =?Windows-1252?Q?OSKKZGJI97HmEx85fqpTHt2oAaTnFTgjW6c4uXwVdxBcO4Sh8j+hMsqe?= =?Windows-1252?Q?hP5BgCatWyCrCFL2EYgpEDUSrhLHm3pTV1h1kEA8rxW6KYgweqTWcXzb?= =?Windows-1252?Q?Y9FCo25m5I3iVBlrHj+77Ev0YqETE/war+K1T4DRE1yDVwvFzQqFHEeq?= =?Windows-1252?Q?7bE3DjjQ11SgZ9sRKcx2p+8fg23/g+Fzapfu7RhkP0ymu5E3RCjGiQV/?= =?Windows-1252?Q?qxKE5VnBMsGcrDuU/vYd0YXFDyfNmxrdr9SNlw6hZSZb8lRTmOFLTGa6?= =?Windows-1252?Q?J2FbuNdavsbzz2GwpNTeXmYcpf6/vPvpKaDRsLWRZqd85jyqr7V/NunX?= =?Windows-1252?Q?ECfWTi98w5CCz0SKRtRU5h48KFObPtgrMLBY4qKMlfnxCCKY7gBQemjA?= =?Windows-1252?Q?sK2wHMFwuikNVHEdQyk+s3NJwzgs2xQWxG2NkOpVMZ3AULRR1sh35FEi?= =?Windows-1252?Q?ySQp6xCgYr6NBcdYlL0rIvOxmRUYxIEQBILPOE70UQpXJHIiNayEvVMR?= =?Windows-1252?Q?wp5douHkFOAR8H88szRcY4LF1d/HnxK4NZQBqG8mpTVDItRFB+kiri4W?= =?Windows-1252?Q?RTJokoii7e1i6QnVeCERolWb3wE1VsqiDihjfjBqxSYYv2nX9FNcc8Qx?= =?Windows-1252?Q?zCnSv/7aMt/eriEHMah5gGj0+2Kg1puDm6nSjSRvDjRJMe+o8jjs8utn?= =?Windows-1252?Q?T86/LR07V3Je+TSbVEQVWz7IsMyMBLnn3bTRTuBdklAPCkmmDkY0I3JX?= =?Windows-1252?Q?SeO6UavSSqCg+auFCBbzQCJP/TSkAMYCS1V6XF4Iq285q2M03TENFJBd?= =?Windows-1252?Q?GuEapw8M5bPFdqnIKb45+9vqcuKaD4gO7SUp1x94Dye2hn9zhjz3ZEMI?= =?Windows-1252?Q?luGcyx9BXU88mzAyePQRu4bKaQsubVTv7Ysj9/HG4ECGUPQJ3VHKv2rf?= =?Windows-1252?Q?BiUcH4i8tZSboU/cXhivp8VDIWiyFBSSXY5yGpslCDAFqCuoxQze3pKB?= =?Windows-1252?Q?z8X8TTZvXPIId3rrC9swz1fgXxY4urPgKjjqTruN6KR9fsNDa818Oq6m?= =?Windows-1252?Q?BTzoD+N6VKw7wRZ8ZznAxyDlYVfb06ywLOLVcuxReHhSKsdbKZJdaQdX?= =?Windows-1252?Q?MZlwdySOGueHJCGMjbEhB6Kq7E90MQ=3D=3D?= x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: netapp.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN4PR0601MB3728.namprd06.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 381f128e-46ad-49e7-189b-08d8fc2e935a X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Apr 2021 14:40:24.2086 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4b0911a0-929b-4715-944b-c03745165b3a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: +6gUlSzk6wUGxPF6aEPx6ZyjbGOeT/BpBwXu7z86QE+V7zYEXlPuASRLFDvctWHGNtUbvYeAX83NITyIfQmZGg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR06MB4621 X-Rspamd-Queue-Id: 4FHd2g3fF9z3Pty X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=netapp.com header.s=selector1 header.b=RxDPGAwK; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=netapp.com; spf=pass (mx1.freebsd.org: domain of Richard.Scheffenegger@netapp.com designates 2a01:111:f400:fe4d::601 as permitted sender) smtp.mailfrom=Richard.Scheffenegger@netapp.com X-Spamd-Result: default: False [-4.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a01:111:f400:fe4d::601:from]; R_DKIM_ALLOW(-0.20)[netapp.com:s=selector1]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f400::/48]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SPAMHAUS_ZRD(0.00)[2a01:111:f400:fe4d::601:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_LOW(-1.00)[netapp.com:dkim]; DKIM_TRACE(0.00)[netapp.com:+]; DMARC_POLICY_ALLOW(-0.50)[netapp.com,none]; NEURAL_SPAM_LONG(1.00)[1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-net] Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Apr 2021 14:40:28 -0000 ________________________________ Von: tuexen@freebsd.org Gesendet: Samstag, April 10, 2021 2:19 PM An: Scheffenegger, Richard Cc: Rick Macklem; Youssef GHORBAL; freebsd-net@freebsd.org Betreff: Re: NFS Mount Hangs NetApp Security WARNING: This is an external email. Do not click links or o= pen attachments unless you recognize the sender and know the content is saf= e. > On 10. Apr 2021, at 11:19, Scheffenegger, Richard wrote: > > Hi Rick, > >> Well, I have some good news and some bad news (the bad is mostly for Ric= hard). >> >> The only message logged is: >> tcpflags 0x4; tcp_do_segment: Timestamp missing, segment processed = normally >> >> But...the RST battle no longer occurs. Just one RST that works and then = the SYN gets SYN,ACK'd by the FreeBSD end and off it goes... >> >> So, what is different? >> >> r367492 is reverted from the FreeBSD server. >> I did the revert because I think it might be what otis@ hang is being ca= used by. (In his case, the Recv-Q grows on the socket for the stuck Linux c= lient, while others work. >> >> Why does reverting fix this? >> My only guess is that the krpc gets the upcall right away and sees a EPI= PE when it does soreceive()->results in soshutdown(SHUT_WR). > > With r367492 you don't get the upcall with the same error state? Or you d= on't get an error on a write() call, when there should be one? My understanding is that he needs this error indication when calling shutdo= wn(). > > From what you describe, this is on writes, isn't it? (I'm asking, at the = original problem that was fixed with r367492, occurs in the read path (drai= ning of ths so_rcv buffer in the upcall right away, which subsequently infl= uences the ACK sent by the stack). > > I only added the so_snd buffer after some discussion, if the WAKESOR shou= ldn't have a symmetric equivalent on WAKESOW.... > > Thus a partial backout (leaving the WAKESOR part inside, but reverting th= e WAKESOW part) would still fix my initial problem about erraneous DSACKs (= which can also lead to extremely poor performance with Linux clients), but = possible address this issue... > > Can you perhaps take MAIN and apply https://reviews.freebsd.org/D29690 fo= r the revert only on the so_snd upcall? Since the release of 13.0 is almost done, can we try to fix the issue inste= ad of reverting the commit? Rs: agree, a good understanding where the interaction btwn stack, socket an= d in kernel tcp user breaks is needed; > > If this doesn't help, some major surgery will be necessary to prevent NFS= sessions with SACK enabled, to transmit DSACKs... My understanding is that the problem is related to getting a local error in= dication after receiving a RST segment too late or not at all. Rs: but the move of the upcall should not materially change that; i don=92t= have a pc here to see if any upcall actually happens on rst... Best regards Michael > > >> I know from a printf that this happened, but whether it caused the RST b= attle to not happen, I don't know. >> >> I can put r367492 back in and do more testing if you'd like, but I think= it probably needs to be reverted? > > Please, I don't quite understand why the exact timing of the upcall would= be that critical here... > > A comparison of the soxxx calls and errors between the "good" and the "ba= d" would be perfect. I don't know if this is easy to do though, as these ca= lls appear to be scattered all around the RPC / NFS source paths. > >> This does not explain the original hung Linux client problem, but does s= hed light on the RST war I could create by doing a network partitioning. >> >> rick > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"