From owner-freebsd-net@freebsd.org Sat Apr 10 09:19:35 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6A8665E094E for ; Sat, 10 Apr 2021 09:19:35 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2067.outbound.protection.outlook.com [40.107.220.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FHTwQ2HgKz4pDd; Sat, 10 Apr 2021 09:19:33 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AZOjHbhxwgegiZ14WBAaFA5aPMq44GQoY+fihzHgohNmvDOyx7+s+zhibD8NHQg+mFZSqRiaep9/ZQBcNlK1Ln84FhV1wGcoI7JmEtXteiCuwPIWP7LJOMT9gUL48hM38J9Z/K7qeAqojaRASbE7ftjMCnEscS+SFLm7mRyXzwCZ1z60CQ1zUMk14q86Bk8DeX5bIyWdtNzdeA2kEj4qGrmJO5eYO1RMnqTahO71XygbwnXq6dfwzTI3XbaJ7iYYGLpwO2feaDBOS3j6hV6/gvXE1CKmaLBW0WLjusDVlusdv2tvZEdAconQ3B1y/pwL2M/VViRCH7IzZk6njLhqEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=icq3c6uni/jTX/HalLxu75mKraOC8uwXjINwSA5Ih8E=; b=heLQU46Zd66tmWRmxMKyqVo67/oG0NQk1grNssNOkqxUmIX1sx0jIJbRBWWt2BW4Wmo9lvt47kZ1dVBRWth2GRiDmpFyRyJ60uMI7njp3xoOUfDXKdZOI14XzUGqLju1KhO4VG4bUOboY2UKkipWXfpGYOkDS4aHZJFUgGEKj720hSps/HmIVQrkGDYttHGJUnH92u3jdi6F5WibrBomSRigAMcjI9Umx5kjCQ0Hkru3BwqWUIryJ7PD9o7ff0f7DFck67ABavx2vlOXHtAWjtyJlz5Rhj757sRlf9kg4cO8jC/p6KXNUeigfraC3ZqZDAqeg3Ec8Jw7yq0CALWI0A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netapp.com; dmarc=pass action=none header.from=netapp.com; dkim=pass header.d=netapp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netapp.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=icq3c6uni/jTX/HalLxu75mKraOC8uwXjINwSA5Ih8E=; b=Lm2OntH1iHTYv1lt2D1glLyNfyjjvyZ3e+WXNDz5XHlW9ouiLzeNmexpA8qeMjuMzqY5wmaOFIKJZr6+Yvm23ihiVM1vz6A6TjkDdvG5ue3SnHhW2TMKyHfnIKTmoJYbXu2cCp/qD+DpCF01pWiT5Tr7xpGLnoPUNdOPO6OhHJBFg7uluQGpszQtVBuo5IG1z7UULejnfNJvb9+GT47/7ZNXEkyPX9xuKlhglX14xCUrbVXdrPdytXWG5Wzvu3rscy+uY4LQHGCFVqVpc+A5DrugFVXKGG/kBBDIwx1JpjuUF9w6TEANZuMv2tiFDaw8hR3+Wb3XmUBKPhD5llJ3Uw== Received: from SN4PR0601MB3728.namprd06.prod.outlook.com (2603:10b6:803:51::24) by SN4PR0601MB3728.namprd06.prod.outlook.com (2603:10b6:803:51::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4020.18; Sat, 10 Apr 2021 09:19:30 +0000 Received: from SN4PR0601MB3728.namprd06.prod.outlook.com ([fe80::ccb:944d:e270:63ef]) by SN4PR0601MB3728.namprd06.prod.outlook.com ([fe80::ccb:944d:e270:63ef%6]) with mapi id 15.20.4020.021; Sat, 10 Apr 2021 09:19:30 +0000 From: "Scheffenegger, Richard" To: Rick Macklem , "tuexen@freebsd.org" CC: Youssef GHORBAL , "freebsd-net@freebsd.org" Subject: AW: NFS Mount Hangs Thread-Topic: NFS Mount Hangs Thread-Index: AQHXG1GB6agsoGWN0UqRoZFo/qoHTaqMDIkAgAL97ICACMXzgIAAsfOAgAfvbwCAAQ5PAIAAWDiAgAKBMZWAAD3WgIAAFNIAgAA/e4CAABvaAIAAEe2AgAEE0ACAAJCpAIAAgu0AgAXcwwCAAH0H4A== Date: Sat, 10 Apr 2021 09:19:30 +0000 Message-ID: References: <3750001D-3F1C-4D9A-A9D9-98BCA6CA65A4@tildenparkcapital.com> <33693DE3-7FF8-4FAB-9A75-75576B88A566@tildenparkcapital.com> <8E745920-1092-4312-B251-B49D11FE8028@pasteur.fr> <765CE1CD-6AAB-4BEF-97C6-C2A1F0FF4AC5@freebsd.org> <2B189169-C0C9-4DE6-A01A-BE916F10BABA@freebsd.org> , In-Reply-To: Accept-Language: de-AT, en-US Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [217.70.211.16] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: e59dc514-d5ac-4586-fbb4-08d8fc01bf51 x-ms-traffictypediagnostic: SN4PR0601MB3728: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:4714; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: mdj7L/PFDcsXiHLoKDgLHconNF/cqO2SxkqXqUOdEyKc+V97Zllr1ACbz1kkFzzjqCUSv0jJjhAtayHAOrvLbpIcX41mnZydrNM13EovyXdOgz+xkqJHMZaVGH/0Pm4X+YA7W0MxywDEd/5HJlW/0AiNbcxtyybVrcBSfbcq1HRZgSrqArICtdjrJGLFz6Q/xjH9HyC6y7fgZbKGYx1Av8MX2KSKclHc+4MEqrPB6xOPCnGBtTf/igOF7T9SQf4InK4X+35BY5ZkC7syMKr/FsUg1fyQ2h9y1F1vBV2Nw5xswIlnzAkSUxQlbFsleTzmk46FFAT9WS25cqx21ojQ4DLPtUEdNqir8Qu/Jl8tiTCbpYt5kAvPKigkLlvVgBMFu/r4LgywyvAOogPOR18s2GmpMgEfT9j9ZYtNo0eUBs8MVdy0w5JTvj+ZW2AKJSoYSpT/nkPQF9O/mAEuE+IJQKiqsLqAGd3ChVy9FdL2xIrZ0B/WFTdcoWvVhaE6f9Sqtf/17CCBk6POoOJ1vMd4Ms7VabyBFS7braFBV+o5jMrguAi7vJVLs1Y9a/L4W+ikIWNwJZxPAV1OElXfiu3hntbSDsCFTVz6VPhbMpyimLmTRdM86CvHZxTldWoVjEx5BCqw8Do1zX1MUZQz6zRE29xVEkXJ+xPxINDdWM8HtfPlWHT9iyM6PF6yG795c4wr x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN4PR0601MB3728.namprd06.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(376002)(39860400002)(136003)(346002)(396003)(64756008)(66946007)(5660300002)(6506007)(2906002)(966005)(76116006)(9686003)(66556008)(83380400001)(86362001)(66446008)(55016002)(38100700002)(66476007)(52536014)(71200400001)(54906003)(110136005)(26005)(7116003)(33656002)(316002)(4326008)(186003)(8676002)(7696005)(8936002)(478600001)(296002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?7taCmjVeVtk0E57fWr85b+Ie4bbRiu5/huq+kKwDct4CoBFlVZRPIwzaqdqf?= =?us-ascii?Q?X57kSazp/0KjXEvtWRWI5RaBZ8GsXEcwuTV1O7+kkQ6zefw9W2p5dsE5PgQZ?= =?us-ascii?Q?06s/IAa8FlViJ6SqiaUAt5e4VoyzEk4AUlxGRZx44JUjQeEzgH1UI4a9BbOg?= =?us-ascii?Q?t+XGXNVIdKKIaIETg23iH3yMeYVRAJUFXGMqDIGu+Sz63OXrpE9wug9UGyUE?= =?us-ascii?Q?tpWioQmuHZ2Ndrs9yceq0AYabYBIgMNmQmvUueWj3UmYVKXCwiOfCypjkI+r?= =?us-ascii?Q?F9vdR9zS1qHH/33YNbD7E1U0K0xpwrHdc1vVyMTcHEJZT+1b9asfCROmzqw0?= =?us-ascii?Q?d/dvUl8XSwo6n/SfWjaRYoJmZlqH2Up1kBXZf+eOdJtzZY4MyxNGardKvxgV?= =?us-ascii?Q?jboRDGWwtR7H4K4AjbGXACXVQ/KcyrpPpxqeVz+d2jXNKFBtaLu7Q1dd3ZPn?= =?us-ascii?Q?FbdGgSCrdVHMiVtZET0xNgKXBPrV1Nd6HgDoQ6fog5vQwrVuPyr9upIKzaHT?= =?us-ascii?Q?PPhGZTI/mon1zUo7nrkc/eUvKO8j+ETmh+6WjZMi8gHbTHK0+ZlFVfY8RLSU?= =?us-ascii?Q?752F1hZ8VS0ys2LMWy9cQ30PK2WFnSqOYzrH2ikw5Kab3YPUPIvCP0iN9NeA?= =?us-ascii?Q?rRG6IiSmVV7opAVeUrZ21WxFAWFyPP5QhevbuGfsyxUMAVa9Rzd4XdqhyiaH?= =?us-ascii?Q?FoweoU5MlgrYkdG0wjZKSlcaDis1Jg5VtMYUDqqa+CJJXhYwrRSxHT1cWUsS?= =?us-ascii?Q?8QYtC6gf8wrWtmzP0bCVSKARXAQvsDNXWQXKOTex/Iqlpp8ua9Yb5lG7DKeU?= =?us-ascii?Q?N6UpgmOt3u79ICLgNgOs0cPtrGdTpskGRc3agt6/vBBpcW2sulkPXZwdp0cK?= =?us-ascii?Q?z/dqCBqFAvn6GevmDERDTkDKr1FkKwybcBdeK1kAG5YNP3FnM1KRGOt0VIQ7?= =?us-ascii?Q?vWJN+nlSYeLSlt+Y3X3WhiMVi3KwSFu6O9UvY0H1QkbESR0arhQpTY6B11hL?= =?us-ascii?Q?vY+9ZTMfkGNmRO7CJPKBBBHn4pmkRZXo93GjZUm1vvVzOD6m0pkHeYFJUMRX?= =?us-ascii?Q?aP6yLAG4ouPKYLLPNPQf+HsAYqldvW/KJIrWHDdUpwZXa6Cf1JhAirsosmAR?= =?us-ascii?Q?2+t/xn/PbC4gUfGqK8CvxQrBA8UWBeWMo8OKdjID2xBycZuGIiBW+ylK+Smb?= =?us-ascii?Q?E5W5Ilch8X+x69sY1bp9gCOonIZBWlN9wQFjOaNRq/OQeiH1kyuFLL2kNlNi?= =?us-ascii?Q?Nb9oKjjBNhl9Nb3jDon0IYKAowqu7hahn5bOisJG4nG65hZzjJt71CzTGi4g?= =?us-ascii?Q?71kuVZd9X6sThTVv2qdtrOKm?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: netapp.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN4PR0601MB3728.namprd06.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: e59dc514-d5ac-4586-fbb4-08d8fc01bf51 X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Apr 2021 09:19:30.4844 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4b0911a0-929b-4715-944b-c03745165b3a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: JrAa0iBQnRlTiSFPgwjhjo6/4tU9+DUdSSa5d2uNrx2zTYtsYpoKlI+Wsq+qYIMrzUFNTODylc82EaohrHEkZQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN4PR0601MB3728 X-Rspamd-Queue-Id: 4FHTwQ2HgKz4pDd X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=netapp.com header.s=selector1 header.b=Lm2OntH1; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=netapp.com; spf=pass (mx1.freebsd.org: domain of Richard.Scheffenegger@netapp.com designates 40.107.220.67 as permitted sender) smtp.mailfrom=Richard.Scheffenegger@netapp.com X-Spamd-Result: default: False [-4.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; HAS_XOIP(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[netapp.com:+]; DMARC_POLICY_ALLOW(-0.50)[netapp.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.220.67:from]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[netapp.com:s=selector1]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_LOW(-1.00)[netapp.com:dkim]; SPAMHAUS_ZRD(0.00)[40.107.220.67:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(1.00)[1.000]; RCVD_IN_DNSWL_NONE(0.00)[40.107.220.67:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.220.67:from]; MAILMAN_DEST(0.00)[freebsd-net] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Apr 2021 09:19:35 -0000 Hi Rick, > Well, I have some good news and some bad news (the bad is mostly for Rich= ard). > > The only message logged is: > tcpflags 0x4; tcp_do_segment: Timestamp missing, segment processed n= ormally > > But...the RST battle no longer occurs. Just one RST that works and then t= he SYN gets SYN,ACK'd by the FreeBSD end and off it goes... > > So, what is different? > > r367492 is reverted from the FreeBSD server. > I did the revert because I think it might be what otis@ hang is being cau= sed by. (In his case, the Recv-Q grows on the socket for the stuck Linux cl= ient, while others work. > > Why does reverting fix this? > My only guess is that the krpc gets the upcall right away and sees a EPIP= E when it does soreceive()->results in soshutdown(SHUT_WR). With r367492 you don't get the upcall with the same error state? Or you don= 't get an error on a write() call, when there should be one? >From what you describe, this is on writes, isn't it? (I'm asking, at the or= iginal problem that was fixed with r367492, occurs in the read path (draini= ng of ths so_rcv buffer in the upcall right away, which subsequently influe= nces the ACK sent by the stack). I only added the so_snd buffer after some discussion, if the WAKESOR should= n't have a symmetric equivalent on WAKESOW.... Thus a partial backout (leaving the WAKESOR part inside, but reverting the = WAKESOW part) would still fix my initial problem about erraneous DSACKs (wh= ich can also lead to extremely poor performance with Linux clients), but po= ssible address this issue... Can you perhaps take MAIN and apply https://reviews.freebsd.org/D29690 for = the revert only on the so_snd upcall? If this doesn't help, some major surgery will be necessary to prevent NFS s= essions with SACK enabled, to transmit DSACKs... > I know from a printf that this happened, but whether it caused the RST ba= ttle to not happen, I don't know. >=20 > I can put r367492 back in and do more testing if you'd like, but I think = it probably needs to be reverted? Please, I don't quite understand why the exact timing of the upcall would b= e that critical here... A comparison of the soxxx calls and errors between the "good" and the "bad"= would be perfect. I don't know if this is easy to do though, as these call= s appear to be scattered all around the RPC / NFS source paths. > This does not explain the original hung Linux client problem, but does sh= ed light on the RST war I could create by doing a network partitioning. > > rick