From owner-freebsd-current@freebsd.org Sun Apr 25 13:06:20 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 80D686221F9 for ; Sun, 25 Apr 2021 13:06:20 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660047.outbound.protection.outlook.com [40.107.66.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FSpF66zvVz3MTD; Sun, 25 Apr 2021 13:06:18 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hYp9Fx+DxwRXBi9r4P75RAT8FMjYAMI9c/gMNd0HM6XIs9TbXMncGx0Bz4r0gTpK8+06uuWpUtnrDSXej9WNtPxkjRpu8/C29zkBI/4vh6uzCNT2cyz10lKPucInfAcsltoOiReBgDVYdEgOwQlR5B4Lh/ljFLjY13QDQdkXqe8W0KAadGgstaB/4o3THgWBdVeDXr7Pqs/7s9fDGvBqFiqysS0I3tOjw0pdVcDwtyB+SZHqlquYYoyiTxO+BnfYbRSyMSfvmOyAB4RIBCtYam67MEACawdFtksklId2z+ltp3QpjQQgNHowuJoOZ725Xe6pyovliBwc9jX11coS5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=biHdlDyOa4hRupM1SI09Q7Q5/AISobTT1tokxPrkK3c=; b=MnoF23zXTaOADZJvC15KWPutOiTGw35evRLucpoKi5Yq2bEyFJfOVgFLiMx1AReby58V3yBGuFAjCkWNufzF3lDmz2OXn7NukIt/LiWDEqWc/hxGfgrOONKmzK9ubqZyCgdKBjEHePu0FQCxDlT2LPcMnjYdnuWPa+eSjzG8BOSDWz20TM7TQJMsyGJYJNQ04ABxXZKwkLf0IphlaGck3/qqs/GKtPuV3ajrpXFtjZHq3U1HjXXJUUQv1teEoe7SD+3C6uIK+zN0S1B60oLfg9IyUUdfBk5c7IDcYpljwQCwVNXsdyImLLMzc6VYQZM3MoPzx36Nm2WhYn7LHNvKog== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=biHdlDyOa4hRupM1SI09Q7Q5/AISobTT1tokxPrkK3c=; b=PKcQfqSg8m9rFl8gYY8gb8UVJJH5m9ipqmTwtFpc6Y9IN8A1TBRrOq+VybhzGGvTrtYhZQq78x0cuW51hrEKWCznnnPcue0/lbi5MGVUnKWGIXOnJ8aYC8KRvDOYatAbCgC7PoWWtENz81SAHdQ03HgN8nHgNgeqlfnF/C2K2Y5N0oslm/PhzBLucDZ5rMlVMddWfRFVC9E6A8Ln09u9HZSdXIK+WzH+KHt6bTXk6wNsM7G2+Pw7Bjd5mpNTXlKT02ZYmqOBUksC3694cRwI39scncHM0sB7jz2uGRLZqttAtumBoKBqTEpIWBQobYYnLNBjttq8wjr7JNfgKu05CQ== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQBPR0101MB4586.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:15::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4065.25; Sun, 25 Apr 2021 13:06:17 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::1c05:585a:132a:f08e]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::1c05:585a:132a:f08e%4]) with mapi id 15.20.4065.026; Sun, 25 Apr 2021 13:06:12 +0000 From: Rick Macklem To: Chris Roose , "freebsd-current@freebsd.org" CC: Juraj Lutter Subject: Re: NFS issues since upgrading to 13-RELEASE Thread-Topic: NFS issues since upgrading to 13-RELEASE Thread-Index: AQHXM4zqt7Eq2SuS9kaQcUjLjgYVY6q+A8aAgAc5CBo= Date: Sun, 25 Apr 2021 13:06:11 +0000 Message-ID: References: , <089ce08a-1c77-b7a5-4e20-b1c22df23176@fastmail.com> In-Reply-To: <089ce08a-1c77-b7a5-4e20-b1c22df23176@fastmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 82e7e0b3-f124-44ac-f449-08d907eae680 x-ms-traffictypediagnostic: YQBPR0101MB4586: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: tZkkgJcl1MhnHGHBhPCwl1CGmG3cMdvEQH8gwd5OOLZW8fyUV0o7hyH7PWHyhazMHj6G6QWIzRtjGxD6DkbPNvCLP56TW2iZOurmlJJWGlNCQc3fysJvjNKFFeoIuG88ny97RdkNYegLhYYL+E+zLNb9YX0H/aanXFr9EeUi9Dn589m8gwxe+iu3uyD4Z17N1418A5wz9ELIjuwz2G8xjOerDgYHR6JoeFw+if+EB/IWc9N9MZkAXDp5ed33Nf+ybgLySap499VdmzGtNC0OgCrpEyesPfyMANA1+5CZUWAAecDKJeS+1UmG7Cn2s/ASENaphzF5EXp+AOv21pfpOqMYTKB/qiT4GbsbPuIDndv0LF4YXbcEXTppI7sQ16cmMGgKURWzFGSoD+EZRoT4hWauRp/qodVva8YDeHyTJgEpFksGrFDbWOoP1zv5J+fbm0wbqeEp8PSokh9xuQnl4tr/lYlzi5su5A5wqI2A+QKBzFmD35RPmiaC5wyU5pkOkGI4/V1VpsPWzo0yvBMR2QB9OtAxu7HsEPuvvGe+citDpIigacNE54F8qPaNft6isKSIyFBnqrwPE+Q7XLWh6qR4W8cwxkgj1BOcqDkju196dMUKufGWlqfr32oRbWT0JU2JvhwsByXg9wfvYfv2d362hd2RggQJt+FHWbU8LMumVUUNiAfWSR74OzjiBalM x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(39850400004)(346002)(366004)(136003)(376002)(396003)(55016002)(9686003)(966005)(66946007)(66556008)(76116006)(66446008)(4326008)(91956017)(110136005)(8676002)(52536014)(7696005)(5660300002)(186003)(6506007)(8936002)(478600001)(33656002)(71200400001)(786003)(316002)(2906002)(86362001)(38100700002)(64756008)(122000001)(66476007); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?au5C8zPt+LgnpEvLPgS0eiLbT5RPs08rvL4jL+dUSvD8UTztHR/ISGuTzC?= =?iso-8859-1?Q?7pR5P+aO4MdNZj1EVt+mlXzoFEfBMZLQ92+ADq1Q47vTEBlW6DTH8igmV+?= =?iso-8859-1?Q?TMxNJX4P0unAOXcsUJzCz9fenCN+suEtk3BBa27rCeowIpNXjMSqeFE5m5?= =?iso-8859-1?Q?M7YS+v7q93RNmkA2mnpfc5NL7v8CmZ6rcPoOylo5ipfq0vgeufAA4A6/wI?= =?iso-8859-1?Q?ihqlih/hK/U43nUDorQDlQIvuoy12463llkx7ggOIoWDJ6UdxqgD3W7oXW?= =?iso-8859-1?Q?UDxGgi0OyK8ANYI2mKFgQ6vPrHWjTPEveGcJ0RSikPtSVDMBcGjp9CwdPt?= =?iso-8859-1?Q?O9QYXV16YX7P5QTwIZredIf5Hor65a52zFXHuEk/c3jJEq/bGwBOQTR18C?= =?iso-8859-1?Q?aDoPObrgVbuc4J4ijiJ+xI9fBoY/vskteojEq5gyKS/SMxTtT1dE2giCSi?= =?iso-8859-1?Q?khF1mAovUX9Voh8NRvveA6i1gKgTPU4yLgTFgEY549J6ukCneBc1LguMi8?= =?iso-8859-1?Q?TNAFrIxIlmqSqSbnQFir1FVOsdyM43SrYyKGzgaPRGd/imm1luRzR6Y9OR?= =?iso-8859-1?Q?B4lF3T3S+0BveDsjegDMip8tFGz64OytEZflhkI4o0iL/u0Cu+Jz7S0B1t?= =?iso-8859-1?Q?3NeFFEFKbny9VL4aRuRc+Uh0i93WlqEPkf6JbTwJcQAfDsfUbYHtUfCqyi?= =?iso-8859-1?Q?OuUlMUhbpFn6z72wT43TKFyChvnsvwBq8/jwZgFD/gDBMtPQD3PYkLQXpW?= =?iso-8859-1?Q?AAy5FxN/HQxaOWWaSz+FqERkdApapCAf2LHzt8vqBBdCVjbpLH9OCwDDyK?= =?iso-8859-1?Q?3WyZ+4Ormdf9HDCzaH/cW2PyuZmKhyP90rSkeDHHWngM6YsfF8lOhV/eVu?= =?iso-8859-1?Q?W83QKrC07lve3KYx7cSvHhGWRFxTuVAwBmxaGD5c58cZGMSsTawXDoOtFt?= =?iso-8859-1?Q?9QUAk4Anfx/6IxUD3pq6FmTOgsu7dBOB4WVvf5R1LDMGDv1BkdQbDYvH0Y?= =?iso-8859-1?Q?MPL4kwvJwn/q1g9LzEM4MJpZMWxTZgVA3hHK9a+o75Q91WP90+ia/EStBs?= =?iso-8859-1?Q?Ocy1PfSKfezYVoX0z83YLehztJ5tD20bB2tfZWRcdn11z6ZmN1cDOnoPy0?= =?iso-8859-1?Q?AaxDc/FaHLLz/otYpFxPhqsV2BIBdaYg99lNGaLgpWUMlks+YL6yv4T0tY?= =?iso-8859-1?Q?CD3mi+RAR+4kS8KxDpZvHKv8TM0FHXwC2AedcgndAi88N7SAPhbnSn06H5?= =?iso-8859-1?Q?A8z+3J1NrdH3ayLFkB/THoYRehItXPEQxMaL7KIQP5dWOfO7YnudixdRx9?= =?iso-8859-1?Q?gcXirqhwzPNUTT4FSC3COqVBmacdc/hO9HgvoqOVquCZ49ztEdQA3KAs5z?= =?iso-8859-1?Q?RfofTtOHv3J3yrEstLBh7IPVlVNaJCbLOFVGKr+IlQ70V8siDIpyM=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 82e7e0b3-f124-44ac-f449-08d907eae680 X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Apr 2021 13:06:12.0058 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: eut2Djbc9jSlNIHsgFQKRMwAvEci1uO2t6poV9m9D3BtSuN5MgoWnK/0NaCM/6br5mRc+yngNTkfZIa6p74hQQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB4586 X-Rspamd-Queue-Id: 4FSpF66zvVz3MTD X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=PKcQfqSg; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.47 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[fastmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.66.47:from]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[40.107.66.47:from:127.0.2.255]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[40.107.66.47:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.66.47:from]; MAILMAN_DEST(0.00)[freebsd-current] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Apr 2021 13:06:20 -0000 Chris Roose wrote:=0A= > Jason Unovitch wrote:=0A= > > Does anything change if you set -tso -lro on the serving NIC on your=0A= > > FreeBSD server side? Do the Linux clients remain responsive then?=0A= >=0A= > Thank you, Jason. This seems to have cleared the problem up for me.=0A= > Since disabling TSO and LRO on the server NIC last night, I haven't seen= =0A= > any timeouts.=0A= I think there might be a couple of reasons that disabling TSO resolves this= :=0A= 1 - The obvious one is that the net chip/driver is broken for certain TSO= =0A= segments. Often the culprit is a NFS read reply of just less than 64K= ,=0A= that is made up of a chain of 33mbufs with a total length just under= =0A= 64K. Then the driver adds a MAC layer header that bumps the size up= =0A= to greater than 64K.=0A= --> This can happen if the driver does not set the TSO sizing paramet= ers=0A= quite correctly, among other things.=0A= =0A= 2 - TSO does work correctly, but results in different timing of the TCP=0A= segments transmitted for the segment compared with non-TSO.=0A= =0A= I believe that, for otis@, disabling TSO reduced the frequency of Linux=0A= client hangs, but did not stop them.=0A= --> reverting the patch in r367492 (this patch is not in FreeBSD12) has=0A= fixed the problem for him.=0A= =0A= rick=0A= =0A= --=0A= Chris=0A= _______________________________________________=0A= freebsd-current@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-current=0A= To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"= =0A= =0A=