From owner-freebsd-current@freebsd.org Thu Apr 15 21:05:28 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4F63C5D7D2F for ; Thu, 15 Apr 2021 21:05:28 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660054.outbound.protection.outlook.com [40.107.66.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FLsLZ6sb6z4rg4; Thu, 15 Apr 2021 21:05:26 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MS39GVkEZ2zlijI4BZ19RGW+0wRK8kMkxoH+OFM8tax41iYUE7vJrobjXCgKg5qf0dAEYhMFjzF/1GiK8G9tGK02YIBWfiDnrYm0SXoFoj2kegvweGL3zzxllRrYqM5GmG2MNOkV6R6QFStXwmpaSEGqdga2fuphsL8P7d4lt3s3gAkvW2I0FamaHSJBFG/yHotZfd/USl2MHNEBUjkWi73JPHmUNB/eXPg3X8B7bA/MgFoBt6utwKuAL/+tB/TMAVaLe7IiJVDraFRJ0kxHAk1InED4e3Ydbq5X7eLVrC35MpqmRl+hMrVxXYGtz52kKh78ZNjl/Bp1abdRANSn0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IjbobT/MIbZeAy/k8ZJqxdE8AraFwgRVjg4qPq1+2PE=; b=gZy/KZ/ptZXD/Zv4p/3jhcyEVmD32gTL6e9tbwibCR5DSVugj0TwePl/+oNcoyG25+YVIRliYDan1cL/IeoFO4OVngGg+bKgYyM9NZTcxMV/kpYWeLiL4Do9pLAPVbtTmOCOw62AjiUKDckKE7AD5ePoxUMbOjSFnfs4G+uHzcTpQW9D/NP9cggjnh/TdefFtqx8vxbpjo6hNd+Wkm7cmeKUar2l+abfHFBihrnv3ptQiSh6HU2uUTIHdTX0/iEhnBs5JhvsHYLmXs7XXzmEc535l4vWgIH1UvN7ciecxyI3OVcZaPYqJMcXJXqP5/EoYrfYgDQdEN9sLHJV89BFhw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IjbobT/MIbZeAy/k8ZJqxdE8AraFwgRVjg4qPq1+2PE=; b=A4j8N28bCAz3jyj0SS6YwcwOXsfCnhNN2D0sPQ64wZZG1ehiVN8wOqQFjSlYi4ycGXO7AnS9+qum6j7+Dlx6mDF+I/GASSnfSTzLsPWNbk+3u848ZVQBqfccmt5kN03vqO1YENcueIw6488vlE994HO+2Zq9sOX6mB8yONNoDcSQ1ayIeRq7RFJJev1yG2cf7Ro0Y8GoNVus5tyblytMepSdFiQWz58pvr+fx0+3Qfd8L6im7c5k2UE+5y0YoDZez7mITkXPt/DP0sAf9TuR6z5wM+I+QhCSR2CgNUF24hTqqkEeoTAivZtSK3fUWDQCfCPh821MutpjyWCMa5i/5A== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.32; Thu, 15 Apr 2021 21:05:24 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::1c05:585a:132a:f08e]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::1c05:585a:132a:f08e%4]) with mapi id 15.20.3999.037; Thu, 15 Apr 2021 21:05:24 +0000 From: Rick Macklem To: Allan Jude , "freebsd-current@freebsd.org" CC: Richard Scheffenegger , Juraj Lutter Subject: Re: NFS issues since upgrading to 13-RELEASE Thread-Topic: NFS issues since upgrading to 13-RELEASE Thread-Index: AQHXMhVsKLCb5bI16E6Ydj5qgkLmjKq16AQAgAAfNj2AAAnjjg== Date: Thu, 15 Apr 2021 21:05:24 +0000 Message-ID: References: <902a3c81-2ce8-49c0-b163-5ffa4b90afe5@www.fastmail.com>, , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: dca080eb-c09a-458f-0f83-08d900523047 x-ms-traffictypediagnostic: YQXPR0101MB0968: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:5797; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: LQM/HGLy1AVfCzpD7cjDq0Qmg65mRSHYdHbIFdHw1N7J+Gte5LzoqgyMaQ6wzgexHvMlYC6RtLpj/9DBq3WqE6NEYVsyJ95d7/UDErQTzVwedAtrBIPXp1po2PtG4sUUrL4a28VXZxJe5yqCh66GiCs9P6OztLzZwcny3aKOHG6CZBibEfNY1FC59YHVJBjoUE4FY98vsxO0y4Fm3A1MOknLPUY38DAO82L/1IsyEXuy/tl/gcqqbJt/IzumIkg3YDyato0lme1cjbKku/3b5Gtszy8zduO3RYsX4gZztfg+pXRBhDzJaGkaTbu0iGQJBu3kbwwOmpSmXgaPWDgnaeRRdUZuFWuFUxyxMwGQJwqDA+78cZt+j1OuHieaEa4/6y4j1gGDa8HHVKsPFKMjiKB7Ms8yODmsp1FNwJOxt83odxYv5PNwueQAjduto9RlWzQrCrkk+68td8Pt4ZkyLkw/6emeiYqZgivYDc6nfhOnxX5OejRazUZPtyqKh++OHAruuzL9BQkkGOCLSt3YwEDFxu2UCqrJHrwg4cDWybblbGtRAMsTSgL97Mr6MaHPCRrhQXkFfDJX3g8dyGaYpAT8TpgzedCNx1lkVlCOhUlEX8Bfn+ltQMXP1EdD7kswAQ2IDIQ4SJIMoxTijDCgmpUfaosycHs76TaWJWZ04aJyMyOz1r3j3AB6xVuNkIX7 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(366004)(346002)(136003)(39860400002)(376002)(396003)(2906002)(55016002)(86362001)(450100002)(38100700002)(54906003)(4326008)(76116006)(8936002)(8676002)(64756008)(71200400001)(33656002)(66446008)(2940100002)(122000001)(91956017)(316002)(66946007)(66556008)(6506007)(52536014)(9686003)(5660300002)(966005)(186003)(786003)(7696005)(110136005)(478600001)(66476007); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?fiw8ZfwsT1HyyRwwZbjYTkAMpvXBE5ZVOzERcR4e35QmPk3iQj+Z/ir5r/?= =?iso-8859-1?Q?9CM2uQKI8AF5d4SxSJp7EoiY3VXtoijkD1FHWpjCCQUod9XbhufTllyWhF?= =?iso-8859-1?Q?onwGjPmk/ZprPANCHHJ4MPEhPtVSxiMFgCT7yK6RnwfEysrr6hfs0cErZ9?= =?iso-8859-1?Q?kldAHz/3HBb4IeInjodfj1WEGkEUERjX6nV6q+/+b4cx0vU/Y4SE/5oY7T?= =?iso-8859-1?Q?FnY9dV7mQPIMnmITE8ckfJI29Ug7Dgo+UPZ22cMQ9/Jb7IB7rAjxBOwHTx?= =?iso-8859-1?Q?Ov93FyybXkrPkwOTUm9w3UeCcFuL2v8IPJ7IvCdS+iBUtd93cjrdxFe8MX?= =?iso-8859-1?Q?x/STDtMFOy6YNjjhi3dP+dR2H6BNwu+9cCU9OCXOVwXMeuD+KP5LZkCoIc?= =?iso-8859-1?Q?Szitb1ysFW7rnUs9d5zXSIGcDNGLFTR3DZ8pcrCsJBF25L9M5eBh8VLDfe?= =?iso-8859-1?Q?bs2n8VEO0uGmFYIHj83I+rJoskMdqe6Cc8x9W30l3a5j05mSzi5/BOHp4/?= =?iso-8859-1?Q?aI4LgKJLaxfNOIA5pfiBE1AKe5FOyB4QFamIxRJC3efRxqy77mKYxgLG2h?= =?iso-8859-1?Q?AwqKwuLzDLxpkLdqmrVQcvKvWf4kJ1oWZLPpldi+3fzk4cbE84YrZSZ0mi?= =?iso-8859-1?Q?yZnn+g7RnktES45HSB+wBWsrbYyieAYo5+u/qz9sZz7z4FWr8zDH2AQUSF?= =?iso-8859-1?Q?+hTG5iWHimpadTDo23HsofTaRnSq/+mVfFp3OcueRPhvMHw2gdYByNOab5?= =?iso-8859-1?Q?dt90JGLo6FmHjdGR2SH2Zb3tTMY5aBv7HmqE1F0TH7oaQBY091sbQE3anY?= =?iso-8859-1?Q?c2QGVy6Od6g6uYuMUBlQ9UzKkej+eTeAjYeUbtUjGz5Sn8Bo5YSnaVgKBU?= =?iso-8859-1?Q?yT+6Gy895Vck33Q1LnBh/QLrajzBb1wc5Mf3orKfjxaQiT09BTo8ivSIZi?= =?iso-8859-1?Q?Wf4Ey0nW4ZILMzdQzklI7p7zt26gbwpyIq8uXBnxYk5m+sag7ipx5Y9WgR?= =?iso-8859-1?Q?qo63GjjgCqUF0ZPdllvMwLVW9vH46NMwGPOFYOChFLiYDNGZrEhESpb8oV?= =?iso-8859-1?Q?adaxY9MBz9q03Et3PQYwrqOK7CW32SS9MhxVXgD1JRZB+5lKWLNKUVXynU?= =?iso-8859-1?Q?cwjeMzNYYdc0tBFLbPNIUXBPInT7TAgVGb9xsOpjEVsfSgxd7yUPzjbGLc?= =?iso-8859-1?Q?NzWt0xtJcOezj5xcJVF5PlawzU+ceoSc61/FCXhSxQzhEka5gx7zJ7afzj?= =?iso-8859-1?Q?ZFroUS1Zjv5OKv3NQPa+KmuvvGpkvNzfU0o0yPekcuNCmYFEEd3gqoCIQT?= =?iso-8859-1?Q?CFpjhiFMrtbWngeeYLTOvTTGkKgPOn9AM/L6aut1rvNiffF2ZS2yU3CVUD?= =?iso-8859-1?Q?tL3hW0CksmfO1cc8GPAEPLpiAZmvXww99IKzT8kO6MGiT2RFNyZsJpjmFk?= =?iso-8859-1?Q?fAH+AM4dyVUiMENa?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: dca080eb-c09a-458f-0f83-08d900523047 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Apr 2021 21:05:24.6497 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: mNUeSgDC/cij24NQ3OzdpR7nnWX2ynmS16OUBEh5x8eiYSCWfZK5fj9wJ3+pZIfqi1uHOHmLszgvZq+kzNbSaw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR0101MB0968 X-Rspamd-Queue-Id: 4FLsLZ6sb6z4rg4 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=A4j8N28b; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.54 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.66.54:from]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; SPAMHAUS_ZRD(0.00)[40.107.66.54:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[40.107.66.54:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.66.54:from]; MAILMAN_DEST(0.00)[freebsd-current] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Apr 2021 21:05:28 -0000 I wrote:=0A= [stuff snipped]=0A= >- Alternately you can try rscheff@'s alternate proposed patch that is at= =0A= > https://reviews.freebsd.og/D29690.=0A= Oops, that's=0A= https:/reviews.freebsd.org/D29690=0A= =0A= rick=0A= =0A= I have not yet had time to test this one, but since I cannot reproduce th= e hang, I can=0A= only do testing of it to see that it is "no worse" than reverting r367492= for my=0A= setup.=0A= =0A= Please let us know which you choose and whether or not it fixes your proble= m.=0A= =0A= >> Any pointers for troubleshooting this? I've been looking through vmstat,= gstat, top, etc. when the problem occurs, but I haven't been able to pinpo= int the issue. I can get pcap, but it would be from the hosts, because I do= n't have a 10G tap or managed switch.=0A= >>=0A= >=0A= >run `nfsstat -d 1` and try to capture a few lines from before, during,=0A= >and after the stall, and that may provide some insight.=0A= >=0A= >Specifically, does the queue length grow, suggesting it is waiting on=0A= >the I/O subsystem, or does it just stop getting traffic all together.=0A= =0A= If the revert of r367492 does not fix the problem, monitor the TCP connecti= on(s)=0A= via "netstat -a" and, if possible, capture packets via=0A= tcpdump -s 0 -w hang.pcap host =0A= or similar, run on the server.=0A= =0A= Ideally the tcpdump would be started before the "hang" occurs, but running= =0A= one while the hang is occurring (until after it recovers) could also be use= ful.=0A= =0A= Thanks for reporting this, rick=0A= =0A= --=0A= Allan Jude=0A= _______________________________________________=0A= freebsd-current@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-current=0A= To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"= =0A= =0A=