From owner-freebsd-current@freebsd.org Thu Apr 15 21:07:24 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6ECBC5D7CF5 for ; Thu, 15 Apr 2021 21:07:24 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-qb1can01on0612.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe5c::612]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FLsNq48pKz4rmv; Thu, 15 Apr 2021 21:07:23 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=K30GAqyhT0CcEVZygXpLyqQrm67Hti8RpCVI7aZ1hxhjCvp6i8DYIaZkugxUMmBvwsWRMfDk9Uj7sQ/+j9sqjuBmOFsa99zWTFlgFiPKFGULM1o9FfT5rEfzULFncz9g7zgj7nKS8O6TDhhybKtgvwAkjXkUdGgNtpSfbXUtwjLwK8p25ZX7RlqPzYxTiKIRf+TWXAfgMJYp0cGEtqnIUd2kZagBNFVODjCFJMCshTrlpyfEvZzDab4er31c3llQ2pSmUfPefa7QYia9e41zMJSU1P2qrprYjHPb9l0Bz0x5H2IIf94JY49UbFnQvbTLVtdAQdY0k3RG8Rcejd27eg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+7Zg3fFtkTBnP1Z1HHIoXaELqZsMEOvBbEb7FSuLpKc=; b=F60kbRCW3vRlGdg7EJpw3Gr0gHzR8cTcQjpgI89cxPj8cUpfJBIgtZgIUBK3QCqpwS/hVubOuOhl9seAqEoE6xQ4oPt5Nr/eaS0rh+bVfcqFuJs0xnrq6RJZ/eKdzDXqbpbShqYsCbEeqySOV04GBgYaJ2/lzX+q9d5kY542p60hf7BlEofn3SNYy47zIIzO/9WMyW2bdDV9oGYxG887eeHk5oW2eL/SfKzaGPdoL48bp6ZQFCrjU4yoDRhHNFc2ZEMMfx8lFSyWihkrcjtkmeftT4U2VNMz4QDHbWKri+Va8Mt/OU7aZ8EmmDy4LLECjgLEokcJsjKpZx1FMUnLWQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+7Zg3fFtkTBnP1Z1HHIoXaELqZsMEOvBbEb7FSuLpKc=; b=phR4PR02f7ffraGNi2FX7+4Fdaa25Fv4rMdNkG3NpaYWON+PEFOTyPj7hSNSuQpBRNdjymT7ID1aClM2s0IP/wNThuRz3Wf7ViyVH0z6QkSa+EZ+dKE8WOfQe3e82GNlHod+Tt2Z9pTD5WLbfjR6WfQZJg2OZwFZaznU/h6VF7+j2Hd3wU9LPSgmmKUEQovE4czFFyP0MMBtmNixIZSN/O+xj3FF257k94HBzxuosAjw0SMjQAYdfqXFKuOH3ddbww7DRPL9l6jJirAx27x/DRNCcdY1omjCu2eO7tB0pDM0Og16YrCn8ptIWM7Nwt26aXtJqizO6CvnGru5D2Bjxw== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQBPR0101MB1364.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:a::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.32; Thu, 15 Apr 2021 21:07:22 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::1c05:585a:132a:f08e]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::1c05:585a:132a:f08e%4]) with mapi id 15.20.3999.037; Thu, 15 Apr 2021 21:07:22 +0000 From: Rick Macklem To: Allan Jude , "freebsd-current@freebsd.org" CC: Richard Scheffenegger , Juraj Lutter Subject: Re: NFS issues since upgrading to 13-RELEASE Thread-Topic: NFS issues since upgrading to 13-RELEASE Thread-Index: AQHXMhVsKLCb5bI16E6Ydj5qgkLmjKq16AQAgAAfNj2AAAnjjoAAAQ2B Date: Thu, 15 Apr 2021 21:07:21 +0000 Message-ID: References: <902a3c81-2ce8-49c0-b163-5ffa4b90afe5@www.fastmail.com>, , , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: c7528aa4-ede1-471b-fb28-08d900527633 x-ms-traffictypediagnostic: YQBPR0101MB1364: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:3044; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 1ZCWBxiYnpFNZTCJndsXG2REGKN9LAcZU5QWwf7YJrXB6BA2+b78Q3VBlGGT1qWZABxlVziCnToAlavbgdTpP2BMsEzi73u2+Waj9ihKu9IBc6PZ7KMjWPS6bvdiPQQsBRLd0M9C/A9p+2QnBIt/mq9UPl15r022PH3pdD5M0W67VJJ05Q6wvFCu9+f1VfIKzl7gjo828kTNqlQYTFxqH8dXJV7/u6Gx5k2AI1yr93yejmCvXas0h+wnB+Scq451PpftZf+MEqT2QAasgt4yOoM6L675pyqM+q+LKIRBgKrtyrZsRTu5wqRTki9krBcB0FA6jekrl+8JxkVB85t2RUlpzjFeUxdtPc4sTzs9TiG5c82E7H+Jc+xk8mXv98pKVaU1QjWHDWDMLyWV5hbqoY8qK206uVyr1lt606u6KE7DU1bxc57zGs/p4RxaSyZF1LnW7EAKgCbsJIsbUEqP8VaCmZhWnEiQ+2C/tAWarbb4K7Xx8wYHd3RHcpIfFyQXwzjUPp1rlJ01Ow7KXsNRQWxWwUdK7BVScD1FI+0GjpLA15SQ4zh8b0hvoU/RezJwXswf/u9LjhDAVT5swzGnw8LuCibwDDgOUaHMEdWh4xImIge21ZhXfOdKGrO5SaKMwJR+zSCNCW+50nMnC4XjQjAobxRxfF4pwUdlZs5upp5yrpnrWsa2qFT3UX/Y4Oay x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(376002)(346002)(39860400002)(136003)(366004)(396003)(6506007)(4326008)(86362001)(33656002)(38100700002)(450100002)(2906002)(66946007)(55016002)(54906003)(7696005)(52536014)(122000001)(76116006)(66556008)(786003)(8936002)(66476007)(966005)(478600001)(64756008)(8676002)(316002)(66446008)(2940100002)(110136005)(71200400001)(186003)(9686003)(45080400002)(5660300002)(91956017); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?VdXu9D7ObWzRkfW43rUrZORY2ALDpb4rtJ0EJWF81FieUZjBg+8rWZwSgj?= =?iso-8859-1?Q?DTx8R1FX6uWrdmbRuRtyLWES+3iwoN5kq9Ww4GWbza+fKdG6E+OYOIwNql?= =?iso-8859-1?Q?NaWMxTEed9QRitch+1+nWnTO1GZqWT23R0v7t50lzMUY5a3R4MtlRXgDBM?= =?iso-8859-1?Q?G5+k0bbsLTzFooCMHMKRUlapVERf76Ry7itlLmtrD9RLBustRxlactiKpq?= =?iso-8859-1?Q?uLOie+zEHK7bezuNwTAbhKGSW+/0Z55BnusxRYoQ7kfXrqcPBtufN0etqf?= =?iso-8859-1?Q?YbavT2EvZNpAjiVLEZxzPnlhictO3yaO4w6W0OKKGBcRpH6gHxBayOopzA?= =?iso-8859-1?Q?6fTxKq5Mo+tHhEt21aJn5GKmfRdyoTsYxPNSfpSlJsv5VIW102tmYrvQWh?= =?iso-8859-1?Q?LjhYxySRE6e+f9jAFtjj0RfjF2kkUaJ5YEjN77uwdUnVWSuU7QDgvo1e+W?= =?iso-8859-1?Q?1MhaGmcwRB/rGZKKVI+x/QiFCiY15cpGQdVtxMiMdgAKQV+241pa9bTJLW?= =?iso-8859-1?Q?Wqbf6g3UFAQSEU5as+26pqy9tqqQZiUF2I1L+FW+nh1xbLwIq27ZQ7YXw3?= =?iso-8859-1?Q?wgBOwHjTQN0dKZhaTBuaEPMd436z856ru32W27UKI082xzlO7aRzxfYnd/?= =?iso-8859-1?Q?sTsSbF0Dc6JDgvEBXR73RqIkxb59G28IFbdzV6+71sVEI3VtCa+vuyZwW9?= =?iso-8859-1?Q?VaQi4KUs2TsinkDownrwJoP6arqAJxw3Maq4EWD04L+nu9muPfDpWgVYUU?= =?iso-8859-1?Q?TIqMRAD0puDuR8B/1vHemuHlrqx2kKNT08tLjHnN0B6JtgQJ89WSnO127z?= =?iso-8859-1?Q?QeeiNwQ3bMgH4fN1UP7wPQ9cPvcFmCFLBtQcM6AZnn2Xv31OUnNjC95vzV?= =?iso-8859-1?Q?4Q3xti25iJlUhkqa/aHdFB/j8lP6lTYcoqBKRiaLNFmdJ2YqgZ5h4+t1sm?= =?iso-8859-1?Q?BE6kOpJbcWW7sBaVD22nOdjaEQ2JhH/aWHMz92rFh6KeKSiqvp8xfO7d7U?= =?iso-8859-1?Q?XLrhW/Ucf7ak977R4ECNOFHk0RNOovBaKjTiwrxMq/1tyFfR49m92ILoEr?= =?iso-8859-1?Q?yxR2vLms1lP2EFkleY7bbpPRxJDJc+rl53ZxHlLr1nX5roBIwT+rvI9ems?= =?iso-8859-1?Q?x0/0rVm+1h7UrbBv/LjAA+W/pn4BFA/6c4ARSyOOrA7e+TASzIrPTA4d4Q?= =?iso-8859-1?Q?unx/UWrSk4cDX/y1CgVhWUq1LJo6AQNrW+b6TXQMu6zh7oiaVd/GZEWCwf?= =?iso-8859-1?Q?QNF4P6MDbnR7IVxvstJDBX/ZSil1NnrRPapFEBXTdJJ/bkXGdyySoXzhw1?= =?iso-8859-1?Q?W8cVnwpneVrUP+1ajRGi5h2iji21OjU8tmtDB5nmNxlypmgldZ+6HIw4/T?= =?iso-8859-1?Q?tPbB4uBhYo3zL4I07G8rzDvt05prbH035dIKneXinBoRSQwmNh9CfyUDIC?= =?iso-8859-1?Q?DiwQa6jC/eOAourZ?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: c7528aa4-ede1-471b-fb28-08d900527633 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Apr 2021 21:07:21.9722 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: mVcGsQskzwVCiNRjK+sN5Prx8DJMOaChGR/H/zgyJHWt5KR9o7WSLO7NwWTsDdmmJMqdDHsqdF7LMx+vCX1MXg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1364 X-Rspamd-Queue-Id: 4FLsNq48pKz4rmv X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=phR4PR02; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 2a01:111:f400:fe5c::612 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a01:111:f400:fe5c::612:from]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f400::/48]; NEURAL_HAM_LONG(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SPAMHAUS_ZRD(0.00)[2a01:111:f400:fe5c::612:from:127.0.2.255]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-current] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Apr 2021 21:07:24 -0000 Stupid Outlook...=0A= I wrote:=0A= [stuff snipped]=0A= >- Alternately you can try rscheff@'s alternate proposed patch that is at= =0A= > https://reviews.freebsd.og/D29690.=0A= Oops, that's=0A= https:/reviews.freebsd.org/D29690=0A= =0A= But you can figure out the link;-), rick=0A= =0A= rick=0A= =0A= I have not yet had time to test this one, but since I cannot reproduce th= e hang, I can=0A= only do testing of it to see that it is "no worse" than reverting r367492= for my=0A= setup.=0A= =0A= Please let us know which you choose and whether or not it fixes your proble= m.=0A= =0A= >> Any pointers for troubleshooting this? I've been looking through vmstat,= gstat, top, etc. when the problem occurs, but I haven't been able to pinpo= int the issue. I can get pcap, but it would be from the hosts, because I do= n't have a 10G tap or managed switch.=0A= >>=0A= >=0A= >run `nfsstat -d 1` and try to capture a few lines from before, during,=0A= >and after the stall, and that may provide some insight.=0A= >=0A= >Specifically, does the queue length grow, suggesting it is waiting on=0A= >the I/O subsystem, or does it just stop getting traffic all together.=0A= =0A= If the revert of r367492 does not fix the problem, monitor the TCP connecti= on(s)=0A= via "netstat -a" and, if possible, capture packets via=0A= tcpdump -s 0 -w hang.pcap host =0A= or similar, run on the server.=0A= =0A= Ideally the tcpdump would be started before the "hang" occurs, but running= =0A= one while the hang is occurring (until after it recovers) could also be use= ful.=0A= =0A= Thanks for reporting this, rick=0A= =0A= --=0A= Allan Jude=0A= _______________________________________________=0A= freebsd-current@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-current=0A= To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"= =0A= =0A= _______________________________________________=0A= freebsd-current@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-current=0A= To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"= =0A=