From owner-freebsd-current@freebsd.org Fri Apr 16 23:10:16 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B16325E4C55 for ; Fri, 16 Apr 2021 23:10:16 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670085.outbound.protection.outlook.com [40.107.67.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FMX474wlCz3FsC; Fri, 16 Apr 2021 23:10:14 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PPiGJN1Cl14lFkxqzF9QZncUMKROVTcR3oLaI1870I1Fjv6Hj5OKnTZgyGScaO+aQPUgkpxGS0IkD2zakugtysnTstaGn1nM+kIx8WBFmJP6WBbKPikd+UXj0WuaGnEsV6RfYOZY1d0v2ctEDtMplpQ2/q0xTpam87kQUmkKiKqdwGidakOju1sgvQg6JoHPWa9yvag+Y8W2FdYVIhA3356bR51a2MWycrH13n7avIt8f2oNlgjzDktsVuOZl9P7TZunIc7/J6dsRuNt7UTKfOB1koZbJFPdzFhl/RY/Vlwn44skDk3sUn9cnj5hBwaRW1/jXH+PiaongfLxrwA36Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OqNoXc+l0y/aWKppaNr4GeMUETav4jbnMZEQRwH68qs=; b=Hg7mws1yiUVeXd+SmAXhgjY2dKi2JfhFd74NNv7d9dloD/x4xbZL19yJSNDCuvT/Cyho8oMsYgjkzfblQfUNFj/Mub8AXiNGL30msfV9GD98hXhGeApw81CiU5EPZ+Oi6uLTJxZ22cxP6ipAYar4Gxo1nvcsfOVB6hBT/RZ/ltoul6IH9KslctDMV1cYqoP2eaIC2lIN7gQxwo/9O8JFQop92OD4u+0RyDQE+rzvrdRXGFOgQ8mypGGJS9TxSjy6kS335RKmfEzXqHiYKo+MlNrwUMp3NHIkTdi5l8hbV+CVGkP9mnip6qmgtrsPxd/gzy19VdNlc4o53AVuzhu1Gw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OqNoXc+l0y/aWKppaNr4GeMUETav4jbnMZEQRwH68qs=; b=K0RcRw0A+bxOC3EPDsqlDaWswODIX2q8o0QJlCvPrG+qF+1oFKRKADDQL/MysWLsWPQJzzZY/kuFs1AG0q1Q0J6hKFjnehjLTCif5zljQ77zOwPKhY3pLz9HYKFtOB+GW2tQjtjbhd5cOY7cSXz2E/iO2gFtWGB8/5hbUPx2lelGFXJ9seL0BdzVNC4xxJ8Y2GlvpOo4YnBap32T+DDiDStx3trI8JNhkzUARRsses+rzzqdEcb04hFd7JQ/BqFgK9xYL4RHeNyFjACvfU36qjTNbZTwYzANiu6lUZ17XhSZmYvgSSibc34rqtv/+w7ibXl9muvVscOxzUUUA4CmFA== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQXPR01MB3912.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:4d::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4020.22; Fri, 16 Apr 2021 23:10:13 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::1c05:585a:132a:f08e]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::1c05:585a:132a:f08e%4]) with mapi id 15.20.3999.038; Fri, 16 Apr 2021 23:10:13 +0000 From: Rick Macklem To: Olav Gjerde , Allan Jude CC: "freebsd-current@freebsd.org" Subject: Re: NFS issues since upgrading to 13-RELEASE Thread-Topic: NFS issues since upgrading to 13-RELEASE Thread-Index: AQHXMhVsKLCb5bI16E6Ydj5qgkLmjKq16AQAgAAI7ACAAAPdgIAB0deP Date: Fri, 16 Apr 2021 23:10:13 +0000 Message-ID: References: <902a3c81-2ce8-49c0-b163-5ffa4b90afe5@www.fastmail.com> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 9bd929d7-ecfe-4534-a1cd-08d9012cca8d x-ms-traffictypediagnostic: YQXPR01MB3912: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2399; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: vm99TogCH5XYnPyqGlw1+hnISi0dRHV4KWxkD6JyDu6yfg5oRgw/omPtpYflewERJY4ifci7C4ZNGdIL6Ns+JmFPnYzUiELRQIgKQYqvYfsgupiWDJd3NNAnOcV4EU7S7XSCY2iVKBx6wNJVJ1eajwAWBClKGBmfWo7zQ5986o5ZI6FeFezotlsfMgzAJrl0ckx5gfySeyO/dhOc8Eg2x21Xicbu1e+7ePdQCJ2rEpnfg9izebJg5JLhJeBXohAKjtW264fnKUqO2HgXj09hH5gS9p6fNOdoYXW4duSKHdmxCixGvFqUxS8OdNNwzBA43eaNiDP8a1nS0OBV2I3/vQBtUyqUXP15gQl1dAhGOlAD4eDFz3fGl0rUG4B6Q9aM8+KeESqgerzJmzlajdfseI5evPRpazdbTmkQIdWrxII+1iD9JvqYYE9znnsdVWDmJK+RFhICX/YoYZZybnE/NzQiNpB6otvqaY9E2qhtBvjYLsRHqXsERwqrWGPNxcbm02+ty445MAxHiu4de5iSB/Ak5rIGQ4O2+1jVzSUmgwrnS90+zCSI49W4w5Ficiklddq5GAa1byA33F7epPv7VlvvBgWJZrM7sAH19PgILn0U3BbRBgDtVrLprjn4MUWdiH3M5jwaeJcOAxhKTmoaL5QP0aeqDKRZYn7mVG2xY2aAhV7xJE3HPt90Xi2v9Syo x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(39860400002)(376002)(136003)(366004)(346002)(396003)(122000001)(38100700002)(71200400001)(64756008)(52536014)(786003)(316002)(55016002)(66946007)(66446008)(9686003)(6506007)(53546011)(7696005)(5660300002)(110136005)(66556008)(91956017)(2906002)(478600001)(76116006)(8936002)(8676002)(4326008)(966005)(66574015)(83380400001)(186003)(66476007)(33656002)(86362001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?voHsZj086ee4tBSV7nC4WsqdhPeBvpFtRQh/ekBExap8fJKV2Qte0IABGg?= =?iso-8859-1?Q?mcIMktWQWA+eoabHEkLVy3D62gjdEkMSw++v6QABHC8IQCfHPN0iJZxMiQ?= =?iso-8859-1?Q?JAtLoBwcNSdfPD0Kx2psEgltcZMeTxxn9LZ9uOrcmazEX3r2q+iHKRseP3?= =?iso-8859-1?Q?rP6OBQcVTItcRqHYiilu1Ak4c2n2AJVVVlHjTLc+0cirgEdoHH+WbeJ1K/?= =?iso-8859-1?Q?i2QWL8s/siEWCNvHB9xile1/+vjGVk27TrvsH27wCs8WsfgGhboSuu5c6I?= =?iso-8859-1?Q?ujHEkISGVmBJ8kc5rjWmEvv6tMSVdEc7gfXinni0R03C3y46eyg85pHaCP?= =?iso-8859-1?Q?sfCDhIGpkijg2IlW5HQ37epRzYHW8S0LLB+HmSsVQlyJdjy3J/KPrgtDVo?= =?iso-8859-1?Q?NLItykUEsQpclhS/IY2RYykfOyPmoomW1HsB1fTqrmSsRw5W+nhzfbi8PZ?= =?iso-8859-1?Q?aIeUWpM80r6ati5KAMM7mgcaILhj8uOJyO7iMkZEAXAw5L46mUBmB5rK0a?= =?iso-8859-1?Q?nfzz9wW5BPaeUbym3ciJn3sozIe411ijEVG+MZiEQn/cGsb2HYqbYx1wfh?= =?iso-8859-1?Q?unSoxVFPc+3462T5q9J/Fy23o1jT/9MaDmQFI8nZ2Tgnsr7APRtQjZGXqT?= =?iso-8859-1?Q?PAj7wXwXu6CfnUJNVdDwYQEz/s1NfhRNr1tlgrPdccd3CVKYV61UabcqAs?= =?iso-8859-1?Q?YSPUXALwjjkEalGDc+nk12PKZaVzri83A7ASZWuPv+FIuBwWem5dv6vAo8?= =?iso-8859-1?Q?UeJMKRlI+GI1amTx18xJAxzmsgjyQj+iAUoeX2xECefRuCXBlQTRz7Zr2b?= =?iso-8859-1?Q?+iZCi42MlLPr7VWFrKQCxaRfXUiE1Wyp17qUNdk1FIH4L391q0ZgY0oMLs?= =?iso-8859-1?Q?zUDCfu3CQMGb5tFLdmQ1UpvfVceRzu9POmPpBJLVFPk9Xq++mPpqFn+B8a?= =?iso-8859-1?Q?M6uuDVGtzqECFMqbqZlcaSVuv96UBnkmXKDTLzzzm9MG8ybioLRo19nbZx?= =?iso-8859-1?Q?XKdTdbaRW32QDJt+lavjiCwBkvNb+M3iR9NuCdcohZXbStpObzGZuC6Q0j?= =?iso-8859-1?Q?EOrAI0w+/JTA3P6r/fKsiO5olbc+spLU/ZvPBfom1d8ieXo6HIq+X0cZf/?= =?iso-8859-1?Q?zilRPe3By940v06TGJT+ylQNeFyopW91nE8du7P0cgftiVf6VzTuNU9Dmm?= =?iso-8859-1?Q?0TwrG8gLqWvakJ9fIpIlDPepJowklcibDDOd5pGfS1ajKOuXb/WrUelT9d?= =?iso-8859-1?Q?iDVPRqyFRH5xK37yjegDSeq034IpWlt8bI1iTSy4lfSKMKH3cMAE0bPbpZ?= =?iso-8859-1?Q?GbdCB9ceq838g42i/C94riv1WHx7YeS/ceDAyNqOteiX0Q1tePwIJjOBXM?= =?iso-8859-1?Q?9b6FELDNeZ94OeKP9jUgUjMCIrZ4IirCJkl8NJM67ukRL0DRxFpEcP9hh2?= =?iso-8859-1?Q?VgcYRmiAn+xgmZrp?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 9bd929d7-ecfe-4534-a1cd-08d9012cca8d X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Apr 2021 23:10:13.7114 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Qr2oI9OyszvCaE7LJ/U4QznU+mckv8sRZjp8Y7sSIoIJjOqJZLN2I/MN3vQECSpOhdVb3CvzbfWsub8YPFMpFw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB3912 X-Rspamd-Queue-Id: 4FMX474wlCz3FsC X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=K0RcRw0A; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.85 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.10 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_IN_DNSWL_LOW(-0.10)[40.107.67.85:from]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.67.85:from]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FROM_EQ_ENVFROM(0.00)[]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; SPAMHAUS_ZRD(0.00)[40.107.67.85:from:127.0.2.255]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.85:from]; MAILMAN_DEST(0.00)[freebsd-current] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Apr 2021 23:10:16 -0000 Just fyi, I just got a "recursed on non-recursed mutex" panic in socantrcvmore() with the D29690 patch, so you might not want to test with that one yet. rick ________________________________________ From: owner-freebsd-current@freebsd.org = on behalf of Olav Gjerde Sent: Thursday, April 15, 2021 3:21 PM To: Allan Jude Cc: freebsd-current@freebsd.org Subject: Re: NFS issues since upgrading to 13-RELEASE CAUTION: This email originated from outside of the University of Guelph. Do= not click links or open attachments unless you recognize the sender and kn= ow the content is safe. If in doubt, forward suspicious emails to IThelp@uo= guelph.ca Well something do happen if I restart NFS Service on FreeBSD , it works for like 10 seconds then it gets unresponsive again. This is my output from `nfsstat -d 1` 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 8.00 1025 8.00 8.02 17170 134.54 2.01 72716 142.54 0.07 51 34 8.00 2273 17.76 7.99 31273 244.07 2.01 133267 261.83 0.14 20 82 8.03 4889 38.33 7.99 25885 202.07 2.06 119340 240.40 0.13 21 81 [=3D=3D=3D=3D=3D Read =3D=3D=3D=3D=3D] [=3D=3D=3D=3D=3D Write =3D=3D=3D=3D= ] [=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Total =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D] KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s ms ql %b 7.98 8811 68.64 8.00 12997 101.54 2.22 78396 170.18 0.15 1 80 7.99 922 7.20 8.00 3798 29.68 2.10 17965 36.87 0.09 0 11 8.07 2959 23.31 0.00 0 0.00 2.67 8938 23.31 0.86 32 72 7.97 7088 55.18 0.00 0 0.00 2.66 21233 55.18 1.05 16 98 7.98 4666 36.38 0.00 0 0.00 2.66 13986 36.38 0.36 9 29 8.00 4513 35.24 8.00 7662 59.86 2.20 44188 95.10 0.27 10 49 7.98 4799 37.40 8.00 11422 89.23 2.16 60076 126.63 0.19 0 51 8.00 4322 33.76 0.00 0 0.00 2.67 12967 33.76 0.89 0 42 8.02 4839 37.91 0.00 0 0.00 2.67 14550 37.91 0.54 17 41 8.01 4516 35.32 0.00 0 0.00 2.67 13569 35.32 0.57 27 38 7.95 4459 34.62 8.00 1195 9.34 2.49 18109 43.96 0.55 0 45 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 On Thu, Apr 15, 2021 at 9:07 PM Olav Gjerde wrote: > I have the same issue, using Ubuntu 20.10 with Linux 5.8 kernel. The Linu= x > NFS client will get unresponsive and it does not recover in my case, even > if I restart NFS on FreeBSD. I upgraded from FreeBSD 12.1-RELEASE though. > > On Thu, Apr 15, 2021 at 8:36 PM Allan Jude wrote: > >> On 4/15/2021 9:22 AM, Chris Roose wrote: >> > I posted this in -questions and someone suggested I post here as well. >> > >> > I'm having NFS availability issues between my Proxmox client and >> FreeBSD server (10G link) since upgrading to 13-RELEASE. And unfortunate= ly >> I upgraded my ZFS pool to v2.0.0 before I noticed the issue, so I'm kind= of >> stuck. >> > >> > Periodically, the NFS server (I've tried both v3 and v4.2 clients) wil= l >> go unresponsive for several minutes. I never had this problem on 12.2, a= nd >> as far as I can tell it's not a disk or network I/O issue. I'll get seve= ral >> "nfs: server not responding, still trying" messages on the client and a = few >> minutes later it usually recovers. It's not clear to me yet what's causi= ng >> the block. Restarting nfsd on the server will resolve the issue if it >> doesn't clear itself. >> > >> > Any pointers for troubleshooting this? I've been looking through >> vmstat, gstat, top, etc. when the problem occurs, but I haven't been abl= e >> to pinpoint the issue. I can get pcap, but it would be from the hosts, >> because I don't have a 10G tap or managed switch. >> > >> >> run `nfsstat -d 1` and try to capture a few lines from before, during, >> and after the stall, and that may provide some insight. >> >> Specifically, does the queue length grow, suggesting it is waiting on >> the I/O subsystem, or does it just stop getting traffic all together. >> >> >> -- >> Allan Jude >> _______________________________________________ >> freebsd-current@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.or= g >> " >> > > > -- > Kind Regards / Med Vennlig Hilsen > > Olav Gr=F8n=E5s Gjerde > > BackupBay Gjerde > Madlaforen 35 > 4042 HAFRSFJORD > Norway > Phone: +47 918 000 59 > -- Kind Regards / Med Vennlig Hilsen Olav Gr=F8n=E5s Gjerde BackupBay Gjerde Madlaforen 35 4042 HAFRSFJORD Norway Phone: +47 918 000 59 _______________________________________________ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"