From owner-freebsd-current@freebsd.org Fri Jan 1 23:00:50 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 771964D713F for ; Fri, 1 Jan 2021 23:00:50 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670084.outbound.protection.outlook.com [40.107.67.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D70qj12V0z4Tc5; Fri, 1 Jan 2021 23:00:48 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=J3uxzkU+5w7DkzmsXvpGsQRjHI1KFFbIdTjiSduRcKI04ftyKRaXmyt9ffV8D7pVzd4ccIiRzGvJm+xCQZH7qX8CY6VqIhTJlMgr4eQ0uGTq1ot87eRZf9v+U6KgG/7sLuztExus+7/D1XrvTvjVOfavStKS+cINrOQd7qWHt6eLFGTqR29q4ZszL7GOp9mfpsyAwJRF5Ph3vUGJXqdi4E+u22O7GwqgrMdhnN2MecXplgYWt2KsjXogIZtrITZp8p2wE1Hh2Fc43R3a6z10x8Y3d2HJVBjs0neCXUgQSOQFpH2yBm41joA9Myj5VnLNciuQGfibFq4R2i5i9qsobw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jfAXfRgb3+jrbxfvKWaSqgWdT2nr1qX8x6p4VAUx7SA=; b=eBsq5uOwM/KQW/bcri68ATpIIpxf7ciyzOsL71IHtYhenP2WJXqm5tuwhKet1zMxLy7WVE/69D325v3oH3RIoK3LeT0C7vyqCOxYNHePxI7a5G+J4YB86oqFay15Tyaz+Ox1n4q0EHqDmPpQqoAV/yreG9BSCMCtYqcKnloYv0REX4qf/Bi0NHe+Slm7MMCkBX/oH3MfpY2Way68CjIJpT2i/d6dlNDy8y/62lUzve9UJrTsCRkbe/S6K9Q/5a6y/Tl8/cQ3hAhcYdZ3ACMnrt6JpXS3KIjK5JNAXl0N7Xcqzz3yiotG2r7xKdEs+y+a88abUHcS71i2gJGIUZmMXw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQBPR0101MB1364.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:a::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3721.19; Fri, 1 Jan 2021 23:00:47 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::3d86:c7f9:bc4c:40c0]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::3d86:c7f9:bc4c:40c0%6]) with mapi id 15.20.3721.023; Fri, 1 Jan 2021 23:00:47 +0000 From: Rick Macklem To: Konstantin Belousov CC: "freebsd-current@freebsd.org" , Alan Somers , Kirk McKusick , Mark Johnston Subject: Re: r367672 broke the NFS server Thread-Topic: r367672 broke the NFS server Thread-Index: AQHW3k3ynlL3lGhXRkC37zjbBa1PUKoPmNaAgAA7NVaAABFYAIAAwg6JgABvToCAALma1IABlq32 Date: Fri, 1 Jan 2021 23:00:47 +0000 Message-ID: References: , , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 0aa77b31-6902-446c-9ee7-08d8aea913a4 x-ms-traffictypediagnostic: YQBPR0101MB1364: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Gt7/DHbwqYjq7h/Bgv9OjCoteoVxvR1ocx+rEEwiNvoDSFqzYJqtDs2RNJTNWevd7X4lupHkEClovPj3ukQrHyMOEqM+S5L0k2YoSNCujX0eOHxuuLc2cyhZrl8M7rVvfAZwuZCBK8hvRB6XdbgsILskzpOJQyK3/gBfCDaSz88E8pLmRCd/s+i3Tfsug5Tk351Ld5cHAoBUz4GcAWMEpzmJKEidekV/6KdASStEmxKCblB4brfU+wAsuzGA2lJUWkxHF60ydMAcVOXk1+HXBNr2Pe/nL7+kbJigxRpCXI7irtxquU6pP4JhlH0ewBqj5dhyk3APWQSEp4IwxPhQYuOvBQ/KQ7fZDkOaLHRo1BnAbyYDgR0xvfATtnKZekJvFvyxhF+9F7xmXHYcvaArQUb9xFfXX2C1TFRIC5FiUC2DJR2tV+B02wB5/TfeNk36O60U8SUi1ZaIzxbc5EhKwg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(366004)(39850400004)(136003)(346002)(376002)(396003)(26005)(186003)(71200400001)(4326008)(33656002)(478600001)(55016002)(6916009)(2906002)(9686003)(66556008)(66476007)(52536014)(66446008)(53546011)(6506007)(5660300002)(8936002)(64756008)(54906003)(966005)(83380400001)(91956017)(8676002)(86362001)(76116006)(7696005)(316002)(786003)(66946007); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?WVWZ6dY/uR4v95iptSTCvzEg9hcnwsiDkhWeQa819AvII/h+bJhgAB/RxPKF?= =?us-ascii?Q?8B2h+uJNQFZLcbH8LrKxFjkQJfFPFzZYv7IzRpD0o9XC8DuSwclRDwcoMDnj?= =?us-ascii?Q?ir6Gx2OD9hdXGIvCjg9nBsu18FLe5xvmPP6o9OQowLgbpHm1iD9kpTbHsAU2?= =?us-ascii?Q?JmJuYy41OzAoD5bYT0zZMRwvkolpkpON0TZH1/ssaDAZKMWxdsTCURG8SOf0?= =?us-ascii?Q?S5xpcv1sXV+RePoTPBoQNT5M7JmNphYTzaQxsX2tFg/WqCt6ahfnfi8MRJEc?= =?us-ascii?Q?JC/aDNReMM1aFvFQGQfl1KghzxFXngX6trWzgC/3RD3VIPXR7/ypeDgEe4t6?= =?us-ascii?Q?CWX59teLQX7Lg/Al2ESPdZGsiSmRmHEe61gq8y5GvbrNSCePrCeeLa3kNU3N?= =?us-ascii?Q?1WJJPMgOHVqF28eF1SSXNb1MkDdH2vJFrdxlxHfl5Cf9mkgd7I48bZ9XnyG9?= =?us-ascii?Q?iKK9lgQlMlQXUamLHskeFqzMbYUB58T4c+P38nYHysFaXaR9Oelc6tLuHIwE?= =?us-ascii?Q?P5DZMMbFBQiZZDU0A2uDoUJvdK6/9Fv5Pz8UNctoPOch+Hr4kXYVOQS61Nd6?= =?us-ascii?Q?QOb8mSuGg5P38zGng7/q/TdpgnRdUonXoKo0HwC7Y/m1j/cR8jfginY06XFD?= =?us-ascii?Q?OAKXN3j98Js4JY6KK3cxaxGM/1g543ubxrj0lU1zF3zsznmuLjj/ZXBpBfzb?= =?us-ascii?Q?6SF/ZFSR4dU/95OKpdlPoKaHxlpnToPcOPfzh04Z+vPdbLtyuj+4urC8uVp4?= =?us-ascii?Q?bYJ9c1iXgRol9b3PidZWxaTbN8p3/SMB0IwQICsn/eqjiTh/U3AMHQRnLsqW?= =?us-ascii?Q?Dv+LrspH593yiIbIIz+rV5hAW2yu5KrllH0TRPLynqei55fy4i9AHvZTiSxB?= =?us-ascii?Q?tUNGwKHe2QRk/qaNmN9yeTWU0/lz8wU44yBeY/fBmVz3AL8uRqDX9xUFlhrO?= =?us-ascii?Q?yZbiPjcMRVL/Nnb4BWvqM+FZ3oPxbkWqdsmucjZI7GM=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 0aa77b31-6902-446c-9ee7-08d8aea913a4 X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Jan 2021 23:00:47.4667 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: aY1BezYk3SkvqmjKNOBbfrw60bHEMSMyslKIKMBpHLk2wI95qa27YeNPpR0bPpHjCPd1dA+y2SQvs1FfGaIbvA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1364 X-Rspamd-Queue-Id: 4D70qj12V0z4Tc5 X-Spamd-Bar: ------ X-Spamd-Result: default: False [-6.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16:c]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.67.84:from]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; SPAMHAUS_ZRD(0.00)[40.107.67.84:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[40.107.67.84:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.84:from]; MAILMAN_DEST(0.00)[freebsd-current] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jan 2021 23:00:50 -0000 The patches that I believe fix this are now committed to head. Have a good 2021, rick ________________________________________ From: owner-freebsd-current@freebsd.org = on behalf of Rick Macklem Sent: Thursday, December 31, 2020 5:56 PM To: Konstantin Belousov Cc: freebsd-current@freebsd.org; Alan Somers; Kirk McKusick; Mark Johnston Subject: Re: r367672 broke the NFS server Just fyi, I have put a patch up on phabricator as D27875 that seems to fix the problem for all NFS client mounts except NFSv4.0. NFSv4.0 will require an additional fix so that the "seqid" is properly maintained during redos of the Open caused by the ERELOOKUP redo. If anyone is running a recent head kernel on a system that NFS exports UFS file systems, please test this patch. Peter, can you test this? If acquiring the patch from phabricator is awkward, just email and I will send you a copy of the patch. Thanks, rick ps: If possible, I'd like to commit this patch in a couple of days, given the FreeBSD release schedule. ________________________________________ From: owner-freebsd-current@freebsd.org = on behalf of Konstantin Belousov Sent: Thursday, December 31, 2020 6:40 AM To: Rick Macklem Cc: freebsd-current@freebsd.org; Alan Somers; Kirk McKusick; Mark Johnston Subject: Re: r367672 broke the NFS server CAUTION: This email originated from outside of the University of Guelph. Do= not click links or open attachments unless you recognize the sender and kn= ow the content is safe. If in doubt, forward suspicious emails to IThelp@uo= guelph.ca On Thu, Dec 31, 2020 at 05:16:27AM +0000, Rick Macklem wrote: > Rick Macklem wrote: > >Kostik wrote: > > > > > >Idea of the change is to restart the syscall at top level. So for NFS > > >server the right approach is to not send a response and also to not > > >free the request mbuf chain, but to restart processing. > > Yes. I took a look and I think restarting the operation by rolling the > > working position in the mbuf lists back and redoing the operation > > is feasible and easier than fixing the individual operations. > > > > For NFSv4, you cannot redo the entire compound, since non-idempotent > > operations like exclusive open may have already been completed. > > However, rolling back to the beginning of the operation should be > > doable. > Turned out to be quite easy. I'll stick a patch up on phabricator > tomorrow, after I do a little more testing. > NFSv4.0 is still broken, because it screws up the seqid, but I can > fix that separately. > > I do see the code looping about 2-3 times before it gets a successful > ufs_create(). Does that sound reasonable? In the simple case, it could be described as is: ERELOOKUP is returned if the parent directory cannot be locked sleep-less, and we have to drop the lock for opened vnode to sleep on it. More elaborate (but still not precise) description is that parent directory might also need to be synced, in which case its parent might need to be locked, and so on recursively. Slightly reformulating, I expect that ERELOOKUPs come out in case several threads create files in the same directory. > Here's some debug printfs for the test run of 4 concurrent compiles. > (proc=3D8 is create and proc=3D12 is remove. Each line is a ERELOOKUP > retry. This is for the 4 threads, but I had the thread tid in another pr= intf > and it showed 2-3 attempts for the same thread. They should be serialize= d > by the exclusive lock on the directory vnode.) I cannot make any conclusion from the output and its description. Are there opens that do not result in ERELOOKUP, i.e. does the op eventually succeed ? What is the ratio of ERELOOKUP vs. success ? Also note that any VOP that modify the volume' metadata might result in ERELOOKUP. > tryag3 stat=3D0 proc=3D8 > tryag3 stat=3D0 proc=3D8 _______________________________________________ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" _______________________________________________ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"