From owner-freebsd-fs@freebsd.org Sun Jan 17 05:03:57 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BCEB94DB645 for ; Sun, 17 Jan 2021 05:03:57 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-lf1-x133.google.com (mail-lf1-x133.google.com [IPv6:2a00:1450:4864:20::133]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DJN9m5wJtz3vlL for ; Sun, 17 Jan 2021 05:03:56 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: by mail-lf1-x133.google.com with SMTP id u25so19290384lfc.2 for ; Sat, 16 Jan 2021 21:03:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ccRefeo4YsgltQRYwQq8iPyHcvntSzXaME7IeRkev8k=; b=UJrG1AMhyysrD+LKPViBM+VUXDVwFxoIGGCgLXK6WdmqajMvZrj3eXo5Ghr4RvGZ0G OtuAGbEZJJE7vWB2cc4+idveCSyK8mF063rjTr8SSxFbpe6FC1Oo6FGgI3h8K0qzoh7g mcAscG0uvi+b6QggqQU9ub9mohMnno3AhqMXaKxGQjBETI6awbARJirEfnJsXd4yn+xl p9ltlaNjcE/NAAaN1BOLc+AKXOCN7n+w7PPx02lCppxX1Vx6narmr1OiowulquotNE9/ 4h3RKNnVO1DvTTG6DG7alaUfU3O++NkUpHYTmzD/50RHfFT+OVocZD34ZsVJwTslbPn5 BH/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ccRefeo4YsgltQRYwQq8iPyHcvntSzXaME7IeRkev8k=; b=iCwGkPtO1fqWvlPiCAlhoTSSb/2DDVgmA9sr+y+V0yRpyknhBjQv06v8jHIAf3RmfC hfXmnMPiswDvs/r40Bhvyf/NPJyPflCWMIfuRxQ4Q+qQoUmrq/bDfwS0PwUSAmhWiKVZ N5cw9s/Ww4S+WZbPHGvDS3Bf8eYZWQiDqlJwBFdQexwuMyEZugdZGQnH36T94UrdnWnl RAgSanypa7jZdVYR9MmBy/Yfn5EApDvHliSIeuJ49ndh5qkAy95jsQO7AocEF6HqBjIv 04Mj+WcQpxYEOs6K5otIh76pwfb4YU92187mmguGYzO3QO0+XDXe3pvLyfHhs7Qw4Acn aX1Q== X-Gm-Message-State: AOAM531anRe4ogKSVxZBPBq1+vrvguR0ZVze11RWrdAMg17y/IW61k5X tlAp+DiDpUs7mo5sGb6GB908evflH/viZATaW0I= X-Google-Smtp-Source: ABdhPJx0zDtf02PPVY0Y+LrdnfuYfCEumbqld2jkqYy2m8tsQu8+3LBhogb52daWAiw3jC0iQ7+qFSpRKdISt1byjbo= X-Received: by 2002:ac2:5450:: with SMTP id d16mr8909784lfn.309.1610859833604; Sat, 16 Jan 2021 21:03:53 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: J David Date: Sun, 17 Jan 2021 00:03:42 -0500 Message-ID: Subject: Re: Major issues with nfsv4 To: Rick Macklem Cc: Konstantin Belousov , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4DJN9m5wJtz3vlL X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=UJrG1AMh; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2a00:1450:4864:20::133 as permitted sender) smtp.mailfrom=jdavidlists@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::133:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::133:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::133:from]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Jan 2021 05:03:57 -0000 On Sat, Jan 16, 2021 at 5:57 PM Rick Macklem wrote: > Opens can be closed until all FreeBSD open file descriptors for the file > are closed. > --> Just the way it is. It is not an unintended leak. They go away once > all file descriptors get closed, [...] > However, none of the above seems unexpected, except maybe for why > "ls" in the chroot opens 3 regular files each time. I don't know what > chroot actually does for something like "ls"? I'll look. It probably isn't anything ls is doing. It's probably just the open of whatever shared libraries ls has in common with the shell running in the other window. And so, on a long-running system with frequently-accessed binaries (or just shared libraries) mounted over NFS, where it's unlikely that all descriptors will ever be closed at the same time, a hang/crash/failure essentially becomes inevitable? Is this a solvable problem? (Outside of oneopenown.) Thanks! From owner-freebsd-fs@freebsd.org Sun Jan 17 14:49:18 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A16524EBD0A for ; Sun, 17 Jan 2021 14:49:18 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-to1can01on0620.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe5d::620]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DJd9948KSz3Ht7 for ; Sun, 17 Jan 2021 14:49:17 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=b+0KvMyl72Ojq4/aaJTkMqvdxTSyI7+5BZC/8cxwUeCh+R4BOY6mXr/zk/QXrBnvUgCyWrfQvc8X1GzZI3uhl9yGcOVh3X2NQxKU/TXyLbtizYXI1R+p4yUW86Pd1Dp/NSTRNzFp5bkfXdd7pHbGnYXiDgk7eEt1JQu9+e2Qj3GLsKSs5RrPiCginDYNr9XHENN8jJY2qmPTNHMjK82sBaOjjWRNWU2Lgn5DJEqw+znOGUytGZ0V7qnFgCrZgv0nLkWlWPlGOReyzAa42/XxliPo01VnvIcm0dQj+479tHB+bfiforzNrV4wCCNE4lessBD4B1kUe5TthHbzOwSlPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rYPyiK98fDSCHVlpKhwECSPe6s4KII1Ilx2hn9mdpVM=; b=jRO3+UmY7jI8kmwTGGnYBlg+R+al+xQNZS78c6tKZTU9CyAJ2IerR4ZjcgQ/xxT3JZz6GuI+gtAY5dIKI20dN4xqgVQ9bZ1lnx7JmGW9IG1Z02/wZ8y2w1dH/6rF+Mftj63MHWyla4fRg33QHsg1PKW2H1iH0FIjZ8bvPcn9kkYMpOTp+zxhLiNcXcCxrCNr7DqcAfgnW3pSphUFXdrNfnsTw/T/+rBPWJ45v7vO1Vh12EFGcldV6BzuiUV4U674L9pdFy7ES1PUko2WCH3hTf7NHru9y57COTfYgU/1W6K1bUF9i76GwrNXQ54gSubgxBBTazo6mp8KaPhKb4gE4A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rYPyiK98fDSCHVlpKhwECSPe6s4KII1Ilx2hn9mdpVM=; b=OtRWPBU8UwYM3x4TkwsB92h9u79j6kowZ4arNnGRQCVq2q3CGwGIBgBFExQaGalj4kH+iHOLh/GULdBVcNdNE4kin1sVnnnIwVaRQSc8bdzymvKxi5wRKiqTGuO/t1HClgZtM+Y2UTU4q37qz5Y1pQvB6XMaxSG6JIuEvSalbw1tGi399pECnVXtSuXEWJu6qvCRT0d7D4tLEhD7Xqf0xxRR3PXwHp2tqKFpukAvrWOOy14wE5jnPaSLdvm4FPApHEX943LwHyP/ozDRbjW+EzGu75LMNRndHnLSXrNqyZv4kEto+z+DVwMAGNh1qhYK80cwIfWaPu7HbXEM61Or5w== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQXPR01MB3909.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:4e::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.10; Sun, 17 Jan 2021 14:49:15 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::3d86:c7f9:bc4c:40c0]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::3d86:c7f9:bc4c:40c0%6]) with mapi id 15.20.3763.014; Sun, 17 Jan 2021 14:49:09 +0000 From: Rick Macklem To: J David CC: Konstantin Belousov , "freebsd-fs@freebsd.org" Subject: Re: Major issues with nfsv4 Thread-Topic: Major issues with nfsv4 Thread-Index: AQHWzw/HDat+dHoH9kKG5K3Xpd53kqnxDteQgAFi0QCAABTa84AALLCAgAAVvcmAAAu0AIAA4wiAgAAI5gCAAO/IAIAAiDrmgABWOgCAAGv4AIAEv9nSgCyfuACAAIn3C4ACDQ+AgAESerWAAHpKAIAAoS7J Date: Sun, 17 Jan 2021 14:49:09 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 98fca50c-401f-48b1-cd2c-08d8baf70c31 x-ms-traffictypediagnostic: YQXPR01MB3909: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: E+qv5Svy96/W++IN9Z9Uf+73EXADKVdHuUNaFMf2wgKasNq0sv42nvvh5GGel9segGyMihD26prZMKRIwXUsg0DkKO9b9p2m3jlQLBloJqHjKg40mA12EfiogH4YBrytdSCX/p24Q4nZUVDB/3E9QFzZ02vjSSj/8SDlnCF+okMVYLnvuzsaPOKvV2aX6KQgRDp9W9uECLBcLBvJvVgCY1lK7c9bD3LeTzyKbQZ8UlSaMDH5PxKmJLvw+lRC7v18AmiUXUaTS1iBfKxW6B6NnIOWUc+Cmb5ucQsu2o9c9KNEVKt1mdRhHqv3jTMUWRRBQFDNOCJAgVrrKqPyow2yJhrTbtJKGsnAT2/UR949Es3CYegAru2WnysLAAJzPlRbsDe+QfyqrRBvVZSf6YuyF1spexpNRaZx9oCT0zihwpfDbpSbUTERGg30eUzlwmRrczI49xHCZdRzUOS1+LYUEGdsITvzQS2sLv0BC54hI12qPf7wen2lWITYDA+Jc8nXSy1UwzlsStM1AdzBlzq5dA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(366004)(39850400004)(376002)(346002)(396003)(136003)(55016002)(33656002)(6916009)(7696005)(186003)(6506007)(86362001)(66556008)(4326008)(83380400001)(71200400001)(5660300002)(91956017)(66446008)(66946007)(66476007)(316002)(2906002)(8936002)(9686003)(478600001)(76116006)(64756008)(52536014)(54906003)(8676002)(786003); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?TznECn8E6F+APV36mC3KzEAC3iLnjHw+Qyu6PNwK0ncuqClLs0axoBQKRn?= =?iso-8859-1?Q?cnZOSTbpZIGuP9m5Hy4/yBFSfIGEmICR6SZEBgLT6PnH7WZvcKvcMo/uPr?= =?iso-8859-1?Q?vim8MkZr+0xL1o6+jXs1OGM36R6a9ABKzT5zgq8Vvx6A/SzaokUzk0MNu9?= =?iso-8859-1?Q?XU+4V8W6EVU5pi7KRVWOO0MYxWp5SA+cJDjSSrwcuklFjKnFsGV+FmL5Gb?= =?iso-8859-1?Q?dz+xaYvMB1KaDYuE7fWenYCEzSRFYwWY7DLZE1L4hDnWoM0oamP/UPgTRw?= =?iso-8859-1?Q?g6gzlKrATmlOOlbQ762RQTRWcd0OJc3+S2E0TW0EauZArmse4MaaIpGgXa?= =?iso-8859-1?Q?4lTwl2nDS38VULZh6kvt8tG3k2TTCzZDA8s8/PJx1qKjtvnGdpO9/ivoJ1?= =?iso-8859-1?Q?hWMBP8yYCF8AJ1HtOaW39ipQEsnBcILm59bkirZ5UCyMqVgqG63ZLu/EHg?= =?iso-8859-1?Q?09KjpfRT0fMWa5zIEZXK6BA+6wDHPNaCJiqnxEdojTuispX0zxTEPvRBqZ?= =?iso-8859-1?Q?DLEvDsap3L2a2uac8hBk8UqPsDHeOjZYf5d6xXSIUTHGN9EMEiu9T6m4hx?= =?iso-8859-1?Q?krShIC/mLQxIubRpS+HTPJzWwKjCGCtZYhGd8fy62RfFwIfg3TMSNLRu/x?= =?iso-8859-1?Q?KHEqQOH3oLJOWu6Jpuv3Ap2ASDv5Wg/D6qLbUqlzz6jFEtUr2Fwvmqzmj4?= =?iso-8859-1?Q?V582fndD9CSsYJZ/boDwkNRY70gFPUi+A6v+JOqqm9B7dPcKmL1KEANAWT?= =?iso-8859-1?Q?+A++cBV461Sl/FXgokr2DtScPMY9KSVQ6Qut+oc7LJgNXwCsYIYNgdCKDe?= =?iso-8859-1?Q?hrQeLaXs5IvajYAtDHi0I1eRSWE87OGjqUTtZCa0gsnb/a33RECGTaxOqt?= =?iso-8859-1?Q?0FT6nxNob0L0f+kWIwZs86JPGiZH7giLhY8k2RCWtbcCELpvSViyJ3+7Y3?= =?iso-8859-1?Q?FHqf8+NqEf9SkILCewBS+YuwOB23dxVFYaKa0u0mMhTprMPIy8Y+wE/njK?= =?iso-8859-1?Q?xwTg3rUXW/UpkgCDdfkqzUq02d5a49rdi0s7dl3ToTokTSg4z+7bSGVUrP?= =?iso-8859-1?Q?o/aq+RnNyLCuUTPSTjhahv0=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 98fca50c-401f-48b1-cd2c-08d8baf70c31 X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Jan 2021 14:49:09.6654 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ktR6KPJcmDIRNknmieZA2rS3HK+YycFzhA+nsLsGF31peKqJALkvKrulthrjTgUMDIWBK5L7KHvXlA/jwWmIMw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB3909 X-Rspamd-Queue-Id: 4DJd9948KSz3Ht7 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=OtRWPBU8; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 2a01:111:f400:fe5d::620 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f400::/48]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a01:111:f400:fe5d::620:from]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; MIME_TRACE(0.00)[0:+]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[2a01:111:f400:fe5d::620:from:127.0.2.255]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Jan 2021 14:49:18 -0000 J David wrote:=0A= >On Sat, Jan 16, 2021 at 5:57 PM Rick Macklem wrote:= =0A= >> Opens can be closed until all FreeBSD open file descriptors for the file= =0A= >> are closed.=0A= >> --> Just the way it is. It is not an unintended leak. They go away once= =0A= >> all file descriptors get closed,=0A= >[...]=0A= >> However, none of the above seems unexpected, except maybe for why=0A= >> "ls" in the chroot opens 3 regular files each time. I don't know what=0A= >> chroot actually does for something like "ls"? I'll look.=0A= >=0A= >It probably isn't anything ls is doing. It's probably just the open=0A= >of whatever shared libraries ls has in common with the shell running=0A= >in the other window.=0A= >=0A= >And so, on a long-running system with frequently-accessed binaries (or=0A= >just shared libraries) mounted over NFS, where it's unlikely that all=0A= >descriptors will ever be closed at the same time, a hang/crash/failure=0A= >essentially becomes inevitable?=0A= Yes. If you have NFSv4 mounted shared libraries and an environment=0A= where there is at least one process running using a given shared library=0A= on the mount at any time, the NFSv4 Opens will accumulate and=0A= eventually constipate the mount.=0A= =0A= >Is this a solvable problem? (Outside of oneopenown.)=0A= For this case, "oneopenown" is your only option.=0A= =0A= I think there might have been a nullfs/NFSv4 interaction issue,=0A= but it appears to be resolved in FreeBSD13.=0A= =0A= rick=0A= =0A= Thanks!=0A= From owner-freebsd-fs@freebsd.org Mon Jan 18 12:53:09 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 495B34EAB6E for ; Mon, 18 Jan 2021 12:53:09 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DKBXh450dz3t4W for ; Mon, 18 Jan 2021 12:53:08 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-pl1-f180.google.com with SMTP id be12so8610646plb.4 for ; Mon, 18 Jan 2021 04:53:08 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=59Rr+q+Req2J5K5AlZjstLAWzFawKDSNB+UxEHgzGlY=; b=nb1YcVQ3Cp3UJgd690OMMFZx5PNGgEeLfthQq49QXzRmdK8FOGHtUNKSAYDHA8sIKM zJAXpMr+LuOPgFRR1Puwd5EzpoXjrGcQONL2JG7NkQmiKkktnW8Yyt2ImT4UDOvYxlup 2JLOtJR4LhA+cxT4OMxWVSyHe9pQEFWjWuihk291rXiYFFbPw4ZBYX2eFKWDLfSb0lsw MYBClK6LKJ0Yj2MRHM3HWtBGRep7mUCMr5qyDcrPLK6gl1AX0rOzpKUSXjynz/oDCMgb 6kjjwUctlfZ6M3ZZhOUttVp7Lso+UCr8BMO5Yu+iyXJcNJg76r+2uYd/gNbgiE5la+fA bHag== X-Gm-Message-State: AOAM53363+tPG7af3AQt23GpGQrVK5cXeXzoRAvNPcGOsH13/YMLoA4O Lg9Gd1CbjJh1yk4NbTXQRCs6neaqAth7yA== X-Google-Smtp-Source: ABdhPJzWvc36VPhoWNWjvOFCJtuIhTVSfBuIGCiSKpUqItVILCrpd9UKQsmKlMT51fBPiRWnw5S9Ow== X-Received: by 2002:a17:90a:6587:: with SMTP id k7mr26726786pjj.154.1610974386480; Mon, 18 Jan 2021 04:53:06 -0800 (PST) Received: from [192.168.0.24] ([195.64.148.76]) by smtp.googlemail.com with ESMTPSA id e72sm8089344pfh.199.2021.01.18.04.53.03 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Jan 2021 04:53:05 -0800 (PST) Subject: Re: zpool remove not working for metadata special devices To: John Delisle , freebsd-fs References: From: Andriy Gapon Message-ID: <07c28c11-325b-d05c-f078-e4a9b3f0be54@FreeBSD.org> Date: Mon, 18 Jan 2021 14:52:59 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0 Thunderbird/78.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4DKBXh450dz3t4W X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[209.85.214.180:from]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; R_DKIM_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[209.85.214.180:from]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; SPAMHAUS_ZRD(0.00)[209.85.214.180:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.214.180:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2021 12:53:09 -0000 On 2020-12-28 05:04, John Delisle wrote: > To experiment, I've completed this same procedure using Ubuntu 20.10 (on > the same "hardware", both are identically configured VMs in Azure) . > > The steps above work fine with ZFS on Ubuntu, and I can successfully remove > both data vdev mirrors and the special vdev mirror. This makes me think my > syntax is correct at least.. but for whatever reason the exact same > procedure fails on FreeBSD (both removing a mirror and removing the special > mirror fail with the same error on FreeBSD). I don't think that you did anything wrong and the syntax is certainly correct. I can reproduce the problem as well. Note that if you insert a reboot between the add special and the remove, then the latter works fine, at least it does for me. It looks like there is an issue with recording of spa_min_ashift internal parameter when a special device is first added. > On Sun, Dec 27, 2020 at 6:55 PM John Delisle wrote: > >> I have a pool of mirrors, and added a mirrored special device. Although >> documentation suggests it should be removable, I cannot get zpool to do so. >> >> All top-level vdevs are mirrors, and all have the same sector size and >> ashift. No raidz. >> >> ## Current 12.2 p2 >> root@jmdtest:/ # freebsd-version >> 12.2-RELEASE-p2 >> >> ## Disk info >> root@jmdtest:/ # diskinfo /dev/da[2-9] /dev/da[0-9][0-9] >> /dev/da2 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da3 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da4 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da5 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da6 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da7 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da8 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da9 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da10 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da11 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da12 512 68719476736 134217728 4096 0 >> 8354 255 63 >> /dev/da13 512 68719476736 134217728 4096 0 >> 8354 255 63 >> >> >> ## Create the pool: >> zpool create nebula mirror da2 da3 mirror da4 da5 mirror da6 da7 mirror >> da8 da9 mirror da10 da11 >> >> ## Add the special mirror >> zpool add nebula special mirror da12 da13 >> >> ## zpool status >> root@jmdtest:/ # zpool list -v nebula >> NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP >> HEALTH ALTROOT >> nebula 1.30T 2.57G 1.30T - - 0% 0% 1.00x >> ONLINE - >> mirror 254G 492M 254G - - 0% 0.18% >> da2 - - - - - - - >> da3 - - - - - - - >> mirror 254G 501M 254G - - 0% 0.19% >> da4 - - - - - - - >> da5 - - - - - - - >> mirror 254G 596M 253G - - 0% 0.22% >> da6 - - - - - - - >> da7 - - - - - - - >> mirror 254G 481M 254G - - 0% 0.18% >> da8 - - - - - - - >> da9 - - - - - - - >> mirror 254G 561M 253G - - 0% 0.21% >> da10 - - - - - - - >> da11 - - - - - - - >> special - - - - - - >> mirror 63.5G 2.62M 63.5G - - 0% 0.00% >> da12 - - - - - - - >> da13 - - - - - - - >> root@jmdtest:/ # >> >> ## Remove the special mirror >> root@jmdtest:/ # zpool remove nebula mirror-5 >> cannot remove mirror-5: invalid config; all top-level vdevs must have the >> same sector size and not be raidz. >> >> ## All mirrors, and all the same ashift: >> root@jmdtest:/ # zdb -C | grep -e child -e ashift >> vdev_children: 6 >> children[0]: >> ashift: 12 >> children[0]: >> children[1]: >> children[1]: >> ashift: 12 >> children[0]: >> children[1]: >> children[2]: >> ashift: 12 >> children[0]: >> children[1]: >> children[3]: >> ashift: 12 >> children[0]: >> children[1]: >> children[4]: >> ashift: 12 >> children[0]: >> children[1]: >> children[5]: >> ashift: 12 >> children[0]: >> children[1]: >> root@jmdtest:/ # >> >> What am I doing wrong? -- Andriy Gapon From owner-freebsd-fs@freebsd.org Mon Jan 18 12:53:50 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E03F74EABE5 for ; Mon, 18 Jan 2021 12:53:50 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net [217.70.183.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DKBYV2vBRz3t6t for ; Mon, 18 Jan 2021 12:53:50 +0000 (UTC) (envelope-from avg@FreeBSD.org) X-Originating-IP: 195.64.148.76 Received: from [192.168.0.24] (unknown [195.64.148.76]) (Authenticated sender: andriy.gapon@uabsd.com) by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id 0BEB2240007; Mon, 18 Jan 2021 12:53:47 +0000 (UTC) Subject: Re: zpool remove not working for metadata special devices To: John Delisle , freebsd-fs References: From: Andriy Gapon Message-ID: <92af08f3-342c-c98c-702f-54ee48bb089f@FreeBSD.org> Date: Mon, 18 Jan 2021 14:53:44 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0 Thunderbird/78.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4DKBYV2vBRz3t6t X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; ASN(0.00)[asn:29169, ipnet:217.70.176.0/20, country:FR] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2021 12:53:50 -0000 On 2020-12-28 05:04, John Delisle wrote: > To experiment, I've completed this same procedure using Ubuntu 20.10 (on > the same "hardware", both are identically configured VMs in Azure) . > > The steps above work fine with ZFS on Ubuntu, and I can successfully remove > both data vdev mirrors and the special vdev mirror. This makes me think my > syntax is correct at least.. but for whatever reason the exact same > procedure fails on FreeBSD (both removing a mirror and removing the special > mirror fail with the same error on FreeBSD). I don't think that you did anything wrong and the syntax is certainly correct. I can reproduce the problem as well. Note that if you insert a reboot between the add special and the remove, then the latter works fine, at least it does for me. It looks like there is an issue with recording of spa_min_ashift internal parameter when a special device is first added. > On Sun, Dec 27, 2020 at 6:55 PM John Delisle wrote: > >> I have a pool of mirrors, and added a mirrored special device. Although >> documentation suggests it should be removable, I cannot get zpool to do so. >> >> All top-level vdevs are mirrors, and all have the same sector size and >> ashift. No raidz. >> >> ## Current 12.2 p2 >> root@jmdtest:/ # freebsd-version >> 12.2-RELEASE-p2 >> >> ## Disk info >> root@jmdtest:/ # diskinfo /dev/da[2-9] /dev/da[0-9][0-9] >> /dev/da2 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da3 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da4 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da5 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da6 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da7 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da8 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da9 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da10 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da11 512 274877906944 536870912 4096 0 >> 33418 255 63 >> /dev/da12 512 68719476736 134217728 4096 0 >> 8354 255 63 >> /dev/da13 512 68719476736 134217728 4096 0 >> 8354 255 63 >> >> >> ## Create the pool: >> zpool create nebula mirror da2 da3 mirror da4 da5 mirror da6 da7 mirror >> da8 da9 mirror da10 da11 >> >> ## Add the special mirror >> zpool add nebula special mirror da12 da13 >> >> ## zpool status >> root@jmdtest:/ # zpool list -v nebula >> NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP >> HEALTH ALTROOT >> nebula 1.30T 2.57G 1.30T - - 0% 0% 1.00x >> ONLINE - >> mirror 254G 492M 254G - - 0% 0.18% >> da2 - - - - - - - >> da3 - - - - - - - >> mirror 254G 501M 254G - - 0% 0.19% >> da4 - - - - - - - >> da5 - - - - - - - >> mirror 254G 596M 253G - - 0% 0.22% >> da6 - - - - - - - >> da7 - - - - - - - >> mirror 254G 481M 254G - - 0% 0.18% >> da8 - - - - - - - >> da9 - - - - - - - >> mirror 254G 561M 253G - - 0% 0.21% >> da10 - - - - - - - >> da11 - - - - - - - >> special - - - - - - >> mirror 63.5G 2.62M 63.5G - - 0% 0.00% >> da12 - - - - - - - >> da13 - - - - - - - >> root@jmdtest:/ # >> >> ## Remove the special mirror >> root@jmdtest:/ # zpool remove nebula mirror-5 >> cannot remove mirror-5: invalid config; all top-level vdevs must have the >> same sector size and not be raidz. >> >> ## All mirrors, and all the same ashift: >> root@jmdtest:/ # zdb -C | grep -e child -e ashift >> vdev_children: 6 >> children[0]: >> ashift: 12 >> children[0]: >> children[1]: >> children[1]: >> ashift: 12 >> children[0]: >> children[1]: >> children[2]: >> ashift: 12 >> children[0]: >> children[1]: >> children[3]: >> ashift: 12 >> children[0]: >> children[1]: >> children[4]: >> ashift: 12 >> children[0]: >> children[1]: >> children[5]: >> ashift: 12 >> children[0]: >> children[1]: >> root@jmdtest:/ # >> >> What am I doing wrong? -- Andriy Gapon From owner-freebsd-fs@freebsd.org Mon Jan 18 13:03:42 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 960FF4EAF6B for ; Mon, 18 Jan 2021 13:03:42 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DKBms6T5Mz3tvS for ; Mon, 18 Jan 2021 13:03:41 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-pj1-f41.google.com with SMTP id j12so849613pjy.5 for ; Mon, 18 Jan 2021 05:03:41 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=sI6JNOxaOX3OqJKSy+vKFItyiWxrTLOx+WXH6bcTgq0=; b=syofk0elbrudMMCoNMsHrDLVAHPpL9fnS+MnTHfOwXm5kSteuRkbPKxlUcsdHhisuj aO1jc+fCh7FTIRFHB4Gh7CGbpUWA3PddQjH54qZuN/C4fcYNtbGcd4zt8IklO7IbaJdR jtL8hFoFYb46NhmT0wp1OIdtMJoJFHDqxpA9SPDVoIqdEvF9Y5u7boc8c38JouKFylBQ WsoGJyxDbRZHQ6Fy/UOLZT6HnfFpCXyNtk0YHwg5X8vEVQznhz6Si+GLs96OUlp6biB5 ChQr/HiuOwFmhw6stfX9NtnWVCk/sdaoxs6oy4LtNqGvExepoMaLmMGi51HGHJqTjX/5 EPBA== X-Gm-Message-State: AOAM533NJfEnRR65zGtt8Jsm50a0T4zph0xQyCHC1MT2zSXzOSq213B0 Ui/BKGuCTOb3p2hrbTb/TvcDLnnyw0xcjg== X-Google-Smtp-Source: ABdhPJzuLu/jEmDN6vndcIz9mjlMr4/WUbvEuGLN1dOEFY1jFATxUnl4wwiIAAf9QWjkSVFz0GiU/w== X-Received: by 2002:a17:90a:658c:: with SMTP id k12mr26332358pjj.31.1610975019415; Mon, 18 Jan 2021 05:03:39 -0800 (PST) Received: from [192.168.0.24] ([195.64.148.76]) by smtp.googlemail.com with ESMTPSA id m4sm16122845pgv.16.2021.01.18.05.03.37 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Jan 2021 05:03:38 -0800 (PST) Subject: Re: zpool remove not working for metadata special devices From: Andriy Gapon To: John Delisle , freebsd-fs References: <92af08f3-342c-c98c-702f-54ee48bb089f@FreeBSD.org> Message-ID: Date: Mon, 18 Jan 2021 15:03:33 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0 Thunderbird/78.6.1 MIME-Version: 1.0 In-Reply-To: <92af08f3-342c-c98c-702f-54ee48bb089f@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4DKBms6T5Mz3tvS X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[209.85.216.41:from]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; R_DKIM_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[209.85.216.41:from]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; SPAMHAUS_ZRD(0.00)[209.85.216.41:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.216.41:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2021 13:03:42 -0000 On 2021-01-18 14:53, Andriy Gapon wrote: > On 2020-12-28 05:04, John Delisle wrote: >> To experiment, I've completed this same procedure using Ubuntu 20.10 (on >> the same "hardware", both are identically configured VMs in Azure) . >> >> The steps above work fine with ZFS on Ubuntu, and I can successfully remove >> both data vdev mirrors and the special vdev mirror. This makes me think my >> syntax is correct at least.. but for whatever reason the exact same >> procedure fails on FreeBSD (both removing a mirror and removing the special >> mirror fail with the same error on FreeBSD). > > I don't think that you did anything wrong and the syntax is certainly > correct. I can reproduce the problem as well. > Note that if you insert a reboot between the add special and the remove, > then the latter works fine, at least it does for me. > > It looks like there is an issue with recording of spa_min_ashift > internal parameter when a special device is first added. > Actually, just creating a pool can create the min/max ashift discrepancy. fbsd12# zpool create tank mirror vtbd1 vtbd2 fbsd12# dtrace -n 'spa_close:entry /pid == $target/ { this->spa = args[0]; printf("min = %d, max = %d\n", this->spa->spa_min_ashift, this->spa->spa_max_ashift); }' -c 'zpool status tank' dtrace: description 'spa_close:entry ' matched 1 probe pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 vtbd1 ONLINE 0 0 0 vtbd2 ONLINE 0 0 0 errors: No known data errors dtrace: pid 737 has exited CPU ID FUNCTION:NAME 1 66818 spa_close:entry min = 9, max = 12 1 66818 spa_close:entry min = 9, max = 12 Both devices have 4K logical block size, so *the* ashift is 12. But somehow spa_min_ashift got set to 9. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Mon Jan 18 13:26:52 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 701074EB8C7 for ; Mon, 18 Jan 2021 13:26:52 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from relay9-d.mail.gandi.net (relay9-d.mail.gandi.net [217.70.183.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DKCHc0CyYz3w6W for ; Mon, 18 Jan 2021 13:26:51 +0000 (UTC) (envelope-from avg@FreeBSD.org) X-Originating-IP: 195.64.148.76 Received: from [192.168.0.24] (unknown [195.64.148.76]) (Authenticated sender: andriy.gapon@uabsd.com) by relay9-d.mail.gandi.net (Postfix) with ESMTPSA id B769BFF803; Mon, 18 Jan 2021 13:26:49 +0000 (UTC) Subject: Re: zpool remove not working for metadata special devices From: Andriy Gapon To: John Delisle , freebsd-fs References: <92af08f3-342c-c98c-702f-54ee48bb089f@FreeBSD.org> Message-ID: <63109762-27df-7677-e71a-b70572af12c8@FreeBSD.org> Date: Mon, 18 Jan 2021 15:26:46 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0 Thunderbird/78.6.1 MIME-Version: 1.0 In-Reply-To: <92af08f3-342c-c98c-702f-54ee48bb089f@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4DKCHc0CyYz3w6W X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; ASN(0.00)[asn:29169, ipnet:217.70.176.0/20, country:FR]; local_wl_from(0.00)[FreeBSD.org] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2021 13:26:52 -0000 On 2021-01-18 14:53, Andriy Gapon wrote: > It looks like there is an issue with recording of spa_min_ashift > internal parameter when a special device is first added. This OpenZFS change explicitly addresses a couple of tangential issues (special + ashift) but it also fixes the issue that you reported: https://github.com/openzfs/zfs/commit/b3f4436d37bb72ffd924132aa4ef2d743762011f -- Andriy Gapon From owner-freebsd-fs@freebsd.org Mon Jan 18 17:27:24 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9B9A44F0C2C for ; Mon, 18 Jan 2021 17:27:24 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-lf1-x12e.google.com (mail-lf1-x12e.google.com [IPv6:2a00:1450:4864:20::12e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DKJd702jCz4j7R for ; Mon, 18 Jan 2021 17:27:22 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: by mail-lf1-x12e.google.com with SMTP id u25so25173346lfc.2 for ; Mon, 18 Jan 2021 09:27:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=EbMaQXyIvnZ+a0YNaWO8nd/s44ib1+PtdQG+6Kcs6Fc=; b=EZR7Hlb1ymHO4Xn09SeSNtn4rpy/S2IfSSrAR4NNmvtr+fvpu9IJF4yG9PFBWvh7fl wiFXyj0CGpDb7PtjntEZXXJ+riXPLaHG/zm6yf5vzm8AR1SxATcFfiBbiqoTxqAsHLh0 fZu3WWswJQduURL5iSjvtuXcLafsopzTc40Be5WHkw2JdiVmG49zvBnnizpHeZsztU+6 hTJNxEnhfLpgR+ZRdViodfX37g9Qi0TgFLZtvarzqABDeEZFDmDvMx60jzCKcplN1Td0 Bl1hkqH14IY10uYnpmw/7H4Lhj/Dn41jg46+HETLEuwuFrMBq3dV4ScnRPQvTlaKtBxF ZKWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=EbMaQXyIvnZ+a0YNaWO8nd/s44ib1+PtdQG+6Kcs6Fc=; b=t4zL4S2n6B7MUJO6zxieWIBtBx7+0kvKjSEZrCXvDzOQbqKiGwmctAasPDKn0f1znF DQVulw6gXgZTWF/wtzwAGYJn7K7gOzF5d+NKXwTeNZL7jpKgni9mq6hcFNqqGH6IjORg y6W6GfuUPC2C2USiNzk9tHzaHI4GBm0BDvjq6uFnp6RAuANkCrt50kPyEfnnwYehtdio OcPKtOlTdMfNYEFD0v6yjeEJLZjP/fI67iTiCZrBQk2FIoO2usVx0MM3W35tp+UwxtPf INdWhC9mjMkfgT0JCrAjS/k656QYkADdhDHyyrS4ng+/n9CKBFW7+L8C7VAXGHawnb67 mlRg== X-Gm-Message-State: AOAM530U36jasN96OmY20phFBVafpU8KhmUQrx92PzuTk3fM9WtYy//t mnAzlNpks8gStkn5nlLIRgEoxz05pGAE0LWoDAQ= X-Google-Smtp-Source: ABdhPJw1F+E+88nvrpo80/4193m6EIfxY3aGJ70uO7zzYdX+Do/IK3OqOssZOutaKDwDccXPlm8H9Jn3QA1TbcSRpUo= X-Received: by 2002:a05:6512:786:: with SMTP id x6mr59655lfr.643.1610990841209; Mon, 18 Jan 2021 09:27:21 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: J David Date: Mon, 18 Jan 2021 12:27:10 -0500 Message-ID: Subject: Re: Major issues with nfsv4 To: Rick Macklem Cc: Konstantin Belousov , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4DKJd702jCz4j7R X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=EZR7Hlb1; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2a00:1450:4864:20::12e as permitted sender) smtp.mailfrom=jdavidlists@gmail.com X-Spamd-Result: default: False [-2.01 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::12e:from]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_SPAM_SHORT(0.99)[0.992]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::12e:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::12e:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2021 17:27:24 -0000 Using oneopenown in even limited testing seems to cause NFS mounts to completely hang (requiring "umount -N") on a fairly regular basis. The problems without oneopenown are worse than that it will "eventually constipate the mount." There also seems to be some sort of garbage collection or something that occurs periodically that causes one nfscl process to jump to 100% CPU and create systemwide disruption, causing delay in many unrelated system calls (30+ second delays observed in at least fork, systemctl, and ioctl via ktrace of processes that do not touch any NFS filesystem). As far as I can tell, stuff like procstat -ka doesn't run until it's already over, at which point there's nothing to see. Sorry I don't have more information about that. Unfortunately, we discussed the status of this today, and my update-to-NFSv4 project has basically been killed; we're backing off to NFSv3. Thanks for the help and information and all the hard work on NFS for many many years! From owner-freebsd-fs@freebsd.org Fri Jan 22 02:27:06 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 978004E796B for ; Fri, 22 Jan 2021 02:27:06 +0000 (UTC) (envelope-from jdelisle@gmail.com) Received: from mail-pf1-x42a.google.com (mail-pf1-x42a.google.com [IPv6:2607:f8b0:4864:20::42a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DMNST5SMxz3mw5; Fri, 22 Jan 2021 02:27:05 +0000 (UTC) (envelope-from jdelisle@gmail.com) Received: by mail-pf1-x42a.google.com with SMTP id o20so2747221pfu.0; Thu, 21 Jan 2021 18:27:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=7ZbXOBi6dpx31AfIMOmZ+aOHbdBW//QoF0F7QllFbLE=; b=N5oKebgHPjXPMSdVdFY8CkxkC8bGsozQL7S7d1it5yxjqx1z2bUjfK217PppwkKRkT ggl8tOQK3OWdtfy7KMOaF4WqRjwjBIJ+4+QBy+pQsy4qtrMn27X5vymrs9w2NJnp565J tFJq5K4O3fcL7LS+Q+lTv4H6YiNcPGNdO9wbVbGsszX/TpGPOkjbr4ye1npI9//Ksmpc MbZMNkFR+Dg7dZnoWQn1FXjMxFVpYWd5WRKI7l9AgBqxeM8uKO7npH9HHgMVs2jlUZ5b 0DCRULCa7Gp49U1Cxf+8+KqEx6LV2sp7oiOIKc4NWWvsHC+kQkkjgAbeuqxPfR37LBQh 0dGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=7ZbXOBi6dpx31AfIMOmZ+aOHbdBW//QoF0F7QllFbLE=; b=b8TpdIk53j+QLRm6eTzuMtHgFJCeVlNvNUocDV+4lhzSWXCUvOwerbVfOpvj7fVFxj 7fja7BpI+FKiYn7qeXnGCbxLk6+utGMkzy98mWmNRbL6MWB4RtZjuBhs/2Q+TX83BpkH 7qti0xKd/SZPRQfuWMSzzox3LNqmTRZQsTn0TVg/DbjMih1vr2Uh2IeBk39fECcxGNBC a7f9gUxPk1Cj0QprCaJnZVdVNZPmYhhWba2B0r48eVdrY848BI8K1byMOcM2eDv12/SQ U+PPtoiWwwbOLiF4DijlH2q2HtO3KBZ9r2jXKV4soYki5lvA/HTK48YPrfLnHtUx2upG tkwA== X-Gm-Message-State: AOAM531vPP+BynIz1tVP/JFPOgSp4urMDmx9xjSkUT9/JVumwPPvrctT zWSvPXb65FC6jYcfiJSQQ5ECQ03BB5yDOu6c4YwY3vTrZnI= X-Google-Smtp-Source: ABdhPJxs56nEiZEdVTGQmrQ+e9nL7eSjBjvDvIKsuUFywMlvXYvFiqg12cHHcom696p/m4m2mE/b7xKUVwLrIWKVSag= X-Received: by 2002:a63:fe13:: with SMTP id p19mr2382759pgh.119.1611282423892; Thu, 21 Jan 2021 18:27:03 -0800 (PST) MIME-Version: 1.0 References: <92af08f3-342c-c98c-702f-54ee48bb089f@FreeBSD.org> <63109762-27df-7677-e71a-b70572af12c8@FreeBSD.org> In-Reply-To: <63109762-27df-7677-e71a-b70572af12c8@FreeBSD.org> From: John Delisle Date: Thu, 21 Jan 2021 20:26:51 -0600 Message-ID: Subject: Re: zpool remove not working for metadata special devices To: Andriy Gapon Cc: freebsd-fs X-Rspamd-Queue-Id: 4DMNST5SMxz3mw5 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=N5oKebgH; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdelisle@gmail.com designates 2607:f8b0:4864:20::42a as permitted sender) smtp.mailfrom=jdelisle@gmail.com X-Spamd-Result: default: False [-2.20 / 15.00]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::42a:from]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_SPAM_SHORT(0.80)[0.795]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::42a:from:127.0.2.255]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::42a:from]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-fs]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2021 02:27:06 -0000 Thank you Andriy for your insight into the issue. I can confirm that after a reboot, everything works fine! Interestingly, after creating the pool and rebooting, I can add and remove special mirrors without needing to reboot in-between operations. Thanks again! Looking forward to trying this in a production setting. On Mon, Jan 18, 2021 at 7:26 AM Andriy Gapon wrote: > On 2021-01-18 14:53, Andriy Gapon wrote: > > It looks like there is an issue with recording of spa_min_ashift > > internal parameter when a special device is first added. > > This OpenZFS change explicitly addresses a couple of tangential issues > (special + ashift) but it also fixes the issue that you reported: > > > https://github.com/openzfs/zfs/commit/b3f4436d37bb72ffd924132aa4ef2d743762011f > > > -- > Andriy Gapon >