From owner-freebsd-fs@freebsd.org Sun Apr 25 21:01:06 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DFCBC5EBE8E for ; Sun, 25 Apr 2021 21:01:06 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4FT0my4Yjvz4cv4 for ; Sun, 25 Apr 2021 21:01:06 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id 890E85EBB83; Sun, 25 Apr 2021 21:01:06 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8859D5EBE8D for ; Sun, 25 Apr 2021 21:01:06 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FT0my1q2gz4cyD for ; Sun, 25 Apr 2021 21:01:06 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 224871B05E for ; Sun, 25 Apr 2021 21:01:06 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 13PL16ag097393 for ; Sun, 25 Apr 2021 21:01:06 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 13PL162I097392 for fs@FreeBSD.org; Sun, 25 Apr 2021 21:01:06 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202104252101.13PL162I097392@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 25 Apr 2021 21:01:06 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Apr 2021 21:01:07 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 211491 | System hangs after "Uptime" on reboot with ZFS Open | 221909 | [ZFS] Add a sysctl to toggle send_corrupt_data Open | 237067 | ZFS: Crash in vdev_dtl_reassess when using GELI w Open | 240831 | zfs: Panic during snapshot on 12.1-STABLE r352648 Open | 243973 | [zfs] rollback segmentation fault Open | 244656 | zfs: resilver doesn't provide enough information Open | 244692 | gjournal: Does not support TRIM Open | 244899 | [PATCH] zfs: xattr on a symlink target > 136 caus Open | 251035 | [zfs] Allow 64 bit ZFS to support 32 bit ioctls ( 11 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Apr 26 11:58:35 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 361D75FCDC6 for ; Mon, 26 Apr 2021 11:58:35 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vtr.rulingia.com (vtr.rulingia.com [IPv6:2001:19f0:5801:ebe:5400:1ff:fe53:30fd]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "vtr.rulingia.com", Issuer "R3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FTNhT6hTWz3Hsr for ; Mon, 26 Apr 2021 11:58:33 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (ppp239-208.static.internode.on.net [59.167.239.208]) by vtr.rulingia.com (8.16.1/8.15.2) with ESMTPS id 13QBwI96053117 (version=TLSv1.3 cipher=AEAD-AES256-GCM-SHA384 bits=256 verify=OK) for ; Mon, 26 Apr 2021 21:58:23 +1000 (AEST) (envelope-from peter@rulingia.com) DKIM-Filter: OpenDKIM Filter v2.10.3 vtr.rulingia.com 13QBwI96053117 X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.16.1/8.16.1) with ESMTPS id 13QBwCXb024110 (version=TLSv1.3 cipher=AEAD-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 26 Apr 2021 21:58:12 +1000 (AEST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.16.1/8.16.1/Submit) id 13QBwCRH024109 for freebsd-fs@freebsd.org; Mon, 26 Apr 2021 21:58:12 +1000 (AEST) (envelope-from peter) Date: Mon, 26 Apr 2021 21:58:12 +1000 From: Peter Jeremy To: freebsd-fs@freebsd.org Subject: Migrating a ZFS pool to use OpenZFS encryption Message-ID: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="OFeBEMU1NA93kTsC" Content-Disposition: inline X-PGP-Key: http://www.rulingia.com/keys/peter.pgp X-Rspamd-Queue-Id: 4FTNhT6hTWz3Hsr X-Spamd-Bar: ------ X-Spamd-Result: default: False [-6.10 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[rulingia.com:s=default]; FREEFALL_USER(0.00)[peter]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2001:19f0:5801:ebe:5400:1ff:fe53:30fd:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[rulingia.com:+]; DMARC_POLICY_ALLOW(-0.50)[rulingia.com,quarantine]; NEURAL_HAM_SHORT(-1.00)[-1.000]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:19f0:5801:ebe:5400:1ff:fe53:30fd:from]; ASN(0.00)[asn:20473, ipnet:2001:19f0:5800::/38, country:US]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Apr 2021 11:58:35 -0000 --OFeBEMU1NA93kTsC Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I'm considering options for remote backups of a ZFS pool, without the remote system having the decryption key, and they seem to be either: a) Export the raw disks and locally run ZFS over geli over ggate. b) Use ZFS send between encrypted pools. The second option has the big advantage that I can do a scrub remotely without the remote system needing the encryption keys. The downside is that the local pool also needs to be encrypted. It's not possible to encrypt in place (native encryption can only be enabled when a pool is created) and there's very little information about how to get from an unencrypted pool to a natively encrypted pool. So far, the best documentation I've found is https://zfsonlinux.topicbox.com/groups/zfs-discuss/Tc9acf1bc1513ea21-M2f797= 7ea237e2f536b967a84/migration-from-unencrypted-to-encrypted-data-set which can be summarised as "it's complicated". (Another downside is that native encryption is relatively new so I'm not sure how battle-hardened it is). Before I reinvent the wheel, has anyone done this sort of thing and is able to offer advice from practical experience? --=20 Peter Jeremy --OFeBEMU1NA93kTsC Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE7rKYbDBnHnTmXCJ+FqWXoOSiCzQFAmCGqs9fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEVF QjI5ODZDMzA2NzFFNzRFNjVDMjI3RTE2QTU5N0EwRTRBMjBCMzQACgkQFqWXoOSi CzQnzQ/+OWCbxqMU9QuUT25yWNnGMZG9tbOnV/DWfU3AmimPnn/oaV2gbBYl3Akd 4rPsFgeFgJHd5QSssuEVa1QIGrzOGrl0TVRaPxdGzkHM1GlwqHLDGlhYUTZ+U7RS 6qPhiv6JkoBTVFPiJbXuoqfem6YqD1rWtXMh9ZUOhNs4SGTypzLn1fMByygRg87b GsVh+6GLRnsMB1kM3k+mPQlJZELJU9QdCEnkGjIMQxyT3NL9lOz229iN9NQUkl6H XqnY0b1XXIJ3y4H6DyKEDqf9G811j03zEOlBUqEs6o+QaVjQEuson8Ov8oT5WNkh /0ovf6xTpaftptZfuZALxmxjGgugCw87race6FUjIKl6MUjhnjJQl4qmG8yjfgI4 RnsjO19OEWcrsV8m1jp/zGr0zKWbKCzfni4knBMP//mV9LChN6dh+zY2o5at6CnZ qkItc9f/Wb+Dkc3L/mTTGBf0r3fbmEGv2T/3QMb9ATlWmL9nUb0nk3pp2tNkNzDh s93YMffQglN7c4QZpSY4YhSksBz3n38RCGiJOoK7W1sZ6ETgBGGtyiV7GBkaJuqL oEaC2/zhAFVsQ5OmOdIRfe+xGeOHcKfj0BE8E4rvUI6ZS07cUN/rL3no96wcatr1 oA7A6NeBq2g6o1l8nE8liejlfw++bVqGeWQTo4CY52S3RQtU8L0= =W7+u -----END PGP SIGNATURE----- --OFeBEMU1NA93kTsC-- From owner-freebsd-fs@freebsd.org Mon Apr 26 12:24:52 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 344255FE4B1 for ; Mon, 26 Apr 2021 12:24:52 +0000 (UTC) (envelope-from SRS0=7Nu3=JX=perdition.city=julien@bebif.be) Received: from orval.bbpf.belspo.be (orval.bbpf.belspo.be [193.191.208.90]) by mx1.freebsd.org (Postfix) with ESMTP id 4FTPGp73QNz3KRK for ; Mon, 26 Apr 2021 12:24:50 +0000 (UTC) (envelope-from SRS0=7Nu3=JX=perdition.city=julien@bebif.be) Received: from x1 (94.105.107.231.dyn.edpnet.net [94.105.107.231]) by orval.bbpf.belspo.be (Postfix) with ESMTPSA id 7A4B71D4FC1B for ; Mon, 26 Apr 2021 14:24:43 +0200 (CEST) Date: Mon, 26 Apr 2021 14:24:40 +0200 From: Julien Cigar To: freebsd-fs@freebsd.org Subject: iSCSI SAN Message-ID: <20210426122440.xirux6bwdztwsm5c@x1> Mail-Followup-To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline X-Rspamd-Queue-Id: 4FTPGp73QNz3KRK X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of SRS0=7Nu3=JX=perdition.city=julien@bebif.be designates 193.191.208.90 as permitted sender) smtp.mailfrom=SRS0=7Nu3=JX=perdition.city=julien@bebif.be X-Spamd-Result: default: False [-2.40 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[193.191.208.90:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[193.191.208.90:from:127.0.2.255]; MID_RHS_NOT_FQDN(0.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_NA(0.00)[perdition.city]; FORGED_SENDER(0.30)[julien@perdition.city,SRS0=7Nu3=JX=perdition.city=julien@bebif.be]; RCVD_NO_TLS_LAST(0.10)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:2611, ipnet:193.191.192.0/19, country:BE]; FROM_NEQ_ENVFROM(0.00)[julien@perdition.city,SRS0=7Nu3=JX=perdition.city=julien@bebif.be]; MAILMAN_DEST(0.00)[freebsd-fs]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Apr 2021 12:24:52 -0000 Hello, I'm wondering if something has already been written to implement a fully redundant and highly available FreeBSD ZFS based iSCSI SAN? I've setup some FreeBSD iSCSI SAN-like in the past (for small structures) and it has always worked well. However upgrades have always been painfull and, although there is ZFS, redundant power supplies, redundant switches with multipath, it's still a SPOF if a non-redundant component dies, like the motherboard. It's not like an HPE MSA-like system where everything is redundant out of the box. So the idea came to me for the iSCSI target to setup 2 physical servers with a bunch of disks, create some raidzx on them and export one ZFS volume per initiator on each target, a bit like on (1) I've tried to setup that in a small "lab", with some jails, gmultipath, two switches, and several VLANs. Unfortunately no 10 gbits to test, but 3x1Gbits LAGG with LACP. The downside of this setup is that "half" of the storage is (temporarily) lost when a target reboots (freebsd-update, upgrades, etc), which de-facto disqualifies gmirror + UFS on the initiator side as a full resync of required and takes ages. With ZFS you don't have this problem as only the delta is resync. For the few tests I did it seems to work, but I'm wondering: zfs over zvol .. is it sane? does it makes sense? could I disable checksum on the initiator side to speed up things? do you see any race condition or ... with this setup (sync, etc)? What do you think? Thanks! Julien (1) https://gist.github.com/silenius/c6d1020aca54c47f71aa9f2a19a55ffe -- Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. From owner-freebsd-fs@freebsd.org Mon Apr 26 15:19:29 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 256845E4433 for ; Mon, 26 Apr 2021 15:19:29 +0000 (UTC) (envelope-from joris.dedieu@gmail.com) Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FTT8J3cFzz3lLs for ; Mon, 26 Apr 2021 15:19:28 +0000 (UTC) (envelope-from joris.dedieu@gmail.com) Received: by mail-ej1-x62b.google.com with SMTP id r20so35285094ejo.11 for ; Mon, 26 Apr 2021 08:19:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=ULDAdnfglKl2NDKirGhaHm+fMnIA2HI8ks0I/B1j+xI=; b=PPUVu5AIG/yHO3Ppwb35IKn+ECoXnnl9mqrz9ow3qL2iScm6UlEzxtBl4ZwdAEDlWO 3Ej38Fn+1aJLGR2JqlN+XvsFIeWhxsMwR5kXjGV6nHCnKQ+4sj4zQfpuZ0nP+d0XH2dI FQvH3F+FZNQrPkATpaYURbGWtMMB8af0rP95OatJ7RvyVTdjQGe9/EVGs1Vq2uSLOLU4 YBQhIBlSkiB8LblzKfGG26RXQy/LjkUniIfmPPxlsr2p0LO4FcGmTAEE8OBGzm1LvkO4 V7Dy2mpy38rMkbmze+tNXAE1i42pV+0/rWOnAHF+5mRNc+ONu5Icx+ZAduAHujYB2IvV JZCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=ULDAdnfglKl2NDKirGhaHm+fMnIA2HI8ks0I/B1j+xI=; b=X9D/+fo7WvzR4jeWaSheHWuDDH4xFFfOE+YiMFVg0Eyod3YePyyyDRG7Znkg6TbMCp NvGqHPLaKX6UHplaUoURPOIA/m3Q/EbTlxE5h/XD5ipMqWv7ZYMza/QAELojZ2ZI7uYb 8floRscKjGVgndNhWIla3wWn0PQehaRGiUJGvWBX7rzpZM5KuoAbhJGKMY5bn0lJFGks UdChcpXxUHMSOpaN/QLFH0LnBDYWetTcpQqY9JAvW2eQ0iynL/7bNu93R2uXSJ0KoBnq GLI73SM5wdRsYwzt5U8iMnauQZE8You4GaRVXC+L2o9kMa3TofqByjyAdTwm/V8SAEmR gFFg== X-Gm-Message-State: AOAM533uADfYJ7PUq/opkc8sAPtTyRxdFB3N35APsohBCqfKr6FK0FdR /gKLaB0sAPbKa64IWBhwxevOXQrFf19Ox2GwrzKflR+yOjs= X-Google-Smtp-Source: ABdhPJwJCf6bmvRIMcphhu/W61S+Dn8U9kiRLgEk2bzCV4vKVnaaRRbom1EE2YrPZJK7YAI2d5T7Q8Yps21l6y8jAHk= X-Received: by 2002:a17:906:5fce:: with SMTP id k14mr19461179ejv.9.1619450365673; Mon, 26 Apr 2021 08:19:25 -0700 (PDT) MIME-Version: 1.0 References: <20210426122440.xirux6bwdztwsm5c@x1> In-Reply-To: <20210426122440.xirux6bwdztwsm5c@x1> From: joris dedieu Date: Mon, 26 Apr 2021 17:19:14 +0200 Message-ID: Subject: Re: iSCSI SAN To: freebsd-fs@freebsd.org X-Rspamd-Queue-Id: 4FTT8J3cFzz3lLs X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=PPUVu5AI; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jorisdedieu@gmail.com designates 2a00:1450:4864:20::62b as permitted sender) smtp.mailfrom=jorisdedieu@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::62b:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::62b:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::62b:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Apr 2021 15:19:29 -0000 Hi, Le lun. 26 avr. 2021 =C3=A0 14:24, Julien Cigar a = =C3=A9crit : > Hello, > > I'm wondering if something has already been written to implement a > fully redundant and highly available FreeBSD ZFS based iSCSI SAN? > > I've setup some FreeBSD iSCSI SAN-like in the past (for small > structures) and it has always worked well. However upgrades have always > been painfull and, although there is ZFS, redundant power supplies, > redundant switches with multipath, it's still a SPOF if a non-redundant > component dies, like the motherboard. It's not like an HPE MSA-like > system where everything is redundant out of the box. > >From my experience HA mechanisms are an infinite source of pain. Don't forget the power of simplicity. What uptime do you get with your simple setup ? When was the last time you see a decent mainboard crash ? Also don't forget that FreeBSD has glusterfs and ceph. If you want to do something similar to proprietary chassis, you should have to look at SAS HBA and JBOD chassis, ATAoE chassis (if it still exists) or stuff like that to attach your disks to your two mainboard. Still (OMG) dealing with zfs (import -f) on failover, cluster STONITH and other voodoo. You will have fully redundant design (see https://i.stack.imgur.com/ijjpk.png ) Cheers Joris > So the idea came to me for the iSCSI target to setup 2 physical servers > with a bunch of disks, create some raidzx on them and export one ZFS > volume per initiator on each target, a bit like on (1) > > I've tried to setup that in a small "lab", with some jails, gmultipath, > two switches, and several VLANs. Unfortunately no 10 gbits to test, but > 3x1Gbits LAGG with LACP. > > The downside of this setup is that "half" of the storage is (temporarily) > lost when a target reboots (freebsd-update, upgrades, etc), which > de-facto disqualifies gmirror + UFS on the initiator side as a full > resync of required and takes ages. With ZFS you don't have this problem > as only the delta is resync. For the few tests I did it seems to work, > but I'm wondering: zfs over zvol .. is it sane? does it makes sense? > could I disable checksum on the initiator side to speed up things? do > you see any race condition or ... with this setup (sync, etc)? > > What do you think? > > Thanks! > Julien > > (1) https://gist.github.com/silenius/c6d1020aca54c47f71aa9f2a19a55ffe > > > -- > Julien Cigar > Belgian Biodiversity Platform (http://www.biodiversity.be) > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 > No trees were killed in the creation of this message. > However, many electrons were terribly inconvenienced. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Mon Apr 26 15:38:05 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 96DA95E57A5 for ; Mon, 26 Apr 2021 15:38:05 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from mailgate2.uni-hannover.de (mailgate2.uni-hannover.de [130.75.2.114]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4FTTYm0vsfz3mGq for ; Mon, 26 Apr 2021 15:38:03 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from intranet.aei.uni-hannover.de (ahin1.aei.uni-hannover.de [130.75.117.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mailgate2.uni-hannover.de (Postfix) with ESMTPS id 6B3FB22CA for ; Mon, 26 Apr 2021 17:38:01 +0200 (CEST) Received: from comet2.terra.ger ([130.75.117.49]) by intranet.aei.uni-hannover.de (IBM Domino Release 9.0.1FP8) with ESMTP id 2021042617380063-249133 ; Mon, 26 Apr 2021 17:38:00 +0200 Date: Mon, 26 Apr 2021 17:37:59 +0200 From: Gerrit Kuehn To: freebsd-fs@freebsd.org Subject: Re: iSCSI SAN Message-ID: <20210426173759.2a4029e9@comet2.terra.ger> In-Reply-To: References: <20210426122440.xirux6bwdztwsm5c@x1> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; amd64-portbld-freebsd12.1) MIME-Version: 1.0 X-MIMETrack: Itemize by SMTP Server on intranet/aei-hannover(Release 9.0.1FP8|February 23, 2017) at 26/04/2021 17:38:00, Serialize by Router on intranet/aei-hannover(Release 9.0.1FP8|February 23, 2017) at 26/04/2021 17:38:00, Serialize complete at 26/04/2021 17:38:00 X-TNEFEvaluated: 1 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Virus-Scanned: clamav-milter 0.102.4 at mailgate2 X-Virus-Status: Clean X-Rspamd-Queue-Id: 4FTTYm0vsfz3mGq X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of gerrit.kuehn@aei.mpg.de has no SPF policy when checking 130.75.2.114) smtp.mailfrom=gerrit.kuehn@aei.mpg.de X-Spamd-Result: default: False [-2.30 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[mpg.de]; RCVD_IN_DNSWL_MED(-0.20)[130.75.2.114:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:680, ipnet:130.75.0.0/16, country:DE]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Apr 2021 15:38:05 -0000 On Mon, 26 Apr 2021 17:19:14 +0200 joris dedieu wrote: > From my experience HA mechanisms are an infinite source of pain. Don't > forget the power of simplicity. What uptime do you get with your > simple setup ? When was the last time you see a decent mainboard > crash ? Also don't forget that FreeBSD has glusterfs and ceph. Yeah, I never got ggated and zfs working together in a stable manner... but maybe that was my fault? Another option to look at for HA might be minio: https://honeyguide.eu/posts/minio-fuse/ HTH cu Gerrit From owner-freebsd-fs@freebsd.org Mon Apr 26 15:59:54 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9544C5E6147 for ; Mon, 26 Apr 2021 15:59:54 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-qk1-x72a.google.com (mail-qk1-x72a.google.com [IPv6:2607:f8b0:4864:20::72a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FTV2x1Lvxz3nCt for ; Mon, 26 Apr 2021 15:59:52 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: by mail-qk1-x72a.google.com with SMTP id v20so2096478qkv.5 for ; Mon, 26 Apr 2021 08:59:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=longcount.org; s=google; h=content-transfer-encoding:from:mime-version:subject:date:message-id :references:cc:in-reply-to:to; bh=axpsCihFIkWKtQ0EPVXXG9BgLcAr/hKnS42PibggrGk=; b=pSIYoRFmKxcUy0h3ilwGlhnP5nFugJBE5SZwRuseteySS5mnlHBgwKyDaMGJNbhxXj AXFP39X/4nWT9SLNH5RF1RfLdTyShCbckubml5oIK74jmhqrbmP/vGmIZdf0a8TWaKj7 Xly3jsZaxOhekOG6HU/Y9drLgoh+jojCc2vQKxD8M+2u4QNvpRiFIqUsKLZV5rU2t5Yv dRtRUjvE+/TKSd20Dw2ha4zl6vzoaImGlkdgH6QsCjnMVpRLtjDkzGl2jgJfMIoAQtWl ULrs9GHA8w6oxsAj4JtByVKvMT502vYmjkG4gt/VwuKjxtyrrKFlu9JWO/ag50qtjpwn QMgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:content-transfer-encoding:from:mime-version :subject:date:message-id:references:cc:in-reply-to:to; bh=axpsCihFIkWKtQ0EPVXXG9BgLcAr/hKnS42PibggrGk=; b=pufTIDuSSLyq4lYwp24lY7k2iBR3eLALYYd/LITSH5YViwNULYfJdSzxBt9NtveQux BeqOAWe8dR3sOGFhK8aV+0+ztVd+GFLUXf6cN2LUHwmnGV+yDpVgFaqWcLFm96DdGsRP CgHEG+KBfjrcaN+EhZrmcZ7954Of62TyYFbmQ+PZFvcc/qk+RHCWOcnnm1YA0mU5aSB2 7loo2PjKlgR0kaaZZO2vzJaw4UCFETVnT6MF36bY2sa7bfGybVHLMFMsvhaF28vuizNk F5i13aqCkd5u3buTK5sOCpnY/aLtjF0KG7lruy4tx4hQUBlGvX2w4d/UqahWYBagCAG4 ggpw== X-Gm-Message-State: AOAM531a8ePUDCuD2DJhRWSzK2icahdhVJtLLsk3N+Kud37J1etMTt9y sg7oH05/gdoqtOeFJO4vkfRjKO0o2+2nmdIcai4= X-Google-Smtp-Source: ABdhPJwFEyXSA3at24jWK5NFsEI8wJWMRFraEA+JhNw09h4S4B+GIYJMjBqlvMmI8LHwgUQqHRpUNA== X-Received: by 2002:a05:620a:70c:: with SMTP id 12mr17936158qkc.377.1619452792056; Mon, 26 Apr 2021 08:59:52 -0700 (PDT) Received: from [192.168.1.50] (pool-96-232-87-72.nycmny.fios.verizon.net. [96.232.87.72]) by smtp.gmail.com with ESMTPSA id 69sm394578qkk.58.2021.04.26.08.59.51 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 26 Apr 2021 08:59:51 -0700 (PDT) From: Mark Saad Mime-Version: 1.0 (1.0) Subject: Re: iSCSI SAN Date: Mon, 26 Apr 2021 11:59:50 -0400 Message-Id: <8FE1DC37-1ACE-4E39-B4AC-9E958BED5BE9@longcount.org> References: <20210426173759.2a4029e9@comet2.terra.ger> Cc: freebsd-fs@freebsd.org In-Reply-To: <20210426173759.2a4029e9@comet2.terra.ger> To: Gerrit Kuehn X-Mailer: iPhone Mail (18D70) X-Rspamd-Queue-Id: 4FTV2x1Lvxz3nCt X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=longcount.org header.s=google header.b=pSIYoRFm; dmarc=none; spf=pass (mx1.freebsd.org: domain of nonesuch@longcount.org designates 2607:f8b0:4864:20::72a as permitted sender) smtp.mailfrom=nonesuch@longcount.org X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[longcount.org:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::72a:from]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[96.232.87.72:received]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[longcount.org:s=google]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[longcount.org]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::72a:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::72a:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Apr 2021 15:59:54 -0000 Julien You should also look at ha zfs. This is a commercial add on for FreeBSD ; I= llumos; Solaris or Linux to build msa like zfs appliances.=20 https://www.high-availability.com/home --- Mark Saad | nonesuch@longcount.org > On Apr 26, 2021, at 11:38 AM, Gerrit Kuehn wrote= : >=20 > =EF=BB=BF >> On Mon, 26 Apr 2021 17:19:14 +0200 >> joris dedieu wrote: >>=20 >> =46rom my experience HA mechanisms are an infinite source of pain. Don't >> forget the power of simplicity. What uptime do you get with your >> simple setup ? When was the last time you see a decent mainboard >> crash ? Also don't forget that FreeBSD has glusterfs and ceph. >=20 > Yeah, I never got ggated and zfs working together in a stable manner... > but maybe that was my fault? > Another option to look at for HA might be minio: > https://honeyguide.eu/posts/minio-fuse/ >=20 >=20 > HTH > cu > Gerrit > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Mon Apr 26 19:54:52 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1BFCE5EE33C for ; Mon, 26 Apr 2021 19:54:52 +0000 (UTC) (envelope-from SRS0=7Nu3=JX=perdition.city=julien@bebif.be) Received: from orval.bbpf.belspo.be (orval.bbpf.belspo.be [193.191.208.90]) by mx1.freebsd.org (Postfix) with ESMTP id 4FTbG30vSDz4W9L for ; Mon, 26 Apr 2021 19:54:50 +0000 (UTC) (envelope-from SRS0=7Nu3=JX=perdition.city=julien@bebif.be) Received: from x1 (94.105.107.231.dyn.edpnet.net [94.105.107.231]) by orval.bbpf.belspo.be (Postfix) with ESMTPSA id 818B81D4FC10; Mon, 26 Apr 2021 21:54:49 +0200 (CEST) Date: Mon, 26 Apr 2021 21:54:46 +0200 From: Julien Cigar To: joris dedieu Cc: freebsd-fs@freebsd.org Subject: Re: iSCSI SAN Message-ID: <20210426195446.ih2s3iyvhdbp2po2@x1> Mail-Followup-To: joris dedieu , freebsd-fs@freebsd.org References: <20210426122440.xirux6bwdztwsm5c@x1> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 4FTbG30vSDz4W9L X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of SRS0=7Nu3=JX=perdition.city=julien@bebif.be designates 193.191.208.90 as permitted sender) smtp.mailfrom=SRS0=7Nu3=JX=perdition.city=julien@bebif.be X-Spamd-Result: default: False [-2.40 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; FORGED_SENDER(0.30)[julien@perdition.city,SRS0=7Nu3=JX=perdition.city=julien@bebif.be]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[perdition.city]; SPAMHAUS_ZRD(0.00)[193.191.208.90:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[193.191.208.90:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; MID_RHS_NOT_FQDN(0.50)[]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_NO_TLS_LAST(0.10)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:2611, ipnet:193.191.192.0/19, country:BE]; FROM_NEQ_ENVFROM(0.00)[julien@perdition.city,SRS0=7Nu3=JX=perdition.city=julien@bebif.be]; MAILMAN_DEST(0.00)[freebsd-fs]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Apr 2021 19:54:52 -0000 On Mon, Apr 26, 2021 at 05:19:14PM +0200, joris dedieu wrote: > Hi, > > Le lun. 26 avr. 2021 à 14:24, Julien Cigar a écrit : > > > Hello, > > > > I'm wondering if something has already been written to implement a > > fully redundant and highly available FreeBSD ZFS based iSCSI SAN? > > > > > I've setup some FreeBSD iSCSI SAN-like in the past (for small > > structures) and it has always worked well. However upgrades have always > > been painfull and, although there is ZFS, redundant power supplies, > > redundant switches with multipath, it's still a SPOF if a non-redundant > > component dies, like the motherboard. It's not like an HPE MSA-like > > system where everything is redundant out of the box. > > > > From my experience HA mechanisms are an infinite source of pain. Don't >From my experence "automatic failover" is an infinite source of pain, not HA > forget the power of simplicity. What uptime do you get with your simple > setup ? When was the last time you see a decent mainboard crash ? As long as I don't touch at it everything is fine. Now, as soon as I have to reboot the server or upgrade (from example from one major release to another) there are long minutes of downtime, which is unacceptable if many servers are connected. > Also don't forget that FreeBSD has glusterfs and ceph. > I haven't looked at CEPH yet, but I don't think everything has been ported (KRDB, CEPH native fs, etc). It is certainly not "production ready". > If you want to do something similar to proprietary chassis, you should have > to look at SAS HBA and JBOD chassis, ATAoE chassis (if it still exists) or Yes, definitively. SAS HBA with a JBOD chassis is something I was looking too .. unfortunately I don't have the hardware to experiment :( > stuff like that to attach your disks to your two mainboard. Still (OMG) > dealing with zfs (import -f) on failover, cluster STONITH and other voodoo. > You will have fully redundant design (see > https://i.stack.imgur.com/ijjpk.png ) > > Cheers > Joris > Thanks, Julien > > > So the idea came to me for the iSCSI target to setup 2 physical servers > > with a bunch of disks, create some raidzx on them and export one ZFS > > volume per initiator on each target, a bit like on (1) > > > > > I've tried to setup that in a small "lab", with some jails, gmultipath, > > two switches, and several VLANs. Unfortunately no 10 gbits to test, but > > 3x1Gbits LAGG with LACP. > > > > The downside of this setup is that "half" of the storage is (temporarily) > > lost when a target reboots (freebsd-update, upgrades, etc), which > > de-facto disqualifies gmirror + UFS on the initiator side as a full > > resync of required and takes ages. With ZFS you don't have this problem > > as only the delta is resync. For the few tests I did it seems to work, > > but I'm wondering: zfs over zvol .. is it sane? does it makes sense? > > could I disable checksum on the initiator side to speed up things? do > > you see any race condition or ... with this setup (sync, etc)? > > > > What do you think? > > > > Thanks! > > Julien > > > > (1) https://gist.github.com/silenius/c6d1020aca54c47f71aa9f2a19a55ffe > > > > > > -- > > Julien Cigar > > Belgian Biodiversity Platform (http://www.biodiversity.be) > > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 > > No trees were killed in the creation of this message. > > However, many electrons were terribly inconvenienced. > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" -- Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. From owner-freebsd-fs@freebsd.org Tue Apr 27 19:42:12 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5C1AF5FF5CE for ; Tue, 27 Apr 2021 19:42:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4FVBx01xZJz4q5v for ; Tue, 27 Apr 2021 19:42:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 42A4F5FF7E3; Tue, 27 Apr 2021 19:42:12 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 427195FF81C for ; Tue, 27 Apr 2021 19:42:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FVBx01LxCz4q4K for ; Tue, 27 Apr 2021 19:42:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 222B520B04 for ; Tue, 27 Apr 2021 19:42:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 13RJgCV2014377 for ; Tue, 27 Apr 2021 19:42:12 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 13RJgCRk014376 for fs@FreeBSD.org; Tue, 27 Apr 2021 19:42:12 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 255436] nfs client panic with non-sleepable locks on 14.0-CURRENT main-n246340-01bad87a76d Date: Tue, 27 Apr 2021 19:42:12 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to keywords Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Apr 2021 19:42:12 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D255436 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org Keywords| |panic --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Apr 27 19:56:05 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E8D4E5FFF52 for ; Tue, 27 Apr 2021 19:56:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4FVCF15zGcz4qtk for ; Tue, 27 Apr 2021 19:56:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id CD112621066; Tue, 27 Apr 2021 19:56:05 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CCDD75FFF51 for ; Tue, 27 Apr 2021 19:56:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FVCF15N2Xz4qfR for ; Tue, 27 Apr 2021 19:56:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id AB99D20D8E for ; Tue, 27 Apr 2021 19:56:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 13RJu5Y5021311 for ; Tue, 27 Apr 2021 19:56:05 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 13RJu5w7021310 for fs@FreeBSD.org; Tue, 27 Apr 2021 19:56:05 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 255436] nfs client panic with non-sleepable locks on 14.0-CURRENT main-n246340-01bad87a76d Date: Tue, 27 Apr 2021 19:56:06 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: rmacklem@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Apr 2021 19:56:06 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D255436 Rick Macklem changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|fs@FreeBSD.org |rmacklem@FreeBSD.org CC| |rmacklem@FreeBSD.org --- Comment #1 from Rick Macklem --- A couple of questions? - Did this happen during normal operation or during a umount? - Are delegations enabled on your server and nfscbd(8) running in the client? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Apr 29 08:24:24 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AE9DD5FD24E for ; Thu, 29 Apr 2021 08:24:24 +0000 (UTC) (envelope-from njm@njm.me.uk) Received: from smtp001-out.apm-internet.net (smtp001-out.apm-internet.net [85.119.248.222]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4FW7nz33YSz4SXV for ; Thu, 29 Apr 2021 08:24:22 +0000 (UTC) (envelope-from njm@njm.me.uk) Received: (qmail 4274 invoked from network); 29 Apr 2021 08:24:20 -0000 X-APM-Out-ID: 16196846600427 X-APM-Authkey: 18389/1(18389/1) 1 Received: from unknown (HELO meld.njm.me.uk) (86.179.69.31) by smtp001.apm-internet.net with SMTP; 29 Apr 2021 08:24:20 -0000 Received: from triton.njm.me.uk (triton.njm.me.uk [192.168.144.133]) by meld.njm.me.uk (8.16.1/8.16.1) with ESMTPS id 13T8OJkB001518 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 29 Apr 2021 09:24:20 +0100 (BST) (envelope-from njm@njm.me.uk) Received: from localhost (localhost [127.0.0.1]) by triton.njm.me.uk (8.16.1/8.16.1) with ESMTP id 13T8OJtZ073848 for ; Thu, 29 Apr 2021 09:24:19 +0100 (BST) (envelope-from njm@njm.me.uk) Date: Thu, 29 Apr 2021 09:24:19 +0100 From: "N.J. Mann" To: freebsd-fs@freebsd.org Subject: readdir() -> d_type always zero on NFS v3 mounted filesystems Message-ID: <2E84A420CCC10A73504624DE@triton.njm.me.uk> X-Mailer: Mulberry/4.0.8 (Linux/x86) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="==========16324982B0D1D3197C7A==========" X-Rspamd-Queue-Id: 4FW7nz33YSz4SXV X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of njm@njm.me.uk has no SPF policy when checking 85.119.248.222) smtp.mailfrom=njm@njm.me.uk X-Spamd-Result: default: False [-0.77 / 15.00]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[85.119.248.222:from]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; HAS_ATTACHMENT(0.00)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[85.119.248.222:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_NONE(0.00)[]; NEURAL_HAM_SHORT(-0.57)[-0.568]; CTYPE_MIXED_BOGUS(1.00)[]; DMARC_NA(0.00)[njm.me.uk]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:+,4:+]; ASN(0.00)[asn:35259, ipnet:85.119.248.0/21, country:GB]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs]; RCVD_IN_DNSWL_LOW(-0.10)[85.119.248.222:from] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Apr 2021 08:24:24 -0000 --==========16324982B0D1D3197C7A========== Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline Hi, I recently changed over from using svn to gitup to update /usr/ports on my local system and have been experiencing problems since. At first I thought it was an issue with gitup itself, but now I believe it is either a kernel issue or a configuration issue. I originally posted about the problem to the freebsd-ports mailing list: https://lists.freebsd.org/pipermail/freebsd-ports/2021-April/120929.html Since then I have dug deeper and come to the conclusion that it is not a problem with gitup. The issue I am seeing is that gitup is unable to delete files and directories, even complete ports, which have been removed from the repository. gitup basically does the following: prune_tree(base_path) { if ((directory = opendir(base_path)) != NULL) { while ((entry = readdir(directory)) != NULL) { snprintf(full_path, sizeof(full_path), "%s/%s", base_path, entry->d_name); if (entry->d_type == DT_DIR) { prune_tree(full_path); } else { if ((remove(full_path) != 0) && (errno != ENOENT)) err(EXIT_FAILURE, "prune_tree: cannot remove %s", full_path); } } closedir(directory); if (rmdir(base_path) != 0) err(EXIT_FAILURE, "prune_tree: cannot remove %s", base_path); } } When gitup is run on either a UFS or ZFS file system this works. However, when I run it on a NFS v3 mounted filesystem it fails. Liberal addition of printf's shows that for non-NFS mounted filesystems d_type contains the correct value for each file/directory, but for NFS mounted file systems it is zero - other fields such as d_name and d_namlen are correct. While debugging this I added a call to stat() and that always returns the correct values. At this point I started digging in libc and quickly found that readdir() is basically a wrapper around a system call. Digging in the kernel I quickly got out of my depth and hence my posting here. I then wrote a simple test programme which also shows the issue. I have attached the source for the test programme and the output from two runs, the first on an NFS mounted file system and the second on a local UFS file system. Before gitup started failing I had not seen any failures like this and that was why I initially assumed this must be a gitup issue. I now believe this is either a kernel issue or a confuration issue. My configuration is as follows: file server: /exports/ports - ZFS file system for FreeBSD ports repo exported via NFS /remote/ports - FreeBSD ports repo NFS (v3 rw,tcp) mounted from /export/ports on file server /usr/ports - symbolic link to /remote/ports client machines: /remote/ports - FreeBSD ports repo NFS (v3 ro,tcp) mounted from /export/ports on file server /usr/ports - symbolic link to /remote/ports I run gitup in /remote/ports on the file server and then pkg, portmaster, &.c, in /usr/ports on the file server and then the client machines. All systems are running 11-STABLE from about 12 days ago - I usually update every couple of weeks. Any assistance, suggestions, patches, clue bats, will be gratefully accepted. Regards, Nick. -- --==========16324982B0D1D3197C7A========== Content-Type: text/plain; charset=us-ascii; name="my_search.c.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="my_search.c.txt"; size=2044 /* * vim:sw=4 sts=4 expandtab * */ #include #include #include #include #include #include #include #include static void my_recurse(char *path) { DIR *directory = NULL; struct dirent *entry = NULL; struct stat sb; char full_path[strlen(path) + 1 + MAXNAMLEN + 1]; if ((directory = opendir(path)) != NULL) { printf("opendir: %s\n", path); while ((entry = readdir(directory)) != NULL) { printf("readdir: %c %u %s\n", entry->d_type == DT_DIR ? 'D' : '_', entry->d_namlen, entry->d_name); snprintf(full_path, sizeof(full_path), "%s/%s", path, entry->d_name); stat(full_path, &sb); printf("stat: %c\n", S_ISDIR(sb.st_mode) ? 'D' : '_'); if (S_ISDIR(sb.st_mode) != 0) { if ((entry->d_namlen == 1) && (strcmp(entry->d_name, "." ) == 0)) continue; if ((entry->d_namlen == 2) && (strcmp(entry->d_name, "..") == 0)) continue; printf("recurse: %s\n", full_path); my_recurse(full_path); } else printf("file: %s\n", full_path); } printf("closedir: %s\n", path); closedir(directory); } } int main(int argc, char *argv[]) { int ch, i; char *path; while ((ch = getopt(argc, argv, "v")) != -1) switch (ch) { case 'v': /* Do nothing, for now */ break; default: fprintf(stderr, "Illegal option -- %c\n", ch); exit (-1); } argc -= optind; argv += optind; if (argc > 1) { fprintf(stderr, "Too many parameters\n"); exit(-1); } else if (argc < 1) { fprintf(stderr, "Not enough parameters\n"); exit(-1); } path = *argv; my_recurse(path); return 0; } --==========16324982B0D1D3197C7A========== Content-Type: text/plain; charset=us-ascii; name="test1.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="test1.txt"; size=458 opendir: /remote/ports/net/gitup readdir: _ 1 . stat: D readdir: _ 2 .. stat: D readdir: _ 9 pkg-descr stat: _ file: /remote/ports/net/gitup/pkg-descr readdir: _ 8 distinfo stat: _ file: /remote/ports/net/gitup/distinfo readdir: _ 8 Makefile stat: _ file: /remote/ports/net/gitup/Makefile readdir: _ 9 pkg-plist stat: _ file: /remote/ports/net/gitup/pkg-plist closedir: /remote/ports/net/gitup --==========16324982B0D1D3197C7A========== Content-Type: text/plain; charset=us-ascii; name="test2.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="test2.txt"; size=8879 opendir: /local/av/radio/6Music/ readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: _ 13 DJ_Shadow.m4a stat: _ file: /local/av/radio/6Music//DJ_Shadow.m4a readdir: D 4 2020 stat: D recurse: /local/av/radio/6Music//2020 opendir: /local/av/radio/6Music//2020 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: D 2 05 stat: D recurse: /local/av/radio/6Music//2020/05 opendir: /local/av/radio/6Music//2020/05 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: D 2 23 stat: D recurse: /local/av/radio/6Music//2020/05/23 opendir: /local/av/radio/6Music//2020/05/23 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: D 4 1100 stat: D recurse: /local/av/radio/6Music//2020/05/23/1100 opendir: /local/av/radio/6Music//2020/05/23/1100 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: _ 44 The_Huey_Show:_Funky_Space_Reincarnation.m4a stat: _ file: /local/av/radio/6Music//2020/05/23/1100/The_Huey_Show:_Funky_Space_Reincarnation.m4a closedir: /local/av/radio/6Music//2020/05/23/1100 closedir: /local/av/radio/6Music//2020/05/23 readdir: D 2 30 stat: D recurse: /local/av/radio/6Music//2020/05/30 opendir: /local/av/radio/6Music//2020/05/30 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: D 4 1100 stat: D recurse: /local/av/radio/6Music//2020/05/30/1100 opendir: /local/av/radio/6Music//2020/05/30/1100 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: _ 29 The_Huey_Show:_Buggin_Out.m4a stat: _ file: /local/av/radio/6Music//2020/05/30/1100/The_Huey_Show:_Buggin_Out.m4a closedir: /local/av/radio/6Music//2020/05/30/1100 closedir: /local/av/radio/6Music//2020/05/30 closedir: /local/av/radio/6Music//2020/05 readdir: D 2 06 stat: D recurse: /local/av/radio/6Music//2020/06 opendir: /local/av/radio/6Music//2020/06 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: D 2 06 stat: D recurse: /local/av/radio/6Music//2020/06/06 opendir: /local/av/radio/6Music//2020/06/06 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: D 4 1100 stat: D recurse: /local/av/radio/6Music//2020/06/06/1100 opendir: /local/av/radio/6Music//2020/06/06/1100 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: _ 40 The_Huey_Show:_Uptown_meets_Downtown.m4a stat: _ file: /local/av/radio/6Music//2020/06/06/1100/The_Huey_Show:_Uptown_meets_Downtown.m4a closedir: /local/av/radio/6Music//2020/06/06/1100 closedir: /local/av/radio/6Music//2020/06/06 readdir: D 2 13 stat: D recurse: /local/av/radio/6Music//2020/06/13 opendir: /local/av/radio/6Music//2020/06/13 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: D 4 1100 stat: D recurse: /local/av/radio/6Music//2020/06/13/1100 opendir: /local/av/radio/6Music//2020/06/13/1100 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: _ 29 The_Huey_Show:_Heads_Know.m4a stat: _ file: /local/av/radio/6Music//2020/06/13/1100/The_Huey_Show:_Heads_Know.m4a closedir: /local/av/radio/6Music//2020/06/13/1100 closedir: /local/av/radio/6Music//2020/06/13 readdir: D 2 27 stat: D recurse: /local/av/radio/6Music//2020/06/27 opendir: /local/av/radio/6Music//2020/06/27 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: D 4 1100 stat: D recurse: /local/av/radio/6Music//2020/06/27/1100 opendir: /local/av/radio/6Music//2020/06/27/1100 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: _ 41 The_Huey_Show:_Louie_Vega_Block_Party.m4a stat: _ file: /local/av/radio/6Music//2020/06/27/1100/The_Huey_Show:_Louie_Vega_Block_Party.m4a closedir: /local/av/radio/6Music//2020/06/27/1100 closedir: /local/av/radio/6Music//2020/06/27 closedir: /local/av/radio/6Music//2020/06 readdir: D 2 07 stat: D recurse: /local/av/radio/6Music//2020/07 opendir: /local/av/radio/6Music//2020/07 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: D 2 04 stat: D recurse: /local/av/radio/6Music//2020/07/04 opendir: /local/av/radio/6Music//2020/07/04 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: D 4 1100 stat: D recurse: /local/av/radio/6Music//2020/07/04/1100 opendir: /local/av/radio/6Music//2020/07/04/1100 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: _ 52 The_Huey_Show:_Brazil_Block_Party_and_Louie_Vega.m4a stat: _ file: /local/av/radio/6Music//2020/07/04/1100/The_Huey_Show:_Brazil_Block_Party_and_Louie_Vega.m4a closedir: /local/av/radio/6Music//2020/07/04/1100 closedir: /local/av/radio/6Music//2020/07/04 readdir: D 2 11 stat: D recurse: /local/av/radio/6Music//2020/07/11 opendir: /local/av/radio/6Music//2020/07/11 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: D 4 1100 stat: D recurse: /local/av/radio/6Music//2020/07/11/1100 opendir: /local/av/radio/6Music//2020/07/11/1100 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: _ 35 The_Huey_Show:_Cuba_Block_Party.m4a stat: _ file: /local/av/radio/6Music//2020/07/11/1100/The_Huey_Show:_Cuba_Block_Party.m4a closedir: /local/av/radio/6Music//2020/07/11/1100 closedir: /local/av/radio/6Music//2020/07/11 readdir: D 2 18 stat: D recurse: /local/av/radio/6Music//2020/07/18 opendir: /local/av/radio/6Music//2020/07/18 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: D 4 1100 stat: D recurse: /local/av/radio/6Music//2020/07/18/1100 opendir: /local/av/radio/6Music//2020/07/18/1100 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: _ 42 The_Huey_Show:_Puerto_Rico_Block_Party.m4a stat: _ file: /local/av/radio/6Music//2020/07/18/1100/The_Huey_Show:_Puerto_Rico_Block_Party.m4a closedir: /local/av/radio/6Music//2020/07/18/1100 closedir: /local/av/radio/6Music//2020/07/18 closedir: /local/av/radio/6Music//2020/07 readdir: D 2 08 stat: D recurse: /local/av/radio/6Music//2020/08 opendir: /local/av/radio/6Music//2020/08 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: D 2 01 stat: D recurse: /local/av/radio/6Music//2020/08/01 opendir: /local/av/radio/6Music//2020/08/01 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: D 4 1100 stat: D recurse: /local/av/radio/6Music//2020/08/01/1100 opendir: /local/av/radio/6Music//2020/08/01/1100 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: _ 35 The_Huey_Show:_Chuck_D_turns_60.m4a stat: _ file: /local/av/radio/6Music//2020/08/01/1100/The_Huey_Show:_Chuck_D_turns_60.m4a closedir: /local/av/radio/6Music//2020/08/01/1100 closedir: /local/av/radio/6Music//2020/08/01 readdir: D 2 08 stat: D recurse: /local/av/radio/6Music//2020/08/08 opendir: /local/av/radio/6Music//2020/08/08 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: D 4 1100 stat: D recurse: /local/av/radio/6Music//2020/08/08/1100 opendir: /local/av/radio/6Music//2020/08/08/1100 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: _ 59 The_Huey_Show:_New_music_from_Felt_Skyzoo_and_Sly5thAve.m4a stat: _ file: /local/av/radio/6Music//2020/08/08/1100/The_Huey_Show:_New_music_from_Felt_Skyzoo_and_Sly5thAve.m4a closedir: /local/av/radio/6Music//2020/08/08/1100 closedir: /local/av/radio/6Music//2020/08/08 readdir: D 2 15 stat: D recurse: /local/av/radio/6Music//2020/08/15 opendir: /local/av/radio/6Music//2020/08/15 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: D 4 1100 stat: D recurse: /local/av/radio/6Music//2020/08/15/1100 opendir: /local/av/radio/6Music//2020/08/15/1100 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: _ 17 The_Huey_Show.m4a stat: _ file: /local/av/radio/6Music//2020/08/15/1100/The_Huey_Show.m4a closedir: /local/av/radio/6Music//2020/08/15/1100 closedir: /local/av/radio/6Music//2020/08/15 readdir: D 2 22 stat: D recurse: /local/av/radio/6Music//2020/08/22 opendir: /local/av/radio/6Music//2020/08/22 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: D 4 1100 stat: D recurse: /local/av/radio/6Music//2020/08/22/1100 opendir: /local/av/radio/6Music//2020/08/22/1100 readdir: D 1 . stat: D readdir: D 2 .. stat: D readdir: _ 17 The_Huey_Show.m4a stat: _ file: /local/av/radio/6Music//2020/08/22/1100/The_Huey_Show.m4a closedir: /local/av/radio/6Music//2020/08/22/1100 closedir: /local/av/radio/6Music//2020/08/22 closedir: /local/av/radio/6Music//2020/08 closedir: /local/av/radio/6Music//2020 closedir: /local/av/radio/6Music/ --==========16324982B0D1D3197C7A==========-- From owner-freebsd-fs@freebsd.org Thu Apr 29 11:18:36 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EE69B623CAF for ; Thu, 29 Apr 2021 11:18:36 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4FWCfz6RR2z4ZTh for ; Thu, 29 Apr 2021 11:18:35 +0000 (UTC) (envelope-from ronald-lists@klop.ws) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=klop.ws; s=mail; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version: Date:Message-ID:From:References:To:Subject:Sender:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=8d/PpHBGaxtRYmrSTUhySYQuPNnbp0eFTxJQ0WVHL2k=; b=n+6HwXCI/8JqVo9U3Ppbe9D5AG NlYW2BdEVhtf64onYHVGBsim2+Zwzt837FmtrB8dIpu92qfBaxuIOyW8ukmiD3sS5b8sruk303kA8 dVvtLtDtPDTkIUGvNMcBaDVTrBFWKnANLHQTY6QDBjRcvZtw+1SDFb8zU+DiTfLFzbPk=; Subject: Re: readdir() -> d_type always zero on NFS v3 mounted filesystems To: freebsd-fs@freebsd.org References: <2E84A420CCC10A73504624DE@triton.njm.me.uk> From: Ronald Klop Message-ID: <78c159ef-947a-8e2f-53bf-8d27b80193bf@klop.ws> Date: Thu, 29 Apr 2021 13:14:53 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.10.0 MIME-Version: 1.0 In-Reply-To: <2E84A420CCC10A73504624DE@triton.njm.me.uk> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Authenticated-As-Hash: 398f5522cb258ce43cb679602f8cfe8b62a256d1 X-Virus-Scanned: by clamav at smarthost1.greenhost.nl X-Spam-Level: / X-Spam-Score: -0.4 X-Spam-Status: No, score=-0.4 required=5.0 tests=ALL_TRUSTED, BAYES_50, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, NICE_REPLY_A autolearn=disabled version=3.4.2 X-Scan-Signature: dd305f484f1ad876e240bae2a9803cbd X-Rspamd-Queue-Id: 4FWCfz6RR2z4ZTh X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=klop.ws header.s=mail header.b=n+6HwXCI; dmarc=pass (policy=none) header.from=klop.ws; spf=pass (mx1.freebsd.org: domain of ronald-lists@klop.ws designates 195.190.28.88 as permitted sender) smtp.mailfrom=ronald-lists@klop.ws X-Spamd-Result: default: False [-4.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[klop.ws:s=mail]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:195.190.28.64/27]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[klop.ws:+]; DMARC_POLICY_ALLOW(-0.50)[klop.ws,none]; RCVD_IN_DNSWL_NONE(0.00)[195.190.28.88:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RWL_MAILSPIKE_VERYGOOD(0.00)[195.190.28.88:from]; ASN(0.00)[asn:47172, ipnet:195.190.28.0/24, country:NL]; MID_RHS_MATCH_FROM(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Apr 2021 11:18:37 -0000 On 4/29/21 10:24 AM, N.J. Mann wrote: > Hi, > > > I recently changed over from using svn to gitup to update /usr/ports > on my local system and have been experiencing problems since. At first > I thought it was an issue with gitup itself, but now I believe it is > either a kernel issue or a configuration issue. I originally posted > about the problem to the freebsd-ports mailing list: > https://lists.freebsd.org/pipermail/freebsd-ports/2021-April/120929.html > > Since then I have dug deeper and come to the conclusion that it is not a > problem with gitup. > > The issue I am seeing is that gitup is unable to delete files and > directories, even complete ports, which have been removed from the > repository. gitup basically does the following: > > prune_tree(base_path) > { > if ((directory = opendir(base_path)) != NULL) { > while ((entry = readdir(directory)) != NULL) { > snprintf(full_path, sizeof(full_path), "%s/%s", base_path, entry->d_name); > if (entry->d_type == DT_DIR) { > prune_tree(full_path); > } else { > if ((remove(full_path) != 0) && (errno != ENOENT)) > err(EXIT_FAILURE, "prune_tree: cannot remove %s", full_path); > } > } > closedir(directory); > if (rmdir(base_path) != 0) > err(EXIT_FAILURE, "prune_tree: cannot remove %s", base_path); > } > } > > When gitup is run on either a UFS or ZFS file system this works. However, > when I run it on a NFS v3 mounted filesystem it fails. Liberal addition > of printf's shows that for non-NFS mounted filesystems d_type contains the > correct value for each file/directory, but for NFS mounted file systems it > is zero - other fields such as d_name and d_namlen are correct. While > debugging this I added a call to stat() and that always returns the correct > values. > > At this point I started digging in libc and quickly found that readdir() > is basically a wrapper around a system call. Digging in the kernel I > quickly got out of my depth and hence my posting here. I then wrote a > simple test programme which also shows the issue. I have attached the > source for the test programme and the output from two runs, the first on > an NFS mounted file system and the second on a local UFS file system. > > Before gitup started failing I had not seen any failures like this and that > was why I initially assumed this must be a gitup issue. I now believe this > is either a kernel issue or a confuration issue. > > My configuration is as follows: > > file server: > /exports/ports - ZFS file system for FreeBSD ports repo exported via NFS > /remote/ports - FreeBSD ports repo NFS (v3 rw,tcp) mounted from /export/ports > on file server > /usr/ports - symbolic link to /remote/ports > client machines: > /remote/ports - FreeBSD ports repo NFS (v3 ro,tcp) mounted from /export/ports > on file server > /usr/ports - symbolic link to /remote/ports > > I run gitup in /remote/ports on the file server and then pkg, portmaster, &.c, > in /usr/ports on the file server and then the client machines. All systems are > running 11-STABLE from about 12 days ago - I usually update every couple of weeks. > > Any assistance, suggestions, patches, clue bats, will be gratefully accepted. > > > Regards, > Nick. > Nice analysis. The manual page dirent(5) says: "BUGS The usage of the member d_type of struct dirent is unportable as it is FreeBSD-specific. It also may fail on certain file systems, for example the cd9660 file system. " I think gitup would be more standards compliant if it used stat(2) for determining the type of a file/dir. An issue about this can best be reported at the upstream project: https://github.com/johnmehr/gitup . If the NFS server/client can implement d_type is outside of my knowledge. Regards, Ronald. From owner-freebsd-fs@freebsd.org Thu Apr 29 11:20:48 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9FC03623E0D for ; Thu, 29 Apr 2021 11:20:48 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4FWCjX0tHXz4ZZH for ; Thu, 29 Apr 2021 11:20:47 +0000 (UTC) (envelope-from ronald-lists@klop.ws) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=klop.ws; s=mail; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version: Date:Message-ID:From:References:To:Subject:Sender:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=1VTlhayvmGIOfLIFxzXd9vb8ZjxuA+ZrYqhoTuWMCxU=; b=hTehty9X6q/hK7uDgoPVCfhqNf x/w0i2nbco7eUb6m9vrUeiMslLqrA8CxRPSH5amPoQw+OgO5ENUyEFQgqCUX6q0FE5KIDyOq59pb1 d3HwADZZP6+a0cbtqrHnLUm5n0HGYMebq3qD4IfV/NezVereMh/4qDxnYS0E/LdMSSXI=; Subject: Re: readdir() -> d_type always zero on NFS v3 mounted filesystems To: freebsd-fs@freebsd.org References: <2E84A420CCC10A73504624DE@triton.njm.me.uk> <78c159ef-947a-8e2f-53bf-8d27b80193bf@klop.ws> From: Ronald Klop Message-ID: <5b55a633-ebd9-30bb-cc0a-66879474a123@klop.ws> Date: Thu, 29 Apr 2021 13:17:13 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.10.0 MIME-Version: 1.0 In-Reply-To: <78c159ef-947a-8e2f-53bf-8d27b80193bf@klop.ws> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Authenticated-As-Hash: 398f5522cb258ce43cb679602f8cfe8b62a256d1 X-Virus-Scanned: by clamav at smarthost1.greenhost.nl X-Spam-Level: / X-Spam-Score: -0.4 X-Spam-Status: No, score=-0.4 required=5.0 tests=ALL_TRUSTED, BAYES_50, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, NICE_REPLY_A autolearn=disabled version=3.4.2 X-Scan-Signature: 4c1f0696016754537de762f13eb96caa X-Rspamd-Queue-Id: 4FWCjX0tHXz4ZZH X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=klop.ws header.s=mail header.b=hTehty9X; dmarc=pass (policy=none) header.from=klop.ws; spf=pass (mx1.freebsd.org: domain of ronald-lists@klop.ws designates 195.190.28.88 as permitted sender) smtp.mailfrom=ronald-lists@klop.ws X-Spamd-Result: default: False [-4.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[klop.ws:s=mail]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:195.190.28.64/27:c]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[klop.ws:+]; DMARC_POLICY_ALLOW(-0.50)[klop.ws,none]; RCVD_IN_DNSWL_NONE(0.00)[195.190.28.88:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RWL_MAILSPIKE_VERYGOOD(0.00)[195.190.28.88:from]; ASN(0.00)[asn:47172, ipnet:195.190.28.0/24, country:NL]; MID_RHS_MATCH_FROM(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Apr 2021 11:20:48 -0000 On 4/29/21 1:14 PM, Ronald Klop wrote: > On 4/29/21 10:24 AM, N.J. Mann wrote: >> Hi, >> >> >> I recently changed over from using svn to gitup to update /usr/ports >> on my local system and have been experiencing problems since.  At first >> I thought it was an issue with gitup itself, but now I believe it is >> either a kernel issue or a configuration issue.  I originally posted >> about the problem to the freebsd-ports mailing list: >> https://lists.freebsd.org/pipermail/freebsd-ports/2021-April/120929.html >> >> Since then I have dug deeper and come to the conclusion that it is not a >> problem with gitup. >> >> The issue I am seeing is that gitup is unable to delete files and >> directories, even complete ports, which have been removed from the >> repository.  gitup basically does the following: >> >> prune_tree(base_path) >> { >> if ((directory = opendir(base_path)) != NULL) { >>      while ((entry = readdir(directory)) != NULL) { >>          snprintf(full_path, sizeof(full_path), "%s/%s", base_path, entry->d_name); >>          if (entry->d_type == DT_DIR) { >>              prune_tree(full_path); >>          } else { >>              if ((remove(full_path) != 0) && (errno != ENOENT)) >>                   err(EXIT_FAILURE, "prune_tree: cannot remove %s", full_path); >>              } >>          } >>          closedir(directory); >>          if (rmdir(base_path) != 0) >>              err(EXIT_FAILURE, "prune_tree: cannot remove %s", base_path); >>      } >> } >> >> When gitup is run on either a UFS or ZFS file system this works.  However, >> when I run it on a NFS v3 mounted filesystem it fails.  Liberal addition >> of printf's shows that for non-NFS mounted filesystems d_type contains the >> correct value for each file/directory, but for NFS mounted file systems it >> is zero - other fields such as d_name and d_namlen are correct.  While >> debugging this I added a call to stat() and that always returns the correct >> values. >> >> At this point I started digging in libc and quickly found that readdir() >> is basically a wrapper around a system call.  Digging in the kernel I >> quickly got out of my depth and hence my posting here.  I then wrote a >> simple test programme which also shows the issue.  I have attached the >> source for the test programme and the output from two runs, the first on >> an NFS mounted file system and the second on a local UFS file system. >> >> Before gitup started failing I had not seen any failures like this and that >> was why I initially assumed this must be a gitup issue.  I now believe this >> is either a kernel issue or a confuration issue. >> >> My configuration is as follows: >> >> file server: >>    /exports/ports - ZFS file system for FreeBSD ports repo exported via NFS >>    /remote/ports  - FreeBSD ports repo NFS (v3 rw,tcp) mounted from /export/ports >>                     on file server >>    /usr/ports     - symbolic link to /remote/ports >> client machines: >>    /remote/ports  - FreeBSD ports repo NFS (v3 ro,tcp) mounted from /export/ports >>                     on file server >>    /usr/ports     - symbolic link to /remote/ports >> >> I run gitup in /remote/ports on the file server and then pkg, portmaster, &.c, >> in /usr/ports on the file server and then the client machines.  All systems are >> running 11-STABLE from about 12 days ago - I usually update every couple of weeks. >> >> Any assistance, suggestions, patches, clue bats, will be gratefully accepted. >> >> >> Regards, >>          Nick. >> > > > Nice analysis. The manual page dirent(5) says: > "BUGS >     The usage of the member d_type of struct dirent is unportable as it is >     FreeBSD-specific.  It also may fail on certain file systems, for example >     the cd9660 file system. > " > > I think gitup would be more standards compliant if it used stat(2) for determining the type of a file/dir. An issue about this can best be reported at the upstream project: https://github.com/johnmehr/gitup . Ah, somebody already found it: https://github.com/johnmehr/gitup/issues/67 You could add the information from the readdir(5) and stat(2) man pages. Regards, Ronald. > If the NFS server/client can implement d_type is outside of my knowledge. > > Regards, > Ronald. > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Thu Apr 29 14:39:19 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5C19162B2B2 for ; Thu, 29 Apr 2021 14:39:19 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-qb1can01on0625.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe5c::625]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FWJ6Z043Lz4kRX for ; Thu, 29 Apr 2021 14:39:17 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=B54O3novzPMeWGLwbQtsEw9qqJavuvbSRt4MnQ+wKkw/UtXbfWt3DPluuN48sx1yMZtw6wUY08KWuD7SbJcMsh+8bfx/lKb9AAXPqqlhhGrNdd6UkN78VtvMY0ATjH0CuMJFySZFMPm7F+fxMX3tq01rliTWjB7ZI5pB0ZCEIf8YtfCYNu803Wrr7kf1ODCH4OlD8gUTbmIYAuKAGYKAgT59KvSWcnYZlvMkDzTqaRiGXBIDE3BOMvQdZQmJtIdU4rL5198chu183gGbuiE2kEnOw318LOSOvrXRIaIRTLoeEW6JW7eFyGp6NJhKPk+lKe/Qmh3aUOF3OpF7zSSE7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sQEre/RAytbQeIdNfm1XpWwP3nNcZ7vO4UbskwG8AJ0=; b=cYRiJt9Bm3cdj0qFscpSjkmaPCr61HxH8prJNp1C/ma2UUbVbzxZZuDZ5O6oRscupT7ZUi9nFQDTodOD51yaT0EyhGYSxsOi2cNXbod0gdDL/b1L5D+RePowbIWy0mFtpUx+4R0O4kOJTIjOXQZUpvsX+VF1057rcXZ9oT4b4mDaV3EcO/Qp/E8LdtvL6ICPF8HFyNy9QXIuFfIT18jZ9qhuNOtXjqRUCyiVY5NTFBwI0KejgKnKVe7m9qVWB30+9enygiO9xK+8/9yJyqyKjStxCMz3XmAO4QY5TPEa7FO5W8VxMieCGHU9maSzulQ4uDjz1tF3Kra9gNSP4tjplA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sQEre/RAytbQeIdNfm1XpWwP3nNcZ7vO4UbskwG8AJ0=; b=CNuR6GOJzD2AYdAPRi5ESZajSTIZHa05feuer0jcLG1SAyyB3CX2ZGHqywqZHOnh7g+MVi4SgRlBGp3od+wFxfbyor0D+dKHvw1aSXnB8KzstR9NHVeVAwg/GHWvWNWuZSWM47KjGsgbSHpPg0Cv+MPVWB+VRxFbOZhvhXWyFqW48A2r1JKheIzIFwenoJOs8fcZT1jUYTk2tqShnWyX5iOhkx4QIr7Jv5nzyfUr9jztIzjh0XAUEu098brjArUqT6uMopWnT7d1dXwfLD8pc5rHbBgXS889lerldE6D7FhVeD2hHaSgqQJwXBAUiH6WStpwrifRsCcFZyxi48T7qA== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQXPR01MB3590.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:46::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4065.25; Thu, 29 Apr 2021 14:39:15 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::1c05:585a:132a:f08e]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::1c05:585a:132a:f08e%4]) with mapi id 15.20.4065.027; Thu, 29 Apr 2021 14:39:10 +0000 From: Rick Macklem To: "N.J. Mann" , "freebsd-fs@freebsd.org" Subject: Re: readdir() -> d_type always zero on NFS v3 mounted filesystems Thread-Topic: readdir() -> d_type always zero on NFS v3 mounted filesystems Thread-Index: AQHXPNEVpMPpA4XUr0WbMbVrubCkgqrLh/S4 Date: Thu, 29 Apr 2021 14:39:10 +0000 Message-ID: References: <2E84A420CCC10A73504624DE@triton.njm.me.uk> In-Reply-To: <2E84A420CCC10A73504624DE@triton.njm.me.uk> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 50c4a044-05c5-447f-d4ca-08d90b1c8d21 x-ms-traffictypediagnostic: YQXPR01MB3590: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:4941; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 4//DHJYSCeNaSZT3dkUjvlCI0Ifxn1YYedTK9ZDtzZl+uECsVhJYpjf53t9WQdrScQntU2E8BWB7exzxkCsH8xWkxYOlXDmmFR8LksT+g4K+eegLKuP0Rd7/Yvn0QPJsv2R2pNAGx+qVJcuD4XlMdu5C14cN3QJUpstCncPDfVDmZKMZh2ieVz7joBj9nyut6zU7ae4c1vXYpssxSpNSaz3Empi53leruakxU28P68XxMuiwfbsQFy7v6M/etYQYCiaRdrx6p6rw0jSoEffKhUhd/JKqC32vaXKx9vLDTjva4mJ0L5GIaYJWzazG6oi12H0xaLM+HIB7M1GeR06aosw9GYS/MFr06kwdPVv+sd8nO3A7Y8EJKBl+I7XVFy4ONmLQW4F7P5ND5Itg++zfwI2uU7yYPLbUJRIhovToZUEEE73hf6+altna0dH0/ve+tk9oosGkyJkhzm1nyVZ+WQ3Q+E6sbXbQB0eldLpUARUzSRRLh4g8AfALaxQJgKzB0Qf7TiUU/jwUvl/6ikJuOXl6XBo034QR8OMSWR74DZBwC4GV5oCk9f8ekS8BGHypGA/7lHhI8xGIrSNzgEW5PfnP6622t0FIEuNJF8hqLrxgDsouf890DaE/nlFlzXZwaCXOQZfmrOFm79bYbMa9qPyAvAMe8ofMOTFwNEvu/hvZ5eeeTkYtmFmyEhJSvxx2 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(376002)(366004)(136003)(396003)(39850400004)(346002)(186003)(76116006)(7696005)(66476007)(66446008)(66556008)(6506007)(122000001)(38100700002)(64756008)(91956017)(66946007)(8936002)(83380400001)(8676002)(786003)(110136005)(316002)(478600001)(71200400001)(9686003)(55016002)(86362001)(33656002)(52536014)(2906002)(5660300002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?6SIIzpRGVX1gGSqaCiKdpml8f8Z/KMZCPU+kZeR4JYmu/VgMYx520j8heR?= =?iso-8859-1?Q?FVNznXSj/FlULHZ6kZ66r5/tofAogU4ERqUs4xtH2Rg75C0t/d4XSCKQAL?= =?iso-8859-1?Q?cBCJg/dFzT8O6kNvp2daLxHS3XK/lT+oJdrmqQ6w2d1eXfu60iKXCFXOb1?= =?iso-8859-1?Q?4ukrEaXeVWao99Pa5gWt7ifdq3xm6XhNtFfuCqDiKL0+fIkjnqqqsR5+Zb?= =?iso-8859-1?Q?xemELzh063pgrN3ETPvaX1TqBj4xgQR5PPwRvYgTjj7ntWBd3mOlAmXqA0?= =?iso-8859-1?Q?3462yaKA6SY1PGBzvyEYGw11BAIb6fcmOfiMiDmUH7aPgBCgNwwpgINj74?= =?iso-8859-1?Q?8Y0QUiyRzLk4YH5BtdH/NknTORIdRx/Va4dEAmb6ft1/BKWkm2I4fxJRv9?= =?iso-8859-1?Q?O/4beNJfKE/apHWEhWV4lyR0hf2MimH5y3Xre+uVjS3c2iOVs3Sco6BjWQ?= =?iso-8859-1?Q?qQNtaIHWmaVpv204G2SdLGY5krQEFQi9NlQryi25CRkdgxNP/dwb8Fp/7N?= =?iso-8859-1?Q?TBRi56TZ2moqqucZfs9ovsthsdkbShfRNpRuTsfpUrg44qMa/d9JiqeO0M?= =?iso-8859-1?Q?c8TnV52xCvMaMPkixLKt65c3u3zG+3RaTCrGa4LgakDcT0RoWJWYGycIDu?= =?iso-8859-1?Q?kRdZoL9Id/ZrxTDRhe+ZJsRsRsTBkvXezrTE7Ir1xFjwAiI52aljQzui77?= =?iso-8859-1?Q?MR8MWnAnlsPusuMgQQf58KQmn9DTv/SulKG0X3zCfRtHXM3fEFiQJfKNh2?= =?iso-8859-1?Q?drTjTgPIW4ktCjuI5lNRl4JdJ+9PnUqggkddU+Gbyi5YtEgPCrDYJi0F4Q?= =?iso-8859-1?Q?oiDGePcW5/ucbEnrf02v45MULzILEPxXjosNEQUms6JVAddJEqaWOMjwTq?= =?iso-8859-1?Q?IogO0qb25HEMRCyq0lnNulwElUg1hBMRHLI8BcBeHUmDPoNaiNpPt0Wcbj?= =?iso-8859-1?Q?XYW8kruAv8Gbb+ODd1fiZDIosdTGYCuFvTx/8//eJ7a3tkCbUHLSHwtOh1?= =?iso-8859-1?Q?xVUcwT1o438agXYS36DOyF7QSy4HePYUFuIY8JM8pmYD9DU2fOEZ0eYT0M?= =?iso-8859-1?Q?Ym4Ejazyy1IEc9c0yfPYGarH0TmclfUEbl3KAtKBMZWMSdcT+fMQXtGMef?= =?iso-8859-1?Q?086kzPGpUfw91tPty9NCRVIkAElZ+7geVZIPcwf62uix418qtIvcA61/V6?= =?iso-8859-1?Q?ECqAFrpgjnxNjMAfl/OiHg8x66MVQSA9DptkfniyP+cwwoS6p7e8Ejuem9?= =?iso-8859-1?Q?/ObSfmtSwXU+XsawolDoBMKAg/Jznf7X0tMKmrB8QyVLjqv+a1wbTFEYfy?= =?iso-8859-1?Q?mOBq++i+qFbDzn1DpQGgxaORD/8R5epcWoUUiJljTj8+7v/moLAmpX6Sbz?= =?iso-8859-1?Q?wShDxF9ekKS1i4IIKYhMC8R1yZgWua31e7wyTRmiu7OIerSCHWn8kkv9Dp?= =?iso-8859-1?Q?4Vl98IUNIWE/1NQs?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 50c4a044-05c5-447f-d4ca-08d90b1c8d21 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Apr 2021 14:39:10.3510 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: wbTw8qCOh3JT9Z23JAsqal/986IzFrr7w3uQd5pduTKI+SLWE0trfDQOy+goi1IDZaoIKIo3ahCSwJ7qodKiRw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB3590 X-Rspamd-Queue-Id: 4FWJ6Z043Lz4kRX X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=CNuR6GOJ; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 2a01:111:f400:fe5c::625 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a01:111:f400:fe5c::625:from]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f400::/48]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; SPAMHAUS_ZRD(0.00)[2a01:111:f400:fe5c::625:from:127.0.2.255]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Apr 2021 14:39:19 -0000 N.J. Mann wrote:=0A= >I recently changed over from using svn to gitup to update /usr/ports=0A= >on my local system and have been experiencing problems since. At first=0A= >either a kernel issue or a configuration issue. I originally posted=0A= >about the problem to the freebsd-ports mailing list:=0A= >https://lists.freebsd.org/pipermail/freebsd-ports/2021-April/120929.html= =0A= >=0A= >Since then I have dug deeper and come to the conclusion that it is not a= =0A= >problem with gitup.=0A= >=0A= >The issue I am seeing is that gitup is unable to delete files and=0A= >directories, even complete ports, which have been removed from the=0A= >repository. gitup basically does the following:=0A= >=0A= >prune_tree(base_path)=0A= >{=0A= >if ((directory =3D opendir(base_path)) !=3D NULL) {=0A= > while ((entry =3D readdir(directory)) !=3D NULL) {=0A= > snprintf(full_path, sizeof(full_path), "%s/%s", base_path, entry->= d_name);=0A= > if (entry->d_type =3D=3D DT_DIR) {=0A= > prune_tree(full_path);=0A= > } else {=0A= > if ((remove(full_path) !=3D 0) && (errno !=3D ENOENT))=0A= > err(EXIT_FAILURE, "prune_tree: cannot remove %s", full_pa= th);=0A= > }=0A= > }=0A= > closedir(directory);=0A= > if (rmdir(base_path) !=3D 0)=0A= > err(EXIT_FAILURE, "prune_tree: cannot remove %s", base_path);= =0A= > }=0A= >}=0A= =0A= The code should check for d_type =3D=3D DT_UNKNOWN and then do=0A= stat() to find out the type, as Ronald noted.=0A= --> The d_type is normally filled in if you use the "rdirplus"=0A= NFS mount option, which might work around the issue.=0A= =0A= rick=0A= =0A= closedir()=0A= =0A= =0A= When gitup is run on either a UFS or ZFS file system this works. However,= =0A= when I run it on a NFS v3 mounted filesystem it fails. Liberal addition=0A= of printf's shows that for non-NFS mounted filesystems d_type contains the= =0A= correct value for each file/directory, but for NFS mounted file systems it= =0A= is zero - other fields such as d_name and d_namlen are correct. While=0A= debugging this I added a call to stat() and that always returns the correct= =0A= values.=0A= =0A= At this point I started digging in libc and quickly found that readdir()=0A= is basically a wrapper around a system call. Digging in the kernel I=0A= quickly got out of my depth and hence my posting here. I then wrote a=0A= simple test programme which also shows the issue. I have attached the=0A= source for the test programme and the output from two runs, the first on=0A= an NFS mounted file system and the second on a local UFS file system.=0A= =0A= Before gitup started failing I had not seen any failures like this and that= =0A= was why I initially assumed this must be a gitup issue. I now believe this= =0A= is either a kernel issue or a confuration issue.=0A= =0A= My configuration is as follows:=0A= =0A= file server:=0A= /exports/ports - ZFS file system for FreeBSD ports repo exported via NFS= =0A= /remote/ports - FreeBSD ports repo NFS (v3 rw,tcp) mounted from /export/= ports=0A= on file server=0A= /usr/ports - symbolic link to /remote/ports=0A= client machines:=0A= /remote/ports - FreeBSD ports repo NFS (v3 ro,tcp) mounted from /export/= ports=0A= on file server=0A= /usr/ports - symbolic link to /remote/ports=0A= =0A= I run gitup in /remote/ports on the file server and then pkg, portmaster, &= .c,=0A= in /usr/ports on the file server and then the client machines. All systems= are=0A= running 11-STABLE from about 12 days ago - I usually update every couple of= weeks.=0A= =0A= Any assistance, suggestions, patches, clue bats, will be gratefully accepte= d.=0A= =0A= =0A= Regards,=0A= Nick.=0A= --=0A=