From owner-freebsd-fs@freebsd.org Mon Apr 26 12:24:52 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 344255FE4B1 for ; Mon, 26 Apr 2021 12:24:52 +0000 (UTC) (envelope-from SRS0=7Nu3=JX=perdition.city=julien@bebif.be) Received: from orval.bbpf.belspo.be (orval.bbpf.belspo.be [193.191.208.90]) by mx1.freebsd.org (Postfix) with ESMTP id 4FTPGp73QNz3KRK for ; Mon, 26 Apr 2021 12:24:50 +0000 (UTC) (envelope-from SRS0=7Nu3=JX=perdition.city=julien@bebif.be) Received: from x1 (94.105.107.231.dyn.edpnet.net [94.105.107.231]) by orval.bbpf.belspo.be (Postfix) with ESMTPSA id 7A4B71D4FC1B for ; Mon, 26 Apr 2021 14:24:43 +0200 (CEST) Date: Mon, 26 Apr 2021 14:24:40 +0200 From: Julien Cigar To: freebsd-fs@freebsd.org Subject: iSCSI SAN Message-ID: <20210426122440.xirux6bwdztwsm5c@x1> Mail-Followup-To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline X-Rspamd-Queue-Id: 4FTPGp73QNz3KRK X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of SRS0=7Nu3=JX=perdition.city=julien@bebif.be designates 193.191.208.90 as permitted sender) smtp.mailfrom=SRS0=7Nu3=JX=perdition.city=julien@bebif.be X-Spamd-Result: default: False [-2.40 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[193.191.208.90:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[193.191.208.90:from:127.0.2.255]; MID_RHS_NOT_FQDN(0.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_NA(0.00)[perdition.city]; FORGED_SENDER(0.30)[julien@perdition.city,SRS0=7Nu3=JX=perdition.city=julien@bebif.be]; RCVD_NO_TLS_LAST(0.10)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:2611, ipnet:193.191.192.0/19, country:BE]; FROM_NEQ_ENVFROM(0.00)[julien@perdition.city,SRS0=7Nu3=JX=perdition.city=julien@bebif.be]; MAILMAN_DEST(0.00)[freebsd-fs]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Apr 2021 12:24:52 -0000 Hello, I'm wondering if something has already been written to implement a fully redundant and highly available FreeBSD ZFS based iSCSI SAN? I've setup some FreeBSD iSCSI SAN-like in the past (for small structures) and it has always worked well. However upgrades have always been painfull and, although there is ZFS, redundant power supplies, redundant switches with multipath, it's still a SPOF if a non-redundant component dies, like the motherboard. It's not like an HPE MSA-like system where everything is redundant out of the box. So the idea came to me for the iSCSI target to setup 2 physical servers with a bunch of disks, create some raidzx on them and export one ZFS volume per initiator on each target, a bit like on (1) I've tried to setup that in a small "lab", with some jails, gmultipath, two switches, and several VLANs. Unfortunately no 10 gbits to test, but 3x1Gbits LAGG with LACP. The downside of this setup is that "half" of the storage is (temporarily) lost when a target reboots (freebsd-update, upgrades, etc), which de-facto disqualifies gmirror + UFS on the initiator side as a full resync of required and takes ages. With ZFS you don't have this problem as only the delta is resync. For the few tests I did it seems to work, but I'm wondering: zfs over zvol .. is it sane? does it makes sense? could I disable checksum on the initiator side to speed up things? do you see any race condition or ... with this setup (sync, etc)? What do you think? Thanks! Julien (1) https://gist.github.com/silenius/c6d1020aca54c47f71aa9f2a19a55ffe -- Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced.