From owner-freebsd-fs@freebsd.org Mon Apr 26 19:54:52 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1BFCE5EE33C for ; Mon, 26 Apr 2021 19:54:52 +0000 (UTC) (envelope-from SRS0=7Nu3=JX=perdition.city=julien@bebif.be) Received: from orval.bbpf.belspo.be (orval.bbpf.belspo.be [193.191.208.90]) by mx1.freebsd.org (Postfix) with ESMTP id 4FTbG30vSDz4W9L for ; Mon, 26 Apr 2021 19:54:50 +0000 (UTC) (envelope-from SRS0=7Nu3=JX=perdition.city=julien@bebif.be) Received: from x1 (94.105.107.231.dyn.edpnet.net [94.105.107.231]) by orval.bbpf.belspo.be (Postfix) with ESMTPSA id 818B81D4FC10; Mon, 26 Apr 2021 21:54:49 +0200 (CEST) Date: Mon, 26 Apr 2021 21:54:46 +0200 From: Julien Cigar To: joris dedieu Cc: freebsd-fs@freebsd.org Subject: Re: iSCSI SAN Message-ID: <20210426195446.ih2s3iyvhdbp2po2@x1> Mail-Followup-To: joris dedieu , freebsd-fs@freebsd.org References: <20210426122440.xirux6bwdztwsm5c@x1> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 4FTbG30vSDz4W9L X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of SRS0=7Nu3=JX=perdition.city=julien@bebif.be designates 193.191.208.90 as permitted sender) smtp.mailfrom=SRS0=7Nu3=JX=perdition.city=julien@bebif.be X-Spamd-Result: default: False [-2.40 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; FORGED_SENDER(0.30)[julien@perdition.city,SRS0=7Nu3=JX=perdition.city=julien@bebif.be]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[perdition.city]; SPAMHAUS_ZRD(0.00)[193.191.208.90:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[193.191.208.90:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; MID_RHS_NOT_FQDN(0.50)[]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_NO_TLS_LAST(0.10)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:2611, ipnet:193.191.192.0/19, country:BE]; FROM_NEQ_ENVFROM(0.00)[julien@perdition.city,SRS0=7Nu3=JX=perdition.city=julien@bebif.be]; MAILMAN_DEST(0.00)[freebsd-fs]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Apr 2021 19:54:52 -0000 On Mon, Apr 26, 2021 at 05:19:14PM +0200, joris dedieu wrote: > Hi, > > Le lun. 26 avr. 2021 à 14:24, Julien Cigar a écrit : > > > Hello, > > > > I'm wondering if something has already been written to implement a > > fully redundant and highly available FreeBSD ZFS based iSCSI SAN? > > > > > I've setup some FreeBSD iSCSI SAN-like in the past (for small > > structures) and it has always worked well. However upgrades have always > > been painfull and, although there is ZFS, redundant power supplies, > > redundant switches with multipath, it's still a SPOF if a non-redundant > > component dies, like the motherboard. It's not like an HPE MSA-like > > system where everything is redundant out of the box. > > > > From my experience HA mechanisms are an infinite source of pain. Don't >From my experence "automatic failover" is an infinite source of pain, not HA > forget the power of simplicity. What uptime do you get with your simple > setup ? When was the last time you see a decent mainboard crash ? As long as I don't touch at it everything is fine. Now, as soon as I have to reboot the server or upgrade (from example from one major release to another) there are long minutes of downtime, which is unacceptable if many servers are connected. > Also don't forget that FreeBSD has glusterfs and ceph. > I haven't looked at CEPH yet, but I don't think everything has been ported (KRDB, CEPH native fs, etc). It is certainly not "production ready". > If you want to do something similar to proprietary chassis, you should have > to look at SAS HBA and JBOD chassis, ATAoE chassis (if it still exists) or Yes, definitively. SAS HBA with a JBOD chassis is something I was looking too .. unfortunately I don't have the hardware to experiment :( > stuff like that to attach your disks to your two mainboard. Still (OMG) > dealing with zfs (import -f) on failover, cluster STONITH and other voodoo. > You will have fully redundant design (see > https://i.stack.imgur.com/ijjpk.png ) > > Cheers > Joris > Thanks, Julien > > > So the idea came to me for the iSCSI target to setup 2 physical servers > > with a bunch of disks, create some raidzx on them and export one ZFS > > volume per initiator on each target, a bit like on (1) > > > > > I've tried to setup that in a small "lab", with some jails, gmultipath, > > two switches, and several VLANs. Unfortunately no 10 gbits to test, but > > 3x1Gbits LAGG with LACP. > > > > The downside of this setup is that "half" of the storage is (temporarily) > > lost when a target reboots (freebsd-update, upgrades, etc), which > > de-facto disqualifies gmirror + UFS on the initiator side as a full > > resync of required and takes ages. With ZFS you don't have this problem > > as only the delta is resync. For the few tests I did it seems to work, > > but I'm wondering: zfs over zvol .. is it sane? does it makes sense? > > could I disable checksum on the initiator side to speed up things? do > > you see any race condition or ... with this setup (sync, etc)? > > > > What do you think? > > > > Thanks! > > Julien > > > > (1) https://gist.github.com/silenius/c6d1020aca54c47f71aa9f2a19a55ffe > > > > > > -- > > Julien Cigar > > Belgian Biodiversity Platform (http://www.biodiversity.be) > > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 > > No trees were killed in the creation of this message. > > However, many electrons were terribly inconvenienced. > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" -- Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced.