From owner-freebsd-fs@freebsd.org Mon Apr 26 15:19:29 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 256845E4433 for ; Mon, 26 Apr 2021 15:19:29 +0000 (UTC) (envelope-from joris.dedieu@gmail.com) Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FTT8J3cFzz3lLs for ; Mon, 26 Apr 2021 15:19:28 +0000 (UTC) (envelope-from joris.dedieu@gmail.com) Received: by mail-ej1-x62b.google.com with SMTP id r20so35285094ejo.11 for ; Mon, 26 Apr 2021 08:19:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=ULDAdnfglKl2NDKirGhaHm+fMnIA2HI8ks0I/B1j+xI=; b=PPUVu5AIG/yHO3Ppwb35IKn+ECoXnnl9mqrz9ow3qL2iScm6UlEzxtBl4ZwdAEDlWO 3Ej38Fn+1aJLGR2JqlN+XvsFIeWhxsMwR5kXjGV6nHCnKQ+4sj4zQfpuZ0nP+d0XH2dI FQvH3F+FZNQrPkATpaYURbGWtMMB8af0rP95OatJ7RvyVTdjQGe9/EVGs1Vq2uSLOLU4 YBQhIBlSkiB8LblzKfGG26RXQy/LjkUniIfmPPxlsr2p0LO4FcGmTAEE8OBGzm1LvkO4 V7Dy2mpy38rMkbmze+tNXAE1i42pV+0/rWOnAHF+5mRNc+ONu5Icx+ZAduAHujYB2IvV JZCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=ULDAdnfglKl2NDKirGhaHm+fMnIA2HI8ks0I/B1j+xI=; b=X9D/+fo7WvzR4jeWaSheHWuDDH4xFFfOE+YiMFVg0Eyod3YePyyyDRG7Znkg6TbMCp NvGqHPLaKX6UHplaUoURPOIA/m3Q/EbTlxE5h/XD5ipMqWv7ZYMza/QAELojZ2ZI7uYb 8floRscKjGVgndNhWIla3wWn0PQehaRGiUJGvWBX7rzpZM5KuoAbhJGKMY5bn0lJFGks UdChcpXxUHMSOpaN/QLFH0LnBDYWetTcpQqY9JAvW2eQ0iynL/7bNu93R2uXSJ0KoBnq GLI73SM5wdRsYwzt5U8iMnauQZE8You4GaRVXC+L2o9kMa3TofqByjyAdTwm/V8SAEmR gFFg== X-Gm-Message-State: AOAM533uADfYJ7PUq/opkc8sAPtTyRxdFB3N35APsohBCqfKr6FK0FdR /gKLaB0sAPbKa64IWBhwxevOXQrFf19Ox2GwrzKflR+yOjs= X-Google-Smtp-Source: ABdhPJwJCf6bmvRIMcphhu/W61S+Dn8U9kiRLgEk2bzCV4vKVnaaRRbom1EE2YrPZJK7YAI2d5T7Q8Yps21l6y8jAHk= X-Received: by 2002:a17:906:5fce:: with SMTP id k14mr19461179ejv.9.1619450365673; Mon, 26 Apr 2021 08:19:25 -0700 (PDT) MIME-Version: 1.0 References: <20210426122440.xirux6bwdztwsm5c@x1> In-Reply-To: <20210426122440.xirux6bwdztwsm5c@x1> From: joris dedieu Date: Mon, 26 Apr 2021 17:19:14 +0200 Message-ID: Subject: Re: iSCSI SAN To: freebsd-fs@freebsd.org X-Rspamd-Queue-Id: 4FTT8J3cFzz3lLs X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=PPUVu5AI; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jorisdedieu@gmail.com designates 2a00:1450:4864:20::62b as permitted sender) smtp.mailfrom=jorisdedieu@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::62b:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::62b:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::62b:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Apr 2021 15:19:29 -0000 Hi, Le lun. 26 avr. 2021 =C3=A0 14:24, Julien Cigar a = =C3=A9crit : > Hello, > > I'm wondering if something has already been written to implement a > fully redundant and highly available FreeBSD ZFS based iSCSI SAN? > > I've setup some FreeBSD iSCSI SAN-like in the past (for small > structures) and it has always worked well. However upgrades have always > been painfull and, although there is ZFS, redundant power supplies, > redundant switches with multipath, it's still a SPOF if a non-redundant > component dies, like the motherboard. It's not like an HPE MSA-like > system where everything is redundant out of the box. > >From my experience HA mechanisms are an infinite source of pain. Don't forget the power of simplicity. What uptime do you get with your simple setup ? When was the last time you see a decent mainboard crash ? Also don't forget that FreeBSD has glusterfs and ceph. If you want to do something similar to proprietary chassis, you should have to look at SAS HBA and JBOD chassis, ATAoE chassis (if it still exists) or stuff like that to attach your disks to your two mainboard. Still (OMG) dealing with zfs (import -f) on failover, cluster STONITH and other voodoo. You will have fully redundant design (see https://i.stack.imgur.com/ijjpk.png ) Cheers Joris > So the idea came to me for the iSCSI target to setup 2 physical servers > with a bunch of disks, create some raidzx on them and export one ZFS > volume per initiator on each target, a bit like on (1) > > I've tried to setup that in a small "lab", with some jails, gmultipath, > two switches, and several VLANs. Unfortunately no 10 gbits to test, but > 3x1Gbits LAGG with LACP. > > The downside of this setup is that "half" of the storage is (temporarily) > lost when a target reboots (freebsd-update, upgrades, etc), which > de-facto disqualifies gmirror + UFS on the initiator side as a full > resync of required and takes ages. With ZFS you don't have this problem > as only the delta is resync. For the few tests I did it seems to work, > but I'm wondering: zfs over zvol .. is it sane? does it makes sense? > could I disable checksum on the initiator side to speed up things? do > you see any race condition or ... with this setup (sync, etc)? > > What do you think? > > Thanks! > Julien > > (1) https://gist.github.com/silenius/c6d1020aca54c47f71aa9f2a19a55ffe > > > -- > Julien Cigar > Belgian Biodiversity Platform (http://www.biodiversity.be) > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 > No trees were killed in the creation of this message. > However, many electrons were terribly inconvenienced. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >