From owner-freebsd-questions@freebsd.org Thu Aug 31 15:34:17 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 27D76E00747 for ; Thu, 31 Aug 2017 15:34:17 +0000 (UTC) (envelope-from julien@perdition.city) Received: from relay-b03.edpnet.be (relay-b03.edpnet.be [212.71.1.220]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "edpnet.email", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CA760814DE for ; Thu, 31 Aug 2017 15:34:15 +0000 (UTC) (envelope-from julien@perdition.city) X-ASG-Debug-ID: 1504192855-0a8818159bc93540001-jLrpzn Received: from mordor.lan (213.211.136.209.static.edpnet.net [213.211.136.209]) by relay-b03.edpnet.be with ESMTP id GyjJVjgCFocmpNE8 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 31 Aug 2017 17:20:57 +0200 (CEST) X-Barracuda-Envelope-From: julien@perdition.city X-Barracuda-Effective-Source-IP: 213.211.136.209.static.edpnet.net[213.211.136.209] X-Barracuda-Apparent-Source-IP: 213.211.136.209 Date: Thu, 31 Aug 2017 17:20:55 +0200 From: Julien Cigar To: Frank Leonhardt Cc: freebsd-questions@freebsd.org Subject: Re: Fun with HAST and inter-host connections Message-ID: <20170831152055.GD27165@mordor.lan> X-ASG-Orig-Subj: Re: Fun with HAST and inter-host connections References: <6cc75798-b7f2-b794-faec-8807616fd7f4@fjl.co.uk> <20170831113330.GA27165@mordor.lan> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="g7w8+K/95kPelPD2" Content-Disposition: inline In-Reply-To: <20170831113330.GA27165@mordor.lan> User-Agent: Mutt/1.8.2 (2017-04-18) X-Barracuda-Connect: 213.211.136.209.static.edpnet.net[213.211.136.209] X-Barracuda-Start-Time: 1504192856 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://212.71.1.220:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at edpnet.be X-Barracuda-Scan-Msg-Size: 3326 X-Barracuda-BRTS-Status: 1 X-Barracuda-Bayes: INNOCENT GLOBAL 0.5000 1.0000 0.0100 X-Barracuda-Spam-Score: 0.01 X-Barracuda-Spam-Status: No, SCORE=0.01 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=6.0 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.42479 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Aug 2017 15:34:17 -0000 --g7w8+K/95kPelPD2 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Aug 31, 2017 at 01:33:30PM +0200, Julien Cigar wrote: > On Wed, Aug 30, 2017 at 10:40:03PM +0100, Frank Leonhardt wrote: > > Please note - I'm pushing what can be done with commodity hardware for= =20 > > amusement here, and trying to do interesting things with FreeBSD. I *do= *=20 > > want to do it this way. > >=20 > > Okay, so the game is this: I've got some identical hosts with disks and= =20 > > I've been working on ways of clustering them for a long time. It's what= =20 > > I do for fun, right? > >=20 > > I'm currently playing with failover storage. Not necessarily with HAST;= =20 > > iSCSI is fun and I'm messing with geom stuff in general. But let's stic= k=20 > > with HAST as it illustrates the dilemma. > >=20 > > When I started this game, 1Gb Ethernet was blistering. Now it's not so= =20 > > hot. How do I "network" the hosts with as much throughput as possible= =20 > > (with IP sockets, preferably)? Options: > >=20 > > 1) 10Gb Ethernet is expensive. 10Gb switches even more so, but I can do= =20 > > point-to-point. > >=20 > > 2) LAG is more about failover than speed. And anyway, you end up needin= g=20 > > a lot of Ethernet ports on each host and it soon gets crazy. > >=20 > > 3) Fibrechannel - if I bought a few old (cheap) fiberchannel cards, I'm= =20 > > not sure how I could use them point-to-point. Is this possible and has= =20 > > anyone done it? > >=20 > > 4) USB 3.1. 10Gb. PCIe cards cost about $30. Now this sounds fun. IP=20 > > over USB anyone? > >=20 > > Now please don't advise me to get a pair of fabric switches and do the= =20 > > job properly. That's not what this is about. I want to see if it's=20 > > possible to make a fast(er) storage solution using cheap components. A= =20 > > sort of Redundant Array of Inexpensive NAS. > >=20 > > Thoughts anyone? In particular, is the USB 3.1 idea crazy? And is anyon= e=20 > > else crazy enough to be trying the same thing? >=20 > I've set up a low-cost redundant storage here at work with ZFS + iSCSI, > basically a zpool mirror over 2 local disks and 2 iSCSI disks. Failover > is done through CARP and devd. It works well (except when Jumbo Frames > are turned on) for almost a year now. Just for extra security, and also > because some people (smarter than me in this area) advised me against=20 > this approach, I've setup a third node with ZFS replication (zrep). > Basically if you can afford to lose "some" data, I would suggest to go > with ZFS replication. (of course by "ZFS replication" I mean snapshot and send/receive) >=20 > >=20 > > Thanks, Frank. > >=20 > >=20 > > _______________________________________________ > > freebsd-questions@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd= =2Eorg" >=20 > --=20 > Julien Cigar > Belgian Biodiversity Platform (http://www.biodiversity.be) > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 > No trees were killed in the creation of this message. > However, many electrons were terribly inconvenienced. --=20 Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. --g7w8+K/95kPelPD2 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEE7vn2l0to0nV7EWolsrs3EKIEI8AFAlmoKVQACgkQsrs3EKIE I8A4Xg//SLFQq7htthZXYrSOCF8I8dDhUfwTKBgHssUA2Oj5qzUf4L0MTu97gHmv zljkG3eOFKDNxEJ4MWCV8xwl+rR6OouRAnVhB8Y/KM26iwhqdLBGs5EZisH3k0r/ e9Y5mQTRKhYCv4iLUt6HBcQ9j/v063gA9w6xlvYJN6lvig400EvcFcl0tcnvrMWZ 0keBwXvJ0YKZXYMYGJuwDC/au/XTMnbQ2sr+3vUuNx/1MCkkX8QDmvXMW4ps2JBR NUeT+lImUO9qqGwHRd3hFnHub55eUD1Y0sMMC67icESo1mT2//r/szUGLu8pXPGg XMrgjtp7co8lbg84QW83yqTnDJsR3FR6f44/B5Y84lrb7XLcR5AF1uHVLXVggqgS b8FgWS5MFHutSHxBHyLerQSWfIu9g2Eao+C75Qhu3msN7BfObzztiacBCh+7qmqm n4EmhYnO6z/oe0ShjqusDe64HdpD5UjCgGg/AfDtCNxZRjjb4mDIFrJYLXSftKjV /3Y5L6e8QFU/y/3JSzk2wMYyT8frBt9LjJ9bHRKTN9C8Vnhb+Itf9yEytil9GG7U C8V8qMTclkM41FY9l//U7JDo5ZoXyPNAK+HGZuu/S8XyWW/4wRNcwpEAqje0C2c3 LN9ghpDjs4+23AX9chQXbsTTl7Uknus02b37jo3aYS8mRZ0J7E8= =VCEG -----END PGP SIGNATURE----- --g7w8+K/95kPelPD2--