From owner-freebsd-questions@freebsd.org Wed Aug 30 21:40:04 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 954FBE0AC1D for ; Wed, 30 Aug 2017 21:40:04 +0000 (UTC) (envelope-from freebsd-doc@fjl.co.uk) Received: from bs1.fjl.org.uk (bs1.fjl.org.uk [84.45.41.196]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "bs1.fjl.org.uk", Issuer "bs1.fjl.org.uk" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4A28E8132B for ; Wed, 30 Aug 2017 21:40:03 +0000 (UTC) (envelope-from freebsd-doc@fjl.co.uk) Received: from [192.168.1.35] (host86-191-18-76.range86-191.btcentralplus.com [86.191.18.76]) (authenticated bits=0) by bs1.fjl.org.uk (8.14.4/8.14.4) with ESMTP id v7ULe1Lh081052 for ; Wed, 30 Aug 2017 22:40:01 +0100 (BST) (envelope-from freebsd-doc@fjl.co.uk) To: freebsd-questions@freebsd.org From: Frank Leonhardt Subject: Fun with HAST and inter-host connections Message-ID: <6cc75798-b7f2-b794-faec-8807616fd7f4@fjl.co.uk> Date: Wed, 30 Aug 2017 22:40:03 +0100 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Aug 2017 21:40:04 -0000 Please note - I'm pushing what can be done with commodity hardware for amusement here, and trying to do interesting things with FreeBSD. I *do* want to do it this way. Okay, so the game is this: I've got some identical hosts with disks and I've been working on ways of clustering them for a long time. It's what I do for fun, right? I'm currently playing with failover storage. Not necessarily with HAST; iSCSI is fun and I'm messing with geom stuff in general. But let's stick with HAST as it illustrates the dilemma. When I started this game, 1Gb Ethernet was blistering. Now it's not so hot. How do I "network" the hosts with as much throughput as possible (with IP sockets, preferably)? Options: 1) 10Gb Ethernet is expensive. 10Gb switches even more so, but I can do point-to-point. 2) LAG is more about failover than speed. And anyway, you end up needing a lot of Ethernet ports on each host and it soon gets crazy. 3) Fibrechannel - if I bought a few old (cheap) fiberchannel cards, I'm not sure how I could use them point-to-point. Is this possible and has anyone done it? 4) USB 3.1. 10Gb. PCIe cards cost about $30. Now this sounds fun. IP over USB anyone? Now please don't advise me to get a pair of fabric switches and do the job properly. That's not what this is about. I want to see if it's possible to make a fast(er) storage solution using cheap components. A sort of Redundant Array of Inexpensive NAS. Thoughts anyone? In particular, is the USB 3.1 idea crazy? And is anyone else crazy enough to be trying the same thing? Thanks, Frank.