From owner-freebsd-questions@FreeBSD.ORG Thu May 31 18:39:09 2012 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC124106566B for ; Thu, 31 May 2012 18:39:09 +0000 (UTC) (envelope-from dg@pki2.com) Received: from btw.pki2.com (btw.pki2.com [IPv6:2001:470:a:6fd::2]) by mx1.freebsd.org (Postfix) with ESMTP id 20EA28FC12 for ; Thu, 31 May 2012 18:39:08 +0000 (UTC) Received: from btw.pki2.com (btw.pki2.com [192.168.23.1]) by btw.pki2.com (8.14.5/8.14.5) with ESMTP id q4VId4c8051572 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Thu, 31 May 2012 11:39:04 -0700 (PDT) (envelope-from dg@pki2.com) Date: Thu, 31 May 2012 11:39:04 -0700 (PDT) From: Dennis Glatting X-X-Sender: dennisg@btw.pki2.com To: Kaya Saman In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="3463506553-125945988-1338489544=:48219" X-yoursite-MailScanner-Information: Dennis Glatting X-yoursite-MailScanner-ID: q4VId4c8051572 X-yoursite-MailScanner: Found to be clean X-MailScanner-From: dg@pki2.com Cc: freebsd-questions@freebsd.org, Oscar Hodgson Subject: Re: Anyone using freebsd ZFS for large storage servers? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2012 18:39:09 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --3463506553-125945988-1338489544=:48219 Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8BIT On Thu, 31 May 2012, Kaya Saman wrote: > On Thu, May 31, 2012 at 5:05 PM, Oscar Hodgson wrote: >> That helps.  Thank you. >> >> This is an academic departmental instructional / research environment. >>  We had a great relationship with Sun, they provided great >> opportunities to put Solaris in front of students.  Oracle, not so >> much, and the Oracle single-tier support model simply isn't affordable >> for this "business" (there's no ROI at the departmental level ). >> Solaris is not a viable option. > > We found Oracle to be the cheapest out of all the solutions we looked > at: Netapp, MSI, et el..... > >> >> FreeBSD looks like the next best available option at the moment, >> particularly considering the use of the storage heads as compute >> machines.  OpenIndiana shows promise.  Nexenta has a great product, >> but the user community expects more flexibility in software options. > > FreeBSD is better then Linux in my opinion though lacking some > software and multimedia functionality that Linux has and not for the > Desktop as it's not as "bleeding edge" as say Fedora 16, however, if > FreeBSD offered Gnome3 and supported my wireless NIC I'd be all over > it like a "bad rash" :-) > >> >> Is there anything like a list of "supported" (known good) SAS HBA's? > > LSI HBA's are really good! > > For my DIY solution at home I used a SuperMicro system board with > non-RAID LSI HBA....... > Similarly: mc => Tyan S8812WGM3NR iirc => Supermicro H8DGi bd3 => Soon another Supermicro H8DGi Others are consumer boards from Gigabyte (preferred). I also have a small collection of Supermicro AOC-USAS2-L8i boards. Generally, I have had no trouble but ESXi 5.0 hated them. For work I looked at two Supermicro 848A chassis with a H8QGL board and 20 3TB disks for two different projects, but they lie in limbo. > It is a similar solution that we will use for our test NAS at work > though we already have a Dell R700 series server. For this setup > however I will need to use an LSI HBA with both internal and external > Mini-SAS ports. > > Instead of Promise we will use NetStor JBOD solutions as they work > with 6Gbps drives and overall give better performance. > >> >> Oscar > > Regards, > > > Kaya > >> >> On Thu, May 31, 2012 at 11:38 AM, Kaya Saman wrote: >>> If this is any consellation I run a 36TB cluster using a self built >>> server with a Promise DAS (VessJBOD 1840) using ZFS at home! to >>> support my OpenSource projects and personal files. >>> >>> As for OS take your pick: NexentaStor, FreeBSD, Solaris 11 >>> >>> >>> All capable, of course Solaris has latest version of ZFS but still..... >>> >>> >>> At work we're looking into getting a StorEdge appliance wich will >>> handle up to 140+ TB. >>> >>> >>> I am also in charge of redesigning one of our virtual SAN's to a >>> FreeBSD ZFS storage system which will run.... well how many JBOD's can >>> you fit on the system?? Probably round ~100TB or so..... >>> >>> >>> Regards, >>> >>> >>> Kaya >>> >>> >>> On Thu, May 31, 2012 at 4:32 PM, Oscar Hodgson wrote: >>>> The subject is pretty much the question.  Perhaps there's a better >>>> place to be asking this question ... >>>> >>>> We have (very briefly) discussed the possibility of using FreeBSD >>>> pizza boxes as a storage heads direct attached to external JBOD arrays >>>> with ZFS.  In perusing the list, I haven't stumbled across indications >>>> of people actually doing this.  External JBODs would be running 24 to >>>> 48TB each, roughly.  There would be a couple of units.  The pizza >>>> boxes would be used for computational tasks, and nominally would have >>>> 8 cores and 96G+ RAM. >>>> >>>> Obvious questions are hardware compatibility and stability.  I've set >>>> up small FreeBSD 9 machines with ZFS roots and simple mirrors for >>>> other tasks here, and those have been successful so far. >>>> >>>> Observations would be appreciated. >>>> >>>> Oscar. >>>> _______________________________________________ >>>> freebsd-questions@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions >>>> To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" > > --3463506553-125945988-1338489544=:48219--