From owner-freebsd-fs@FreeBSD.ORG Tue Sep 20 19:25:46 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 85DF2106566B for ; Tue, 20 Sep 2011 19:25:46 +0000 (UTC) (envelope-from jusher71@yahoo.com) Received: from nm23.bullet.mail.ne1.yahoo.com (nm23.bullet.mail.ne1.yahoo.com [98.138.90.86]) by mx1.freebsd.org (Postfix) with SMTP id 504438FC1D for ; Tue, 20 Sep 2011 19:25:46 +0000 (UTC) Received: from [98.138.90.55] by nm23.bullet.mail.ne1.yahoo.com with NNFMP; 20 Sep 2011 19:25:45 -0000 Received: from [98.138.89.164] by tm8.bullet.mail.ne1.yahoo.com with NNFMP; 20 Sep 2011 19:25:45 -0000 Received: from [127.0.0.1] by omp1020.mail.ne1.yahoo.com with NNFMP; 20 Sep 2011 19:25:45 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 845336.8903.bm@omp1020.mail.ne1.yahoo.com Received: (qmail 7554 invoked by uid 60001); 20 Sep 2011 19:25:45 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1316546745; bh=nXWEW4O+hjHeIa68zo2/nKjkvSUpNdeGTdRGlbRh71U=; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:MIME-Version:Content-Type; b=5dE7vuj1KVZo8kRELywrnDoBWV4F5MvpY7Oow1MN8bJG9tK8eCiK551MQqPCU4S7B2kXPrA32Y8i3ZlwtvFsrpkIvdX0X841KfELa2vgXQ8u2MN1ktDSVIH8DvllDX/QVGF0m+sMsWBhCyDqBvFECnL4P5puCsYVqkzmyYmR8+4= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:MIME-Version:Content-Type; b=T7jfVqbF7o4NHQ4JyiYALmMedNUlvLdK13gxFP18n8/A1pHjRAX+8MLtFhfPB0YNlA3RXQJis+exzr2zwsWqPJZmrYQDyHdhE4qPeDf8AUKm8Iaw5o1URh6y+UNTk84eZp2hKiYc29wZaFs4f/ww777YRfqmyJV4OLi3Lm+43XQ=; X-YMail-OSG: AiF9w.IVM1neVL33uwCXCtlxmD1XPU6G6F0uMZfWYcMvVzV qz0D5drKn6LZJFdDj2DH4g3nRILxcmBRoX.tDktnHcQizKzKXN1GbZZXAWjL qbSwt.7vwxzcJ6JiJ.UVDJLYClH1uolEFJwB0Tv1o7hJjTw2JGOXPcdoj_aH _Pmp56gCLk2ljITGcEsWU9yvGk98kSotvcxWUjc_OL2tSRzFU7QKNGb2mL3s 9ef7K8PRLtNGxAVZdWQljomIMsPGdGjy95OiFlVPYdXcaMSRCzAp3g.l5hDy NIXk_jjfFTKGHQ3Iu5rDp61K3bYDjEosCabD5naBWNe15LXdW2BNGG8NtQdq Lo8PyX0LKGCS8PwfYbJNi3.xzKzR3E2Y80nlMli_frZ7XCZaDwImMiEFv3_b a5VnzgDocWXl9ZhthnWA4I1f2g0nfdoH.ZXjfyHwN2bt8gi0- Received: from [46.105.26.30] by web121208.mail.ne1.yahoo.com via HTTP; Tue, 20 Sep 2011 12:25:45 PDT X-Mailer: YahooMailClassic/14.0.5 YahooMailWebService/0.8.114.317681 Message-ID: <1316546745.96947.YahooMailClassic@web121208.mail.ne1.yahoo.com> Date: Tue, 20 Sep 2011 12:25:45 -0700 (PDT) From: Jason Usher To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailman-Approved-At: Tue, 20 Sep 2011 21:03:13 +0000 Subject: Re: ZFS obn FreeBSD hardware model for 48 or 96 sata3 paths... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Sep 2011 19:25:46 -0000 Hi Julian, --- On Mon, 9/19/11, Julian Elischer wrote: > jason, you still haven't said what > the reason for all this is.. speed, capacity, both or > some other reason.. > (or if you did, I missed it). I did, but I replied badly and it didn't thread right - sorry. The use case is very simple - nothing interesting at all - just a big giant local fileserver that will get hit by a lot of big, long rsync and sftp jobs, as well as some simple, but intensive, local housekeeping jobs (file culling with 'find', legacy hardlink "snapshots" and other things that could be done in better ways, but won't be). In fact, the only interesting aspect of the whole operation is that there are a few hundred million inodes in use and the average file size is between 150 and 200 KB. So why am I going on about pcie paths and dedicated drive paths, etc. ? No reason - I just thought it was a simple and cheap optimization that would allow me to never worry about a certain class of problems - admittedly, problems I might not ever run into. I'm not going to double the cost of 48 drives to get this, nor am I going to double the cost of 6 adaptor cards to do this, but I *would* be willing to double the cost of a single motherboard to do this. But now I see it's not that practical, and probably doesn't exist. The latest, greatest 32 lane pcie 2.0 motherboards tend to have just four ports, or other such complications. So, if there isn't a better suggestion, I think I will economize a bit and get the Supermicro X8DTH-6F ... 8 core / 192 GB / 7 8x slots ... or the X8DAH+-F ... 8 core / 288 GB / 2 x16, 4 x8, 1 x4 slots. The other questions regarding the ZIL/L2arc and so on have, I think, been answered - many thanks for all of the good suggestions and warnings.