From owner-freebsd-fs@FreeBSD.ORG Sat Oct 30 18:23:49 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 16345106564A for ; Sat, 30 Oct 2010 18:23:49 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta13.emeryville.ca.mail.comcast.net (qmta13.emeryville.ca.mail.comcast.net [76.96.27.243]) by mx1.freebsd.org (Postfix) with ESMTP id EC7778FC14 for ; Sat, 30 Oct 2010 18:23:48 +0000 (UTC) Received: from omta17.emeryville.ca.mail.comcast.net ([76.96.30.73]) by qmta13.emeryville.ca.mail.comcast.net with comcast id R5ZK1f0031afHeLAD6Pof8; Sat, 30 Oct 2010 18:23:48 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta17.emeryville.ca.mail.comcast.net with comcast id R6Pm1f00G3LrwQ28d6Pn6X; Sat, 30 Oct 2010 18:23:47 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id CA45A9B422; Sat, 30 Oct 2010 11:23:46 -0700 (PDT) Date: Sat, 30 Oct 2010 11:23:46 -0700 From: Jeremy Chadwick To: Peter =?iso-8859-1?Q?Ankerst=E5l?= Message-ID: <20101030182346.GA61519@icarus.home.lan> References: <86693036-9351-4303-BADA-C99F7A4C375C@pean.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <86693036-9351-4303-BADA-C99F7A4C375C@pean.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: Raid + zfs performace. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Oct 2010 18:23:49 -0000 On Sat, Oct 30, 2010 at 07:56:56PM +0200, Peter Ankerstål wrote: > > > On 30 okt 2010, at 19.39, Sean wrote: > > >> I have a question about raid and zfs. I Have a hardware-raid running. > >> A mirror thats the only storage in my zfs pool. Im going to > >> add another mirror to the machine and my question is, what is the best option performace-wise? > > > > The best performance option is to get rid of the hardware-raid, and > > present each disc to ZFS in a JBOD fashion. > > Ok. RIght now thats not an option. I have a da0 device thats a hardware-raid mirror. and it is currently > the only device in the only pool on the machine. > > > > >> Is it to add the other mirror to the same pool or create another separate pool for that mirror? > >> Btw. Today my disk are quite saturated r/w wise. > > > > RAID functionality only exists within the context of a single pool. > > You don't create a new pool and then try to mirror the two pools. You > > add the storage to an existing pool, unless you have a reason to start > > a new pool. When I already have a mirror, I like to add new mirror > > sets. It's the equivalent of a RAID 10 scenario. > > Yes I know. > I thought maybe because the existing pool is kind of r/w saturated it should be better > to create a new independent pool for the new drives. In that way the heavy activity > would not "spread" to the new drives. > > Now you presented me with a third option. So you think I should skip to create > a new hardware-raid mirror and instead use two single drives and add these as > a mirror to the existing pool? How will zfs handle howswap of these drives? > I've seen a few crashes due to ata-detach in other systems. As stated previously: ideally you should turn off all RAID capabilities on the controller itself and just use JBOD + let FreeBSD deal with the RAID'ing (in this case, ZFS). I realise you can't do this for the current hardware RAID'd "da0" mirror. I also understand the reason you might have done this. I too would trust a hardware RAID controller to handle booting from a mirror more than the FreeBSD bootstraps. (I choose not to deal with things like gptzfsboot, zfsloader, etc. etc. -- these should all just become part of the standard bootstraps and stop requiring people to jump through hoops to achieve such capabilities) As for hot-swapping of drives: there are well-established procedures for this sort of thing. I blogged about it[1] not too long ago, with real-world server hardware. I think it will address your concerns, since I spent the time to document *everything*, including all output. What confuses me about your statement is that you're talking about "ata-detach" when your drives are appearing as daX, which implies use of CAM. Are you talking about hot-swapping failing on a completely different system which was using ata(4) without AHCI? If so, I'm not surprised. You need AHCI to achieve this. Finally, you should be aware that there are two AHCI "methods" implemented in FreeBSD: ataahci.ko and ahci.ko. ataahci is the older of the two and uses ata(4). ahci.ko is the newer of the two and is significantly better, since it makes use of CAM, and thus provides performance improvements and handles hot-swapping a lot better (IMHO). ahci.ko "trumps" ataahci.ko (meaning if your kernel includes ataahci and you do "load ahci" from loader or loader.conf, ahci.ko will be used instead). Be aware that ahci.ko disks appear as adaX (this is not a typo). [1]: http://koitsu.wordpress.com/2010/07/22/freebsd-and-zfs-hot-swapping-sata-disks-with-ahci/ -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |