From owner-freebsd-questions@FreeBSD.ORG Fri Nov 7 10:06:51 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 59E711065690 for ; Fri, 7 Nov 2008 10:06:51 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA09.westchester.pa.mail.comcast.net (qmta09.westchester.pa.mail.comcast.net [76.96.62.96]) by mx1.freebsd.org (Postfix) with ESMTP id F24748FC14 for ; Fri, 7 Nov 2008 10:06:50 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from OMTA12.westchester.pa.mail.comcast.net ([76.96.62.44]) by QMTA09.westchester.pa.mail.comcast.net with comcast id cA2i1a0030xGWP859A6qvt; Fri, 07 Nov 2008 10:06:50 +0000 Received: from koitsu.dyndns.org ([69.181.141.110]) by OMTA12.westchester.pa.mail.comcast.net with comcast id cA6n1a0082P6wsM3YA6obU; Fri, 07 Nov 2008 10:06:49 +0000 X-Authority-Analysis: v=1.0 c=1 a=gDos0kX76BEA:10 a=tWmolMFHfhMA:10 a=6I5d2MoRAAAA:8 a=QycZ5dHgAAAA:8 a=x4uUgcpHUNeuCcu5CB8A:9 a=X9wY_EeQV1QUlyEeSrAA:7 a=nhOcXDZk1xYVsGW6uamPzbDw5GAA:4 a=EoioJ0NPDVgA:10 a=LY0hPdMaydYA:10 Received: by icarus.home.lan (Postfix, from userid 1000) id AD32B5C19; Fri, 7 Nov 2008 02:06:47 -0800 (PST) Date: Fri, 7 Nov 2008 02:06:47 -0800 From: Jeremy Chadwick To: Pieter Donche Message-ID: <20081107100647.GA2064@icarus.home.lan> References: <20081106215647.A28070@wojtek.tensor.gdynia.pl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.18 (2008-05-17) Cc: Wojciech Puchar , freebsd-questions@freebsd.org Subject: Re: Hardware Raid + hot-replace failed disk X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Nov 2008 10:06:51 -0000 On Fri, Nov 07, 2008 at 10:35:39AM +0100, Pieter Donche wrote: > On Thu, 6 Nov 2008, Wojciech Puchar wrote: > >>> Suppose you have a system with multiple disks managed by a >>> hardware RAID controller in a RAID5 of RAID6 configuration, >>> To FreeBSD it will look like e.g. a single large drive. >> >> what is "RAID5 of RAID6"??? > RAID5 or RAID6 (sorry, typing error) > >>> If you want to extend your disk space by plugging in an extra >>> disk, the hardware RAID controller will probably detect it and >>> add it in his management, but will it be seen by FreeBSD? >> FreeBSD will see larger drive. > > With what command can you see that FreeBSD had 'seen' it ? The answer is: "it depends". The below applies to SATA, SAS, and SCSI only; you cannot hot-swap PATA disks. If you have a hot-swap enclosure or a hot-swap backplane, and are using a hardware RAID controller (and I do mean *real* hardware RAID, not BIOS-level RAID like Intel MatrixRAID or Adaptec HostRAID), then the FreeBSD controller driver should report the disk falling off the bus (if a disk is removed), or a disk appearing on the bus (if a disk is added). If the driver does not handle this natively, you will have to rely on command-line utilities from the RAID card vendor to manage this. If you have a hot-swap enclosure or a hot-swap backplane, and are using software/OS-based RAID (such as gvinum, ccd, or ZFS), then it depends on the underlying type of disk you're using. With SATA disks, you rely on the FreeBSD ata(4) layer. You are at the whim of the ata(4) layer and its support for your motherboard chipset, assuming that's what you're using (there are exceptions; see below). Removal of a SATA disk should show the disk falling off the bus, and you will need to perform "atacontrol detach " to ensure the kernel knows the disk has been removed (this is not done automatically, despite what you see on the console; I recommend you do the "detach" prior to disk removal). Addition of a SATA disk will require you to perform "atacontrol attach ", and hopefully you will see the disk make and model show up moments later. With SCSI or SAS disks, you rely on the FreeBSD da(4) layer, backed by the FreeBSD CAM(4) layer. This layer is proven reliable, and even some SATA RAID controllers use it (such as Areca controllers; yes, they're SATA disks on a hardware RAID controller, but the FreeBSD driver for the Areca card uses da(4) and CAM(4)). Removal of a SCSI disk should show the disk falling off the bus. You can use "camcontrol" to examine the state of things; you may need to use start/stop (it's been a while since I've used camcontrol). Addition of a SCSI disk might require "camcontrol rescan"; again, it's been a while since I've used camcontrol. In general, there is no easy way to describe every single scenario under the sun. It greatly depends upon what hardware you're using, and what kind of disk you're using. If you choose to use a hardware RAID card, the card user manual should describe *exactly* how to accomplish additions and removals. Chances are you're talking about generic SATA disks hooked up to your generic motherboard. You should be aware that FreeBSD is somewhat "flaky" in this regard. I've recently written about a disk swap gone bad (while using a Promise TX4310 controller), which should give you some idea of the chaos that can happen as a result of shoddy driver support: http://wiki.freebsd.org/JeremyChadwick/ZFS_disk_upgrade_gone_bad This article is followed-up by a fully-working example when using an Intel ICH-based board with Intel AHCI enabled (meaning, everything worked flawlessly and exactly how it should've): http://wiki.freebsd.org/JeremyChadwick/ZFS_disk_upgrade_gone_bad_part_2 I'm still in the process of writing the details that make up Part 2. > Or is the the bsdlabel command? bsdlabel(8) is what creates filesystems. To format filesystems, you use newfs(8). > Is bsdlabel a partition management program (such as GParted, Partition > Magic)? No, that's fdisk(8). FreeBSD calls these "slices", not "partitions", but they're the same thing. If you want to "keep it simple", I recommend you use sade(8), which is the text-based interface for partitioning and filesystem creation that you see when you install FreeBSD. If you don't have the "sade" command, just run "sysinstall" and choose post-configuration. > Is there any document (besides the manual pages bsdlabel, growfs, ..) > that describes step-by-step what to do to grow an existing file system > of to add a new file system on newly added disk space ? What everyone else is telling you is sending you on a wild goose chase. I'm sitting here imagining you clicking your mouse at 6000 clicks per second, eyeballs rolling around, sweating profusely. :-) I wish FreeBSD mailing list people wouldn't do this to new folks, because all it's doing is confusing you. The simple answer is this: on FreeBSD, there is not a reliable way to grow an existing filesystem without taking the machine down, bringing it into single-user, or rebooting it. In fact, I would go so far to say there is not a reliable way on FreeBSD to grow a filesystem at all; you are generally forced to back up all of your data, then recreate the filesystem. ZFS doesn't make this any easier, by the way -- you cannot "grow" a zpool. You can add disks in real-time (no rebooting/single-user) to a separate zpool (think: separate filesystem), but you cannot add disks to an existing zpool. Yes, I am aware Linux has LVM, which provides all of this capability for you, without having to reboot. If you're really wanting something like a end-user NAS, which allows you to remove/add disks as you please, and the filesystem shrinks/grows with it, then you need to be looking at a product from Data Robotics called Drobo. Note that Drobo does not work under FreeBSD, as far as I know (it probably shows up as a umass/da device, but if you remove/add disks, I'm absolutely certain the filesystem does not grow. Linux Drobo users have been dealing with that problem for quite some time too, and Data Robotics is *just now* getting around to addressing it) Hope this helps. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |