From owner-freebsd-scsi@FreeBSD.ORG Fri Jan 14 11:12:07 2011 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 71375106566B for ; Fri, 14 Jan 2011 11:12:07 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop1b.sarenet.es (proxypop1b.sarenet.es [194.30.0.104]) by mx1.freebsd.org (Postfix) with ESMTP id 319E28FC19 for ; Fri, 14 Jan 2011 11:12:06 +0000 (UTC) Received: from [172.16.1.55] (ssglan.sare.net [192.148.167.100]) by proxypop1b.sarenet.es (Postfix) with ESMTP id 378165DB9 for ; Fri, 14 Jan 2011 11:55:09 +0100 (CET) From: Borja Marcos Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Fri, 14 Jan 2011 11:55:09 +0100 Message-Id: <522B2387-1BE2-4AF7-B250-B1272D301C59@sarenet.es> To: freebsd-scsi@freebsd.org Mime-Version: 1.0 (Apple Message framework v1082) X-Mailer: Apple Mail (2.1082) Subject: ZFS and "internal" mfi RAID cards, the eternal question X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Jan 2011 11:12:07 -0000 Hello, I've been using FreeBSD with ZFS for some time, with varied degrees of = success. Unfortunately, turns out that Dell, which is the preferred = manufacturer here, doesn't offer an appropiate configuration. They insist on selling those "RAID cards" that of course get in the way = of running ZFS in the right way. With these cards you are left basicly = with two options: either defining a large RAID5 logical volume and use = ZFS as a file system, or define a RAID0 volume for each physical disk. I don't like the RAID5 option because you really lose some of the = interesting ZFS functionality. Also, I'm not sure how robust it would = be. (I've done the following tests with a Dell PERC H700) And the RAID0 volumes "solution" is not valid. The RAID card stays = between the disks and ZFS, and some things simply don't work. For = example, if you remove one of the disks (or it experiences a failure) = the RAID card will cry foul and disable the RAID0 volume on it = completely. Once this happens, there's no way to replace the disk or put = in online again. You must reboot the machine and go through the RAID = configuration, including an "IMPORT FOREIGN CONFIGURATION". It's plain = insane. And there's another option I've been using on one machine recently, with = some success. When I asked about these problems long ago, Scott Long = gave me instructions to patch mfi_cam.c so that I can see the disks as = SAS targets. As it had a "beware, at your own risk" warning attached, I = limited it to a single machine. And, so far so good, although I must = remember to repatch the mfi_cam.c file whenever I update the system. There are indeed some problems using it. Of course I avoided to define = any logical volumes on the RAID card, assuming that it wouldn't touch = them. At least it senses them, and places them in "UNCONFIGURED_GOOD" = state. I haven seen also interference (unexpected sense status) if I = try, for example, to put a disk offline with "camcontrol stop". I = suspect that there might be other problems. But so far I have had no problems with ZFS running in this way. Today = I've been trying a new machine (with a PERC H700) and I've been doing = some silly tests like removing a disk, swapping them (I am using gpt = labels for the disks so that ZFS can recognize them regardless of = controller/backplane position) and I haven't managed to damage the ZFS = pool. Everything worked like a charm. Two problems remain with the mfip approach, though. First, it's risky in = the present state. And I'm not sure if future versions of the mfi = firmware could cause problems by touching disks I'm accessing=20 through mfip. Second, there is some problem with the driver shutdown. If = I try to shutdown or reboot the machine (I mean, good old command-line = shutdown -r or shutdown -p) the driver gets stuck and the machine = requires a manual power-down. Would it be possible to have this option better integrated, so that it = would export disks not being actually used by the RAID subsystem (I = assume that means disks in UNCONFIGURED_GOOD state) and, of course, = making it possible to reboot/halt the system without problems. Maybe there would be a way to just tell the card firmware to shut up and = let it work like a simple and well behaved SAS card?=20 Borja.