From owner-freebsd-fs@FreeBSD.ORG Thu Sep 23 02:57:06 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 74B7F1065670 for ; Thu, 23 Sep 2010 02:57:06 +0000 (UTC) (envelope-from fbsd@dannysplace.net) Received: from mailgw.dannysplace.net (mailgw.dannysplace.net [204.109.56.184]) by mx1.freebsd.org (Postfix) with ESMTP id 3FC878FC16 for ; Thu, 23 Sep 2010 02:57:05 +0000 (UTC) Received: from [203.206.171.212] (helo=[192.168.10.10]) by mailgw.dannysplace.net with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.72 (FreeBSD)) (envelope-from ) id 1Oyc00-0000Xe-Do for freebsd-fs@freebsd.org; Thu, 23 Sep 2010 12:57:30 +1000 Message-ID: <4C9AC1F6.90305@dannysplace.net> Date: Thu, 23 Sep 2010 12:56:54 +1000 From: Danny Carroll User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.9) Gecko/20100915 Lightning/1.0b2 Thunderbird/3.1.4 MIME-Version: 1.0 To: freebsd-fs Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Authenticated-User: danny X-Authenticator: plain X-Exim-Version: 4.72 (build at 12-Jul-2010 18:31:29) X-Date: 2010-09-23 12:57:28 X-Connected-IP: 203.206.171.212:58285 X-Message-Linecount: 106 X-Body-Linecount: 94 X-Message-Size: 4314 X-Body-Size: 3789 X-Received-Count: 1 X-Recipient-Count: 1 X-Local-Recipient-Count: 1 X-Local-Recipient-Defer-Count: 0 X-Local-Recipient-Fail-Count: 0 X-SA-Exim-Connect-IP: 203.206.171.212 X-SA-Exim-Mail-From: fbsd@dannysplace.net X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on damka.dannysplace.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.3.1 X-SA-Exim-Version: 4.2 X-SA-Exim-Scanned: Yes (on mailgw.dannysplace.net) Subject: Devices disappeared after drive shuffle - or - how to recover and mount a slice with UFS partitions. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: fbsd@dannysplace.net List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Sep 2010 02:57:06 -0000 I had a strange problem this week and although it's already solved, I am still not sure what went wrong. This is a long email so bear with me ;-) There is a nice recovery procedure at the end and a few questions along the way for someone who knows more than I. I have 15 drives attached to my home fileserver. 12 are connected via an areca controller (configured as pass-through). Data drives. 3 are connected directly to the motherboard SATA controllers. OS Drives. The motherboard is a Supermicro X7SBE with 8Gb RAM. The 3 OS drives are 80Gb seagate sata drives and up till this week one of them was running 7.3-Stable. I wanted to reinstall so I decided to pull the OS drive, re-install 8.1 on one of the other 2 drives and use PJD's procedure for ZFS mirrored root. (http://blogs.freebsdish.org/pjd/2010/08/06/from-sysinstall-to-zfs-only-configuration/) I installed on ad6 and mirrored to ad4. Ad8 was the old 7.3 drive which was pulled from the server to make sure I did not accidentally overwrite it. Install went fine and I was able to plug the old drive back in after the upgrade and see the old 7.3 data (it was again ad8). My problem arose when I decided to test if the ARECA passthrough drives are really 100% passed through. The 12 drives are configured in a zfs raidz array. I use GPT partitions with labels that are used. I powered down, pulled one of the drives (da0) and plugged it into one of the SATA ports. ZFS came up perfectly and a scrub revealed no errors. This tells me that a drive connected to an Areca controller and configured as a pass-through is no different to a drive connected to a standard SATA port. This is good to know if my areca card ever fails. (Areca Pass-through differs from JBOD in that JBOD ignores the Areca's cache (and battery backup) and passthrough does indeed use the cache.) The da0 drive is now ad4, the 2 OS drives are now ad6 and ad8 and the old OS drive (which I plugged back in after the upgrade) is now ad10. All looked good but the 2 slices on the ad10 drive did not show up in /dev/. My data was on the second slice with 4 UFS partitions on that slice. Running fdisk on ad10 showed the slices were there, but just not in /dev/. I had backups but I wanted the data anyway so I used testdisk to create an image of the second slice. (dd would have worked just as well I guess). I then mounted the raw slice via a md0 device. As expected md0a, md0d, md0e and md0f showed up in /dev/ Great... I had my data back! My only real question is. Why did the devices fail to be created in /dev from the original disk? And is there a way to force devfs to rescan a device and create those sub-devices? And now, a procedure for anyone who finds themselves having to recover a slice containing UFS partitions. I kinda guessed my way though this and was surprised that it was so easy. Use sysutils/testdisk to dump the partition to a flat file. mdconfig -a -t vnode -f /data/image.dd -u 0 mount /dev/md0a /oldserv mount /dev/md0f /oldserv/usr mount /dev/md0d /oldserv/var mount /dev/md0e /oldserv/tmp done! Interestingly enough, I tried this with a zvol as well. Use sysutils/testdisk to dump the partition to a flat file. Called image.dd (it was about 70G) Create the zvol: zfs create -V 70G data/oldserv (make sure size is the same) dd if=/data/image.dd of=/dev/zvol/data/oldserv bs=1M Unfortunately it did not work out. The slices were created but I had trouble mounting them. I am not sure why. I learned a little about UFS data recovery, a little about md0 and a little about zvols today. I assume if I had have done the same with the whole disk device instead of the slice, I would have seen md0p2a md0p2d etc in /dev/... -D