Date: Fri, 16 Nov 2007 02:01:20 -0700 From: Edward Sutton <mirror176@hotmail.com> To: Ulf Lilleengen <lulf@stud.ntnu.no> Cc: freebsd-geom@freebsd.org Subject: RE: gvinum raid5 cannot find file system superblock after removing a disk Message-ID: <BAY110-W250A3B0690AFE0C9101253E6830@phx.gbl> In-Reply-To: <20071115224851.GB30409@stud.ntnu.no> References: <BAY110-W41B96F9B17F8E787B23965E6850@phx.gbl> <20071115224851.GB30409@stud.ntnu.no>
next in thread | previous in thread | raw e-mail | index | archive | help
Ulf Lilleengen wrote: >Your description was a bit hard to understand, so please post a more detai= led >description of what you did, because all I got out from this was: =20 >1) You have a RAID-5 volume, and ad8 fails. >2) You dd'ed the first 64 mb of ad8...=20 >3) You tried to repartition ad8... and then what? did you try rebuildparit= y? =20 >Could you also please post your gvinum configuration (gvinum printconfig) = and >listing (gvinum list)? Hopefully word wrap does not kill the output of the configuration (at the= end of this email). On a Promise TX4 SATA150 , I have 3 400GB drives which= i partitioned (sliced actually if I recall my BSD titles correctly) in hal= f. Seemed appropriate because I do not know of any utilities that let me ma= nipulate the layout like I can with windows and linux pieces. If I do not l= ike my layout, I can always try again on the other half; space is available= to migrate back and forth. On the first half, I setup a bootable system; Swap striped across the 3 di= sks, root mirrored across the 3 disks (yes, 2 copies. I could not think of = a better idea for the leftover space on the 3rd disk than make another copy= ), usr var and tmp are striped with parity across the 3 drives. I have had trouble with system crashes. One appeared to be drive related = and the best I tracked from more recent crashes appeared to be with swap sp= ace; after I use so much, I'd get a crash. I have been using swap space fro= m an unrelated hard drive and have achieved uptimes of>1 month again. Only = 2 of the 3 entries for the root mirroring ever appear to work once somethin= g goes down and I cannot start it because its in use after booting (which i= t sounds will go away with future patches) I was trying to move away from my vinum play to work a bit more with gmir= ror (which I setup a server with for my uncle). Instead of moving over to t= he ad*s2 parts of the disk, I figured I would just use all of one of the th= ree disks. I had a disk that was currently out of sync (ad8), so I figured = I would pull that disk out. I made a backup copy of the first 64MB with dd = because I figured using fdisk only touches the start so I could put back an= y of those changes. I proceeded by rebooting from a FreeBSD copy on another= drive (which does not load geom_vinum by default but does have it availabl= e) and repartitioned the drive to use the entire 400GB together. I did not = format that space yet and after finidng I could not reboot my gvinum config= ured system, I restored the 64MB and found that I still could not boot the = gvinum configured system. The root partition was okay (on 2 of the 3 disks = as expected), but all raid5 disks errored out with the superblock message f= or both mounts and fsck. I played with editing the on disk configuration which can be gathered by = `dd if=3D/dev/ad4s1h skip=3D9 count=3D5` in the case of my configuration. I= 'd use my favorite text editor to change 'sd' lines to contain the plex and= plexoffset and change it from 'down' to 'up' (and reversing that on other = lines when disabling disks) and found that I was able to get other response= s on partitions and even get to where 'some' data could be read. Changing s= kip to seek makes it so I can write the new output back to disk. I had a dd= of more than a count of 5 that I could put back (in hopes of always being = to undo that damage). It was strange that the changes did not always produc= e predictable results. Changing tmp to where I could partially read it afte= r mounting it (read only) was done 1 way, Different changes were required f= or the other raid5s and combining those changes lead to more results as to = the errors I would get on bootup. I found the needed combination to be: changing all the subdisks to a good= state, booting the computer (or loading geom_vinum), mounting the partitio= ns, then unplugging ad8 leads to a disk that I can access as it was before = I tampered with the 'down, but plugged in' ad8 in my attempts to consider m= oving from gvinum. A missing ad8 leads to not being able to mount, and an a= vailable and up ad8 leads to much corruption on the file system. Mounting a= nd then unplugging the disk makes it happy though by my testing of filesyst= em corruption I saw not being present and I can play videos from disk (whic= h can be gigs in size without any sign of any additional corruption). Now that I could read the data, I used a `ccdconfig ccd0 64 none /dev/ad4= s2 /dev/ad6s2` and a newfs across it to copy all but the swap partition by = using a dump/restore. Is it just me, or does everyone else always forget th= at restore restores to the current directory instead of a directory added a= s an ignored paramater? `dump -0 -f - /dev/gvinum/usr|buffer -S 2058K -p 75= |restore -r -f -` was one used command (and took about 13 hours to complete= ). By using ccd0 instead of ad8, I was able to keep ad8 in tact in case I n= eeded additional attempts to access/copy the data. Now that I had a copy of the data, I tested a fsck_ffs across /dev/gvinum= /tmp and all I ended up with was one lost+found directory which took up 410= K when the old one took up 2K, but there was 747MB in the tmp directory, so= a lot was missing. I expected bad things to happen, which is why I try to = copy/backup what I can read 'before' I try to fix a broken file system. I should be able to newfs /dev/gvinum/ partitions and dump/restore to the= m from my ad*s2 partitions and follow that with a rebuildparity of ad8 to s= afely get things back to how they were for my previousely semi-stable but b= ooting system right? Before I destroy/overwrite any more data, would any ot= her information be useful to examine/debug? If ad8 had really failed, could= I still have gotten back that data? Below is the configuration. the disks go in reverse order from the device= nodes numerically because I reversed them while playing with testing vinum= back when I first set it up. disk1 used to be ad4 when I first set up gvin= um. The configuration of `gvinum list` reflects the state of the raid5 part= itions having been set to up (by manually edititng), mounting the partition= s, then unplugging the drive. This is still on the boot that I made the cop= y to the ccd layout with. darkstar# gvinum printconfig # Vinum configuration of darkstar.localhost, saved at Fri Nov 16 00:17:31 2= 007 drive disk3 device /dev/ad4s1h drive disk2 device /dev/ad6s1h drive disk1 device /dev/ad8s1h volume tmp =20 volume var volume usr volume swap volume root plex name tmp.p0 org raid5 8126s vol tmp plex name var.p0 org raid5 8126s vol var plex name usr.p0 org raid5 8126s vol usr plex name swap.p0 org concat vol swap plex name root.p0 org concat vol root plex name root.p1 org concat vol root plex name root.p2 org concat vol root sd name tmp.p0.s2 drive disk3 len 3071628s driveoffset 387628721s plex tmp.= p0 plexoffset 16252s sd name tmp.p0.s1 drive disk2 len 3071628s driveoffset 387628721s plex tmp.= p0 plexoffset 8126s sd name tmp.p0.s0 drive disk1 len 3071628s driveoffset 387628721s plex tmp.= p0 plexoffset 0s sd name var.p0.s2 drive disk3 len 5119380s driveoffset 382508721s plex var.= p0 plexoffset 16252s sd name var.p0.s1 drive disk2 len 5119380s driveoffset 382508721s plex var.= p0 plexoffset 8126s sd name var.p0.s0 drive disk1 len 5119380s driveoffset 382508721s plex var.= p0 plexoffset 0s sd name usr.p0.s2 drive disk3 len 379882374s driveoffset 2621424s plex usr.= p0 plexoffset 16252s sd name usr.p0.s1 drive disk2 len 379882374s driveoffset 2621424s plex usr.= p0 plexoffset 8126s sd name usr.p0.s0 drive disk1 len 379882374s driveoffset 2621424s plex usr.= p0 plexoffset 0s sd name swap.p0.s2 drive disk3 len 1572583s driveoffset 265s plex swap.p0 p= lexoffset 3145166s sd name swap.p0.s1 drive disk2 len 1572583s driveoffset 265s plex swap.p0 p= lexoffset 1572583s sd name swap.p0.s0 drive disk1 len 1572583s driveoffset 265s plex swap.p0 p= lexoffset 0s sd name root.p0.s0 drive disk1 len 1048576s driveoffset 1572848s plex root.= p0 plexoffset 0s sd name root.p1.s0 drive disk2 len 1048576s driveoffset 1572848s plex root.= p1 plexoffset 0s sd name root.p2.s0 drive disk3 len 1048576s driveoffset 1572848s plex root.= p2 plexoffset 0s darkstar# gvinum list 3 drives: D disk3 State: up /dev/ad4s1h A: 2/190771 MB (0%) D disk2 State: up /dev/ad6s1h A: 2/190771 MB (0%) D disk1 State: down /dev/ad8s1h A: 2/190771 MB (0%) 5 volumes: V tmp State: up Plexes: 1 Size: 2999 MB V var State: up Plexes: 1 Size: 4999 MB V usr State: up Plexes: 1 Size: 362 GB V swap State: down Plexes: 1 Size: 2303 MB V root State: up Plexes: 3 Size: 512 MB 7 plexes: P tmp.p0 R5 State: degraded Subdisks: 3 Size: 2999 MB P var.p0 R5 State: degraded Subdisks: 3 Size: 4999 MB P usr.p0 R5 State: degraded Subdisks: 3 Size: 362 GB P swap.p0 C State: down Subdisks: 3 Size: 2303 MB P root.p0 C State: down Subdisks: 1 Size: 512 MB P root.p1 C State: up Subdisks: 1 Size: 512 MB P root.p2 C State: up Subdisks: 1 Size: 512 MB 15 subdisks: S tmp.p0.s2 State: up D: disk3 Size: 1499 MB S tmp.p0.s1 State: up D: disk2 Size: 1499 MB S tmp.p0.s0 State: down D: disk1 Size: 1499 MB S var.p0.s2 State: up D: disk3 Size: 2499 MB S var.p0.s1 State: up D: disk2 Size: 2499 MB S var.p0.s0 State: down D: disk1 Size: 2499 MB S usr.p0.s2 State: up D: disk3 Size: 181 GB S usr.p0.s1 State: up D: disk2 Size: 181 GB S usr.p0.s0 State: down D: disk1 Size: 181 GB S swap.p0.s2 State: up D: disk3 Size: 767 MB S swap.p0.s1 State: up D: disk2 Size: 767 MB S swap.p0.s0 State: down D: disk1 Size: 767 MB S root.p0.s0 State: down D: disk1 Size: 512 MB S root.p1.s0 State: up D: disk2 Size: 512 MB S root.p2.s0 State: up D: disk3 Size: 512 MB --=20 Ulf Lilleengen _________________________________________________________________ Boo!=A0Scare away worms, viruses and so much more! Try Windows Live OneCare= ! http://onecare.live.com/standard/en-us/purchase/trial.aspx?s_cid=3Dwl_hotma= ilnews=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BAY110-W250A3B0690AFE0C9101253E6830>