From owner-freebsd-geom@FreeBSD.ORG Mon Jul 27 11:06:54 2009 Return-Path: Delivered-To: freebsd-geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 275B61065679 for ; Mon, 27 Jul 2009 11:06:54 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 13E128FC3E for ; Mon, 27 Jul 2009 11:06:54 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n6RB6rqQ018940 for ; Mon, 27 Jul 2009 11:06:53 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n6RB6r37018936 for freebsd-geom@FreeBSD.org; Mon, 27 Jul 2009 11:06:53 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 27 Jul 2009 11:06:53 GMT Message-Id: <200907271106.n6RB6r37018936@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-geom@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-geom@FreeBSD.org X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Jul 2009 11:06:54 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/136467 geom [geom] glabel(8) destroys access to GEOM tree if volum o kern/135898 geom [geom] Severe filesystem corruption - large files or l o kern/134922 geom [gmirror] [panic] kernel panic when use fdisk on disk o kern/134113 geom [geli] Problem setting secondary GELI key o kern/134044 geom [geom] gmirror(8) overwrites fs with stale data from r o kern/133931 geom [geli] [request] intentionally wrong password to destr o bin/132845 geom [geom] [patch] ggated(8) does not close files opened a o kern/132273 geom glabel(8): [patch] failing on journaled partition o kern/132242 geom [gmirror] gmirror.ko fails to fully initialize o kern/131353 geom [geom] gjournal(8) kernel lock o kern/131037 geom [geli] Unable to create disklabel on .eli-Device p docs/130548 geom [patch] gjournal(8) man page is missing sysctls o kern/130528 geom gjournal fsck during boot o kern/129674 geom [geom] gjournal root did not mount on boot o kern/129645 geom gjournal(8): GEOM_JOURNAL causes system to fail to boo o kern/129245 geom [geom] gcache is more suitable for suffix based provid f kern/128276 geom [gmirror] machine lock up when gmirror module is used o kern/126902 geom [geom] geom_label: kernel panic during install boot o kern/124973 geom [gjournal] [patch] boot order affects geom_journal con o kern/124969 geom gvinum(8): gvinum raid5 plex does not detect missing s o kern/124294 geom [geom] gmirror(8) have inappropriate logic when workin o kern/124130 geom [gmirror] [usb] gmirror fails to start usb devices tha o kern/123962 geom [panic] [gjournal] gjournal (455Gb data, 8Gb journal), o kern/123630 geom [patch] [gmirror] gmirror doesnt allow the original dr o kern/123122 geom [geom] GEOM / gjournal kernel lock o kern/122738 geom [geom] gmirror list "losts consumers" after gmirror de f kern/122415 geom [geom] UFS labels are being constantly created and rem o kern/122067 geom [geom] [panic] Geom crashed during boot o kern/121559 geom [patch] [geom] geom label class allows to create inacc o kern/121364 geom [gmirror] Removing all providers create a "zombie" mir o kern/120231 geom [geom] GEOM_CONCAT error adding second drive o kern/120091 geom [geom] [geli] [gjournal] geli does not prompt for pass o kern/120044 geom [msdosfs] [geom] incorrect MSDOSFS label fries adminis o kern/120021 geom [geom] [panic] net-p2p/qbittorrent crashes system when o kern/119743 geom [geom] geom label for cds is keeped after dismount and o kern/115856 geom [geli] ZFS thought it was degraded when it should have o kern/115547 geom [geom] [patch] [request] let GEOM Eli get password fro o kern/114532 geom [geom] GEOM_MIRROR shows up in kldstat even if compile o kern/113957 geom [gmirror] gmirror is intermittently reporting a degrad o kern/113885 geom [gmirror] [patch] improved gmirror balance algorithm o kern/113837 geom [geom] unable to access 1024 sector size storage o kern/113419 geom [geom] geom fox multipathing not failing back p bin/110705 geom gmirror(8) control utility does not exit with correct o kern/107707 geom [geom] [patch] [request] add new class geom_xbox360 to o kern/104389 geom [geom] [patch] sys/geom/geom_dump.c doesn't encode XML o kern/98034 geom [geom] dereference of NULL pointer in acd_geom_detach o kern/94632 geom [geom] Kernel output resets input while GELI asks for o kern/90582 geom [geom] [panic] Restore cause panic string (ffs_blkfree o bin/90093 geom fdisk(8) incapable of altering in-core geometry a kern/89660 geom [vinum] [patch] [panic] due to g_malloc returning null o kern/89546 geom [geom] GEOM error o kern/88601 geom [geli] geli cause kernel panic under heavy disk usage o kern/87544 geom [gbde] mmaping large files on a gbde filesystem deadlo o kern/84556 geom [geom] [panic] GBDE-encrypted swap causes panic at shu o kern/79251 geom [2TB] newfs fails on 2.6TB gbde device o kern/79035 geom [vinum] gvinum unable to create a striped set of mirro o bin/78131 geom gbde(8) "destroy" not working. s kern/73177 geom kldload geom_* causes panic due to memory exhaustion 58 problems total. From owner-freebsd-geom@FreeBSD.ORG Fri Jul 31 06:23:08 2009 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F1E2C106564A for ; Fri, 31 Jul 2009 06:23:08 +0000 (UTC) (envelope-from acc@hexadecagram.org) Received: from mail.itproficiency.com (hexadecagram.org [166.70.126.65]) by mx1.freebsd.org (Postfix) with ESMTP id 8589E8FC15 for ; Fri, 31 Jul 2009 06:23:08 +0000 (UTC) (envelope-from acc@hexadecagram.org) Received: from localhost (unknown [127.0.0.2]) by mail.itproficiency.com (Postfix) with ESMTP id 4242DC106FD; Fri, 31 Jul 2009 00:06:13 -0600 (MDT) X-Virus-Scanned: amavisd-new at itproficiency.com Received: from mail.itproficiency.com ([127.0.0.2]) by localhost (mail.itproficiency.com [127.0.0.2]) (amavisd-new, port 10024) with LMTP id ccYsQIQ-M1uU; Fri, 31 Jul 2009 00:05:57 -0600 (MDT) Received: from ares.aegaeum.hexadecagram.org (ares.aegaeum.hexadecagram.org [192.168.133.220]) by mail.itproficiency.com (Postfix) with ESMTP id 214C5C0FF81; Fri, 31 Jul 2009 00:05:54 -0600 (MDT) Message-ID: <4A7289B9.2060907@hexadecagram.org> Date: Fri, 31 Jul 2009 00:05:45 -0600 From: Anthony Chavez Organization: hexadecagram.org User-Agent: Mozilla-Thunderbird 2.0.0.19 (X11/20090103) MIME-Version: 1.0 To: freebsd-geom@freebsd.org References: <4A62E0CE.1000508@hexadecagram.org> <20090729140436.GG1586@garage.freebsd.pl> In-Reply-To: <20090729140436.GG1586@garage.freebsd.pl> X-Enigmail-Version: 0.95.7 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigB401B05976538240EEF8BBEA" Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: Re-starting a gjournal provider X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Jul 2009 06:23:09 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigB401B05976538240EEF8BBEA Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks very much for responding, Pawel. I'm moving this discussion to freebsd-geom, which is where I probably should have posted in the first place. Lack of sleep and coffee on Sunday morning were partly to blame, I'm sure. ;-) Pawel Jakub Dawidek wrote: > On Sun, Jul 19, 2009 at 03:01:02AM -0600, Anthony Chavez wrote: >> Hello freebsd-fs, >> >> I'm trying to get gjournal working on a "removable" hard disk. I use >> the term loosely, because I'm using a very simple eSATA enclosure: an >> AMS Venus DS5 [1]. >> >> If I swap out disks, atacontrol cap ad0 seems sufficient enough to >> detect the new drive: the reported device model, serial number, firmwa= re >> revision, and CHS values change as one would expect. >> >> My interpretation of [2] section 5.3 and gjournal(8) is that the >> following sequence of commands should ensure me that all write buffers= >> have been flushed and bring the system to a point where it is safe to >> remove a disk. >> >> sync; sync; sync >> gjournal sync >> umount /dev/ad0s1.journal >> gjournal stop ad0s1.journal >=20 > You should first unmount and then call 'gjournal sync'. Thank you for clarifying that. You mention this again later on in your response, and I respond below. >> However, once they are executed, /dev/ad0s1.journal disappears and whe= n >> I swap out the disk it doesn't come back. The only way I've found to >> bring it back is atacontrol detach ata0; atacontrol attach ata0, which= >> doesn't seem like a wise thing to do if I have another device on the >> same channel. >=20 > It doesn't come back because something (ATA layer?) doesn't properly > remove ad0 provider. When you remove the disk, /dev/ad0 should disappea= r > and reappear once you insert it again. >=20 > You can still do this trick after you insert the disk again so the GEOM= > can schedule retaste: >=20 > # true > /dev/ad0 Thank you for informing me of that trick. I tried using it after "gjournal stop" but unfortunately, nothing changed. My terminology might have been a bit off in my initial post (gjournal is still a bit new to me). So I will attempt to clarify a bit more. Here is an example of a typical session (the only difference this time being "gjournal sync" following umount as you prescribed). % sudo atacontrol info ata0 Master: ad0 SATA revision 1.x Slave: no device present % ls /dev/ad0* /dev/ad0 /dev/ad0s1 /dev/ad0s1.journal % mount | grep ad0s1 /dev/ad0s1.journal on /mnt/ad0s1 (ufs, local, gjournal) % ( subsh> set -o errexit subsh> sync subsh> sync subsh> sync subsh> sudo umount /dev/ad0s1.journal subsh> gjournal sync subsh> sudo gjournal stop ad0s1.journal subsh> ) % ls /dev/ad0* /dev/ad0 /dev/ad0s1 % sudo true \> /dev/ad0; ls /dev/ad0* /dev/ad0 /dev/ad0s1 % sudo true \> /dev/ad0s1; ls /dev/ad0* /dev/ad0 /dev/ad0s1 % sudo atacontrol detach ata0 && sudo atacontrol attach ata0 Master: ad0 SATA revision 1.x Slave: no device present % ls /dev/ad0* /dev/ad0 /dev/ad0s1 /dev/ad0s1.journal Here are the points to note. 1) When I physically remove a drive from the enclosure, /dev/ad0 does not disappear. /dev/ad0 *always* exists until I "atacontrol detach." Even when the device is powered off, /dev/ad0 continues to exist. 2) /dev/ad0s1.journal disappears when I "gjournal stop." /dev/ad0s1.journal is the device that, AFAIK, will only come back after "atacontrol detach ata0; atacontrol attach ata0". 3) When I swap drives, "atacontrol cap ad0" will produce a report for the newly-inserted drive. If I attempt to "atacontrol info ata0" before issuing that command, it continues to display the drive model and firmware revision from the drive that was previously inserted. However, "atacontrol cap" does not appear to provoke the return of /dev/ad0s1.journal. >> My question is, do I need to issue gjournal stop before I swap disks? >> And if so, is there any way that I can avoid the atacontrol >> detach/attach cycle that would need to take place before any mount is >> attempted so that /dev/ad0s1.journal appears (if in the drive inserted= >> at the time does in fact utilize gjournal; I may want to experiment wi= th >> having disks with either gjournal or soft updates)? This paragraph (above) and the one that that proceeded it in my original post contains the following 2 questions that remain unanswered (I've added another which was implied previously at best). 1) Is "atacontrol detach ata0 && atacontrol attach ata0" in fact a safe operation to perform in any circumstance? My better judgment has me thinking that the answer to this question is almost certainly "no." However, I am hypothesizing that it would safe enough if all devices on ata0 are properly unmounted first, but if I can avoid that, I will. It feels clumsy and seems to defeat the purpose of hot-swapping. 2) Is it *necessary* to "gjournal stop" before hot-swapping? In such a scenario, I would opt to simply "umount; gjournal sync," swap disks, and then "atacontrol cap ad0; mount" (or even just "mount"). It seems quite likely, however, that all drives that undergo this treatment would be *required* to have gjournal labels since /dev/ad0s1.journal would never disappear (although I've yet to actually test that). 3) If the answer to question 2 is "yes," then how can I handle the case of inserting a drive that does *not* have a gjournal label? >> And while I'm on the subject, are the (gjournal) syncs commands >> preceeding umount absolutely necessary in the case of removable media?= >=20 > 'gjournal sync' should follow unmount, not the other way around. And it= s > better to do it, but 'gjournal stop' should do the same. If that is indeed the case then the article I referenced as [2], "Implementing UFS Journaling on a Desktop PC," should be updated to reflect that ordering (section 5.3 prescribes a "umount" followed by "gjournal sync"). I'm submitting a PR that addresses this. In any case, the question I was asking here is actually twofold: 1) Is it really necessary to perform 3 "sync" commands before "umount"? Line 94 of src/sbin/umount/umount.c,v 1.45.20.1 has me thinking that the answer is "no," since it calls sync() itself, albeit only once. I got the idea for executing "sync" three times from /etc/rc.suspend. 2) Is it necessary to "gjournal sync" if I'm going to "gjournal stop" anyway? (You answered this one already.) Thank you for the assistance. --=20 Anthony Chavez http://hexadecagram.org/ mailto:acc@hexadecagram.org xmpp:acc@hexadecagram.org --------------enigB401B05976538240EEF8BBEA Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkpyicAACgkQbZTbIaRBRXFQPgCfemQsbeTMnQsFfSkLIR7VMvjM f1UAmwWwuO8tnWGM3hmXOqp5F7nbFo3c =rY6f -----END PGP SIGNATURE----- --------------enigB401B05976538240EEF8BBEA-- From owner-freebsd-geom@FreeBSD.ORG Fri Jul 31 06:49:32 2009 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5AF47106564A; Fri, 31 Jul 2009 06:49:32 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello087206049004.chello.pl [87.206.49.4]) by mx1.freebsd.org (Postfix) with ESMTP id ADF7E8FC21; Fri, 31 Jul 2009 06:49:31 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 90F8E45CA0; Fri, 31 Jul 2009 08:49:29 +0200 (CEST) Received: from localhost (chello087206049004.chello.pl [87.206.49.4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 1B41445685; Fri, 31 Jul 2009 08:49:24 +0200 (CEST) Date: Fri, 31 Jul 2009 08:49:48 +0200 From: Pawel Jakub Dawidek To: Anthony Chavez Message-ID: <20090731064948.GG1584@garage.freebsd.pl> References: <4A62E0CE.1000508@hexadecagram.org> <20090729140436.GG1586@garage.freebsd.pl> <4A7289B9.2060907@hexadecagram.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Sw7tCqrGA+HQ0/zt" Content-Disposition: inline In-Reply-To: <4A7289B9.2060907@hexadecagram.org> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 8.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs@freebsd.org, freebsd-geom@freebsd.org Subject: Re: Re-starting a gjournal provider X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Jul 2009 06:49:32 -0000 --Sw7tCqrGA+HQ0/zt Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jul 31, 2009 at 12:05:45AM -0600, Anthony Chavez wrote: > > It doesn't come back because something (ATA layer?) doesn't properly > > remove ad0 provider. When you remove the disk, /dev/ad0 should disappear > > and reappear once you insert it again. > >=20 > > You can still do this trick after you insert the disk again so the GEOM > > can schedule retaste: > >=20 > > # true > /dev/ad0 >=20 > Thank you for informing me of that trick. I tried using it after > "gjournal stop" but unfortunately, nothing changed. This is because it should be /dev/ad0s1 and not /dev/ad0. Try with /dev/ad0s1. > Here are the points to note. >=20 > 1) When I physically remove a drive from the enclosure, /dev/ad0 does > not disappear. /dev/ad0 *always* exists until I "atacontrol detach." > Even when the device is powered off, /dev/ad0 continues to exist. This might be three things: 1. Your enclosure/controller doesn't report back about disk being removed. 2. Your enclosure does report back, but ATA ignores such report. This will be a bug in ATA. 3. Your controller doesn't support hot-swap or it supports warm-swap, which means you have to detach it by hand before removing it. > 2) /dev/ad0s1.journal disappears when I "gjournal stop." > /dev/ad0s1.journal is the device that, AFAIK, will only come back after > "atacontrol detach ata0; atacontrol attach ata0". It should also get back after 'true > /dev/ad0s1'. What this command do is to open provider for writing (it doesn't write anything). In GEOM it will trigger spoil event and then, once command completes, it will trigger retaste event. This mean that GEOM will inform gjournal to check /dev/ad0s1 again and this will allow gjournal to find its metadata and create /dev/ad0s1.journal once again. One more test would be in place. If you could try the command below before removing disk and after inserting different disk: # diskinfo -v /dev/ad0 If it shows exactly the same in two cases, it means that it is not aware that disk was replaced and detach/attach cycle is needed. > 1) Is "atacontrol detach ata0 && atacontrol attach ata0" in fact a safe > operation to perform in any circumstance? >=20 > My better judgment has me thinking that the answer to this question is > almost certainly "no." However, I am hypothesizing that it would safe > enough if all devices on ata0 are properly unmounted first, but if I can > avoid that, I will. It feels clumsy and seems to defeat the purpose of > hot-swapping. It should be safe, but there were plenty of bugs related to disappearing disk from under mount file system, etc. If nothing is mounted you should be fine (if there are no ATA bugs in this area). But for full hot-swap the disk controller should discover disk being removed and ATA code should remove it from /dev/. > 2) Is it *necessary* to "gjournal stop" before hot-swapping? >=20 > In such a scenario, I would opt to simply "umount; gjournal sync," swap > disks, and then "atacontrol cap ad0; mount" (or even just "mount"). It > seems quite likely, however, that all drives that undergo this treatment > would be *required* to have gjournal labels since /dev/ad0s1.journal > would never disappear (although I've yet to actually test that). I'd go with 'umount; gjournal stop' and drop 'gjournal sync'. Controler should inform ATA that disk is gone. ATA should inform GEOM that ad0 is gone. If that would be the case, simple 'umount; gjournal sync' will be enough. But because it isn't the case, you have to stop gjournal and detach ad0. > 3) If the answer to question 2 is "yes," then how can I handle the case > of inserting a drive that does *not* have a gjournal label? There's nothing special here. Let's see how diskinfo test will go first. > 1) Is it really necessary to perform 3 "sync" commands before "umount"? >=20 > Line 94 of src/sbin/umount/umount.c,v 1.45.20.1 has me thinking that the > answer is "no," since it calls sync() itself, albeit only once. I got > the idea for executing "sync" three times from /etc/rc.suspend. The idea is that unmount should take case of syncing data. There should be not need for even one sync. It is called "just in case". > 2) Is it necessary to "gjournal sync" if I'm going to "gjournal stop" > anyway? (You answered this one already.) No, stop should be sufficient. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --Sw7tCqrGA+HQ0/zt Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFKcpQMForvXbEpPzQRAvBrAJ4l7CA4pYXbRsfVDqT4vdzeSqIYggCdHrxs FlRRqlOfTP6dD/n2dVzEBsQ= =dGOD -----END PGP SIGNATURE----- --Sw7tCqrGA+HQ0/zt-- From owner-freebsd-geom@FreeBSD.ORG Fri Jul 31 14:17:11 2009 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BC3EC106566B for ; Fri, 31 Jul 2009 14:17:11 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Received: from lennier.cc.vt.edu (lennier.cc.vt.edu [198.82.162.213]) by mx1.freebsd.org (Postfix) with ESMTP id 768348FC08 for ; Fri, 31 Jul 2009 14:17:11 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Received: from freya.cc.vt.edu (freya.cc.vt.edu [198.82.163.115]) by lennier.cc.vt.edu (8.13.8/8.13.8) with ESMTP id n6VDNXIh016859 for ; Fri, 31 Jul 2009 09:23:33 -0400 Received: from auth3.smtp.vt.edu (EHLO auth3.smtp.vt.edu) ([198.82.161.152]) by freya.cc.vt.edu (MOS 3.10.4-GA FastPath queued) with ESMTP id DYT87496; Fri, 31 Jul 2009 09:23:32 -0400 (EDT) Received: from gromit.tower.lib.vt.edu (gromit.tower.lib.vt.edu [128.173.51.22]) (authenticated bits=0) by auth3.smtp.vt.edu (8.13.8/8.13.8) with ESMTP id n6VDP4Hf017526 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO) for ; Fri, 31 Jul 2009 09:25:04 -0400 Message-Id: <431FC16E-A25C-4BC3-A283-B1DAF2E3E46E@gromit.dlib.vt.edu> From: Paul Mather To: freebsd-geom@freebsd.org Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v935.3) Date: Fri, 31 Jul 2009 09:23:32 -0400 X-Mailer: Apple Mail (2.935.3) X-Mirapoint-Received-SPF: 198.82.161.152 auth3.smtp.vt.edu paul@gromit.dlib.vt.edu 5 none Subject: ZFS ignores some labels, now pool is corrupted. X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Jul 2009 14:17:12 -0000 I recently repurposed a motley assortment of hardware that used to be a JBOD ad hoc backup mirror to use FreeBSD 7-STABLE and ZFS. When I say motley I mean motley: it has four internal SATA 1 TB drives and three external Maxtor OneTouch 1 TB USB drives. I aggregated together all of these drives as a single raidz1 using ZFS. Following a recent suggestion on here, before creating the raidz1 vdev I labelled each drive as "driveN" using glabel, e.g., "glabel label drive1 /dev/ad4". (I figured this would be important especially for the external USB drives, which might get plugged into different USB ports and thus probed in a different order to the one when the pool was created and hence shuffle device names.) When creating the pool, I used "zpool create backups raidz label/drive1 label/drive2 ...". That all worked for a week or so until today when I rebooted. One of the USB drives was not probed during boot and so was flagged as "REMOVED" when doing a "zpool status.": pool: backups state: DEGRADED scrub: none requested config: NAME STATE READ WRITE CKSUM backups DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 label/drive1 ONLINE 0 0 0 label/drive2 ONLINE 0 0 0 label/drive3 ONLINE 0 0 0 label/drive4 ONLINE 0 0 0 label/drive5 REMOVED 0 0 0 label/drive6 ONLINE 0 0 0 label/drive7 ONLINE 0 0 0 errors: No known data errors I unplugged and plugged in the REMOVED drive's cable to get it to probe. Eventually, the system appeared to recognise the drive and resilver: pool: backups state: ONLINE scrub: resilver completed after 0h0m with 0 errors on Fri Jul 31 07:54:22 2009 config: NAME STATE READ WRITE CKSUM backups ONLINE 0 0 0 raidz1 ONLINE 0 0 0 label/drive1 ONLINE 0 0 0 11.5K resilvered label/drive2 ONLINE 0 0 0 11K resilvered label/drive3 ONLINE 0 0 0 12K resilvered label/drive4 ONLINE 0 0 0 11.5K resilvered label/drive5 ONLINE 0 0 0 17.5K resilvered label/drive6 ONLINE 0 0 0 13K resilvered label/drive7 ONLINE 0 0 0 11.5K resilvered errors: No known data errors I rebooted again, but, once more, the drive did not probe during boot, so I had to force it to probe by unplugging and plugging in its USB cable. This time, however, the drive was mis-identified in the pool as "da2" instead of "label/drive5" and, in fact, /dev/label/drive5 was missing: pool: backups state: ONLINE scrub: resilver completed after 0h0m with 0 errors on Fri Jul 31 07:59:43 2009 config: NAME STATE READ WRITE CKSUM backups ONLINE 0 0 0 raidz1 ONLINE 0 0 0 label/drive1 ONLINE 0 0 0 8.50K resilvered label/drive2 ONLINE 0 0 0 10K resilvered label/drive3 ONLINE 0 0 0 9K resilvered label/drive4 ONLINE 0 0 0 10K resilvered da2 ONLINE 0 0 0 11.5K resilvered label/drive6 ONLINE 0 0 0 7.50K resilvered label/drive7 ONLINE 0 0 0 8.50K resilvered errors: No known data errors $ ls /dev/label drive1 drive2 drive3 drive4 drive6 drive7 For some reason, the label was not being detected properly. When I rebooted, things went from bad to worse. I now have two "da2" devices show up in my raidz vdev and this time my label/drive7 has disappeared. This seems to have thrown ZFS for a loop and my vdev is corrupted: pool: backups state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-4J scrub: none requested config: NAME STATE READ WRITE CKSUM backups DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 label/drive1 ONLINE 0 0 0 label/drive2 ONLINE 0 0 0 label/drive3 ONLINE 0 0 0 label/drive4 ONLINE 0 0 0 da2 FAULTED 0 0 0 corrupted data label/drive6 ONLINE 0 0 0 da2 ONLINE 0 0 0 errors: No known data errors $ ls /dev/label drive1 drive2 drive3 drive4 drive5 drive6 When I boot up in single-user mode all of my original "driveN" labels (1-7) show up. However, right now, with ZFS active, label/drive7 refused to appear. Is there a problem with ZFS and labels? Does anyone have any suggestions for how to repair this pool? I'm presuming I can't do a "zpool replace backups da2 /dev/label/drive5" to repair the faulted drive because I now have two "da2" devices in my vdev. As a sort of related question, is there a better way to create a pool out of these devices yet still maximise the amount of storage (allowing for some redundancy)? For example, would it be better to do something like this: zpool create backups raidz label/sata1 label/sata2 label/sata3 label/ sata4 \ raidz label/usb1 label/usb2 label/usb3 (where "sataN" are the internal SATA drives and "usbN" are the external USB drives) to place the internal and external drives into separate vdevs (albeit losing an extra drive of storage space to parity)? (Would that improve I/O speeds? I'm guessing it should.) Or, is it just storing up trouble to try and mix these USB devices into the pool as I am now and I'd be best off trying to lobby for an eSATA enclosure if I want to use external drives? Cheers, Paul. From owner-freebsd-geom@FreeBSD.ORG Fri Jul 31 14:52:37 2009 Return-Path: Delivered-To: geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 32447106564A for ; Fri, 31 Jul 2009 14:52:37 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mail.digiware.nl (mail.ip6.digiware.nl [IPv6:2001:4cb8:1:106::2]) by mx1.freebsd.org (Postfix) with ESMTP id E8D538FC08 for ; Fri, 31 Jul 2009 14:52:36 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from localhost (localhost.digiware.nl [127.0.0.1]) by mail.digiware.nl (Postfix) with ESMTP id 3D8B9153434 for ; Fri, 31 Jul 2009 16:52:36 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from mail.digiware.nl ([127.0.0.1]) by localhost (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jDt9lP6KMOYw; Fri, 31 Jul 2009 16:52:34 +0200 (CEST) Received: from [192.168.10.67] (opteron [192.168.10.67]) by mail.digiware.nl (Postfix) with ESMTP id 5B625153433 for ; Fri, 31 Jul 2009 16:52:34 +0200 (CEST) Message-ID: <4A7305A9.3080506@digiware.nl> Date: Fri, 31 Jul 2009 16:54:33 +0200 From: Willem Jan Withagen Organization: Digiware User-Agent: Thunderbird 2.0.0.22 (Windows/20090605) MIME-Version: 1.0 To: geom@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Gmirror rebuilding X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Jul 2009 14:52:37 -0000 Hi, I lost one of my disk in a gmirror, so I inserted a fresh one. And thusfar things went rather smoothly,it started rebuilding automagically. That's good,but what isn't is: Jul 31 16:43:15 www kernel: ad2: FAILURE - READ_DMA status=51 error=40 LBA=16344448 Jul 31 16:43:15 www kernel: GEOM_MIRROR: Synchronization request failed (error=5). mirror/mirror[READ(offset=8368291840, length=131072)] Jul 31 16:43:40 www kernel: ad2: FAILURE - READ_DMA status=51 error=40 LBA=16910976 Jul 31 16:43:40 www kernel: GEOM_MIRROR: Synchronization request failed (error=5). mirror/mirror[READ(offset=8658354176, length=131072)] and ad2 is the original disk. So somewhere I'm left with corrupt files. And what's worse, once this happens, geom_mirror does not continue with the remainder of the disk... It claims it is, but there is no activity at all on the disks. So what to do???? Hard way out would be to make a backup, reinstall the basic system with another/second fresh harddisk, and recover the backup. But that is a lot of work. Why doesn't geom_mirror continue with the remainder of the disk? --WjW From owner-freebsd-geom@FreeBSD.ORG Fri Jul 31 15:39:20 2009 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A71E5106566B for ; Fri, 31 Jul 2009 15:39:20 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from ns1.jnielsen.net (ns1.jnielsen.net [69.55.238.237]) by mx1.freebsd.org (Postfix) with ESMTP id 8BECC8FC18 for ; Fri, 31 Jul 2009 15:39:20 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from jnielsen.socialserve.com (office.socialserve.com [208.60.89.34]) (authenticated bits=0) by ns1.jnielsen.net (8.12.9p2/8.12.9) with ESMTP id n6VFIdBc029828; Fri, 31 Jul 2009 11:18:40 -0400 (EDT) (envelope-from lists@jnielsen.net) From: John Nielsen To: freebsd-geom@freebsd.org Date: Fri, 31 Jul 2009 11:18:33 -0400 User-Agent: KMail/1.9.10 References: <4A7305A9.3080506@digiware.nl> In-Reply-To: <4A7305A9.3080506@digiware.nl> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200907311118.33490.lists@jnielsen.net> X-Virus-Scanned: ClamAV version 0.88.4, clamav-milter version 0.88.4 on ns1.jnielsen.net X-Virus-Status: Clean Cc: Willem Jan Withagen Subject: Re: Gmirror rebuilding X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Jul 2009 15:39:20 -0000 On Friday 31 July 2009 10:54:33 Willem Jan Withagen wrote: > I lost one of my disk in a gmirror, so I inserted a fresh one. > And thusfar things went rather smoothly,it started rebuilding > automagically. > > That's good,but what isn't is: > > Jul 31 16:43:15 www kernel: ad2: FAILURE - READ_DMA > status=51 error=40 LBA=16344448 > Jul 31 16:43:15 www kernel: GEOM_MIRROR: Synchronization request failed > (error=5). mirror/mirror[READ(offset=8368291840, length=131072)] > Jul 31 16:43:40 www kernel: ad2: FAILURE - READ_DMA > status=51 error=40 LBA=16910976 > Jul 31 16:43:40 www kernel: GEOM_MIRROR: Synchronization request failed > (error=5). mirror/mirror[READ(offset=8658354176, length=131072)] > > and ad2 is the original disk. So somewhere I'm left with corrupt files. > > And what's worse, once this happens, geom_mirror does not continue with the > remainder of the disk... > It claims it is, but there is no activity at all on the disks. > > So what to do???? > > Hard way out would be to make a backup, reinstall the basic system with > another/second fresh harddisk, and recover the backup. > But that is a lot of work. If you have a backup already then restoring from it would be the best option. If you don't then the "hard" way above is a good second (assuming you can read enough of your remaining disk to get your backup tool to cooperate). You should make a backup in any case, but if you want to try to avoid reinstalling you could do some dd trickery. Remove the new disk from the mirror. Create a new mirror containing only the new disk. Run dd if=/dev/mirror/ of=/dev/mirror/ conv=noerror,sync This will take a long time with the default block size (512 bytes, one sector), but the plus side is that you only lose the data from the sectors that cannot be read. Depending on the extent of the damage to the original disk and your level of desperation, you may want to make note of which sectors fail to copy on the first run and try to copy them again (dd if=... of=... skip=NN seek=NN count=1). See also sysutils/ddrescue, sysutils/recoverdm and similar (I haven't used any of them). If you get a (mostly) viable clone, run fsck -f on it, assess the damage, update your fstab(s), reboot and add a second new disk to your new mirror. > Why doesn't geom_mirror continue with the remainder of the disk? I'll leave this question for someone else, but I suspect the behavior is intentional. As you say, you now have corruption on your volume so the best recourse is to restore from a known good backup. JN