From owner-freebsd-current@FreeBSD.ORG Sun Oct 27 15:10:40 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id B1BE023C for ; Sun, 27 Oct 2013 15:10:40 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6F8552BF3 for ; Sun, 27 Oct 2013 15:10:40 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.80.1) with esmtp (envelope-from ) id <1VaRz5-002cCK-T0>; Sun, 27 Oct 2013 16:10:31 +0100 Received: from g225187133.adsl.alicedsl.de ([92.225.187.133] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.80.1) with esmtpsa (envelope-from ) id <1VaRz5-000uzf-Nn>; Sun, 27 Oct 2013 16:10:31 +0100 Date: Sun, 27 Oct 2013 16:10:26 +0100 From: "O. Hartmann" To: "O. Hartmann" Subject: Re: ZFS buggy in CURRENT? Stuck in [zio->io_cv] forever! Message-ID: <20131027161026.6ec57034@thor.walstatt.dyndns.org> In-Reply-To: <20131027134039.574849f5@thor.walstatt.dyndns.org> References: <20131027134039.574849f5@thor.walstatt.dyndns.org> Organization: FU Berlin X-Mailer: Claws Mail 3.9.2 (GTK+ 2.24.19; amd64-portbld-freebsd11.0) Mime-Version: 1.0 Content-Type: multipart/signed; micalg=PGP-SHA1; boundary="Sig_/BPBJIUgc+FbpFs=SCfyINjI"; protocol="application/pgp-signature" X-Originating-IP: 92.225.187.133 Cc: FreeBSD CURRENT X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Oct 2013 15:10:40 -0000 --Sig_/BPBJIUgc+FbpFs=SCfyINjI Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On Sun, 27 Oct 2013 13:40:39 +0100 "O. Hartmann" wrote: >=20 > I have setup a RAIDZ pool comprised from 4 3TB HDDs. To maintain 4k > block alignment, I followed the instructions given on several sites > and I'll sketch them here for the protocol. >=20 > The operating system is 11.0-CURRENT AND 10.0-BETA2. >=20 > create a GPT partition on each drive and add one whole-covering > partition with the option >=20 > gpart add -t freebsd-zfs -b 1M -l disk0[0-3] ada[3-6] >=20 > gnop create -S4096 gtp/disk[3-6] >=20 > Because I added a disk to an existing RAIDZ, I exported the former > ZFS pool, then I deleted on each disk the partition and then destroyed > the GPT scheme. The former pool had a ZIL and CACHE residing on the > same SSD, partioned. I didn't kill or destroy the partitions on that > SSD. To align 4k blocks, I also created on the existing gpt/log00 and > gpt/cache00 via=20 >=20 > gnop create -S4096 gpt/log00|gpt/cache00 >=20 > the NOP overlays. >=20 > After I created a new pool via zpool create POOL gpt/disk0[0-3].nop > log gpt/log00.nop cache gpt/cache00.nop It is, of course, a "zpool create POOL raidz ..." >=20 > I "received" a snapshot taken and sent to another storage array, after > I the newly created pool didn't show up any signs of illness or > corruption. >=20 > After ~10 hours of receiving the backup, I exported that pool amongst > the backup pool, destroyed the appropriate .nop device entries via=20 >=20 > gnop destroy gpt/disk0[0-3] >=20 > and the same for cache and log and tried to check via=20 >=20 > zpool import >=20 > whether my pool (as well as the backup pool) shows up. And here the > nasty mess starts! >=20 > The "zpool import" command issued on console is now stuck for hours > and can not be interrupted via Ctrl-C! No pool shows up! Hitting > Ctrl-T shows a state like >=20 > ... cmd: zpool 4317 [zio->io_cv]: 7345.34r 0.00 [...] >=20 > Looking with=20 >=20 > systat -vm 1 >=20 > at the trhoughput of the CAM devices I realise that two of the four > RAIDZ-comprising drives show activities, having 7000 - 8000 tps and ~ > 30 MB/s bandwidth - the other two zero! >=20 > And the pool is still inactive, the console is stuck. >=20 > Well, this made my day! At this point, I try to understand what's > going wrong and try to recall what I did the last time different when > the same procedure on three disks on the same hardware worked for me. >=20 > Now after 10 hours copy orgy and the need for the working array I > start believing that using ZFS is still peppered with too many > development-like flaws rendering it risky on FreeBSD. Colleagues > working on SOLARIS on ZFS I consulted never saw those stuck-behaviour > like I realise this moment. >=20 > I don not want to repeat the procedure again. There must be a > possibility to import the pool - even the backup pool, which is > working, untouched by the work, should be able to import - but it > doesn't. If I address that pool, while this crap "zpool import" > command is still blocking the console, not willing to die even with > "killall -9 zpool", I can not import the backup pool via "zpool > import BACKUP00". The console gets stuck immediately and for the > eternity without any notice. Htting Ctrl-T says something like=20 >=20 > load: 3.59 cmd: zpool 46199 [spa_namespace_lock] 839.18r 0.00u 0.00s > 0% 3036k >=20 > which means I can not even import the backup facility and this means > really no fun. --Sig_/BPBJIUgc+FbpFs=SCfyINjI Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQEcBAEBAgAGBQJSbSzmAAoJEOgBcD7A/5N83nAH/RJf5QOJYTjyIrwzxr4AGyuE MdVeaRKOcsiURaFDWsfe9ovFB//FaSFkdjYmEJZMGfK1KR/GthXLtAadCXfQ3wlM xC9CLF+WPqbf87HV4Y5Z8xypCp/kfGBvkJpLOvgRGBm+xZcjftnTj0Q7VH1M/DLr RHXRu1u8NxRBTsBVPPVELcWDZIbeKIbHRCakdjM298OshRJ9IEm78+AcUXcy5ovn xsbzu6EUSRjsxeFr59iaIFP6C1aI4Ba8mh6mi6RsKFqS7VwFX+WXY8CwR/TysRg1 rJUZnJJb/51IKhcHPntD6D0vEmk3JKgdjGaDgpDY1lLl8wOz4kr1XsoVGoehm70= =74M9 -----END PGP SIGNATURE----- --Sig_/BPBJIUgc+FbpFs=SCfyINjI--