From owner-freebsd-stable@FreeBSD.ORG Sun Nov 3 08:03:23 2013 Return-Path: <owner-freebsd-stable@FreeBSD.ORG> Delivered-To: stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 7F40A3F4; Sun, 3 Nov 2013 08:03:23 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 45B482284; Sun, 3 Nov 2013 08:03:18 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id KAA00333; Sun, 03 Nov 2013 10:03:17 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1VcseT-0001oz-40; Sun, 03 Nov 2013 10:03:17 +0200 Message-ID: <5276030E.5040100@FreeBSD.org> Date: Sun, 03 Nov 2013 10:02:22 +0200 From: Andriy Gapon <avg@FreeBSD.org> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.1.0 MIME-Version: 1.0 To: Artem Belevich <art@FreeBSD.org>, fs@FreeBSD.org, "stable@freebsd.org" <stable@FreeBSD.org> Subject: Re: Can't mount root from raidz2 after r255763 in stable/9 References: <CAFqOu6jfZc5bGF4n0tLa+Y7=UkqmbsK589o6G+UiP3OTdyLdTg__13033.8046853014$1383448959$gmane$org@mail.gmail.com> In-Reply-To: <CAFqOu6jfZc5bGF4n0tLa+Y7=UkqmbsK589o6G+UiP3OTdyLdTg__13033.8046853014$1383448959$gmane$org@mail.gmail.com> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-stable>, <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable> List-Post: <mailto:freebsd-stable@freebsd.org> List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>, <mailto:freebsd-stable-request@freebsd.org?subject=subscribe> X-List-Received-Date: Sun, 03 Nov 2013 08:03:23 -0000 on 03/11/2013 05:22 Artem Belevich said the following: > Hi, > > I have a box with root mounted from 8-disk raidz2 ZFS volume. > After recent buildworld I've ran into an issue that kernel fails to > mount root with error 6. > r255763 on stable/9 is the first revision that fails to mount root on > mybox. Preceding r255749 boots fine. > > Commit r255763 (http://svnweb.freebsd.org/base?view=revision&revision=255763) > MFCs bunch of changes from 10 but I don't see anything that obviously > impacts ZFS. Indeed. > Attempting to boot with vfs.zfs.debug=1 shows that order in which geom > providers are probed by zfs has apparently changed. Kernels that boot, > show "guid match for provider /dev/gpt/<valid pool slice>" while > failing kernels show "guid match for provider /dev/daX" -- the raw > disks that are *not* the right geom provider for my pool slices. Beats > me why ZFS picks raw disks over GPT partitions it should have. Perhaps the kernel gpart code fails to recognize the partitions and thus ZFS can't see them? > Pool configuration: > #zpool status z0 > pool: z0 > state: ONLINE > scan: scrub repaired 0 in 8h57m with 0 errors on Sat Oct 19 20:23:52 2013 > config: > > NAME STATE READ WRITE CKSUM > z0 ONLINE 0 0 0 > raidz2-0 ONLINE 0 0 0 > gpt/da0p4-z0 ONLINE 0 0 0 > gpt/da1p4-z0 ONLINE 0 0 0 > gpt/da2p4-z0 ONLINE 0 0 0 > gpt/da3p4-z0 ONLINE 0 0 0 > gpt/da4p4-z0 ONLINE 0 0 0 > gpt/da5p4-z0 ONLINE 0 0 0 > gpt/da6p4-z0 ONLINE 0 0 0 > gpt/da7p4-z0 ONLINE 0 0 0 > logs > mirror-1 ONLINE 0 0 0 > gpt/ssd-zil-z0 ONLINE 0 0 0 > gpt/ssd1-zil-z0 ONLINE 0 0 0 > cache > gpt/ssd1-l2arc-z0 ONLINE 0 0 0 > > errors: No known data errors > > Here are screen captures from a failed boot: > https://plus.google.com/photos/+ArtemBelevich/albums/5941857781891332785 I don't have permission to view this album. > And here's boot log from successful boot on the same system: > http://pastebin.com/XCwebsh7 > > Removing ZIL and L2ARC makes no difference -- r255763 still fails to mount root. > > I'm thoroughly baffled. Is there's something wrong with the pool -- > some junk metadata somewhere on the disk that now screws with the root > mounting? Changed order in geom provider enumeration? Something else? > Any suggestions on what I can do to debug this further? gpart. -- Andriy Gapon