From owner-freebsd-fs@FreeBSD.ORG Sun Nov 3 03:22:21 2013 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id A21B339F; Sun, 3 Nov 2013 03:22:21 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-vc0-x22e.google.com (mail-vc0-x22e.google.com [IPv6:2607:f8b0:400c:c03::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 52BEC284D; Sun, 3 Nov 2013 03:22:21 +0000 (UTC) Received: by mail-vc0-f174.google.com with SMTP id ld13so3781195vcb.5 for ; Sat, 02 Nov 2013 20:22:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=/v+7IEryTuWSNaT5MXvVcdpiMlmiBAI958uhdhRl84A=; b=mVspGHshDB20l8f+RpTdwMpkwH/jKQMMbB/dvWCqRAWT1J9D08Iiuyyw/FGHuA5Ybm Q7Mf3pXuLSPviEKDPt4bfrR2+LylXwozqa2J23s4+OQgCGf5b3rENUOxRCQ+ouJH2LnA m+n+0XGiX4cFIQxOGwdp21N13C7oenYtA2iI29upPiK3G8igKEgx2wpBdG4Q95aNmMjZ 27BjOM3IJnE6C5tgXgKoB7r4VHtDoWmh92sUPTqSWCM31sGUo8/pOebALqn/LApQXk4z AoIJGMc7BVSvGx6HsRiPNWATl4NFQd0A+oAd/LeyegikUHZvheAXFEwvN2Dr+A3ccIs3 PVzg== MIME-Version: 1.0 X-Received: by 10.220.184.70 with SMTP id cj6mr1461044vcb.23.1383448939996; Sat, 02 Nov 2013 20:22:19 -0700 (PDT) Sender: artemb@gmail.com Received: by 10.221.9.2 with HTTP; Sat, 2 Nov 2013 20:22:19 -0700 (PDT) Date: Sat, 2 Nov 2013 20:22:19 -0700 X-Google-Sender-Auth: ALZlonwybfmhgWGXaPYz6AJI_cQ Message-ID: Subject: Can't mount root from raidz2 after r255763 in stable/9 From: Artem Belevich To: fs@freebsd.org, "stable@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Nov 2013 03:22:21 -0000 Hi, I have a box with root mounted from 8-disk raidz2 ZFS volume. After recent buildworld I've ran into an issue that kernel fails to mount root with error 6. r255763 on stable/9 is the first revision that fails to mount root on mybox. Preceding r255749 boots fine. Commit r255763 (http://svnweb.freebsd.org/base?view=revision&revision=255763) MFCs bunch of changes from 10 but I don't see anything that obviously impacts ZFS. Attempting to boot with vfs.zfs.debug=1 shows that order in which geom providers are probed by zfs has apparently changed. Kernels that boot, show "guid match for provider /dev/gpt/" while failing kernels show "guid match for provider /dev/daX" -- the raw disks that are *not* the right geom provider for my pool slices. Beats me why ZFS picks raw disks over GPT partitions it should have. Pool configuration: #zpool status z0 pool: z0 state: ONLINE scan: scrub repaired 0 in 8h57m with 0 errors on Sat Oct 19 20:23:52 2013 config: NAME STATE READ WRITE CKSUM z0 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/da0p4-z0 ONLINE 0 0 0 gpt/da1p4-z0 ONLINE 0 0 0 gpt/da2p4-z0 ONLINE 0 0 0 gpt/da3p4-z0 ONLINE 0 0 0 gpt/da4p4-z0 ONLINE 0 0 0 gpt/da5p4-z0 ONLINE 0 0 0 gpt/da6p4-z0 ONLINE 0 0 0 gpt/da7p4-z0 ONLINE 0 0 0 logs mirror-1 ONLINE 0 0 0 gpt/ssd-zil-z0 ONLINE 0 0 0 gpt/ssd1-zil-z0 ONLINE 0 0 0 cache gpt/ssd1-l2arc-z0 ONLINE 0 0 0 errors: No known data errors Here are screen captures from a failed boot: https://plus.google.com/photos/+ArtemBelevich/albums/5941857781891332785 And here's boot log from successful boot on the same system: http://pastebin.com/XCwebsh7 Removing ZIL and L2ARC makes no difference -- r255763 still fails to mount root. I'm thoroughly baffled. Is there's something wrong with the pool -- some junk metadata somewhere on the disk that now screws with the root mounting? Changed order in geom provider enumeration? Something else? Any suggestions on what I can do to debug this further? --Artem