From owner-freebsd-stable@FreeBSD.ORG Tue Sep 9 15:09:58 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 70CB0C4E for ; Tue, 9 Sep 2014 15:09:58 +0000 (UTC) Received: from st11p09mm-asmtp001.mac.com (st11p09mm-asmtp001.mac.com [17.164.24.96]) (using TLSv1 with cipher DES-CBC3-SHA (112/168 bits)) (Client CN "smtp.me.com", Issuer "VeriSign Class 3 Extended Validation SSL SGC CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3CEED8C8 for ; Tue, 9 Sep 2014 15:09:57 +0000 (UTC) Received: from [10.71.14.16] (dsl-hkibrasgw1-58c380-33.dhcp.inet.fi [88.195.128.33]) by st11p09mm-asmtp001.mac.com (Oracle Communications Messaging Server 7u4-27.10(7.0.4.27.9) 64bit (built Jun 6 2014)) with ESMTPSA id <0NBN0065Q3FNY300@st11p09mm-asmtp001.mac.com> for freebsd-stable@freebsd.org; Tue, 09 Sep 2014 15:09:27 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.12.52,1.0.28,0.0.0000 definitions=2014-09-09_05:2014-09-09,2014-09-09,1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=21 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1402240000 definitions=main-1409090132 Content-type: text/plain; charset=utf-8 MIME-version: 1.0 (Mac OS X Mail 8.0 \(1973.6\)) Subject: Re: ZFS on root booting broken somewhere after r270020 From: Kimmo Paasiala In-reply-to: Date: Tue, 09 Sep 2014 18:09:22 +0300 Content-transfer-encoding: quoted-printable Message-id: <7643523D-3839-42FC-A9CB-EAF3E6B028DB@icloud.com> References: <51AD1F36-1089-481F-8784-8BD8E6EF020F@icloud.com> <71DEB316-3CDD-4403-A397-BCE684725ABD@icloud.com> <25886C53-39C1-47A8-95F7-494FA6E7ABA2@icloud.com> <20140819071045.GS2737@kib.kiev.ua> <99FB0662-1954-4ECB-939B-06D0AA49C1A1@icloud.com> <20140819074643.GU2737@kib.kiev.ua> To: "freebsd-stable@freebsd.org" X-Mailer: Apple Mail (2.1973.6) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Sep 2014 15:09:58 -0000 > On 9.9.2014, at 17.23, Kimmo Paasiala wrote: >=20 >=20 >> On 19.8.2014, at 10.46, Konstantin Belousov = wrote: >>=20 >> On Tue, Aug 19, 2014 at 10:32:08AM +0300, Kimmo Paasiala wrote: >>>=20 >>> On 19.8.2014, at 10.10, Konstantin Belousov = wrote: >>>=20 >>>> On Tue, Aug 19, 2014 at 07:40:46AM +0300, Kimmo Paasiala wrote: >>>>>=20 >>>>> On 18.8.2014, at 8.54, Kimmo Paasiala wrote: >>>>>=20 >>>>>>=20 >>>>>> On 18.8.2014, at 2.32, Kimmo Paasiala = wrote: >>>>>>=20 >>>>>>> System is: >>>>>>>=20 >>>>>>> FreeBSD freebsd10.rdnzl.info 10.0-STABLE FreeBSD 10.0-STABLE #2 = r270020: Fri Aug 15 20:38:59 EEST 2014 = kimmo@buildstable10amd64.rdnzl.info:/usr/obj/usr/src/sys/GENERIC amd64 >>>>>>>=20 >>>>>>> This version still works fine. The one that didn?t work was = r270097. The kernel boots but gets stuck at the line: >>>>>>>=20 >>>>>>> Trying to mount root from zfs:pool/ROOT/default [] >>>>>>>=20 >>>>>>> I tried pressing enter at this point but got a panic, I don?t = have a screenshot of the panic at the moment. >>>>>>>=20 >>>>>>> Could this problem be related the this commit? : >>>>>>>=20 >>>>>>> = http://svnweb.freebsd.org/base?view=3Drevision&sortby=3Drev&sortdir=3Ddown= &revision=3D270095 >>>>>>>=20 >>>>>>> -Kimmo >>>>>>=20 >>>>>> Trying to bisect this I backed to r270050 and that version still = works. More to come. >>>>>>=20 >>>>>> -Kimmo >>>>>=20 >>>>> Version r270094 still works but commit r270095 definitely does = break booting from ZFS on root on my system. The error message I see on = the console is ?Mounting from failed with error 5? = and I?m given the mountroot prompt. I don?t see the ZFS pool among the = listed GEOM devices on the mountroot prompt. >>>>>=20 >>>>> Adding the committer of r270095 (kib@) to this discussion. >>>>=20 >>>> I have no idea about ZFS, but the fact that things, which are = lower-level >>>> than ZFS filesystem itself are missing, suggests that the issue is = unrelated. >>>> At the very least, start with providing the verbose boot dmesg for = successful >>>> and failed boots. >>>=20 >>> This looks like to be unrelated to ZFS as you?re saying. I just = remembered that I?m loading fuse.ko in my /boot/loader.conf. I commented = that one out and what do you know? The system boots fine with the = r270095 kernel. >>>=20 >>> Very very strange?=20 >> This probably indicates KBI mismatch. >>=20 >> Is your fuse.ko installed from the kernel build ? >> Still, I think the verbose dmesg is the start to look. >>=20 >>>=20 >>> Thanks for the input anyway, >>>=20 >>> -Kimmo >>>=20 >>>=20 >>>=20 >>=20 >>=20 >=20 > Hi it=E2=80=99s me again. Something that was committed in stable/10 = after r271213 up to and including r271288 broke ZFS on Root booting in = exactly the same way again. I know the problem is no longer related to = extra kernel modules loaded in /boot/loader.conf because I=E2=80=99m = loading only the required zfs.ko and opensolaris.ko modules. Also, the = new vt(4) console that I=E2=80=99m using is not the culprit because the = same thing happens with kern.vty set to =E2=80=9Csc=E2=80=9D. >=20 > -Kimmo Verbose dmesg logs from both versions at pastebin if anyone wants to = take a look: r271213 (the one that still works): http://pastebin.com/A6rAkDpB r271288 (the one where booting from ZFS is now broken): = http://pastebin.com/sq6FnVr0 Both are with the sc console, the new vt console doesn=E2=80=99t quite = work at mountroot console for me. Running diff(1) on those I didn=E2=80=99t spot any obvious differences = except in some memory addresses that is to be expected. -Kimmo