From owner-freebsd-arm@freebsd.org Fri Dec 1 21:53:55 2017 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 66B38E6C0F4 for ; Fri, 1 Dec 2017 21:53:55 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io0-x236.google.com (mail-io0-x236.google.com [IPv6:2607:f8b0:4001:c06::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3259772986 for ; Fri, 1 Dec 2017 21:53:55 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io0-x236.google.com with SMTP id d16so12794025iob.4 for ; Fri, 01 Dec 2017 13:53:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=v0J7ha1Llo4zf5t0skenV3NOwG8sqBvKamPXAMxqDTY=; b=Bk4fjPgrxHanYjQNZUZIUfucOWzIpzlM+CSUes2s/zfsZ7G5UELlnkZ7YSZFQI34mr Slc9dA0JoHOBSF5sagetp6VQrnKgC9EXTYLKdWe4SCuaUJT2fxQuOSM/uet5kGSaVI1g HbVzP2y+BSuGiaEpz+QSYmRXusLPjUehTmA5oXPBFKUCBLfh7QYgrMoHRd4GLnjNv/5S h9Xkd1QFJb0+OSr3nVWE5qPob9t9d6n98A6q1NMxwSr082VBe/tRjYhzpb72iBDoDyEh 2HEIiwVnzgIuKHdJTF+Z4dFq/Uihi5FfNMMRy9WeW6SdsnlGMHDZP1avTwOxtw5Aht5B XOWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=v0J7ha1Llo4zf5t0skenV3NOwG8sqBvKamPXAMxqDTY=; b=ToU326eZxhYoreCFWFoniGm1EPpV68J2NGDxzfKaG0eQuQis5A01QZYGqYDIeKJiff jid9z43PMIu7nwJ2R4MBMeSqqQD2IFQQ0HU1D+4BL46TpEltWI30mi4lR7uRPMU5vmZJ sosB7TjzzLgZGDsN6hG4j7Q72NWmO9oL7Q/DEotXNyCkKhRflYLZPB7PM55q3GDjvjS/ pEYTAMmZcY+Oqu3GG7/O5LckuIsz7YSu2Fe9bPwV20yejik1WFU+YQLQqeZD6r61TflB vZT6E98o0B3wQRoY2Jdck2RfpmqypsQ9xT1hxVrjX4BHXvce9PTaFe17QyIo/kM2L1+6 JSdw== X-Gm-Message-State: AKGB3mKbufgBRcaZRbNkwiIKg44IkuBjX6gv971sCLDU0qBkeP0tJFwi aJv+yIBJtRX1G94Tz4C0RejUEa64JrRO3Vs1mga5ng== X-Google-Smtp-Source: AGs4zMYShFlbLx5OCs+EKA2fCcAUKtFC8BpnU3TvWA6BmZJL7HVd3gqdhcnuU0Pw5n6fSQ9AjrgvAsQIRA7rdBeqgz8= X-Received: by 10.107.52.140 with SMTP id b134mr1715801ioa.291.1512165234381; Fri, 01 Dec 2017 13:53:54 -0800 (PST) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.108.204 with HTTP; Fri, 1 Dec 2017 13:53:53 -0800 (PST) X-Originating-IP: [2603:300b:6:5100:1052:acc7:f9de:2b6d] In-Reply-To: <20171201214955.zopa6v2f2mx3rkt2@mutt-hbsd> References: <20171130002135.q7p27hh6qkog4slr@mutt-hbsd> <20171130003458.bnltotlgfdbke5ue@mutt-hbsd> <20171130004358.usia2jrvman4cvvz@mutt-hbsd> <20171201214955.zopa6v2f2mx3rkt2@mutt-hbsd> From: Warner Losh Date: Fri, 1 Dec 2017 14:53:53 -0700 X-Google-Sender-Auth: GnENBzzMvPJUa1FIQuWQfhGN_FQ Message-ID: Subject: Re: Booting UEFI ZFS is broken on arm64 To: Shawn Webb Cc: "freebsd-arm@freebsd.org" , FreeBSD Current Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 21:53:55 -0000 On Fri, Dec 1, 2017 at 2:49 PM, Shawn Webb wrote: > On Wed, Nov 29, 2017 at 07:31:17PM -0700, Warner Losh wrote: > > On Wed, Nov 29, 2017 at 5:54 PM, Warner Losh wrote: > > > > > > > > > > > On Wed, Nov 29, 2017 at 5:43 PM, Shawn Webb < > shawn.webb@hardenedbsd.org> > > > wrote: > > > > > >> On Wed, Nov 29, 2017 at 05:42:52PM -0700, Warner Losh wrote: > > >> > On Wed, Nov 29, 2017 at 5:34 PM, Shawn Webb < > shawn.webb@hardenedbsd.org > > >> > > > >> > wrote: > > >> > > > >> > > On Wed, Nov 29, 2017 at 05:33:46PM -0700, Warner Losh wrote: > > >> > > > On Wed, Nov 29, 2017 at 5:21 PM, Shawn Webb < > > >> shawn.webb@hardenedbsd.org> > > >> > > > wrote: > > >> > > > > > >> > > > > It appears that in the latest FreeBSD 12-CURRENT/arm64 > snapshot, > > >> > > > > booting UEFI GPT ZFS on my OverDrive 1000 is broken. It boots > up > > >> to > > >> > > > > this line: > > >> > > > > > > >> > > > > Using DTB provided by EFI at 0x801fe00000. > > >> > > > > > >> > > > > > >> > > > Which snapshot is that? Boot1 was broken until recently. > > >> > > > > >> > > FreeBSD-12.0-CURRENT-arm64-aarch64-20171121-r326056-memstick.img > > >> > > > > >> > > It also happens on latest HEAD, so it would appear to still be > broken. > > >> > > > >> > > > >> > Is this boot1.efi producing the output, or loader.efi? I'm guessing > the > > >> > latter, but wanted to make sure. If so, then we're past the point > where > > >> > boot1.efi would have failed (besides, it was fixed before that > > >> snapshot). > > >> > > >> With DEBUG turned on for stand/fdt: > > >> > > >> Booting [/boot/kernel/kernel]... > > >> fdt_copy(): fdt_copy va 0x01208000 > > >> fdt_setup_fdtp(): fdt_setup_fdtp() > > >> fdt_load_dtb_addr(): fdt_load_dtb_addr(0x801fe00000) > > >> Using DTB provided by EFI at 0x801fe00000. > > >> Loaded the platform dtb: 0x81f56f1630. > > >> fdt_fixup(): fdt_fixup() > > >> > > >> ^ hangs after that message > > > > > > > > > That doesn't sound like anything I've changed, but it could well be... > I > > > think to find this breakage, you may need to bisect backwards along > stand / > > > sys/boot until we find the spot where it broke. > > > > > > > There's been several conversations on IRC about how others are hitting a > > scheduler bug, at least on x86. hps' fix seems to do the trick for their > > issues. > > > > Author: hselasky > > Date: Wed Nov 29 23:28:40 2017 +0000 > > > > The sched_add() function is not only used when the thread is > initially > > started, but also by the turnstiles to mark a thread as runnable for > > all locks, for instance sleepqueues do: > > setrunnable()->sched_wakeup()->sched_add() > > > > In r326218 code was added to allow booting from non-zero CPU numbers > > by setting the ts_cpu field inside the ULE scheduler's sched_add() > > function. This had an undesired side-effect that prior sched_pin() > and > > sched_bind() calls got disregarded. This patch fixes the > > initialization of the ts_cpu field for the ULE scheduler to only > > happen once when the initial thread is constructed during system > > init. Forking will then later on ensure that a valid ts_cpu value > gets > > copied to all children. > > > > Reviewed by: jhb, kib > > Discussed with: nwhitehorn > > MFC after: 1 month > > Differential revision: https://reviews.freebsd.org/D13298 > > Sponsored by: Mellanox Technologies > > > > > > git-svn-id: svn+ssh://svn.freebsd.org/base/head@326376 > > ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f > > > > is the fix.... But the bug it fixes post-dates the snapshot, so maybe > this > > isn't the same thing... > > Definitely is not the same thing. I've so far got it printf'd to where > the uefi loader jumps into the kernel's entry point. So the loader > itself might be fine. Something in the kernel, then, is going funky. > > Booting in verbose mode does not provide any additional input. > > Here's the output I get (some of the output is from printf's I've > done): > > FreeBSD/arm64 EFI loader, Revision 1.1 > (Wed Nov 29 21:51:14 EST 2017 shawn@hbsd-dev-laptop) > EFI boot environment > Loading /boot/defaults/loader.conf > /boot/kernel/kernel text=0x7e0a78 data=0xaad80+0x443f62 > syms=[0x8+0x10ec78+0x8+0x1021d4] > /boot/entropy size=0x1000 > /boot/kernel/zfs.ko text=0x99070 text=0x130390 data=0x21ff8+0x9ef98 > syms=[0x8+0x22c68+0x8+0x1b99b] > /boot/kernel/opensolaris.ko text=0x1330 text=0xd00 data=0x10160+0x125d0 > syms=[0x8+0xff0+0x8+0x8d8] > > Hit [Enter] to boot immediately, or any other key for command prompt. > Booting [/boot/kernel/kernel]... > Using DTB provided by EFI at 0x801fe00000. > fdt_copy returned. dtb_size is 9060. > bi_load finished. err: 0 > dev_cleanup finished > About to call into the entry point at 0x81ee601000 > You might try booting the same kernel off a small UFS partition. There's a tiny chance that the loader didn't load it right, but more likely the kernel is borked. Maybe DTB issues? Maybe something else... A quick test like that would remove ZFS from the equation, even if it's just a USB stick... Warner