From owner-freebsd-arm@freebsd.org Fri Dec 1 21:55:59 2017 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B0829E6C22F for ; Fri, 1 Dec 2017 21:55:59 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-wr0-x230.google.com (mail-wr0-x230.google.com [IPv6:2a00:1450:400c:c0c::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3E8FB72B50 for ; Fri, 1 Dec 2017 21:55:59 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-wr0-x230.google.com with SMTP id o2so11505464wro.5 for ; Fri, 01 Dec 2017 13:55:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=nJOil9JJd2wzxv0PujzibHIHszsTmNoZi0+vgYTIkNI=; b=iRzxmD6TFpJGZWstvKPfw7o/3DXLnHbRsZ7C6entpQpqcmr4GV4HGoPqwRhDzBC9O5 1xdTnogad0SKni4yJFe5L8fHgHrLLrYq/vtJfKmihlrH6QzCwL0ADU9sjCxAbTBa9PmQ dAqp5bmKPjWCLZWmu333zQaYD8K0laGjQdTBgNXw7RyvLfUQmMv7dqLLOiv8OZQ6/hex wS9/Eersxns2j8AfRtlUXQM0Lp4drKMbGgDh5OnEUqLhKCV1ANaaw1cZvfOvKD35syQb 1kXpTRLrHxF2B5xXA46T+tilKSor8tWZ1vQ8gdMfy36/CAHHCr5rRhbFqivUtPL5GXoC hURg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=nJOil9JJd2wzxv0PujzibHIHszsTmNoZi0+vgYTIkNI=; b=ftTSWjqxwbIoMN1s5VdqVZPcR4ggO93SC0Sjel81EOg+DETktwC0cZ0gT1YQRjG3lm RCUEFJdJlYfCGZ1QqwTJuyqPF72mLGjW2U2WIo3UWbBCJ5M9togfIG+Oh63fGygo390w 1g6Qde+NC1UllGhXB82kYY8ID8VpRd0IN0N77afewigxRpPnW3c1tsXdhE2psYSi9wAH bUCkZEd5YUrX79qIu+k2B6qArzsMD4bQv7lsH1/44UPOr6kxFyi34CcQQktbaxeGIHhl 2mZ+zfRIFKnomNU4ZMA0gJcXcsmGhLQ8LVIB9nQR0BgxNT1p6Fi/+qgKNBh9Wi7tqhPi BDQA== X-Gm-Message-State: AJaThX7ZgeUCinBjkqxHg+DzNTUNJa8W74QEBXFt/85yYXEiE+2oLVnU jTnaG2rROtJXOz5+M0I909MGqw== X-Google-Smtp-Source: AGs4zMaeFPtHlWM8CsbCG5O8VgGKuoaIoAo6NGU2RkkhrE77NOsmN6WPTzLFf1fWbc/4xruckP6kXQ== X-Received: by 10.223.202.1 with SMTP id o1mr2132835wrh.233.1512165357151; Fri, 01 Dec 2017 13:55:57 -0800 (PST) Received: from mutt-hbsd ([65.19.167.130]) by smtp.gmail.com with ESMTPSA id h7sm6844047wrb.35.2017.12.01.13.55.51 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 01 Dec 2017 13:55:56 -0800 (PST) Date: Fri, 1 Dec 2017 16:55:31 -0500 From: Shawn Webb To: Warner Losh Cc: "freebsd-arm@freebsd.org" , FreeBSD Current Subject: Re: Booting UEFI ZFS is broken on arm64 Message-ID: <20171201215531.u4f2c4eno7z5irxj@mutt-hbsd> References: <20171130002135.q7p27hh6qkog4slr@mutt-hbsd> <20171130003458.bnltotlgfdbke5ue@mutt-hbsd> <20171130004358.usia2jrvman4cvvz@mutt-hbsd> <20171201214955.zopa6v2f2mx3rkt2@mutt-hbsd> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="yxqbpbqi3jogdpsy" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD mutt-hbsd 12.0-CURRENT FreeBSD 12.0-CURRENT X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: NeoMutt/20171027 X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 21:55:59 -0000 --yxqbpbqi3jogdpsy Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Dec 01, 2017 at 02:53:53PM -0700, Warner Losh wrote: > On Fri, Dec 1, 2017 at 2:49 PM, Shawn Webb > wrote: >=20 > > On Wed, Nov 29, 2017 at 07:31:17PM -0700, Warner Losh wrote: > > > On Wed, Nov 29, 2017 at 5:54 PM, Warner Losh wrote: > > > > > > > > > > > > > > > On Wed, Nov 29, 2017 at 5:43 PM, Shawn Webb < > > shawn.webb@hardenedbsd.org> > > > > wrote: > > > > > > > >> On Wed, Nov 29, 2017 at 05:42:52PM -0700, Warner Losh wrote: > > > >> > On Wed, Nov 29, 2017 at 5:34 PM, Shawn Webb < > > shawn.webb@hardenedbsd.org > > > >> > > > > >> > wrote: > > > >> > > > > >> > > On Wed, Nov 29, 2017 at 05:33:46PM -0700, Warner Losh wrote: > > > >> > > > On Wed, Nov 29, 2017 at 5:21 PM, Shawn Webb < > > > >> shawn.webb@hardenedbsd.org> > > > >> > > > wrote: > > > >> > > > > > > >> > > > > It appears that in the latest FreeBSD 12-CURRENT/arm64 > > snapshot, > > > >> > > > > booting UEFI GPT ZFS on my OverDrive 1000 is broken. It bo= ots > > up > > > >> to > > > >> > > > > this line: > > > >> > > > > > > > >> > > > > Using DTB provided by EFI at 0x801fe00000. > > > >> > > > > > > >> > > > > > > >> > > > Which snapshot is that? Boot1 was broken until recently. > > > >> > > > > > >> > > FreeBSD-12.0-CURRENT-arm64-aarch64-20171121-r326056-memstick.i= mg > > > >> > > > > > >> > > It also happens on latest HEAD, so it would appear to still be > > broken. > > > >> > > > > >> > > > > >> > Is this boot1.efi producing the output, or loader.efi? I'm guess= ing > > the > > > >> > latter, but wanted to make sure. If so, then we're past the point > > where > > > >> > boot1.efi would have failed (besides, it was fixed before that > > > >> snapshot). > > > >> > > > >> With DEBUG turned on for stand/fdt: > > > >> > > > >> Booting [/boot/kernel/kernel]... > > > >> fdt_copy(): fdt_copy va 0x01208000 > > > >> fdt_setup_fdtp(): fdt_setup_fdtp() > > > >> fdt_load_dtb_addr(): fdt_load_dtb_addr(0x801fe00000) > > > >> Using DTB provided by EFI at 0x801fe00000. > > > >> Loaded the platform dtb: 0x81f56f1630. > > > >> fdt_fixup(): fdt_fixup() > > > >> > > > >> ^ hangs after that message > > > > > > > > > > > > That doesn't sound like anything I've changed, but it could well be= =2E.. > > I > > > > think to find this breakage, you may need to bisect backwards along > > stand / > > > > sys/boot until we find the spot where it broke. > > > > > > > > > > There's been several conversations on IRC about how others are hittin= g a > > > scheduler bug, at least on x86. hps' fix seems to do the trick for th= eir > > > issues. > > > > > > Author: hselasky > > > Date: Wed Nov 29 23:28:40 2017 +0000 > > > > > > The sched_add() function is not only used when the thread is > > initially > > > started, but also by the turnstiles to mark a thread as runnable = for > > > all locks, for instance sleepqueues do: > > > setrunnable()->sched_wakeup()->sched_add() > > > > > > In r326218 code was added to allow booting from non-zero CPU numb= ers > > > by setting the ts_cpu field inside the ULE scheduler's sched_add() > > > function. This had an undesired side-effect that prior sched_pin() > > and > > > sched_bind() calls got disregarded. This patch fixes the > > > initialization of the ts_cpu field for the ULE scheduler to only > > > happen once when the initial thread is constructed during system > > > init. Forking will then later on ensure that a valid ts_cpu value > > gets > > > copied to all children. > > > > > > Reviewed by: jhb, kib > > > Discussed with: nwhitehorn > > > MFC after: 1 month > > > Differential revision: https://reviews.freebsd.org/D13298 > > > Sponsored by: Mellanox Technologies > > > > > > > > > git-svn-id: svn+ssh://svn.freebsd.org/base/head@326376 > > > ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f > > > > > > is the fix.... But the bug it fixes post-dates the snapshot, so maybe > > this > > > isn't the same thing... > > > > Definitely is not the same thing. I've so far got it printf'd to where > > the uefi loader jumps into the kernel's entry point. So the loader > > itself might be fine. Something in the kernel, then, is going funky. > > > > Booting in verbose mode does not provide any additional input. > > > > Here's the output I get (some of the output is from printf's I've > > done): > > > > FreeBSD/arm64 EFI loader, Revision 1.1 > > (Wed Nov 29 21:51:14 EST 2017 shawn@hbsd-dev-laptop) > > EFI boot environment > > Loading /boot/defaults/loader.conf > > /boot/kernel/kernel text=3D0x7e0a78 data=3D0xaad80+0x443f62 > > syms=3D[0x8+0x10ec78+0x8+0x1021d4] > > /boot/entropy size=3D0x1000 > > /boot/kernel/zfs.ko text=3D0x99070 text=3D0x130390 data=3D0x21ff8+0x9ef= 98 > > syms=3D[0x8+0x22c68+0x8+0x1b99b] > > /boot/kernel/opensolaris.ko text=3D0x1330 text=3D0xd00 data=3D0x10160+0= x125d0 > > syms=3D[0x8+0xff0+0x8+0x8d8] > > > > Hit [Enter] to boot immediately, or any other key for command prompt. > > Booting [/boot/kernel/kernel]... > > Using DTB provided by EFI at 0x801fe00000. > > fdt_copy returned. dtb_size is 9060. > > bi_load finished. err: 0 > > dev_cleanup finished > > About to call into the entry point at 0x81ee601000 > > >=20 > You might try booting the same kernel off a small UFS partition. There's a > tiny chance that the loader didn't load it right, but more likely the > kernel is borked. Maybe DTB issues? Maybe something else... A quick test > like that would remove ZFS from the equation, even if it's just a USB > stick... UFS works fine and dandy. It's ZFS that's b0rked. Thanks, --=20 Shawn Webb Cofounder and Security Engineer HardenedBSD GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --yxqbpbqi3jogdpsy Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEKrq2ve9q9Ia+iT2eaoRlj1JFbu4FAlohz9IACgkQaoRlj1JF bu4oXw/9EOPJzkBWeiYIsnLFVg05wSQVIIuMyhYE/oEmnY0lK5H+muPK6t9ETqM+ AVEGxjEpmgeOCkzt3tOOrC5jul/PSR/XGpmVi68ho5KEbuOFD+UPjtN50cZJraSD Uusjl8jtFWBDAybQ78O43XIrr2710NQ0PTbeZYJouQnTvLz+MkVhMgKF37YglZOK 4CLqEMzEqwNIxj/tKRvJt+G6NB12xpN3mp8w9Svq/G10v7GjxZC7ikqsd5FLH6iP hWpyqqRtoucbCpn66Ahc98c32EHpMvwx2XffsddWb8g1Z/dSjxAD1j7mBLLHBVbV 1e/CB5S7Fyexei66/UZ47q0ZIqTtVGwhYE1LqMVdvLUimhtd4t06nWJeRGUDrr0C NZuLRBOUiY8suaI5oYizyg84m4hjfCoEK27Hye5aYizF6gAh/eodllfm7YHde+oT Nos1YTb2/axrS60ibUdhom3a1gD1TvUqVVkrdRCf1+ds0RfIgAHiAAsZ9qogEoIP PBeumYz6+4Waq44VUjD1QKoEy9AG6yj0plLp/n/Dt6AoViM0nOT4puYMuhbgps+i RjBIjn8JiToBYsNppn3Gv7GTn7vwEiqU2OyfFs4CH8SZ6MIvhhLOrQ16B8BuBEGm 7erXmEi8ZtfOZHtJ6MZlJTXj48OYP1TwUuvVwuvEmPAO9LF4Mqk= =+uXh -----END PGP SIGNATURE----- --yxqbpbqi3jogdpsy--