From owner-freebsd-current@freebsd.org Mon Feb 12 22:25:45 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A73EF22F5B for ; Mon, 12 Feb 2018 22:25:45 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io0-x22a.google.com (mail-io0-x22a.google.com [IPv6:2607:f8b0:4001:c06::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9F75371C9E for ; Mon, 12 Feb 2018 22:25:44 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io0-x22a.google.com with SMTP id 72so19087549iom.10 for ; Mon, 12 Feb 2018 14:25:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=OCjxLaknND19iV4aWf/kzbMniI5lEGN+ui2BvsR6ZTo=; b=tau21zr7HfH9rz5nC+IWFTntM5zkG65IFBZjVmYITZf9erxz1SoQ/E3GnUN8GHp2YS 7vzCvUZqwOTkZ5/6eDHzyV8jYo3zzE6JcjwMVNOsijc2ZiemMrjzVqzmYt1sBIsgRVkE Gjon6BiRk1+hDo4+I1IB0iDZck/7E/uBr1ZpnvJNK92Bh1+yWSEeeYIWm3vHp7/vjfui J+S4802M403mdWi3Q/6vQbDNx5IAnfWfCF3sdSt1S7848dHa1kExHhbmIhiQItdQZn7n iU4Rbl2Rxs+cvSIwyqbbtD3u5QPcswy6XsZnMfm2KIptZNd4B9D9MCTQBnueaLzsK/6J XwEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=OCjxLaknND19iV4aWf/kzbMniI5lEGN+ui2BvsR6ZTo=; b=gOp9NF2Rg1RZBMGLpiA5mN0jH2uQrxJOlFl4oEWh2vp4ytaNicRYEwixZtYT5lDA8+ 9eyNFC+6PM8NpmkIGHIZnCQ6w4FNYejPTBMsVAnnu1x/OFQsFOSUsME+V5UtGaRU6idx NhZpq+k86JFi8oIU648NT2K4JUxe14yMHoW6V+owLSoj4uHD0hD3orNO1wzi5gRhe0qx OkmbbQaQE9362AyOdGNt2/pZY8q+aOneTG6dilQMDhde6KBIGBezv31NIkIbjRFcw89G jn/eGFfof0R/E6kBQnFD+bi8SacxFvFahXeHj4/3sPdgH12uhK2sZ6CBBn6NBGS4NmDR EYfw== X-Gm-Message-State: APf1xPC/LBFLLImOmSJ26pwwmsOx29n7bO75jcChB7VaQ4jxx8qWClcR BpqhzTekRwxMsLoQpFGac8+OZEyZddxjTzXq+6kWVA== X-Google-Smtp-Source: AH8x226BfOmSY8tIFr6QB8txAC3Q+gQOTuULnxdjT13ISU17RXQCRuATVOntK99R0Z4avcoImDfn4DevsLNDW/zi4TU= X-Received: by 10.107.88.12 with SMTP id m12mr13763694iob.136.1518474343568; Mon, 12 Feb 2018 14:25:43 -0800 (PST) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.201.67 with HTTP; Mon, 12 Feb 2018 14:25:42 -0800 (PST) X-Originating-IP: [50.253.99.174] In-Reply-To: <27682255.x3aPd9zKdx@ralph.baldwin.cx> References: <2333272.J6kp8cXjfh@ralph.baldwin.cx> <27682255.x3aPd9zKdx@ralph.baldwin.cx> From: Warner Losh Date: Mon, 12 Feb 2018 15:25:42 -0700 X-Google-Sender-Auth: N5gTaZknxEsubZX3KBCwYbXaVk4 Message-ID: Subject: Re: LUA boot loader coming very soon To: John Baldwin Cc: FreeBSD Current Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Feb 2018 22:25:45 -0000 On Mon, Feb 12, 2018 at 2:53 PM, John Baldwin wrote: > On Monday, February 12, 2018 02:31:46 PM Warner Losh wrote: > > On Mon, Feb 12, 2018 at 10:12 AM, John Baldwin wrote: > > > > > On Monday, February 12, 2018 08:27:27 AM Warner Losh wrote: > > > > Greetings, > > > > > > > > As you may know, the Lua (http://www.lua.org) boot loader has been > in > > > the > > > > works for some time. It started out life as a GSoC in 2014 by Pedro > Souza > > > > mentored by Wojciech A. Koszek. Rui Paulo created a svn project > branch to > > > > try to integrate it. I rebased that effort into a github branch which > > > Pedro > > > > Arthur fixed up. Over the past year, I've been cleaning up the boot > > > loader > > > > for other reasons, and found the time was ripe to start integrating > this > > > > into the tree. However, those integration efforts have taken a while > as > > > my > > > > day-job work on the boot loader took priority. In the mean time, Ed > Maste > > > > and the FreeBSD Foundation funded Zakary Nafziger to enhance the > original > > > > GSoC Lua scripts to bring it closer to parity with the evolution of > the > > > > FORTH menu system since the GSoC project started. > > > > > > > > I'm pleased to announce that all these threads of development have > > > > converged and I'll be pushing the FreeBSD Lua Loader later today. > This > > > > loader uses Lua as its scripting language instead of FORTH. While > > > > co-existance is planned, the timeline for it is looking to be a few > weeks > > > > and I didn't want to delay pushing this into the tree for that. > > > > > > > > To try the loader, you'll need to build WITHOUT_FORTH=yes and > > > > WITH_LOADER_LUA=yes. Fortunately, you needn't do a full world to do > this, > > > > you can do it in src/stand and install the result (be sure to have > the > > > > options for both the build and the install). This will replace your > > > current > > > > /boot/loader that is scripted with FORTH to one that's scripted with > Lua. > > > > It will install the lua scripts in /boot/lua. The boot is scripted > with > > > > /boot/lua/loader.lua instead of /boot/loader.rc. You are strongly > advised > > > > to create a backup copy of /boot/loader before testing (eg cp > > > /boot/loader > > > > /boot/loader_forth), since you'll need to boot that from boot2 if > > > something > > > > goes wrong. I've tested it extensively, though, with userboot.so and > it's > > > > test program, so all the initial kinks of finding the lua scripts, > etc > > > have > > > > been worked out. > > > > > > > > While it's possible to build all the /boot/loader variants with Lua, > I've > > > > just tested a BIOS booting /boot/loader both with and without menus > > > > enabled. I've not tested any of the other variants and the > instructions > > > for > > > > testing some of them may be rather tedious (especially UEFI, if you > want > > > a > > > > simple path to back out). Since there's not been full convergence > > > testing, > > > > you'll almost certainly find bumps in this system. Also, all the > > > > build-system APIs are likely not yet final. > > > > > > > > I put MFC after a month on the commit. Due to the heroic (dare I say > > > > almost crazy) work of Kyle Evans on merging all the revs from > -current to > > > > 11, I'm planning a MFC to 11 after the co-existence issues are > hammered > > > > out. In 11, FORTH will be the default, and Lua will be built by > default, > > > > but users will have to do something to use it. 12, both FORTH and Lua > > > will > > > > be built and installed, with Lua as default (barring unforeseen > > > > complications). Once the co-existence stuff goes in, I imagine we'll > make > > > > the switch to Lua by default shortly after that. In 13, FORTH will be > > > > removed unless there's a really really compelling case made to keep > it. > > > > > > > > So please give it a spin and give me any feedback, documentation > updates > > > > and/or bug fixes. I'm especially interested in reviews from people > that > > > > have embedded Lua in other projects or experts in Lua that can > improve > > > the > > > > robustness of the menu code. > > > > > > Do you have some memory usage numbers for LUA vs forth for the > different > > > BIOS loaders (text/data/bss sizes)? For the EFI case we probably have > lots > > > of room, but for the non-EFI case we are constrained to 0xa0000 - > 0xa000 > > > for the text/data/bss and stack (in some cases like PXE booting the top > > > can be lower than 0xa0000). I'm not sure if we have any other > platforms > > > with similar memory constraints. > > > > > > > Lua is about 60-70k larger than FORTH for this application: > > > > Forth based: > > -r-xr-xr-x 1 root wheel 385024 Feb 12 14:14 > > /home/imp/roots/amd64/boot/loader > > -r-xr-xr-x 1 root wheel 450560 Feb 12 14:14 > > /home/imp/roots/amd64/boot/zfsloader > > Lua based: > > -r-xr-xr-x 1 root wheel 450560 Feb 12 14:20 > > /home/imp/roots/amd64/boot/loader > > -r-xr-xr-x 1 root wheel 520192 Feb 12 14:20 > > /home/imp/roots/amd64/boot/zfsloader > > > > So with Lua: > > % size loader.bin > > text data bss dec hex filename > > 411840 22264 48304 482408 0x75c68 loader.bin > > % size zfsloader.bin > > text data bss dec hex filename > > 478712 22968 52368 554048 0x87440 zfsloader.bin > > > > so we're at 555k out of 640k before we look at heap usage. Of course, > these > > numbers are only going to get a lot worse as we pile-in new crypto > > features... I know that the normal loader works, but I've not tried the > > zfsloader. > > Woof, on stable/11 the forth ones are quite a bit smaller: > > -r-xr-xr-x 1 root wheel 315392 Dec 16 13:39 /boot/loader > -r-xr-xr-x 1 root wheel 356352 Dec 16 13:39 /boot/zfsloader > > % size /usr/obj/usr/src/sys/boot/i386/loader/loader.bin > text data bss dec hex filename > 274913 22624 46560 344097 0x54021 /usr/obj/usr/src/sys/boot/ > i386/loader/loader.bin > % size /usr/obj/usr/src/sys/boot/i386/zfsloader/zfsloader.bin > text data bss dec hex filename > 313865 23184 50624 387673 0x5ea59 /usr/obj/usr/src/sys/boot/ > i386/zfsloader/zfsloader.bin > > Note that it's not 640k, it's 0xa0000 (640k) - the start address of the > loader: 0xa000 (40k), so text/data/bss + stack has 600k of room. The heap > isn't in low memory anymore, so we don't have to worry about that. 555k > for > ZFS loader leaves around 45k for stack. If the ZFS loader code is anything > like the kernel code then ZFS probably uses deeper stacks than UFS. :( > Yea. We support more FS types than in 11 now I think (at least more than in 10), so we might want to look to trimming that. I don't think we need 3 different compressed filesystem formats compiled in by default, for example... > One of the larger uses of stack in the loader is probably gzipfs, so > loading > a mfsroot.gz from the loader might be a good test. Unfortunately, there > isn't > a good way to really detect stack overflow other than silent corruption in > bss. If we could create a 'volatile int canary' in the bss and force it > to be > the last thing in the bss we could check the value of 'canary' in the > loader's > main loop perhaps and complain if it is ever non-zero. > That's not a terrible idea... Assuming that the corruption is such that it's right at the end of bss, and not something a bit before it... Warner