Date: Sat, 6 Nov 2010 13:33:04 -0700 From: Devin Teske <dteske@vicor.com> To: Nathan Whitehorn <nwhitehorn@freebsd.org> Cc: jpaetzel@freebsd.org, freebsd-hackers@freebsd.org, Devin Teske <dteske@vicor.com>, Garrett Cooper <gcooper@FreeBSD.org> Subject: Re: txt-sysinstall scrapped Message-ID: <7D12F133-FBFD-4664-A5B2-D8DC6D189F9B@vicor.com> In-Reply-To: <4CD58136.6070509@freebsd.org> References: <201011052316.27839.jpaetzel@freebsd.org> <AANLkTi=62rRhZsN4wUi6p_yokSxG0tkjUHK7gosLtTRZ@mail.gmail.com> <20101105.230617.74669306.imp@bsdimp.com> <AANLkTi=G2UEj4P=h=B7Tr58vg7RC9McMZq-q73ArDWOZ@mail.gmail.com> <4CD58136.6070509@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Nov 6, 2010, at 9:24 AM, Nathan Whitehorn wrote: > On 11/06/10 01:04, Garrett Cooper wrote: >> On Fri, Nov 5, 2010 at 10:06 PM, Warner Losh <imp@bsdimp.com> wrote: >>>> Just to add to that (because I do find it a novel idea), 1) how >>>> are you going to properly prevent man in the middle attacks (SSL, = TLS, >>>> etc?), and 2) what webserver would you use? >>>=20 >>> https or ssh. >>>=20 >>> We're also toying with the idea of having a partition that you could >>> 'dd' your certs and keys to (so any system can customize the image >>> with keys to make sure you were talking to who you think you are). >>> We'd just reserve 1MB of space on partition s3. We'd then check to >>> see if there was a tar ball. If so, we'd extract it and do the >>> intelligent thing with the keys we find there. >>=20 >> Wouldn't it be better just to go with a read-write media solution >> (USB) like Matt Dillon was suggesting at today then? Then again, >> determining the root device to date is still a bit kludgy isn't it? >=20 > But this breaks badly for people who don't own USB sticks of = sufficient > size, are installing on machines without USB ports, can't boot from = USB, > want to install from a shared medium like PXE, are installing on = blades > with convenient shared CD drives but not USB etc. etc. Everything in = the > world can boot from CD, and we have to ensure that continues working. >=20 > I also have mixed feelings about needing to use a web browser to > instruct a web app inside a bundled web server to write a config file = to > be interpreted by shell scripts just in order to run gpart, newfs, and > tar. But if you get it working, it's better than sysinstall no matter > how baroque. > -Nathan I find myself identifying a lot with this sentiment. We have FreeBSD = running on 1,000+ systems (modestly) within our organization. We've = found that some older BIOSes don't boot from USB sticks so-well. It's = also hit-or-miss in varied combinations of old-BIOS and off-brand sticks = -- had great luck with those super-tiny PNY's that rotate-out, though = again, old BIOSes say pre-2003 P4 or AMD K6, have issues). And since we're shipping at a rate of at least 100+ servers a year that = are now coming without any optical drives... our solution has been to = purchase one of these for each/every field engineer: = http://www.lg.com/us/computer-products/optical-media/LG-external-dvd-burne= r-GP08LU30.jsp This great little device has worked wonders for over 30 field engineers = (modestly). Out of thousands upon thousands of installs with the thing, = we've really only had a problem with about a half-dozen individual PCs = -- but after a BIOS upgrade, they tended to work else we slapped = internal ATAPI CD-ROM's in there and they worked). Though, to-date what we've really found great-success in, is employing = the latest-and-greatest ISOLINUX boot-loader w/ their `isohybrid' ISO = post-processing program which encapsulates the disk-image contents = within a hardware-emulation structure (you can make your ISO -- when = written to USB stick -- appear to the BIOS as a ZIP disk, Hard Disk, = etc.). We currently tell all of our field engineers to download a single ISO = that is built with our custom build-process (which employs ISOLINUX and = `isohybrid' from http://syslinux.zytor.com/). This ISO can then = optionally -- at the field engineer's discretion -- either burn that ISO = to optical media and use the LG AC-Adapterless USB optical drive (linked = to above), OR ... they can use dd(1) to write the ISO directly to a USB = stick (though again... some older BIOSes prefer the optical over flash = media). This was, of course, quite a challenge to achieve. Here's a brief (lol -- it started that way, you got to believe me) list = of what we did to accomplish installs with one ISO on two mediums: 1. Entire swaths of sysinstall(8) were patched Out of the 48 source files that make up sysinstall (that I count = currently within RELENG_8), we've thus far patched 28 of them and added = 1 new file. We even added a new feature ... ability to install from a directory = rather than a device. In install.cfg, we have something akin to: nullfs=3D/path/to/freebsd/release mediaSetNullFS As implied by the name, a nullfs is done from the directory specified to = /dist -- where sysinstall(8) expects to find the mounted distribution, = regardless of what media you select. 2. One tiny kernel patch There's a feature in the kernel for forcing the boot-medium into = read-write mode (which is not the default... the default is to mount = read-only and switch it during boot-up). When we're mounting an mfsroot, = especially one that's been customized to run scripts which build the = `install.cfg' script to hand-off to sysinstall(8), you _want_ to mount = read-write out of the gate. This feature is enabled by setting the environment variable = vfs.root.mountfrom.options to "rw" in the loader's Forth name-space (in = loader.rc for example). Well, this was broken. : ( So we patched it : ) 3. One tiny patch to the release(7) Makefile Since we rely on the release(7) process to compile both our customized = sysinstall(8) and also our custom mfsroot (explained below), it became a = real problem that WORLD_FLAGS was not being passed to installworld from = within `/usr/obj'. For example, when passing in WORLD_FLAGS=3D"-DWITHOUT_OPENSSL" (see = src.conf(5)) we kept bombing out on the installworld phase which = installs your newly compiled object-files from /usr/obj to an = installbase of /usr/release (we pass CHROOTDIR=3D/usr/release to `make = release'). This was because installworld was descending into a directory like = gssapi which it shouldn't, had WORLD_FLAGS been properly replicated down = the chain to installworld. We fixed this with a one-liner patch to /usr/src/release/Makefile 4. A custom dialog(3) based menu system for selecting pre-scripted = installs of many different flavors 5. A custom highly customized mfsroot We start by patching /usr/src/release/i386/boot_crunch.conf (heavily). = We add the following utilities: cat(1), chflags(1), chmod(1), cp(1), kill(1), ln(1), mkdir(1), mv(1), = rmdir(1), sleep(1), bsdlabel(8) (aka disklabel), init(8), ldconfig(8), = mount(8), mount_cd9660(8), reboot(8) (aka halt), at(1) (aka atq, atrm, = batch), awk(1), bzip2(1) (aka bunzip2, bzcat), ee(1), passwd(1), = printf(1), tar(1), tail(1), uniq(1), chown(8) (aka chgrp), moused(8), = mtree(8), pw(8), pwd_mkdb(8), sendmail(8) (aka newaliases, mailq), = tzsetup(8), vidcontrol(1), dialog(1), grep(1), sort(1), and crontab(1) NOTE: Yes, all those fit into the same amount of disk-space currently = allocated for the mfsroot, so no patching was necessary to give us a = larger mfsroot. First, you might be asking yourself... OMG, why so many utilities added? With the exception of dialog(1), mount_cd9660(8), and sleep(1), ALL of = these utilities were added because they are called directly by = sysinstall(8). We wanted to do something absolutely crazy... we wanted to install = FreeBSD-8.1 amd64 while booted under FreeBSD-8.1 i386. This proved to be = a challenge because once sysinstall(8) is done laying-down the = distribution-sets chose, it then does a chroot(2) into the = newly-installed OS and starts calling all these lovely binaries (most = listed above are called when performing optional setup routines, like = configuring sendmail, setting up the mouse, etc. etc.). Well, by adding all these utilities to the mfsroot and patching = sysinstall(8) to keep /stand around, we can have sysinstall(8) -- when = told to do so -- only call executables out of /stand after the chroot. = This way, all the normal optional setup routines can be performed, but = now with the 32-bit binaries that we trust, rather than the 64-bit = binaries which cause the system to panic and reboot when invoked after = the chroot normally. This heavy patching affords the user to -- once in our custom dialog(3) = based menu system -- choose to install either 64-bit or 32-bit despite = having booted from 32-bit kernel whereas currently sysinstall(8) must be = run in an environment that can execute 64-bit binaries if you want to = install the 64-bit distributions. It also affords us the ability to put legacy OSes on the disk as = optional installs. Because our patched sysinstall will never call a = single binary within the installed base, we're safe from problems = involving incompatible binaries. Still talking about the customized mfsroot... once the `make release' = process is finished and we have our highly customized mfsroot, we then = patch it some more! We add the binary nullfs.ko kernel module -- to support our custom = sysinstall(8) feature of mediaSetNullFS -- which will load the nullfs = kernel module if this VFS type is not already compiled into the kernel = (more on that later -- see description of our custom boot loader below). Then we throw a very tiny binary custom init(8) that chain-loads to = sysinstall(8), but not before simply putting some dots up on the screen = for 4 seconds -- allowing the user to quickly hit Scroll-Lock and peruse = the kernel messages if necessary before sysinstall launches (this was a = real problem in 4.x where the scrollback history was not preserved so = well). We add `/install.cfg' (searched-for and read automatically by = sysinstall(8) on startup) which invokes a script which uses mount_cd9660 = to re-mount the device we booted off of to `/cdrom' That last part is important. Remember that I mentioned that we're using = ISOLINUX's `isohybrid' to post-process the ISO. A very nice effect is = that our post-processed ISO is probed by the cd9660 geom provider and = /dev/iso9660/VOLID appears in devfs when probed, despite the fact that = we booted using zip-disk or hard-disk emulation from the BIOS. This makes the process of re-mounting the media we booted from (from = within the mfsroot) a piece of cake, regardless of whether the field = engineer wrote the ISO to optical media or USB stick. Lastly, before our custom mfsroot stops interrupting sysinstall(8)'s = progress (remember, we invoked this custom shell script via = `/install.cfg' which used mount_cd9660(8) to mount `/dev/iso9660/VOLID' = to `/cdrom'), we invoke `/cdrom/freebsd/menu/menu' which is an ELF = executable based on dialog(3) which presents our custom menu to the user = for them to choose "Which OS?" to install (among which we have both = 32-bit variants AND 64-bit variants of FreeBSD). 6. Because there's always that ONE package you'd like to have = pre-installed along with your OS, we have an elaborate post-install = configuration system. However, we later realized that (as randi once so eloquently voiced) = installing Packages from sysinstall(8) is a bad bad idea. We patched sysinstall(8) to no longer call system executables for = configuration setups, allowing safe install/configuration of a foreign = binary system, though we couldn't patch the package installation process = for one very good reason... We can't predict what +INSTALL, +POST-INSTALL, +DEINSTALL, = +POST-DEINSTALL, and in-truth, even +CONTENTS (via @exec/@unexec) were = going to do. Some developers have even been caught writing these scripts = in perl (which is no longer part of the base distribution). So, for that ONE package that we just _had to have_ pre-installed before = first-boot (for us it's perl), we make use of DIST_LOCAL. sysinstall has = a feature allowing you to create your own custom distribution-set = (local). We ended up throwing a few packages in there. Though to be honest, we felt that perl and kernels deserved special = attention. Our patched sysinstall(8) also has a selector for our own = custom kernel (we have in our release, /usr/release/R/stage/kernels/FIS = which ends up in the ISO at = /freebsd/repos/8.1-RELEASE{,-amd64}/kernels/fis.?? w/ accompanying = files). It also has a selector for installing perl -- our release has = /usr/release/R/stage/trees/perl which is actually an unpacked = perl-5.10.1_1 package, which ends up on the iso at = /freebsd/repos/8.1-RELEASE{,-amd64}/perl/perl.?? w/ accompanying dist = files). I have to say, it's nice being able to install FreeBSD-8.1 = _with_ perl (and if I later decide to uninstall it, I can, because we = retained the /var/db/pkg records). Oh, and before anyone asks, yes, I hand-recoded the +INSTALL, etc. files = from the packages we converted to distribution-sets into post-install = routines that are performed automatically. 7. And if that all wasn't enough (phew -- yeah, even that was abridged) = ... now we get to the custom boot loader. This part isn't as important to the whole "boot from both optical _and_ = USB stick" schtick, as it is important to our own internal operations. We have found a common problem to be that we _don't_ currently upgrade = our Operating System often enough to keep up with hardware demands. We = quite often find that we can't order a part anymore because it went = End-of-Life (EOL). Then we're forced to get some new hardware that = performs the same functionality. This can continue for some time until = the operating system you're working with too becomes End-of-Life. If you're good, you'll still be OK... you'll just hire someone to = backport drivers (HINT: that was my job for several years), from later = revisions of the OS which _aren't_ EOL'd. However, that gravy-train comes to an end eventually when the gap = between your current OS and the next-non-EOL OS is ... oh... say, 4 = major revisions (compare FreeBSD-4.11 to FreeBSD-8.1). Well, as an enterprise model, we can't just say "well, sorry, the = community has stopped supporting our OS, can't help you" as our = customers (the ones running the 1000+ FreeBSD = workstations/pedestals/servers) look to us for = solutions/upgrades/everything FreeBSD. The proper solution is to get the customers to upgrade more often -- but = our customers are the type that don't like to upgrade more than once = every few years (and as my co-workers point-out, the FreeBSD community = has worked with us on that in the past -- delaying the EOL of certain = releases for example). However, we found a permanent solution. By building a custom boot-loader that allows us to select from a list of = custom kernels and custom mfsroots and change boot parameters visually, = we were able to solve the problem of not being able to install a = particular OS because the GENERIC kernel used by the Walnut = Creek/FreeBSD Mall CD/DVD's couldn't effectively probe the newer = hardware. Of course, the boot-loader alone was only half the solution. Because = naturally, just slapping a custom kernel and boot-loader onto the disc = to get around your boot problems won't change the fact that the = installer will still lay-down the GENERIC kernel. Your very first boot = off the internal system disk will fail if the hardware is something that = the GENERIC kernel can't utilize. The other half of the solution was to have our scripted installs lay = down a custom kernel and configure that kernel to boot. FYI: The boot-loader I developed for this is no slouch... it's = 1930-lines of ANS FICL Forth -- the reverse polish stack-based language = for which there is an embedded interpreter inside FreeBSD's = boot-loader). I did not customize the boot-loader itself, so much as I = wrote extensible Forth modules capable of managing a hierarchical and/or = deep menu structure (something far more advanced than what is displayed = by today's boot-loader). In the full-picture, this has allowed us to do crazy things... like = install FreeBSD-4.11 to hardware that is not currently supported by = RELENG_4 even in it's current state (we've done crazy things like = regress drivers from FreeBSD-6.x back to 4.x, mainly mpt, isp, mfi, em, = fxp, and others as we use a lot of varied RAID controllers and NICs). But none of this could have been possible without the build process that = we developed to develop the install platform itself. The build-platform consists of: autoconf-based configure.in and GNUMakefile.in configure script config.guess, config.sub, install-sh Though don't be fooled. The GNUMakefile.in (sorry BSD makefiles -- you = don't support `override' in the target deps which means length of = makefile would have tripled if not quadrupled) is a heavy-hitter. I got deathly tired of ripping open mfsroot's to modify the boot-loader = Forth modules, only to be forced to close them, deconfigure the md = device, run my tests, then repeat. I ended up developing the makefile to = generate an ISO that contains the boot-loader, kernel(s), forth modules, = etc. and have the menu-ing system load the mfsroot. This opened up the = door for a whole new set up possibilities. We are now using vesamenu.c32 from ISOLINUX to throw up an advanced menu = of options, allowing the user to select from booting memtest86, windiag, = dban, spinrite, tufftest pro, hdt.c32, seatools, firmware updaters, or = ... our freebsd installer. ISOLINUX these days has support for chain-loading (via memdisk module) = to ISO files. So our makefile makes the ISO containing all the FreeBSD = boot stuffs, and that chain-loads to mfsroot, and that re-mounts the = CD-ROM where all the releases are. Sounds convoluted, but it dramatically increased development time. It = also allowed the core of development to move off of FreeBSD as the = development platform. The building of the ISO(s) can be done on any OS, = including the Forth modules, which kernels to load into the menu, which = kernels period, and everything about the boot process with the exception = of the mfsroot itself, can be worked-on on some other OS like Linux, Mac = OS X, Windows, or whatever as long as it supports CVS, has gmake, and = has mkisofs (and perl for isohybrid ability -- though that can be = disabled via ./configure). Not to mention that the disc can also install other OSes. We have in the = works right now, a multi-distro Linux installer being developed on the = same platform, and an N-Lite based slipstreamed Windows install too. The = build process is actively allowing development of all three in parallel = (FreeBSD developers say "make freebsd", linux guys say "make linux", = windows guys say "make windows", and when we all coalesce to a = release-point, we'll say "make all" and it will make the all-in-one = super-disc capable of installing multiple versions of FreeBSD, multiple = versions of Linux, and multiple versions of Windows -- all scripted = installs with minimal user-interaction). Now... for the best part (which I've been saving the best for last): I've already started generalizing this stuff and releasing it to the = outside world: http://druidbsd.sf.net/ Right now, if you were to pull down the code using anonymous pserver CVS = access, here's what you would get today: 1. A development platform already capable of producing an ISO that boots = FreeBSD either from optical media or USB stick 2. The underpinnings of our very own installer that we use to perform = cross-platform 64-bit while booted under 32-bit 3. Many (if not all) of the FreeBSD patches mentioned above 4. The excellent build process that solves all the difficult problems of = how to get FreeBSD booted under ISOLINUX and also the abstraction layer = that allows you to develop the boot process on platforms other than = FreeBSD (achieved by building the chain-loaded ISO at compile-time) 5. A graphical boot menu system And I do think I left all the tools in there, so it should come complete = with memtest86, memtest86+, dban, windiag, seatools, etc. etc. though = you can compile the ISO without them if size is a concern. The resulting ISO right now is only 32MB with all the tools enabled, or = 24MB w/o. You can get it even smaller (sub 16MB) if you rip out all the = rescue tools that I have added too. .... I am so sad that I could not be at the MeetBSD unConference this year. = Me and a co-worker were going to come down and meet up with you all. We = already know many of you and many of you know us, but we've been down = for the count for a few years. We really wanted to show you what we have been developing for the past = 3-5 years, because we feel it's reached a really stable level where it's = worth showing. I guess we'll have to work at making an appearance for = next year. It's just with all the mergers, we've not been able to get away from the = grind-stone (especially with down-sizing). But don't worry, we'll be = back in the community soon. Many may know us by the name VICOR. Our = signatures have changed in our e-mail footers, but we are still the same = core set of developers. Though I do miss the annual release parties we = used to throw for FreeBSD (gosh, that seems like so long ago now). -- Cheers, Devin Teske -> CONTACT INFORMATION <- Business Solutions Consultant II FIS - fisglobal.com 510-735-5650 Mobile 510-621-2038 Office 510-621-2020 Office Fax 909-477-4578 Home/Fax devin.teske@fisglobal.com -> LEGAL DISCLAIMER <- This message contains confidential and proprietary information of the sender, and is intended only for the person(s) to whom it is addressed. Any use, distribution, copying or disclosure by any other person is strictly prohibited. If you have received this message in error, please notify the e-mail sender immediately, and delete the original message without making a copy. -> FUN STUFF <- -----BEGIN GEEK CODE BLOCK----- Version 3.1 GAT/CS d(+) s: a- C++(++++) UB++++$ P++(++++) L++(++++) !E--- W++ N? o? = K- w O M+ V- PS+ PE Y+ PGP- t(+) 5? X+(++) R>++ tv(+) b+(++) DI+(++) D(+) G+>++ = e>+ h r>++ y+=20 ------END GEEK CODE BLOCK------ http://www.geekcode.com/ -> END TRANSMISSION <- P.S. Sorry if I rambled a bit... this stuff has been bottled up in my = head for a long time. For a time I was the sole developer on this = project.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7D12F133-FBFD-4664-A5B2-D8DC6D189F9B>