Date: Sat, 6 Nov 2010 16:37:25 -0700 From: Garrett Cooper <yanegomi@gmail.com> To: Devin Teske <dteske@vicor.com> Cc: "jpaetzel@freebsd.org" <jpaetzel@freebsd.org>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, Devin Teske <dteske@vicor.com>, Nathan Whitehorn <nwhitehorn@freebsd.org>, Garrett Cooper <gcooper@FreeBSD.org> Subject: Re: txt-sysinstall scrapped Message-ID: <150A8520-3DC3-4978-916F-F51667482A50@gmail.com> In-Reply-To: <7D12F133-FBFD-4664-A5B2-D8DC6D189F9B@vicor.com> References: <201011052316.27839.jpaetzel@freebsd.org> <AANLkTi=62rRhZsN4wUi6p_yokSxG0tkjUHK7gosLtTRZ@mail.gmail.com> <20101105.230617.74669306.imp@bsdimp.com> <AANLkTi=G2UEj4P=h=B7Tr58vg7RC9McMZq-q73ArDWOZ@mail.gmail.com> <4CD58136.6070509@freebsd.org> <7D12F133-FBFD-4664-A5B2-D8DC6D189F9B@vicor.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Nov 6, 2010, at 1:33 PM, Devin Teske <dteske@vicor.com> wrote: >=20 > On Nov 6, 2010, at 9:24 AM, Nathan Whitehorn wrote: >=20 >> On 11/06/10 01:04, Garrett Cooper wrote: >>> On Fri, Nov 5, 2010 at 10:06 PM, Warner Losh <imp@bsdimp.com> wrote: >>>>> Just to add to that (because I do find it a novel idea), 1) how >>>>> are you going to properly prevent man in the middle attacks (SSL, TLS,= >>>>> etc?), and 2) what webserver would you use? >>>>=20 >>>> https or ssh. >>>>=20 >>>> We're also toying with the idea of having a partition that you could >>>> 'dd' your certs and keys to (so any system can customize the image >>>> with keys to make sure you were talking to who you think you are). >>>> We'd just reserve 1MB of space on partition s3. We'd then check to >>>> see if there was a tar ball. If so, we'd extract it and do the >>>> intelligent thing with the keys we find there. >>>=20 >>> Wouldn't it be better just to go with a read-write media solution >>> (USB) like Matt Dillon was suggesting at today then? Then again, >>> determining the root device to date is still a bit kludgy isn't it? >>=20 >> But this breaks badly for people who don't own USB sticks of sufficient >> size, are installing on machines without USB ports, can't boot from USB, >> want to install from a shared medium like PXE, are installing on blades >> with convenient shared CD drives but not USB etc. etc. Everything in the >> world can boot from CD, and we have to ensure that continues working. >>=20 >> I also have mixed feelings about needing to use a web browser to >> instruct a web app inside a bundled web server to write a config file to >> be interpreted by shell scripts just in order to run gpart, newfs, and >> tar. But if you get it working, it's better than sysinstall no matter >> how baroque. >> -Nathan >=20 >=20 > I find myself identifying a lot with this sentiment. We have FreeBSD runni= ng on 1,000+ systems (modestly) within our organization. We've found that so= me older BIOSes don't boot from USB sticks so-well. It's also hit-or-miss in= varied combinations of old-BIOS and off-brand sticks -- had great luck with= those super-tiny PNY's that rotate-out, though again, old BIOSes say pre-20= 03 P4 or AMD K6, have issues). >=20 > And since we're shipping at a rate of at least 100+ servers a year that ar= e now coming without any optical drives... our solution has been to purchase= one of these for each/every field engineer: >=20 > http://www.lg.com/us/computer-products/optical-media/LG-external-dvd-burne= r-GP08LU30.jsp >=20 > This great little device has worked wonders for over 30 field engineers (m= odestly). Out of thousands upon thousands of installs with the thing, we've r= eally only had a problem with about a half-dozen individual PCs -- but after= a BIOS upgrade, they tended to work else we slapped internal ATAPI CD-ROM's= in there and they worked). >=20 > Though, to-date what we've really found great-success in, is employing the= latest-and-greatest ISOLINUX boot-loader w/ their `isohybrid' ISO post-proc= essing program which encapsulates the disk-image contents within a hardware-= emulation structure (you can make your ISO -- when written to USB stick -- a= ppear to the BIOS as a ZIP disk, Hard Disk, etc.). >=20 > We currently tell all of our field engineers to download a single ISO that= is built with our custom build-process (which employs ISOLINUX and `isohybr= id' from http://syslinux.zytor.com/). This ISO can then optionally -- at the= field engineer's discretion -- either burn that ISO to optical media and us= e the LG AC-Adapterless USB optical drive (linked to above), OR ... they can= use dd(1) to write the ISO directly to a USB stick (though again... some ol= der BIOSes prefer the optical over flash media). >=20 > This was, of course, quite a challenge to achieve. >=20 > Here's a brief (lol -- it started that way, you got to believe me) list of= what we did to accomplish installs with one ISO on two mediums: >=20 > 1. Entire swaths of sysinstall(8) were patched >=20 > Out of the 48 source files that make up sysinstall (that I count currently= within RELENG_8), we've thus far patched 28 of them and added 1 new file. >=20 > We even added a new feature ... ability to install from a directory rather= than a device. In install.cfg, we have something akin to: >=20 > nullfs=3D/path/to/freebsd/release > mediaSetNullFS >=20 > As implied by the name, a nullfs is done from the directory specified to /= dist -- where sysinstall(8) expects to find the mounted distribution, regard= less of what media you select. >=20 >=20 >=20 > 2. One tiny kernel patch >=20 > There's a feature in the kernel for forcing the boot-medium into read-writ= e mode (which is not the default... the default is to mount read-only and sw= itch it during boot-up). When we're mounting an mfsroot, especially one that= 's been customized to run scripts which build the `install.cfg' script to ha= nd-off to sysinstall(8), you _want_ to mount read-write out of the gate. >=20 > This feature is enabled by setting the environment variable vfs.root.mount= from.options to "rw" in the loader's Forth name-space (in loader.rc for exam= ple). >=20 > Well, this was broken. : ( >=20 > So we patched it : ) >=20 >=20 >=20 > 3. One tiny patch to the release(7) Makefile >=20 > Since we rely on the release(7) process to compile both our customized sys= install(8) and also our custom mfsroot (explained below), it became a real p= roblem that WORLD_FLAGS was not being passed to installworld from within `/u= sr/obj'. >=20 > For example, when passing in WORLD_FLAGS=3D"-DWITHOUT_OPENSSL" (see src.co= nf(5)) we kept bombing out on the installworld phase which installs your new= ly compiled object-files from /usr/obj to an installbase of /usr/release (we= pass CHROOTDIR=3D/usr/release to `make release'). >=20 > This was because installworld was descending into a directory like gssapi w= hich it shouldn't, had WORLD_FLAGS been properly replicated down the chain t= o installworld. >=20 > We fixed this with a one-liner patch to /usr/src/release/Makefile >=20 >=20 >=20 > 4. A custom dialog(3) based menu system for selecting pre-scripted install= s of many different flavors >=20 >=20 >=20 > 5. A custom highly customized mfsroot >=20 > We start by patching /usr/src/release/i386/boot_crunch.conf (heavily). We a= dd the following utilities: >=20 > cat(1), chflags(1), chmod(1), cp(1), kill(1), ln(1), mkdir(1), mv(1), rmdi= r(1), sleep(1), bsdlabel(8) (aka disklabel), init(8), ldconfig(8), mount(8),= mount_cd9660(8), reboot(8) (aka halt), at(1) (aka atq, atrm, batch), awk(1)= , bzip2(1) (aka bunzip2, bzcat), ee(1), passwd(1), printf(1), tar(1), tail(1= ), uniq(1), chown(8) (aka chgrp), moused(8), mtree(8), pw(8), pwd_mkdb(8), s= endmail(8) (aka newaliases, mailq), tzsetup(8), vidcontrol(1), dialog(1), gr= ep(1), sort(1), and crontab(1) >=20 > NOTE: Yes, all those fit into the same amount of disk-space currently allo= cated for the mfsroot, so no patching was necessary to give us a larger mfsr= oot. >=20 > First, you might be asking yourself... OMG, why so many utilities added? >=20 > With the exception of dialog(1), mount_cd9660(8), and sleep(1), ALL of the= se utilities were added because they are called directly by sysinstall(8). >=20 > We wanted to do something absolutely crazy... we wanted to install FreeBSD= -8.1 amd64 while booted under FreeBSD-8.1 i386. This proved to be a challeng= e because once sysinstall(8) is done laying-down the distribution-sets chose= , it then does a chroot(2) into the newly-installed OS and starts calling al= l these lovely binaries (most listed above are called when performing option= al setup routines, like configuring sendmail, setting up the mouse, etc. etc= .). >=20 > Well, by adding all these utilities to the mfsroot and patching sysinstall= (8) to keep /stand around, we can have sysinstall(8) -- when told to do so -= - only call executables out of /stand after the chroot. This way, all the no= rmal optional setup routines can be performed, but now with the 32-bit binar= ies that we trust, rather than the 64-bit binaries which cause the system to= panic and reboot when invoked after the chroot normally. >=20 > This heavy patching affords the user to -- once in our custom dialog(3) ba= sed menu system -- choose to install either 64-bit or 32-bit despite having b= ooted from 32-bit kernel whereas currently sysinstall(8) must be run in an e= nvironment that can execute 64-bit binaries if you want to install the 64-bi= t distributions. >=20 > It also affords us the ability to put legacy OSes on the disk as optional i= nstalls. Because our patched sysinstall will never call a single binary with= in the installed base, we're safe from problems involving incompatible binar= ies. >=20 > Still talking about the customized mfsroot... once the `make release' proc= ess is finished and we have our highly customized mfsroot, we then patch it s= ome more! >=20 > We add the binary nullfs.ko kernel module -- to support our custom sysinst= all(8) feature of mediaSetNullFS -- which will load the nullfs kernel module= if this VFS type is not already compiled into the kernel (more on that late= r -- see description of our custom boot loader below). >=20 > Then we throw a very tiny binary custom init(8) that chain-loads to sysins= tall(8), but not before simply putting some dots up on the screen for 4 seco= nds -- allowing the user to quickly hit Scroll-Lock and peruse the kernel me= ssages if necessary before sysinstall launches (this was a real problem in 4= .x where the scrollback history was not preserved so well). >=20 > We add `/install.cfg' (searched-for and read automatically by sysinstall(8= ) on startup) which invokes a script which uses mount_cd9660 to re-mount the= device we booted off of to `/cdrom' >=20 > That last part is important. Remember that I mentioned that we're using IS= OLINUX's `isohybrid' to post-process the ISO. A very nice effect is that our= post-processed ISO is probed by the cd9660 geom provider and /dev/iso9660/V= OLID appears in devfs when probed, despite the fact that we booted using zip= -disk or hard-disk emulation from the BIOS. >=20 > This makes the process of re-mounting the media we booted from (from withi= n the mfsroot) a piece of cake, regardless of whether the field engineer wro= te the ISO to optical media or USB stick. >=20 > Lastly, before our custom mfsroot stops interrupting sysinstall(8)'s progr= ess (remember, we invoked this custom shell script via `/install.cfg' which u= sed mount_cd9660(8) to mount `/dev/iso9660/VOLID' to `/cdrom'), we invoke `/= cdrom/freebsd/menu/menu' which is an ELF executable based on dialog(3) which= presents our custom menu to the user for them to choose "Which OS?" to inst= all (among which we have both 32-bit variants AND 64-bit variants of FreeBSD= ). >=20 >=20 >=20 > 6. Because there's always that ONE package you'd like to have pre-installe= d along with your OS, we have an elaborate post-install configuration system= . >=20 > However, we later realized that (as randi once so eloquently voiced) insta= lling Packages from sysinstall(8) is a bad bad idea. >=20 > We patched sysinstall(8) to no longer call system executables for configur= ation setups, allowing safe install/configuration of a foreign binary system= , though we couldn't patch the package installation process for one very goo= d reason... >=20 > We can't predict what +INSTALL, +POST-INSTALL, +DEINSTALL, +POST-DEINSTALL= , and in-truth, even +CONTENTS (via @exec/@unexec) were going to do. Some de= velopers have even been caught writing these scripts in perl (which is no lo= nger part of the base distribution). >=20 > So, for that ONE package that we just _had to have_ pre-installed before f= irst-boot (for us it's perl), we make use of DIST_LOCAL. sysinstall has a fe= ature allowing you to create your own custom distribution-set (local). We en= ded up throwing a few packages in there. >=20 > Though to be honest, we felt that perl and kernels deserved special attent= ion. Our patched sysinstall(8) also has a selector for our own custom kernel= (we have in our release, /usr/release/R/stage/kernels/FIS which ends up in t= he ISO at /freebsd/repos/8.1-RELEASE{,-amd64}/kernels/fis.?? w/ accompanying= files). It also has a selector for installing perl -- our release has /usr/= release/R/stage/trees/perl which is actually an unpacked perl-5.10.1_1 packa= ge, which ends up on the iso at /freebsd/repos/8.1-RELEASE{,-amd64}/perl/per= l.?? w/ accompanying dist files). I have to say, it's nice being able to ins= tall FreeBSD-8.1 _with_ perl (and if I later decide to uninstall it, I can, b= ecause we retained the /var/db/pkg records). >=20 > Oh, and before anyone asks, yes, I hand-recoded the +INSTALL, etc. files f= rom the packages we converted to distribution-sets into post-install routine= s that are performed automatically. >=20 >=20 >=20 > 7. And if that all wasn't enough (phew -- yeah, even that was abridged) ..= . now we get to the custom boot loader. >=20 > This part isn't as important to the whole "boot from both optical _and_ US= B stick" schtick, as it is important to our own internal operations. >=20 > We have found a common problem to be that we _don't_ currently upgrade our= Operating System often enough to keep up with hardware demands. We quite of= ten find that we can't order a part anymore because it went End-of-Life (EOL= ). Then we're forced to get some new hardware that performs the same functio= nality. This can continue for some time until the operating system you're wo= rking with too becomes End-of-Life. >=20 > If you're good, you'll still be OK... you'll just hire someone to backport= drivers (HINT: that was my job for several years), from later revisions of t= he OS which _aren't_ EOL'd. >=20 > However, that gravy-train comes to an end eventually when the gap between y= our current OS and the next-non-EOL OS is ... oh... say, 4 major revisions (= compare FreeBSD-4.11 to FreeBSD-8.1). >=20 > Well, as an enterprise model, we can't just say "well, sorry, the communit= y has stopped supporting our OS, can't help you" as our customers (the ones r= unning the 1000+ FreeBSD workstations/pedestals/servers) look to us for solu= tions/upgrades/everything FreeBSD. >=20 > The proper solution is to get the customers to upgrade more often -- but o= ur customers are the type that don't like to upgrade more than once every fe= w years (and as my co-workers point-out, the FreeBSD community has worked wi= th us on that in the past -- delaying the EOL of certain releases for exampl= e). >=20 > However, we found a permanent solution. >=20 > By building a custom boot-loader that allows us to select from a list of c= ustom kernels and custom mfsroots and change boot parameters visually, we we= re able to solve the problem of not being able to install a particular OS be= cause the GENERIC kernel used by the Walnut Creek/FreeBSD Mall CD/DVD's coul= dn't effectively probe the newer hardware. >=20 > Of course, the boot-loader alone was only half the solution. Because natur= ally, just slapping a custom kernel and boot-loader onto the disc to get aro= und your boot problems won't change the fact that the installer will still l= ay-down the GENERIC kernel. Your very first boot off the internal system dis= k will fail if the hardware is something that the GENERIC kernel can't utili= ze. >=20 > The other half of the solution was to have our scripted installs lay down a= custom kernel and configure that kernel to boot. >=20 > FYI: The boot-loader I developed for this is no slouch... it's 1930-lines o= f ANS FICL Forth -- the reverse polish stack-based language for which there i= s an embedded interpreter inside FreeBSD's boot-loader). I did not customize= the boot-loader itself, so much as I wrote extensible Forth modules capable= of managing a hierarchical and/or deep menu structure (something far more a= dvanced than what is displayed by today's boot-loader). >=20 > In the full-picture, this has allowed us to do crazy things... like instal= l FreeBSD-4.11 to hardware that is not currently supported by RELENG_4 even i= n it's current state (we've done crazy things like regress drivers from Free= BSD-6.x back to 4.x, mainly mpt, isp, mfi, em, fxp, and others as we use a l= ot of varied RAID controllers and NICs). >=20 > But none of this could have been possible without the build process that w= e developed to develop the install platform itself. >=20 > The build-platform consists of: >=20 > autoconf-based configure.in and GNUMakefile.in > configure script > config.guess, config.sub, install-sh >=20 > Though don't be fooled. The GNUMakefile.in (sorry BSD makefiles -- you don= 't support `override' in the target deps which means length of makefile woul= d have tripled if not quadrupled) is a heavy-hitter. >=20 > I got deathly tired of ripping open mfsroot's to modify the boot-loader Fo= rth modules, only to be forced to close them, deconfigure the md device, run= my tests, then repeat. I ended up developing the makefile to generate an IS= O that contains the boot-loader, kernel(s), forth modules, etc. and have the= menu-ing system load the mfsroot. This opened up the door for a whole new s= et up possibilities. >=20 > We are now using vesamenu.c32 from ISOLINUX to throw up an advanced menu o= f options, allowing the user to select from booting memtest86, windiag, dban= , spinrite, tufftest pro, hdt.c32, seatools, firmware updaters, or ... our f= reebsd installer. >=20 > ISOLINUX these days has support for chain-loading (via memdisk module) to I= SO files. So our makefile makes the ISO containing all the FreeBSD boot stuf= fs, and that chain-loads to mfsroot, and that re-mounts the CD-ROM where all= the releases are. >=20 > Sounds convoluted, but it dramatically increased development time. It also= allowed the core of development to move off of FreeBSD as the development p= latform. The building of the ISO(s) can be done on any OS, including the For= th modules, which kernels to load into the menu, which kernels period, and e= verything about the boot process with the exception of the mfsroot itself, c= an be worked-on on some other OS like Linux, Mac OS X, Windows, or whatever a= s long as it supports CVS, has gmake, and has mkisofs (and perl for isohybri= d ability -- though that can be disabled via ./configure). >=20 > Not to mention that the disc can also install other OSes. We have in the w= orks right now, a multi-distro Linux installer being developed on the same p= latform, and an N-Lite based slipstreamed Windows install too. The build pro= cess is actively allowing development of all three in parallel (FreeBSD deve= lopers say "make freebsd", linux guys say "make linux", windows guys say "ma= ke windows", and when we all coalesce to a release-point, we'll say "make al= l" and it will make the all-in-one super-disc capable of installing multiple= versions of FreeBSD, multiple versions of Linux, and multiple versions of W= indows -- all scripted installs with minimal user-interaction). >=20 > Now... for the best part (which I've been saving the best for last): >=20 > I've already started generalizing this stuff and releasing it to the outsi= de world: >=20 > http://druidbsd.sf.net/ >=20 > Right now, if you were to pull down the code using anonymous pserver CVS a= ccess, here's what you would get today: >=20 > 1. A development platform already capable of producing an ISO that boots Fre= eBSD either from optical media or USB stick > 2. The underpinnings of our very own installer that we use to perform cros= s-platform 64-bit while booted under 32-bit > 3. Many (if not all) of the FreeBSD patches mentioned above > 4. The excellent build process that solves all the difficult problems of h= ow to get FreeBSD booted under ISOLINUX and also the abstraction layer that a= llows you to develop the boot process on platforms other than FreeBSD (achie= ved by building the chain-loaded ISO at compile-time) > 5. A graphical boot menu system >=20 > And I do think I left all the tools in there, so it should come complete w= ith memtest86, memtest86+, dban, windiag, seatools, etc. etc. though you can= compile the ISO without them if size is a concern. >=20 > The resulting ISO right now is only 32MB with all the tools enabled, or 24= MB w/o. You can get it even smaller (sub 16MB) if you rip out all the rescue= tools that I have added too. >=20 > .... >=20 > I am so sad that I could not be at the MeetBSD unConference this year. Me a= nd a co-worker were going to come down and meet up with you all. We already k= now many of you and many of you know us, but we've been down for the count f= or a few years. >=20 > We really wanted to show you what we have been developing for the past 3-5= years, because we feel it's reached a really stable level where it's worth s= howing. I guess we'll have to work at making an appearance for next year. >=20 > It's just with all the mergers, we've not been able to get away from the g= rind-stone (especially with down-sizing). But don't worry, we'll be back in t= he community soon. Many may know us by the name VICOR. Our signatures have c= hanged in our e-mail footers, but we are still the same core set of develope= rs. Though I do miss the annual release parties we used to throw for FreeBSD= (gosh, that seems like so long ago now). Being brief (sadly it would take a millennia to abbreviate the above tex= t with my phone :/) -- the suggestion was to replace the DVD with a fat USB i= mage. That was all :). Thanks! -Garrett=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?150A8520-3DC3-4978-916F-F51667482A50>