From owner-freebsd-libh Sun Oct 22 22:35:30 2000 Delivered-To: freebsd-libh@freebsd.org Received: from atlas.dnai.com (atlas.dnai.com [207.181.194.95]) by hub.freebsd.org (Postfix) with ESMTP id 63B5437B479 for ; Sun, 22 Oct 2000 22:35:26 -0700 (PDT) Received: from azoth.dnai.com (azoth.dnai.com [207.181.194.94]) by atlas.dnai.com (8.9.3/8.9.3) with ESMTP id WAA10745; Sun, 22 Oct 2000 22:35:23 -0700 (PDT) Received: from acm.org (207-172-166-179.s179.tnt1.sfrn.ca.dialup.rcn.com [207.172.166.179]) by azoth.dnai.com (8.9.3/8.9.3) with ESMTP id WAA34598; Sun, 22 Oct 2000 22:35:21 -0700 (PDT) Message-ID: <39F3CDD7.15B889E7@acm.org> Date: Sun, 22 Oct 2000 22:34:15 -0700 From: Tim Kientzle Reply-To: kientzle@acm.org X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 3.3-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: Alexander Langer Cc: libh@FreeBSD.ORG Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG The installer BOF didn't happen, purely because of my own ignorance. (I didn't know that all BOFs were on Wednesday night, and hence didn't arrive until mid-day Thursday.) However, I did talk to a few people about the installer and came up with at least one very concrete suggestion: The installer must be scriptable. Since installer size is such a major concern, the language needs to be something very compact to implement. I know of three languages that have extremely compact implementations: Forth, Lisp, and BASIC. My favorite is BASIC, because it's generally more accessible than the alternatives. If there's general agreement that a BASIC-like langauge would be appropriate, I might find time to whip up something in the next couple of weeks. This also implies that the installer must be very modular. In particular, the major dialogs need to be script language commands and the primary logic flow needs to be controlled by scripts. That will allow people to completely override the installation sequence. Ideally, the script language would also provide access to a collection of generic dialogs, but I think that's less essential. The primary use for the scripting will be to eliminate dialogs (automated installation), not to add new ones. If anyone agrees this is a good idea, let me know. I don't really have time to build code that nobody wants... - Tim Kientzle Alexander Langer wrote: > > Thus spake Tim Kientzle (kientzle@acm.org): > > > I'll be at BSDCon in Monterey on Thursday evening > > and all day Friday. I'd like to propose a BOF > > meeting on FreeBSD installation and packaging. > > Hello! > > Please summarize what you've discussed, if you have time. > > Thanks > > Alex > > -- > cat: /home/alex/.sig: No such file or directory > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-libh" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Mon Oct 23 10: 4:22 2000 Delivered-To: freebsd-libh@freebsd.org Received: from mailout06.sul.t-online.com (mailout06.sul.t-online.com [194.25.134.19]) by hub.freebsd.org (Postfix) with ESMTP id 9C12937B479 for ; Mon, 23 Oct 2000 10:04:20 -0700 (PDT) Received: from fmrl00.sul.t-online.de by mailout06.sul.t-online.com with smtp id 13nl1G-0001c9-03; Mon, 23 Oct 2000 19:04:18 +0200 Received: from neutron.cichlids.com (520050424122-0001@[62.225.194.153]) by fmrl00.sul.t-online.com with esmtp id 13nl1C-25n5E0C; Mon, 23 Oct 2000 19:04:14 +0200 Received: from cichlids.cichlids.com (cichlids.cichlids.com [192.168.0.10]) by neutron.cichlids.com (Postfix) with ESMTP id EE00EAB91; Mon, 23 Oct 2000 19:06:42 +0200 (CEST) Received: by cichlids.cichlids.com (Postfix, from userid 1001) id 55DFF14A7C; Mon, 23 Oct 2000 19:04:12 +0200 (CEST) Date: Mon, 23 Oct 2000 19:04:12 +0200 From: Alexander Langer To: Tim Kientzle Cc: libh@FreeBSD.ORG Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System Message-ID: <20001023190412.B507@cichlids.cichlids.com> References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <39F3CDD7.15B889E7@acm.org>; from kientzle@acm.org on Sun, Oct 22, 2000 at 10:34:15PM -0700 X-PGP-Fingerprint: 44 28 CA 4C 46 5B D3 A8 A8 E3 BA F3 4E 60 7D 7F X-PGP-at: finger alex@big.endian.de X-Verwirrung: Dieser Header dient der allgemeinen Verwirrung. X-Sender: 520050424122-0001@t-dialin.net Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Thus spake Tim Kientzle (kientzle@acm.org): > The installer must be scriptable. Isn't that what libh provides? > be very modular. In particular, the major dialogs > need to be script language commands and the primary > logic flow needs to be controlled by scripts. That They are already. Or better - "will be", since noone has written the scripts yet. Alex To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Mon Oct 23 12: 3:36 2000 Delivered-To: freebsd-libh@freebsd.org Received: from winston.osd.bsdi.com (winston.osd.bsdi.com [204.216.27.229]) by hub.freebsd.org (Postfix) with ESMTP id 97A4437B479 for ; Mon, 23 Oct 2000 12:03:33 -0700 (PDT) Received: from winston.osd.bsdi.com (jkh@localhost [127.0.0.1]) by winston.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id e9NJ38407135; Mon, 23 Oct 2000 12:03:09 -0700 (PDT) (envelope-from jkh@winston.osd.bsdi.com) To: kientzle@acm.org Cc: Alexander Langer , libh@FreeBSD.ORG Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System In-Reply-To: Message from Tim Kientzle of "Sun, 22 Oct 2000 22:34:15 PDT." <39F3CDD7.15B889E7@acm.org> Date: Mon, 23 Oct 2000 12:03:08 -0700 Message-ID: <7131.972327788@winston.osd.bsdi.com> From: Jordan Hubbard Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > However, I did talk to a few people about the installer > and came up with at least one very concrete suggestion: > > The installer must be scriptable. We're waaaaay ahead of you on that one. :) That's one of the major reasons that libh is TCL based and mandates a TCL API for all major functionality in addition to the C/C++ APIs already offered by various back-end libraries. You want to be able to compile up monolithic installer/package front-ends for embedded solution stuff (think flash) AND you want to be able to craft more comprehensive solutions, such as that which would go on a generic boot floppy, with a TCL interpreter embedded and lots of nice hooks for jumping to optional user scripts which can pretty much do anything they damn well please. > Since installer size is such a major concern, the > language needs to be something very compact to implement. > I know of three languages that have extremely compact > implementations: Forth, Lisp, and BASIC. My favorite Heh. Well, fortunately, TCL is both compact (at one of my last jobs we even got it into a telephone switch with only 64K of memory) and well-known to many. Forth, well, it's suited to boot loaders and such (being *really* compact) but has such an acceptance problem for more general programming that I don't think we could use it. I'm a big forth fan myself, don't get me wrong, but it's a really hard sell. Likewise lisp, even though you're starting to see more of it in window managers and such (I believe sawfish uses guile), is a pretty hard sell to people more used to procedural programming languages. I also like lisp and think it's a great language but, like forth, you usually hear little but loud whining when you adopt it for a mainstream project. The GIMP uses it, for example, and you see hardly anyone but the actual principal developers writing plug-ins for it. As to BASIC, well, ICK. It might be great for training neophytes in programming but it's got too many deficiencies as a general programming language unless you go all the way and attempt to write an HP business BASIC clone or something. There's no scoping, e.g. nothing like TCL's namespace mechanism, and the language itself is not very extensible. With TCL, for example, you can drop in one module and voila', you have a full OOP paradigm now available ([incr TCL]). There are quite a few other examples of this which make BASIC, especially a quick hand-rolled BASIC, simply way too simplistic for the job. Also, people tend to have a strong gag reflex when you mention BASIC and I'd rather not have to sell such a project over the sounds of gagging, fair or not. > This also implies that the installer must > be very modular. In particular, the major dialogs > need to be script language commands and the primary I think you really really need to look into libh. Much of what's on your wish list is already implemented, to say nothing of designed. :) - Jordan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Mon Oct 23 12:20:49 2000 Delivered-To: freebsd-libh@freebsd.org Received: from dnai.com (dnai.com [207.181.194.98]) by hub.freebsd.org (Postfix) with ESMTP id CFECF37B479; Mon, 23 Oct 2000 12:20:46 -0700 (PDT) Received: from azoth.dnai.com (azoth.dnai.com [207.181.194.94]) by dnai.com (8.9.3/8.9.3) with ESMTP id MAA28613; Mon, 23 Oct 2000 12:20:46 -0700 (PDT) Received: from acm.org (207-172-166-9.s9.tnt1.sfrn.ca.dialup.rcn.com [207.172.166.9]) by azoth.dnai.com (8.9.3/8.9.3) with ESMTP id MAA64963; Mon, 23 Oct 2000 12:20:44 -0700 (PDT) Message-ID: <39F48F4A.38D458C2@acm.org> Date: Mon, 23 Oct 2000 12:19:38 -0700 From: Tim Kientzle Reply-To: kientzle@acm.org X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 3.3-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: Alexander Langer , libh@freebsd.org Cc: "Jordan K. Hubbard" Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Alexander Langer wrote: > I've set up a small webpage what describes everything you need to > begin with libh. > http://people.freebsd.org/~alex/libh/ Well, looks like I'll not be contributing for a while, then, given that upgrading to 4.1 is not yet a real possibility for me. (Downtime plus the inevitably painful package upgrades that have followed every update I've done. ;-) However, I noticed that you repeated Jordan's initial draft specification. I can give a little feedback on that: * ZIP access? Have you written your own, or just incorporated pre-written libraries? The official ZIP specification puts all of the archive directory information at the _end_ of the archive; which is not particularly compatible with streaming. (In particular, you can't easily get a list of all files from a streamed archive without simply going through the entire archive; in this respect, ZIP is not an improvement over tar.gz.) Also, have you done anything to address ZIP's relatively poor compression? (tar.gz files are consistently much smaller than ZIP files; this is very important for package/distribution files). Finally, does the ZIP implementation you're using support hard/soft links? (Some packages rely on links heavily; just duplicating the files is not a good idea.) A better approach, in my opinion, is to stick with tar.gz, but with a slight twist: put the manifest/package definition information first (possibly just a free-form text file and/or install script?) followed by the tar.gz data. That gives you the package/distribution info up front where you need it, while preserving tar.gz's tight compression for the bulk of the data. tar.bz2 might be even better, though I'm unsure of the status of the bzip2 compression algorithms. * Consolidating package/distribution formats needs to be done carefully. In particular, there are different security issues: e.g., packages should generally be prohibited from dropping bits into /bin or /etc. * An idea that gets floated around periodically, but never apparently taken seriously: packages should install into private directories. /usr/local is becoming a real tarpit. Instead, package foo-3.4 should be contained _entirely_ (with few exceptions) in /usr/packages/foo-3.4/ Many people (including myself) have experimented with this type of scheme, and it simplifies a lot of issues: * it removes the need for a temporary installation directory, while still permitting easy rollback * simplifies a number of security issues (for example, you can compute an MD5 checksum of the tar.gz data while unpacking it to the target directory and simply delete the entire directory if the checksum fails) Handling these issues while still supporting streaming is tricky without this sort of framework. * It removes much of the need for a separate "package database". In particular, very few individual files ever need to go into such a database. Feedback appreciated... - Tim To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Mon Oct 23 13:40:58 2000 Delivered-To: freebsd-libh@freebsd.org Received: from oberon.dnai.com (oberon.dnai.com [207.181.194.97]) by hub.freebsd.org (Postfix) with ESMTP id 0725F37B479 for ; Mon, 23 Oct 2000 13:40:47 -0700 (PDT) Received: from azoth.dnai.com (azoth.dnai.com [207.181.194.94]) by oberon.dnai.com (8.9.3/8.9.3) with ESMTP id NAA13405; Mon, 23 Oct 2000 13:40:46 -0700 (PDT) Received: from acm.org (207-172-123-229.s483.tnt1.sfrn.ca.dialup.rcn.com [207.172.123.229]) by azoth.dnai.com (8.9.3/8.9.3) with ESMTP id NAA71067; Mon, 23 Oct 2000 13:40:45 -0700 (PDT) Message-ID: <39F4A24B.F421AF5B@acm.org> Date: Mon, 23 Oct 2000 13:40:43 -0700 From: Tim Kientzle Reply-To: kientzle@acm.org X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 3.3-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: Jordan Hubbard , libh@freebsd.org Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System References: <7131.972327788@winston.osd.bsdi.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Jordan Hubbard wrote: > Well, fortunately, TCL is both compact ... and well-known to many. My impression is that most of the "off-the-shelf" script languages (Python, Perl, Tcl, etc, etc) started off small but have since become quite bloated. I just compiled a short C program (given below) with an embedded Tcl interpreter and got a 530k stripped executable. Is that acceptable for sysinstall? #include #include int main(int argc, char **argv) { char *script = "set a 44"; Tcl_Interp * interp = Tcl_CreateInterp(); int result = Tcl_Eval(interp,script); puts(interp->result); if(result != TCL_OK) exit(1); exit(0); } > Forth ... has such an acceptance problem ... Likewise > lisp ... is a pretty hard sell ... True, which is why I didn't push either one. I spent a few years doing Forth development and really liked it, but I've learned not to champion it too loudly. > As to BASIC, well, ICK. ... Apparently, another hard sell. ;-) I was thinking, of course, of the more modern structured BASIC implementations (Visual Basic is the best-known) which draw heavily on Pascal and C. (Perl is heavily influenced by structured Basic, for example.) If Tcl is sufficiently compact, it's a good choice (even despite it's somewhat "unique" syntax); I'm just not sure that the current Tcl systems are sufficiently compact. (I guess you could dig up an old version...) - Tim To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Mon Oct 23 15: 5:36 2000 Delivered-To: freebsd-libh@freebsd.org Received: from winston.osd.bsdi.com (winston.osd.bsdi.com [204.216.27.229]) by hub.freebsd.org (Postfix) with ESMTP id AA9DE37B479; Mon, 23 Oct 2000 15:05:33 -0700 (PDT) Received: from winston.osd.bsdi.com (jkh@localhost [127.0.0.1]) by winston.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id e9NM5M409951; Mon, 23 Oct 2000 15:05:22 -0700 (PDT) (envelope-from jkh@winston.osd.bsdi.com) To: kientzle@acm.org Cc: Alexander Langer , libh@freebsd.org, "Jordan K. Hubbard" Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System In-Reply-To: Message from Tim Kientzle of "Mon, 23 Oct 2000 12:19:38 PDT." <39F48F4A.38D458C2@acm.org> Date: Mon, 23 Oct 2000 15:05:22 -0700 Message-ID: <9947.972338722@winston.osd.bsdi.com> From: Jordan Hubbard Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > * ZIP access? Have you written your own, or just incorporated > pre-written libraries? The official ZIP specification puts We've written our own. > Also, have you done anything to address ZIP's relatively poor > compression? (tar.gz files are consistently much smaller than ZIP I don't think this is such a big deal anymore since disk space has quintupled while prices fell proportionately. It's a measurable difference, yes, but by no means as important a difference as it was 10 years ago. > A better approach, in my opinion, is to stick with tar.gz, but > with a slight twist: put the manifest/package definition information > first (possibly just a free-form text file and/or install script?) That sort of works (for a grotty definition of "works") but the problem is also in getting things out selectively from that point forward. Using tar.gz format, the very idea of multi-architecture "fat packages" or extract-by-property features pretty much go out the window given that you can't just apply a name-mangling algorithm and have the random-access you need to just pluck the relevant "hashed" item out. > * Consolidating package/distribution formats needs to be done > carefully. In particular, there are different security issues: > e.g., packages should generally be prohibited from dropping bits > into /bin or /etc. That's all covered under the libh security model which allows you to restrict, override or augment any of the mutagenic file primitives. > * An idea that gets floated around periodically, but never apparently > taken seriously: packages should install into private directories. > /usr/local is becoming a real tarpit. Instead, package foo-3.4 should It's not so much that it isn't taken seriously as that it's hard. A lot of software (like emacs) hard-codes its path information into the compiled binaries and makes the resulting packages non-relocatable without the services of a binary editor and a hell of a lot of luck. > * it removes the need for a temporary installation directory, > while still permitting easy rollback Roll-back should be handled by the history stack, as described in my paper. That way a roll-back of n (where n > 1) is supported to cover some of the scenarios also given in my paper. - Jordan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Mon Oct 23 15:15: 0 2000 Delivered-To: freebsd-libh@freebsd.org Received: from winston.osd.bsdi.com (winston.osd.bsdi.com [204.216.27.229]) by hub.freebsd.org (Postfix) with ESMTP id 513CC37B479 for ; Mon, 23 Oct 2000 15:14:54 -0700 (PDT) Received: from winston.osd.bsdi.com (jkh@localhost [127.0.0.1]) by winston.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id e9NMEo410356; Mon, 23 Oct 2000 15:14:50 -0700 (PDT) (envelope-from jkh@winston.osd.bsdi.com) To: kientzle@acm.org Cc: libh@FreeBSD.ORG Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System In-Reply-To: Message from Tim Kientzle of "Mon, 23 Oct 2000 13:40:43 PDT." <39F4A24B.F421AF5B@acm.org> Date: Mon, 23 Oct 2000 15:14:49 -0700 Message-ID: <10351.972339289@winston.osd.bsdi.com> From: Jordan Hubbard Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > My impression is that most of the "off-the-shelf" script languages > (Python, Perl, Tcl, etc, etc) started off small but have since > become quite bloated. I just compiled a short C program (given below) Some are more bloated than others (let's not even talk about PERL) but we can still manage it. > with an embedded Tcl interpreter and got a 530k stripped executable. > Is that acceptable for sysinstall? Well, the current one is 820K stripped (separate, not with all the other crunched stuff) so I'd say just being able to separate large chunks of its current functionality into separate-to-the-executable scripts will win well beyond that, even starting at 530k and adding back some of the media handling stuff necessary to go find an install script and start it up. > ... Apparently, another hard sell. ;-) I was thinking, > of course, of the more modern structured BASIC implementations > (Visual Basic is the best-known) which draw heavily on Pascal and C. > (Perl is heavily influenced by structured Basic, for example.) Be honest: Do you really want to have to write something like this as a prerequisite to even starting the new installation program? I think we want to steal technology rather than implement it as much as possible or we'll never get this item off our TODO lists. :) - Jordan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Mon Oct 23 16:18:24 2000 Delivered-To: freebsd-libh@freebsd.org Received: from dnai.com (dnai.com [207.181.194.98]) by hub.freebsd.org (Postfix) with ESMTP id D719137B479 for ; Mon, 23 Oct 2000 16:18:19 -0700 (PDT) Received: from neptune.dnai.com (neptune.dnai.com [207.181.194.93]) by dnai.com (8.9.3/8.9.3) with ESMTP id QAA42349; Mon, 23 Oct 2000 16:18:19 -0700 (PDT) Received: from acm.org (207-172-166-6.s6.tnt1.sfrn.ca.dialup.rcn.com [207.172.166.6]) by neptune.dnai.com (8.9.3/8.9.3) with ESMTP id QAA74114; Mon, 23 Oct 2000 16:18:17 -0700 (PDT) Message-ID: <39F4C6F7.679F562D@acm.org> Date: Mon, 23 Oct 2000 16:17:11 -0700 From: Tim Kientzle Reply-To: kientzle@acm.org X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 3.3-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: Jordan Hubbard , libh@freebsd.org Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System References: <9947.972338722@winston.osd.bsdi.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Jordan Hubbard wrote: > > ... have you done anything to address ZIP's relatively poor compression? > > I don't think this is such a big deal anymore since disk space has > quintupled while prices fell proportionately. Yes, but CD-ROMs are still only 650MB. Are you willing to cut maybe 25% of the packages out of the CD-ROM distro? (I've seen many cases where ZIP files are nearly double the size of .tar.gz files. Have you compared ZIP and tar.gz versions of the ports collection?) And there are still a lot of people running over 28.8kbps modems; historically, bandwidth grows a lot more slowly than hard disk space. As for selective install, I'm very confused. Any streamable archive format will allow you to choose on-the-fly whether or not to write the next file to disk. If you have the manifest information up front, then you can decide on a file-by-file basis. If you're assuming that the archive is always available on a local disk (which is a requisite for single-file random access), then you're requiring a chunk of spare disk to store the package file, which I find confusing: why is a local copy of the package file so much worse than a local copy of the unpacked tree? (Which I understood to be one of your initial objections to tar.gz; to get random access, you had to first unpack it to a temp directory. Or was that purely a performance concern and not a disk-space concern?) The model I have in mind for package handling (much of which applies to distributions as well) is a three-phase model: * "Pre-install": Read manifest and installation requirements data. Among other things, this specifies which files/directories will be touched. (Having a single "package directory" simplifies this by eliminating the need to list every single file. Only files outside of the package directory need explicit checking.) It also specifies users that might need to be created, etc. Anything controversial can be verified with the user in advance; any installation options can also be settled now. * "Install": Based on options determined earlier, unpack the archived data; unneeded files are simply skipped. Streaming this is also faster than doing selective seeks and repeated single-file extractions. * "Post-Install": Some packages will have special requirements (such as editing /etc/rc.conf) This can be handled after unpacking by executing scripts in the package directory. This works very well with the idea of a "package file" consisting of manifest/installation data followed by a streamable archive. It also works nicely with the "directory per package" model. It requires no additional disk space (apart from space required for post-install scripts) either for storing the package file itself or a temporary unpacked version, which makes it very efficient for installing packages on-the-fly as they are downloaded. > > * An idea that gets floated around periodically, but never apparently > > taken seriously: packages should install into private directories. > > /usr/local is becoming a real tarpit. Instead, package foo-3.4 should > > It's not so much that it isn't taken seriously as that it's hard. A > lot of software (like emacs) hard-codes its path information into the > compiled binaries and makes the resulting packages non-relocatable ... Huh? You have to _compile_ a program to run in /usr/local; why not just _compile_ it to run in /usr/package/xxxx/? E.g., for any program that uses GNU autoconf (which definitely includes emacs): ./configure --prefix=/usr/packages/foo-3.4 make ... ... Then when the package is installed, it installs into the right directory and all of the compiled-in paths work just fine. As I mentioned, I used a hand-rolled version of such a system for several years; I largely abandoned it because I got tired of compiling everything myself. The 'packages' collection is a great convenience, even if I have to rm -rf /usr/local every year or two to keep the brambles down. (Yes, I really do this at regular intervals, and have since 1995 when I started using FreeBSD.) > > * it removes the need for a temporary installation directory, > > while still permitting easy rollback > > Roll-back should be handled by the history stack ... I'm not a fan of "big machinery" when naming conventions will often suffice. But, if you've already built the machinery, I suppose it's worth trying out. (Given my previous experience with "history stack" mechanisms, I'll remain dubious.) Plus, I should point out that having a directory per package doesn't rule out your "history stack". However, it does simplify it, since you don't need to track every individual file, only a single "package directory" and possibly a handful of files outside of that directory. BTW, one of my major concerns: packages must cohabitate with manually-compiled software. I've heard a lot of complaints from folks about the fact that RPM files only have "package" dependencies, not "file" dependencies. The current FreeBSD package system does dependencies much better: I can manually compile a certain library, then install a package that depends on that library without the package installer freaking out (either refusing to install or worse, overwriting my manually-built library). This is an essential feature. - Tim To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Tue Oct 24 9:48:40 2000 Delivered-To: freebsd-libh@freebsd.org Received: from manta.mayn.de (manta.mayn.de [194.145.150.45]) by hub.freebsd.org (Postfix) with ESMTP id 5971137B479 for ; Tue, 24 Oct 2000 09:48:38 -0700 (PDT) Received: by manta.mayn.de (Postfix, from userid 1001) id 9B81D9B005; Tue, 24 Oct 2000 18:52:56 +0200 (CEST) Date: Tue, 24 Oct 2000 18:52:56 +0200 To: Tim Kientzle Cc: Jordan Hubbard , libh@freebsd.org Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System Message-ID: <20001024185256.B92779@manta.mayn.de> References: <7131.972327788@winston.osd.bsdi.com> <39F4A24B.F421AF5B@acm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <39F4A24B.F421AF5B@acm.org>; from kientzle@acm.org on Mon, Oct 23, 2000 at 01:40:43PM -0700 X-PGP-Fingerprint: 44 28 CA 4C 46 5B D3 A8 A8 E3 BA F3 4E 60 7D 7F X-PGP-at: finger alex@big.endian.de X-Verwirrung: Dieser Header dient der allgemeinen Verwirrung. From: alex@manta.mayn.de (Alexander Langer) Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Thus spake Tim Kientzle (kientzle@acm.org): > with an embedded Tcl interpreter and got a 530k stripped executable. > Is that acceptable for sysinstall? Together with the Qt libs its even about 8 MB (stripped, but not compressed) It's ok, though, since it includes stuff for everything. > If Tcl is sufficiently compact, it's a good choice (even despite it's > somewhat "unique" syntax); I'm just not sure that the current Tcl ^^^^^^^^^^^^^^^^^^^^^^^^^^ seconded :I) Alex To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Thu Oct 26 5:37: 2 2000 Delivered-To: freebsd-libh@freebsd.org Received: from winston.osd.bsdi.com (winston.osd.bsdi.com [204.216.27.229]) by hub.freebsd.org (Postfix) with ESMTP id AF91437B4CF for ; Thu, 26 Oct 2000 05:36:48 -0700 (PDT) Received: from winston.osd.bsdi.com (jkh@localhost [127.0.0.1]) by winston.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id e9QCaY449572; Thu, 26 Oct 2000 05:36:39 -0700 (PDT) (envelope-from jkh@winston.osd.bsdi.com) To: kientzle@acm.org Cc: libh@freebsd.org Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System In-Reply-To: Message from Tim Kientzle of "Mon, 23 Oct 2000 16:17:11 PDT." <39F4C6F7.679F562D@acm.org> Date: Thu, 26 Oct 2000 05:36:34 -0700 Message-ID: <49568.972563794@winston.osd.bsdi.com> From: Jordan Hubbard Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > Yes, but CD-ROMs are still only 650MB. Are you willing to cut maybe 25% > of the packages out of the CD-ROM distro? (I've seen many cases > where ZIP files are nearly double the size of .tar.gz files. Have you > compared ZIP and tar.gz versions of the ports collection?) And there > are still a lot of people running over 28.8kbps modems; historically, > bandwidth grows a lot more slowly than hard disk space. Hmmm. I guess I'm not really sure yet; it's not an insignificant point you raise there, the jury is simply still out on how serious it would be in practice. > As for selective install, I'm very confused. Any streamable archive > format will allow you to choose on-the-fly whether or not to write > the next file to disk. If you have the manifest information up front, > then you can decide on a file-by-file basis. If you're assuming that > the archive is always available on a local disk (which is a requisite > for single-file random access). I'm not assuming it's always available, but it will be available in a large number of cases since a lot of people install from CDROM or NFS as well as FTP. > Huh? You have to _compile_ a program to run in /usr/local; why not > just _compile_ it to run in /usr/package/xxxx/? E.g., for any > program that uses GNU autoconf (which definitely includes emacs): The issue is that you're NOT compiling it when you install it as a package, and that's the point where many people would like to change the prefix. What if I don't like /usr/package but prefer /pkg? Or what if I still like /usr/local? Saying "well, you gotta build it from the port then" is to miss the whole point of ports/packages transparency and trying to make them essentially equivalent as far as the end result is concerned. - Jordan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Thu Oct 26 10:46:17 2000 Delivered-To: freebsd-libh@freebsd.org Received: from dnai.com (dnai.com [207.181.194.98]) by hub.freebsd.org (Postfix) with ESMTP id 1BAED37B479 for ; Thu, 26 Oct 2000 10:46:16 -0700 (PDT) Received: from azoth.dnai.com (azoth.dnai.com [207.181.194.94]) by dnai.com (8.9.3/8.9.3) with ESMTP id KAA20995; Thu, 26 Oct 2000 10:46:12 -0700 (PDT) Received: from acm.org (207-172-123-118.s372.tnt1.sfrn.ca.dialup.rcn.com [207.172.123.118]) by azoth.dnai.com (8.9.3/8.9.3) with ESMTP id KAA42375; Thu, 26 Oct 2000 10:46:10 -0700 (PDT) Message-ID: <39F86DDD.98678B9D@acm.org> Date: Thu, 26 Oct 2000 10:46:05 -0700 From: Tim Kientzle Reply-To: kientzle@acm.org X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 3.3-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: Jordan Hubbard Cc: libh@FreeBSD.ORG Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System References: <49568.972563794@winston.osd.bsdi.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Jordan Hubbard wrote: > Saying "well, you gotta build it from the port then" is to miss the > whole point of ports/packages transparency and trying to make them > essentially equivalent as far as the end result is concerned. On the other hand, saying that "you can do anything with a package you can do with a port" is simple naivete. People have tried it, it doesn't really work, let's move on. - Tim To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Thu Oct 26 20:29:56 2000 Delivered-To: freebsd-libh@freebsd.org Received: from winston.osd.bsdi.com (winston.osd.bsdi.com [204.216.27.229]) by hub.freebsd.org (Postfix) with ESMTP id B53C037B479 for ; Thu, 26 Oct 2000 20:29:54 -0700 (PDT) Received: from winston.osd.bsdi.com (jkh@localhost [127.0.0.1]) by winston.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id e9R3Th452557; Thu, 26 Oct 2000 20:29:44 -0700 (PDT) (envelope-from jkh@winston.osd.bsdi.com) To: kientzle@acm.org Cc: libh@FreeBSD.ORG Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System In-Reply-To: Message from Tim Kientzle of "Thu, 26 Oct 2000 10:46:05 PDT." <39F86DDD.98678B9D@acm.org> Date: Thu, 26 Oct 2000 20:29:43 -0700 Message-ID: <52554.972617383@winston.osd.bsdi.com> From: Jordan Hubbard Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > On the other hand, saying that "you can do anything with a package > you can do with a port" is simple naivete. People have tried > it, it doesn't really work, let's move on. Erm, but that's the entire point - it's SUPPOSED to work and if it doesn't, that port can be considered broken because it doesn't build a correct package. Let's not just casually dismiss something which is actually a hard-and-fast part of the functional spec before "moving on" here either. :-) - Jordan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Thu Oct 26 22:59: 6 2000 Delivered-To: freebsd-libh@freebsd.org Received: from puck.firepipe.net (mcut-b-167.resnet.purdue.edu [128.211.209.167]) by hub.freebsd.org (Postfix) with ESMTP id 5108D37B479 for ; Thu, 26 Oct 2000 22:59:05 -0700 (PDT) Received: by puck.firepipe.net (Postfix, from userid 1000) id 0A0E51A9F; Fri, 27 Oct 2000 00:59:23 -0500 (EST) Date: Fri, 27 Oct 2000 00:59:23 -0500 From: Will Andrews To: Jordan Hubbard Cc: kientzle@acm.org, libh@FreeBSD.ORG Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System Message-ID: <20001027005923.D3713@puck.firepipe.net> Reply-To: Will Andrews Mail-Followup-To: Will Andrews , Jordan Hubbard , kientzle@acm.org, libh@FreeBSD.ORG References: <52554.972617383@winston.osd.bsdi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <52554.972617383@winston.osd.bsdi.com>; from jkh@winston.osd.bsdi.com on Thu, Oct 26, 2000 at 08:29:43PM -0700 X-Operating-System: FreeBSD 4.1-STABLE i386 Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Thu, Oct 26, 2000 at 08:29:43PM -0700, Jordan Hubbard wrote: > Erm, but that's the entire point - it's SUPPOSED to work and if it > doesn't, that port can be considered broken because it doesn't build a > correct package. Let's not just casually dismiss something which is > actually a hard-and-fast part of the functional spec before "moving > on" here either. :-) No Jordan, I think he was talking about options that are available in ports. Currently there's no intuitive way to put options in packages and know it's there without having to hack the damn package to find out whether an option was enabled or not. Besides renaming the package file, of course, and that can get ugly. -- Will Andrews - Physics Computer Network wench To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Fri Oct 27 6:50:15 2000 Delivered-To: freebsd-libh@freebsd.org Received: from mailout06.sul.t-online.com (mailout06.sul.t-online.com [194.25.134.19]) by hub.freebsd.org (Postfix) with ESMTP id DFAAE37B479 for ; Fri, 27 Oct 2000 06:50:12 -0700 (PDT) Received: from fmrl03.sul.t-online.de by mailout06.sul.t-online.com with smtp id 13p9tW-0001SB-01; Fri, 27 Oct 2000 15:50:06 +0200 Received: from neutron.cichlids.com (520050424122-0001@[62.225.192.31]) by fmrl03.sul.t-online.com with esmtp id 13p9tS-2HbuYyC; Fri, 27 Oct 2000 15:50:02 +0200 Received: from cichlids.cichlids.com (cichlids.cichlids.com [192.168.0.10]) by neutron.cichlids.com (Postfix) with ESMTP id 9DC02AB91; Fri, 27 Oct 2000 15:52:46 +0200 (CEST) Received: by cichlids.cichlids.com (Postfix, from userid 1001) id 2342514A5C; Fri, 27 Oct 2000 15:50:07 +0200 (CEST) Date: Fri, 27 Oct 2000 15:50:07 +0200 To: Tim Kientzle Cc: libh@freebsd.org Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System Message-ID: <20001027155007.A42972@cichlids.cichlids.com> References: <7131.972327788@winston.osd.bsdi.com> <39F4A24B.F421AF5B@acm.org> <20001024185256.B92779@manta.mayn.de> <39F5D6CC.24A923C4@acm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <39F5D6CC.24A923C4@acm.org>; from kientzle@acm.org on Tue, Oct 24, 2000 at 11:37:00AM -0700 X-PGP-Fingerprint: 44 28 CA 4C 46 5B D3 A8 A8 E3 BA F3 4E 60 7D 7F X-PGP-at: finger alex@big.endian.de X-Verwirrung: Dieser Header dient der allgemeinen Verwirrung. From: alex@big.endian.de (Alexander Langer) X-Sender: 520050424122-0001@t-dialin.net Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Thus spake Tim Kientzle (kientzle@acm.org): > > Together with the Qt libs its even about 8 MB (stripped, but not > > compressed) > > It's ok, though, since it includes stuff for everything. > No, 8MB is not ok. The initial system needs to be installable > from floppy disk, and a floppy disk is 1.44MB. That leaves two > options: Yes. But a floppy-install will not include the Qt library, and gzipped it's only some 1.2 MB. > b) Not include sysinstall on the installation boot floppies. > Instead, pull it as the first step in the install process > (e.g., read it from CD, over the network, etc.) Yeah, that's the proposed way. I wanted to to that already, but had problems to gunzip and execute the new binary. The libh/bin/setup dir is what you want. The same is for the 8 MB RAM guys: The TVision-only one is smaller. Alex -- cat: /home/alex/.sig: No such file or directory To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message