From owner-freebsd-libh Sun Oct 29 20: 1:53 2000 Delivered-To: freebsd-libh@freebsd.org Received: from peach.ocn.ne.jp (peach.ocn.ne.jp [210.145.254.87]) by hub.freebsd.org (Postfix) with ESMTP id A85F237B479; Sun, 29 Oct 2000 20:01:50 -0800 (PST) Received: from newsguy.com (p05-dn03kiryunisiki.gunma.ocn.ne.jp [210.232.224.134]) by peach.ocn.ne.jp (8.9.1a/OCN/) with ESMTP id NAA12001; Mon, 30 Oct 2000 13:01:33 +0900 (JST) Message-ID: <39FCF244.5A8C8E59@newsguy.com> Date: Mon, 30 Oct 2000 13:00:04 +0900 From: "Daniel C. Sobral" X-Mailer: Mozilla 4.7 [en] (Win98; I) X-Accept-Language: en,pt-BR MIME-Version: 1.0 To: kientzle@acm.org Cc: Alexander Langer , libh@FreeBSD.ORG, "Jordan K. Hubbard" Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Tim Kientzle wrote: > > * ZIP access? Have you written your own, or just incorporated > pre-written libraries? The official ZIP specification puts > all of the archive directory information at the _end_ of the > archive; which is not particularly compatible with streaming. > (In particular, you can't easily get a list of all files from > a streamed archive without simply going through the entire archive; > in this respect, ZIP is not an improvement over tar.gz.) Heh. I also thought the ZIP thing wasn't very well thought out. I wonder what other formats we could use... ARJ? RAR? LHA? > A better approach, in my opinion, is to stick with tar.gz, but > with a slight twist: put the manifest/package definition information > first (possibly just a free-form text file and/or install script?) > followed by the tar.gz data. That gives you the package/distribution What this does NOT allow is search a specific point in the archive, and start reading from there. Granted, this kind of use sucks with streams, but I gather it is wanted. > * Consolidating package/distribution formats needs to be done > carefully. In particular, there are different security issues: > e.g., packages should generally be prohibited from dropping bits > into /bin or /etc. Non-sense. Packages should do what they are told, period. Do not "forbid" anything. Besides, all paths in a package MUST be relative to the base directory. I might want it installed on /usr/local, /opt, or even /. Thus, distribution formats need not be any different from package format. > * An idea that gets floated around periodically, but never apparently > taken seriously: packages should install into private directories. > /usr/local is becoming a real tarpit. Instead, package foo-3.4 should > be contained _entirely_ (with few exceptions) in > /usr/packages/foo-3.4/ Err... no. This is a worse nightmare. Ok, you might disagree. You have the option of changing the base path of the port to be installed. Write a shell script that makes the install path get it's name from the port. -- Daniel C. Sobral (8-DCS) dcs@newsguy.com dcs@freebsd.org capo@world.wide.bsdconspiracy.net He has been convicted of criminal possession of a clue with intent to distribute. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Mon Oct 30 10:43:49 2000 Delivered-To: freebsd-libh@freebsd.org Received: from dnai.com (dnai.com [207.181.194.98]) by hub.freebsd.org (Postfix) with ESMTP id 0632637B4C5; Mon, 30 Oct 2000 10:43:46 -0800 (PST) Received: from azoth.dnai.com (azoth.dnai.com [207.181.194.94]) by dnai.com (8.9.3/8.9.3) with ESMTP id KAA10126; Mon, 30 Oct 2000 10:43:41 -0800 (PST) Received: from acm.org (207-172-166-38.s38.tnt1.sfrn.ca.dialup.rcn.com [207.172.166.38]) by azoth.dnai.com (8.9.3/8.9.3) with ESMTP id KAA96268; Mon, 30 Oct 2000 10:43:31 -0800 (PST) Message-ID: <39FDC12E.304B0011@acm.org> Date: Mon, 30 Oct 2000 10:42:54 -0800 From: Tim Kientzle Reply-To: kientzle@acm.org X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 3.3-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: "Daniel C. Sobral" Cc: Alexander Langer , libh@FreeBSD.ORG, "Jordan K. Hubbard" Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG "Daniel C. Sobral" wrote: > Tim Kientzle wrote: > > * ZIP access? Have you written your own, or just incorporated > > pre-written libraries? [ List of problems with ZIP ] > > Heh. I also thought the ZIP thing wasn't very well thought out. I wonder > what other formats we could use... ARJ? RAR? LHA? Unfortunately, all of those share ZIP's problem with poor compression (relative to tar.gz, at least). ZIP, ARJ, RAR, LHA, ZOO, ARC, StuffIt, etc, etc, all compress individual files and then create an archive from them. tar.gz creates the archive then compresses it all as a single stream. I've seen many instances in which tar.gz files are half the size of ZIP files. This is most pronounced with archives containing a lot of small text files. Though I haven't tested it, I wouldn't be surprised if the ports tree was more than twice as large in ZIP format as in tar.gz format. There's a very simple trade-off: archive-then-compress (tar.gz) gives smaller files (and therefore fits more stuff onto the installation CD-ROMs) at the cost of more tedious single-file access. compress-then-archive (ZIP, etc.) gives quick single-file access at the cost of bigger archives. Pick one. Personally, I think the most important issue is to be able to build full-featured distributions, which means sticking with high-compression formats such as tar.gz. Package installation is not a particularly time-critical operation, so I don't see fast single-file access as a sufficient imperative to justify losing as much as 20% of the current CD-ROM distribution. (That 20% figure is just a guess based on my experience with the different formats; it could be lower, could be much higher.) > > * Consolidating package/distribution formats needs to be done > > carefully. In particular, there are different security issues: > > e.g., packages should generally be prohibited from dropping bits > > into /bin or /etc. > > Non-sense. Packages should do what they are told, period. Do not > "forbid" anything. > > Besides, all paths in a package MUST be relative to the base directory. > I might want it installed on /usr/local, /opt, or even /. Thus, > distribution formats need not be any different from package format. Having a package just be a list of instructions to be blindly followed is acceptable as long as you assume everyone always pulls their packages from a "trusted" source. My understanding was that people wanted to move away from the assumption of a single, universally-available trusted source. That doesn't mean the basic format for distributions and packages can't be the same, but it does mean that there need to be different security models for them. The simplest security models I've been able to come up with are: * "distributions" can put anything anywhere they want * "packages" can only put files into a single directory owned by that package; any exceptions must be specifically listed in the manifest and may be subject to manual approval by the user. Either, of course, may be subject to cryptographic authentication. (The expiration of the RSA patent simplifies this issue considerably.) If you can suggest an equally simple security model, I'd be interested in seeing it. I think this model is simple, robust, easy to understand, and allows for several different implementation approaches. Putting all packages into /usr/local begs problems with file overwrites (which can't realistically be resolved manually; even when you can tell the user what package each file came from, few users will be able to tell whether it's "safe" for a particular file to be replaced) and creates other ugly problems because /usr/local must generally be shared with locally-compiled software. > > * An idea that gets floated around periodically, but never apparently > > taken seriously: packages should install into private directories. > > /usr/local is becoming a real tarpit. Instead, package foo-3.4 should > > be contained _entirely_ (with few exceptions) in > > /usr/packages/foo-3.4/ > > Err... no. This is a worse nightmare. Ok, you might disagree. You have > the option of changing the base path of the port to be installed. Write > a shell script that makes the install path get it's name from the port. "a worse nightmare"? I'm curious to know why you think this. I used exactly this approach on my own FreeBSD system for several years, and found it quite managable. I'd be interested in knowing what problems you've had with it, as I've found it a big improvement over the current /usr/local tarpit. For the last two years, as an experiment, I've tried using the packages system "straight" to see how it works. In short, it's rather a mess and something I'll soon be dumping. The current packages system is, in my opinion, acceptable only if you assume that noone ever manually compiles their own software. In particular, some of us would like to use /usr/local for locally-developed software, or need to track certain open-source projects (I'm currently tracking "Tomcat" and have closely followed "latex" in the past), which means that /usr/local becomes a mix of package-installed software and manually-compiled software. This is not a viable situation. As far as I can tell, the /usr/local mess is a somewhat delicate issue exactly because so much currently-distributed software currently installs by default into /usr/local. By using /usr/local for packages, that simplifies building ports, since many software packages require little or no change; on the other hand, most locally-compiled software is also going to end up in /usr/local, which argues for keeping packages out of /usr/local. - Tim Kientzle To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Mon Oct 30 13: 6:22 2000 Delivered-To: freebsd-libh@freebsd.org Received: from dnai.com (dnai.com [207.181.194.98]) by hub.freebsd.org (Postfix) with ESMTP id 7ABC237B4C5; Mon, 30 Oct 2000 13:06:17 -0800 (PST) Received: from neptune.dnai.com (neptune.dnai.com [207.181.194.93]) by dnai.com (8.9.3/8.9.3) with ESMTP id NAA38332; Mon, 30 Oct 2000 13:06:17 -0800 (PST) Received: from acm.org (207-172-166-73.s73.tnt1.sfrn.ca.dialup.rcn.com [207.172.166.73]) by neptune.dnai.com (8.9.3/8.9.3) with ESMTP id NAA11324; Mon, 30 Oct 2000 13:06:11 -0800 (PST) Message-ID: <39FDE2A0.C2CEF041@acm.org> Date: Mon, 30 Oct 2000 13:05:36 -0800 From: Tim Kientzle Reply-To: kientzle@acm.org X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 3.3-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: "Daniel C. Sobral" , Alexander Langer , libh@FreeBSD.ORG, "Jordan K. Hubbard" Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Tim Kientzle wrote: > Though I haven't tested it, I wouldn't be surprised if > the ports tree was more than twice as large in ZIP format as > in tar.gz format. I just did a few quick tests against my FreeBSD 3.3 system to see how much you lose by switching from tar.gz to ZIP. I simply archived a couple of directories and compared the sizes: Directory tar.gz ZIP /usr/ports 7,601,675 15,008,530 /usr/src 50,896,742 62,536,891 /usr/bin 3,892,391 6,192,116 /usr/share/man 28,449,979 22,518,970 (!) I think it's pretty clear that building a single archive and then compressing the whole thing is necessary if you really want to build full-featured CD-ROM distributions. - Tim Kientzle P.S. /usr/share/man is an interesting example which works out larger in tar.gz format because the individual files are already gzipped. I suspect that you could get an archive smaller than 22MB by un-gzipping all the individual files and then building a tar.gz archive. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Mon Oct 30 17: 2:17 2000 Delivered-To: freebsd-libh@freebsd.org Received: from oberon.dnai.com (oberon.dnai.com [207.181.194.97]) by hub.freebsd.org (Postfix) with ESMTP id 9FF5137B4FE; Mon, 30 Oct 2000 17:02:11 -0800 (PST) Received: from neptune.dnai.com (neptune.dnai.com [207.181.194.93]) by oberon.dnai.com (8.9.3/8.9.3) with ESMTP id RAA90914; Mon, 30 Oct 2000 17:02:10 -0800 (PST) Received: from acm.org (207-172-166-2.s2.tnt1.sfrn.ca.dialup.rcn.com [207.172.166.2]) by neptune.dnai.com (8.9.3/8.9.3) with ESMTP id RAA28297; Mon, 30 Oct 2000 17:02:08 -0800 (PST) Message-ID: <39FE19EA.5346F798@acm.org> Date: Mon, 30 Oct 2000 17:01:30 -0800 From: Tim Kientzle Reply-To: kientzle@acm.org X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 3.3-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: "Daniel C. Sobral" , Alexander Langer , libh@FreeBSD.ORG, "Jordan K. Hubbard" Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> <39FDE2A0.C2CEF041@acm.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Hmmm.. As I suspected, if you first gunzip -r /usr/share/man then a tar.gz archive is only 9MB. That suggests a couple of ways to save space in the distribution archives. One, obviously, is to store the data un-gzipped and, after unpacking, go back through and gzip appropriate files. (This is tricky with the man tree because of multiply-linked files. Best is to auto-build a shell script that gzips files and creates links; then you can build a very compact archive with just one copy of each man file.) Another approach is to build a custom archive format that permits you to store the actual file data un-gzipped but mark the entry so that the de-archiver will re-gzip the data as it's written. Sounds roundabout, I know, but if you think carefully about how gzip works internally, you'll understand why this generally gives better compression. It's similar to HTTP "transfer-encoding", if you want to think of it that way. A custom archive format is no big deal; I have a favorite one I've used for a couple of years now that's extremely easy to implement, extensible, etc. It discards tar's tape-centric heritage and in the process discards most of tar's limitations. - Tim Tim Kientzle wrote: > > Tim Kientzle wrote: > > Though I haven't tested it, I wouldn't be surprised if > > the ports tree was more than twice as large in ZIP format as > > in tar.gz format. > > I just did a few quick tests against my FreeBSD 3.3 > system to see how much you lose by switching from > tar.gz to ZIP. I simply archived a couple of directories > and compared the sizes: > > Directory tar.gz ZIP > /usr/ports 7,601,675 15,008,530 > /usr/src 50,896,742 62,536,891 > /usr/bin 3,892,391 6,192,116 > /usr/share/man 28,449,979 22,518,970 (!) > > I think it's pretty clear that building a single > archive and then compressing the whole thing is > necessary if you really want to build full-featured > CD-ROM distributions. > > - Tim Kientzle > > P.S. /usr/share/man is an interesting example > which works out larger in tar.gz format because the > individual files are already gzipped. I suspect that > you could get an archive smaller than 22MB by un-gzipping > all the individual files and then building a tar.gz archive. > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-libh" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Mon Oct 30 17:46:28 2000 Delivered-To: freebsd-libh@freebsd.org Received: from peach.ocn.ne.jp (peach.ocn.ne.jp [210.145.254.87]) by hub.freebsd.org (Postfix) with ESMTP id E436C37B4C5; Mon, 30 Oct 2000 17:46:22 -0800 (PST) Received: from newsguy.com (p23-dn03kiryunisiki.gunma.ocn.ne.jp [210.232.224.152]) by peach.ocn.ne.jp (8.9.1a/OCN/) with ESMTP id KAA02833; Tue, 31 Oct 2000 10:46:09 +0900 (JST) Message-ID: <39FE2406.150CA3B1@newsguy.com> Date: Tue, 31 Oct 2000 10:44:38 +0900 From: "Daniel C. Sobral" X-Mailer: Mozilla 4.7 [en] (Win98; I) X-Accept-Language: en,pt-BR MIME-Version: 1.0 To: kientzle@acm.org Cc: Alexander Langer , libh@FreeBSD.ORG, "Jordan K. Hubbard" Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Tim Kientzle wrote: > > > Heh. I also thought the ZIP thing wasn't very well thought out. I wonder > > what other formats we could use... ARJ? RAR? LHA? > > Unfortunately, all of those share ZIP's problem with poor > compression (relative to tar.gz, at least). ZIP, ARJ, RAR, LHA, > ZOO, ARC, StuffIt, etc, etc, all compress individual files and then > create an archive from them. tar.gz creates the archive then > compresses it all as a single stream. I've seen many instances Precisely. You cannot do random access with single stream compression. We _need_ to avoid single stream compression if we are to have random access, which we happen to want. Well, Jordan does, I just work here. :-) > in which tar.gz files are half the size of ZIP files. This is most > pronounced with archives containing a lot of small text files. This very unusual for ports. Few small text files, or many text files, some small some large, are the common cases. We should target the common cases. > Though I haven't tested it, I wouldn't be surprised if > the ports tree was more than twice as large in ZIP format as > in tar.gz format. It would surprise me. I think you are grossly overestimating tar.gz advantage. Anyway, the ports tree is very different from the source tree or the ports sources. > There's a very simple trade-off: > archive-then-compress (tar.gz) gives smaller files (and > therefore fits more stuff onto the installation CD-ROMs) > at the cost of more tedious single-file access. I predict that in two or three years, we'll be using DVD. The argument still holds true for network connection, as I *do not* predict wideband access _world wide_ in two or three years. > compress-then-archive (ZIP, etc.) gives quick single-file > access at the cost of bigger archives. Yep. > Pick one. Personally, I think the most important issue is to > be able to build full-featured distributions, which means sticking > with high-compression formats such as tar.gz. Package installation > is not a particularly time-critical operation, so I don't see > fast single-file access as a sufficient imperative to justify > losing as much as 20% of the current CD-ROM distribution. Package installation can be painfully slow because of tar.gz problems. Compare the speed with which our packages are installed with the speed of the most popular Linux package formats. We can be two times slower than them. > (That 20% figure is just a guess based on my experience with > the different formats; it could be lower, could be much higher.) I think that 20% figure is way out of proportion. I doubt the difference would go as high as 10%, based on my experience. > > Non-sense. Packages should do what they are told, period. Do not > > "forbid" anything. > > > > Besides, all paths in a package MUST be relative to the base directory. > > I might want it installed on /usr/local, /opt, or even /. Thus, > > distribution formats need not be any different from package format. > > Having a package just be a list of instructions to be blindly followed > is acceptable as long as you assume everyone always pulls their packages > from a "trusted" source. My understanding was that people wanted > to move away from the assumption of a single, universally-available > trusted source. That doesn't mean the basic format for distributions Nope, as far as I know, we DO NOT want to move away from our model. I, and a lot of people I have heard from, consider the centralized model we use to be one of FreeBSD's main advantages over some Linux distributions. Then again, maybe the mood has changed from the last time I saw this issue discussed, but I'd rather believe to still be in the majority. :-) Besides, that doesn't really address what I said. It's the USER who specifies what should be done. The package says "install to /bin", and the installation installs to "${PREFIX}/bin". By the default, for ports, that prefix is /usr/local. The distributions would have a prefix of /. If the user makes it / for a package install, do what he says. > Putting all packages into /usr/local begs problems with file overwrites > (which can't realistically be resolved manually; even when you can > tell the user what package each file came from, few users will be > able to tell whether it's "safe" for a particular file to be replaced) > and creates other ugly problems because /usr/local must generally be > shared > with locally-compiled software. Not true. We don't have much of a problem with overwrites, except for multiple versions of the same software. The later is an upgrade target problem, not a problem in itself. Programs, like tcl, which one would naturally want multiple versions of are installed without conflict. As for /usr/local being shared with locally compiled software... well, that's only if you want it to! :-) You may either install the locally compiled software elsewhere, or you may put a different base path for ports/package installation on /etc/make.conf, and still preserve the single directory for all ports model. > > Err... no. This is a worse nightmare. Ok, you might disagree. You have > > the option of changing the base path of the port to be installed. Write > > a shell script that makes the install path get it's name from the port. > > "a worse nightmare"? I'm curious to know why you think this. > I used exactly this approach on my own FreeBSD system for several > years, and found it quite managable. I'd be interested in knowing > what problems you've had with it, as I've found it a big improvement > over the current /usr/local tarpit. Somehow I dislike PATHs, MANPATHs and LIBPATHs with 40 or 50 entries, I dislike a /usr that won't fit in one screen, I dislike having programs all over the place, I dislike having to edit /etc every time I want to make a new program available, and I specially dislike having to instruct users in setting up their accounts to be able to use a new program . Install/deinstall is simple. It's in the database, so I have no problem telling one software apart from the other in the directories they share. I don't even _need_ to, as it's automated. It's the non-automated tasks that get more complex going from the single-directory model to the multiple-directories model, which I used on AIX, Ultrix and SunOS. Using both at the same time, I would sometimes sigh in relief when I had to deal with something on FreeBSD instead of the others. > For the last two years, as an experiment, I've tried using the packages > system "straight" to see how it works. In short, it's rather a > mess and something I'll soon be dumping. The current packages system > is, in my opinion, acceptable only if you assume that noone ever > manually compiles their own software. > > In particular, some of us would like to use /usr/local for > locally-developed > software, or need to track certain open-source projects (I'm currently > tracking "Tomcat" and have closely followed "latex" in the past), which > means that /usr/local becomes a mix of package-installed software and > manually-compiled software. This is not a viable situation. Well, *don't use* /usr/local! Use something else. Or install the ports on /opt or whatever, and keep local to yourself, though that seems to me to be needlessly straying away from the standard way we do things. This argument is without merit, imo. If you are mixing locally compiled software, over which you have complete control over install path, with the ports, that's exclusively *your* choice, with which our ports model have *no* connection. > As far as I can tell, the /usr/local mess is a somewhat delicate > issue exactly because so much currently-distributed software > currently installs by default into /usr/local. By using /usr/local > for packages, that simplifies building ports, since many software > packages require little or no change; on the other hand, most > locally-compiled software is also going to end up in /usr/local, > which argues for keeping packages out of /usr/local. Not really. All ports are built and installed in the directory ${PREFIX}. If you want elm+ME to be installed on /usr/elm+ME, you type PREFIX=/usr/elm+ME make all install So, as you see, the install place is *NOT* fixed at all! The default, in the absence of PREFIX, is /usr/local, but the argument that having this as default simplifies things from a build/install perspective is incorrect. -- Daniel C. Sobral (8-DCS) dcs@newsguy.com dcs@freebsd.org capo@world.wide.bsdconspiracy.net He has been convicted of criminal possession of a clue with intent to distribute. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Mon Oct 30 17:58:36 2000 Delivered-To: freebsd-libh@freebsd.org Received: from peach.ocn.ne.jp (peach.ocn.ne.jp [210.145.254.87]) by hub.freebsd.org (Postfix) with ESMTP id E69A637B4CF; Mon, 30 Oct 2000 17:58:33 -0800 (PST) Received: from newsguy.com (p23-dn03kiryunisiki.gunma.ocn.ne.jp [210.232.224.152]) by peach.ocn.ne.jp (8.9.1a/OCN/) with ESMTP id KAA05605; Tue, 31 Oct 2000 10:58:25 +0900 (JST) Message-ID: <39FE26E5.BDCC6CCD@newsguy.com> Date: Tue, 31 Oct 2000 10:56:53 +0900 From: "Daniel C. Sobral" X-Mailer: Mozilla 4.7 [en] (Win98; I) X-Accept-Language: en,pt-BR MIME-Version: 1.0 To: kientzle@acm.org Cc: Alexander Langer , libh@FreeBSD.ORG, "Jordan K. Hubbard" Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> <39FDE2A0.C2CEF041@acm.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Tim Kientzle wrote: > > I just did a few quick tests against my FreeBSD 3.3 > system to see how much you lose by switching from > tar.gz to ZIP. I simply archived a couple of directories > and compared the sizes: > > Directory tar.gz ZIP > /usr/ports 7,601,675 15,008,530 > /usr/src 50,896,742 62,536,891 > /usr/bin 3,892,391 6,192,116 > /usr/share/man 28,449,979 22,518,970 (!) I assume you used maximum compression for tar.gz. Did you use maximum compression for ZIP? -- Daniel C. Sobral (8-DCS) dcs@newsguy.com dcs@freebsd.org capo@world.wide.bsdconspiracy.net He has been convicted of criminal possession of a clue with intent to distribute. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Mon Oct 30 20:14:18 2000 Delivered-To: freebsd-libh@freebsd.org Received: from modemcable101.200-201-24.mtl.mc.videotron.ca (modemcable030.183-200-24.mtl.mc.videotron.ca [24.200.183.30]) by hub.freebsd.org (Postfix) with SMTP id EAED037B4C5 for ; Mon, 30 Oct 2000 20:14:12 -0800 (PST) Received: (qmail 11471 invoked from network); 31 Oct 2000 04:14:12 -0000 Received: from patrak.local.mindstep.com (HELO PATRAK) (192.168.10.4) by jacuzzi.local.mindstep.com with SMTP; 31 Oct 2000 04:14:12 -0000 Message-ID: <00cb01c042f1$1a347190$040aa8c0@local.mindstep.com> From: "Patrick Bihan-Faou" To: "\"Daniel C. Sobral\"" Cc: References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> <39FE2406.150CA3B1@newsguy.com> Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System Date: Mon, 30 Oct 2000 23:14:45 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Hi, > > Though I haven't tested it, I wouldn't be surprised if > > the ports tree was more than twice as large in ZIP format as > > in tar.gz format. > > It would surprise me. I think you are grossly overestimating tar.gz > advantage. Anyway, the ports tree is very different from the source tree > or the ports sources. Well somebody sent in results that show exactly that... > > Pick one. Personally, I think the most important issue is to > > be able to build full-featured distributions, which means sticking > > with high-compression formats such as tar.gz. Package installation > > is not a particularly time-critical operation, so I don't see > > fast single-file access as a sufficient imperative to justify > > losing as much as 20% of the current CD-ROM distribution. > > Package installation can be painfully slow because of tar.gz problems. > Compare the speed with which our packages are installed with the speed > of the most popular Linux package formats. We can be two times slower > than them. Are you so sure that the tar.gz format is the culprit in the installation slowness ? I build some custom packages using the "pkg_*" tools as they are today. These packages are really large (30M) since they correspond to a full (customized) FreeBSD install. Now the slow part is definitely not installing the content of the package, the really really painfuly slow part is the cleaning up of the temp directory that holds the uncompressed files. I think that with a format like ZIP, this is what you are trying to optimize right ? If I am not mistaken, most linux based packaging system also you the tar.gz format, and if as you claim they are faster than the FreeBSD system, it only means that they do it better. One way to be fast would be as one suggested the "one directory per package" approach. It is messy to set-up correctly if you do it manually, but this is precisely why tools are nice when they exist... > > > Besides, all paths in a package MUST be relative to the base directory. > > > I might want it installed on /usr/local, /opt, or even /. Thus, > > > distribution formats need not be any different from package format. > > > > Having a package just be a list of instructions to be blindly followed > > is acceptable as long as you assume everyone always pulls their packages > > from a "trusted" source. My understanding was that people wanted > > to move away from the assumption of a single, universally-available > > trusted source. That doesn't mean the basic format for distributions > > Nope, as far as I know, we DO NOT want to move away from our model. I, > and a lot of people I have heard from, consider the centralized model we > use to be one of FreeBSD's main advantages over some Linux > distributions. Then again, maybe the mood has changed from the last time > I saw this issue discussed, but I'd rather believe to still be in the > majority. :-) > > Besides, that doesn't really address what I said. It's the USER who > specifies what should be done. The package says "install to /bin", and > the installation installs to "${PREFIX}/bin". By the default, for ports, > that prefix is /usr/local. The distributions would have a prefix of /. > If the user makes it / for a package install, do what he says. Except that it is not likely to work. At this time, the motto would be more along the lines "install in /usr/local or you are doomed". This is not because of a deficiency of the package system, it is simply because most (if not all) software compiled in will contain the hardcoded "prefix" value somewhere in the code, and since the default for building the ports is to have "PREFIX=/usr/local", the resulting package should (must ?) be installed in /usr/local or it won't work properly. The only thing that the prefix option of pkg_add is usefull for (IMHO) is to install on a disk that is mounted at a different place than what is "production" location will be (i.e. boot from a CD, mount the HD on /mnt install all the packages in "/mnt/usr/local", reboot, the HD is now mounted as "/", done). > > Putting all packages into /usr/local begs problems with file overwrites > > (which can't realistically be resolved manually; even when you can > > tell the user what package each file came from, few users will be > > able to tell whether it's "safe" for a particular file to be replaced) > > and creates other ugly problems because /usr/local must generally be > > shared > > with locally-compiled software. > > Not true. We don't have much of a problem with overwrites, except for > multiple versions of the same software. The later is an upgrade target > problem, not a problem in itself. Programs, like tcl, which one would > naturally want multiple versions of are installed without conflict. > > As for /usr/local being shared with locally compiled software... well, > that's only if you want it to! :-) You may either install the locally > compiled software elsewhere, or you may put a different base path for > ports/package installation on /etc/make.conf, and still preserve the > single directory for all ports model. > > > > Err... no. This is a worse nightmare. Ok, you might disagree. You have > > > the option of changing the base path of the port to be installed. Write > > > a shell script that makes the install path get it's name from the port. > > > > "a worse nightmare"? I'm curious to know why you think this. > > I used exactly this approach on my own FreeBSD system for several > > years, and found it quite managable. I'd be interested in knowing > > what problems you've had with it, as I've found it a big improvement > > over the current /usr/local tarpit. > > Somehow I dislike PATHs, MANPATHs and LIBPATHs with 40 or 50 entries, I > dislike a /usr that won't fit in one screen, I dislike having programs > all over the place, I dislike having to edit /etc every time I want to > make a new program available, and I specially dislike having to instruct > users in setting up their accounts to be able to use a new program . Well this is precisely why we need good tools that can handle all of this automatically. Hell, these things are not really complicated, they are highly repitive, that's about it. If the situation is not good with AIX or Solaris, maybe it is because the tools are not good enough on these platforms. To keep /usr "small" we can use a convention: /usr/package/xyz-version or something similar to the ports layout (combining the structure of /var/db/pkg and /usr/ports). Also the use of symlinks (can be really hugly) can solve the PATH and friends issue. Maybe it would be time to revisit how these variables are handled (i.e. created, maintained etc.), maybe a system wide list of prefixes that trigger the proper instantiation of these variables when needed could be the solution. Of course then this is not exactly "tradditional" unix any more... > Not really. All ports are built and installed in the directory > ${PREFIX}. If you want elm+ME to be installed on /usr/elm+ME, you type > > PREFIX=/usr/elm+ME make all install > > So, as you see, the install place is *NOT* fixed at all! The default, in > the absence of PREFIX, is /usr/local, but the argument that having this > as default simplifies things from a build/install perspective is > incorrect. If you install a pre-compiled package that you obtained from the website, you DO NOT have a choice: you have to install in /usr/local Thinking about it, this little war is even a bit silly. The current port system is already smart enough to let you use a custom prefix. Why not implement 2 "standard" profiles: - "tradditional": default PREFIX=/usr/local - "split": default PREFIX=/usr/package/$PACKAGE_NAME-$VERSION Set the profile you like best in /etc/make.conf, and now everybody is happy. Of course we also need to develop the smarts that will handle the "split" layout effectively (i.e. updating the PATH and friends), but this is not rocket science and a couple of shell or perl scripts can tackle that easily. One of the advantages of the "split" version is that binary package installs can then become really fast no matter what the format of the archive is. Cheers, Patrick. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Mon Oct 30 21:20:44 2000 Delivered-To: freebsd-libh@freebsd.org Received: from peach.ocn.ne.jp (peach.ocn.ne.jp [210.145.254.87]) by hub.freebsd.org (Postfix) with ESMTP id A2D6037B479 for ; Mon, 30 Oct 2000 21:20:40 -0800 (PST) Received: from newsguy.com (p23-dn03kiryunisiki.gunma.ocn.ne.jp [210.232.224.152]) by peach.ocn.ne.jp (8.9.1a/OCN/) with ESMTP id OAA27469; Tue, 31 Oct 2000 14:20:09 +0900 (JST) Message-ID: <39FE562C.714DBE7C@newsguy.com> Date: Tue, 31 Oct 2000 14:18:36 +0900 From: "Daniel C. Sobral" X-Mailer: Mozilla 4.7 [en] (Win98; I) X-Accept-Language: en,pt-BR MIME-Version: 1.0 To: Patrick Bihan-Faou Cc: libh@FreeBSD.ORG Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> <39FE2406.150CA3B1@newsguy.com> <00cb01c042f1$1a347190$040aa8c0@local.mindstep.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Patrick Bihan-Faou wrote: > > > It would surprise me. I think you are grossly overestimating tar.gz > > advantage. Anyway, the ports tree is very different from the source tree > > or the ports sources. > > Well somebody sent in results that show exactly that... The same person that made the claim. I'll be surprised if he used maximum compression for ZIP, or default compression for gzip (though the later is not to the point, anyway). Of course, if he _does_ confirm he used maximum compression for ZIP, then I will, as I said, be surprised. :-) > > Package installation can be painfully slow because of tar.gz problems. > > Compare the speed with which our packages are installed with the speed > > of the most popular Linux package formats. We can be two times slower > > than them. > > Are you so sure that the tar.gz format is the culprit in the installation > slowness ? I build some custom packages using the "pkg_*" tools as they are > today. These packages are really large (30M) since they correspond to a full > (customized) FreeBSD install. Now the slow part is definitely not installing > the content of the package, the really really painfuly slow part is the > cleaning up of the temp directory that holds the uncompressed files. I think > that with a format like ZIP, this is what you are trying to optimize right ? The temporary directory is a by-product of having to decompress everything, and ZIP is a solution to this problem. Or so I understood from jkh's rants. > One way to be fast would be as one suggested the "one directory per package" > approach. It is messy to set-up correctly if you do it manually, but this is > precisely why tools are nice when they exist... I fail to see what it could gain us. > Except that it is not likely to work. At this time, the motto would be more > along the lines "install in /usr/local or you are doomed". This is not > because of a deficiency of the package system, it is simply because most (if > not all) software compiled in will contain the hardcoded "prefix" value > somewhere in the code, and since the default for building the ports is to > have "PREFIX=/usr/local", the resulting package should (must ?) be installed > in /usr/local or it won't work properly. Err... "most (if not all)" is not at all true! That view is seriously biased by the kind of application you run. Anyway, if you need it outside /usr/local, then, yes, you are better installing from ports. But it *will* work correctly from ports, and it will also work correctly for a lot of pre-compiled packages. The whole problem with binary packages is that they screw everyone who is not satisfied with the defaults. It needed not even be mentioned (since Jordan had already done, anyway :). > > Somehow I dislike PATHs, MANPATHs and LIBPATHs with 40 or 50 entries, I > > dislike a /usr that won't fit in one screen, I dislike having programs > > all over the place, I dislike having to edit /etc every time I want to > > make a new program available, and I specially dislike having to instruct > > users in setting up their accounts to be able to use a new program . > > Well this is precisely why we need good tools that can handle all of this > automatically. Hell, these things are not really complicated, they are > highly repitive, that's about it. If the situation is not good with AIX or > Solaris, maybe it is because the tools are not good enough on these > platforms. Installing and removing, that's easily automated. Even checking for filename conflict at install. Handling paths all over the place, that's very hard to automate. What's the point of moving to the system most difficult to automate? Unless, of course, you are voluntering to write all these tools before we make the change? > To keep /usr "small" we can use a convention: /usr/package/xyz-version or > something similar to the ports layout (combining the structure of > /var/db/pkg and /usr/ports). > > Also the use of symlinks (can be really hugly) can solve the PATH and > friends issue. > > Maybe it would be time to revisit how these variables are handled (i.e. > created, maintained etc.), maybe a system wide list of prefixes that trigger > the proper instantiation of these variables when needed could be the > solution. Of course then this is not exactly "tradditional" unix any more... Yes, these are all good ideas. We could also install everything under /usr/local, with application-specific subdirectories only for the applications that need them. Oh, wait, we already do that... What people won't suggest for the sake of doing it the hard way! :-) > If you install a pre-compiled package that you obtained from the website, > you DO NOT have a choice: you have to install in /usr/local Granted. > Thinking about it, this little war is even a bit silly. The current port > system is already smart enough to let you use a custom prefix. Why not > implement 2 "standard" profiles: > - "tradditional": default PREFIX=/usr/local > - "split": default PREFIX=/usr/package/$PACKAGE_NAME-$VERSION That's an idea. Of course, it requires more machine, more disk space, and _definitely_ overloads the install cd-roms (or dvds :). If such a suggestion is made with enough people backing "split", it might have a chance, though. > Set the profile you like best in /etc/make.conf, and now everybody is happy. > Of course we also need to develop the smarts that will handle the "split" > layout effectively (i.e. updating the PATH and friends), but this is not > rocket science and a couple of shell or perl scripts can tackle that easily. Easily said than done. > One of the advantages of the "split" version is that binary package installs > can then become really fast no matter what the format of the archive is. Only if we changed the tools to take advantage of the split version. -- Daniel C. Sobral (8-DCS) dcs@newsguy.com dcs@freebsd.org capo@world.wide.bsdconspiracy.net He has been convicted of criminal possession of a clue with intent to distribute. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Tue Oct 31 22:53:38 2000 Delivered-To: freebsd-libh@freebsd.org Received: from oberon.dnai.com (oberon.dnai.com [207.181.194.97]) by hub.freebsd.org (Postfix) with ESMTP id D8DDD37B4C5 for ; Tue, 31 Oct 2000 22:53:36 -0800 (PST) Received: from azoth.dnai.com (azoth.dnai.com [207.181.194.94]) by oberon.dnai.com (8.9.3/8.9.3) with ESMTP id WAA97150; Tue, 31 Oct 2000 22:53:36 -0800 (PST) Received: from acm.org (207-172-166-222.s222.tnt1.sfrn.ca.dialup.rcn.com [207.172.166.222]) by azoth.dnai.com (8.9.3/8.9.3) with ESMTP id WAA95887; Tue, 31 Oct 2000 22:53:31 -0800 (PST) Message-ID: <39FFBDA8.8E278D48@acm.org> Date: Tue, 31 Oct 2000 22:52:24 -0800 From: Tim Kientzle Reply-To: kientzle@acm.org X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 3.3-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: "Daniel C. Sobral" Cc: Patrick Bihan-Faou , libh@FreeBSD.ORG Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> <39FE2406.150CA3B1@newsguy.com> <00cb01c042f1$1a347190$040aa8c0@local.mindstep.com> <39FE562C.714DBE7C@newsguy.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG "Daniel C. Sobral" wrote: > Patrick Bihan-Faou wrote: > > If you install a pre-compiled package that you obtained from the website, > > you DO NOT have a choice: you have to install in /usr/local > > Granted. I just want to make sure everyone agrees with this point, since I've heard several people claim the opposite. - Tim Kientzle To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Wed Nov 1 0: 1:10 2000 Delivered-To: freebsd-libh@freebsd.org Received: from dnai.com (dnai.com [207.181.194.98]) by hub.freebsd.org (Postfix) with ESMTP id 2143037B4C5 for ; Wed, 1 Nov 2000 00:01:03 -0800 (PST) Received: from azoth.dnai.com (azoth.dnai.com [207.181.194.94]) by dnai.com (8.9.3/8.9.3) with ESMTP id AAA03922; Wed, 1 Nov 2000 00:01:02 -0800 (PST) Received: from acm.org (207-172-166-54.s54.tnt1.sfrn.ca.dialup.rcn.com [207.172.166.54]) by azoth.dnai.com (8.9.3/8.9.3) with ESMTP id AAA96926; Wed, 1 Nov 2000 00:00:54 -0800 (PST) Message-ID: <39FFCD73.7364C2BF@acm.org> Date: Tue, 31 Oct 2000 23:59:47 -0800 From: Tim Kientzle Reply-To: kientzle@acm.org X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 3.3-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: "Daniel C. Sobral" Cc: Patrick Bihan-Faou , libh@FreeBSD.ORG Subject: Making the Packages System Better References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> <39FE2406.150CA3B1@newsguy.com> <00cb01c042f1$1a347190$040aa8c0@local.mindstep.com> <39FE562C.714DBE7C@newsguy.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG "Daniel C. Sobral" wrote: > Somehow I dislike PATHs, MANPATHs and LIBPATHs with 40 or 50 entries, I > dislike a /usr that won't fit in one screen, I dislike having programs > all over the place, I dislike having to edit /etc every time I want to > make a new program available, and I specially dislike having to instruct > users in setting up their accounts to be able to use a new program . The following is an outline of an approach to package management that addresses all of these concerns, does not require a package database, and involves minimal automation. Just to make things clear, I'm specifically talking about pre-compiled packages such as would be downloaded from the FreeBSD website or installed from a CD-ROM. 1) Each package resides in it's own private directory. E.g., "foo" version 3.4 resides in /usr/packages/foo-3.4/ From here on, I'll refer to /usr/packages/foo-3.4/ as PREFIX. 2) Certain directories within PREFIX have special meaning. PREFIX/bin - user binaries PREFIX/lib - programming libraries PREFIX/man - man pages PREFIX/include - C/C++ include files PREFIX/rc.d - boot-time startup scripts Directories not on this list have no special meaning and can be used however the package wants to use them. 3) Special link directories are created for those recognized directories: /usr/packages/bin contains symlinks to /usr/packages/*/bin/* /usr/packages/lib contains symlinks to /usr/packages/*/lib/* /usr/pacakges/man/... tree contains symlinks to corresponding files in /usr/packages/*/man/... /usr/packages/include contains symlinks to /usr/packages/*/include/* ... etc ... Only recognized directories have their contents symlinked. For example, PREFIX/private is never symlinked, nor is PREFIX/libexec or PREFIX/libdata or any other directory not on a particular list. This is an important point. Several important things to note here: * User paths, MANPATH, LIBPATH, etc, do not ever need to be augmented when new directories are added, since those paths will all use the link directories. * Package installation/deinstallation/upgrading are pretty trivial. The only significant automation needed is to build/maintain the link dirs. * Tentative upgrades are fairly simple: /usr/packages/foo-3.5/ can be installed without first removing /usr/packages/foo-3.4/ (A little intelligence in building the link directories obviously helps here, but nothing unreasonable.) In comparison, supporting tentative upgrades with the current approach requires the ability to archive all updated files individually and restore them for a rollback. (Question: do you correctly handle files that existed in the previous version but not the new version?) * Rolling back an incomplete install is trivial: just delete the directory. With the current system, rolling back an incomplete install requires that you make copies of any overwritten or altered files and keep track of each file that does get installed. A crashed incomplete install is an ugly mess with the current defaults. * No temporary directory is needed, since a new directory is created for each package. If the package name isn't known in advance, you can create a temporary directory in /usr/packages and just rename it when the final name is known. Similarly, package archive creation is much simplified (no need to fake a directory heirarchy). * Some packages will obviously have to touch files outside of their directory; this could be handled in several ways, the easiest is to have an "install" script in the PREFIX dir and execute it after unpacking. * Dependency handling is very efficient, even with networked installs. You can pull down a package and unpack it directly into it's PREFIX dir, then test dependencies and recursively install those dependencies. By executing the "install" script and updating link directories as the recursion unwinds, you ensure that the visible parts of the installation are done in the correct order without requiring any temporary storage and without ever having to pause a download in progress. In contrast, the current system requires you to either stop the current download to handle the dependency or have multiple pending installs sitting in temporary directories requiring additional disk space. (This is admittedly a subtle point. ;-) * The symlink directories should only contain symlinks and never contain actual files. Otherwise, you run into some confusing maintenance issues. (Learn from database design: don't mix derived data and primary data.) * Packages should be compiled directly for their specific PREFIX. Files outside of the "special" directories do not get symlinked. Among other things, this avoids a lot of filename conflicts. (Two different spelling packages that have a "dictionary" file in PREFIX/private/dictionary won't conflict since that file doesn't need to be symlinked.) The basic principle: packages should keep their guts to themselves. In essence, the symlink directories are just how a package publicizes it's public interfaces. * This does not require any changes to the existing "ports" collection as long as all ports are PREFIX-aware. They can just be compiled with package-specific PREFIXes. * The symlink dirs should probably build shadows of any directories. E.g., /usr/packages/tcl-8.0/lib/tcl/xxxx should result in a real dir /usr/packages/lib/tcl/ with symlinks to corresponding files. That allows several tcl packages to have files within .../lib/tcl/ * Often, you need no special utilities at all with this scheme. For example, a list of installed packages: ls /usr/packages Building/maintaining the link directories is a pretty simple exercise in walking a directory heirarchy that can be cribbed from any number of sources. As a crude first cut, you can simply: cd /usr/packages/bin && ln -s ../*/bin/* . But that rapidly runs into argument-length limits. Besides, it's much faster to compare the existing symlinks to the desired symlinks and only alter the ones that have changed (delete symlinks that point to deleted packages or add new symlinks). Updating all of the symlink directories shouldn't take more than a second or two. I have Perl code that does most of this if you're interested (although it could be improved by adding full shadow-dir support). In short, I'd argue that this approach is significantly simpler than the current package tools: it doesn't require a database; it makes it easy to provide such advanced features as rollback, recovering failed installs and tentative upgrades. I'm fairly confident that tools to support this approach would be much simpler than overhauling the existing tools to provide such features. However, note that such tools would not work with the current /usr/local tarpit, since they rely heavily on the assumption that each package goes into its very own directory. If you must support the /usr/local tarpit, then you've got some ugly coding ahead. Comments? - Tim Kientzle To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Wed Nov 1 0: 9:19 2000 Delivered-To: freebsd-libh@freebsd.org Received: from dnai.com (dnai.com [207.181.194.98]) by hub.freebsd.org (Postfix) with ESMTP id 850AC37B4CF for ; Wed, 1 Nov 2000 00:09:16 -0800 (PST) Received: from azoth.dnai.com (azoth.dnai.com [207.181.194.94]) by dnai.com (8.9.3/8.9.3) with ESMTP id AAA06404; Wed, 1 Nov 2000 00:09:16 -0800 (PST) Received: from acm.org (207-172-166-54.s54.tnt1.sfrn.ca.dialup.rcn.com [207.172.166.54]) by azoth.dnai.com (8.9.3/8.9.3) with ESMTP id AAA97000; Wed, 1 Nov 2000 00:09:14 -0800 (PST) Message-ID: <39FFCFA8.BCF5425@acm.org> Date: Wed, 01 Nov 2000 00:09:12 -0800 From: Tim Kientzle Reply-To: kientzle@acm.org X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 3.3-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: "Daniel C. Sobral" Cc: Patrick Bihan-Faou , libh@FreeBSD.ORG Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> <39FE2406.150CA3B1@newsguy.com> <00cb01c042f1$1a347190$040aa8c0@local.mindstep.com> <39FE562C.714DBE7C@newsguy.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG "Daniel C. Sobral" wrote: > It would surprise me. I think you are grossly overestimating tar.gz > advantage. Anyway, the ports tree is very different from the source tree > or the ports sources. My earlier numbers used default compression for all tests. Using maximum compression for both ZIP and GZip gives different numbers, but the same conclusion: /usr/ports tar.gz size: 7,454,638 ZIP size: 14,947,231 If you don't trust my numbers, feel free to try it yourself: cd && zip -r9 - . | wc cd && tar -cf - . | gzip -9 | wc The ports tree is the most extreme example, but it's a general fact that archives of lots of relatively small text files will compress much better with tar.gz than with ZIP. This is intrinsic to how the compression works. (If you're unfamiliar with the nitty-gritty of modern compression algorithms, Mark Nelson's "The Data Compression Book" is a good place to start.) I originally thought the ports distribution was unique in this way, but it's not. The source tree, manpages, info, and doc distributions are also all collections of relatively small text files. The Perl and Emacs packages (both of which contain large directory trees of textual source code) are other good examples. Any source code archive will fit this kind of profile. To a somewhat lesser degree, this effect occurs in any archive of similar small files, which includes /usr/bin, for instance. If compression is important, tar.gz has a big advantage over ZIP. > Package installation can be painfully slow because of tar.gz problems. What exactly are these problems? I strongly suspect they are artifacts not of the file format itself but rather of the pkg_install program architecture. If you're doing multiple scans through the tar.gz archive to extract all files then, yes, that will be slow. The answer to this is to only do one pass. (Admittedly, that's not always easy to arrange, but noone said software optimization was simple. ;-) > The temporary directory is a by-product of having to decompress > everything, and ZIP is a solution to this problem. Huh? I don't follow this. I use tar quite often to unpack directly into a target directory. I think what you might be saying is that you currently unpack into a temporary directory in order to locate the manifest, then move everything from the temp dir according to the manifest instructions. Is this correct? This is why I suggested attaching the manifest separately to the beginning of the package file. Then, you can decide where to put files and just unpack the stream in a single pass directly to it's target location. The directory-per-package model allows some further optimizations (especially related to dependency loading when installing over a network connection) that aren't really practical with the current big /usr/local tarpit. (In particular, it lets you uncouple archiving and "installation" without requiring a temp directory, which allows you to perform fully-streaming installs with all dependencies without having to stop any stream.) > > One way to be fast would be as one suggested the "one directory per package" > > approach. It is messy to set-up correctly if you do it manually, but this is > > precisely why tools are nice when they exist... > > I fail to see what it could gain us. Because the package directory _is_ the temporary directory. There's no need to unpack into a separate location first. > The whole problem with binary packages is that they screw everyone who > is not satisfied with the defaults. Succinctly put. > > > Somehow I dislike PATHs, MANPATHs and LIBPATHs with 40 or 50 entries, I > > > dislike a /usr that won't fit in one screen, I dislike having programs > > > all over the place, I dislike having to edit /etc every time I want to > > > make a new program available, and I specially dislike having to instruct > > > users in setting up their accounts to be able to use a new program . I confess that when I first started playing with this scheme, I didn't see the obvious solution either. In a separate message I'll outline the full system that I've experimented with. It addresses all of these issues and simplifies a lot of related problems. > Installing and removing, that's easily automated. > Even checking for filename conflict at install. Could you explain how these are "easily automated"? I can see a couple of ways to handle these, but I want to make sure I understand which specific approach you have in mind. Maybe you see something I've overlooked. BTW, I notice you said "checking" for filename conflicts, and not "resolving" filename conflicts. > What's the point of moving to the system most difficult to automate? > Unless, of course, you are voluntering to write all these tools before > we make the change? I have written a set of tools to automate the maintenance of link directories. They're really very simple. Would you like to see them? - Tim Kientzle To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Wed Nov 1 0:26:22 2000 Delivered-To: freebsd-libh@freebsd.org Received: from modemcable101.200-201-24.mtl.mc.videotron.ca (modemcable030.183-200-24.mtl.mc.videotron.ca [24.200.183.30]) by hub.freebsd.org (Postfix) with SMTP id E19C437B479 for ; Wed, 1 Nov 2000 00:26:20 -0800 (PST) Received: (qmail 6117 invoked from network); 1 Nov 2000 08:26:20 -0000 Received: from patrak.local.mindstep.com (HELO PATRAK) (192.168.10.4) by jacuzzi.local.mindstep.com with SMTP; 1 Nov 2000 08:26:20 -0000 Message-ID: <01b601c043dd$75345050$040aa8c0@local.mindstep.com> From: "Patrick Bihan-Faou" To: , "Daniel C. Sobral" Cc: References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> <39FE2406.150CA3B1@newsguy.com> <00cb01c042f1$1a347190$040aa8c0@local.mindstep.com> <39FE562C.714DBE7C@newsguy.com> <39FFCD73.7364C2BF@acm.org> Subject: Re: Making the Packages System Better Date: Wed, 1 Nov 2000 03:26:39 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Well Tim I really thank you for that clear description of what I had in mind when I initialy posted in this thread. I for one would really be interested in seeing your scripts. And if anybody is interested, I will try to implement support for this scheme in the current port system and of course supply back the patches to the necessary files (most likely the various *.mk files for the port collection). As mentioned there is no reason why both approaches can not be supported by the ports collection. I am sure that a global PORT_INSTALL_METHOD make variable could satisfy proponents of both approaches. Now let's not start a holly-flame-war on this issue. I really believe that the approach that has been described is sound. However I would also like to hear about the arguments of people who oppose this. Maybe there is something that we missed and that will byte us badly at some point ? Patrick. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Wed Nov 1 1: 8:16 2000 Delivered-To: freebsd-libh@freebsd.org Received: from winston.osd.bsdi.com (winston.osd.bsdi.com [204.216.27.229]) by hub.freebsd.org (Postfix) with ESMTP id CE2A937B479 for ; Wed, 1 Nov 2000 01:08:14 -0800 (PST) Received: from winston.osd.bsdi.com (jkh@localhost [127.0.0.1]) by winston.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id eA197VU21922; Wed, 1 Nov 2000 01:07:32 -0800 (PST) (envelope-from jkh@winston.osd.bsdi.com) To: kientzle@acm.org Cc: "Daniel C. Sobral" , Patrick Bihan-Faou , libh@FreeBSD.ORG Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System In-Reply-To: Message from Tim Kientzle of "Wed, 01 Nov 2000 00:09:12 PST." <39FFCFA8.BCF5425@acm.org> Date: Wed, 01 Nov 2000 01:07:31 -0800 Message-ID: <21917.973069651@winston.osd.bsdi.com> From: Jordan Hubbard Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > pkg_install program architecture. If you're doing multiple scans > through the tar.gz archive to extract all files then, yes, that > will be slow. The answer to this is to only do one pass. pkg_install does it in one pass. - Jordan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Wed Nov 1 1:10:58 2000 Delivered-To: freebsd-libh@freebsd.org Received: from winston.osd.bsdi.com (winston.osd.bsdi.com [204.216.27.229]) by hub.freebsd.org (Postfix) with ESMTP id 289B737B4C5 for ; Wed, 1 Nov 2000 01:10:57 -0800 (PST) Received: from winston.osd.bsdi.com (jkh@localhost [127.0.0.1]) by winston.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id eA19AJU21947; Wed, 1 Nov 2000 01:10:32 -0800 (PST) (envelope-from jkh@winston.osd.bsdi.com) To: "Patrick Bihan-Faou" Cc: kientzle@acm.org, "Daniel C. Sobral" , libh@FreeBSD.ORG Subject: Re: Making the Packages System Better In-Reply-To: Message from "Patrick Bihan-Faou" of "Wed, 01 Nov 2000 03:26:39 EST." <01b601c043dd$75345050$040aa8c0@local.mindstep.com> Date: Wed, 01 Nov 2000 01:10:19 -0800 Message-ID: <21943.973069819@winston.osd.bsdi.com> From: Jordan Hubbard Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > the approach that has been described is sound. However I would also like to > hear about the arguments of people who oppose this. Maybe there is something > that we missed and that will byte us badly at some point ? There are a number of more complex packages which "find" themselves by examining argv[0] (or $0 if, more typically, they're scripts which front-end executables which require significant environmental pollution to run). These packages are often confused by finding their base to be a symlink and will do parent-directory (..) relative path smashing to find their other bits. If this fails to work, you're hosed. These "complex packages" are also not very rare - both emacs 20.x as well as java come immediately to mind, and there are others. - Jordan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Wed Nov 1 1:37:31 2000 Delivered-To: freebsd-libh@freebsd.org Received: from peach.ocn.ne.jp (peach.ocn.ne.jp [210.145.254.87]) by hub.freebsd.org (Postfix) with ESMTP id B5E5A37B4C5 for ; Wed, 1 Nov 2000 01:37:29 -0800 (PST) Received: from newsguy.com (p15-dn02kiryunisiki.gunma.ocn.ne.jp [211.0.245.80]) by peach.ocn.ne.jp (8.9.1a/OCN/) with ESMTP id SAA19492; Wed, 1 Nov 2000 18:36:59 +0900 (JST) Message-ID: <39FFE3E1.C769524D@newsguy.com> Date: Wed, 01 Nov 2000 18:35:29 +0900 From: "Daniel C. Sobral" X-Mailer: Mozilla 4.7 [en] (Win98; I) X-Accept-Language: en,pt-BR MIME-Version: 1.0 To: kientzle@acm.org Cc: Patrick Bihan-Faou , libh@FreeBSD.ORG Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> <39FE2406.150CA3B1@newsguy.com> <00cb01c042f1$1a347190$040aa8c0@local.mindstep.com> <39FE562C.714DBE7C@newsguy.com> <39FFBDA8.8E278D48@acm.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Tim Kientzle wrote: > > "Daniel C. Sobral" wrote: > > Patrick Bihan-Faou wrote: > > > If you install a pre-compiled package that you obtained from the website, > > > you DO NOT have a choice: you have to install in /usr/local > > > > Granted. > > I just want to make sure everyone agrees with this point, since > I've heard several people claim the opposite. Well, it *does* depend on the package, since it does not need to have compiled-in paths. But the kind of application that would fall under third-party binary only distribution is usually the kind that has them. -- Daniel C. Sobral (8-DCS) dcs@newsguy.com dcs@freebsd.org capo@world.wide.bsdconspiracy.net He has been convicted of criminal possession of a clue with intent to distribute. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Wed Nov 1 1:44:58 2000 Delivered-To: freebsd-libh@freebsd.org Received: from peach.ocn.ne.jp (peach.ocn.ne.jp [210.145.254.87]) by hub.freebsd.org (Postfix) with ESMTP id 3EFE237B4C5 for ; Wed, 1 Nov 2000 01:44:56 -0800 (PST) Received: from newsguy.com (p15-dn02kiryunisiki.gunma.ocn.ne.jp [211.0.245.80]) by peach.ocn.ne.jp (8.9.1a/OCN/) with ESMTP id SAA21744; Wed, 1 Nov 2000 18:44:49 +0900 (JST) Message-ID: <39FFE5B7.8FC4216B@newsguy.com> Date: Wed, 01 Nov 2000 18:43:19 +0900 From: "Daniel C. Sobral" X-Mailer: Mozilla 4.7 [en] (Win98; I) X-Accept-Language: en,pt-BR MIME-Version: 1.0 To: kientzle@acm.org Cc: Patrick Bihan-Faou , libh@FreeBSD.ORG Subject: Re: Making the Packages System Better References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> <39FE2406.150CA3B1@newsguy.com> <00cb01c042f1$1a347190$040aa8c0@local.mindstep.com> <39FE562C.714DBE7C@newsguy.com> <39FFCD73.7364C2BF@acm.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Tim Kientzle wrote: > [diet quote] > > Comments? To be brief, I don't like it. :-) Too SysV-ishy. Anyway, don't mind me. If you bring this up elsewhere you'll see plenty objections and downright flames, so I'll spare you any here. -- Daniel C. Sobral (8-DCS) dcs@newsguy.com dcs@freebsd.org capo@world.wide.bsdconspiracy.net He has been convicted of criminal possession of a clue with intent to distribute. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Wed Nov 1 6:47:10 2000 Delivered-To: freebsd-libh@freebsd.org Received: from hyde.ssec.wisc.edu (hyde.ssec.wisc.edu [144.92.108.217]) by hub.freebsd.org (Postfix) with ESMTP id 03C0D37B4CF for ; Wed, 1 Nov 2000 06:47:08 -0800 (PST) Received: from hyde.ssec.wisc.edu (dglo@localhost [127.0.0.1]) by hyde.ssec.wisc.edu (8.9.3/8.9.3) with ESMTP id IAA07292; Wed, 1 Nov 2000 08:46:25 -0600 (CST) Message-Id: <200011011446.IAA07292@hyde.ssec.wisc.edu> X-Mailer: exmh version 2.1.1 10/15/1999 To: kientzle@acm.org Cc: "Daniel C. Sobral" , Patrick Bihan-Faou , libh@FreeBSD.ORG Subject: Re: BOF at BSDCon: FreeBSD Installer, Packages System In-reply-to: Your message of "Wed, 01 Nov 2000 00:09:12 PST." <39FFCFA8.BCF5425@acm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 01 Nov 2000 08:46:25 -0600 From: Dave Glowacki Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Tim Kientzle wrote: > "Daniel C. Sobral" wrote: > > It would surprise me. I think you are grossly overestimating tar.gz > > advantage. Anyway, the ports tree is very different from the source tree > > or the ports sources. > > My earlier numbers used default compression for all tests. > Using maximum compression for both ZIP and GZip gives different > numbers, but the same conclusion: > > /usr/ports > tar.gz size: 7,454,638 > ZIP size: 14,947,231 > > If you don't trust my numbers, feel free to try it yourself: > > cd && zip -r9 - . | wc > cd && tar -cf - . | gzip -9 | wc One point I haven't seen anyone else make is that any compression method wins with more raw data. If your above is /usr/ports, you're unfairly biasing things in favor of tar.gz. Individual packages are *much* smaller than /usr/ports and thus won't get the same compression rate. The tar.gz versions will be smaller because they're compressing the entire package rather than compressing individual files, but I don't think the savings will be quite as dramatic, except possibly for mega-packages like emacs or X11. > I have written a set of tools to automate the maintenance of link > directories. They're really very simple. Would you like to see them? Your earlier description sounds a lot like the GNU 'stow' system. You might want to look at that: http://www.gnu.org/software/stow/manual.html To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Wed Nov 1 10:42:39 2000 Delivered-To: freebsd-libh@freebsd.org Received: from atlas.dnai.com (atlas.dnai.com [207.181.194.95]) by hub.freebsd.org (Postfix) with ESMTP id B6EB437B4C5 for ; Wed, 1 Nov 2000 10:42:37 -0800 (PST) Received: from azoth.dnai.com (azoth.dnai.com [207.181.194.94]) by atlas.dnai.com (8.9.3/8.9.3) with ESMTP id KAA05586; Wed, 1 Nov 2000 10:42:36 -0800 (PST) Received: from acm.org (207-172-123-6.s260.tnt1.sfrn.ca.dialup.rcn.com [207.172.123.6]) by azoth.dnai.com (8.9.3/8.9.3) with ESMTP id KAA15348; Wed, 1 Nov 2000 10:42:35 -0800 (PST) Message-ID: <3A0063F5.4798055B@acm.org> Date: Wed, 01 Nov 2000 10:41:57 -0800 From: Tim Kientzle Reply-To: kientzle@acm.org X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 3.3-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: "Daniel C. Sobral" Cc: Patrick Bihan-Faou , libh@FreeBSD.ORG Subject: Re: Making the Packages System Better References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> <39FE2406.150CA3B1@newsguy.com> <00cb01c042f1$1a347190$040aa8c0@local.mindstep.com> <39FE562C.714DBE7C@newsguy.com> <39FFCD73.7364C2BF@acm.org> <39FFE5B7.8FC4216B@newsguy.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG "Daniel C. Sobral" wrote: > > Tim Kientzle wrote: > > > [diet quote] > > > > Comments? > > To be brief, I don't like it. :-) > > Too SysV-ishy. Anyway, don't mind me. If you bring this up elsewhere > you'll see plenty objections and downright flames, so I'll spare you any > here. Actually, I've brought it up in several forums and the response has been interesting. Generally, I get one of three responses: * Agreement, often from people who have actually used something like this. * Worries about path explosion, which is a non-issue. * Vague "I don't like it" comments. So far, I've seen exactly two valid technical objections: * symlinks slow down file lookups. I don't know any good way to solve this one. Hardlinks would solve this issue, but would make link directory maintenance considerably uglier. * symlinks confuse programs that look for their data files relative to the executable location. One past culprit was the JDK, but that's been fixed in later versions. For other programs, you can mindlessly create a one-line shell-script wrapper that invokes the real program in it's package directory. This is more-or-less the system I'm going to be using; if the FreeBSD "package" tools don't support it, I won't be using them. No great loss, personally, but I just thought it would be nice if other FreeBSD users could benefit as well. - Tim To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Wed Nov 1 10:50:54 2000 Delivered-To: freebsd-libh@freebsd.org Received: from oberon.dnai.com (oberon.dnai.com [207.181.194.97]) by hub.freebsd.org (Postfix) with ESMTP id 8447137B4CF for ; Wed, 1 Nov 2000 10:50:51 -0800 (PST) Received: from azoth.dnai.com (azoth.dnai.com [207.181.194.94]) by oberon.dnai.com (8.9.3/8.9.3) with ESMTP id KAA56164; Wed, 1 Nov 2000 10:50:38 -0800 (PST) Received: from acm.org (207-172-123-6.s260.tnt1.sfrn.ca.dialup.rcn.com [207.172.123.6]) by azoth.dnai.com (8.9.3/8.9.3) with ESMTP id KAA16139; Wed, 1 Nov 2000 10:50:22 -0800 (PST) Message-ID: <3A0065EC.C4ABA4AD@acm.org> Date: Wed, 01 Nov 2000 10:50:20 -0800 From: Tim Kientzle Reply-To: kientzle@acm.org X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 3.3-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: Jordan Hubbard Cc: Patrick Bihan-Faou , "Daniel C. Sobral" , libh@FreeBSD.ORG Subject: Re: Making the Packages System Better References: <21943.973069819@winston.osd.bsdi.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Jordan Hubbard wrote: > > > the approach that has been described is sound. However I would also like to > > hear about the arguments of people who oppose this. Maybe there is something > > that we missed and that will byte us badly at some point ? > > There are a number of more complex packages which "find" themselves by > examining argv[0] (or $0 if, more typically, they're scripts which > front-end executables which require significant environmental > pollution to run). These packages are often confused by finding their > base to be a symlink and will do parent-directory (..) relative path > smashing to find their other bits. If this fails to work, you're > hosed. > > These "complex packages" are also not very rare - both emacs 20.x as > well as java come immediately to mind, and there are others. > > - Jordan Later JDK releases are symlink-aware and do not have this problem. (I know for a fact that the native JDK 1.2.2 is symlink-aware. The Blackdown Linux JDK 1.2.2 is also symlink-aware, so I think it's a fix that's been incorporated in Sun's sources, not just a FreeBSD-specific fix.) There's also a stock workaround that should work for any such program: $ cd /usr/packages/emacs-20.35 $ mkdir bin-private $ mv bin/emacs bin-private/emacs $ cd bin $ cat >emacs #!/bin/sh exec /usr/packages/emacs-20.35/bin-private/emacs "$@" ^D $ chmod +x emacs Now, a symlink from /usr/packages/bin/emacs to /usr/packages/emacs-20.35/bin/emacs will invoke this shell script which will exec the "real" emacs with all of the expected path information. Gee, Jordan, I think even you could automate this one. ;-) - Tim To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Wed Nov 1 18: 9:21 2000 Delivered-To: freebsd-libh@freebsd.org Received: from peach.ocn.ne.jp (peach.ocn.ne.jp [210.145.254.87]) by hub.freebsd.org (Postfix) with ESMTP id 44B3637B4CF for ; Wed, 1 Nov 2000 18:09:18 -0800 (PST) Received: from newsguy.com (p09-dn03kiryunisiki.gunma.ocn.ne.jp [210.232.224.138]) by peach.ocn.ne.jp (8.9.1a/OCN/) with ESMTP id LAA07012; Thu, 2 Nov 2000 11:09:09 +0900 (JST) Message-ID: <3A00CC6B.39522CF0@newsguy.com> Date: Thu, 02 Nov 2000 11:07:39 +0900 From: "Daniel C. Sobral" X-Mailer: Mozilla 4.7 [en] (Win98; I) X-Accept-Language: en,pt-BR MIME-Version: 1.0 To: kientzle@acm.org Cc: Patrick Bihan-Faou , libh@FreeBSD.ORG Subject: Re: Making the Packages System Better References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> <39FE2406.150CA3B1@newsguy.com> <00cb01c042f1$1a347190$040aa8c0@local.mindstep.com> <39FE562C.714DBE7C@newsguy.com> <39FFCD73.7364C2BF@acm.org> <39FFE5B7.8FC4216B@newsguy.com> <3A0063F5.4798055B@acm.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Tim Kientzle wrote: > > * Vague "I don't like it" comments. Symlinks are annoying for many people, which is one of the reasons they prefer BSD rc over SysV init. > So far, I've seen exactly two valid technical objections: > * symlinks slow down file lookups. I don't know > any good way to solve this one. Hardlinks would > solve this issue, but would make link directory > maintenance considerably uglier. > * symlinks confuse programs that look for their > data files relative to the executable location. > One past culprit was the JDK, but that's been > fixed in later versions. For other programs, > you can mindlessly create a one-line shell-script > wrapper that invokes the real program in it's > package directory. > > This is more-or-less the system I'm going to be using; > if the FreeBSD "package" tools don't support it, I won't > be using them. No great loss, personally, but I > just thought it would be nice if other FreeBSD users > could benefit as well. I plain don't like having dozens of directories under a single path. Your "solution" of moving the stuff from /usr to /usr/packages just move the problem. Furthermore, the main advantage you claim is forfeiting the need for the database of installed files, but having symlinks negates this disadvantage. Though removal/upgrade of packages could be done by search ALL symlink directories for invalid symlinks, and this is slow, these directories of symlinks are a database in themselves, only with disadvantages. Another advantage you claim is name clash avoidance. Again, having symlinks negates this advantage. It does make it faster to install packages under the current scheme we use for packages, *BUT*, that is only an artifact of how we do things. Since either method requires a change in this respect, this isn't really an advantage. It does make it easy to "test-upgrade" a package, *IF* you solve the symlink name clash problem, which you didn't. OTOH, you *CAN* test-install a port on an alternate directory of your choice, without any problems. Though this does precludes use of pre-compiled binaries for this purpose, you'll notice that many people never use pre-compiled packages at all. They have problems OTHER than PREFIX path. Also, symlinks make it harder to "eye-ball" things like /usr/local/etc when searching for problems, as they do not contain all information a file does (particularly access time and access permissions). Another big problem is that it would require retraining of personel and invalidate all existing documentation on our ports systems, and you can bet *THAT* will make people very pissed off. And the fact is that name clashes are _not_ a problem for most of us, and the application database is convenient when using pkg_info to find out information about a package. If you give it up, for example, you cannot find out if a file is part of a package or not, as it's presence in a directory is assumed to mean it is part of that package. So, you are basically proposing to replace something that exists, works, whose problems can be fixed without departing from the model, and with which people are familiar and comfortable with by something which would require new tools, new ways of doing things, bring new bugs for a while, introduce new problems and bring no advantage that cannot achieved by improving the existing method. The _only_ advantage your method bring is that people who like it like it. Well, people who don't like it, don't like it. And people who like the present method like the present method. So, the way I see it, this proposal brings a new set of problems, requires us to adjust to a new way of doing things, offers little short term advantages and no long term ones, and proposes a way of doing things I plainly don't like, having used it elsewhere. And, then, there is inertia. You are proposing a huge change in the way we do things. As long as you are just talking, people who think your idea is absurd will mostly ignore you. That will change if it gets closer to adoption. Most people are HAPPY with the present way of doing things. If you want to solve name clashes, test-upgrades, faster installs and the like, there are plenty paths to take within the existing framework, which would make it much more likely the adoption of the code. You do whatever you will, but I think the path you are taken is unlikely to be adopted. I'm not core, I'm not a ports committer, and I'm speaking only for myself, so feel free to disregard my opinions on this topic. -- Daniel C. Sobral (8-DCS) dcs@newsguy.com dcs@freebsd.org capo@world.wide.bsdconspiracy.net He has been convicted of criminal possession of a clue with intent to distribute. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Thu Nov 2 11:18:16 2000 Delivered-To: freebsd-libh@freebsd.org Received: from oberon.dnai.com (oberon.dnai.com [207.181.194.97]) by hub.freebsd.org (Postfix) with ESMTP id E8D5037B479 for ; Thu, 2 Nov 2000 11:18:10 -0800 (PST) Received: from neptune.dnai.com (neptune.dnai.com [207.181.194.93]) by oberon.dnai.com (8.9.3/8.9.3) with ESMTP id LAA95977; Thu, 2 Nov 2000 11:18:10 -0800 (PST) Received: from acm.org (207-172-166-25.s25.tnt1.sfrn.ca.dialup.rcn.com [207.172.166.25]) by neptune.dnai.com (8.9.3/8.9.3) with ESMTP id LAA86144; Thu, 2 Nov 2000 11:18:05 -0800 (PST) Message-ID: <3A01BDA9.C637C0AA@acm.org> Date: Thu, 02 Nov 2000 11:16:57 -0800 From: Tim Kientzle Reply-To: kientzle@acm.org X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 3.3-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: "Daniel C. Sobral" Cc: Patrick Bihan-Faou , libh@FreeBSD.ORG Subject: Re: Making the Packages System Better References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> <39FE2406.150CA3B1@newsguy.com> <00cb01c042f1$1a347190$040aa8c0@local.mindstep.com> <39FE562C.714DBE7C@newsguy.com> <39FFCD73.7364C2BF@acm.org> <39FFE5B7.8FC4216B@newsguy.com> <3A0063F5.4798055B@acm.org> <3A00CC6B.39522CF0@newsguy.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG "Daniel C. Sobral" wrote: > I plain don't like having dozens of directories under a single path. You could, of course, have /usr/packages mimic the layout of /usr/ports. That would largely address that issue and more tightly bind the ports and packages collections. I think a flat heirarchy is more appropriate here, but I could be convinced otherwise. Personally, I generally prefer more structured heirarchies, but the purpose of any directory organization is to simplify locating and/or managing files. Sometimes a flat heirarchy is the best way to do that. There are many directories in a standard FreeBSD system whose listings don't fit on one screen. > Furthermore, the main advantage you claim is forfeiting the need for the > database of installed files, but ... these > directories of symlinks are a database in themselves, only with > disadvantages. Here's an exercise: * You type the following on your system: rm -rf /var/db/pkg/* * I'll type the following on mine: rm -rf /usr/packages/{bin,lib} Now, who can recover more quickly? The symlink "database" is purely derived data, and hence can always be completely reconstructed from the filesystem. The package database is additional data on top of the filesystem; it cannot be easily reconstructed and the filesystem information is considerably less useful without it. On of my big concerns with the current package database is that there is no mechanism that enforces consistency between the database and the filesystem to which it refers. As a result, inadvertent change to either one can make the whole system much less effective. I've also seen people develop "object-oriented" filesystems which relied on an auxiliary database of information about each file; such systems tend to be brittle (files get moved or renamed without the database being updated). The fact that the current packages install by default into /usr/local which is assumed by so many source-code distributions makes it rather easy to change the filesystem without corresponding changes to the package database. Using link dirs, you simply update those directories; with a database, there's no easy recovery. (Imagine someone who installs something from the package collection, then installs the next version directly from source code; subsequent versions often dramatically change the file lists. The package/file information in the database now has many errors and there's no easy way to fix it.) Put simply: The package database is subject to irreversible bit-rot; the link directory is not. > Another advantage you claim is name clash avoidance. Again, having > symlinks negates this advantage. Nothing, of course, can completely eliminate name clashes. With a single shared /usr/local, name clashes must be handled at install time by deleting one of the offending files. (Or through some sort of backup mechanism.) Either makes it difficult to address the issue later on. Also, name clashes can be reduced because not every file needs to be linked; only publically-visible files. Private executables, data files, configuration files remain in the package directories and therefore avoid name clash issues. Thus, the system I propose provides another way to deal with name clashes; one that is not available with the current system. > It does make it easy to "test-upgrade" a package, *IF* you solve the > symlink name clash problem, which you didn't. One easy way to address this is to only link the most recent version of a package. (e.g., if it finds foo-3.4 and foo-3.5 it can only link foo-3.5) With this convention, tentative upgrades and version rollbacks are routine. With the current system, a tentative upgrade of a package requires repeated de-installs and re-installs, losing any customized settings in the process. Alternatively, it requires a backup system to be implemented within the package tools. From where I sit, that looks like an awfully complex answer to a simple problem. > OTOH, you *CAN* test-install a port on an alternate directory of > your choice, without any problems. Though this does precludes use > of pre-compiled binaries for this purpose, you'll notice that many > people never use pre-compiled packages at all. That's true, you can circumvent any limitation of the package system by simply not using it. What I'm doing is trying to make the package system itself more useful so that more people can use it. > Also, symlinks make it harder to "eye-ball" things like /usr/local/etc > when searching for problems, as they do not contain all information a > file does (particularly access time and access permissions). Having separate package directories does change things, yes. Instead of looking at "all configuration files in /usr/local/etc", you look at "all files associated with package foo". The current /usr/local approach leads naturally to the first, the separate directory-per-package approach leads naturally to the second. Each has it's benefits and drawbacks. In my experience, it's more common to discover "something is wrong with emacs" than to have a user report that "some package is mis-configured". > Another big problem is that it would require retraining of personel and > invalidate all existing documentation on our ports systems, and you can > bet *THAT* will make people very pissed off. I never thought I'd see the day that an open-source project requires discussion of "retraining of personel". My, how times change. ;-) I'm actually suggesting no changes to the ports system at all. Very little documentation should require changing. Ports that are PREFIX-clean will require no changes at all. Only the package tools require any changes. > If you give it up, for example, you > cannot find out if a file is part of a package or not, as it's presence > in a directory is assumed to mean it is part of that package. Yes, being in a certain directory is equivalent to being a part of that package. This makes it very easy to tell if a particular file is part of a particular package. On the other hand, it also largely renders such a question moot. With the current system, you sometimes need to know whether a particular file is part of a package so that you can manually take steps to preserve it. A less-experienced sysadmin is likely to delete that file and then later find out it was a bad idea. Automatic backup tools such as Jordan has been proposing don't save you from that. I concede that some form of package database is probably a good thing, even with my proposed scheme, since it does allow you to track the inevitable exceptional cases. But it's no longer an absolute requirement. > The _only_ advantage your method bring is that people who like it like > it. Well, people who don't like it, don't like it. And people who like > the present method like the present method. I believe that many people who "like the present method" have either not yet encountered some of its limitations or never considered alternatives; that's my whole reason in continuing this discussion. > ... and proposes a way of doing things I plainly don't like, > having used it elsewhere. I've also used Solaris' "/opt" convention, and I don't like it either. > Most people are HAPPY with the present way of doing things. Earlier, you claimed there were many people who never use the package tools. I suspect those people aren't all that happy with them. ;-) I used a simpler of my proposed system for several years, then tried using the package tools for a few years. The package tools are convenient, but /usr/local tends to become unmanagable after a few years. > If you want to solve name clashes, test-upgrades, faster installs > and the like, there are plenty paths to take within the existing > framework, which would make it much more likely the adoption of the code. Please explain how to fix the current tools. Your solution above to tentative upgrades was to not use pre-compiled packages. I've read Jordan's paper, and his suggestion of a complete versioning database on top of /usr/local sounds both needlessly complex and very brittle. (Certainly more complex than building shadow heirarchies and symlinks.) My most serious concern with the current approach relates to the lack of any enforced relation between the package database and the filesystem. There's no real solution for that, other than to contradict several decades of standard Unix practice and not use /usr/local for local software. Such a solution seems unenforcable to me. (The other solution is to move /usr/local off of ufs and onto a custom filesystem, but that seems even less practical, although no doubt someone has considered it.) At the very least, the default PREFIX should be changed to something other than /usr/local. That would reduce the brittleness of the current system. It wouldn't address many other issues, of course. - Tim To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Thu Nov 2 14:44:11 2000 Delivered-To: freebsd-libh@freebsd.org Received: from rucus.ru.ac.za (rucus.ru.ac.za [146.231.29.2]) by hub.freebsd.org (Postfix) with SMTP id C412137B4CF for ; Thu, 2 Nov 2000 14:44:04 -0800 (PST) Received: (qmail 70997 invoked by uid 1003); 2 Nov 2000 22:44:01 -0000 Date: Fri, 3 Nov 2000 00:44:01 +0200 From: Neil Blakey-Milner To: Tim Kientzle Cc: Jordan Hubbard , Patrick Bihan-Faou , "Daniel C. Sobral" , libh@FreeBSD.ORG Subject: Re: Making the Packages System Better Message-ID: <20001103004401.A60432@mithrandr.moria.org> References: <21943.973069819@winston.osd.bsdi.com> <3A0065EC.C4ABA4AD@acm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <3A0065EC.C4ABA4AD@acm.org>; from kientzle@acm.org on Wed, Nov 01, 2000 at 10:50:20AM -0800 X-Operating-System: FreeBSD 4.1-STABLE i386 X-URL: http://mithrandr.moria.org/nbm/ Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Wed 2000-11-01 (10:50), Tim Kientzle wrote: > > > the approach that has been described is sound. However I would also like to > > > hear about the arguments of people who oppose this. Maybe there is something > > > that we missed and that will byte us badly at some point ? > > > > There are a number of more complex packages which "find" themselves by > > examining argv[0] (or $0 if, more typically, they're scripts which > > front-end executables which require significant environmental > > pollution to run). These packages are often confused by finding their > > base to be a symlink and will do parent-directory (..) relative path > > smashing to find their other bits. If this fails to work, you're > > hosed. > > > > These "complex packages" are also not very rare - both emacs 20.x as > > well as java come immediately to mind, and there are others. > > > > - Jordan > > Later JDK releases are symlink-aware and do not have this problem. > (I know for a fact that the native JDK 1.2.2 is symlink-aware. > The Blackdown Linux JDK 1.2.2 is also symlink-aware, so I think > it's a fix that's been incorporated in Sun's sources, not just > a FreeBSD-specific fix.) > > There's also a stock workaround that should work for any such program: > > $ cd /usr/packages/emacs-20.35 > $ mkdir bin-private > $ mv bin/emacs bin-private/emacs > $ cd bin > $ cat >emacs > #!/bin/sh > exec /usr/packages/emacs-20.35/bin-private/emacs "$@" > ^D > $ chmod +x emacs > > Now, a symlink from /usr/packages/bin/emacs to > /usr/packages/emacs-20.35/bin/emacs will invoke this > shell script which will exec the "real" emacs with > all of the expected path information. If anyone cares, I actually like Tim's suggestion in theory, and think it's worth having some parallel testing going once someone has got it implemented. I'm talking maybe dedicating a few cycles to building these packages and having people try it out a bit in testing, maybe mixing it up with ports built with STOWLIKE_PREFIX set. I wrote a reasonable amount of stuff necessary for this about this time last year, but I don't seem to have it anywhere. I think the joke was 'FIRE - FreeBSD Install Relocation Engine' to which was extended "Fireman" and many other amusing sidetracks. It might be with the rest of the portconf stuff I was playing with on at the time (and thus got left behind when I moved on and couldn't get any comments) on one of my machines that currently isn't up, though. The only disadvantages are really human-relations stuff, which I'm pretty sure won't be as bad as some worry. I remember battling with whether to have a consolidated "etc" directory which were actually files, not symlinks - it's more consistent with most other systems. We could symlink "foo.conf-dist" as usual, though. This means local configuration stuff is always there, and may even improve the shared-vs-local filesystem situation. I'm not religious on this one, but I think it makes sense (with the obvious "add a default config if there isn't one, delete the config when you uninstall/upgrade and the config is unchanged" rules being taken care of by the package tools). Since there's going to be a massive shift with the new system, I think it could slip in reasonably with the rest of the stuff, especially if we decide to go for /usr/packages. New package format, new package database, new installation and package tools are more likely to be the hot items. Neil -- Neil Blakey-Milner nbm@mithrandr.moria.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Thu Nov 2 18:21: 7 2000 Delivered-To: freebsd-libh@freebsd.org Received: from peach.ocn.ne.jp (peach.ocn.ne.jp [210.145.254.87]) by hub.freebsd.org (Postfix) with ESMTP id 386CD37B4CF for ; Thu, 2 Nov 2000 18:21:00 -0800 (PST) Received: from newsguy.com (p04-dn03kiryunisiki.gunma.ocn.ne.jp [210.232.224.133]) by peach.ocn.ne.jp (8.9.1a/OCN/) with ESMTP id LAA11422; Fri, 3 Nov 2000 11:20:41 +0900 (JST) Message-ID: <3A02209D.5E7137F8@newsguy.com> Date: Fri, 03 Nov 2000 11:19:09 +0900 From: "Daniel C. Sobral" X-Mailer: Mozilla 4.7 [en] (Win98; I) X-Accept-Language: en,pt-BR MIME-Version: 1.0 To: kientzle@acm.org Cc: Patrick Bihan-Faou , libh@FreeBSD.ORG Subject: Re: Making the Packages System Better References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> <39FE2406.150CA3B1@newsguy.com> <00cb01c042f1$1a347190$040aa8c0@local.mindstep.com> <39FE562C.714DBE7C@newsguy.com> <39FFCD73.7364C2BF@acm.org> <39FFE5B7.8FC4216B@newsguy.com> <3A0063F5.4798055B@acm.org> <3A00CC6B.39522CF0@newsguy.com> <3A01BDA9.C637C0AA@acm.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Tim Kientzle wrote: > > Personally, I generally prefer more structured heirarchies, but > the purpose of any directory organization is to simplify locating > and/or managing files. Sometimes a flat heirarchy is the best > way to do that. There are many directories in a standard FreeBSD > system whose listings don't fit on one screen. Me too. Why is it that to me this means cd /usr/local/etc to configure programs, and to you it means cd /usr/packages/X/etc; configure; cd /usr/packages/Y/etc; configure; etc...? > Here's an exercise: > * You type the following on your system: rm -rf /var/db/pkg/* > * I'll type the following on mine: rm -rf /usr/packages/{bin,lib} > Now, who can recover more quickly? The symlink "database" I'll tell you when such an accident happens to me. Until then, this exercise is futile. > is purely derived data, and hence can always be completely reconstructed > from the filesystem. The package database is additional data > on top of the filesystem; it cannot be easily reconstructed and > the filesystem information is considerably less useful without it. BTW, a hacker inserted some trojan binaries among the installed packages. Who can find out faster those trojans? > On of my big concerns with the current package database is > that there is no mechanism that enforces consistency between > the database and the filesystem to which it refers. As a result, > inadvertent change to either one can make the whole system > much less effective. I've also seen people develop "object-oriented" > filesystems which relied on an auxiliary database of information about > each file; such systems tend to be brittle (files get moved or > renamed without the database being updated). The fact that the > current packages install by default into /usr/local which is > assumed by so many source-code distributions makes it rather > easy to change the filesystem without corresponding changes to > the package database. Using link dirs, you simply update those > directories; with a database, there's no easy recovery. (Imagine > someone who installs something from the package collection, then > installs the next version directly from source code; subsequent > versions often dramatically change the file lists. The package/file > information in the database now has many errors and there's no easy > way to fix it.) Err, no. The versions keep their file list, so there is no error. Of course, one version has probably overwritten other version's files, but that is an implementation problem. The upgrade must correctly remove the previous version and install the new version, and make any format corrections or preserve configuration files. That this doesn't work has _nothing_ to do with the database being used. > Put simply: The package database is subject to irreversible bit-rot; > the link directory is not. I'll grant you that. > > Another advantage you claim is name clash avoidance. Again, having > > symlinks negates this advantage. > > Nothing, of course, can completely eliminate name clashes. With My point. > a single shared /usr/local, name clashes must be handled at > install time by deleting one of the offending files. (Or through > some sort of backup mechanism.) Either makes it difficult to address > the issue later on. Also, name clashes can be reduced because not I'd rather address the problem at _install_. *IF* a name clashes, and this has never happened to me, I'd rather be notified then and there, and not have the install proceed until some has been done about it. > every file needs to be linked; only publically-visible files. > Private executables, data files, configuration files remain in the > package directories and therefore avoid name clash issues. Thus, Duh! Like, say, we presently do? You *DID* realize that we use directories named after the port for private files, didn't you? > > It does make it easy to "test-upgrade" a package, *IF* you solve the > > symlink name clash problem, which you didn't. > > One easy way to address this is to only link the most recent version > of a package. (e.g., if it finds foo-3.4 and foo-3.5 it can only link Clashes between the same version is an upgrade target problem, not a general problem. > are routine. With the current system, a tentative upgrade of a package > requires repeated de-installs and re-installs, losing any customized > settings in the process. Alternatively, it requires a backup system Unless, of course, you install them in a temporary directory for test? > to be implemented within the package tools. From where I sit, that > looks like an awfully complex answer to a simple problem. The funny thing is that this is what your proposal looks like to me. :-) > > OTOH, you *CAN* test-install a port on an alternate directory of > > your choice, without any problems. Though this does precludes use > > of pre-compiled binaries for this purpose, you'll notice that many > > people never use pre-compiled packages at all. > > That's true, you can circumvent any limitation of the package system > by simply not using it. What I'm doing is trying to make the > package system itself more useful so that more people can use it. cd /usr/ports/mail/elm; PREFIX=/testinstall make all install How's that not using the package system? > approach leads naturally to the second. Each has it's benefits and > drawbacks. In my experience, it's more common to discover "something is > wrong with emacs" than to have a user report that "some package is > mis-configured". In my experience it is more often necessary to make changes to various packages to reflect an environmental change than change multiple configuration files of a single package. > > Another big problem is that it would require retraining of personel and > > invalidate all existing documentation on our ports systems, and you can > > bet *THAT* will make people very pissed off. > > I never thought I'd see the day that an open-source > project requires discussion of "retraining of personel". My, how > times change. ;-) Huh? You think the people who have 4000 FreeBSD box installed has any less of a support/maintenance problem than the people who have 4000 Windows NT box installed? They both need to have people trained to give user support, to solve simple problems, and generally keep things going. > I'm actually suggesting no changes to the ports system at all. > Very little documentation should require changing. Ports that > are PREFIX-clean will require no changes at all. Only the package > tools require any changes. You completely miss the point. Go back up there and think again. > > The _only_ advantage your method bring is that people who like it like > > it. Well, people who don't like it, don't like it. And people who like > > the present method like the present method. > > I believe that many people who "like the present method" have > either not yet encountered some of its limitations or never considered > alternatives; that's my whole reason in continuing this discussion. Libh is not the place to do it. This is a small backwater list, subscribed by just a few people, most of whom have a lot experience with the different ways of dealing with packages used by everything from NT to Debian. > > ... and proposes a way of doing things I plainly don't like, > > having used it elsewhere. > > I've also used Solaris' "/opt" convention, and I don't like it either. Actually, I have little experience with Solaris. :-) > > Most people are HAPPY with the present way of doing things. > > Earlier, you claimed there were many people who never use the > package tools. I suspect those people aren't all that happy Because they prefer to use ports. Ports is the way of doing things, packages is a side-kick for quick and dirty results. My remark stands. > > If you want to solve name clashes, test-upgrades, faster installs > > and the like, there are plenty paths to take within the existing > > framework, which would make it much more likely the adoption of the code. > > Please explain how to fix the current tools. Your solution above > to tentative upgrades was to not use pre-compiled packages. I've > read Jordan's paper, and his suggestion of a complete versioning > database on top of /usr/local sounds both needlessly complex and > very brittle. (Certainly more complex than building shadow heirarchies > and symlinks.) Pre-compiled packages suck. They are, as far as I'm concerned, a non-issue, because they simply CANNOT be targetted at _your_ environment, compiled to take advantage of _your_ hardware, use the libs that _you_ have installed, the defaults _you_ like, etc. "Package installation," for me, is cd /usr/ports/Category/Application; make all install clean. Optimize for pkg_add is optimizing for the wrong target. You solve name clashes at the port, because that's really the only way of doing it. Otherwise, EVERYONE has to deal with it, individually. Upgrades you solve by keeping a list of what files need to be taken special attention of, scripts to handle them if necessary, and just paying attention to what IS in the database before installing. Faster installs are dealt with by removing the requirement of using a temporary directory, and this is done by using a smarter package format. > My most serious concern with the current approach relates to the > lack of any enforced relation between the package database and > the filesystem. There's no real solution for that, other than > to contradict several decades of standard Unix practice and > not use /usr/local for local software. Such a solution seems Err... /usr/local is NOT several decades of standard Unix practice. If you search the mailing lists, you'll find that /usr/local has been used for DIFFERENT purposes by different vendor and people. :-) I'll grant you that the solution you propose is the only one available if your most serious concern is the enforced relation between the package database and the filesystem. Since this is not a concern of mine, it's understandable that we see things different. > unenforcable to me. (The other solution is to move /usr/local > off of ufs and onto a custom filesystem, but that seems even > less practical, although no doubt someone has considered it.) I don't see how any custom filesystem could help, as a matter of fact. > At the very least, the default PREFIX should be changed to something > other than /usr/local. That would reduce the brittleness of the > current system. It wouldn't address many other issues, of course. Huh? Why something other than /usr/local? If you claim /usr/local is the place to install locally compiled software, you'll find plenty people disagreeing with you. And it would most certainly be easier for you to change such perception than for the hundreds of thousands of FreeBSD users to adapt to a new PREFIX. -- Daniel C. Sobral (8-DCS) dcs@newsguy.com dcs@freebsd.org capo@world.wide.bsdconspiracy.net He has been convicted of criminal possession of a clue with intent to distribute. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Fri Nov 3 6:35:16 2000 Delivered-To: freebsd-libh@freebsd.org Received: from weirdo.netcraft.com (weirdo.netcraft.com [195.188.192.47]) by hub.freebsd.org (Postfix) with ESMTP id 412CC37B4C5 for ; Fri, 3 Nov 2000 06:35:13 -0800 (PST) Received: (from sketchy@localhost) by weirdo.netcraft.com (8.11.1/8.11.1) id eA3EZ3Y94463 for libh@freebsd.org; Fri, 3 Nov 2000 14:35:03 GMT (envelope-from sketchy) Date: Fri, 3 Nov 2000 14:35:03 +0000 From: Jonathan Perkin To: libh@freebsd.org Subject: Re: Making the Packages System Better Message-ID: <20001103143503.B94079@netcraft.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i X-Operating-System: FreeBSD/i386 4.1.1-STABLE Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Talking about symlinks and stuff [snip] Just to say this has already been done, and I know quite a few people who use it on their linux systems. http://encap.cso.uiuc.edu/ Known as the encap package manager, it uses a userland tool "epkg" to manage the symlinks etc. Quite nifty, but I side with DES in that the symlinks get annoying after a while - basic stuff like viewing file permissions is impractical as you have to follow symlinks first just to get to the file. Debian's /etc/alternatives system is even worse: $ ls -l /usr/bin/vi lrwxrwxrwx 1 root root 20 Sep 27 10:32 /usr/bin/vi -> /etc/alternatives/vi $ ls -l /etc/alternatives/vi rwxrwxrwx 1 root root 12 Sep 29 17:06 /etc/alternatives/vi -> /usr/bin/nvi $ ls -l /usr/bin/nvi -rwxr-xr-x 3 root root 315248 Mar 26 2000 /usr/bin/nvi 3 commands just to get basic file perms? I can see a *lot* of people hating that very quickly. -- Jonathan Perkin Voice: +44 (01225) 404422 ech`echo xiun | tr nu oc | sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol System Administrator - Netcraft, Bath, UK - http://www.netcraft.com/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Fri Nov 3 6:46: 9 2000 Delivered-To: freebsd-libh@freebsd.org Received: from weirdo.netcraft.com (weirdo.netcraft.com [195.188.192.47]) by hub.freebsd.org (Postfix) with ESMTP id 3A8AC37B479 for ; Fri, 3 Nov 2000 06:46:07 -0800 (PST) Received: (from sketchy@localhost) by weirdo.netcraft.com (8.11.1/8.11.1) id eA3Ek3K94485 for libh@freebsd.org; Fri, 3 Nov 2000 14:46:03 GMT (envelope-from sketchy) Date: Fri, 3 Nov 2000 14:46:03 +0000 From: Jonathan Perkin To: libh@freebsd.org Subject: Re: Making the Packages System Better Message-ID: <20001103144603.C94079@netcraft.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i X-Operating-System: FreeBSD/i386 4.1.1-STABLE Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG I wrote: > manage the symlinks etc. Quite nifty, but I side with DES in that the s/DES/DCS/ :) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Fri Nov 3 6:59:26 2000 Delivered-To: freebsd-libh@freebsd.org Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by hub.freebsd.org (Postfix) with ESMTP id 2045437B4C5 for ; Fri, 3 Nov 2000 06:59:24 -0800 (PST) Received: (from des@localhost) by flood.ping.uio.no (8.9.3/8.9.3) id PAA40036; Fri, 3 Nov 2000 15:59:21 +0100 (CET) (envelope-from des@ofug.org) X-URL: http://www.ofug.org/~des/ X-Disclaimer: The views expressed in this message do not necessarily coincide with those of any organisation or company with which I am or have been affiliated. To: Jonathan Perkin Cc: libh@FreeBSD.ORG Subject: Re: Making the Packages System Better References: <20001103144603.C94079@netcraft.com> From: Dag-Erling Smorgrav Date: 03 Nov 2000 15:59:20 +0100 In-Reply-To: Jonathan Perkin's message of "Fri, 3 Nov 2000 14:46:03 +0000" Message-ID: Lines: 10 User-Agent: Gnus/5.0802 (Gnus v5.8.2) Emacs/20.4 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Jonathan Perkin writes: > > manage the symlinks etc. Quite nifty, but I side with DES in that the > s/DES/DCS/ Oh, I was starting to worry that I had an evil twin who posted to freebsd-libh while I was sleeping, or something. That, or my cats. DES -- Dag-Erling Smorgrav - des@ofug.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Fri Nov 3 11:57:52 2000 Delivered-To: freebsd-libh@freebsd.org Received: from atlas.dnai.com (atlas.dnai.com [207.181.194.95]) by hub.freebsd.org (Postfix) with ESMTP id 30DCD37B4F9 for ; Fri, 3 Nov 2000 11:57:45 -0800 (PST) Received: from azoth.dnai.com (azoth.dnai.com [207.181.194.94]) by atlas.dnai.com (8.9.3/8.9.3) with ESMTP id LAA54740; Fri, 3 Nov 2000 11:57:39 -0800 (PST) Received: from acm.org (207-172-166-33.s33.tnt1.sfrn.ca.dialup.rcn.com [207.172.166.33]) by azoth.dnai.com (8.9.3/8.9.3) with ESMTP id LAA55042; Fri, 3 Nov 2000 11:57:35 -0800 (PST) Message-ID: <3A0318AB.A0417517@acm.org> Date: Fri, 03 Nov 2000 11:57:31 -0800 From: Tim Kientzle Reply-To: kientzle@acm.org X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 3.3-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: Jonathan Perkin Cc: libh@FreeBSD.ORG Subject: Re: Making the Packages System Better References: <20001103143503.B94079@netcraft.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Jonathan Perkin wrote: > ... I side with DES in that the symlinks get annoying after a while > - basic stuff like viewing file permissions is impractical as you > have to follow symlinks first just to get to the file. man ls I don't know if the -H option is available on most Linux systems, but it's been a standard part of BSD for quite a while. For example, on my system right now, I could follow the symlink trail for /usr/local/bin/java: % ls -l /usr/local/bin/java lrwxr-xr-x 1 root wheel 20 Sep 1 20:30 /usr/local/bin/java@ -> ../jdk1.2.2/bin/java % ls -l /usr/local/jdk1.2.2/bin/java lrwxr-xr-x 1 root 1050 13 Aug 24 13:01 /usr/local/jdk1.2.2/bin/java@ -> .java_wrapper % ls -l /usr/local/jdk1.2.2/bin/.java_wrapper -r-xr-xr-x 1 root 1050 3228 Aug 23 17:33 /usr/local/jdk1.2.2/bin/.java_wrapper* Or I can type ls -lH and get the final file details in one shot: % ls -lH /usr/local/bin/java -r-xr-xr-x 1 root 1050 3228 Aug 23 17:33 /usr/local/bin/java* - Tim Kientzle To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Fri Nov 3 13: 0:45 2000 Delivered-To: freebsd-libh@freebsd.org Received: from oberon.dnai.com (oberon.dnai.com [207.181.194.97]) by hub.freebsd.org (Postfix) with ESMTP id 47EC737B4D7 for ; Fri, 3 Nov 2000 13:00:43 -0800 (PST) Received: from neptune.dnai.com (neptune.dnai.com [207.181.194.93]) by oberon.dnai.com (8.9.3/8.9.3) with ESMTP id NAA38065; Fri, 3 Nov 2000 13:00:42 -0800 (PST) Received: from acm.org (207-172-166-112.s112.tnt1.sfrn.ca.dialup.rcn.com [207.172.166.112]) by neptune.dnai.com (8.9.3/8.9.3) with ESMTP id NAA54389; Fri, 3 Nov 2000 13:00:41 -0800 (PST) Message-ID: <3A032753.E596C98D@acm.org> Date: Fri, 03 Nov 2000 13:00:03 -0800 From: Tim Kientzle Reply-To: kientzle@acm.org X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 3.3-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: "Daniel C. Sobral" Cc: libh@FreeBSD.ORG Subject: Re: Making the Packages System Better References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> <39FE2406.150CA3B1@newsguy.com> <00cb01c042f1$1a347190$040aa8c0@local.mindstep.com> <39FE562C.714DBE7C@newsguy.com> <39FFCD73.7364C2BF@acm.org> <39FFE5B7.8FC4216B@newsguy.com> <3A0063F5.4798055B@acm.org> <3A00CC6B.39522CF0@newsguy.com> <3A01BDA9.C637C0AA@acm.org> <3A02209D.5E7137F8@newsguy.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG "Daniel C. Sobral" wrote: > Actually, I have little experience with Solaris. :-) Lucky you. ;-( > "Package installation," for me, is cd /usr/ports/Category/Application; > make all install clean. Optimize for pkg_add is optimizing for the wrong > target. The current ports are already optimized for pkg_add, and I have yet to find anything in the existing ports system that would require changing current ports to support the scheme I suggest. If you don't use pre-compiled software packages, you will be entirely unaffected by any change to how they work. > Upgrades you solve by keeping a list of what files need to be taken > special attention of, scripts to handle them if necessary, and just > paying attention to what IS in the database before installing. The problem isn't just upgrades, but repeated upgrades/downgrades. The scheme I propose would allow the pkg_* tools to deal with that cleanly in most cases. (For example, I recently had five different versions of java installed on my system and had to switch between them a number of times to find the one whose bugs and/or deficiencies least affected my work. The current FreeBSD packages generally don't handle this situation gracefully. I had to deal with this manually--with some assistance from the ports collection---and I would rather not have to.) > > ... several decades of standard Unix practice ... > > use /usr/local for local software. > > ... /usr/local has been used for DIFFERENT purposes by different > vendor and people. :-) Exactly my point. > > At the very least, the default PREFIX should be changed to something > > other than /usr/local. That would reduce the brittleness of the > > current system. It wouldn't address many other issues, of course. > > Huh? Why something other than /usr/local? Because, as you pointed out, "/usr/local has been used for DIFFERENT purposes by different vendor and people". People have different (strongly-held) opinions about the use of /usr/local; FreeBSD's package system should steer clear of such issues and leave /usr/local alone for individual sysadmins to use as they see fit. If they want to fix PREFIX to /usr/local, let them; but, if they'd rather use it some other way, FreeBSD's packages should not dissuade that. Also, something to remember: installation/upgrade/deinstallation isn't just a question of what's in the database. It's more importantly a question of what's in the filesystem. If those two get out of sync for any reason, the filesystem should be the definitive source, not the database. (I've heard many complaints about RPM, for example, in part because it doesn't consider the filesystem as a place where dependencies might live. As a result, it will often refuse to install something if the dependency wasn't installed from an RPM.) - Tim Kientzle To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message From owner-freebsd-libh Fri Nov 3 19:57:57 2000 Delivered-To: freebsd-libh@freebsd.org Received: from peach.ocn.ne.jp (peach.ocn.ne.jp [210.145.254.87]) by hub.freebsd.org (Postfix) with ESMTP id 59A3937B4D7 for ; Fri, 3 Nov 2000 19:57:54 -0800 (PST) Received: from newsguy.com (p03-dn03kiryunisiki.gunma.ocn.ne.jp [210.232.224.132]) by peach.ocn.ne.jp (8.9.1a/OCN/) with ESMTP id MAA16510; Sat, 4 Nov 2000 12:56:39 +0900 (JST) Message-ID: <3A038899.681CB9DD@newsguy.com> Date: Sat, 04 Nov 2000 12:55:05 +0900 From: "Daniel C. Sobral" X-Mailer: Mozilla 4.7 [en] (Win98; I) X-Accept-Language: en,pt-BR MIME-Version: 1.0 To: kientzle@acm.org Cc: libh@FreeBSD.ORG Subject: Re: Making the Packages System Better References: <39DCC860.B04F7D50@acm.org> <20001006155542.A29218@cichlids.cichlids.com> <39F3CDD7.15B889E7@acm.org> <20001023190412.B507@cichlids.cichlids.com> <39F47E98.4BB647AA@acm.org> <20001023202244.B10374@cichlids.cichlids.com> <39F48F4A.38D458C2@acm.org> <39FCF244.5A8C8E59@newsguy.com> <39FDC12E.304B0011@acm.org> <39FE2406.150CA3B1@newsguy.com> <00cb01c042f1$1a347190$040aa8c0@local.mindstep.com> <39FE562C.714DBE7C@newsguy.com> <39FFCD73.7364C2BF@acm.org> <39FFE5B7.8FC4216B@newsguy.com> <3A0063F5.4798055B@acm.org> <3A00CC6B.39522CF0@newsguy.com> <3A01BDA9.C637C0AA@acm.org> <3A02209D.5E7137F8@newsguy.com> <3A032753.E596C98D@acm.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-libh@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Tim Kientzle wrote: > > > "Package installation," for me, is cd /usr/ports/Category/Application; > > make all install clean. Optimize for pkg_add is optimizing for the wrong > > target. > > The current ports are already optimized for pkg_add, and I have > yet to find anything in the existing ports system that would > require changing current ports to support the scheme I suggest. > If you don't use pre-compiled software packages, you will be > entirely unaffected by any change to how they work. I beg your pardon, but having pkg_info, installing things elsewhere, etc, *DO* change the way things work. Your changes are *not* related to pre-compiled packages! Whether you get your stuff from a .tgz (or whatever), or you get your stuff from a directory that just has been compiled is irrelevant wrt your changes. It's the *management* from there on that changes. > > > ... several decades of standard Unix practice ... > > > use /usr/local for local software. > > > > ... /usr/local has been used for DIFFERENT purposes by different > > vendor and people. :-) > > Exactly my point. And, in FreeBSD in particular, it has been used for ports since 1994. > > Huh? Why something other than /usr/local? > > Because, as you pointed out, "/usr/local has been used for DIFFERENT > purposes by different vendor and people". People have different > (strongly-held) opinions about the use of /usr/local; FreeBSD's Some do. In many systems /usr/local has never been used for anything. We have two main targets here. One, our userbase, which has been using /usr/local our way since they started using FreeBSD, for how long that may have been. Two, new users, who, by and large, have not been exposed to any Unix hierarchy, with the possibly exception of one of the many Linux distributions. The people who have been using /usr/local for something other than we do and have strongly-held opinions are in minority. In particular, they are most likely in minority compared to our user base. > package system should steer clear of such issues and leave /usr/local > alone for individual sysadmins to use as they see fit. If they > want to fix PREFIX to /usr/local, let them; but, if they'd rather > use it some other way, FreeBSD's packages should not dissuade that. That would have been an option six years ago. Now, it is not. > Also, something to remember: installation/upgrade/deinstallation > isn't just a question of what's in the database. It's more > importantly a question of what's in the filesystem. If those > two get out of sync for any reason, the filesystem should > be the definitive source, not the database. (I've heard > many complaints about RPM, for example, in part because > it doesn't consider the filesystem as a place where > dependencies might live. As a result, it will often > refuse to install something if the dependency wasn't > installed from an RPM.) That way lies autoconf, which I'm not even dignifying to comment on. -- Daniel C. Sobral (8-DCS) dcs@newsguy.com dcs@freebsd.org capo@world.wide.bsdconspiracy.net He has been convicted of criminal possession of a clue with intent to distribute. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-libh" in the body of the message