From owner-freebsd-arch Sun Jul 7 19: 0:52 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A5CEC37B400; Sun, 7 Jul 2002 19:00:47 -0700 (PDT) Received: from mail.rpi.edu (mail.rpi.edu [128.113.22.40]) by mx1.FreeBSD.org (Postfix) with ESMTP id EDE1743E31; Sun, 7 Jul 2002 19:00:46 -0700 (PDT) (envelope-from drosih@rpi.edu) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by mail.rpi.edu (8.12.1/8.12.1) with ESMTP id g6820jiI066802; Sun, 7 Jul 2002 22:00:45 -0400 Mime-Version: 1.0 X-Sender: drosih@mail.rpi.edu Message-Id: In-Reply-To: <3D27A296.D58FB4B4@softweyr.com> References: <20020706220511.GA88651@scoobysnax.jaded.net> <3D27A296.D58FB4B4@softweyr.com> Date: Sun, 7 Jul 2002 22:00:43 -0400 To: Wes Peters , Dan Moschuk From: Garance A Drosihn Subject: Re: Package system flaws? Cc: arch@FreeBSD.ORG Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-Scanned-By: MIMEDefang 2.3 (www dot roaringpenguin dot com slash mimedefang) Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG At 7:08 PM -0700 7/6/02, Wes Peters wrote: >Dan Moschuk wrote: > > I'm not sure that this classifies as an architectual discussion > > (for now) so if you feel its appropriate please feel free to > > redirect to ports@. > >I'd certainly like to see it evolve into an architectural >discussion, even if all it accomplishes is to layout the work >that needs to be done and provide some sort of road map of what >the next step or two might be. There is probably room for several different architectural discussions. One is whether we use tar vs some other archive format to hold the data and the metadata for each port. I agree that is an important topic, but I have no strong opinions on it. Whatever is decided upon by others would probably work fine for me. I do find the ports collection one of the most useful things in the freebsd community, but the more comfortable you get with depending on it, the more you see little cracks where various problems crop up -- over and over again. - - - - Separate from that, one thing I think we need is some mechanism which makes certain that the generated package-list for a port is exactly correct. I install about 70-80 ports, and I know several of them install files which are not included in their package-list. Try running find /usr/local /usr/X11R6 -type f -print0 | \ xargs -0 /usr/local/sbin/pkg_which -v | fgrep '?' Whatever we do for this, it should be something which is simple enough to use that it does not add work for the ports maintainers. Conceptually what I would like to do is have ports *built* for /usr/local (just as they are now), but then when it comes time to do the make-install, the make-install would do something like: (chroot /usr/local/TEMP ; make install) and then copy all files from /usr/local/TEMP/usr/local to the real /usr/local, based on the package-list for the port. It obviously wouldn't be quite *that* simple to do, but the idea is that you could be certain that the package list included every file that was actually installed. It would also be easy to see every file which SHOULD have been installed, because they would all be still sitting in the /usr/local/TEMP/usr/local directories. I believe that openbsd has something like this, but I do not know the details. I have also thought that maybe it could be done with Kirk's filesystem snapshots, where you'd do a snapshot before building a port, and another one afterwards, and find out what files it changed. This would be a rather heavy-handed way to approach the problem, but it avoids some of the technical issues that come up with the chroot approach. - - - - Portupgrade is a great step forward in rebuilding ports, but it also has it's limitations. It has an idea of dependences between ports, but it only knows about the dependences as of the last time the port was built. If progAA was built a month ago, and at that time it only depended on progBB, then 'portupgrade -Rr -n progAA' will only consider progAA and progBB. If the *new* version of progAA (the one you will build) now depends on progCC, you will not find out about that until portupgrade does the 'make', and it turns around to build progCC. This also shows up if you have NOCLEANDEPENDS=true in your /etc/make.conf file. Portupgrade will do a: make clean ; make ; make install; make clean for every port that it builts -- but it only knows to do that for the ports that it *already* knows about. If you ask it to install progAA, that depends on progBB and progCC, then those 'make clean's are only done for progAA. The work-directories for progBB and progCC are not cleaned out before they are made, and they are not cleaned out after they are installed. - - - - Portupgrade also looses track of what dependencies are, as various things get rebuilt. I just pkg_deinstall'ed a number of ports, and pkg_deinstall missed a good number of package dependencies. By that I mean it quite happy removed some port, even though there were other ports still installed which in fact depended on that port. [that was fine for me in this case, but it was still a little disappointing...] That's a problem when things get new versions. It gets even more confusing when ports are renamed, or entirely removed. - - - - The current ports mechanism has the idea "this port depends on file XYZZY, and if that file does not exist then you should first build port PLOUGH". It also needs the idea that "this port depends on XYZZY, but it must the XYZZY built by XFree86-4, and not the one built by XFree86-3". - - - - As we move into more platforms (sparc64, ia-64, ppc), then I expect that will add even more to the work of the ports collection. - - - - In addition to the above issues, there is also the issue of how large and unwieldy the ports collection is, and how it's "one single giant collection" (as far as something like the INDEX is concerned, for instance). It would be nice to rearrange it a bit, such that I can easily clip off all foreign-language ports that I know I don't want. Right now I can kinda do that by changing my cvsup file, and changing my portupgrade-config file, and editing /usr/ports/Makefile, and... >I'm more interested in the binary package side personally, a >holdover from a previous professional involvement in this area. >I have a number of ideas for things that could be improved in >relatively small projects if someone wants to discuss those >with me. I have had a number of ideas, but no time to really pursue them. I think we try to stuff too much information into the name of a port, and we try to do too much to shoehorn all ports-processing into standard makefile variables and standard make-cmd processing. [in the background, the sound of people screaming is heard...] And lastly, one of the problems with our ports and package system is that it has grown so large without addressing some of these issues. Any attempt to switch to a significantly different mechanism would have to include a plan to EASE INTO the new approach, probably over months of time. There is no way we are going to convert 7,000 ports to any new mechanism, on multiple HW platforms, on a wide-range of freebsd-versions, in some single one-week cut-over period. -- Garance Alistair Drosehn = gad@gilead.netel.rpi.edu Senior Systems Programmer or gad@freebsd.org Rensselaer Polytechnic Institute or drosih@rpi.edu To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message