From owner-freebsd-arch Sun Jul 7 15: 2:35 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CFAFD37B400; Sun, 7 Jul 2002 15:02:29 -0700 (PDT) Received: from spqr.osg.gov.bc.ca (spqr.osg.gov.bc.ca [142.32.102.24]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4862C43E31; Sun, 7 Jul 2002 15:02:29 -0700 (PDT) (envelope-from Cy.Schubert@osg.gov.bc.ca) Received: from passer.osg.gov.bc.ca (passer.osg.gov.bc.ca [142.32.110.29]) by spqr.osg.gov.bc.ca (Postfix) with ESMTP id 0AA729EF16; Sun, 7 Jul 2002 15:02:29 -0700 (PDT) Received: from cwsys.cwsent.com (cwsys2 [10.1.2.1]) by passer.osg.gov.bc.ca (8.12.5/8.12.3) with ESMTP id g67M2KOX026307; Sun, 7 Jul 2002 15:02:20 -0700 (PDT) (envelope-from cy@cwsent.com) Received: from cwsys (localhost [127.0.0.1]) by cwsys.cwsent.com (8.12.5/8.12.3) with ESMTP id g67M2IfP015705; Sun, 7 Jul 2002 15:02:19 -0700 (PDT) (envelope-from cy@cwsys.cwsent.com) Message-Id: <200207072202.g67M2IfP015705@cwsys.cwsent.com> X-Mailer: exmh version 2.5 07/13/2001 with nmh-1.0.4 Reply-To: Cy Schubert - CITS Open Systems Group From: Cy Schubert - CITS Open Systems Group X-os: FreeBSD X-Sender: cy@cwsent.com To: Wes Peters Cc: Dag-Erling Smorgrav , Dan Moschuk , arch@FreeBSD.ORG Subject: Re: Package system flaws? In-Reply-To: Message from Wes Peters of "Sun, 07 Jul 2002 12:23:21 PDT." <3D289529.95C1EC8A@softweyr.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sun, 07 Jul 2002 15:02:18 -0700 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG In message <3D289529.95C1EC8A@softweyr.com>, Wes Peters writes: > Dag-Erling Smorgrav wrote: > > > > Wes Peters writes: > > > Dan Moschuk wrote: > > > > Where can we improve? > > > In all of the above areas, plus all the ones we haven't addressed yet. > > > > *grin* > > > > One thing we could improve a lot is the package file format. We > > currently use gzipped tarballs, which have to be completely unpacked > > before processing can begin. One improvement we can make is to use an > > archive format such as zip that allows us to extract individual files > > quickly, so we can extract the metadata and check dependencies > > etc. without unpacking the entire package. > > I'd rather go one step further and separate the metadata from the files > completely. I'm thinking an XML-ish format for the metadata, with > blobs of files stored in whatever format makes the most sense inter- > spersed. Zip format makes a lot of sense and is available in library > format, but I could easily tolerate tar or cpio, as well as a variety > of compression formats. The metadata should not be compressed for > speed of package processing. This seems like a reasonable approach. I don't think that the format of the files matters that much. The zip file format has a couple of advantages. First, as you mention, it's available in library format. Secondly, vendors like Sun are using this format to distribute patches. I think this format will quickly become the UNIX standard, superseding tar. Whether metadata is compressed or not is immaterial. With the speed of processors and the price of disk these days, I don't think it matters either way, except for possibly network congestion. I suspect that metadata would not be the lions share of the data anyhow. > > This would also allow us to separate the files in the package into > distinct pieces, and apply metadata to the collections of files. > Picture, for instance, a package that installs an X application. We > have executable files that need to go into $PKG_BIN, perhaps a library > or two that needs to in $PKG_LIB, an application resource file that > needs to go into $X_APPRESDIR and some fonts that need to be installed > in $X_FONTDIR. We provide defaults for each of these, let the user > override them as necessary. So far so good. I would also include the possibility of source being distributed with a package. In my previous life as an MVS systems programmer, packages (in MVS they're called functions) may be delivered in source form, binary form, or both. For example, JES/2 (Job Entry Subsystem/2), one of the MVS spooling subsystems, was distributed in source and object form. When JES would be installed, sources and binaries were installed by the same package (in different locations of course). I as the systems programmer for a site could apply fixes (PTF's and APARFIXES) or apply my own modifications to the code (USERMODS) to the source. SMP/E would automatically invoke the assembler to rebuild any affected binaries. Alternatively I could apply fixes or modifications in binary form (using an application called superzap -- IMASPZAP). (Pretty scary that I can remember this stuff after not having looked at it for 10 years.) When upgrading SMP/E would flag my USERMODS that were being backed out in order to accommodate the upgrade. This way I as the systems programmer for a site wouldn't have to remember a lot of details about my system, SMP/E did all that for me. I personally used SMP/E to update system config files too. That saved me from having to write a lot of stuff down because it was already recorded in my target zone (database). Dates and times and all that good stuff were recored. I don't think we need anything as elaborate as this. IMO not many FreeBSD users make custom modifications to their systems. Those of us who do, myself included, probably need our heads examined (or in my case I should make a case to have them applied to the source tree). However in the MVS world, I have never worked at a site that did not customize the source or apply binary patches in any way. Different requirements and different culture I guess. All of this helped make SMP/E the monstrosity it became. It was great for techno-wizards but for the average MVS guy, I've been told it was too much. I think we need to find the happy medium between functionality and scaryness. Much of this stuff is pretty useless here, but if you want to use any of these ideas, be my guest. > > > This would save both time > > and space (the current system can fail if the temp directory doesn't > > have room for the unpacked package, even if the installation directory > > has room to spare). A further improvement is to use a custom archive > > format that always stores the metadata at the beginning of the archive > > so we can install packages from the net without having to download and > > store them locally first (zip isn't suitable for this as it stores the > > content directory at the end of the archive, and the files within the > > archive can appear in any order). > > Yes, an excellent idea. With a format as I've proposed, it would be > possible to specify the file collections as URLs rather than encoded > archives. You could download the entire package collection in about > the same amount of space as the current ports collection. Of course > this means pkg_add would have to be linked with libfetch, too. ;^) This too is a good idea. However I personally like to keep local copy of package distribution files locally just in case they're needed for DRP recovery. In a DRP situation one may not want to install the latest version of a package, because the only focus should be to get the application(s) working again, not upgrade the system, and there is no guarantee that a certain version of a package will be there days, months, or even years from now. Keeping one's own copy is the only way to guarantee that. The option to fetch and store the files that comprise a package might be worthwhile looking at. -- Cheers, Phone: 250-387-8437 Cy Schubert Fax: 250-387-5766 Team Leader, Sun/Alpha Team Email: Cy.Schubert@osg.gov.bc.ca Open Systems Group, CITS Ministry of Management Services Province of BC FreeBSD UNIX: cy@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message