Date: Sun, 07 Jul 2002 15:02:18 -0700 From: Cy Schubert - CITS Open Systems Group <Cy.Schubert@uumail.gov.bc.ca> To: Wes Peters <wes@softweyr.com> Cc: Dag-Erling Smorgrav <des@ofug.org>, Dan Moschuk <dan@FreeBSD.ORG>, arch@FreeBSD.ORG Subject: Re: Package system flaws? Message-ID: <200207072202.g67M2IfP015705@cwsys.cwsent.com> In-Reply-To: Message from Wes Peters <wes@softweyr.com> of "Sun, 07 Jul 2002 12:23:21 PDT." <3D289529.95C1EC8A@softweyr.com>
next in thread | previous in thread | raw e-mail | index | archive | help
In message <3D289529.95C1EC8A@softweyr.com>, Wes Peters writes:
> Dag-Erling Smorgrav wrote:
> >
> > Wes Peters <wes@softweyr.com> writes:
> > > Dan Moschuk wrote:
> > > > Where can we improve?
> > > In all of the above areas, plus all the ones we haven't addressed yet.
> >
> > *grin*
> >
> > One thing we could improve a lot is the package file format. We
> > currently use gzipped tarballs, which have to be completely unpacked
> > before processing can begin. One improvement we can make is to use an
> > archive format such as zip that allows us to extract individual files
> > quickly, so we can extract the metadata and check dependencies
> > etc. without unpacking the entire package.
>
> I'd rather go one step further and separate the metadata from the files
> completely. I'm thinking an XML-ish format for the metadata, with
> blobs of files stored in whatever format makes the most sense inter-
> spersed. Zip format makes a lot of sense and is available in library
> format, but I could easily tolerate tar or cpio, as well as a variety
> of compression formats. The metadata should not be compressed for
> speed of package processing.
This seems like a reasonable approach. I don't think that the format
of the files matters that much. The zip file format has a couple of
advantages. First, as you mention, it's available in library format.
Secondly, vendors like Sun are using this format to distribute patches.
I think this format will quickly become the UNIX standard, superseding
tar.
Whether metadata is compressed or not is immaterial. With the speed of
processors and the price of disk these days, I don't think it matters
either way, except for possibly network congestion. I suspect that
metadata would not be the lions share of the data anyhow.
>
> This would also allow us to separate the files in the package into
> distinct pieces, and apply metadata to the collections of files.
> Picture, for instance, a package that installs an X application. We
> have executable files that need to go into $PKG_BIN, perhaps a library
> or two that needs to in $PKG_LIB, an application resource file that
> needs to go into $X_APPRESDIR and some fonts that need to be installed
> in $X_FONTDIR. We provide defaults for each of these, let the user
> override them as necessary.
So far so good.
I would also include the possibility of source being distributed with a
package. In my previous life as an MVS systems programmer, packages
(in MVS they're called functions) may be delivered in source form,
binary form, or both. For example, JES/2 (Job Entry Subsystem/2), one
of the MVS spooling subsystems, was distributed in source and object
form. When JES would be installed, sources and binaries were installed
by the same package (in different locations of course). I as the
systems programmer for a site could apply fixes (PTF's and APARFIXES)
or apply my own modifications to the code (USERMODS) to the source.
SMP/E would automatically invoke the assembler to rebuild any affected
binaries. Alternatively I could apply fixes or modifications in binary
form (using an application called superzap -- IMASPZAP). (Pretty scary
that I can remember this stuff after not having looked at it for 10
years.)
When upgrading SMP/E would flag my USERMODS that were being backed out
in order to accommodate the upgrade. This way I as the systems
programmer for a site wouldn't have to remember a lot of details about
my system, SMP/E did all that for me. I personally used SMP/E to
update system config files too. That saved me from having to write a
lot of stuff down because it was already recorded in my target zone
(database). Dates and times and all that good stuff were recored.
I don't think we need anything as elaborate as this. IMO not many
FreeBSD users make custom modifications to their systems. Those of us
who do, myself included, probably need our heads examined (or in my
case I should make a case to have them applied to the source tree).
However in the MVS world, I have never worked at a site that did not
customize the source or apply binary patches in any way. Different
requirements and different culture I guess.
All of this helped make SMP/E the monstrosity it became. It was great
for techno-wizards but for the average MVS guy, I've been told it was
too much. I think we need to find the happy medium between
functionality and scaryness.
Much of this stuff is pretty useless here, but if you want to use any
of these ideas, be my guest.
>
> > This would save both time
> > and space (the current system can fail if the temp directory doesn't
> > have room for the unpacked package, even if the installation directory
> > has room to spare). A further improvement is to use a custom archive
> > format that always stores the metadata at the beginning of the archive
> > so we can install packages from the net without having to download and
> > store them locally first (zip isn't suitable for this as it stores the
> > content directory at the end of the archive, and the files within the
> > archive can appear in any order).
>
> Yes, an excellent idea. With a format as I've proposed, it would be
> possible to specify the file collections as URLs rather than encoded
> archives. You could download the entire package collection in about
> the same amount of space as the current ports collection. Of course
> this means pkg_add would have to be linked with libfetch, too. ;^)
This too is a good idea. However I personally like to keep local copy
of package distribution files locally just in case they're needed for
DRP recovery. In a DRP situation one may not want to install the
latest version of a package, because the only focus should be to get
the application(s) working again, not upgrade the system, and there is
no guarantee that a certain version of a package will be there days,
months, or even years from now. Keeping one's own copy is the only way
to guarantee that. The option to fetch and store the files that
comprise a package might be worthwhile looking at.
--
Cheers, Phone: 250-387-8437
Cy Schubert Fax: 250-387-5766
Team Leader, Sun/Alpha Team Email: Cy.Schubert@osg.gov.bc.ca
Open Systems Group, CITS
Ministry of Management Services
Province of BC
FreeBSD UNIX: cy@FreeBSD.org
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200207072202.g67M2IfP015705>