Date: Mon, 15 Jul 2013 08:23:08 +0200 From: Baptiste Daroussin <bapt@freebsd.org> To: Devin Teske <dteske@freebsd.org> Cc: Matthew Seaman <matthew@FreeBSD.org>, "current@FreeBSD.org" <current@freebsd.org>, Peter Wemm <peter@wemm.org> Subject: Re: [HEADSUP] No more pkg_install on HEAD by default Message-ID: <20130715062308.GE1516@ithaqua.etoilebsd.net> In-Reply-To: <13CA24D6AB415D428143D44749F57D7201FC547A@ltcfiswmsgmb21> References: <13CA24D6AB415D428143D44749F57D7201FC2DBD@ltcfiswmsgmb21> <20130713080732.GV85556@ithaqua.etoilebsd.net> <13CA24D6AB415D428143D44749F57D7201FC3AA2@ltcfiswmsgmb21> <13CA24D6AB415D428143D44749F57D7201FC3C92@ltcfiswmsgmb21> <CA%2B7WWSe7C2UXAsEX3OZH7nPJJbxotpQ_iYBxgake4R68sbUtLA@mail.gmail.com> <13CA24D6AB415D428143D44749F57D7201FC3FAA@ltcfiswmsgmb21> <CAGE5yCoH2auer_kKpUT_caFUZPpVM5TdAFH5tJcGgF4Ji12f0g@mail.gmail.com> <13CA24D6AB415D428143D44749F57D7201FC4E0D@ltcfiswmsgmb21> <51E26FD3.4020208@FreeBSD.org> <13CA24D6AB415D428143D44749F57D7201FC547A@ltcfiswmsgmb21>
next in thread | previous in thread | raw e-mail | index | archive | help
--jKBxcB1XkHIR0Eqt Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jul 14, 2013 at 10:12:19AM +0000, Teske, Devin wrote: >=20 > On Jul 14, 2013, at 2:30 AM, Matthew Seaman wrote: >=20 > > On 14/07/2013 06:48, Teske, Devin wrote: > >> Question: Where can I learn more about the actual format of what's in > >> the new tarballs? This is going to be important not for bsdconfig, > >> but $work (we have our own build platform; I'm going to have to > >> rewrite it from mastering PLIST files to mastering YAML MANIFEST > >> files and I want to know all the gritty details). > >=20 > > We do need a pkg-manifest(5) man page, which can double as a > > pkg-tarball(5) page since the manifest file is where most of the > > interesting bits are. > >=20 > > A pkg tarball is a compressed tar archive like so: > >=20 > > lucid-nonsense:...cache/pkg/All:% tar -tvf pkg-1.1.4.txz > > -rw-r--r-- 0 root wheel 530 Jan 1 1970 +COMPACT_MANIFEST > > -rw-r--r-- 0 root wheel 6385 Jan 1 1970 +MANIFEST > > -rw-r--r-- 0 root wheel 17567 Jan 1 1970 +MTREE_DIRS > > -r--r--r-- 0 root wheel 19453 Jul 7 12:26 > > /usr/local/etc/bash_completion.d/_pkg.bash > > -r-xr-xr-x 0 root wheel 629 Jul 7 12:26 > > /usr/local/etc/periodic/daily/400.status-pkg > > -r-xr-xr-x 0 root wheel 823 Jul 7 12:26 > > /usr/local/etc/periodic/daily/411.pkg-backup > > -r-xr-xr-x 0 root wheel 678 Jul 7 12:26 > > /usr/local/etc/periodic/daily/490.status-pkg-changes > > -r-xr-xr-x 0 root wheel 2558 Jul 7 12:26 > > /usr/local/etc/periodic/security/410.pkg-audit > > -r-xr-xr-x 0 root wheel 611 Jul 7 12:26 > > /usr/local/etc/periodic/security/460.pkg-checksum > > -r--r--r-- 0 root wheel 839 Jul 7 12:26 > > /usr/local/etc/pkg.conf.sample > > -r--r--r-- 0 root wheel 43432 Jul 7 12:26 /usr/local/include/pkg.h > > -r--r--r-- 0 root wheel 727558 Jul 7 12:26 /usr/local/lib/libpkg.a > > lrwxr-xr-x 0 root wheel 0 Jul 7 12:26 /usr/local/lib/libpkg.so > > -> libpkg.so.1 > > -r--r--r-- 0 root wheel 1227064 Jul 7 12:26 /usr/local/lib/libpkg.s= o.1 > > -rw-r--r-- 0 root wheel 187 Jul 7 12:26 > > /usr/local/libdata/pkgconfig/pkg.pc > > [... etc ...] > >=20 >=20 > Interesting. I notice that (while looking ahead to see a prefix: of /usr/= local in the +MANIFEST), the tarball itself has files that include /usr/loc= al in their path. >=20 > Does pkg handle packages where the prefix (in +MANIFEST) is "/usr/local" = _but_ "tar tf" of the unxz'd tarball is _not_ "/usr/local"? (e.g. pkg_insta= ll style) or is this the new way going forward? >=20 >=20 >=20 > > There must at least be a +MANIFEST -- other meta data files are > > optional. +COMPACT_MANIFEST is a subset of the full +MANIFEST. They're > > both YAML documents. +COMPACT_MANIFEST looks like this, for example: > >=20 > > --- > > name: pkg > > version: 1.1.4 > > origin: ports-mgmt/pkg > > comment: New generation package manager > > arch: freebsd:9:x86:64 > > www: https://urldefense.proofpoint.com/v1/url?u=3Dhttp://wiki.freebsd.o= rg/pkgng&k=3D%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=3DMrjs6vR4%2Faj2Ns9%2Fss= HJjg%3D%3D%0A&m=3DLPoff3ddLDoHE30AU6R5vgBklqxsjVAosXGjJop6uO4%3D%0A&s=3D30b= 27cf33d7e594144fc38f20bc75d84464788e715f7fc82b2495f813fd4be2d > > maintainer: portmgr@FreeBSD.org > > prefix: /usr/local > > licenselogic: single > > licenses: > > - BSD > > flatsize: 6311507 > > desc: | > > New Generation package management tool for FreeBSD > >=20 > > WWW: http://wiki.freebsd.org/pkgng > > categories: > > - ports-mgmt > > shlibs_required: > > - libpkg.so.1 > > shlibs_provided: > > - libpkg.so.1 > > message: | > > If you are upgrading from the old package format, first run: > >=20 > > # pkg2ng > >=20 >=20 > Nice. Very nice. >=20 >=20 >=20 > > +MTREE_DIRS is a compatibility thing with the old pkg_tools. It's not > > needed in general as +MANIFEST can provide all that meta data itself. > > It isn't going to be deprecated for at least as long as the ports tree > > continues to support pkg_tools though. > >=20 > > Beyond that, the rest of the pkg tarball just contains a tar archive of > > all the files, directories, sym-links etc to be installed by the > > package. Note that pkg(8) has no problem with creating an empty > > directory for a package, unlike pkg_tools. > >=20 >=20 > Excellent! >=20 >=20 > > Now, while you can grovel through the details of pkg tarballs and > > internal data formats like this, be aware: the format of the manifest > > and the details of the meta-data included in the pkg-tarballs is subject > > to change without warning as we develop pkg(8) further. We only promise > > API stability through the pkg(8) commands or for the libpkg.so library > > functions; reports of breakage from usage outside those APIs will > > receive little sympathy. > >=20 >=20 > Understood. >=20 > So just to give you a better idea of how we manage packages here. >=20 > We've realized that if you want to package for FreeBSD (in 9 and older), = you could get by alright if you knew how to create a +CONTENTS file from sc= ratch. For RedHat, it's the SPECFILE. And now for FreeBSD 10, it looks an a= wful lot like a SPECFILE (they are both in-fact YAML). The only thing you need if you really want to package is a small subset of = the manifest written in YAML: Have a look at what the ports tree do to create the minimalistic manifest a= nd what form it has and you will see how easy it is. It is so awful and complicated that there are already a load of custom scri= pts to create packages with pkg without using the ports tree. As an example: https://github.com/z0nt/pkg , but there is way more. I for example wrote in the past a plugin for pkg to be able to parse and USE RPM's specfiles as an input for pkg. >=20 > So rather than teach the people here how to use tools, I taught them what= I think is more important... how to manage +CONTENTS, SPECFILE, and now up= -and-coming, +MANIFEST. The problem is that you compare 2 things that cannot be compared, SPECFILE = is the equivalent of pkg + the ports tree. In fact specfiles are the equivalent of a given "port" !!! Are you writing = rpm files directly by hand? there is no SPECFILE inside RPM but rather a header container metadata in binary format and a body which is a cpio.(gz,bz2,xz). You are not creating the RPM file directly but rather create a SPECFILE, the equivalent on FreeBSD is to create a port!!! the +MANIFEST as used to be the +CONTENTS are the equivalent of the RPM metadata. We wrote them in YAML so = that they are easy to debug and that we can use third party well used and tested parsers instead of writting our own. >=20 > However, I'd be lying if I said I taught them *all* the gory details. In = reality, for +CONTENTS they edit a PLIST which is essentially +CONTENTS wit= hout the "@comment MD5:" entries. For SPECFILE, they manage the full thing = except there's a small section that is the standard RPM bootstrap that is l= abeled as "do not touch". >=20 > From what you posted of +COMPACT_MANIFEST was nice, except it lacks the l= ist of files. >=20 > The basic principle here is that if you have to maintain a list of files,= and that list ends up being compiled into a +MANIFEST, why not just teach = your peers how to read/write/maintain +MANIFEST files (in version control o= f course) so that there are never any mysteries as to how the package perfo= rms. Have a closer look at pkg create, you will see that we are still able to us= e the plist or +CONTENTS as a possible input for pkg create, it is even the main = way to create packages. Once again, the main and most important requirement to build packages was t= o be compatible with the ports tree, and the ports tree is using plist/+CONTENTS. and pkg2ng does only works by using the +CONTENTS. >=20 > I know this sounds screwy... but (as with our other YAML files), it's rea= lly nice because (as is the case with SPECFILE), we version-control things = as-close to the end-product as possible (take for example the case of "pre-= install" or "post-install" et cetera scripts -- I'm sure a tool like "pkg c= reate" or "pkg_create" or other could do a good job of assembling the piece= s into the +MANIFEST, but then it's not being version-controlled as closely. You can keep what you have already done. Once again pkg does not have the equivalent of the SPECFILE because we have= the ports tree! so we do not need to provide a custom package building system. We could, as it is really easy, but we do not need so it is not there yet, = as said earlier I already wrote a pkg plugin that is able to build packages fr= om a RPM specfile, I also modified archlinux's makepkg to be able to output pkgn= g files. I did the same on slackare, etc. Here the limitation is not technical at al= l. Bapt --jKBxcB1XkHIR0Eqt Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlHjlUwACgkQ8kTtMUmk6Ew9UACggBbNenoJv7wzsrk3sTnaH6nu 4osAnjmGCuSJWFKVktv5ZnRPPWi2nTmj =QPsZ -----END PGP SIGNATURE----- --jKBxcB1XkHIR0Eqt--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130715062308.GE1516>