From owner-freebsd-current@FreeBSD.ORG Sun Jul 14 10:12:21 2013 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8D3AA242; Sun, 14 Jul 2013 10:12:21 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id 5006FBE2; Sun, 14 Jul 2013 10:12:21 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.15]) by ltcfislmsgpa07.fnfis.com (8.14.5/8.14.5) with ESMTP id r6EACKRK021768 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sun, 14 Jul 2013 05:12:20 -0500 Received: from LTCFISWMSGMB21.FNFIS.com ([10.132.99.23]) by LTCFISWMSGHT04.FNFIS.com ([10.132.206.15]) with mapi id 14.02.0309.002; Sun, 14 Jul 2013 05:12:19 -0500 From: "Teske, Devin" To: Matthew Seaman Subject: Re: [HEADSUP] No more pkg_install on HEAD by default Thread-Topic: [HEADSUP] No more pkg_install on HEAD by default Thread-Index: AQHOf1XlEhI/XtjETUan8/QjEhM8ppliCtaAgACKXACAAHg7gIAAGMOAgAAZGQCAAAqhgIAAnyqAgAAXnwCAAD4lgIAAC4sA Date: Sun, 14 Jul 2013 10:12:19 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201FC547A@ltcfiswmsgmb21> References: <20130712231637.GS85556@ithaqua.etoilebsd.net> <13CA24D6AB415D428143D44749F57D7201FC2DBD@ltcfiswmsgmb21> <20130713080732.GV85556@ithaqua.etoilebsd.net> <13CA24D6AB415D428143D44749F57D7201FC3AA2@ltcfiswmsgmb21> <13CA24D6AB415D428143D44749F57D7201FC3C92@ltcfiswmsgmb21> <13CA24D6AB415D428143D44749F57D7201FC3FAA@ltcfiswmsgmb21> <13CA24D6AB415D428143D44749F57D7201FC4E0D@ltcfiswmsgmb21> <51E26FD3.4020208@FreeBSD.org> In-Reply-To: <51E26FD3.4020208@FreeBSD.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.121] Content-Type: text/plain; charset="iso-8859-1" Content-ID: <2BC6EED7240B1440985D4D176E20F3BB@fisglobal.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8794, 1.0.431, 0.0.0000 definitions=2013-07-14_01:2013-07-12,2013-07-14,1970-01-01 signatures=0 Cc: Baptiste Daroussin , Devin Teske , "current@FreeBSD.org" , Peter Wemm X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jul 2013 10:12:21 -0000 On Jul 14, 2013, at 2:30 AM, Matthew Seaman wrote: > On 14/07/2013 06:48, Teske, Devin wrote: >> Question: Where can I learn more about the actual format of what's in >> the new tarballs? This is going to be important not for bsdconfig, >> but $work (we have our own build platform; I'm going to have to >> rewrite it from mastering PLIST files to mastering YAML MANIFEST >> files and I want to know all the gritty details). >=20 > We do need a pkg-manifest(5) man page, which can double as a > pkg-tarball(5) page since the manifest file is where most of the > interesting bits are. >=20 > A pkg tarball is a compressed tar archive like so: >=20 > lucid-nonsense:...cache/pkg/All:% tar -tvf pkg-1.1.4.txz > -rw-r--r-- 0 root wheel 530 Jan 1 1970 +COMPACT_MANIFEST > -rw-r--r-- 0 root wheel 6385 Jan 1 1970 +MANIFEST > -rw-r--r-- 0 root wheel 17567 Jan 1 1970 +MTREE_DIRS > -r--r--r-- 0 root wheel 19453 Jul 7 12:26 > /usr/local/etc/bash_completion.d/_pkg.bash > -r-xr-xr-x 0 root wheel 629 Jul 7 12:26 > /usr/local/etc/periodic/daily/400.status-pkg > -r-xr-xr-x 0 root wheel 823 Jul 7 12:26 > /usr/local/etc/periodic/daily/411.pkg-backup > -r-xr-xr-x 0 root wheel 678 Jul 7 12:26 > /usr/local/etc/periodic/daily/490.status-pkg-changes > -r-xr-xr-x 0 root wheel 2558 Jul 7 12:26 > /usr/local/etc/periodic/security/410.pkg-audit > -r-xr-xr-x 0 root wheel 611 Jul 7 12:26 > /usr/local/etc/periodic/security/460.pkg-checksum > -r--r--r-- 0 root wheel 839 Jul 7 12:26 > /usr/local/etc/pkg.conf.sample > -r--r--r-- 0 root wheel 43432 Jul 7 12:26 /usr/local/include/pkg.h > -r--r--r-- 0 root wheel 727558 Jul 7 12:26 /usr/local/lib/libpkg.a > lrwxr-xr-x 0 root wheel 0 Jul 7 12:26 /usr/local/lib/libpkg.so > -> libpkg.so.1 > -r--r--r-- 0 root wheel 1227064 Jul 7 12:26 /usr/local/lib/libpkg.so.1 > -rw-r--r-- 0 root wheel 187 Jul 7 12:26 > /usr/local/libdata/pkgconfig/pkg.pc > [... etc ...] >=20 Interesting. I notice that (while looking ahead to see a prefix: of /usr/lo= cal in the +MANIFEST), the tarball itself has files that include /usr/local= in their path. Does pkg handle packages where the prefix (in +MANIFEST) is "/usr/local" _b= ut_ "tar tf" of the unxz'd tarball is _not_ "/usr/local"? (e.g. pkg_install= style) or is this the new way going forward? > There must at least be a +MANIFEST -- other meta data files are > optional. +COMPACT_MANIFEST is a subset of the full +MANIFEST. They're > both YAML documents. +COMPACT_MANIFEST looks like this, for example: >=20 > --- > name: pkg > version: 1.1.4 > origin: ports-mgmt/pkg > comment: New generation package manager > arch: freebsd:9:x86:64 > www: https://urldefense.proofpoint.com/v1/url?u=3Dhttp://wiki.freebsd.org= /pkgng&k=3D%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=3DMrjs6vR4%2Faj2Ns9%2FssHJ= jg%3D%3D%0A&m=3DLPoff3ddLDoHE30AU6R5vgBklqxsjVAosXGjJop6uO4%3D%0A&s=3D30b27= cf33d7e594144fc38f20bc75d84464788e715f7fc82b2495f813fd4be2d > maintainer: portmgr@FreeBSD.org > prefix: /usr/local > licenselogic: single > licenses: > - BSD > flatsize: 6311507 > desc: | > New Generation package management tool for FreeBSD >=20 > WWW: http://wiki.freebsd.org/pkgng > categories: > - ports-mgmt > shlibs_required: > - libpkg.so.1 > shlibs_provided: > - libpkg.so.1 > message: | > If you are upgrading from the old package format, first run: >=20 > # pkg2ng >=20 Nice. Very nice. > +MTREE_DIRS is a compatibility thing with the old pkg_tools. It's not > needed in general as +MANIFEST can provide all that meta data itself. > It isn't going to be deprecated for at least as long as the ports tree > continues to support pkg_tools though. >=20 > Beyond that, the rest of the pkg tarball just contains a tar archive of > all the files, directories, sym-links etc to be installed by the > package. Note that pkg(8) has no problem with creating an empty > directory for a package, unlike pkg_tools. >=20 Excellent! > Now, while you can grovel through the details of pkg tarballs and > internal data formats like this, be aware: the format of the manifest > and the details of the meta-data included in the pkg-tarballs is subject > to change without warning as we develop pkg(8) further. We only promise > API stability through the pkg(8) commands or for the libpkg.so library > functions; reports of breakage from usage outside those APIs will > receive little sympathy. >=20 Understood. So just to give you a better idea of how we manage packages here. We've realized that if you want to package for FreeBSD (in 9 and older), yo= u could get by alright if you knew how to create a +CONTENTS file from scra= tch. For RedHat, it's the SPECFILE. And now for FreeBSD 10, it looks an awf= ul lot like a SPECFILE (they are both in-fact YAML). So rather than teach the people here how to use tools, I taught them what I= think is more important... how to manage +CONTENTS, SPECFILE, and now up-a= nd-coming, +MANIFEST. However, I'd be lying if I said I taught them *all* the gory details. In re= ality, for +CONTENTS they edit a PLIST which is essentially +CONTENTS witho= ut the "@comment MD5:" entries. For SPECFILE, they manage the full thing ex= cept there's a small section that is the standard RPM bootstrap that is lab= eled as "do not touch". >From what you posted of +COMPACT_MANIFEST was nice, except it lacks the lis= t of files. The basic principle here is that if you have to maintain a list of files, a= nd that list ends up being compiled into a +MANIFEST, why not just teach yo= ur peers how to read/write/maintain +MANIFEST files (in version control of = course) so that there are never any mysteries as to how the package perform= s. I know this sounds screwy... but (as with our other YAML files), it's reall= y nice because (as is the case with SPECFILE), we version-control things as= -close to the end-product as possible (take for example the case of "pre-in= stall" or "post-install" et cetera scripts -- I'm sure a tool like "pkg cre= ate" or "pkg_create" or other could do a good job of assembling the pieces = into the +MANIFEST, but then it's not being version-controlled as closely. Before this workflow, mistakes were rampant and there wasn't much hope of w= rangling the "oops, forgot to touch the PLIST" or "oops, forgot to update t= he post-install script" mistakes. I see a nice bright future in re-working my "pkgbase" to be able to supplan= t metadata into a revision-controlled +MANIFEST. Ideally, I don't want them to have to be burdened with maintaining both a += MANIFEST and +COMPACT_MANIFEST, so it seems most logical to generate one fr= om the other (the latter being the revision-controlled copy sans-meta-data;= the meta-data is added at "make" time before then running validations and = generating the tarball). The problem of breakage to the system by API changes is less important, bec= ause we track security releng branches and use binaries so changes are very= slow to creep in. But since *can* use a different framework for each/every= major release branch, we can track API changes quite easily. I'm aware that only slow moving dinosaurs like us are interested in such ba= ckward-thinking mechanics... but this stuff really works for us, and I'm ce= rtain that digesting +MANIFEST will be very beneficial to our growth. --=20 Devin _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you.