From owner-freebsd-current@FreeBSD.ORG Sun Jul 14 09:06:51 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DFA858EB; Sun, 14 Jul 2013 09:06:51 +0000 (UTC) (envelope-from matthew@FreeBSD.org) Received: from smtp.infracaninophile.co.uk (smtp6.infracaninophile.co.uk [IPv6:2001:8b0:151:1:3cd3:cd67:fafa:3d78]) by mx1.freebsd.org (Postfix) with ESMTP id 5C137A3A; Sun, 14 Jul 2013 09:06:51 +0000 (UTC) Received: from seedling.black-earth.co.uk (seedling.black-earth.co.uk [81.2.117.99]) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.14.7/8.14.7) with ESMTP id r6E95QEr024120 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sun, 14 Jul 2013 10:05:26 +0100 (BST) (envelope-from matthew@FreeBSD.org) DKIM-Filter: OpenDKIM Filter v2.8.3 smtp.infracaninophile.co.uk r6E95QEr024120 Authentication-Results: smtp.infracaninophile.co.uk/r6E95QEr024120; dkim=none reason="no signature"; dkim-adsp=none (unprotected policy) Message-ID: <51E269CD.8030603@FreeBSD.org> Date: Sun, 14 Jul 2013 10:05:17 +0100 From: Matthew Seaman User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: Devin Teske Subject: Re: [HEADSUP] No more pkg_install on HEAD by default References: <20130712231637.GS85556@ithaqua.etoilebsd.net> <13CA24D6AB415D428143D44749F57D7201FC2DBD@ltcfiswmsgmb21> <20130713080732.GV85556@ithaqua.etoilebsd.net> <13CA24D6AB415D428143D44749F57D7201FC3AA2@ltcfiswmsgmb21> <13CA24D6AB415D428143D44749F57D7201FC3C92@ltcfiswmsgmb21> <13CA24D6AB415D428143D44749F57D7201FC3FAA@ltcfiswmsgmb21> <1373759460.29471.140661255344697.6A5745F8@webmail.messagingengine.com> <13CA24D6AB415D428143D44749F57D7201FC48D7@ltcfiswmsgmb21> In-Reply-To: <13CA24D6AB415D428143D44749F57D7201FC48D7@ltcfiswmsgmb21> X-Enigmail-Version: 1.5.1 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="----enig2ROIXFAASMNHNHKAJFHBI" X-Virus-Scanned: clamav-milter 0.97.8 at lucid-nonsense.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-1.6 required=5.0 tests=AWL,BAYES_00,SPF_HELO_FAIL, SPF_SOFTFAIL autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on lucid-nonsense.infracaninophile.co.uk Cc: Mark Felder , "freebsd-current@freebsd.org" , "Teske, Devin" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jul 2013 09:06:51 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) ------enig2ROIXFAASMNHNHKAJFHBI Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 14/07/2013 04:18, Teske, Devin wrote: > ASIDE: For efficiency, I will actually need three things: (1) a list > of all packages (2) their descriptions and (3) their run-time dependenc= ies. That would be the repository catalogue, which you can download by 'pkg update' (or it will happen automatically before pkg operations.) Or, more precisely, the repository /catalogues/ for each of the package repositories enabled in pkg.conf > NOTE: I'll need run-time dependencies so that as they're checking > boxes in the UI, I can update accordingly (has nothing to do with how > the dependencies are handled when the packages get installed; I'll let > pkg handle that when it comes time, but for the UI and for the "review"= > screen, I want the user to be informed and I can do that more > efficiently if I have a master-list and don't have to make continuous > queries). In 1.1 the repository catalogue is just the collection of all the YAML format metadata from all the packages in the repository, plus optional cryptographic signatures. While you could try and parse that directly, there's little point in re-inventing that wheel -- pkg will load the catalogue data into a sqlite database, and it provides 'pkg search' or 'pkg rquery' specifically for searching catalogue data. ('pkg rquery' is aimed at scripting use, 'pkg search' is more for interactive use.) > That being said, I was planning on doing a "pkg rquery" to get that > master list of [minimally] 3-pieces of information. Something like: pkg rquery -a '%n-%v %o\n%e\n%dn-%dv %do\n---\n' > What protocol does that "rquery" run on? What would one have to do > to set up their own server that can accept an "rquery"? (our customers > don't have Internet access) It's built around SQL queries against the sqlite database built from the repository catalogues. Once you've got the repo catalgoue, it doesn't need network access to just query the data. > Last but not least... >=20 > Can you do an "rquery" on a local repository? (say, one that has > been mounted via NFS or some other media, local or otherwise but lookin= g like > a local filesystem once-mounted). What would be required to get a local= > repository like that going? You can certainly build a local repository -- essentially you just assemble a set of packages in some sort of directory heirarchy, and then just run 'pkg repo' to create the catalogue. You can then access the repo by any of the protocols supported by libfetch -- which includes file://, suitable for a repo on a local or NFS mounted directory. Beyond what libfetch provides, you can also use ssh://. You'ld usually use poudriere or similar to build your own set of packages to populate your private repository, but there's nothing to stop you eg. mirroring some selection of packages from a public package repository and building your private repo from them. Or configuring your systems to use a local package repository /and/ a public repository. See pkg-repository(5). >> I do suspect that HTTP_PROXY support is probably not available as I >> recall seeing a post where someone was asking about that getting >> implemented for fetch. I'll have to research that again, though. > Thanks. Keep me up to date on that. pkg will use the proxing capabilities of libfetch -- so it will abide by any HTTP_PROXY or FTP_PROXY or any of a number of other environment variables as described in fetch(3). You can use the PKG_ENV setting in pkg.conf(5) to set the environment use by pkg commands. > Neither sysinstall nor bsdinstall really give HTTP_PROXY access much > thought (they support it, but only minimally). They just construct a > raw HTTP header with the FTP URL and send that directly to the proxy. > One cute thing it does (when initializing the connection) is test for > Squid and if-so, appends ";type=3Di" to the URL (to force binary > download versus ascii). Everything that pkg downloads is currently in the form of an xz(1) compressed tar archive[*] so definitely needs to downloaded in binary mod= e. [*] Even in some cases where the tarball contains only one file, which is a bit of a nonsense, but not hugely so. That may well be changed in future versions. > No support for proxy-server authentication (however user/pass > authentication for the FTP server is passed through to the > proxy-server). pkg uses libfetch's proxy auth support. You can also authenticate access to a repository via FTP or HTTP or HTTPS by encoding a username and password in the repository URL in the usual way. For access over SSH, use any of the authentication mechanisms provided by the ssh server -- use of ssh-agent(1) and key-based auth is recommended, to avoid having to type in passphrases repeatedly. I think if you pkgng-ify a test system and have a play with the capabilities pkg(8) provides (and maybe poudriere(8) too), you will be pleased, maybe even pleasantly surprised, by what it can do. Cheers, Matthew --=20 Dr Matthew J Seaman MA, D.Phil. PGP: http://www.infracaninophile.co.uk/pgpkey ------enig2ROIXFAASMNHNHKAJFHBI Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHiadUACgkQ8Mjk52CukIzV/gCePa/XNq6hY75R08tDNIzdspC0 hDAAn0OMG9xQ/uwPXP73iVfxLGnwYHGC =33Ny -----END PGP SIGNATURE----- ------enig2ROIXFAASMNHNHKAJFHBI--