From owner-freebsd-ports@FreeBSD.ORG Wed Dec 3 23:00:15 2008 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B2BAA106564A; Wed, 3 Dec 2008 23:00:15 +0000 (UTC) (envelope-from scode@hyperion.scode.org) Received: from hyperion.scode.org (cl-1361.ams-04.nl.sixxs.net [IPv6:2001:960:2:550::2]) by mx1.freebsd.org (Postfix) with ESMTP id 36EFA8FC12; Wed, 3 Dec 2008 23:00:14 +0000 (UTC) (envelope-from scode@hyperion.scode.org) Received: by hyperion.scode.org (Postfix, from userid 1001) id 7890123C46E; Thu, 4 Dec 2008 00:00:05 +0100 (CET) Date: Thu, 4 Dec 2008 00:00:05 +0100 From: Peter Schuller To: Doug Barton Message-ID: <20081203230005.GB57472@hyperion.scode.org> References: <20081203210233.GA55633@hyperion.scode.org> <493705E2.907@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="oLBj+sq0vYjzfsbl" Content-Disposition: inline In-Reply-To: <493705E2.907@FreeBSD.org> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-ports@freebsd.org Subject: Re: Deterministic package building with ports X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Dec 2008 23:00:15 -0000 --oLBj+sq0vYjzfsbl Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello, > One thing I'm confused about here, what are your goals? You don't > mention why you need to build the packages. The answer to this might > inform the rest of the conversation. At the very highest level, the goal is to keep a system, or a set of system (large or small) up-to-date in an efficient manner, while minimizing downtime. I want to build packages separately not just to minimize the time window, but also to ensure that the full set of packages can successfully be built - avoiding a half-upgraded system that has issues to be resolved. The system(s) is/are assumed to need custom port options; thus use of bulk built packages is not an option. > > For the past half year or > > so I have ended up using a hacked together shell script which does > > approximately what I need, which is: > >=20 > > To build deterministically as a function of > > (a) base system > > (b) /etc/make.conf > > (c) /usr/local/etc/ports.conf (sysutils/portconf) >=20 > These are givens, but it never hurts to mention requirements explicitly. My intent was also to convey that I want it to be a function of *only* the list of things I provided, rather than an unknown set of factors. > > (d) a list of specifically desired origins/packages which is manual= ly > > mainteined (NOT a complete list of dependencies, this is critic= al) >=20 > Why is not maintaining the list of dependencies critical? I'm guessing > the answer is related to the dependencies being fluid over time. Because it becomes a huge pain when your complete package list is 1000 packages. What always ended up happening back when I tried keeping machines up to date by portupgrade/portmanager, was that every now and then I would just get fed up with all the built-up issues and re-install everything from scratch. The actual real-life goals are often very easily expressed in terms of a pretty small set of packages that one is actively interested in. Any manual maintenance of a very very significantly longer list of dependencies is just manual labor that you want to automate. I view it as the package management equivalent of maintaining a set of patches against a source tree instead of just forking it. I have something that I can realistically maintain manually which expresses exactly my intent, nothing more, nothing less. When writing pkgmanager (pkgsrc only) this was a major goal. You maintained your want list, and your pkgsrc tree, and you just told it "sync". It then did whatever it needed to do to produce a system which is identical to what you would have gotten by taking your list of desired packages and "bmake install":ing them on a clean system. It was all a function of a defined set of things, and the process was 100% restartable because all necessary discovery happened based on persistent information on disk. In short, you told it what you wanted, rather than what to do. Of course, this didn't work perfectly - mostly due to non-determinism. > > (e) the ports tree [keeping the above to keep the quoted list complete] > > Is it the intent of the design of ports that the behavior of packages > > be implicitly dependent on the state of installed packages? > > > > If the answer is yes, can someone suggest a practically viable that > > consistently works, method of building binary packages under the above > > conditions? >=20 > Yes, this is a feature. You could solve the problem you describe above > by building the X stuff first. The problem is that I do not want to care about that. I know that "I want aumix" and that "I want firefox". I do not want to discover by a long period of trial and error that I need to fudge my tools to install aumix prior to firefox, or the other way around. We have even more trikier issues like "I want subversion" and "I want apache", and having to specify which apr has to be used. I see no general and simple way of implementing detection of such conflicts in a package manager built on top of ports, without modifying ports to accomodate. If being impacted by installed packages is considered a feature, it may be that my preferences/goals are just fundamentally incompatible with that of ports. Would there be interest/opinions on having ports support a mode of operation specifically targetted at automated tools and determinism? I suspect having a separate mode will just lead to too much complexity and breakage. > > (I know of portupgrade/portmaster of course. I have never found a tool > > that both (1) does what I want=20 >=20 > What you want is beyond the scope of current tools. I have a proposal > for adding this functionality to portmaster but it is a considerable > amount of work and therefore I was hoping to get some funding to support > it. Do you think it is appropriate to solve this elaborately in tools, as opposed to making changes to the ports infrastructure to facilitate these things? Am I barking up the wrong tree, hoping to see the latter? I have a genuine interest in having something which works and is reasonable simple to use for an administrator, and am willing to spend time on it. But not if the work is going to be linear with respect to various quirks and oddities of the underlying system. In my personal ideal world, the fact that the installed set of packages affects the dependency decision of a port would be a bug. I take it =66rom the above that this is not consistent with the generally accepted view. I have planned for a long time to re-write pkgmanager and support both ports and pkgsrc; but fundamental issues such as determinism are putting me off (other than lazyness) because I know that it will never really work as well as I would want it to in order to be truly useful and more effective than the hacks I am currently using as a poor man's alternative. > > and (2) actually works consistently. >=20 > If you wanted to list the ways that portmaster does not work > consistently in a different thread I'd be interested. I want to clarify right here for the record that I am not accusing portmaster of not working consistently. However, it did fall into (1) - that is, that it does not do what I personally want. portupgrade however does fall into both categories in my book. I'd be very very interesting in discussing package management and tool issues if you are willing. However, I'm not sure whether my personal goals are sufficiently in synch with yours or others for it to be very fruitful. I could try to recall past experience and issues I have run into administratively and when developing pkgmanager and try to state more clearly what my biased view is, with respect to what I would like tools to do, and how I think ports could facilitate those tools. Would freebsd-ports@ be a suitable forum for that? --=20 / Peter Schuller PGP userID: 0xE9758B7D or 'Peter Schuller ' Key retrieval: Send an E-Mail to getpgpkey@scode.org E-Mail: peter.schuller@infidyne.com Web: http://www.scode.org --oLBj+sq0vYjzfsbl Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkk3D3UACgkQDNor2+l1i31syQCgpxXy3AeHla+KmwE62wRES1lO /NsAoKPMkphMvtGl8LCIPPYP44JfXwKs =kY7Y -----END PGP SIGNATURE----- --oLBj+sq0vYjzfsbl--