From owner-freebsd-ports@FreeBSD.ORG Mon Sep 19 01:37:48 2011 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 129FE1065672 for ; Mon, 19 Sep 2011 01:37:48 +0000 (UTC) (envelope-from lstewart@freebsd.org) Received: from lauren.room52.net (lauren.room52.net [210.50.193.198]) by mx1.freebsd.org (Postfix) with ESMTP id 7C3AE8FC08 for ; Mon, 19 Sep 2011 01:37:47 +0000 (UTC) Received: from lstewart1.loshell.room52.net (ppp59-167-184-191.static.internode.on.net [59.167.184.191]) by lauren.room52.net (Postfix) with ESMTPSA id ECF217E84A; Mon, 19 Sep 2011 11:37:44 +1000 (EST) Message-ID: <4E769CE8.4010405@freebsd.org> Date: Mon, 19 Sep 2011 11:37:44 +1000 From: Lawrence Stewart User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0.2) Gecko/20110914 Thunderbird/6.0.2 MIME-Version: 1.0 To: bf1783@gmail.com References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=2.3 required=5.0 tests=FSL_RU_URL,UNPARSEABLE_RELAY autolearn=no version=3.3.1 X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on lauren.room52.net Cc: "b. f." , freebsd-ports@freebsd.org Subject: Re: Detecting dependencies X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Sep 2011 01:37:48 -0000 On 09/17/11 18:09, b. f. wrote: >> On 09/15/11 07:06, chukharev at mail.ru wrote: >>> Hi, >>> >>> There have been a discussion about finding interdependencies of ports. >>> I have a relatively simple Python script for that. There is a pr >>> ports/160007 >>> to add its early version. Unfortunately, I missed a reply to it, so >>> there is >>> an issue which I have not yet addressed... >>> >>> Since that time, I added reverse dependencies with full ports tree scanning >>> (1 h on my 2.5GHz notebook) and saving the tree (directed graph, actually) >>> to a file, so that rescanning all ports tree is not needed. >>> >>> See http://code.google.com/p/porttree/ >>> >>> If there will be interest, scanning packages interdependencies could >>> also be added. >>> >> >> On a related subtopic, we also need a tool that identifies implicit >> dependencies not captured in the ports Makefiles. I hacked the following >> together earlier this year to smooth over the updating process when key >> libraries get bumped (e.g. the gettext update at the time I wrote the >> script was a nightmare). There were a tonne of ports which needed to be >> updated even though they didn't explicitly record a dependency on gettext. >> >> https://lauren.room52.net/hg/scripts/raw-file/tip/libdepend/libdepend.sh >> >> It's still quite rough and manually driven and is tied to portmaster at >> the moment, but I use it routinely after a "portmaster -ad" to check >> that no libs are missing dependencies. It works pretty well most of the >> time, but definitely needs more finessing. I share it mostly to prove >> the feasibility of the approach and in case anyone is curious. > > What, no check to see if the libraries listed in the DT_NEEDED tags > are actually needed? And no kitchen sink? ;) err... look, over there! A dog with a puffy tail chasing a kitchen sink! *runs* > There are scripts in ports/Tools/scripts that were intended to perform > similar tasks, although they may be rougher than your script. Yes they provide various bits and pieces of the puzzle. >> I haven't thought the following ideas through a great deal and welcome >> feedback, but I think the basic functionality/premise of this script >> could be integrated into the ports framework so that at package >> registration time, implicit deps are identified and marked in the >> package database. A warning could also be generated that the port is >> using deps not identified in the Makefile, and perhaps trigger a send-pr >> to the port maintainer to let them know. > > ... > >> A script like this could also be integrated/called somehow from a tool >> like portmaster during an update to ensure ports with implicit >> dependencies on another port which has been updated are identified and >> recompiled too so that we avoid the nasty problems that crop up with >> missing library dependencies. > > Just as in the other *_DEPENDS lists, it was a conscious policy > decision, for the sake of brevity and efficiency, that if port B > requires port C, and port A requires port B, > then libraries from port C will not be listed in the LIB_DEPENDS of > port A, even if port A links directly to those libraries. But because Right, which is fine (and more desirable - simpler is better) if we have a way built into the system to actually derive them at upgrade/install time and ensure we don't leave the system broken. Given that the information can be derived, it seems sensible not to have ports' Makefiles define all deps explicitly, and instead use tools at install/upgrade time to do the heavy lifting automatically. Going for brevity without the infrastructure in place to automagically compensate for the information not being explicitly codified in the makefiles means certain brokeness, which is not cool. > of recurring problems with partial port updates, this policy has been > criticized. I think that the last time the matter was raised, the > consensus seemed to lean toward listing all needed libraries, but the > amount of work involved in, and the likely disruption arising from, > refactoring all LIB_DEPENDS in the tree dissuaded anyone doing so. I can't see a reason why the policy can't stay as it is if the smarts can be added to generate the implicit dependency info when needed, and more importantly use that generated information to avoid leaving the system broken. Whether we argue the smarts belong solely in a tool like portmaster or should be integrated into the ports infrastructure itself is fair game. My gut feeling is that the deps list stored on disk by the ports system at registration time should be complete with explicit and implicit deps, even though the port's Makefile only lists those which are explicit. Tools like portmaster then only need to use the information as is to do their part of the job and the system should be left intact post upgrade cycle, at least from a broken lib deps perspective. If my gut feeling is valid, then that implies the ports infrastructure itself should do a step post install but pre registration where implicit deps are identified and added to the port's +CONTENTS file. I'm very unfamiliar with the behind the scenes mechanics of ports, so I'm sure I'm oversimplifying things and would be keen to learn what pitfalls exist with the idea outlined above. Cheers, Lawrence