From owner-freebsd-current@FreeBSD.ORG Thu Aug 23 20:31:59 2012 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 59E391065670 for ; Thu, 23 Aug 2012 20:31:59 +0000 (UTC) (envelope-from ohauer@gmx.de) Received: from mailout-de.gmx.net (mailout-de.gmx.net [213.165.64.22]) by mx1.freebsd.org (Postfix) with SMTP id 8EE898FC0C for ; Thu, 23 Aug 2012 20:31:58 +0000 (UTC) Received: (qmail invoked by alias); 23 Aug 2012 20:31:57 -0000 Received: from p578be941.dip0.t-ipconnect.de (EHLO [192.168.0.100]) [87.139.233.65] by mail.gmx.net (mp036) with SMTP; 23 Aug 2012 22:31:57 +0200 X-Authenticated: #1956535 X-Provags-ID: V01U2FsdGVkX1+ErnVkAA6E1jZbczPG4aSt6FLYbYj3i4hyYrqobt uZ3S3Z2VQf7BXy Message-ID: <5036933B.8030603@gmx.de> Date: Thu, 23 Aug 2012 22:31:55 +0200 From: olli hauer User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:14.0) Gecko/20120713 Thunderbird/14.0 MIME-Version: 1.0 To: Kris Moore References: <1345739186.30848.YahooMailClassic@web111307.mail.gq1.yahoo.com> <50365F37.7040601@pcbsd.org> <50368973.5040202@pcbsd.org> In-Reply-To: <50368973.5040202@pcbsd.org> X-Enigmail-Version: 1.4.3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Y-GMX-Trusted: 0 Cc: freebsd-current@freebsd.org, FreeBSD Ports Subject: Re: pkgng default schedule... registering a few reasons for rethinking the final implementation... X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 20:31:59 -0000 On 2012-08-23 21:50, Kris Moore wrote: > On 08/23/2012 13:10, Jeremy Messenger wrote: >> On Thu, Aug 23, 2012 at 11:49 AM, Kris Moore wrote: >>> On 08/23/2012 12:26, Jeffrey Bouquet wrote: >>>> I am following with dread the planned implementation of the deprecation of /var/db/pkg as a package registry... I use each /var/db/pkg directory as a database into the port installation/status, using sed/grep/portmaster/portmanager/.sh scripts/find/pipes etc... to fix stuff. For instance, an upgrade py26 > py27. >>>> cd /var/db/pkg >>>> ls -lac | grep py26 >>>> ls -lac | grep python >>>> as the more simple example. >>>> .... >>>> With due respect to its developers and the persons who agree that >>>> the package tools could be upgraded, the mandatory >>>> usage of a front-end database to a file directory one >>>> is here viewd as mutt-only-mbox, registry-and-bsod rather >>>> than /etc/local/rc files, deprecation of sed/grep/find/locate/.sh/portmaster/portmanager as tools to fixup/upgrade the ports that are registered; >>>> ... >>>> I see concurrently too few tests on lower-end p2, p3 as to whether >>>> pkg can run with lesser memory machines (routers...) (pfsense) >>>> ... >>>> I suspect stalling of successful frontends to bsd (pc-bsd, ghostbsd, >>>> pfsense..) due to less-reliability, more-possibility of bugs.. >>>> >>> This is of some concern to me as well. A number of our utilities / >>> scripts rely on checking /var/db/pkg as a means to test if a particular >>> package is installed. This is often much faster than running the pkg_* >>> commands, especially when we may be checking thousands of packages in a >>> single run. It will be some work to adjust our utilities to using the >>> various "pkg" commands now, but it can be done. What worries me is >>> performance. If this is significantly slower, it may cause some issues >>> on our end. >> Guys, please test it before you say anything. Otherwise it's going to >> be moved forward without you. >> >> > > Well, it was about time I got to doing a benchmark of this anyway :) > > I did quick benchmark of how one of our utilities parses through a list > of 1k packages on a newer i5 system: > > First test, using /var/db/pkg/ check we have been doing: > > 0.178s 0:00.31 54.8% > 0.123s 0:00.26 61.5% > 0.099s 0:00.15 60.0% > > Second test, using "pkg info ": > > 5.347s 0:11.41 91.7% > 5.444s 0:11.52 91.3% > 5.878s 0:11.32 91.4% > > The pkg info command is quite a bit slower in this case, but 5 seconds > isn't horrible. > > Now I ran the same benchmark on a slower 1.66gz Atom system, checking > about 1200~ packages: > > First test, using /var/db/pkg/ check we have been doing: > > 0.604s 0:00.76 86.8% > 0.622s 0:00.77 84.4% > 0.614s 0:00.73 90.4% > > Second test, using "pkg info ": > > 28.507s 0:54.80 99.1% > 28.282s 0:54.60 99.4% > 28.302s 0:54.52 99.4% > > Now this is what concerns me a bit. It took closer to 30 seconds, which > is quite a while to wait, especially if a utility like ours has to run > these checks when it starts up, to show the user whats installed / not > installed on the system. > > The only way around It I've found is to do a quick "pkg info" on the > entire DB, dump that to a list, then begin to grep through that list for > each item, but it still takes 10~ seconds on the atom. That may be what > I end up having to do, but it still stinks to go from a half a second > startup, to 10 seconds each time. Any other ideas on how to do this > faster with the new pkgng? > Hi Kris, can you describe what exactly the script is doing. Are you aware that you can feed direct SQL to pkg ? $> echo 'select origin,name,version,comment from packages;' | pkg shell At the beginning I was also a little skeptic, but even for older (slow) machines it works well here. One note, on small systems keep an eye on /var/cache/pkg -- Regards, olli