From owner-freebsd-ports@FreeBSD.ORG Thu Aug 23 20:09:35 2012 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 16112106564A; Thu, 23 Aug 2012 20:09:35 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 555298FC0A; Thu, 23 Aug 2012 20:09:33 +0000 (UTC) Received: by lbbgg13 with SMTP id gg13so829566lbb.13 for ; Thu, 23 Aug 2012 13:09:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=0bz0E+s/P0KLZp18sqOoLPR5ciGKkDK9Yg+jodRoUZA=; b=nWzHVmUiui78n1CMVreePkCTs2oZqaPoRQl3ed675/d96JJTHzF1uYjISbqZFWXjti m2pyY4PkSGxf7yrhmi09tKMNjVBYVis13b3cFzmjfZmhe3hl2euuEWgjENcQyRBKilxt X0DouZSdUkwcHnj1HvmqTOjrnooWXk7bq4do1TLpY+svdf0Uvw+DhCqxbSMjwl4RFwKx x+oeJSKaPXSInTmedfeqCoeX2TN8mOECj8I++vNT6kcVgpegiQGj1unrKVe+p2MD6Rae 3CV3MXLNSaCdbNfpzt7AaWZGNRCRzQD8Us5Koqwxo7aht1CtPPzCOcjsKm+DHPEjIXDy jOAQ== Received: by 10.152.146.163 with SMTP id td3mr3143492lab.26.1345752572835; Thu, 23 Aug 2012 13:09:32 -0700 (PDT) MIME-Version: 1.0 Sender: utisoft@gmail.com Received: by 10.112.60.227 with HTTP; Thu, 23 Aug 2012 13:09:00 -0700 (PDT) In-Reply-To: <50368973.5040202@pcbsd.org> References: <1345739186.30848.YahooMailClassic@web111307.mail.gq1.yahoo.com> <50365F37.7040601@pcbsd.org> <50368973.5040202@pcbsd.org> From: Chris Rees Date: Thu, 23 Aug 2012 21:09:00 +0100 X-Google-Sender-Auth: vCGal1TnIHQchit3mlrWyrBLmBQ Message-ID: To: Kris Moore Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Jeremy Messenger , freebsd-current@freebsd.org, FreeBSD Ports Subject: Re: pkgng default schedule... registering a few reasons for rethinking the final implementation... X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 20:09:35 -0000 On 23 August 2012 20:50, Kris Moore wrote: > On 08/23/2012 13:10, Jeremy Messenger wrote: >> On Thu, Aug 23, 2012 at 11:49 AM, Kris Moore wrote: >>> On 08/23/2012 12:26, Jeffrey Bouquet wrote: >>>> I am following with dread the planned implementation of the deprecatio= n of /var/db/pkg as a package registry... I use each /var/db/pkg directory = as a database into the port installation/status, using sed/grep/portmaster/= portmanager/.sh scripts/find/pipes etc... to fix stuff. For instance, an u= pgrade py26 > py27. >>>> cd /var/db/pkg >>>> ls -lac | grep py26 >>>> ls -lac | grep python >>>> as the more simple example. >>>> .... >>>> With due respect to its developers and the persons who agree that >>>> the package tools could be upgraded, the mandatory >>>> usage of a front-end database to a file directory one >>>> is here viewd as mutt-only-mbox, registry-and-bsod rather >>>> than /etc/local/rc files, deprecation of sed/grep/find/locate/.sh/port= master/portmanager as tools to fixup/upgrade the ports that are registered; >>>> ... >>>> I see concurrently too few tests on lower-end p2, p3 as to whether >>>> pkg can run with lesser memory machines (routers...) (pfsense) >>>> ... >>>> I suspect stalling of successful frontends to bsd (pc-bsd, ghostbsd, >>>> pfsense..) due to less-reliability, more-possibility of bugs.. >>>> >>> This is of some concern to me as well. A number of our utilities / >>> scripts rely on checking /var/db/pkg as a means to test if a particular >>> package is installed. This is often much faster than running the pkg_* >>> commands, especially when we may be checking thousands of packages in a >>> single run. It will be some work to adjust our utilities to using the >>> various "pkg" commands now, but it can be done. What worries me is >>> performance. If this is significantly slower, it may cause some issues >>> on our end. >> Guys, please test it before you say anything. Otherwise it's going to >> be moved forward without you. >> >> > > Well, it was about time I got to doing a benchmark of this anyway :) > > I did quick benchmark of how one of our utilities parses through a list > of 1k packages on a newer i5 system: > > First test, using /var/db/pkg/ check we have been doing: > > 0.178s 0:00.31 54.8% > 0.123s 0:00.26 61.5% > 0.099s 0:00.15 60.0% > > Second test, using "pkg info ": > > 5.347s 0:11.41 91.7% > 5.444s 0:11.52 91.3% > 5.878s 0:11.32 91.4% > > The pkg info command is quite a bit slower in this case, but 5 seconds > isn't horrible. > > Now I ran the same benchmark on a slower 1.66gz Atom system, checking > about 1200~ packages: > > First test, using /var/db/pkg/ check we have been doing: > > 0.604s 0:00.76 86.8% > 0.622s 0:00.77 84.4% > 0.614s 0:00.73 90.4% > > Second test, using "pkg info ": > > 28.507s 0:54.80 99.1% > 28.282s 0:54.60 99.4% > 28.302s 0:54.52 99.4% > > Now this is what concerns me a bit. It took closer to 30 seconds, which > is quite a while to wait, especially if a utility like ours has to run > these checks when it starts up, to show the user whats installed / not > installed on the system. > > The only way around It I've found is to do a quick "pkg info" on the > entire DB, dump that to a list, then begin to grep through that list for > each item, but it still takes 10~ seconds on the atom. That may be what > I end up having to do, but it still stinks to go from a half a second > startup, to 10 seconds each time. Any other ideas on how to do this > faster with the new pkgng? Perhaps refactor your method a little; # Get sample list of packages [crees@pegasus]~% pkg info -qa > pkglist [crees@pegasus]~% wc -l pkglist 712 pkglist # Call pkg info once for each package [crees@pegasus]~% time /dev/null xargs -n 1 pkg info < pkglist > /dev/null 2.18s user 2.32s system 98% cpu 4.544 total # Call pkg info once for all packages [crees@pegasus]~% time pkg info $(cat pkglist) > /dev/null pkg info $(cat pkglist) > /dev/null 0.37s user 0.02s system 98% cpu 0.400 = total Summary; pkgng is awesomely fast... but don't call it lots of times since it has a higher startup time. I presume your code is something like; list_of_packages_to_test=3D$@; for pkg in ${list_of_packages_to_test} do [ -d /var/db/pkg/${pkg} ] && echo Success! Package installed! done ? I don't know exactly what you're testing and how, so I can't offer too much specific advice. Chris