Date: Thu, 23 Aug 2012 21:09:00 +0100 From: Chris Rees <crees@FreeBSD.org> To: Kris Moore <kris@pcbsd.org> Cc: Jeremy Messenger <mezz.freebsd@gmail.com>, freebsd-current@freebsd.org, FreeBSD Ports <freebsd-ports@freebsd.org> Subject: Re: pkgng default schedule... registering a few reasons for rethinking the final implementation... Message-ID: <CADLo838zpCZ_eqtpGocXv2v1FEK7UE3jGgk6X7e4Hz2Zb0TOzA@mail.gmail.com> In-Reply-To: <50368973.5040202@pcbsd.org> References: <1345739186.30848.YahooMailClassic@web111307.mail.gq1.yahoo.com> <50365F37.7040601@pcbsd.org> <CADLFttcOfBvH6rpyjTQrV12mRBCy1VKDbJJ0aYZW1bLm154v4Q@mail.gmail.com> <50368973.5040202@pcbsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 23 August 2012 20:50, Kris Moore <kris@pcbsd.org> wrote:
> On 08/23/2012 13:10, Jeremy Messenger wrote:
>> On Thu, Aug 23, 2012 at 11:49 AM, Kris Moore <kris@pcbsd.org> wrote:
>>> On 08/23/2012 12:26, Jeffrey Bouquet wrote:
>>>> I am following with dread the planned implementation of the deprecatio=
n of /var/db/pkg as a package registry... I use each /var/db/pkg directory =
as a database into the port installation/status, using sed/grep/portmaster/=
portmanager/.sh scripts/find/pipes etc... to fix stuff. For instance, an u=
pgrade py26 > py27.
>>>> cd /var/db/pkg
>>>> ls -lac | grep py26
>>>> ls -lac | grep python
>>>> as the more simple example.
>>>> ....
>>>> With due respect to its developers and the persons who agree that
>>>> the package tools could be upgraded, the mandatory
>>>> usage of a front-end database to a file directory one
>>>> is here viewd as mutt-only-mbox, registry-and-bsod rather
>>>> than /etc/local/rc files, deprecation of sed/grep/find/locate/.sh/port=
master/portmanager as tools to fixup/upgrade the ports that are registered;
>>>> ...
>>>> I see concurrently too few tests on lower-end p2, p3 as to whether
>>>> pkg can run with lesser memory machines (routers...) (pfsense)
>>>> ...
>>>> I suspect stalling of successful frontends to bsd (pc-bsd, ghostbsd,
>>>> pfsense..) due to less-reliability, more-possibility of bugs..
>>>>
>>> This is of some concern to me as well. A number of our utilities /
>>> scripts rely on checking /var/db/pkg as a means to test if a particular
>>> package is installed. This is often much faster than running the pkg_*
>>> commands, especially when we may be checking thousands of packages in a
>>> single run. It will be some work to adjust our utilities to using the
>>> various "pkg" commands now, but it can be done. What worries me is
>>> performance. If this is significantly slower, it may cause some issues
>>> on our end.
>> Guys, please test it before you say anything. Otherwise it's going to
>> be moved forward without you.
>>
>>
>
> Well, it was about time I got to doing a benchmark of this anyway :)
>
> I did quick benchmark of how one of our utilities parses through a list
> of 1k packages on a newer i5 system:
>
> First test, using /var/db/pkg/<pkg> check we have been doing:
>
> 0.178s 0:00.31 54.8%
> 0.123s 0:00.26 61.5%
> 0.099s 0:00.15 60.0%
>
> Second test, using "pkg info <pkg>":
>
> 5.347s 0:11.41 91.7%
> 5.444s 0:11.52 91.3%
> 5.878s 0:11.32 91.4%
>
> The pkg info command is quite a bit slower in this case, but 5 seconds
> isn't horrible.
>
> Now I ran the same benchmark on a slower 1.66gz Atom system, checking
> about 1200~ packages:
>
> First test, using /var/db/pkg/<pkg> check we have been doing:
>
> 0.604s 0:00.76 86.8%
> 0.622s 0:00.77 84.4%
> 0.614s 0:00.73 90.4%
>
> Second test, using "pkg info <pkg>":
>
> 28.507s 0:54.80 99.1%
> 28.282s 0:54.60 99.4%
> 28.302s 0:54.52 99.4%
>
> Now this is what concerns me a bit. It took closer to 30 seconds, which
> is quite a while to wait, especially if a utility like ours has to run
> these checks when it starts up, to show the user whats installed / not
> installed on the system.
>
> The only way around It I've found is to do a quick "pkg info" on the
> entire DB, dump that to a list, then begin to grep through that list for
> each item, but it still takes 10~ seconds on the atom. That may be what
> I end up having to do, but it still stinks to go from a half a second
> startup, to 10 seconds each time. Any other ideas on how to do this
> faster with the new pkgng?
Perhaps refactor your method a little;
# Get sample list of packages
[crees@pegasus]~% pkg info -qa > pkglist
[crees@pegasus]~% wc -l pkglist
712 pkglist
# Call pkg info once for each package
[crees@pegasus]~% time <pkglist xargs -n 1 pkg info > /dev/null
xargs -n 1 pkg info < pkglist > /dev/null 2.18s user 2.32s system 98%
cpu 4.544 total
# Call pkg info once for all packages
[crees@pegasus]~% time pkg info $(cat pkglist) > /dev/null
pkg info $(cat pkglist) > /dev/null 0.37s user 0.02s system 98% cpu 0.400 =
total
Summary; pkgng is awesomely fast... but don't call it lots of times
since it has a higher startup time.
I presume your code is something like;
list_of_packages_to_test=3D$@;
for pkg in ${list_of_packages_to_test}
do [ -d /var/db/pkg/${pkg} ] && echo Success! Package installed!
done
? I don't know exactly what you're testing and how, so I can't offer
too much specific advice.
Chris
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CADLo838zpCZ_eqtpGocXv2v1FEK7UE3jGgk6X7e4Hz2Zb0TOzA>