Date: Thu, 12 Jul 2007 17:01:22 -0700 From: Garrett Cooper <youshi10@u.washington.edu> To: Tim Kientzle <kientzle@freebsd.org> Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Finding slowdowns in pkg_install (continuations of previous threads) Message-ID: <4696C0D2.6010809@u.washington.edu> In-Reply-To: <468E7192.8030105@freebsd.org> References: <468C96C0.1040603@u.washington.edu> <468C9718.1050108@u.washington.edu> <468E60E9.80507@freebsd.org> <468E6C81.4060908@u.washington.edu> <468E7192.8030105@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Tim Kientzle wrote:
>> -I tried ... buffering ... the +CONTENTS file parsing function,
>> and the
>> majority of the time it yielded good results ....
>
> One approach I prototyped sometime back was to use
> libarchive in pkg_add as follows:
> * Open the archive
> * Read +CONTENTS directly into memory (it's
> guaranteed to always be first in the archive)
> * Parse all of +CONTENTS at once
> * Continue scanning the archive, disposing
> of each file as it appears in the archive.
>
> Based on my experience with this, I would
> suggest you just read all of +CONTENTS
> directly into memory at once and parse
> the whole thing in a single shot.
> fopen(), then fstat() to get the size,
> then allocate a buffer and read the whole
> thing, then fclose(). You can then
> parse it all at once.
>
> As a bonus, your parser then becomes a nice
> little bit of reusable code that reads
> a block of memory and returns a structure describing
> the package metadata.
>
> Tim Kientzle
I'm not 100% sure because I'm not comparing apples (virtual disk on
desktop via VMware) to apples (real disk on server), but I'm showing a
2.5-fold speedup after adding the simple parser:
Virtual disk:
4.42 real 1.37 user 1.47 sys
Real disk:
10.26 real 5.36 user 0.99 sys
I'll run a battery of tests just to ensure whether or not that's the case.
Be back with results in a few more days.
-Garrett
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4696C0D2.6010809>