From owner-freebsd-ports@FreeBSD.ORG Fri Jul 6 21:17:17 2007 Return-Path: X-Original-To: ports@freebsd.org Delivered-To: freebsd-ports@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0908E16A421; Fri, 6 Jul 2007 21:17:17 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from mxout3.cac.washington.edu (mxout3.cac.washington.edu [140.142.32.166]) by mx1.freebsd.org (Postfix) with ESMTP id DE86B13C45E; Fri, 6 Jul 2007 21:17:16 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from hymn09.u.washington.edu (hymn09.u.washington.edu [140.142.12.183]) by mxout3.cac.washington.edu (8.13.7+UW06.06/8.13.7+UW07.06) with ESMTP id l66LHG8S000473 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 6 Jul 2007 14:17:16 -0700 Received: from localhost (localhost [127.0.0.1]) by hymn09.u.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l66LHGvY000562; Fri, 6 Jul 2007 14:17:16 -0700 X-Auth-Received: from [192.55.52.3] by hymn09.u.washington.edu via HTTP; Fri, 06 Jul 2007 14:17:16 PDT Date: Fri, 6 Jul 2007 14:17:16 -0700 (PDT) From: youshi10@u.washington.edu To: ports@freebsd.org In-Reply-To: <20070706170750.GA99504@lpthe.jussieu.fr> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-PMX-Version: 5.3.2.304607, Antispam-Engine: 2.5.1.298604, Antispam-Data: 2007.7.6.135432 X-Uwash-Spam: Gauge=IIIIIII, Probability=7%, Report='SUPERLONG_LINE 0.05, NO_REAL_NAME 0, __CP_URI_IN_BODY 0, __CT 0, __CT_TEXT_PLAIN 0, __HAS_MSGID 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0' Cc: hackers@freebsd.org Subject: Re: Finding slowdowns in pkg_install X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Jul 2007 21:17:17 -0000 On Fri, 6 Jul 2007, Michel Talon wrote: > Tim Kientzle said: > >> One approach I prototyped sometime back was to use >> libarchive in pkg_add as follows: >> * Open the archive >> * Read +CONTENTS directly into memory (it's >> guaranteed to always be first in the archive) > > I can only concur with that. In my program > http://www.lpthe.jussieu.fr/~talon/check_pkg.py > i discovered that memory mapping +CONTENTS and then working > in memory before rewriting it was around 5 times faster > than reading line by line and parsing each line. See function > fix_pkg_database(). By the way i am writing a new +CONTENTS > fileand then renaming it, which avoids leaving a mess if > something goes astray like portupgrade does. > > > -- > > Michel TALON Ok, excellent I'll try that then. I'll work on an improved parser this weekend and probably will have more conclusive results for next week, but for the immediate point in time I'll post results on how slow / fast the critical sections were once I return home and post process my data again for averages and standard deviations. I'll use this as my basis for further conclusions this summer. -Garrett