From owner-freebsd-hackers@FreeBSD.ORG Sun Jun 17 22:20:40 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EE73B16A400; Sun, 17 Jun 2007 22:20:40 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from mxout2.cac.washington.edu (mxout2.cac.washington.edu [140.142.33.4]) by mx1.freebsd.org (Postfix) with ESMTP id C8C4A13C469; Sun, 17 Jun 2007 22:20:40 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from smtp.washington.edu (smtp.washington.edu [140.142.33.9] (may be forged)) by mxout2.cac.washington.edu (8.13.7+UW06.06/8.13.7+UW07.05) with ESMTP id l5HMKdWv026072 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Sun, 17 Jun 2007 15:20:40 -0700 X-Auth-Received: from [192.168.10.45] (c-24-10-12-194.hsd1.ca.comcast.net [24.10.12.194]) (authenticated authid=youshi10) by smtp.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l5HMKds9021258 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sun, 17 Jun 2007 15:20:39 -0700 Message-ID: <4675B3B6.2040603@u.washington.edu> Date: Sun, 17 Jun 2007 15:20:38 -0700 From: Garrett Cooper User-Agent: Thunderbird 2.0.0.4 (Windows/20070604) MIME-Version: 1.0 To: Tim Kientzle References: <46703EE9.1030804@freebsd.org> <4674B268.4030502@u.washington.edu> <4674BE32.300@freebsd.org> <20070617203640.334524fc@hermies.int.fubar.geek.nz> <467560F4.9050007@u.washington.edu> <4675B088.2040408@freebsd.org> In-Reply-To: <4675B088.2040408@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-PMX-Version: 5.3.1.294258, Antispam-Engine: 2.5.1.298604, Antispam-Data: 2007.6.17.150433 X-Uwash-Spam: Gauge=IIIIIII, Probability=7%, Report='__CP_URI_IN_BODY 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_MSGID 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0, __USER_AGENT 0' Cc: hackers@freebsd.org, Andrew Turner Subject: Re: Rearchitecting pkg_install (was Re: Using shell commands versus C equivalents) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Jun 2007 22:20:41 -0000 Tim Kientzle wrote: >>> I've also seen a 3x speedup by using my reimplementation of pkg_add >>> using my package management library, libpkg (http://libpkg.berlios.de). >>> It is not production ready yet as if it fails partway through an >>> installation it won't clean up and installed files. > > I *think* a good way to do this is to: > 1) Write each file to a temporary name (as Peter Jeremy > suggested earlier in this thread) > 2) Write a +INPROGRESS file to the package dir with > the temporary name of each file as it is written. > 3) Recursively install dependencies. > 4) Rename the files to their final names. > > This sequence allows you to: > * Do conflict checking on the fly. If a single file > conflicts at any point, you can detect that, ask > the user for a resolution at that point and drop > the single file or back out the entire package. > * Recover crashed installations. Just scan for all > +INPROGRESS files and remove every file mentioned. > > There are a few additional steps you can take to provide > even better failure recovery, but this covers the big > concerns. > >>>> I haven't looked closely at pkg_delete, but I doubt there's >>>> much that can be done to speed it up... >>> >>> I ran a quick test ... difference was too small to be noticeable ... > > Yeah, that's not surprising. > >> 2. I can't go and graft your libs, or do something similar with the >> current source because I don't want to break production code >> (pkg_install) of this importance and muck up a lot of user's systems >> irrevocably. > > If you can get enough people involved in testing, you may > be able to pull off a full rearchitecture. Requires some > careful political groundwork, though. ;-) Be friendly, talk about > your work in different mailing lists, make snapshot versions > of your code available, try to build a reputation as someone > who responds quickly to reported problems. I managed to get > support for replacing 'tar' this way, which is arguably just > as critical as pkg_install. > > Tim Kientzle Well, this certainly changes the problem description from what I thought it was quite a bit. I need to seriously brainstorm what needs to be done for a complete rewrite then, and talk to Kirill, and some of the other main players in the ports/pkg maintainers groups. For now I'm going to run quite a few iterations with a gprof'ed copy of my modified binaries and see if there's another point where things are slow, other than some of the code in deps.c that Stephen's submitted: , and speed it up as much as possible. -Garrett