Date: Sun, 17 Jun 2007 15:20:38 -0700 From: Garrett Cooper <youshi10@u.washington.edu> To: Tim Kientzle <kientzle@freebsd.org> Cc: hackers@freebsd.org, Andrew Turner <andrew@fubar.geek.nz> Subject: Re: Rearchitecting pkg_install (was Re: Using shell commands versus C equivalents) Message-ID: <4675B3B6.2040603@u.washington.edu> In-Reply-To: <4675B088.2040408@freebsd.org> References: <Pine.LNX.4.43.0706131018120.25469@hymn01.u.washington.edu> <46703EE9.1030804@freebsd.org> <4674B268.4030502@u.washington.edu> <4674BE32.300@freebsd.org> <20070617203640.334524fc@hermies.int.fubar.geek.nz> <467560F4.9050007@u.washington.edu> <4675B088.2040408@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Tim Kientzle wrote: >>> I've also seen a 3x speedup by using my reimplementation of pkg_add >>> using my package management library, libpkg (http://libpkg.berlios.de). >>> It is not production ready yet as if it fails partway through an >>> installation it won't clean up and installed files. > > I *think* a good way to do this is to: > 1) Write each file to a temporary name (as Peter Jeremy > suggested earlier in this thread) > 2) Write a +INPROGRESS file to the package dir with > the temporary name of each file as it is written. > 3) Recursively install dependencies. > 4) Rename the files to their final names. > > This sequence allows you to: > * Do conflict checking on the fly. If a single file > conflicts at any point, you can detect that, ask > the user for a resolution at that point and drop > the single file or back out the entire package. > * Recover crashed installations. Just scan for all > +INPROGRESS files and remove every file mentioned. > > There are a few additional steps you can take to provide > even better failure recovery, but this covers the big > concerns. > >>>> I haven't looked closely at pkg_delete, but I doubt there's >>>> much that can be done to speed it up... >>> >>> I ran a quick test ... difference was too small to be noticeable ... > > Yeah, that's not surprising. > >> 2. I can't go and graft your libs, or do something similar with the >> current source because I don't want to break production code >> (pkg_install) of this importance and muck up a lot of user's systems >> irrevocably. > > If you can get enough people involved in testing, you may > be able to pull off a full rearchitecture. Requires some > careful political groundwork, though. ;-) Be friendly, talk about > your work in different mailing lists, make snapshot versions > of your code available, try to build a reputation as someone > who responds quickly to reported problems. I managed to get > support for replacing 'tar' this way, which is arguably just > as critical as pkg_install. > > Tim Kientzle Well, this certainly changes the problem description from what I thought it was quite a bit. I need to seriously brainstorm what needs to be done for a complete rewrite then, and talk to Kirill, and some of the other main players in the ports/pkg maintainers groups. For now I'm going to run quite a few iterations with a gprof'ed copy of my modified binaries and see if there's another point where things are slow, other than some of the code in deps.c that Stephen's submitted: <http://www.freebsd.org/cgi/query-pr.cgi?pr=112630>, and speed it up as much as possible. -Garrett
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4675B3B6.2040603>