From owner-freebsd-hackers@FreeBSD.ORG Wed Jul 18 10:08:01 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 95DBE16A404; Wed, 18 Jul 2007 10:08:01 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from mxout1.cac.washington.edu (mxout1.cac.washington.edu [140.142.32.134]) by mx1.freebsd.org (Postfix) with ESMTP id 7009613C4C6; Wed, 18 Jul 2007 10:08:01 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from smtp.washington.edu (smtp.washington.edu [140.142.33.7] (may be forged)) by mxout1.cac.washington.edu (8.13.7+UW06.06/8.13.7+UW07.06) with ESMTP id l6IA80wP020046 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 18 Jul 2007 03:08:01 -0700 X-Auth-Received: from [192.168.10.45] (c-24-10-12-194.hsd1.ca.comcast.net [24.10.12.194]) (authenticated authid=youshi10) by smtp.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l6IA7xeN017308 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 18 Jul 2007 03:08:00 -0700 Message-ID: <469DE67F.5030801@u.washington.edu> Date: Wed, 18 Jul 2007 03:07:59 -0700 From: Garrett Cooper User-Agent: Thunderbird 2.0.0.4 (Windows/20070604) MIME-Version: 1.0 To: krion@freebsd.org, hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-PMX-Version: 5.3.2.304607, Antispam-Engine: 2.5.1.298604, Antispam-Data: 2007.7.18.24433 X-Uwash-Spam: Gauge=IIIIIII, Probability=7%, Report='__CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_MSGID 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0, __USER_AGENT 0' Cc: Subject: Proposal for alleviating disk read / write time X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jul 2007 10:08:01 -0000 Hello Kirill and Hackers, After reviewing the changes I made last weekend to pkg_install I'm seeing slight improvements, but not a large amount of improvement in overall program operation. I haven't implemented mmap(2) yet as John Baldwin suggested (need to do some reading about fcntl(2) tomorrow because I want to implement locking), so I should see a large boost then, but I was wondering if I should do something similar to the following to reduce redundant disk accesses. 1. Lock +CONTENTS and related files exclusively for writes. Allow shared reads among separate pkg_install (and derivative programs) processes, and derivative packing lists with a limited set and use a LRU type algorithm to move packing lists in and out of shared memory. 2. If shared memory amongst separate processes is possible, share the ports INDEX's entire package list as a global object, and update only if the INDEX's real and virtual copy get out of sync (i.e. local copy is replaced, modified, or touch(1)'ed). +CONTENTS and INDEX are read frequently (once per process set) and although modifying pkg_install won't benefit overall operation speed without implementing a different data structure setup (still aiming for something like red-black trees for larger package lists like INDEX has, and linked lists for the smaller package lists), I think that the above changes would reduce wait times as pkg_install and installing ports becomes more and more concurrent. Clarifications and comments are more than welcome. I really appreciate it. Also, if someone can point me to the unofficial locking guidelines page (I remember it coming up twice in the past, but I'm busy now and wondering if someone can quickly point me to the page), it'd be more than appreciated. Thanks! -Garrett