From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 22 01:16:38 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 96DFB1065692 for ; Mon, 22 Feb 2010 01:16:38 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from monday.kientzle.com (kientzle.com [66.166.149.50]) by mx1.freebsd.org (Postfix) with ESMTP id 6830F8FC0C for ; Mon, 22 Feb 2010 01:16:38 +0000 (UTC) Received: (from root@localhost) by monday.kientzle.com (8.14.3/8.14.3) id o1M1GkbM019713; Mon, 22 Feb 2010 01:16:46 GMT (envelope-from kientzle@freebsd.org) Received: from dark.x.kientzle.com (fw2.kientzle.com [10.123.1.2]) by kientzle.com with SMTP id wk34rd24n52t4c6sqpatdq5v96; Mon, 22 Feb 2010 01:16:45 +0000 (UTC) (envelope-from kientzle@freebsd.org) Message-ID: <4B81DB31.70201@freebsd.org> Date: Sun, 21 Feb 2010 17:17:37 -0800 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.1.21) Gecko/20090601 SeaMonkey/1.1.16 MIME-Version: 1.0 To: Juergen Lock References: <20100217215940.GA19713@triton8.kn-bremen.de> <20100219181247.GA35702@triton8.kn-bremen.de> <4B7F711E.6040402@freebsd.org> <201002200100.48161.jkim@FreeBSD.org> <20100220103329.GA35467@triton8.kn-bremen.de> In-Reply-To: <20100220103329.GA35467@triton8.kn-bremen.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Duane.Hesser@gmail.com Subject: Re: "tar tfv /dev/cd0" speedup patch X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2010 01:16:38 -0000 Juergen, Could you try the current version of read_open_filename from: http://libarchive.googlecode.com/svn/trunk/libarchive/archive_read_open_filename.c You should be able to just copy it into your FreeBSD source tree and recompile. Duane Hesser's comments clarified for me that disk and tape I/O do require fundamentally different strategies, so I refactored this code to clearly break out those strategies. This in turn allowed me to do some straightforward I/O optimization of the disk case, which sped up your example by about 100x. I have a 2G DVD here on which "tar tvf /dev/acd0" took 90s originally, takes 14s with your patch, and now takes only 0.6s. For comparison, after mounting the disk, "find /mnt" takes 1.6s (which drops to .1s after the cache is warmed). It would be interesting to also compare two ways of copying all of the files: "tar xvf /dev/acd0" on an unmounted disk and "cp -R" on a mounted disk. The non-disk cases are all still handled generically, but there are now clearly-labeled holes where someone could add optimized I/O strategies for tapes, sockets, etc. I've even included a number of TODO comments about what strategies I think are worth exploring, including tricks like using MTIO ioctls on tape, mmap() for disk, and nonblocking I/O for sockets and pipes. I expect any one of these is around a dozen lines of code, so should be within the reach of even fairly junior developers. Tim