From owner-freebsd-current@FreeBSD.ORG Fri Aug 26 10:14:43 2005 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CC9B816A41F for ; Fri, 26 Aug 2005 10:14:43 +0000 (GMT) (envelope-from PeterJeremy@optushome.com.au) Received: from mail18.syd.optusnet.com.au (mail18.syd.optusnet.com.au [211.29.132.199]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2EA4A43D45 for ; Fri, 26 Aug 2005 10:14:42 +0000 (GMT) (envelope-from PeterJeremy@optushome.com.au) Received: from cirb503493.alcatel.com.au (c220-239-19-236.belrs4.nsw.optusnet.com.au [220.239.19.236]) by mail18.syd.optusnet.com.au (8.12.11/8.12.11) with ESMTP id j7QAEcio024410 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Fri, 26 Aug 2005 20:14:40 +1000 Received: from cirb503493.alcatel.com.au (localhost.alcatel.com.au [127.0.0.1]) by cirb503493.alcatel.com.au (8.12.10/8.12.10) with ESMTP id j7QAEcSR065030; Fri, 26 Aug 2005 20:14:38 +1000 (EST) (envelope-from pjeremy@cirb503493.alcatel.com.au) Received: (from pjeremy@localhost) by cirb503493.alcatel.com.au (8.12.10/8.12.9/Submit) id j7QAEaKe065029; Fri, 26 Aug 2005 20:14:36 +1000 (EST) (envelope-from pjeremy) Date: Fri, 26 Aug 2005 20:14:36 +1000 From: Peter Jeremy To: Marian Hettwer Message-ID: <20050826101436.GJ37107@cirb503493.alcatel.com.au> References: <430E06AA.2000907@kernel32.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <430E06AA.2000907@kernel32.de> User-Agent: Mutt/1.4.2i Cc: freebsd-current@freebsd.org Subject: Re: filesystem performance with lots of small files X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Aug 2005 10:14:43 -0000 On Thu, 2005-Aug-25 19:58:02 +0200, Marian Hettwer wrote: >Back to the topic. I have a directory with several thousands (800k and >more) small files. UFS2 shows a pretty low performance. Is your problem lots of small files or lots of files in a single directory? These are totally different problems. And what do you mean by "pretty low performance"? What are you measuring? Unix filesystems use linear searching of directories. UFS and UFS_DIRHASH have some performance improvements but at some point you need to scan the entire directory to determine if a filename is or is not present. Your solution is to avoid having lots of files in a single UFS directory: Either use a directory tree (like squid and some inn options) or use an inode filesystem (which I thought had been committed but I can't see it in NOTES). For "lots of small files", any filesystem is going to have relatively low I/O performance because the overheads involved in accessing the first block of a file are fixed and you don't get any benefit from large-block sequential read-ahead that means that reading 64K-128K isn't much slower than reading 1K. -- Peter Jeremy