From owner-freebsd-questions@FreeBSD.ORG Wed Aug 13 14:32:35 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 39A1E1065688 for ; Wed, 13 Aug 2008 14:32:35 +0000 (UTC) (envelope-from wmoran@potentialtech.com) Received: from mail.potentialtech.com (internet.potentialtech.com [66.167.251.6]) by mx1.freebsd.org (Postfix) with ESMTP id 169118FC16 for ; Wed, 13 Aug 2008 14:32:34 +0000 (UTC) (envelope-from wmoran@potentialtech.com) Received: from vanquish.ws.pitbpa0.priv.collaborativefusion.com (pr40.pitbpa0.pub.collaborativefusion.com [206.210.89.202]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.potentialtech.com (Postfix) with ESMTPSA id 0BA36EBC09; Wed, 13 Aug 2008 10:32:34 -0400 (EDT) Date: Wed, 13 Aug 2008 10:32:44 -0400 From: Bill Moran To: Laszlo Nagy Message-Id: <20080813103244.d9c76715.wmoran@potentialtech.com> In-Reply-To: <48A2EBD7.9000903@shopzeus.com> References: <48A2EBD7.9000903@shopzeus.com> X-Mailer: Sylpheed 2.5.0 (GTK+ 2.12.11; i386-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-questions@freebsd.org Subject: Re: Max. number of opened files, efficiency X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Aug 2008 14:32:35 -0000 In response to Laszlo Nagy : > How many files can I open under FreeBSD, at the same time? > > Problem: I'm making a pivot table, and when I drill down the facts, I > would like to create a new temporary file for each possible dimension > value. In most cases, there will be less than 1000 dimension values. I > tried to open 1000 temporary files and I could do so within one second. > > But how efficient is that? What happens when I open 1000 temporary > files, and write data into them randomly, 10 million times. (avg. 10 000 > write operations per file) Will this be handled efficiently by the OS? > Is efficiency affected by the underlying filesystem? > > I also tried to create 10 000 temporary files, but performance dropped down. > > Example in Python: > > import tempfile > import time > N = 10000 > start = time.time() > files = [ tempfile.TemporaryFile() for i in range(N)] > stop = time.time() > print "created %s files/second" % ( int(N/(stop-start)) ) > > On my computer this program prints "3814 files/second" for N=1000, and > "1561 files/second" for N=10000. Directories generally start to perform poorly when you put too many files in them (i.e. the time required to add a new directory entry or find an existing name in the entry goes up) If you're going to be making 10s of 1000s of files, I'd recommend making a tree of directories. I.e., make directories 1 - 10, then put files 0-999 in directory 1 and files 1000-1999 in directory 2, etc -- Bill Moran http://www.potentialtech.com