Date: Wed, 13 Aug 2008 10:32:44 -0400 From: Bill Moran <wmoran@potentialtech.com> To: Laszlo Nagy <gandalf@shopzeus.com> Cc: freebsd-questions@freebsd.org Subject: Re: Max. number of opened files, efficiency Message-ID: <20080813103244.d9c76715.wmoran@potentialtech.com> In-Reply-To: <48A2EBD7.9000903@shopzeus.com> References: <48A2EBD7.9000903@shopzeus.com>
next in thread | previous in thread | raw e-mail | index | archive | help
In response to Laszlo Nagy <gandalf@shopzeus.com>: > How many files can I open under FreeBSD, at the same time? > > Problem: I'm making a pivot table, and when I drill down the facts, I > would like to create a new temporary file for each possible dimension > value. In most cases, there will be less than 1000 dimension values. I > tried to open 1000 temporary files and I could do so within one second. > > But how efficient is that? What happens when I open 1000 temporary > files, and write data into them randomly, 10 million times. (avg. 10 000 > write operations per file) Will this be handled efficiently by the OS? > Is efficiency affected by the underlying filesystem? > > I also tried to create 10 000 temporary files, but performance dropped down. > > Example in Python: > > import tempfile > import time > N = 10000 > start = time.time() > files = [ tempfile.TemporaryFile() for i in range(N)] > stop = time.time() > print "created %s files/second" % ( int(N/(stop-start)) ) > > On my computer this program prints "3814 files/second" for N=1000, and > "1561 files/second" for N=10000. Directories generally start to perform poorly when you put too many files in them (i.e. the time required to add a new directory entry or find an existing name in the entry goes up) If you're going to be making 10s of 1000s of files, I'd recommend making a tree of directories. I.e., make directories 1 - 10, then put files 0-999 in directory 1 and files 1000-1999 in directory 2, etc -- Bill Moran http://www.potentialtech.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080813103244.d9c76715.wmoran>