Date: Wed, 30 May 2012 02:29:57 +0800 From: Marcelo Araujo <araujobsdport@gmail.com> To: Julian Elischer <julian@freebsd.org> Cc: freebsd-fs@freebsd.org Subject: Re: Millions of small files: best filesystem / best options Message-ID: <CAOfEmZhu_CziW9Sb9ZAtaaC1R=wUd8eJNpsWUQanmDeBfpAa2g@mail.gmail.com> In-Reply-To: <4FC4FB3F.8030203@freebsd.org> References: <49722655.1520.1338282954302.JavaMail.root@zimbra.interconnessioni.it> <4FC4FB3F.8030203@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
2012/5/30 Julian Elischer <julian@freebsd.org> > On 5/29/12 2:15 AM, Alessio Focardi wrote: > >> I ran a Usenet server this way for quite a while with fairly good >>>> results, though the average file size was a bit bigger, about 2K or >>>> so. >>>> I found that if I didn't use "-o space" that space optimization >>>> wouldn't >>>> kick in soon enough and I'd tend to run out of full blocks that >>>> would be >>>> needed for larger files. >>>> >>> Fragmentation is not a problem for me, mostly I will have a write >> once-read many situation, still is not clear to me if "-o space" works in >> the constraints of the block/fragment ratio, that in my case it would still >> mean that I will have to use a 512 bytes subblock for every 200 byte files. >> >> ps >> >> really thank you for all of your help! >> >> Maybe try use something other than a filesystem. The problem you will > have is that the physical media is going to be formatted to 512 bytes or > larger so every operation will be a read/modify/write if you try do to much > packing.. As others have said it makes a big difference if you are talking > about 20MB of data or 200GB of data too.. If it fits in ram, you could use > 'mailbox format' and pack them all in one file and then just load it into > ram when you need to access it. > > You could always buy one of our flash cards in key-value-store mode if you > need 1.3 TB of 512 byte values at 400,000 values per second(read/write), > but the pricetag might be a bit scary.. :-) (10k-ish) > > > Maybe you could take a look on Fusion-IO[1]. [1] http://www.fusionio.com/platforms/ -- Marcelo Araujo araujo@FreeBSD.org
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOfEmZhu_CziW9Sb9ZAtaaC1R=wUd8eJNpsWUQanmDeBfpAa2g>