From owner-freebsd-fs@FreeBSD.ORG Tue May 29 18:30:00 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BCF26106566B; Tue, 29 May 2012 18:30:00 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id 60EAF8FC08; Tue, 29 May 2012 18:30:00 +0000 (UTC) Received: by yenl8 with SMTP id l8so2905629yen.13 for ; Tue, 29 May 2012 11:29:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=V9UJ8SG6CcNdoDeeNHlrRufaFBwQDkC51ejsZQmBgtU=; b=o6276NSvMHBDuhxphTDYudNlCrLec+MBRYI+IlKUpwbPFES/D5yFRe6DHSEI8xoyVm t7S71nqnjWYe60vZA4AKn+Pyoyw6u69Jy5cyt1sOHpc25m63UHkfuFDO/SUtE4C+L/CK z7rKyLlHndWA66CTDbjBNN7QMW/I80CW6sRDLSWLfNAsCto8od1RMOnjngZtdDMPAAJl +JeX7Od45gMSoBTVI8PXozPwdpC8qyPdh55ElRi+j+cUCUxephYqljvWkOukPX7Ws6ms d5BT/Wt/6E2Zvq4l4/mnCOjVQFFmWwN2npOt52Hx4mV5rdItCWv3y5D2KYVAvvrWdini 5GZQ== MIME-Version: 1.0 Received: by 10.50.203.41 with SMTP id kn9mr4642406igc.72.1338316197812; Tue, 29 May 2012 11:29:57 -0700 (PDT) Received: by 10.231.244.8 with HTTP; Tue, 29 May 2012 11:29:57 -0700 (PDT) In-Reply-To: <4FC4FB3F.8030203@freebsd.org> References: <49722655.1520.1338282954302.JavaMail.root@zimbra.interconnessioni.it> <4FC4FB3F.8030203@freebsd.org> Date: Wed, 30 May 2012 02:29:57 +0800 Message-ID: From: Marcelo Araujo To: Julian Elischer Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: araujo@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 18:30:00 -0000 2012/5/30 Julian Elischer > On 5/29/12 2:15 AM, Alessio Focardi wrote: > >> I ran a Usenet server this way for quite a while with fairly good >>>> results, though the average file size was a bit bigger, about 2K or >>>> so. >>>> I found that if I didn't use "-o space" that space optimization >>>> wouldn't >>>> kick in soon enough and I'd tend to run out of full blocks that >>>> would be >>>> needed for larger files. >>>> >>> Fragmentation is not a problem for me, mostly I will have a write >> once-read many situation, still is not clear to me if "-o space" works in >> the constraints of the block/fragment ratio, that in my case it would still >> mean that I will have to use a 512 bytes subblock for every 200 byte files. >> >> ps >> >> really thank you for all of your help! >> >> Maybe try use something other than a filesystem. The problem you will > have is that the physical media is going to be formatted to 512 bytes or > larger so every operation will be a read/modify/write if you try do to much > packing.. As others have said it makes a big difference if you are talking > about 20MB of data or 200GB of data too.. If it fits in ram, you could use > 'mailbox format' and pack them all in one file and then just load it into > ram when you need to access it. > > You could always buy one of our flash cards in key-value-store mode if you > need 1.3 TB of 512 byte values at 400,000 values per second(read/write), > but the pricetag might be a bit scary.. :-) (10k-ish) > > > Maybe you could take a look on Fusion-IO[1]. [1] http://www.fusionio.com/platforms/ -- Marcelo Araujo araujo@FreeBSD.org