From owner-freebsd-fs@FreeBSD.ORG Sun May 27 11:59:37 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B5462106564A for ; Sun, 27 May 2012 11:59:37 +0000 (UTC) (envelope-from Martin.Birgmeier@aon.at) Received: from email.aon.at (smtpout03.highway.telekom.at [195.3.96.115]) by mx1.freebsd.org (Postfix) with ESMTP id 050AF8FC12 for ; Sun, 27 May 2012 11:59:36 +0000 (UTC) Received: (qmail 19809 invoked from network); 27 May 2012 11:59:29 -0000 X-Spam-Checker-Version: SpamAssassin 3.2.0 (2007-05-01) on WARSBL605.highway.telekom.at X-Spam-Level: Received: from 91-115-52-194.adsl.highway.telekom.at (HELO gandalf.xyzzy) ([91.115.52.194]) (envelope-sender ) by smarthub79.res.a1.net (qmail-ldap-1.03) with AES256-SHA encrypted SMTP for ; 27 May 2012 11:59:27 -0000 Received: from mizar-v1.xyzzy (mizar-v1.xyzzy [192.168.1.51]) by gandalf.xyzzy (8.14.5/8.14.5) with ESMTP id q4RBwt0P022102; Sun, 27 May 2012 13:59:08 +0200 (CEST) (envelope-from Martin.Birgmeier@aon.at) Message-ID: <4FC21702.6070005@aon.at> Date: Sun, 27 May 2012 13:58:58 +0200 From: Martin Birgmeier User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:12.0) Gecko/20120502 Thunderbird/12.0.1 MIME-Version: 1.0 To: Andrey Simonenko References: <201205200810.q4K8A4KP087730@freefall.freebsd.org> <20120522080456.GA40365@pm513-1.comsys.ntu-kpi.kiev.ua> In-Reply-To: <20120522080456.GA40365@pm513-1.comsys.ntu-kpi.kiev.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org Subject: Re: kern/136865: [nfs] [patch] NFS exports atomic and on-the-fly atomic updates X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 May 2012 11:59:37 -0000 Hi Andrey, One more question: I am running 8.3, 9.0, and 7.4 on various machines. Do you have patches for these versions, too? Regards, Martin On 05/22/12 10:04, Andrey Simonenko wrote: > Hello, > > On Sun, May 20, 2012 at 08:10:04AM +0000, Martin Birgmeier wrote: >> The following reply was made to PR kern/136865; it has been noted by GNATS. >> >> From: Martin Birgmeier >> To: bug-followup@FreeBSD.org, simon@comsys.ntu-kpi.kiev.ua >> Cc: >> Subject: Re: kern/136865: [nfs] [patch] NFS exports atomic and on-the-fly >> atomic updates >> Date: Sun, 20 May 2012 10:04:01 +0200 >> >> Dear Andrey, >> >> It seems that you have done some great work here, and I would really >> like to see this integrated into the core FreeBSD distribution (I was >> the submitter of http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/131342). >> >> I would like to try out your patches and have two questions: >> >> - Do your patches support multiple zfs sharenfs specifications as >> proposed in http://www.freebsd.org/cgi/query-pr.cgi?pr=147881 (I am >> using this)? > The exports(5) manual page says that address specifications must be specified > after options. The nfs.exports(5) file format allows to use options after > address specifications, so they can overwrite previously specified options. > > It is possible to specify all settings for one file system in one line, > no ';' like separators are required. > > For example line: > > /fs -ro -sec krb5 1.1.1.1 -nfsv4 no -rw 2.2.2.2 -sec sys -nfsv4 yes 3.3.3.3 > > will be translated to ("nfse -t ..." output): > > Pathname /fs > Export specifications: > -rw -sec sys -maproot=-2:-2 -host 3.3.3.3 > -rw -sec krb5 -maproot=-2:-2 -nfsv4 no -host 2.2.2.2 > -ro -sec krb5 -maproot=-2:-2 -host 1.1.1.1 > >> >> - Could you give a concise list of incompatibilities (and even >> regressions if they should exist at all) of your solution compared to >> the standard one? - As to the advantages, I am already convinced. :-) > In short: if nfse is run in compatible mode with mountd ("nfse -C ..."), > then it is more compatible with exports(5) than mountd is. If one did > not follow rules of exports(5), then "nfse -C ..." can be incompatible > with mountd. > > If nfse is run in native nfs.export(5) configuration file format mode, > then logic of configuration looks like exports(5), but differs in some > places. > > So, when we speak about "incompatibilities" then it is necessary to > distinguish incompatibilities of "nfse native mode" vs mountd and > incompatibilities of "nfse compatible mode" vs mountd. > > I suggest to check whether "nfse -C ..." is compatible with mountd > using instructions described here: > > http://lists.freebsd.org/pipermail/freebsd-fs/2010-May/008421.html > > You do not need to install anything or modify existent system for > testing. Can you try "nfse -Ct ..." and tell me whether "nfse -C ..." > is compatible enough with mountd (try correct configurations and > configurations with mistakes). > > I have list of difference somewhere, I'll try to find it. > > From owner-freebsd-fs@FreeBSD.ORG Mon May 28 01:38:12 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2A6F2106564A for ; Mon, 28 May 2012 01:38:12 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id D74478FC08 for ; Mon, 28 May 2012 01:38:11 +0000 (UTC) Received: by vbmv11 with SMTP id v11so2084675vbm.13 for ; Sun, 27 May 2012 18:38:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:date:message-id:subject:from:to:content-type; bh=yGIEFNfT+MFCPVIzUq1lDTVJ2lryu2gIGfBKFFz4nFw=; b=XOT8AQXtNmg0FvRFxZjZioi2dcJTGcSHknkI3wx0h5NXGzvovmgEUslmo0hxlFHb+E prjEyETuzIf4gllOHPr+ip3IgM+3f9EiNOustpZZnYNc7+KRbY39koU52a+4bWCPUzyj jyUswbIicp+4auioisYL8Ktv/zIyjrFIfr8QBzzgTKEp3yXtSbQY9MCTu1V0TgqG/enH t/DXTZBiKIofnJioVsYMxowtdfrg8pKBhWX8cWK/E/DrURXyf1LpgL8OqB849zfRnUP1 ufxY6Zwj9EsU3Xvddtv5LZeGIvPEcQ9/1wk8EswziSjHkOSfHMnk7XGc2k/eKf5G36Bt mU1Q== MIME-Version: 1.0 Received: by 10.52.94.36 with SMTP id cz4mr5976655vdb.10.1338169091122; Sun, 27 May 2012 18:38:11 -0700 (PDT) Received: by 10.220.160.15 with HTTP; Sun, 27 May 2012 18:38:11 -0700 (PDT) Date: Mon, 28 May 2012 09:38:11 +0800 Message-ID: From: Marcelo Araujo To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: ZFS new SPA X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: araujo@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 01:38:12 -0000 Hello all, I'm wondering if there is someone working on the new SPA released some days ago. Just to let everybody knows, I'm working on it from last week till now, partially it has been integrated with FreeBSD code(HEAD), but I need couple days or more to finish the code integration as well as perform some tests. In case there is someone else working on it, please let me know, because I don't want wasting my time. 2619 asynchronous destruction of ZFS file systems 2747 SPA versioning with zfs feature flags Best Regards, -- Marcelo Araujo araujo@FreeBSD.org From owner-freebsd-fs@FreeBSD.ORG Mon May 28 04:23:54 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F1DBB106566C for ; Mon, 28 May 2012 04:23:54 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: from mail-wi0-f170.google.com (mail-wi0-f170.google.com [209.85.212.170]) by mx1.freebsd.org (Postfix) with ESMTP id 8AA0D8FC08 for ; Mon, 28 May 2012 04:23:54 +0000 (UTC) Received: by wibhm6 with SMTP id hm6so1095538wib.1 for ; Sun, 27 May 2012 21:23:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=c/1ayi7/fz6vj8RyldBf1++W76QGSjdm41q5Pd/zvh4=; b=K2j97Hr02XUWshU9N79vhujNL9JQic6LkMPumvX8eUvClTfqH3HLIEpVAqqoSo5RBd EV9+FzKF5goOCg8RV8Y4BqSQwokpGbpgFDksKuC88P5HEVauCoy/gMSzu4MuDBkoB5yh s+cMyjxFfvVA2rtQe4qovGonCdm4CYZPixNzHlRoFqT9gHDpZnzQeKWP1ACjFjLmori9 QXADnimKFL0t64QmLFa36l+qs0TLT4kA9guv4lkYNdUj7onzQjboxCivvNT+HKpt64Bg tiuaVoWZReqisbxluixUtl6m64RkZFassWmH0WsPV5uj6rJxfk3ttKPkkCFIY1eDsHTr g8cA== MIME-Version: 1.0 Received: by 10.180.86.194 with SMTP id r2mr12363274wiz.15.1338179028328; Sun, 27 May 2012 21:23:48 -0700 (PDT) Received: by 10.180.84.39 with HTTP; Sun, 27 May 2012 21:23:48 -0700 (PDT) Date: Mon, 28 May 2012 00:23:48 -0400 Message-ID: From: grarpamp To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 Subject: ZFS new SPA X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 04:23:55 -0000 Though it seems unknown [to me] if Oracle [after they worked over Sun] will ever release the source again... ...it's nice to see the community coming together somewhere... https://www.illumos.org/projects/illumos-gate/issues ...and those updates starting to trickle down. Thanks! Mostly just posting the link, because this work [SPA, etc] was news to me and surely other passers by. From owner-freebsd-fs@FreeBSD.ORG Mon May 28 06:50:39 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BE69A1065672; Mon, 28 May 2012 06:50:39 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (mail.vx.sk [IPv6:2a01:4f8:150:6101::4]) by mx1.freebsd.org (Postfix) with ESMTP id 7B9298FC15; Mon, 28 May 2012 06:50:39 +0000 (UTC) Received: from core2.vx.sk (localhost [127.0.0.2]) by mail.vx.sk (Postfix) with ESMTP id D0CBB11F1E; Mon, 28 May 2012 08:50:38 +0200 (CEST) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk by core2.vx.sk (amavisd-new, unix socket) with LMTP id P5W1BNGf9uGJ; Mon, 28 May 2012 08:50:36 +0200 (CEST) Received: from [10.9.8.1] (188-167-78-15.dynamic.chello.sk [188.167.78.15]) by mail.vx.sk (Postfix) with ESMTPSA id C7DFF11F10; Mon, 28 May 2012 08:50:36 +0200 (CEST) Message-ID: <4FC3203B.1040205@FreeBSD.org> Date: Mon, 28 May 2012 08:50:35 +0200 From: Martin Matuska User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: araujo@FreeBSD.org References: In-Reply-To: X-Enigmail-Version: 1.4.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Marcelo Araujo , freebsd-fs@freebsd.org Subject: Re: ZFS new SPA X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 06:50:39 -0000 On 28.5.2012 3:38, Marcelo Araujo wrote: > Hello all, > > I'm wondering if there is someone working on the new SPA released some days > ago. > Just to let everybody knows, I'm working on it from last week till now, > partially it has been integrated with FreeBSD code(HEAD), but I need couple > days or more to finish the code integration as well as perform some tests. > > In case there is someone else working on it, please let me know, because I > don't want wasting my time. > > 2619 asynchronous destruction of ZFS file systems > 2747 SPA versioning with zfs feature flags > > > Best Regards, I am working on this, too. That's why I pushed r236143, r236145, r236146, r236155 - that makes integration much easier. I got a working patch for Illumos 13700 and 13701 in about 2-3 hrs yesterday. The hardest part to cope with was fnvpair.c in userland mode. Next, Illumos has a typo in the code, maybe they didn't compile it? In addition it leaks memory, so we have to find the leak and plug it. Cheers, mm -- Martin Matuska FreeBSD committer http://blog.vx.sk From owner-freebsd-fs@FreeBSD.ORG Mon May 28 06:53:07 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E61BB1065675; Mon, 28 May 2012 06:53:07 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-gh0-f182.google.com (mail-gh0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 8B8C18FC1D; Mon, 28 May 2012 06:53:07 +0000 (UTC) Received: by ghbz22 with SMTP id z22so1239240ghb.13 for ; Sun, 27 May 2012 23:53:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=BxZJ6D7yZ4sAcx4PCR5nbn0VYaGS/J6lF4tuStdY1uQ=; b=ETrMS13ddEMmRBtJKTeLbVeckP1W8zE0DETJUiPBzYqcGX45poo0RVExmj6JKV2UBP D6cvVuaxlX8sqK5Hyuyn17PQR6EV3xzfgAU1X/1c1qUWog6GkF5oJugjl6alMYBFuJto AX0/t1LZcWuQVJStsucYznpoUN5pSG41KADdk0Yd7I5ldMmwumGTxGdDZljN5MDMK44n c1wmmvCzpCxB6T7CHLHZCXausSgRdLxVifLovyBowMWOGeDR33Mn3Nbnh5HGrx/BIF7q vcl26PNMtfH7OHBp0zK+//tA6PGxllT3A2vDo5mHGel/y01Vp0X1lGEO77DN9C8PyoMt amnA== MIME-Version: 1.0 Received: by 10.50.157.194 with SMTP id wo2mr3762440igb.72.1338187985928; Sun, 27 May 2012 23:53:05 -0700 (PDT) Received: by 10.231.244.8 with HTTP; Sun, 27 May 2012 23:53:05 -0700 (PDT) In-Reply-To: <4FC3203B.1040205@FreeBSD.org> References: <4FC3203B.1040205@FreeBSD.org> Date: Mon, 28 May 2012 14:53:05 +0800 Message-ID: From: Marcelo Araujo To: Martin Matuska Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS new SPA X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: araujo@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 06:53:08 -0000 2012/5/28 Martin Matuska > > I am working on this, too. > That's why I pushed r236143, r236145, r236146, r236155 - that makes > integration much easier. > > I got a working patch for Illumos 13700 and 13701 in about 2-3 hrs > yesterday. > The hardest part to cope with was fnvpair.c in userland mode. > Next, Illumos has a typo in the code, maybe they didn't compile it? > In addition it leaks memory, so we have to find the leak and plug it. > > Cheers, > mm > > Well, in this case, should us join effort? I=92m gonna stop to make the integration, due you also are doing the same tasks. Best Regards, --=20 Marcelo Araujo araujo@FreeBSD.org From owner-freebsd-fs@FreeBSD.ORG Mon May 28 07:17:13 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BA3441065672 for ; Mon, 28 May 2012 07:17:13 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (ns1.bitblocks.com [173.228.5.8]) by mx1.freebsd.org (Postfix) with ESMTP id 948A18FC12 for ; Mon, 28 May 2012 07:17:13 +0000 (UTC) Received: from bitblocks.com (localhost [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id A917AB827 for ; Mon, 28 May 2012 00:09:47 -0700 (PDT) To: freebsd-fs@freebsd.org Date: Mon, 28 May 2012 00:09:47 -0700 From: Bakul Shah Message-Id: <20120528070947.A917AB827@mail.bitblocks.com> Subject: zfs disk replace issue X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 07:17:13 -0000 I have a zpool with 2 mirrors of two disks each. 3 of the disks are 1TB. I replaced three original 300GB disks with the TB disks and there were no problems Recently I upgraded to a new machine and trasferred the old zfs disks to the new machine and everything was ok. I then replaced the final 300GB disk with a 1TB disk. I noticed that after resilver finished (in two hours), "zpool status" kept showing 'replacing 0' and showed the old and new disk in the pool. I thought it would automatically take out the old disk? So then I manually "zpool detach"ed the old disk but the size of the mirror has not changed. Is this a bug or did I miss some step? I'd appreciate any help to make the extra space usable! This pool is root so it was mounted when I did this. May be that was the problem? Thanks, Bakul Rough transcript follows: $ zpool iostat -v capacity operations bandwidth pool alloc free read write read write ---------- ----- ----- ----- ----- ----- ----- z 832G 330G 49 98 887K 471K mirror 603G 327G 37 64 800K 329K ada2p1 - - 19 20 425K 330K ada3p1 - - 19 20 409K 330K mirror 229G 3.39G 12 33 86.9K 142K ada4p1 - - 6 33 47.0K 143K ada1p1 - - 6 33 64.7K 143K ---------- ----- ----- ----- ----- ----- ----- $ gpart list ada1 ada2 ada3 ada4 | grep -A2 p1 1. Name: ada1p1 Mediasize: 1000204851712 (931G) Sectorsize: 512 -- 1. Name: ada2p1 Mediasize: 1000204851712 (931G) Sectorsize: 512 -- 1. Name: ada3p1 Mediasize: 1000204851712 (931G) Sectorsize: 512 -- 1. Name: ada4p1 Mediasize: 1000204851712 (931G) Sectorsize: 512 From owner-freebsd-fs@FreeBSD.ORG Mon May 28 07:21:00 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 70E1A1065672 for ; Mon, 28 May 2012 07:21:00 +0000 (UTC) (envelope-from simon@comsys.ntu-kpi.kiev.ua) Received: from comsys.kpi.ua (comsys.kpi.ua [77.47.192.42]) by mx1.freebsd.org (Postfix) with ESMTP id C53538FC0A for ; Mon, 28 May 2012 07:20:59 +0000 (UTC) Received: from pm513-1.comsys.kpi.ua ([10.18.52.101] helo=pm513-1.comsys.ntu-kpi.kiev.ua) by comsys.kpi.ua with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1SYuGA-0002Zm-Gz; Mon, 28 May 2012 10:20:58 +0300 Received: by pm513-1.comsys.ntu-kpi.kiev.ua (Postfix, from userid 1001) id 3B3D91CC21; Mon, 28 May 2012 10:20:58 +0300 (EEST) Date: Mon, 28 May 2012 10:20:58 +0300 From: Andrey Simonenko To: Martin Birgmeier Message-ID: <20120528072058.GA1322@pm513-1.comsys.ntu-kpi.kiev.ua> References: <201205200810.q4K8A4KP087730@freefall.freebsd.org> <20120522080456.GA40365@pm513-1.comsys.ntu-kpi.kiev.ua> <4FC21702.6070005@aon.at> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4FC21702.6070005@aon.at> User-Agent: Mutt/1.5.21 (2010-09-15) X-Authenticated-User: simon@comsys.ntu-kpi.kiev.ua X-Authenticator: plain X-Sender-Verify: SUCCEEDED (sender exists & accepts mail) X-Exim-Version: 4.63 (build at 28-Apr-2011 07:11:12) X-Date: 2012-05-28 10:20:58 X-Connected-IP: 10.18.52.101:36223 X-Message-Linecount: 24 X-Body-Linecount: 6 X-Message-Size: 1032 X-Body-Size: 269 Cc: freebsd-fs@FreeBSD.org Subject: Re: kern/136865: [nfs] [patch] NFS exports atomic and on-the-fly atomic updates X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 07:21:00 -0000 On Sun, May 27, 2012 at 01:58:58PM +0200, Martin Birgmeier wrote: > > One more question: I am running 8.3, 9.0, and 7.4 on various machines. > Do you have patches for these versions, too? I just try to keep my changes for sys/ in sync with the CURRENT version only. From owner-freebsd-fs@FreeBSD.ORG Mon May 28 09:31:09 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 10C50106564A for ; Mon, 28 May 2012 09:31:09 +0000 (UTC) (envelope-from petri@helenius.fi) Received: from mail.helenius.fi (unknown [IPv6:2001:67c:164:40::91]) by mx1.freebsd.org (Postfix) with ESMTP id 91B048FC1B for ; Mon, 28 May 2012 09:31:08 +0000 (UTC) Received: from mail.helenius.fi (localhost [127.0.0.1]) by mail.helenius.fi (Postfix) with ESMTP id 97CD04D40 for ; Mon, 28 May 2012 09:31:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at helenius.fi Received: from mail.helenius.fi ([127.0.0.1]) by mail.helenius.fi (mail.helenius.fi [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 74nUiPzCzhlw for ; Mon, 28 May 2012 09:30:57 +0000 (UTC) Received: from dynamic.helenius.fi (unknown [IPv6:2001:1bc8:1018:0:3138:7495:2ba6:d6d0]) (Authenticated sender: pete) by mail.helenius.fi (Postfix) with ESMTPA for ; Mon, 28 May 2012 09:30:57 +0000 (UTC) From: Petri Helenius Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Mon, 28 May 2012 12:30:56 +0300 Message-Id: <979BE42A-6964-4FBD-8C44-B24E38DAC73C@helenius.fi> To: freebsd-fs@freebsd.org Mime-Version: 1.0 (Apple Message framework v1278) X-Mailer: Apple Mail (2.1278) Subject: zvol questions X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 09:31:09 -0000 Hi, How do I make the device entry in the /dev/zvol disappear after a = snapshot has been deleted? And how I make GEOM to ignore all zvol's so that some clever iscsi-user = does not exploit the system by trickery with GPT labels? Pete From owner-freebsd-fs@FreeBSD.ORG Mon May 28 11:07:26 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D827D1065674 for ; Mon, 28 May 2012 11:07:26 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id C1A1A8FC17 for ; Mon, 28 May 2012 11:07:26 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q4SB7QeB063332 for ; Mon, 28 May 2012 11:07:26 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q4SB7QSl063330 for freebsd-fs@FreeBSD.org; Mon, 28 May 2012 11:07:26 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 28 May 2012 11:07:26 GMT Message-Id: <201205281107.q4SB7QSl063330@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 11:07:26 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/168158 fs [zfs] incorrect parsing of sharenfs options in zfs (fs o kern/167979 fs [ufs] DIOCGDINFO ioctl does not work on 8.2 file syste o kern/167977 fs [smbfs] mount_smbfs results are differ when utf-8 or U o kern/167905 fs [zfs] zfs set canmount=on UNMOUNTS dataset o kern/167688 fs [fusefs] Incorrect signal handling with direct_io o kern/167685 fs [zfs] ZFS on USB drive prevents shutdown / reboot o kern/167612 fs [portalfs] The portal file system gets stuck inside po o kern/167272 fs [zfs] ZFS Disks reordering causes ZFS to pick the wron o kern/167266 fs [zfs] [nfs] ZFS + new NFS export (sharenfs) leads to N o kern/167260 fs [msdosfs] msdosfs disk was mounted the second time whe o kern/167109 fs [zfs] [panic] zfs diff kernel panic Fatal trap 9: gene o kern/167105 fs [nfs] mount_nfs can not handle source exports wiht mor o kern/167067 fs [zfs] [panic] ZFS panics the server o kern/167066 fs [zfs] ZVOLs not appearing in /dev/zvol o kern/167065 fs [zfs] boot fails when a spare is the boot disk o kern/167048 fs [nfs] [patch] RELEASE-9 crash when using ZFS+NULLFS+NF o kern/166912 fs [ufs] [panic] Panic after converting Softupdates to jo o kern/166851 fs [zfs] [hang] Copying directory from the mounted UFS di o kern/166566 fs [zfs] zfs split renders 2 disk (MBR based) mirror unbo o kern/166477 fs [nfs] NFS data corruption. o kern/165950 fs [ffs] SU+J and fsck problem o kern/165923 fs [nfs] Writing to NFS-backed mmapped files fails if flu o kern/165521 fs [zfs] [hang] livelock on 1 Gig of RAM with zfs when 31 o kern/165392 fs Multiple mkdir/rmdir fails with errno 31 o kern/165087 fs [unionfs] lock violation in unionfs o kern/164472 fs [ufs] fsck -B panics on particular data inconsistency o kern/164370 fs [zfs] zfs destroy for snapshot fails on i386 and sparc o kern/164261 fs [nullfs] [patch] fix panic with NFS served from NULLFS o kern/164256 fs [zfs] device entry for volume is not created after zfs o kern/164184 fs [ufs] [panic] Kernel panic with ufs_makeinode o kern/163801 fs [md] [request] allow mfsBSD legacy installed in 'swap' o kern/163770 fs [zfs] [hang] LOR between zfs&syncer + vnlru leading to o kern/163501 fs [nfs] NFS exporting a dir and a subdir in that dir to o kern/162944 fs [coda] Coda file system module looks broken in 9.0 o kern/162860 fs [zfs] Cannot share ZFS filesystem to hosts with a hyph o kern/162751 fs [zfs] [panic] kernel panics during file operations o kern/162591 fs [nullfs] cross-filesystem nullfs does not work as expe o kern/162519 fs [zfs] "zpool import" relies on buggy realpath() behavi o kern/162362 fs [snapshots] [panic] ufs with snapshot(s) panics when g o kern/161968 fs [zfs] [hang] renaming snapshot with -r including a zvo o kern/161897 fs [zfs] [patch] zfs partition probing causing long delay o kern/161864 fs [ufs] removing journaling from UFS partition fails on o bin/161807 fs [patch] add option for explicitly specifying metadata o kern/161579 fs [smbfs] FreeBSD sometimes panics when an smb share is o kern/161533 fs [zfs] [panic] zfs receive panic: system ioctl returnin o kern/161438 fs [zfs] [panic] recursed on non-recursive spa_namespace_ o kern/161424 fs [nullfs] __getcwd() calls fail when used on nullfs mou o kern/161280 fs [zfs] Stack overflow in gptzfsboot o kern/161205 fs [nfs] [pfsync] [regression] [build] Bug report freebsd o kern/161169 fs [zfs] [panic] ZFS causes kernel panic in dbuf_dirty o kern/161112 fs [ufs] [lor] filesystem LOR in FreeBSD 9.0-BETA3 o kern/160893 fs [zfs] [panic] 9.0-BETA2 kernel panic o kern/160860 fs [ufs] Random UFS root filesystem corruption with SU+J o kern/160801 fs [zfs] zfsboot on 8.2-RELEASE fails to boot from root-o o kern/160790 fs [fusefs] [panic] VPUTX: negative ref count with FUSE o kern/160777 fs [zfs] [hang] RAID-Z3 causes fatal hang upon scrub/impo o kern/160706 fs [zfs] zfs bootloader fails when a non-root vdev exists o kern/160591 fs [zfs] Fail to boot on zfs root with degraded raidz2 [r o kern/160410 fs [smbfs] [hang] smbfs hangs when transferring large fil o kern/160283 fs [zfs] [patch] 'zfs list' does abort in make_dataset_ha o kern/159930 fs [ufs] [panic] kernel core o kern/159402 fs [zfs][loader] symlinks cause I/O errors o kern/159357 fs [zfs] ZFS MAXNAMELEN macro has confusing name (off-by- o kern/159356 fs [zfs] [patch] ZFS NAME_ERR_DISKLIKE check is Solaris-s o kern/159351 fs [nfs] [patch] - divide by zero in mountnfs() o kern/159251 fs [zfs] [request]: add FLETCHER4 as DEDUP hash option o kern/159077 fs [zfs] Can't cd .. with latest zfs version o kern/159048 fs [smbfs] smb mount corrupts large files o kern/159045 fs [zfs] [hang] ZFS scrub freezes system o kern/158839 fs [zfs] ZFS Bootloader Fails if there is a Dead Disk o kern/158802 fs amd(8) ICMP storm and unkillable process. o kern/158231 fs [nullfs] panic on unmounting nullfs mounted over ufs o f kern/157929 fs [nfs] NFS slow read o kern/157399 fs [zfs] trouble with: mdconfig force delete && zfs strip o kern/157179 fs [zfs] zfs/dbuf.c: panic: solaris assert: arc_buf_remov o kern/156797 fs [zfs] [panic] Double panic with FreeBSD 9-CURRENT and o kern/156781 fs [zfs] zfs is losing the snapshot directory, p kern/156545 fs [ufs] mv could break UFS on SMP systems o kern/156193 fs [ufs] [hang] UFS snapshot hangs && deadlocks processes o kern/156039 fs [nullfs] [unionfs] nullfs + unionfs do not compose, re o kern/155615 fs [zfs] zfs v28 broken on sparc64 -current o kern/155587 fs [zfs] [panic] kernel panic with zfs p kern/155411 fs [regression] [8.2-release] [tmpfs]: mount: tmpfs : No o kern/155199 fs [ext2fs] ext3fs mounted as ext2fs gives I/O errors o bin/155104 fs [zfs][patch] use /dev prefix by default when importing o kern/154930 fs [zfs] cannot delete/unlink file from full volume -> EN o kern/154828 fs [msdosfs] Unable to create directories on external USB o kern/154491 fs [smbfs] smb_co_lock: recursive lock for object 1 p kern/154228 fs [md] md getting stuck in wdrain state o kern/153996 fs [zfs] zfs root mount error while kernel is not located o kern/153753 fs [zfs] ZFS v15 - grammatical error when attempting to u o kern/153716 fs [zfs] zpool scrub time remaining is incorrect o kern/153695 fs [patch] [zfs] Booting from zpool created on 4k-sector o kern/153680 fs [xfs] 8.1 failing to mount XFS partitions o kern/153520 fs [zfs] Boot from GPT ZFS root on HP BL460c G1 unstable o kern/153418 fs [zfs] [panic] Kernel Panic occurred writing to zfs vol o kern/153351 fs [zfs] locking directories/files in ZFS o bin/153258 fs [patch][zfs] creating ZVOLs requires `refreservation' s kern/153173 fs [zfs] booting from a gzip-compressed dataset doesn't w o kern/153126 fs [zfs] vdev failure, zpool=peegel type=vdev.too_small o kern/152022 fs [nfs] nfs service hangs with linux client [regression] o kern/151942 fs [zfs] panic during ls(1) zfs snapshot directory o kern/151905 fs [zfs] page fault under load in /sbin/zfs o bin/151713 fs [patch] Bug in growfs(8) with respect to 32-bit overfl o kern/151648 fs [zfs] disk wait bug o kern/151629 fs [fs] [patch] Skip empty directory entries during name o kern/151330 fs [zfs] will unshare all zfs filesystem after execute a o kern/151326 fs [nfs] nfs exports fail if netgroups contain duplicate o kern/151251 fs [ufs] Can not create files on filesystem with heavy us o kern/151226 fs [zfs] can't delete zfs snapshot o kern/151111 fs [zfs] vnodes leakage during zfs unmount o kern/150503 fs [zfs] ZFS disks are UNAVAIL and corrupted after reboot o kern/150501 fs [zfs] ZFS vdev failure vdev.bad_label on amd64 o kern/150390 fs [zfs] zfs deadlock when arcmsr reports drive faulted o kern/150336 fs [nfs] mountd/nfsd became confused; refused to reload n o kern/149208 fs mksnap_ffs(8) hang/deadlock o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147560 fs [zfs] [boot] Booting 8.1-PRERELEASE raidz system take o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an f bin/145309 fs bsdlabel: Editing disk label invalidates the whole dev o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o bin/143572 fs [zfs] zpool(1): [patch] The verbose output from iostat o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs p bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139564 fs [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdo o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/138662 fs [panic] ffs_blkfree: freeing free block o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic p kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/127787 fs [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfs o bin/127270 fs fsck_msdosfs(8) may crash if BytesPerSec is zero o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o kern/118318 fs [nfs] NFS server hangs under special circumstances o bin/118249 fs [ufs] mv(1): moving a directory changes its mtime o kern/118126 fs [nfs] [patch] Poor NFS server write performance o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o kern/117954 fs [ufs] dirhash on very large directories blocks the mac o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with o kern/116583 fs [ffs] [hang] System freezes for short time when using o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes s bin/97498 fs [request] newfs(8) has no option to clear the first 12 o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [cd9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o bin/94810 fs fsck(8) incorrectly reports 'file system marked clean' o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88555 fs [panic] ffs_blkfree: freeing free frag on AMD 64 o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o bin/87966 fs [patch] newfs(8): introduce -A flag for newfs to enabl o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o bin/85494 fs fsck_ffs: unchecked use of cg_inosused macro etc. o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o bin/74779 fs Background-fsck checks one filesystem twice and omits o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o bin/70600 fs fsck(8) throws files away when it can't grow lost+foun o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o bin/27687 fs fsck(8) wrapper is not properly passing options to fsc o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 277 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon May 28 11:51:48 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C384F106564A; Mon, 28 May 2012 11:51:48 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (mail.vx.sk [IPv6:2a01:4f8:150:6101::4]) by mx1.freebsd.org (Postfix) with ESMTP id 58CB28FC0A; Mon, 28 May 2012 11:51:48 +0000 (UTC) Received: from core2.vx.sk (localhost [127.0.0.2]) by mail.vx.sk (Postfix) with ESMTP id 71C9C31812; Mon, 28 May 2012 13:51:47 +0200 (CEST) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk by core2.vx.sk (amavisd-new, unix socket) with LMTP id XURcEAaoC4F9; Mon, 28 May 2012 13:51:45 +0200 (CEST) Received: from [10.9.8.1] (188-167-78-15.dynamic.chello.sk [188.167.78.15]) by mail.vx.sk (Postfix) with ESMTPSA id 1870D317FE; Mon, 28 May 2012 13:51:45 +0200 (CEST) Message-ID: <4FC366D0.6030206@FreeBSD.org> Date: Mon, 28 May 2012 13:51:44 +0200 From: Martin Matuska User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: In-Reply-To: X-Enigmail-Version: 1.4.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Marcelo Araujo Subject: [CFT][PREVIEW] ZFS new SPA X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 11:51:48 -0000 Hello all, I have ported the ZFS features support (SPA version 5000) from Illumos to FreeBSD-current. What is still missing is boot support - needs to be implemented. Patch against CURRENT: http://www.vx.sk/download/patches/freebsd/zfs/head-zfs-features.patch Amd64 ISO images for testing (bootable, work well in VirtualBox): Basic: http://www.vx.sk/download/ISO-images/mfsbsd/head-zfs-features.iso (86MB) With full installworld: http://www.vx.sk/download/ISO-images/mfsbsd/head-se-zfs-features.iso (239MB) TODO: boot support (check feature availability from ZFS boot code) References: https://hg.openindiana.org/upstream/illumos/illumos-gate/rev/2889e2596bd6 https://hg.openindiana.org/upstream/illumos/illumos-gate/rev/1949b688d5fb -- Martin Matuska FreeBSD committer http://blog.vx.sk From owner-freebsd-fs@FreeBSD.ORG Mon May 28 12:00:36 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA9B61065670; Mon, 28 May 2012 12:00:36 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id 502E38FC0A; Mon, 28 May 2012 12:00:36 +0000 (UTC) Received: by yhgm50 with SMTP id m50so1863862yhg.13 for ; Mon, 28 May 2012 05:00:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=VfVX6x7iiU2llZTDUnHDIfRDEQJ5FQcqx1DHOoYiwpU=; b=vz0RNND7qfXoQF4ZV87fcXQQBHGUtQzeQL04IkalAnj8YSGxBe1emhjj1SmbYp7ZFF whgpsJm/lnNcp6hrertksy8cGxtNHVOEuBrR5p1+l97PIC85n1yRHxMP9xEvWxPmUCGS yl5fF9VYpxOuEVkskTb8fsBHF4KXv4hXkzuym+9SnBPd7r4P15555kgC4RpdGzvCed6N ZtTzOyXuzTLk/ubMgPXFfywg2udkbec1c3/7sZ2SD2B4s6n7n5GhOSTJ1sPyWRBl7rMJ QoG1jnjOAtj6AoOumOCsBv0ka7PX8MQh9P5vG72OtBfwwzx+6whtUCOMDaMxzHT29XX3 XjVQ== MIME-Version: 1.0 Received: by 10.50.85.201 with SMTP id j9mr4310049igz.16.1338206429945; Mon, 28 May 2012 05:00:29 -0700 (PDT) Received: by 10.231.244.8 with HTTP; Mon, 28 May 2012 05:00:29 -0700 (PDT) In-Reply-To: <4FC366D0.6030206@FreeBSD.org> References: <4FC366D0.6030206@FreeBSD.org> Date: Mon, 28 May 2012 20:00:29 +0800 Message-ID: From: Marcelo Araujo To: Martin Matuska Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: [CFT][PREVIEW] ZFS new SPA X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: araujo@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 12:00:36 -0000 Hello Martin, Sounds great, I'm gonna give a try and if I found some issue, I'll report. Congratz. 2012/5/28 Martin Matuska > Hello all, > > I have ported the ZFS features support (SPA version 5000) from Illumos > to FreeBSD-current. > What is still missing is boot support - needs to be implemented. > > Patch against CURRENT: > http://www.vx.sk/download/patches/freebsd/zfs/head-zfs-features.patch > > Amd64 ISO images for testing (bootable, work well in VirtualBox): > Basic: http://www.vx.sk/download/ISO-images/mfsbsd/head-zfs-features.iso > (86MB) > With full installworld: > http://www.vx.sk/download/ISO-images/mfsbsd/head-se-zfs-features.iso(239MB) > > TODO: boot support (check feature availability from ZFS boot code) > > References: > https://hg.openindiana.org/upstream/illumos/illumos-gate/rev/2889e2596bd6 > https://hg.openindiana.org/upstream/illumos/illumos-gate/rev/1949b688d5fb > > -- > Martin Matuska > FreeBSD committer > http://blog.vx.sk > > -- Marcelo Araujo araujo@FreeBSD.org From owner-freebsd-fs@FreeBSD.ORG Mon May 28 13:28:12 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 76A53106566C for ; Mon, 28 May 2012 13:28:12 +0000 (UTC) (envelope-from alessio@interconnessioni.it) Received: from zimbra.interconnessioni.it (zimbra.interconnessioni.it [194.126.148.30]) by mx1.freebsd.org (Postfix) with ESMTP id 295FD8FC16 for ; Mon, 28 May 2012 13:28:11 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by zimbra.interconnessioni.it (Postfix) with ESMTP id B838B64011 for ; Mon, 28 May 2012 15:21:57 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.interconnessioni.it.interconnessioni.it Received: from zimbra.interconnessioni.it ([127.0.0.1]) by localhost (zimbra.interconnessioni.it [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0zAG4uCdZFen for ; Mon, 28 May 2012 15:21:57 +0200 (CEST) Received: from zimbra.interconnessioni.it (localhost.localdomain [127.0.0.1]) by zimbra.interconnessioni.it (Postfix) with ESMTP id 9035A64007 for ; Mon, 28 May 2012 15:21:57 +0200 (CEST) Date: Mon, 28 May 2012 15:21:57 +0200 (CEST) From: Alessio Focardi To: freebsd-fs@freebsd.org Message-ID: <2134924725.5040.1338211317460.JavaMail.root@zimbra.interconnessioni.it> In-Reply-To: <588211375.4794.1338210497900.JavaMail.root@zimbra.interconnessioni.it> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [83.149.137.176] X-Mailer: Zimbra 7.1.4_GA_2555 (ZimbraWebClient - FF3.0 (Win)/7.1.4_GA_2555) Subject: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 13:28:12 -0000 Hi, I'm pretty new to BSD, but I do have some knowledge in Linux. I'm looking for some advice to efficiently pack millions of small files (200 bytes or less) over a freebsd fs. Those files will be stored in an hierarchical directory structure to limit the number of files for any directory and so (I hope!) speed up file lookups/deletion. I have to say that I'm looking at fbsd for my project because both UFS2 and ZFS have some flavour of "block suballocation" "tail packing" "variable record size", at least documentation says so. My hope is to waste as less space as possible, even sacrificing some speed: can't use a full block for a single file: I will end up wasting 99% of the space! Do someone got some experience in a similar situation, and it's willing to give some advice on which fs I should choose and how to tune it for this particular scenario? Thank you very much, appreciated! ps I know that probably a database will fit better in this situation, but in my case I can't take that route :( Alessio Focardi ------------------ From owner-freebsd-fs@FreeBSD.ORG Mon May 28 16:00:34 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 00B9A106564A for ; Mon, 28 May 2012 16:00:34 +0000 (UTC) (envelope-from nowakpl@platinum.linux.pl) Received: from platinum.linux.pl (platinum.edu.pl [81.161.192.4]) by mx1.freebsd.org (Postfix) with ESMTP id A2FFD8FC1B for ; Mon, 28 May 2012 16:00:33 +0000 (UTC) Received: by platinum.linux.pl (Postfix, from userid 87) id 6DA3C5FD0A; Mon, 28 May 2012 17:54:07 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on platinum.linux.pl X-Spam-Level: X-Spam-Status: No, score=-1.5 required=3.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.3.2 Received: from [172.19.191.4] (unknown [83.151.38.73]) by platinum.linux.pl (Postfix) with ESMTPA id 3F29F5FD06 for ; Mon, 28 May 2012 17:54:05 +0200 (CEST) Message-ID: <4FC39F99.5010906@platinum.linux.pl> Date: Mon, 28 May 2012 17:54:01 +0200 From: Adam Nowacki User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.28) Gecko/20120306 Thunderbird/3.1.20 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <2134924725.5040.1338211317460.JavaMail.root@zimbra.interconnessioni.it> In-Reply-To: <2134924725.5040.1338211317460.JavaMail.root@zimbra.interconnessioni.it> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 16:00:34 -0000 I don't think any regular file system will be able to cope. If you really need this as files start with sysutils/fusefs-sqlfs and then maybe look for postgresql or mysql fusefs modules. On 2012-05-28 15:21, Alessio Focardi wrote: > Hi, > > I'm pretty new to BSD, but I do have some knowledge in Linux. > > I'm looking for some advice to efficiently pack millions of small files (200 bytes or less) over a freebsd fs. > > Those files will be stored in an hierarchical directory structure to limit the number of files for any directory and so (I hope!) speed up file lookups/deletion. > > I have to say that I'm looking at fbsd for my project because both UFS2 and ZFS have some flavour of "block suballocation" "tail packing" "variable record size", at least documentation says so. > > My hope is to waste as less space as possible, even sacrificing some speed: can't use a full block for a single file: I will end up wasting 99% of the space! > > > Do someone got some experience in a similar situation, and it's willing to give some advice on which fs I should choose and how to tune it for this particular scenario? > > > Thank you very much, appreciated! > > > ps > > I know that probably a database will fit better in this situation, but in my case I can't take that route :( > > > > Alessio Focardi > ------------------ > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Mon May 28 16:11:11 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6E7A5106567B for ; Mon, 28 May 2012 16:11:11 +0000 (UTC) (envelope-from feld@feld.me) Received: from feld.me (unknown [IPv6:2607:f4e0:100:300::2]) by mx1.freebsd.org (Postfix) with ESMTP id 434C38FC0C for ; Mon, 28 May 2012 16:11:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=feld.me; s=blargle; h=In-Reply-To:Message-Id:From:Mime-Version:Date:References:Subject:To:Content-Type; bh=7UNzVFrF4Q//B8C19AM997cFFGEj/4Jr5i7uDxJCUkc=; b=Rh335cv23RFmAjK13tVHjWWluDmwTVNdR+hVGzayV7OW697q6bnt296AUeIou1vUkXSMtmO5UoMIAL3XSm+awsJ39jivT8NDJv6JplR1zJuT86ydhYkbxU60Zx2Vqic+; Received: from localhost ([127.0.0.1] helo=mwi1.coffeenet.org) by feld.me with esmtp (Exim 4.77 (FreeBSD)) (envelope-from ) id 1SZ2XG-0005KK-AV for freebsd-fs@freebsd.org; Mon, 28 May 2012 11:11:11 -0500 Received: from feld@feld.me by mwi1.coffeenet.org (Archiveopteryx 3.1.4) with esmtpa id 1338221469-3288-3287/5/36; Mon, 28 May 2012 16:11:09 +0000 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: freebsd-fs@freebsd.org References: <2134924725.5040.1338211317460.JavaMail.root@zimbra.interconnessioni.it> Date: Mon, 28 May 2012 11:11:09 -0500 Mime-Version: 1.0 From: Mark Felder Message-Id: In-Reply-To: <2134924725.5040.1338211317460.JavaMail.root@zimbra.interconnessioni.it> User-Agent: Opera Mail/11.64 (FreeBSD) X-SA-Score: -1.5 Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 16:11:11 -0000 ZFS is heavy, but if you have the resources it could possibly fit your needs if tuned correctly. You can change the blocksize for any ZFS filesystem which might help. It also deals with filesystems that have lots of files quite well -- we have some customer backups that sprawl to 20 million+ files and ZFS doesn't seem to care. As far as UFS -- I'm not sure which newfs options you will want to use, but you definitely will want to tune vfs.ufs.dirhash_maxmem hth From owner-freebsd-fs@FreeBSD.ORG Mon May 28 16:25:25 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 952061065672 for ; Mon, 28 May 2012 16:25:25 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id 594B38FC21 for ; Mon, 28 May 2012 16:25:25 +0000 (UTC) Received: from pps.filterd (ltcfislmsgpa03 [127.0.0.1]) by ltcfislmsgpa03.fnfis.com (8.14.4/8.14.4) with SMTP id q4SGPImr004827; Mon, 28 May 2012 11:25:18 -0500 Received: from smtp.fisglobal.com ([10.132.206.31]) by ltcfislmsgpa03.fnfis.com with ESMTP id 154b3vrcrp-1 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Mon, 28 May 2012 11:25:18 -0500 Received: from [10.0.0.105] (10.14.152.61) by smtp.fisglobal.com (10.132.206.31) with Microsoft SMTP Server (TLS) id 14.2.283.3; Mon, 28 May 2012 11:24:49 -0500 MIME-Version: 1.0 (Apple Message framework v1257) From: Devin Teske In-Reply-To: <2134924725.5040.1338211317460.JavaMail.root@zimbra.interconnessioni.it> Date: Mon, 28 May 2012 09:24:51 -0700 Content-Transfer-Encoding: quoted-printable Message-ID: <922B261C-4AB8-49A9-96CE-16C98B265604@fisglobal.com> References: <2134924725.5040.1338211317460.JavaMail.root@zimbra.interconnessioni.it> To: Alessio Focardi X-Mailer: Apple Mail (2.1257) X-Originating-IP: [10.14.152.61] X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.6.7580, 1.0.260, 0.0.0000 definitions=2012-05-28_02:2012-05-21, 2012-05-28, 1970-01-01 signatures=0 Content-Type: text/plain; charset="windows-1252" Cc: freebsd-fs@freebsd.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Devin Teske List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 16:25:25 -0000 On May 28, 2012, at 6:21 AM, Alessio Focardi wrote: > Hi, >=20 > I'm pretty new to BSD, but I do have some knowledge in Linux.=20 >=20 > I'm looking for some advice to efficiently pack millions of small files (= 200 bytes or less) over a freebsd fs. >=20 This is something we've been doing (on FreeBSD) for almost 15 years now (st= arting with FreeBSD 2.1.5; now 8.1, and soon 8.3). We started with UFS1 and= have been evaluating ZFS (we don't think SU+J is ready for production at t= his scale yet). We haven't used UFS2 yet but have no doubt that it's just a= s strong as UFS1. > Those files will be stored in an hierarchical directory structure to limi= t the number of files for any directory and so (I hope!) speed up file look= ups/deletion. >=20 FreeBSD handles this wonderfully thanks to all the people that have put in = time and effort over the years. Ten years ago (circa FreeBSD 4.0-RELEASE) people at the company I work at n= ow, back then commonly: - fiddled with the dirhash sysctl(8) MIB - modified fsck(8) to make it more efficient - modified tar(1) to handle high numbers of hard-links without falling over - modified du(1) in a similar fashion to tar above - more; all in the name of doing what you're describing (but on steroids) but all those patches eventually made their way back into FreeBSD and we ge= nerally haven't had to worry about even tens-of-millions of JPEG-sized (~20= 0KB) files on a RAID formatted in UFS (1 or 2) since, say, FreeBSD-6 (but s= omeone in FS will be able to give a more accurate release when things reall= y started to stabilize). Either way, 6, 7, 8, and 9 all had very stable fil= esystems w/respect to millions-of-small-files. > I have to say that I'm looking at fbsd for my project because both UFS2 a= nd ZFS have some flavour of "block suballocation" "tail packing" "variable = record size", at least documentation says so. >=20 > My hope is to waste as less space as possible, even sacrificing some spee= d: can't use a full block for a single file: I will end up wasting 99% of t= he space! >=20 I wasn't aware that FreeBSD was unique in this respect, but yes, FreeBSD ha= s a block size and a fragment size. While formatting a UFS filesystem you c= an specify these sizes with the "-b SIZE" and "-f SIZE" arguments to newfs(= 8), for example: newfs -b 16384 -f 2048 /dev/da0s1a Will format a RAID (/dev/da0s1a) with a 16K block size but a 2K fragment si= ze. Using touch(1) to create an empty file will use only 2K of disk space. = This is the "block suballocation" you speak of. The above parameters are ex= actly what we use formatting our RAIDs when storing millions of JPEG-sized = (~200KB as you describe) files. >=20 > Do someone got some experience in a similar situation, and it's willing t= o give some advice on which fs I should choose and how to tune it for this = particular scenario? >=20 Choose your hardware wisely. After you have chosen your hardware wisely, se= t it up even more wisely. For example, we go threw a multi-day burn-in process on RAIDs that have dou= ble-digit numbers of disks. Be smart about how you allocate the logical versus physical media in a way = that reduces bottlenecks. Go through any/all failure/recovery test procedures before putting data on = the device if you don't already trust the hardware. Trust in the hardware i= s very important. If you don't trust your hardware's battery backed DIMM fo= r write-back cache (for example), then I have one very important recommenda= tion when it comes to UFS: disable the SoftUpdates feature. Disabling SoftUpdates on a UFS filesystem cause a huge performance impact b= ut it will allow you to sleep at night. In 15 years, UFS has never barfed o= n us unless maybe 3 memorable events in which entire groups-of-individuals = can recount with amazing clarity debugging horked filesystems late in the n= ight after SoftUpdates ate the kid's homework (leaving tens- to hundreds-of= -thousands of files in lost+found). We routinely use SoftUpdates on _other_= UFS filesystems (like system partitions including "/var" and "/usr"), but = _never_ on the RAIDs housing those millions-of-little-files. Other's mileage may vary. >=20 > Thank you very much, appreciated! >=20 >=20 No problem. > ps >=20 > I know that probably a database will fit better in this situation, but in= my case I can't take that route :( >=20 Not necessarily. A database has the immediate-and-clear down-side that if o= ne bit in the database changes, a backup tool like bacula has to backup the= entire database again. =85and the database administrator is not necessarily the same person as the= backup administrator (just sayin'). --=20 Devin _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. From owner-freebsd-fs@FreeBSD.ORG Mon May 28 17:01:38 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0295B106564A for ; Mon, 28 May 2012 17:01:38 +0000 (UTC) (envelope-from alessio@interconnessioni.it) Received: from zimbra.interconnessioni.it (zimbra.interconnessioni.it [194.126.148.30]) by mx1.freebsd.org (Postfix) with ESMTP id AE6988FC0C for ; Mon, 28 May 2012 17:01:37 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by zimbra.interconnessioni.it (Postfix) with ESMTP id 503A464012 for ; Mon, 28 May 2012 19:01:08 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.interconnessioni.it.interconnessioni.it Received: from zimbra.interconnessioni.it ([127.0.0.1]) by localhost (zimbra.interconnessioni.it [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wpXDy+DIJeVf for ; Mon, 28 May 2012 19:01:08 +0200 (CEST) Received: from zimbra.interconnessioni.it (localhost.localdomain [127.0.0.1]) by zimbra.interconnessioni.it (Postfix) with ESMTP id 25DC564006 for ; Mon, 28 May 2012 19:01:08 +0200 (CEST) Date: Mon, 28 May 2012 19:01:08 +0200 (CEST) From: Alessio Focardi To: freebsd-fs@freebsd.org Message-ID: <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it> In-Reply-To: <922B261C-4AB8-49A9-96CE-16C98B265604@fisglobal.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [83.149.137.176] X-Mailer: Zimbra 7.1.4_GA_2555 (ZimbraWebClient - FF3.0 (Win)/7.1.4_GA_2555) Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 17:01:38 -0000 > FreeBSD handles this wonderfully thanks to all the people that have > put in time and effort over the years. That's a great news and I recognize the effort of the community! > I wasn't aware that FreeBSD was unique in this respect, but yes, > FreeBSD has a block size and a fragment size. While formatting a UFS > filesystem you can specify these sizes with the "-b SIZE" and "-f > SIZE" arguments to newfs(8), for example: > > newfs -b 16384 -f 2048 /dev/da0s1a -b block-size The block size of the file system, in bytes. It must be a power of 2. The default size is 16384 bytes, and the smallest allow- able size is 4096 bytes. -f frag-size The fragment size of the file system in bytes. It must be a power of two ranging in value between blocksize/8 and blocksize. The default is 2048 bytes. So in my case I would have to use -b 4096 -f 512 It's an improvement, but still is not ideal: still a big waste with 200 bytes files! ZFS with compression, maybe? > Choose your hardware wisely. After you have chosen your hardware > wisely, set it up even more wisely. That's a good advice! I'm still working on the theory of the system, trying to find a solution for the "slack" problem, then it will come the time to look at a storage platform and surely we will choose something we can trust! Tnx for you reply, informative and well written! From owner-freebsd-fs@FreeBSD.ORG Mon May 28 17:11:44 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B03CE106568D for ; Mon, 28 May 2012 17:11:44 +0000 (UTC) (envelope-from jhellenthal@dataix.net) Received: from mail-gg0-f182.google.com (mail-gg0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 56C468FC18 for ; Mon, 28 May 2012 17:11:44 +0000 (UTC) Received: by ggnm2 with SMTP id m2so2401882ggn.13 for ; Mon, 28 May 2012 10:11:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dataix.net; s=rsa; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to; bh=YOezDvKR1VCebd9GzVBKOe4NXXDACZTRShQks76eqoc=; b=a3xOBQRKo1SN+KsOsIjTZUCfaUJwaOAfTx0+d/CquL/Z8LIbR9Jfq7QRl54KFc/BT9 drRh1gaw02cUNt49B0x9fxCOz8rNi2IAtLqhrLYYYXIavihbmmTxrPZFACqWJnDAJKet B6H+clRNbqZFNEZKk28/DtXFrv4hF9s0IYPqg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:x-gm-message-state; bh=YOezDvKR1VCebd9GzVBKOe4NXXDACZTRShQks76eqoc=; b=pYTc1h0KbVcZvGI1P3BQ+YOGBgWuKCD1om4Rswdu6CmGvE0vOdfg+JOYElzZ4ygLFV m+CUvYqUU54kxSV4iD81lebL6ehSCPC0lB1qaajhTuyss0hI/S1ZaEY2lv8AY3/YVZGn xcDqZoaCV6EMKPNGLup+z0c9TWbZWC4iZHESVSckznz4Mks2veva0LfZdCKZItcB7pzx P0r3jd27jQv+rYSj8cY2GPFouBMwIdi62LkoWZNlXA0zTUzWoiAZ5zSeHtLsKE/XivUE P9wmxZuSoyBB+T9Xteulmtq9s5cfchm6YsnJ9Q6ORjZTEJ6wGOhn/DvLAOtNKLauZNaq K/dw== Received: by 10.50.42.136 with SMTP id o8mr5031671igl.16.1338225097893; Mon, 28 May 2012 10:11:37 -0700 (PDT) Received: from DataIX.net (24-247-238-117.dhcp.aldl.mi.charter.com. [24.247.238.117]) by mx.google.com with ESMTPS id k4sm11575269igq.16.2012.05.28.10.11.36 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 28 May 2012 10:11:37 -0700 (PDT) Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.5/8.14.5) with ESMTP id q4SHBYCN025351 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 28 May 2012 13:11:34 -0400 (EDT) (envelope-from jhellenthal@DataIX.net) Received: (from jh@localhost) by DataIX.net (8.14.5/8.14.5/Submit) id q4SHBYCa025350; Mon, 28 May 2012 13:11:34 -0400 (EDT) (envelope-from jhellenthal@DataIX.net) Date: Mon, 28 May 2012 13:11:34 -0400 From: Jason Hellenthal To: Alessio Focardi Message-ID: <20120528171134.GA10333@DataIX.net> References: <922B261C-4AB8-49A9-96CE-16C98B265604@fisglobal.com> <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it> X-Gm-Message-State: ALoCoQlR2FsX6ju7AKfVvKlliVzhKH24J91prgCn5TRoovPjTwCsHLBMDwleog40NE9ko/xgK/Bz Cc: freebsd-fs@freebsd.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 17:11:44 -0000 On Mon, May 28, 2012 at 07:01:08PM +0200, Alessio Focardi wrote: > > FreeBSD handles this wonderfully thanks to all the people that have > > put in time and effort over the years. > > That's a great news and I recognize the effort of the community! > > > I wasn't aware that FreeBSD was unique in this respect, but yes, > > FreeBSD has a block size and a fragment size. While formatting a UFS > > filesystem you can specify these sizes with the "-b SIZE" and "-f > > SIZE" arguments to newfs(8), for example: > > > > newfs -b 16384 -f 2048 /dev/da0s1a > > -b block-size > The block size of the file system, in bytes. It must be a power > of 2. The default size is 16384 bytes, and the smallest allow- > able size is 4096 bytes. > > -f frag-size > The fragment size of the file system in bytes. It must be a > power of two ranging in value between blocksize/8 and blocksize. > The default is 2048 bytes. > > So in my case I would have to use -b 4096 -f 512 > > It's an improvement, but still is not ideal: still a big waste with 200 bytes files! > > ZFS with compression, maybe? > ZFSv28 with dedup turned on... Be careful of new instances of SPA working its way into the source tree though. An upgrade could possibly make you incompatible with Solaris ZFS systems if for some reason you ever wanted to change from FreeBSD. -- - (2^(N-1)) From owner-freebsd-fs@FreeBSD.ORG Mon May 28 17:14:51 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 496861065740 for ; Mon, 28 May 2012 17:14:51 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id 092578FC12 for ; Mon, 28 May 2012 17:14:50 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id q4SHD057025491; Mon, 28 May 2012 12:13:00 -0500 (CDT) Date: Mon, 28 May 2012 12:13:00 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Mark Felder In-Reply-To: Message-ID: References: <2134924725.5040.1338211317460.JavaMail.root@zimbra.interconnessioni.it> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Mon, 28 May 2012 12:13:00 -0500 (CDT) Cc: freebsd-fs@freebsd.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 17:14:51 -0000 On Mon, 28 May 2012, Mark Felder wrote: > ZFS is heavy, but if you have the resources it could possibly fit your needs > if tuned correctly. You can change the blocksize for any ZFS filesystem which > might help. It also deals with filesystems that have lots of files quite well > -- we have some customer backups that sprawl to 20 million+ files and ZFS > doesn't seem to care. ZFS will work but its metadata size requirements will likely be twice the amount required by the actual file data. It is not necessary to change the ZFS blocksize in order for it to work. I have a directory in ZFS with 1 million files in one directory (not broken out into some heirarchy) that I use for testing application software. ZFS does not mind a million files in one directory but applications can take quite a long time to obtain a file listing (fault of the application, not ZFS), especially if they want to sort that listing. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@FreeBSD.ORG Mon May 28 17:17:57 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1CC13106566B for ; Mon, 28 May 2012 17:17:57 +0000 (UTC) (envelope-from alessio@interconnessioni.it) Received: from zimbra.interconnessioni.it (zimbra.interconnessioni.it [194.126.148.30]) by mx1.freebsd.org (Postfix) with ESMTP id C83728FC1A for ; Mon, 28 May 2012 17:17:56 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by zimbra.interconnessioni.it (Postfix) with ESMTP id A81A564012 for ; Mon, 28 May 2012 19:17:28 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.interconnessioni.it.interconnessioni.it Received: from zimbra.interconnessioni.it ([127.0.0.1]) by localhost (zimbra.interconnessioni.it [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5Vr6+hn8ll3v for ; Mon, 28 May 2012 19:17:28 +0200 (CEST) Received: from zimbra.interconnessioni.it (localhost.localdomain [127.0.0.1]) by zimbra.interconnessioni.it (Postfix) with ESMTP id 133E364006 for ; Mon, 28 May 2012 19:17:28 +0200 (CEST) Date: Mon, 28 May 2012 19:17:28 +0200 (CEST) From: Alessio Focardi To: freebsd-fs@freebsd.org Message-ID: <1539956470.7148.1338225448013.JavaMail.root@zimbra.interconnessioni.it> In-Reply-To: <20120528171134.GA10333@DataIX.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [83.149.137.176] X-Mailer: Zimbra 7.1.4_GA_2555 (ZimbraWebClient - FF3.0 (Win)/7.1.4_GA_2555) Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 17:17:57 -0000 > > It's an improvement, but still is not ideal: still a big waste with > > 200 bytes files! > > > > ZFS with compression, maybe? > > > > ZFSv28 with dedup turned on... Data is very little "deduplicabile" in my case: is this a trick to enable packing of > than 1 file in a single block/fragment? I tough compression would turn this on ... From owner-freebsd-fs@FreeBSD.ORG Mon May 28 17:25:45 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AAA121065677 for ; Mon, 28 May 2012 17:25:45 +0000 (UTC) (envelope-from alessio@interconnessioni.it) Received: from zimbra.interconnessioni.it (zimbra.interconnessioni.it [194.126.148.30]) by mx1.freebsd.org (Postfix) with ESMTP id 532FB8FC08 for ; Mon, 28 May 2012 17:25:45 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by zimbra.interconnessioni.it (Postfix) with ESMTP id 2AB9664012 for ; Mon, 28 May 2012 19:25:18 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.interconnessioni.it.interconnessioni.it Received: from zimbra.interconnessioni.it ([127.0.0.1]) by localhost (zimbra.interconnessioni.it [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jHtFUIdAaAcK for ; Mon, 28 May 2012 19:25:18 +0200 (CEST) Received: from zimbra.interconnessioni.it (localhost.localdomain [127.0.0.1]) by zimbra.interconnessioni.it (Postfix) with ESMTP id 051F964006 for ; Mon, 28 May 2012 19:25:18 +0200 (CEST) Date: Mon, 28 May 2012 19:25:18 +0200 (CEST) From: Alessio Focardi To: freebsd-fs@freebsd.org Message-ID: <1236761185.7172.1338225917997.JavaMail.root@zimbra.interconnessioni.it> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [83.149.137.176] X-Mailer: Zimbra 7.1.4_GA_2555 (ZimbraWebClient - FF3.0 (Win)/7.1.4_GA_2555) Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 17:25:45 -0000 > ZFS will work but its metadata size requirements will likely be twice > the amount required by the actual file data. It is not necessary to > change the ZFS blocksize in order for it to work. So if I understand correctly you are suggesting that I use as few directories as possible to save space (less metadata for them), but still a single file will weight more in metadata than it's size? That suggests that I cant have no metadatata suballocation, in any case. Correct? So ... what's the size of metadata for a regular file, even empty? A full block? Really tnx! From owner-freebsd-fs@FreeBSD.ORG Mon May 28 22:34:11 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5DD61106564A for ; Mon, 28 May 2012 22:34:11 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id 1D5518FC14 for ; Mon, 28 May 2012 22:34:11 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id q4SMY96V026495; Mon, 28 May 2012 17:34:09 -0500 (CDT) Date: Mon, 28 May 2012 17:34:09 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Alessio Focardi In-Reply-To: <1236761185.7172.1338225917997.JavaMail.root@zimbra.interconnessioni.it> Message-ID: References: <1236761185.7172.1338225917997.JavaMail.root@zimbra.interconnessioni.it> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Mon, 28 May 2012 17:34:10 -0500 (CDT) Cc: freebsd-fs@freebsd.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 22:34:11 -0000 On Mon, 28 May 2012, Alessio Focardi wrote: >> ZFS will work but its metadata size requirements will likely be twice >> the amount required by the actual file data. It is not necessary to >> change the ZFS blocksize in order for it to work. > > So if I understand correctly you are suggesting that I use as few > directories as possible to save space (less metadata for them), but > still a single file will weight more in metadata than it's size? No, I am not suggesting that at all. It is just my own test case to see if my application can open a huge directory. Each ZFS data block has associated redundant metadata data blocks and they can not consume less space than the logical sector size. It is wise to subdivide the millions of files into a heirarchy so that ordinary application software can still work well with it (without being slow or crashing). > That suggests that I cant have no metadatata suballocation, in any > case. Correct? So ... what's the size of metadata for a regular > file, even empty? A full block? There is metadata for each data block and zfs always stores multiple copies of the metadata. With 128K blocks and larger files, this overhead is no big deal but if your file is 200 bytes then at least three sectors are consumed just for the data itself and there is surely other metadata at the file level. The storage space efficiency would not be very high. The good news is that ZFS makes it very easy to test various scenarios to see how well solutions really work. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@FreeBSD.ORG Tue May 29 01:55:58 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 55E17106566C for ; Tue, 29 May 2012 01:55:58 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (ns1.bitblocks.com [173.228.5.8]) by mx1.freebsd.org (Postfix) with ESMTP id 36A948FC0C for ; Tue, 29 May 2012 01:55:58 +0000 (UTC) Received: from bitblocks.com (localhost [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id B4A8CB82E; Mon, 28 May 2012 18:55:57 -0700 (PDT) To: kpneal@pobox.com In-reply-to: Your message of "Mon, 28 May 2012 21:35:22 EDT." <20120529013522.GB86408@neutralgood.org> References: <20120528070947.A917AB827@mail.bitblocks.com> <20120529013522.GB86408@neutralgood.org> Comments: In-reply-to kpneal@pobox.com message dated "Mon, 28 May 2012 21:35:22 -0400." Date: Mon, 28 May 2012 18:55:57 -0700 From: Bakul Shah Message-Id: <20120529015557.B4A8CB82E@mail.bitblocks.com> Cc: freebsd-fs@freebsd.org Subject: Re: zfs disk replace issue X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 01:55:58 -0000 On Mon, 28 May 2012 21:35:22 EDT kpneal@pobox.com wrote: > On Mon, May 28, 2012 at 12:09:47AM -0700, Bakul Shah wrote: > > I have a zpool with 2 mirrors of two disks each. 3 of the > > disks are 1TB. I replaced three original 300GB disks with > > the TB disks and there were no problems > > > > Recently I upgraded to a new machine and trasferred the old > > zfs disks to the new machine and everything was ok. > > > > I then replaced the final 300GB disk with a 1TB disk. I > > noticed that after resilver finished (in two hours), "zpool > > status" kept showing 'replacing 0' and showed the old and new > > disk in the pool. I thought it would automatically take out > > the old disk? So then I manually "zpool detach"ed the old > > disk but the size of the mirror has not changed. > > > > Is this a bug or did I miss some step? I'd appreciate any help > > to make the extra space usable! This pool is root so it > > was mounted when I did this. May be that was the problem? > > I don't know if the 'autoexpand' property works on FreeBSD or not. If not > then you may have to export the pool and then reimport it. I don't know > how to do that when you are booting from the pool. Thanks. I did discover this property earlier today. At least for the older pool the default seems to be off. I turned it on but haven't had a chance to work on this... I just did zpool detach z ada4p1 zpool attach z ada1p1 ada4p1 That fixed it but now it is resilvering! Luckily it only takes about a couple hours! > Oh, and I don't know if this is needed or not, but the practice seems > to be to always do a scrub after a resilver has completed to a new, > larger, disk. Yeah, I already did that. Thanks for your response. From owner-freebsd-fs@FreeBSD.ORG Tue May 29 05:00:41 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [69.147.83.53]) by hub.freebsd.org (Postfix) with ESMTP id 08C7B106567D for ; Tue, 29 May 2012 05:00:41 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from [127.0.0.1] (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 9B0E3151FE3; Tue, 29 May 2012 05:00:40 +0000 (UTC) Message-ID: <4FC457F7.9000800@FreeBSD.org> Date: Mon, 28 May 2012 22:00:39 -0700 From: Doug Barton Organization: http://www.FreeBSD.org/ User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: Alessio Focardi References: <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it> In-Reply-To: <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it> X-Enigmail-Version: 1.4.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 05:00:41 -0000 On 5/28/2012 10:01 AM, Alessio Focardi wrote: > So in my case I would have to use -b 4096 -f 512 > > It's an improvement, but still is not ideal: still a big waste with 200 bytes files! Are all of the files exactly 200 bytes? If so that's likely the best you can do. The good news is that it's a big improvement (I've done similar stuff in the past). You'll also want to tweak the -i (inode) value to insure that you have sufficient inodes for the number of files you plan to store. The default is not likely to be adequate for your needs. Don't use journaling of any kind. If your primary access is going to be reads, you should be Ok with softupdates. If you're going to be doing a lot of writing consider no softupdates and async. Of course, all of this assumes that you have a reasonable disk array setup (RAID or similar, power backup, etc.) From the rest of your thread it seems that you have this under control. I think you already referenced this, but you also want to shard the file system so that you don't have any one directory with more than 10k files. Even with dirhash that's still a limitation. You don't have to worry too much if you go a few files over 10k, but you don't want to stretch it much more than that. Make sure you plan for growth over time. hth, Doug -- This .signature sanitized for your protection From owner-freebsd-fs@FreeBSD.ORG Tue May 29 06:38:18 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 49DB7106566C for ; Tue, 29 May 2012 06:38:18 +0000 (UTC) (envelope-from petri@helenius.fi) Received: from mail.helenius.fi (unknown [IPv6:2001:67c:164:40::91]) by mx1.freebsd.org (Postfix) with ESMTP id CC4F28FC0A for ; Tue, 29 May 2012 06:38:17 +0000 (UTC) Received: from mail.helenius.fi (localhost [127.0.0.1]) by mail.helenius.fi (Postfix) with ESMTP id 70C31414A for ; Tue, 29 May 2012 06:38:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at helenius.fi Received: from mail.helenius.fi ([127.0.0.1]) by mail.helenius.fi (mail.helenius.fi [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id D6ZVtoNxqYLD for ; Tue, 29 May 2012 06:38:10 +0000 (UTC) Received: from dynamic.helenius.fi (unknown [IPv6:2001:1bc8:1018:0:549b:d96a:5d5a:de26]) (Authenticated sender: pete) by mail.helenius.fi (Postfix) with ESMTPA for ; Tue, 29 May 2012 06:38:10 +0000 (UTC) From: Petri Helenius Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Tue, 29 May 2012 09:38:09 +0300 Message-Id: To: freebsd-fs@freebsd.org Mime-Version: 1.0 (Apple Message framework v1278) X-Mailer: Apple Mail (2.1278) Subject: ZVOL snapshots and dev files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 06:38:18 -0000 Hi, Figured out that if ZVOL snapshots are removed using -r, the entries in = /dev/zvol never get removed. (until reboot) Should I file a bug? Pete From owner-freebsd-fs@FreeBSD.ORG Tue May 29 06:38:33 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 967C7106566B; Tue, 29 May 2012 06:38:33 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-gh0-f182.google.com (mail-gh0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 3C0948FC16; Tue, 29 May 2012 06:38:33 +0000 (UTC) Received: by ghbz22 with SMTP id z22so1876603ghb.13 for ; Mon, 28 May 2012 23:38:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=zQAfRJj2cvL6UzEWnrGr1TtiR6maWum4HGNu+DI8YU0=; b=Rza7QPGzGNLGTtzgMFk2ckyDgybyTmLQYF7adh0YeozcW6s5zujsr4KkYryv/4R8qI spgRV+kLeiR46gITIdltxDCIh0dAjH01E3nQzEUlUemgCxip85WJ3uTFu5MgqJE5TtvR UfZa6FF6ZQePb9EaRvoTbcrmoWJKhjQlKedaEMpKxUvA2dcKgbY9JEP/iXaFsXASEEfz bINxy0ZmO6tEq+yXN1672bZcU/z/PqontN5CvFtnX6ybI4MIr2fEp/IW8m+ozxVnq3IN xkxXL3QAUMRrw+7rwiUTphEE0w2T53bhxqEzgrxJkVx25mrOXsbaP5Xk4xEwoyrO5oce zkQA== MIME-Version: 1.0 Received: by 10.50.203.41 with SMTP id kn9mr2340322igc.72.1338273512358; Mon, 28 May 2012 23:38:32 -0700 (PDT) Received: by 10.231.244.8 with HTTP; Mon, 28 May 2012 23:38:32 -0700 (PDT) In-Reply-To: <4FC366D0.6030206@FreeBSD.org> References: <4FC366D0.6030206@FreeBSD.org> Date: Tue, 29 May 2012 14:38:32 +0800 Message-ID: From: Marcelo Araujo To: Martin Matuska Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: [CFT][PREVIEW] ZFS new SPA X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: araujo@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 06:38:33 -0000 Dear Martin and All, The patch works like a charm.... root@controllerB:/home/araujo # zpool create -f tank da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11 da12 da13 root@controllerB:/home/araujo # zpool set feature@async_destroy=enabled tank root@controllerB:/home/araujo # zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT tank 18.1T 148K 18.1T 0% 1.00x ONLINE - When do you believe we gonna have it on HEAD? Best Regards, - Araujo 2012/5/28 Martin Matuska > Hello all, > > I have ported the ZFS features support (SPA version 5000) from Illumos > to FreeBSD-current. > What is still missing is boot support - needs to be implemented. > > Patch against CURRENT: > http://www.vx.sk/download/patches/freebsd/zfs/head-zfs-features.patch > > Amd64 ISO images for testing (bootable, work well in VirtualBox): > Basic: http://www.vx.sk/download/ISO-images/mfsbsd/head-zfs-features.iso > (86MB) > With full installworld: > http://www.vx.sk/download/ISO-images/mfsbsd/head-se-zfs-features.iso(239MB) > > TODO: boot support (check feature availability from ZFS boot code) > > References: > https://hg.openindiana.org/upstream/illumos/illumos-gate/rev/2889e2596bd6 > https://hg.openindiana.org/upstream/illumos/illumos-gate/rev/1949b688d5fb > > -- > Martin Matuska > FreeBSD committer > http://blog.vx.sk > > -- Marcelo Araujo araujo@FreeBSD.org From owner-freebsd-fs@FreeBSD.ORG Tue May 29 07:04:36 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D36B8106564A for ; Tue, 29 May 2012 07:04:36 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vps.rulingia.com (host-122-100-2-194.octopus.com.au [122.100.2.194]) by mx1.freebsd.org (Postfix) with ESMTP id 44DD98FC16 for ; Tue, 29 May 2012 07:04:35 +0000 (UTC) Received: from server.rulingia.com (c220-239-254-65.belrs5.nsw.optusnet.com.au [220.239.254.65]) by vps.rulingia.com (8.14.5/8.14.5) with ESMTP id q4T74Yke050015 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 29 May 2012 17:04:35 +1000 (EST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from aspire.rulingia.com (aspire.rulingia.com [192.168.123.161]) by server.rulingia.com (8.14.5/8.14.5) with ESMTP id q4T74Rtn046867 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 29 May 2012 17:04:27 +1000 (EST) (envelope-from peter@rulingia.com) Received: from aspire.rulingia.com (localhost [127.0.0.1]) by aspire.rulingia.com (8.14.5/8.14.5) with ESMTP id q4T6c5Qx038725 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 29 May 2012 16:38:10 +1000 (EST) (envelope-from peter@aspire.rulingia.com) Received: (from peter@localhost) by aspire.rulingia.com (8.14.5/8.14.5/Submit) id q4T6c48D038724; Tue, 29 May 2012 16:38:04 +1000 (EST) (envelope-from peter) Date: Tue, 29 May 2012 16:38:03 +1000 From: Peter Jeremy To: Alessio Focardi Message-ID: <20120529063803.GG2675@aspire.rulingia.com> References: <588211375.4794.1338210497900.JavaMail.root@zimbra.interconnessioni.it> <2134924725.5040.1338211317460.JavaMail.root@zimbra.interconnessioni.it> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="iAL9S67WQOXgEPD9" Content-Disposition: inline In-Reply-To: <2134924725.5040.1338211317460.JavaMail.root@zimbra.interconnessioni.it> X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 07:04:36 -0000 --iAL9S67WQOXgEPD9 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2012-May-28 15:21:57 +0200, Alessio Focardi wrote: >I'm looking for some advice to efficiently pack millions of small files (2= 00 bytes or less) over a freebsd fs. "Millions of files" isn't an issue - any self-respecting filesystem will manage that. The "200 bytes or less" is a real problem. The best you will be able to do is a 4K/512 UFS1 filesystem (UFS1 because the inodes are half the size) - that gives you a minimum of 640 bytes (plus directory entry) per file (it would be 768 bytes for UFS2 and 1536 or 2048 bytes for ZFS). I would suggest that no normal filesystem will be a good match for your requirements. IMHO, you would be better off storing your data as an array of "file+metadata" objects within a single physical file, together with some sort of index structure that suits your "file" names. --=20 Peter Jeremy --iAL9S67WQOXgEPD9 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAk/EbsoACgkQ/opHv/APuIdrtACgs5rseqWRzW/8nsc+XxE3SCM6 fOEAoK3c77JusLQomJ+YygpKhYCh9XO8 =gO6N -----END PGP SIGNATURE----- --iAL9S67WQOXgEPD9-- From owner-freebsd-fs@FreeBSD.ORG Tue May 29 07:35:51 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DF4ED1065687; Tue, 29 May 2012 07:35:51 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail09.syd.optusnet.com.au (mail09.syd.optusnet.com.au [211.29.132.190]) by mx1.freebsd.org (Postfix) with ESMTP id ED6138FC16; Tue, 29 May 2012 07:35:27 +0000 (UTC) Received: from c122-106-171-232.carlnfd1.nsw.optusnet.com.au (c122-106-171-232.carlnfd1.nsw.optusnet.com.au [122.106.171.232]) by mail09.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id q4T7ZIww012490 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 29 May 2012 17:35:19 +1000 Date: Tue, 29 May 2012 17:35:18 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Doug Barton In-Reply-To: <4FC457F7.9000800@FreeBSD.org> Message-ID: <20120529161802.N975@besplex.bde.org> References: <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it> <4FC457F7.9000800@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@FreeBSD.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 07:35:52 -0000 On Mon, 28 May 2012, Doug Barton wrote: > On 5/28/2012 10:01 AM, Alessio Focardi wrote: >> So in my case I would have to use -b 4096 -f 512 >> >> It's an improvement, but still is not ideal: still a big waste with 200 bytes files! > > Are all of the files exactly 200 bytes? If so that's likely the best you > can do. It is easy to do better by using a file system that supports small block sizes. This might be slow, but it reduces the wastage. Possible file systems: - msdosfs has a minimum block size of 512 and handles caching for this fairly well for a small number of files, but is probably even slower than ffs for a large number of files. Especially when directories are involved. - ext2fs has a minimum block size of 1024 and handles caching for this fairly poorly. - it is easy to fix ffs to support a minimum block size of 512 (by reducing its gratuitous limit of MINBSIZE and fixing the few things that break: % magic 19540119 (UFS2) time Tue May 29 16:16:20 2012 % superblock location 65536 id [ 4fc46886 2007c27b ] % ncg 4 size 1200 blocks 947 % bsize 512 shift 9 mask 0xfffffe00 % fsize 512 shift 9 mask 0xfffffe00 % frag 1 shift 0 fsbtodb 0 % minfree 8% optim time symlinklen 120 % maxbsize 512 maxbpg 64 maxcontig 256 contigsumsize 16 % nbfree 944 ndir 2 nifree 75 nffree 0 % bpg 301 fpg 301 ipg 20 % nindir 64 inopb 2 maxfilesize 136353791 % sbsize 1536 cgsize 512 csaddr 171 cssize 512 % sblkno 144 cblkno 160 iblkno 161 dblkno 171 % cgrotor 0 fmod 0 ronly 0 clean 1 % avgfpdir 64 avgfilesize 16384 % flags none % fsmnt /mnt % volname swuid 0 Note that sbsize is now larger than bsize. Most of the things that break involve wrong checks that sbsize <= bsize. sbsize is not limited by bsize in either direction, since the super block is accessed in DEV_BSIZE-blocks, not bsize-blocks and the upper limit on its size is not the same as upper limit on bsize. > The good news is that it's a big improvement (I've done similar > stuff in the past). You'll also want to tweak the -i (inode) value to > insure that you have sufficient inodes for the number of files you plan > to store. The default is not likely to be adequate for your needs. Big is relative. 4K-blocks with 200-byte files gives a wastage factor of 20. Metadata alone will be 256 bytes for the inode alone with ffs2. Only 128 bytes with ffs1. Only 32 bytes with msdosfs. > ... But I expect using a file system would be so slow for lots of really small files that I wouldn't try it. Caching is already poor for 4K-files, and a factor of 20 loss won't improve it. If you don't want to use a database, maybe you can use tar.[gz] files. These at least reduce the wastage (but still waste about twice as much as msdosfs with 512 byte blocks), unless they are compressed. I think there are ways to treat tar files as file systems and to avoid reading the whole file to find files in it (zip format is better for this). Bruce From owner-freebsd-fs@FreeBSD.ORG Tue May 29 07:59:52 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6F13C106566C for ; Tue, 29 May 2012 07:59:52 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (ns1.bitblocks.com [173.228.5.8]) by mx1.freebsd.org (Postfix) with ESMTP id 501D38FC08 for ; Tue, 29 May 2012 07:59:52 +0000 (UTC) Received: from bitblocks.com (localhost [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id C4ADFB82A; Tue, 29 May 2012 00:59:46 -0700 (PDT) To: Bruce Evans In-reply-to: Your message of "Tue, 29 May 2012 17:35:18 +1000." <20120529161802.N975@besplex.bde.org> References: <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it> <4FC457F7.9000800@FreeBSD.org> <20120529161802.N975@besplex.bde.org> Comments: In-reply-to Bruce Evans message dated "Tue, 29 May 2012 17:35:18 +1000." Date: Tue, 29 May 2012 00:59:46 -0700 From: Bakul Shah Message-Id: <20120529075946.C4ADFB82A@mail.bitblocks.com> Cc: freebsd-fs@FreeBSD.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 07:59:52 -0000 On Tue, 29 May 2012 17:35:18 +1000 Bruce Evans wrote: > > But I expect using a file system would be so slow for lots of really > small files that I wouldn't try it. Caching is already poor for > 4K-files, and a factor of 20 loss won't improve it. If you don't want > to use a database, maybe you can use tar.[gz] files. These at least > reduce the wastage (but still waste about twice as much as msdosfs with > 512 byte blocks), unless they are compressed. I think there are ways > to treat tar files as file systems and to avoid reading the whole file > to find files in it (zip format is better for this). As someone else pointed out, the right thing for Alessio may be to just use fusefs-sqlfs or may be even roll his own! Metadata can be generated on the fly. If performance is an issue he can slurp in the whole file and use write-through for any updates. A million 200 bytes files would take less than 512MB. Another alternative: 9pfuse (from plan9ports). There is even an sqfs written in 339 lines of python on github that'd bolt right on 9pfuse! He can use it as a template to build exactly what he wants. There is also tarfs etc. in plan9ports but it provides readonly support. From owner-freebsd-fs@FreeBSD.ORG Tue May 29 08:00:25 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BBCE5106566C; Tue, 29 May 2012 08:00:24 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail07.syd.optusnet.com.au (mail07.syd.optusnet.com.au [211.29.132.188]) by mx1.freebsd.org (Postfix) with ESMTP id F3DF28FC0C; Tue, 29 May 2012 08:00:23 +0000 (UTC) Received: from c122-106-171-232.carlnfd1.nsw.optusnet.com.au (c122-106-171-232.carlnfd1.nsw.optusnet.com.au [122.106.171.232]) by mail07.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id q4T80LLc017581 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 29 May 2012 18:00:21 +1000 Date: Tue, 29 May 2012 18:00:20 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Bruce Evans In-Reply-To: <20120529161802.N975@besplex.bde.org> Message-ID: <20120529175504.K1291@besplex.bde.org> References: <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it> <4FC457F7.9000800@FreeBSD.org> <20120529161802.N975@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@FreeBSD.org, Doug Barton Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 08:00:26 -0000 On Tue, 29 May 2012, Bruce Evans wrote: > ... > - it is easy to fix ffs to support a minimum block size of 512 (by > reducing its gratuitous limit of MINBSIZE and fixing the few things > that break: > On Mon, 28 May 2012, Doug Barton wrote: >> The good news is that it's a big improvement (I've done similar >> stuff in the past). You'll also want to tweak the -i (inode) value to >> insure that you have sufficient inodes for the number of files you plan >> to store. The default is not likely to be adequate for your needs. > > Big is relative. 4K-blocks with 200-byte files gives a wastage factor > of 20. Metadata alone will be 256 bytes for the inode alone with ffs2. > Only 128 bytes with ffs1. Only 32 bytes with msdosfs. Oops, only a wastage factor of 2.5 with the 512-byte fragments that are normally used with 4K-blocks by ffs. 512-byte blocks with ffs only give a small reduction in metadata size and better block allocation. Bruce From owner-freebsd-fs@FreeBSD.ORG Tue May 29 08:06:35 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 79CA6106566C; Tue, 29 May 2012 08:06:35 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (gw.catspoiler.org [75.1.14.242]) by mx1.freebsd.org (Postfix) with ESMTP id 56B138FC18; Tue, 29 May 2012 08:06:35 +0000 (UTC) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.13.3/8.13.3) with ESMTP id q4T86K8M007099; Tue, 29 May 2012 01:06:24 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <201205290806.q4T86K8M007099@gw.catspoiler.org> Date: Tue, 29 May 2012 01:06:20 -0700 (PDT) From: Don Lewis To: brde@optusnet.com.au In-Reply-To: <20120529161802.N975@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Cc: freebsd-fs@FreeBSD.org, dougb@FreeBSD.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 08:06:35 -0000 On 29 May, Bruce Evans wrote: > On Mon, 28 May 2012, Doug Barton wrote: > >> On 5/28/2012 10:01 AM, Alessio Focardi wrote: >>> So in my case I would have to use -b 4096 -f 512 >>> >>> It's an improvement, but still is not ideal: still a big waste with 200 bytes files! >> >> Are all of the files exactly 200 bytes? If so that's likely the best you >> can do. > > It is easy to do better by using a file system that supports small block > sizes. This might be slow, but it reduces the wastage. Possible file > systems: > - it is easy to fix ffs to support a minimum block size of 512 (by > reducing its gratuitous limit of MINBSIZE and fixing the few things > that break: That shouldn't be necessary, especially if you newfs with the "-o space" option to force the fragments for multiple files to be allocated out of the same block right from the start unstead of waiting to do this once the filesystem starts getting full. I ran a Usenet server this way for quite a while with fairly good results, though the average file size was a bit bigger, about 2K or so. I found that if I didn't use "-o space" that space optimization wouldn't kick in soon enough and I'd tend to run out of full blocks that would be needed for larger files. The biggest performance problem that I ran into was that as the directories shrank and grew, they would tend to get badly fragmented, causing lookups to get slow. This was in the days before dirhash ... From owner-freebsd-fs@FreeBSD.ORG Tue May 29 08:11:32 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 11D981065673 for ; Tue, 29 May 2012 08:11:32 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from [127.0.0.1] (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 0BB9E14E75E; Tue, 29 May 2012 08:11:31 +0000 (UTC) Message-ID: <4FC484B1.2080202@FreeBSD.org> Date: Tue, 29 May 2012 01:11:29 -0700 From: Doug Barton Organization: http://www.FreeBSD.org/ User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: Bruce Evans References: <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it> <4FC457F7.9000800@FreeBSD.org> <20120529161802.N975@besplex.bde.org> In-Reply-To: <20120529161802.N975@besplex.bde.org> X-Enigmail-Version: 1.4.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 08:11:32 -0000 On 5/29/2012 12:35 AM, Bruce Evans wrote: > On Mon, 28 May 2012, Doug Barton wrote: > >> On 5/28/2012 10:01 AM, Alessio Focardi wrote: >>> So in my case I would have to use -b 4096 -f 512 >>> >>> It's an improvement, but still is not ideal: still a big waste with >>> 200 bytes files! >> >> Are all of the files exactly 200 bytes? If so that's likely the best you >> can do. > > It is easy to do better by using a file system that supports small block > sizes. This might be slow, but it reduces the wastage. Possible file > systems: > - msdosfs has a minimum block size of 512 and handles caching for this > fairly well for a small number of files, but is probably even slower > than ffs for a large number of files. Especially when directories > are involved. > - ext2fs has a minimum block size of 1024 and handles caching for this > fairly poorly. I wouldn't choose either of those for a mission-critical system. I use both of them daily, and while they work, the performance and reliability are not something I'd bet a business on. >> The good news is that it's a big improvement (I've done similar >> stuff in the past). You'll also want to tweak the -i (inode) value to >> insure that you have sufficient inodes for the number of files you plan >> to store. The default is not likely to be adequate for your needs. > > Big is relative. 4K-blocks with 200-byte files gives a wastage factor > of 20. Metadata alone will be 256 bytes for the inode alone with ffs2. > Only 128 bytes with ffs1. Only 32 bytes with msdosfs. I'm talking about "big" in the sense of how much better it performed. Changing the file system defaults as I described made a 4-fold decrease in the load time for a busy BIND server that had to load hundreds of thousands of tiny zone files. I was surprised actually at how much better it worked, given that (as you correctly describe) the math is certainly not 4 times better. > But I expect using a file system would be so slow for lots of really > small files that I wouldn't try it. Caching is already poor for > 4K-files, and a factor of 20 loss won't improve it. If you don't want > to use a database, maybe you can use tar.[gz] files. These at least > reduce the wastage (but still waste about twice as much as msdosfs with > 512 byte blocks), unless they are compressed. I think there are ways > to treat tar files as file systems and to avoid reading the whole file > to find files in it (zip format is better for this). There are some uses cases where you have to have a flat file on disk. If I were the OP I'd be looking at any excuse I could find to get this stuff into a db too, but I think he said that was out of scope. Doug -- This .signature sanitized for your protection From owner-freebsd-fs@FreeBSD.ORG Tue May 29 08:20:14 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx2.freebsd.org (mx2.freebsd.org [69.147.83.53]) by hub.freebsd.org (Postfix) with ESMTP id 82D35106564A; Tue, 29 May 2012 08:20:14 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from [127.0.0.1] (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 2C5DB14E75E; Tue, 29 May 2012 08:20:14 +0000 (UTC) Message-ID: <4FC486BC.3050808@FreeBSD.org> Date: Tue, 29 May 2012 01:20:12 -0700 From: Doug Barton Organization: http://www.FreeBSD.org/ User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: Don Lewis References: <201205290806.q4T86K8M007099@gw.catspoiler.org> In-Reply-To: <201205290806.q4T86K8M007099@gw.catspoiler.org> X-Enigmail-Version: 1.4.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 08:20:14 -0000 On 5/29/2012 1:06 AM, Don Lewis wrote: > On 29 May, Bruce Evans wrote: >> On Mon, 28 May 2012, Doug Barton wrote: >> >>> On 5/28/2012 10:01 AM, Alessio Focardi wrote: >>>> So in my case I would have to use -b 4096 -f 512 >>>> >>>> It's an improvement, but still is not ideal: still a big waste with 200 bytes files! >>> >>> Are all of the files exactly 200 bytes? If so that's likely the best you >>> can do. >> >> It is easy to do better by using a file system that supports small block >> sizes. This might be slow, but it reduces the wastage. Possible file >> systems: > >> - it is easy to fix ffs to support a minimum block size of 512 (by >> reducing its gratuitous limit of MINBSIZE and fixing the few things >> that break: > > That shouldn't be necessary, especially if you newfs with the "-o space" > option to force the fragments for multiple files to be allocated out of > the same block right from the start unstead of waiting to do this once > the filesystem starts getting full. > > I ran a Usenet server this way for quite a while with fairly good > results, though the average file size was a bit bigger, about 2K or so. > I found that if I didn't use "-o space" that space optimization wouldn't > kick in soon enough and I'd tend to run out of full blocks that would be > needed for larger files. The biggest performance problem that I ran > into was that as the directories shrank and grew, they would tend to get > badly fragmented, causing lookups to get slow. This was in the days > before dirhash ... Yeah, I started to write something about that, and stopped because I was afraid I was getting too far in the weeds, and the strategy used nowadays is so much better by default. If, as it was in my case, the model is almost exclusively write-once read-many, using -o space as the default works really well. I don't recall the OP well enough to know if that's what's happening here or not. It would be really cool if there was a way to tell the filesystem to do something in between ... like in this case shove 2 files into a 4k block so that performance is better, but it won't get too badly fragmented if the files have to grow. Doug -- This .signature sanitized for your protection From owner-freebsd-fs@FreeBSD.ORG Tue May 29 08:22:06 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D8731065673 for ; Tue, 29 May 2012 08:22:06 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 5438D8FC0A for ; Tue, 29 May 2012 08:22:04 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [192.92.129.5]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.5/8.14.5) with ESMTP id q4T8M1PZ015018 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Tue, 29 May 2012 11:22:01 +0300 (EEST) (envelope-from daniel@digsys.bg) Message-ID: <4FC48729.5050302@digsys.bg> Date: Tue, 29 May 2012 11:22:01 +0300 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0.4) Gecko/20120528 Thunderbird/10.0.4 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it> <4FC457F7.9000800@FreeBSD.org> <20120529161802.N975@besplex.bde.org> <20120529175504.K1291@besplex.bde.org> In-Reply-To: <20120529175504.K1291@besplex.bde.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 08:22:06 -0000 On 29.05.12 11:00, Bruce Evans wrote: > On Tue, 29 May 2012, Bruce Evans wrote: > >> On Mon, 28 May 2012, Doug Barton wrote: >>> The good news is that it's a big improvement (I've done similar >>> stuff in the past). You'll also want to tweak the -i (inode) value to >>> insure that you have sufficient inodes for the number of files you plan >>> to store. The default is not likely to be adequate for your needs. >> >> Big is relative. 4K-blocks with 200-byte files gives a wastage factor >> of 20. Metadata alone will be 256 bytes for the inode alone with ffs2. >> Only 128 bytes with ffs1. Only 32 bytes with msdosfs. > > Oops, only a wastage factor of 2.5 with the 512-byte fragments that are > normally used with 4K-blocks by ffs. 512-byte blocks with ffs only > give a small reduction in metadata size and better block allocation. > But how big the entire filesystem is going to be, anyway? Say, 10 million 200 byte files is some 2GB of real data. Let's say we have 4x waste and with UFS this will take some 8GB. Let's even say with ZFS there will be 20x waste and it grows to 40GB. (with data validation, no need to wait eons for fsck etc). Grow it to 100 million and it will eat say 400GB on ZFS. These are trivial file system sizes today, unless the data needs to fit on a thumb drive or is for an embedded system. Otherwise, the discussion is good reading :) Daniel From owner-freebsd-fs@FreeBSD.ORG Tue May 29 08:34:30 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id A58511065745 for ; Tue, 29 May 2012 08:34:30 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from [127.0.0.1] (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id C5C662019F3; Tue, 29 May 2012 08:33:22 +0000 (UTC) Message-ID: <4FC489D1.6070609@FreeBSD.org> Date: Tue, 29 May 2012 01:33:21 -0700 From: Doug Barton Organization: http://www.FreeBSD.org/ User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: Daniel Kalchev References: <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it> <4FC457F7.9000800@FreeBSD.org> <20120529161802.N975@besplex.bde.org> <20120529175504.K1291@besplex.bde.org> <4FC48729.5050302@digsys.bg> In-Reply-To: <4FC48729.5050302@digsys.bg> X-Enigmail-Version: 1.4.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 08:34:30 -0000 On 5/29/2012 1:22 AM, Daniel Kalchev wrote: > But how big the entire filesystem is going to be, anyway? Your math is good, but the problem isn't how big the data is going to be on disk, it's how to get some kind of reasonable performance. Just because you can jam something onto a disk doesn't mean you can get it back off again in any kind of a timely manner. :) This is even more true if you have a large data set combined with a highly random access pattern that doesn't repeat often enough to benefit from the cache. Doug -- This .signature sanitized for your protection From owner-freebsd-fs@FreeBSD.ORG Tue May 29 09:16:25 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7761B106564A for ; Tue, 29 May 2012 09:16:25 +0000 (UTC) (envelope-from alessio@interconnessioni.it) Received: from zimbra.interconnessioni.it (zimbra.interconnessioni.it [194.126.148.30]) by mx1.freebsd.org (Postfix) with ESMTP id 2B4428FC17 for ; Tue, 29 May 2012 09:16:24 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by zimbra.interconnessioni.it (Postfix) with ESMTP id 817E664019 for ; Tue, 29 May 2012 11:15:58 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.interconnessioni.it.interconnessioni.it Received: from zimbra.interconnessioni.it ([127.0.0.1]) by localhost (zimbra.interconnessioni.it [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sdUxJsfsIXn8 for ; Tue, 29 May 2012 11:15:54 +0200 (CEST) Received: from zimbra.interconnessioni.it (localhost.localdomain [127.0.0.1]) by zimbra.interconnessioni.it (Postfix) with ESMTP id 5C77664012 for ; Tue, 29 May 2012 11:15:54 +0200 (CEST) Date: Tue, 29 May 2012 11:15:54 +0200 (CEST) From: Alessio Focardi To: freebsd-fs@FreeBSD.org Message-ID: <49722655.1520.1338282954302.JavaMail.root@zimbra.interconnessioni.it> In-Reply-To: <4FC486BC.3050808@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.16.199.68] X-Mailer: Zimbra 7.1.4_GA_2555 (ZimbraWebClient - FF3.0 (Win)/7.1.4_GA_2555) Cc: Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 09:16:25 -0000 > > I ran a Usenet server this way for quite a while with fairly good > > results, though the average file size was a bit bigger, about 2K or > > so. > > I found that if I didn't use "-o space" that space optimization > > wouldn't > > kick in soon enough and I'd tend to run out of full blocks that > > would be > > needed for larger files. Fragmentation is not a problem for me, mostly I will have a write once-read many situation, still is not clear to me if "-o space" works in the constraints of the block/fragment ratio, that in my case it would still mean that I will have to use a 512 bytes subblock for every 200 byte files. ps really thank you for all of your help! Alessio Focardi ------------------ From owner-freebsd-fs@FreeBSD.ORG Tue May 29 09:41:44 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E3C1A106566C; Tue, 29 May 2012 09:41:44 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail02.syd.optusnet.com.au (mail02.syd.optusnet.com.au [211.29.132.183]) by mx1.freebsd.org (Postfix) with ESMTP id 0A0A68FC08; Tue, 29 May 2012 09:41:43 +0000 (UTC) Received: from c122-106-171-232.carlnfd1.nsw.optusnet.com.au (c122-106-171-232.carlnfd1.nsw.optusnet.com.au [122.106.171.232]) by mail02.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id q4T9fctV003703 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 29 May 2012 19:41:40 +1000 Date: Tue, 29 May 2012 19:41:38 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Don Lewis In-Reply-To: <201205290806.q4T86K8M007099@gw.catspoiler.org> Message-ID: <20120529182711.Y1436@besplex.bde.org> References: <201205290806.q4T86K8M007099@gw.catspoiler.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@FreeBSD.org, dougb@FreeBSD.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 09:41:45 -0000 On Tue, 29 May 2012, Don Lewis wrote: > On 29 May, Bruce Evans wrote: >> On Mon, 28 May 2012, Doug Barton wrote: >> >>> On 5/28/2012 10:01 AM, Alessio Focardi wrote: >>>> So in my case I would have to use -b 4096 -f 512 >>>> >>>> It's an improvement, but still is not ideal: still a big waste with 200 bytes files! >>> >>> Are all of the files exactly 200 bytes? If so that's likely the best you >>> can do. >> >> It is easy to do better by using a file system that supports small block >> sizes. This might be slow, but it reduces the wastage. Possible file >> systems: > >> - it is easy to fix ffs to support a minimum block size of 512 (by >> reducing its gratuitous limit of MINBSIZE and fixing the few things >> that break: I realized just after writing this that it doesn't save much space. > That shouldn't be necessary, especially if you newfs with the "-o space" > option to force the fragments for multiple files to be allocated out of > the same block right from the start unstead of waiting to do this once > the filesystem starts getting full. But this may pessimize the allocation even further. Even without -o space, IIRC ffs likes to fill in fragments. It does this even on nearly empty file systems. This tends to give backwards seeks, which drive caches might not handle very well (FreeBSD caches don't even attempt to cache nearby blocks in other files, so for packed small files FreeBSD depends on driver caches for the i/o performance to not be too bad). For example, according to my version of prtblknos: --- % fs_bsize = 8192 % fs_fsize = 1024 % 4: lbn 0 blkno 41 % 5: lbn 0 blkno 42-45 % 6: lbn 0 blkno 64-71 % 7: lbn 0 blkno 46 4: is the inode number of ".". Its data is allocated in the single blkno 41. These blknos are in ffs allocation units (fragments of size fs_fsize = 1024). Note that 41 is not a multiple of 8. It is the second fragment of the ffs block consisting of fragments with blkno's 40-47. Blkno 40 is the first fragment of this block. It is allocated somewhere in "..". After creating ".", I created a 4K file. This has inode 5, and is allocated in the 4 fragments after blkno 41. Then I created an 8K file. This has inode 6. Since its size is >= the block size, it is allocated in the full ffs block consisting of the 8 fragments with blkno's 64-71. Then I created a 512 byte file. This has inode 7. ffs "seeks back" and allocates it in the next free fragment (#46) in the full block 40-47. --- The backwards seeks are worst with mixtures of small and large files. Then reading of a small file typically results in the drive reading all nearby blocks but FreeBSD only reading 1 of these. Then reading a large file causes the blocks near the small file to be discarded from the drive's cache. Then reading a small file causes a seek back to near the first small file and the drive reading all nearby blocks again, an FreeBSD only reading 1 of these again... If ffs didn't seek back like this, then there would always be relatively large gaps between small files and locality would be defeated in another way. Using a block size of 512 results in not really using fragments. The allocation problem is simpler. Then, normally, no gaps are left between related files, unless multiple processes are creating and deleting related files concurrently, and backwards seeks are not needed to read back files that were created sequentially, when the read order is the same as the write order. > I ran a Usenet server this way for quite a while with fairly good > results, though the average file size was a bit bigger, about 2K or so. > I found that if I didn't use "-o space" that space optimization wouldn't > kick in soon enough and I'd tend to run out of full blocks that would be > needed for larger files. The biggest performance problem that I ran > into was that as the directories shrank and grew, they would tend to get > badly fragmented, causing lookups to get slow. This was in the days > before dirhash ... Perhaps FreeBSD ffs now does the backwards seek space optimization more, or I changed it to do so (the above is with my version). I tried changing my version to do the opposite (avoid filling in holes before the current preferred block), but this gave worse results. But I think you just saw a side effect of an old pessimization in ffs block allocation that was fixed about 10 years ago: ffs used to change the preferred block too often (for every directory or something like that, so that directories were allocated far away in another cylinder group). The backwards seeks shouldn't go so far back that they reach another cylinder group. So they will have to go forward more often, and start new blocks, and thus run out of full blocks faster. ffs still has too many cylinder groups, but they are not so harmful provided the block preference doesn't switch between them so often. Bruce From owner-freebsd-fs@FreeBSD.ORG Tue May 29 10:45:20 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id EC7DB1065673 for ; Tue, 29 May 2012 10:45:20 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from [127.0.0.1] (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id A797D14FE2C; Tue, 29 May 2012 10:45:20 +0000 (UTC) Message-ID: <4FC4A8BE.2000507@FreeBSD.org> Date: Tue, 29 May 2012 03:45:18 -0700 From: Doug Barton Organization: http://www.FreeBSD.org/ User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: Alessio Focardi References: <49722655.1520.1338282954302.JavaMail.root@zimbra.interconnessioni.it> In-Reply-To: <49722655.1520.1338282954302.JavaMail.root@zimbra.interconnessioni.it> X-Enigmail-Version: 1.4.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 10:45:21 -0000 On 5/29/2012 2:15 AM, Alessio Focardi wrote: >>> I ran a Usenet server this way for quite a while with fairly >>> good results, though the average file size was a bit bigger, >>> about 2K or so. I found that if I didn't use "-o space" that >>> space optimization wouldn't kick in soon enough and I'd tend to >>> run out of full blocks that would be needed for larger files. > > Fragmentation is not a problem for me, mostly I will have a write > once-read many situation, still is not clear to me if "-o space" > works in the constraints of the block/fragment ratio, that in my case > it would still mean that I will have to use a 512 bytes subblock for > every 200 byte files. TMK you can't have more than one file per fragment, but due to metadata you're not really wasting as much space as it sounds. If your data is truly WORM, I'd definitely give -o space a try. You'll probably want to benchmark various combinations anyway. Doug -- This .signature sanitized for your protection From owner-freebsd-fs@FreeBSD.ORG Tue May 29 12:52:33 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 02F08106564A for ; Tue, 29 May 2012 12:52:33 +0000 (UTC) (envelope-from resdev80@gmail.com) Received: from mail-lpp01m010-f54.google.com (mail-lpp01m010-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 6E2FE8FC14 for ; Tue, 29 May 2012 12:52:32 +0000 (UTC) Received: by laai10 with SMTP id i10so3401970laa.13 for ; Tue, 29 May 2012 05:52:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:subject:from:reply-to:to:date:content-type:x-mailer :content-transfer-encoding:mime-version; bh=ugjUEmceh4BiHZSw9Eq9UMBSLdaWNHXUo2U5xxwVjuw=; b=zEtc0rUQoH77rg4DmdVzZcD4x7epXfWdbldTS6dNqBEalD3lDG/V4YH6FIbgV5ZHye KGdyvUMSF5HnkvZ0UXzo1M9XI1O85Wj5HGkLnZzSo5WhAder6GmpexnPFaxXSSJrxK8S rf5RXsi9BTiGxmbNcLUFAG6VzpB8E2DJNRg7zzNJghFCTx680LUbsVQ6F1ehrGuvNsvT Dq4L1I5TY/wJRqnjoZzdHPp1XzbGyScBAmnVhP45AXIPNDobaB9H95C1vIR0EXD1sH3o tZFmr47gkyIhWg+uejtebaZHS8b8GyGiq781REtfrU8LRgD+9ILM3Ducni0mTEqnEQTn M7bw== Received: by 10.112.44.233 with SMTP id h9mr5505089lbm.26.1338295951097; Tue, 29 May 2012 05:52:31 -0700 (PDT) Received: from [217.21.170.21] ([217.21.170.21]) by mx.google.com with ESMTPS id j3sm9911156lbh.0.2012.05.29.05.52.28 (version=SSLv3 cipher=OTHER); Tue, 29 May 2012 05:52:29 -0700 (PDT) Message-ID: <1338295953.17714.12.camel@brainbug-nb.office.ctco.lv> From: Juris Krumins To: freebsd-fs@freebsd.org Date: Tue, 29 May 2012 15:52:33 +0300 Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.2.3 (3.2.3-3.fc16) Content-Transfer-Encoding: 7bit Mime-Version: 1.0 Subject: Hastd activemap cache flush and "request failed: FLUSH.: Operation not supported by device" messages. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: resdev80@gmail.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 12:52:33 -0000 Hi everybody. I've recently setup two FreeBSD nodes to use them as a storage. So I'm using hast over "MegaRAID SAS 1078" and ZFS filesystem created on HAST resource devices from /dev/hast/*. Everything looks good, but when I execute iozone tests, I'm constantly getting following messages in /var/log/message: primary node: May 29 15:44:15 rook hastd[2313]: [mfid2] (primary) Local request failed (Operation not supported by device): FLUSH. May 29 15:44:15 rook hastd[2311]: [mfid0] (primary) Local request failed (Operation not supported by device): FLUSH. May 29 15:44:15 rook hastd[2312]: [mfid1] (primary) Local request failed (Operation not supported by device): FLUSH. May 29 15:44:15 rook hastd[2312]: [mfid1] (primary) Remote request failed (Operation not supported by device): FLUSH. May 29 15:44:15 rook hastd[2313]: [mfid2] (primary) Remote request failed (Operation not supported by device): FLUSH. May 29 15:44:15 rook hastd[2311]: [mfid0] (primary) Remote request failed (Operation not supported by device): FLUSH. secondary node: May 29 15:43:41 rook2 hastd[2458]: [mfid1] (secondary) Request failed: FLUSH.: Operation not supported by device. May 29 15:43:41 rook2 hastd[2457]: [mfid0] (secondary) Request failed: FLUSH.: Operation not supported by device. May 29 15:43:41 rook2 hastd[2462]: [mfid2] (secondary) Request failed: FLUSH.: Operation not supported by device. I'v also found following topic http://lists.freebsd.org/pipermail/freebsd-fs/2012-January/013385.html. I've updated src, rebuild world and kernel and rebooted. I've also added metaflush "off" options to /etc/hast.conf configuration file. But still getting previously mentioned messages. Can somebody point me to the right direction how to configure hastd to throw this warning only once. I'm using: FreeBSD primary. 9.0-STABLE FreeBSD 9.0-STABLE #0: Tue May 29 12:53:44 EEST 2012 root@primary.:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD secondary. 9.0-STABLE FreeBSD 9.0-STABLE #0: Tue May 29 12:53:44 EEST 2012 root@secondary.:/usr/obj/usr/src/sys/GENERIC amd64 and hast.conf configuration from both machines: I've decided to put metaflush option on every single level just to be sure. No effect. metaflush "off" resource mfid0 { metaflush "off" on primary { local /dev/mfid0 remote 10.0.0.2 metaflush "off" } on secondary { local /dev/mfid0 remote 10.0.0.1 metaflush "off" } } resource mfid1 { metaflush "off" on primary { local /dev/mfid1 remote 10.0.0.2 metaflush "off" } on secondary { local /dev/mfid1 remote 10.0.0.1 metaflush "off" } } resource mfid2 { metaflush "off" on primary { local /dev/mfid2 remote 10.0.0.2 metaflush "off" } on secondary { local /dev/mfid2 remote 10.0.0.1 metaflush "off" } } Thank in advance. Juris Krumins From owner-freebsd-fs@FreeBSD.ORG Tue May 29 13:57:22 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5A7C91065673; Tue, 29 May 2012 13:57:22 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 2CC098FC0C; Tue, 29 May 2012 13:57:22 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q4TDvMkF063910; Tue, 29 May 2012 13:57:22 GMT (envelope-from rmacklem@freefall.freebsd.org) Received: (from rmacklem@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q4TDvLdM063906; Tue, 29 May 2012 13:57:21 GMT (envelope-from rmacklem) Date: Tue, 29 May 2012 13:57:21 GMT Message-Id: <201205291357.q4TDvLdM063906@freefall.freebsd.org> To: ob@e-Gitt.net, rmacklem@FreeBSD.org, freebsd-fs@FreeBSD.org From: rmacklem@FreeBSD.org Cc: Subject: Re: kern/167266: [zfs] [nfs] ZFS + new NFS export (sharenfs) leads to NAMEI leak X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 13:57:22 -0000 Synopsis: [zfs] [nfs] ZFS + new NFS export (sharenfs) leads to NAMEI leak State-Changed-From-To: open->closed State-Changed-By: rmacklem State-Changed-When: Tue May 29 13:55:17 UTC 2012 State-Changed-Why: Fix is in head as r234740 and has been MFC'd to stable/9 and stable/8 as r236134 and r236147 respectively. http://www.freebsd.org/cgi/query-pr.cgi?pr=167266 From owner-freebsd-fs@FreeBSD.ORG Tue May 29 14:47:36 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B2314106564A for ; Tue, 29 May 2012 14:47:36 +0000 (UTC) (envelope-from bryan@shatow.net) Received: from secure.xzibition.com (secure.xzibition.com [173.160.118.92]) by mx1.freebsd.org (Postfix) with ESMTP id 5787E8FC16 for ; Tue, 29 May 2012 14:47:36 +0000 (UTC) DomainKey-Signature: a=rsa-sha1; c=nofws; d=shatow.net; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type; q=dns; s=sweb; b=x0NmWC4oznw32ndqBHJekT4LYPcESp15 fJjrqdEQQ+FHw8LOogBGXqd1iOkdXdAjUoVgoU9Un5gnvxc2kdLzQhJUheSEIwn9 iUtcuzLA8ZJC4cKyb4XVkDSav4NR6VwjWG0CU3ouoWy1wyk9ncf1HPLmf9Yof0wc 5fh+qersSkc= DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=shatow.net; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type; s=sweb; bh=gcLpWI7ED2DidDua3NE6DxCfbNruFUuLBzSPh/ WKLVk=; b=yqxO7+rU/ppgl2HZVCI4VwhBZzJCFvidYUFBhi3Yc1cqk9mkDQRzHh Oyu/GHFiQqndY2GbIoP1HJEZjsKggO8Cy7x3O9GYILjWZnK+FbjTUKajRcyjJ70Y Kg9cb/nH1jv7OoGs7TMU05yEJdRow0cQISpgPPRrnQ2qmFQIMdRJ8= Received: (qmail 93555 invoked from network); 29 May 2012 09:47:29 -0500 Received: from unknown (HELO ?192.168.21.109?) (bryan@shatow.net@74.94.87.209) by sweb.xzibition.com with ESMTPA; 29 May 2012 09:47:29 -0500 Message-ID: <4FC4E17F.7060008@shatow.net> Date: Tue, 29 May 2012 09:47:27 -0500 From: Bryan Drewery User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: Petri Helenius References: In-Reply-To: X-Enigmail-Version: 1.4.1 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig466BE3B4C767AE5420F577DD" Cc: freebsd-fs@freebsd.org Subject: Re: ZVOL snapshots and dev files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 14:47:36 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig466BE3B4C767AE5420F577DD Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 5/29/2012 1:38 AM, Petri Helenius wrote: >=20 > Hi, >=20 > Figured out that if ZVOL snapshots are removed using -r, the entries in= /dev/zvol never get removed. > (until reboot) >=20 > Should I file a bug? >=20 Yes Regards, Bryan Drewery --------------enig466BE3B4C767AE5420F577DD Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJPxOGEAAoJEG54KsA8mwz5/LgQAJLL+BS96Qii6QxSzbHk8T/L BgHf0HOQI0AM89Di7EoK2+ggxFY6OO36gAiAx6HcMN3Gz0JKuCqHxlnozdNReGmN qJW4SoJtfSVCfn2nm4wVdPLRToiX2q1l97uutZh9JI81Wai2YQdH9ojPbrG23AQ3 SxLb+5UXalEKtDKkH4Xr3522frrAlITq4vd02FBYh3HVmBeMgMMQO1YqcPbjbAWl PlZa3ygJz7qdVKDeWKDqeM7dBoPKGfd9ZzmWGU9uH0tOAIz2qt7wdBmkPVj1R+5h nSqMSGiWoKMLmXK7Lhfb7q7Mg0AZN5scllmkV5UWjOseliEASc9k6r1ZbPf8p4gf U4wKvzGF+cFfuVlz5t6cDkuxYG4qNdfaF3s4U8rCegW2G9w/GtBGGve8rbvoodiY /algUIKcsuoRa0jZ0exU8nbCrWsvWaXq9yhEmCEVUTSOtvzyTywQvw3O+gE72mip p4UU7xHj9VQpcvxy/LhzllXdOKk2uXxpMc+j2Xpf6uNPvd7+9enyRE1Z/OoTXGgk ePDWYq2gOhEHORWLkdGMRWUwHo4Bvx3qSJm7k5LUTWXkbrKuRBwMWQ2r2vOe0IFn 1TFZ5Qitqyou+WbId0+Oq7NxZ1cMRfDi08lqwhsyAnF1Nui1pF9iJsN849gjgjAM VRwuebLiLagkhRL4jf1v =rLnY -----END PGP SIGNATURE----- --------------enig466BE3B4C767AE5420F577DD-- From owner-freebsd-fs@FreeBSD.ORG Tue May 29 16:37:07 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1FFFE106564A for ; Tue, 29 May 2012 16:37:07 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id E2C188FC1A for ; Tue, 29 May 2012 16:37:06 +0000 (UTC) Received: from julian-mac.elischer.org (c-67-180-24-15.hsd1.ca.comcast.net [67.180.24.15]) (authenticated bits=0) by vps1.elischer.org (8.14.5/8.14.5) with ESMTP id q4TGb5eh095379 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 29 May 2012 09:37:06 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <4FC4FB3F.8030203@freebsd.org> Date: Tue, 29 May 2012 09:37:19 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.28) Gecko/20120306 Thunderbird/3.1.20 MIME-Version: 1.0 To: Alessio Focardi References: <49722655.1520.1338282954302.JavaMail.root@zimbra.interconnessioni.it> In-Reply-To: <49722655.1520.1338282954302.JavaMail.root@zimbra.interconnessioni.it> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 16:37:07 -0000 On 5/29/12 2:15 AM, Alessio Focardi wrote: >>> I ran a Usenet server this way for quite a while with fairly good >>> results, though the average file size was a bit bigger, about 2K or >>> so. >>> I found that if I didn't use "-o space" that space optimization >>> wouldn't >>> kick in soon enough and I'd tend to run out of full blocks that >>> would be >>> needed for larger files. > Fragmentation is not a problem for me, mostly I will have a write once-read many situation, still is not clear to me if "-o space" works in the constraints of the block/fragment ratio, that in my case it would still mean that I will have to use a 512 bytes subblock for every 200 byte files. > > ps > > really thank you for all of your help! > > > Alessio Focardi > ------------------ > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > Maybe try use something other than a filesystem. The problem you will have is that the physical media is going to be formatted to 512 bytes or larger so every operation will be a read/modify/write if you try do to much packing.. As others have said it makes a big difference if you are talking about 20MB of data or 200GB of data too.. If it fits in ram, you could use 'mailbox format' and pack them all in one file and then just load it into ram when you need to access it. You could always buy one of our flash cards in key-value-store mode if you need 1.3 TB of 512 byte values at 400,000 values per second(read/write), but the pricetag might be a bit scary.. :-) (10k-ish) From owner-freebsd-fs@FreeBSD.ORG Tue May 29 18:30:00 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BCF26106566B; Tue, 29 May 2012 18:30:00 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id 60EAF8FC08; Tue, 29 May 2012 18:30:00 +0000 (UTC) Received: by yenl8 with SMTP id l8so2905629yen.13 for ; Tue, 29 May 2012 11:29:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=V9UJ8SG6CcNdoDeeNHlrRufaFBwQDkC51ejsZQmBgtU=; b=o6276NSvMHBDuhxphTDYudNlCrLec+MBRYI+IlKUpwbPFES/D5yFRe6DHSEI8xoyVm t7S71nqnjWYe60vZA4AKn+Pyoyw6u69Jy5cyt1sOHpc25m63UHkfuFDO/SUtE4C+L/CK z7rKyLlHndWA66CTDbjBNN7QMW/I80CW6sRDLSWLfNAsCto8od1RMOnjngZtdDMPAAJl +JeX7Od45gMSoBTVI8PXozPwdpC8qyPdh55ElRi+j+cUCUxephYqljvWkOukPX7Ws6ms d5BT/Wt/6E2Zvq4l4/mnCOjVQFFmWwN2npOt52Hx4mV5rdItCWv3y5D2KYVAvvrWdini 5GZQ== MIME-Version: 1.0 Received: by 10.50.203.41 with SMTP id kn9mr4642406igc.72.1338316197812; Tue, 29 May 2012 11:29:57 -0700 (PDT) Received: by 10.231.244.8 with HTTP; Tue, 29 May 2012 11:29:57 -0700 (PDT) In-Reply-To: <4FC4FB3F.8030203@freebsd.org> References: <49722655.1520.1338282954302.JavaMail.root@zimbra.interconnessioni.it> <4FC4FB3F.8030203@freebsd.org> Date: Wed, 30 May 2012 02:29:57 +0800 Message-ID: From: Marcelo Araujo To: Julian Elischer Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: araujo@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 18:30:00 -0000 2012/5/30 Julian Elischer > On 5/29/12 2:15 AM, Alessio Focardi wrote: > >> I ran a Usenet server this way for quite a while with fairly good >>>> results, though the average file size was a bit bigger, about 2K or >>>> so. >>>> I found that if I didn't use "-o space" that space optimization >>>> wouldn't >>>> kick in soon enough and I'd tend to run out of full blocks that >>>> would be >>>> needed for larger files. >>>> >>> Fragmentation is not a problem for me, mostly I will have a write >> once-read many situation, still is not clear to me if "-o space" works in >> the constraints of the block/fragment ratio, that in my case it would still >> mean that I will have to use a 512 bytes subblock for every 200 byte files. >> >> ps >> >> really thank you for all of your help! >> >> Maybe try use something other than a filesystem. The problem you will > have is that the physical media is going to be formatted to 512 bytes or > larger so every operation will be a read/modify/write if you try do to much > packing.. As others have said it makes a big difference if you are talking > about 20MB of data or 200GB of data too.. If it fits in ram, you could use > 'mailbox format' and pack them all in one file and then just load it into > ram when you need to access it. > > You could always buy one of our flash cards in key-value-store mode if you > need 1.3 TB of 512 byte values at 400,000 values per second(read/write), > but the pricetag might be a bit scary.. :-) (10k-ish) > > > Maybe you could take a look on Fusion-IO[1]. [1] http://www.fusionio.com/platforms/ -- Marcelo Araujo araujo@FreeBSD.org From owner-freebsd-fs@FreeBSD.ORG Wed May 30 07:29:40 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 998B0106566C for ; Wed, 30 May 2012 07:29:40 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 164138FC17 for ; Wed, 30 May 2012 07:29:39 +0000 (UTC) Received: by lbon10 with SMTP id n10so3726273lbo.13 for ; Wed, 30 May 2012 00:29:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:organization:references:sender:date:in-reply-to :message-id:user-agent:mime-version:content-type; bh=WWVDZ/LozHiF0HWYrgpgbP1Ka08AU3Pn1qEqan5vYgY=; b=LA2Eqkg/ETAxvNkV5fHAJgsH196h6UyUbeLXLskhSOzwxQ3Ar2LNiuOPLEyFKYcmBB RO4nLlne79OXtQ42ShyR7yN++qyyODc+1Ax8CEm5ifMwiDOuEF+zTqXqzkZYVkFIpgR2 XuJa8gazu2TDs9Dn+LGTxDVXCBFoUocwMV0we5yallR/i1IRWRs4XxlnXC8ynVJ4t+YL WZYQBPhReOsbUfXqjsmmKZKJhzirrL3kvT/z05EBhjSJ72x8QDa2MEfHI9pMobfHmzP/ +f02X5q5orXXQwQ0TejnT36W1ia45OmeZcEqS1F5IySnb9INpJuQWHxocjwjoagG3mq9 1zpA== Received: by 10.112.101.97 with SMTP id ff1mr6299156lbb.69.1338362978775; Wed, 30 May 2012 00:29:38 -0700 (PDT) Received: from localhost ([188.230.122.226]) by mx.google.com with ESMTPS id p2sm11115438lbj.4.2012.05.30.00.29.36 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 30 May 2012 00:29:37 -0700 (PDT) From: Mikolaj Golub To: resdev80@gmail.com Organization: TOA Ukraine References: <1338295953.17714.12.camel@brainbug-nb.office.ctco.lv> Sender: Mikolaj Golub Date: Wed, 30 May 2012 10:29:34 +0300 In-Reply-To: <1338295953.17714.12.camel@brainbug-nb.office.ctco.lv> (Juris Krumins's message of "Tue, 29 May 2012 15:52:33 +0300") Message-ID: <86ehq2nn35.fsf@in138.ua3> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.4 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-fs@freebsd.org Subject: Re: Hastd activemap cache flush and "request failed: FLUSH.: Operation not supported by device" messages. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2012 07:29:40 -0000 On Tue, 29 May 2012 15:52:33 +0300 Juris Krumins wrote: JK> Hi everybody. JK> I've recently setup two FreeBSD nodes to use them as a storage. JK> So I'm using hast over "MegaRAID SAS 1078" and ZFS filesystem created on JK> HAST resource devices from /dev/hast/*. JK> Everything looks good, but when I execute iozone tests, I'm constantly JK> getting following messages in /var/log/message: JK> primary node: JK> May 29 15:44:15 rook hastd[2313]: [mfid2] (primary) Local request failed JK> (Operation not supported by device): FLUSH. JK> May 29 15:44:15 rook hastd[2311]: [mfid0] (primary) Local request failed JK> (Operation not supported by device): FLUSH. JK> May 29 15:44:15 rook hastd[2312]: [mfid1] (primary) Local request failed JK> (Operation not supported by device): FLUSH. JK> May 29 15:44:15 rook hastd[2312]: [mfid1] (primary) Remote request JK> failed (Operation not supported by device): FLUSH. JK> May 29 15:44:15 rook hastd[2313]: [mfid2] (primary) Remote request JK> failed (Operation not supported by device): FLUSH. JK> May 29 15:44:15 rook hastd[2311]: [mfid0] (primary) Remote request JK> failed (Operation not supported by device): FLUSH. JK> secondary node: JK> May 29 15:43:41 rook2 hastd[2458]: [mfid1] (secondary) Request failed: JK> FLUSH.: Operation not supported by device. JK> May 29 15:43:41 rook2 hastd[2457]: [mfid0] (secondary) Request failed: JK> FLUSH.: Operation not supported by device. JK> May 29 15:43:41 rook2 hastd[2462]: [mfid2] (secondary) Request failed: JK> FLUSH.: Operation not supported by device. JK> I'v also found following topic JK> http://lists.freebsd.org/pipermail/freebsd-fs/2012-January/013385.html. JK> I've updated src, rebuild world and kernel and rebooted. I've also added JK> metaflush "off" options to /etc/hast.conf configuration file. But still JK> getting previously mentioned messages. Note, there are two sources of BIO_FLUSH operations in HAST: 1) BIO_FLUSH operations generated by HAST on metadata (activemap) update, if enabled (not disabled) in config. 2) BIO_FLUSH operations sent by a HAST user (filesystem). If the underlying provider doesn't support BIO_FLUSH, in the first case a warning "The provider doesn't support flushing write cache. Disabling it." will be generated and there will be no more attempt to flush on activemap update. In the second case the error 'Request failed: FLUSH.: Operation not supported by device' will be observed (as in your case). Before r225832 it logged on every BIO_FLUSH coming. Since r225832 HAST will try only once and only one error in logs should be observed after hastd start. JK> Can somebody point me to the right direction how to configure hastd to JK> throw this warning only once. JK> I'm using: JK> FreeBSD primary. 9.0-STABLE FreeBSD 9.0-STABLE #0: Tue May 29 12:53:44 JK> EEST 2012 root@primary.:/usr/obj/usr/src/sys/GENERIC amd64 JK> FreeBSD secondary. 9.0-STABLE FreeBSD 9.0-STABLE #0: Tue May 29 12:53:44 JK> EEST 2012 root@secondary.:/usr/obj/usr/src/sys/GENERIC amd64 r225832 has been merged to 9-STABLE on 4 Jan 2012 as r229509 and you shouldn't observe 'FLUSH: Operation not supported by device' constantly repeating. If you do this is very strange and you should recheck your build. Please provide the output of ident /sbin/hastd command. JK> and hast.conf configuration from both machines: JK> I've decided to put metaflush option on every single level just to be JK> sure. No effect. No need in disabling metaflush in your case. It will disable automatically after the first attempt failed. -- Mikolaj Golub From owner-freebsd-fs@FreeBSD.ORG Wed May 30 07:58:46 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 45AE0106566C; Wed, 30 May 2012 07:58:46 +0000 (UTC) (envelope-from resdev80@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6E5548FC1A; Wed, 30 May 2012 07:58:45 +0000 (UTC) Received: by lbon10 with SMTP id n10so3746211lbo.13 for ; Wed, 30 May 2012 00:58:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:subject:from:reply-to:to:cc:date:in-reply-to:references :content-type:x-mailer:content-transfer-encoding:mime-version; bh=pJjlhXKTrbjjuABc35bXQRWp4Bfh4uyhAUplG96FWDQ=; b=tdL2xBadHE0saUe9xPTNx2saxodXyEhp9uV/g3ttgPXg7ffATbStOHW3OQ+5lJn2ZR TxhHvqZ0p+39eOik+LBhX4wlp6GBrMsJF2tDpUlLvktAiofuTs3JcobdMfbwk6QmBpQC 35Gw8F9BYD2Rrmvmyt8kDhG4d3rFIu2JUccJvW/bpdgFh/gtIiAjOUAamZwPkuTCTK59 Q4AIgAb+v5nd91LyYShMIHyrirMM4kHwChE0yR6Hpwyk0fTtAdMwx5JWRBBY5pqT3qQu iYApKza41Goo66FG16MVJ4WxuJmZhIdbE8WcDPtncYMnLtPU1AWeiYbXqctS56CnjJNc nd+w== Received: by 10.112.86.132 with SMTP id p4mr6136518lbz.22.1338364724376; Wed, 30 May 2012 00:58:44 -0700 (PDT) Received: from [217.21.170.21] ([217.21.170.21]) by mx.google.com with ESMTPS id gv8sm26772889lab.14.2012.05.30.00.58.42 (version=SSLv3 cipher=OTHER); Wed, 30 May 2012 00:58:43 -0700 (PDT) Message-ID: <1338364726.2417.10.camel@brainbug-nb.office.ctco.lv> From: Juris Krumins To: Mikolaj Golub Date: Wed, 30 May 2012 10:58:46 +0300 In-Reply-To: <86ehq2nn35.fsf@in138.ua3> References: <1338295953.17714.12.camel@brainbug-nb.office.ctco.lv> <86ehq2nn35.fsf@in138.ua3> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.2.3 (3.2.3-3.fc16) Content-Transfer-Encoding: 7bit Mime-Version: 1.0 Cc: freebsd-fs@freebsd.org Subject: Re: Hastd activemap cache flush and "request failed: FLUSH.: Operation not supported by device" messages. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: resdev80@gmail.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2012 07:58:46 -0000 Thanks for your answer Mikolaj. I'v found later last night that BIO_FLUSH request comes from ZFS and You prove it (thank You). More than that, I've also found that it's possible to disable BIO_FLUSH requests as ZFS level by setting following sysctl setting: vfs.zfs.vdev.bio_flush_disable: 1 The only concert I have right now is how this setting can potentially effect ZFS stability and ability to recover in case of power failures and/or other issues ? P.S. Underlying hardware is IBM EXS3000 disk bay with 12 disks combined into 3 RAID5 volumes (4 disks in each) with writeback, disk cache enabled and BBU. On Wed, 2012-05-30 at 10:29 +0300, Mikolaj Golub wrote: > On Tue, 29 May 2012 15:52:33 +0300 Juris Krumins wrote: > > JK> Hi everybody. > JK> I've recently setup two FreeBSD nodes to use them as a storage. > JK> So I'm using hast over "MegaRAID SAS 1078" and ZFS filesystem created on > JK> HAST resource devices from /dev/hast/*. > JK> Everything looks good, but when I execute iozone tests, I'm constantly > JK> getting following messages in /var/log/message: > > JK> primary node: > JK> May 29 15:44:15 rook hastd[2313]: [mfid2] (primary) Local request failed > JK> (Operation not supported by device): FLUSH. > JK> May 29 15:44:15 rook hastd[2311]: [mfid0] (primary) Local request failed > JK> (Operation not supported by device): FLUSH. > JK> May 29 15:44:15 rook hastd[2312]: [mfid1] (primary) Local request failed > JK> (Operation not supported by device): FLUSH. > JK> May 29 15:44:15 rook hastd[2312]: [mfid1] (primary) Remote request > JK> failed (Operation not supported by device): FLUSH. > JK> May 29 15:44:15 rook hastd[2313]: [mfid2] (primary) Remote request > JK> failed (Operation not supported by device): FLUSH. > JK> May 29 15:44:15 rook hastd[2311]: [mfid0] (primary) Remote request > JK> failed (Operation not supported by device): FLUSH. > > JK> secondary node: > JK> May 29 15:43:41 rook2 hastd[2458]: [mfid1] (secondary) Request failed: > JK> FLUSH.: Operation not supported by device. > JK> May 29 15:43:41 rook2 hastd[2457]: [mfid0] (secondary) Request failed: > JK> FLUSH.: Operation not supported by device. > JK> May 29 15:43:41 rook2 hastd[2462]: [mfid2] (secondary) Request failed: > JK> FLUSH.: Operation not supported by device. > > JK> I'v also found following topic > JK> http://lists.freebsd.org/pipermail/freebsd-fs/2012-January/013385.html. > JK> I've updated src, rebuild world and kernel and rebooted. I've also added > JK> metaflush "off" options to /etc/hast.conf configuration file. But still > JK> getting previously mentioned messages. > > Note, there are two sources of BIO_FLUSH operations in HAST: > > 1) BIO_FLUSH operations generated by HAST on metadata (activemap) update, if > enabled (not disabled) in config. > > 2) BIO_FLUSH operations sent by a HAST user (filesystem). > > If the underlying provider doesn't support BIO_FLUSH, in the first case a warning > "The provider doesn't support flushing write cache. Disabling it." will be > generated and there will be no more attempt to flush on activemap update. > > In the second case the error 'Request failed: FLUSH.: Operation not supported > by device' will be observed (as in your case). Before r225832 it logged on > every BIO_FLUSH coming. Since r225832 HAST will try only once and only one > error in logs should be observed after hastd start. > > JK> Can somebody point me to the right direction how to configure hastd to > JK> throw this warning only once. > > JK> I'm using: > JK> FreeBSD primary. 9.0-STABLE FreeBSD 9.0-STABLE #0: Tue May 29 12:53:44 > JK> EEST 2012 root@primary.:/usr/obj/usr/src/sys/GENERIC amd64 > > JK> FreeBSD secondary. 9.0-STABLE FreeBSD 9.0-STABLE #0: Tue May 29 12:53:44 > JK> EEST 2012 root@secondary.:/usr/obj/usr/src/sys/GENERIC amd64 > > r225832 has been merged to 9-STABLE on 4 Jan 2012 as r229509 and you shouldn't > observe 'FLUSH: Operation not supported by device' constantly repeating. If > you do this is very strange and you should recheck your build. Please provide > the output of > > ident /sbin/hastd > > command. > > JK> and hast.conf configuration from both machines: > JK> I've decided to put metaflush option on every single level just to be > JK> sure. No effect. > > No need in disabling metaflush in your case. It will disable automatically > after the first attempt failed. > From owner-freebsd-fs@FreeBSD.ORG Thu May 31 19:32:11 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9021F1065691 for ; Thu, 31 May 2012 19:32:11 +0000 (UTC) (envelope-from warinthepocket@gmail.com) Received: from mail-wi0-f172.google.com (mail-wi0-f172.google.com [209.85.212.172]) by mx1.freebsd.org (Postfix) with ESMTP id 21D768FC1A for ; Thu, 31 May 2012 19:32:10 +0000 (UTC) Received: by wibhj8 with SMTP id hj8so4283502wib.13 for ; Thu, 31 May 2012 12:32:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=ThqNn62AlYkTyG5xMWUhsBKM/r/nI2kBZNDVmahSGLc=; b=mHaY74PkoKkkOUhRC4ZnV8Oqd2K8X/kk4V0rrKy1p6oTm0soEP0o47HG5pB1Bu45cX loa44okKbVcxsACAViDPPOUvWgda2xdfbl2Y0niwalD/+Feyc9AsAE7Fxdtr8uSv6zPT JWF8X/NYEEouLjKkamxUz4Qg6iTQq6qWVzH7fQCIVBY2jVxAN/vJNLpIX+xlJw+Ndy+v r2stYPAGtqriz6DVSoclF4igOajuwsZpJHQab/81G+MisUTGzp1/i2nGyoAkUqeoHTRO GXczJjFtauUOiCCy0KAOvalq0p34vtmeiqb8PNrbh0BBWa8zDjNkOsmuF46psrXaPppr RxNQ== MIME-Version: 1.0 Received: by 10.216.208.221 with SMTP id q71mr14047419weo.174.1338492729170; Thu, 31 May 2012 12:32:09 -0700 (PDT) Received: by 10.216.178.69 with HTTP; Thu, 31 May 2012 12:32:08 -0700 (PDT) Date: Thu, 31 May 2012 14:32:08 -0500 Message-ID: From: Dan Davis To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: ZFS raidz pool recovery - vdev X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2012 19:32:11 -0000 I'm trying to recover a ZFS raidz 3x1.5TB pool. Two of the disks are down, I'm attempting to recover at least one of those to rebuild the pool. All disks seem to be healthy according to smartctl. I know that generally the answer is "if the pool is corrupt, there's nothing you can do," but seeing that zfs isn't a black box, and looking at http://mbruning.blogspot.com/ and looking at Oracles documentation and source, I don't think this is hopeless. The disk I have the most hope for seems to have been overwritten by the bios, Specifically, the first vdev label is corrupted - from what I've read, there are 4 copies of the vdev label, I should have an exact copy of this corrupted vdev label at the end of the disk. I'm hoping that someone with more knowledge of zfs can help me a bit. I've been digging through the source a little, but haven't been able to find how zfs goes about mounting a pool - if a vdev label is corrupt or seemingly non-existent, does zfs look for the other labels, or does it simple fail? Should I be trying to copy one of the intact vdev labels in place of the corrupt one? The other corrupt disk, which I have much less hope for, was quick formatted with NTFS in an attempt to resilver the pool... prior to this major mistake ZFS had refused to do anything with the disk. I asked around ##freebsd on freenode and still was unable to find a solution. I couldn't take it out of the pool, I figured trying to wipe it would prevent zfs from identifying the disk, and I could move forward from there. I'm not exactly sure how, but I physically removed the wrong disk to format - I checked atleast a dozen times with zpool status to make sure that the pool was still functioning, and it was. It wasn't until I reattached the disk that zfs started telling me the truth. Does anyone know what a "quick" NTFS format does, what data would have been written? Any help would be much appreciated! Dan -- http://alpha2delta.blogspot.com/ Is it faith to understand nothing, and merely submit your convictions implicitly to the Church? -John Calvin From owner-freebsd-fs@FreeBSD.ORG Thu May 31 19:40:02 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EFA7E106566C for ; Thu, 31 May 2012 19:40:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D54948FC14 for ; Thu, 31 May 2012 19:40:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q4VJe2sR085866 for ; Thu, 31 May 2012 19:40:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q4VJe2nl085865; Thu, 31 May 2012 19:40:02 GMT (envelope-from gnats) Date: Thu, 31 May 2012 19:40:02 GMT Message-Id: <201205311940.q4VJe2nl085865@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Allen Landsidel Cc: Subject: Re: kern/160790: [fusefs] [panic] VPUTX: negative ref count with FUSE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Allen Landsidel List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2012 19:40:03 -0000 The following reply was made to PR kern/160790; it has been noted by GNATS. From: Allen Landsidel To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/160790: [fusefs] [panic] VPUTX: negative ref count with FUSE Date: Thu, 31 May 2012 15:30:47 -0400 This bug continues to occur on a new system, different 'hardware'. The system is running MooseFS and was migrated some time ago to an VMWare ESX 5 VM, 64bit. OS version is: 8.2-STABLE FreeBSD 8.2-STABLE #0: Fri Oct 14 00:11:10 UTC 2011 root@freebsd-82-64.master.concord.internal:/usr/obj/usr/src/sys/GENERIC amd64 My initial suspicion is there is some kind of race involved, as the error only seems to occur during times of heavy write load. The virtualized system does reboot on the panic however, though the physical machine the initial bug report regards did not. Panic: panic: vputx: negative ref cnt cpuid = 1 KDB: stack backtrace: #0 0xffffffff80607050 at kdb_backtrace+0x60 #1 0xffffffff805d4e74 at panic+0x1b4 #2 0xffffffff80666e31 at vputx+0x101 #3 0xffffffff8066702e at vput+0xe #4 0xffffffff8067206c at vn_open_cred+0x56c #5 0xffffffff8067210c at vn_open+0x1c #6 0xffffffff8066fc03 at kern_openat+0x163 #7 0xffffffff8066ff89 at kern_open+0x19 #8 0xffffffff8066ffa8 at open+0x18 #9 0xffffffff808c6301 at amd64_syscall+0x301 #10 0xffffffff808aee0c at Xfast_syscall+0xfc panic: bufwrite: buffer is not busy??? From owner-freebsd-fs@FreeBSD.ORG Thu May 31 23:26:10 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7A8B2106566C for ; Thu, 31 May 2012 23:26:10 +0000 (UTC) (envelope-from randy@psg.com) Received: from ran.psg.com (ran.psg.com [IPv6:2001:418:1::36]) by mx1.freebsd.org (Postfix) with ESMTP id 5CD168FC0C for ; Thu, 31 May 2012 23:26:10 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=rair.psg.com.psg.com) by ran.psg.com with esmtp (Exim 4.77 (FreeBSD)) (envelope-from ) id 1SaEkr-000J1B-3K for freebsd-fs@freebsd.org; Thu, 31 May 2012 23:26:09 +0000 Date: Fri, 01 Jun 2012 08:26:07 +0900 Message-ID: From: Randy Bush To: FreeBSD FS User-Agent: Wanderlust/2.15.9 (Almost Unreal) Emacs/22.3 Mule/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Subject: hptrr disk labeling X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2012 23:26:10 -0000 i have an hptrr controller with 12-16 2tb satas on it. a picture before i started cleaning up some disk failures. NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror ONLINE 0 0 0 label/m00-d01 ONLINE 0 0 0 label/m00-d00 ONLINE 0 0 0 mirror ONLINE 0 0 0 label/m01-d00 ONLINE 0 0 0 label/m01-d01 ONLINE 0 0 0 mirror ONLINE 0 0 0 label/m02-d00 ONLINE 0 0 0 label/m02-d01 ONLINE 0 0 0 mirror ONLINE 0 0 0 label/m03-d00 ONLINE 0 0 0 label/m03-d01 ONLINE 0 0 0 mirror ONLINE 0 0 0 label/m04-d00 ONLINE 0 0 0 label/m04-d01 ONLINE 0 0 0 mirror ONLINE 0 0 0 label/m05-d00 ONLINE 0 0 0 label/m05-d01 ONLINE 0 0 0 the reason i used glabels was because when the system boots or the controller rescans, it assigns da0 to the first drive it finds alive on the controller, da1 to the second, etc. this means that drive addition or removal changes the daX numbering. so the labels are so that zfs can find its ass when assembling the array. the kernel opt options ATA_STATIC_ID # Static device numbering is for ata, not sata, yes? is there a nice way out of this? randy From owner-freebsd-fs@FreeBSD.ORG Fri Jun 1 05:56:33 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5D27A1065672 for ; Fri, 1 Jun 2012 05:56:33 +0000 (UTC) (envelope-from ae@FreeBSD.org) Received: from mail-new.kirov.so-ups.ru (mail-new.kirov.so-ups.ru [178.74.170.12]) by mx1.freebsd.org (Postfix) with ESMTP id 0C8D68FC18 for ; Fri, 1 Jun 2012 05:56:33 +0000 (UTC) Received: from kirov.so-ups.ru (unknown [172.21.81.1]) by mail-new.kirov.so-ups.ru (Postfix) with ESMTP id CEC65A1E37; Fri, 1 Jun 2012 09:56:12 +0400 (MSK) Received: by ns.kirov.so-ups.ru (Postfix, from userid 1010) id 1AB4AB9FD7; Fri, 1 Jun 2012 09:56:22 +0400 (MSK) Received: from [127.0.0.1] (unknown [10.118.3.52]) by ns.kirov.so-ups.ru (Postfix) with ESMTP id D989EB9FD2; Fri, 1 Jun 2012 09:56:21 +0400 (MSK) Message-ID: <4FC85985.3000903@FreeBSD.org> Date: Fri, 01 Jun 2012 09:56:21 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla Thunderbird 1.5 (FreeBSD/20051231) MIME-Version: 1.0 To: kpneal@pobox.com References: <20120601033945.GA37797@neutralgood.org> In-Reply-To: <20120601033945.GA37797@neutralgood.org> X-Enigmail-Version: 1.4.1 Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: 7bit Cc: Randy Bush , FreeBSD FS Subject: Re: hptrr disk labeling X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jun 2012 05:56:33 -0000 On 01.06.2012 7:39, kpneal@pobox.com wrote: > Can you use GPT for partitioning? Put a single partition on each disk and > set the GPT label (which is not the glabel). See gpart's add and modify > subcommands. Actually GPT labels are implemented with the GLABEL class too. -- WBR, Andrey V. Elsukov From owner-freebsd-fs@FreeBSD.ORG Fri Jun 1 12:11:36 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9579E10657AA; Fri, 1 Jun 2012 12:11:36 +0000 (UTC) (envelope-from Kashyap.Desai@lsi.com) Received: from na3sys009aog128.obsmtp.com (na3sys009aog128.obsmtp.com [74.125.149.141]) by mx1.freebsd.org (Postfix) with ESMTP id E62C18FC08; Fri, 1 Jun 2012 12:11:35 +0000 (UTC) Received: from paledge01.lsi.com ([192.19.193.42]) (using TLSv1) by na3sys009aob128.postini.com ([74.125.148.12]) with SMTP ID DSNKT8ixd7gFagsq0EeI9wxz+siDPKs7LS8s@postini.com; Fri, 01 Jun 2012 05:11:36 PDT Received: from PALCAS01.lsi.com (128.94.213.117) by PALEDGE01.lsi.com (192.19.193.42) with Microsoft SMTP Server (TLS) id 8.3.213.0; Fri, 1 Jun 2012 08:06:56 -0400 Received: from inbexch01.lsi.com (135.36.98.37) by PALCAS01.lsi.com (128.94.213.117) with Microsoft SMTP Server (TLS) id 8.3.213.0; Fri, 1 Jun 2012 08:00:45 -0400 Received: from inbmail01.lsi.com ([135.36.98.64]) by inbexch01.lsi.com ([135.36.98.37]) with mapi; Fri, 1 Jun 2012 17:30:42 +0530 From: "Desai, Kashyap" To: "freebsd-scsi@freebsd.org" , "freebsd-fs@freebsd.org" Date: Fri, 1 Jun 2012 17:30:39 +0530 Thread-Topic: Kernel panic in FreeBSD-8.3 from UFS Thread-Index: Ac0/7imu82JsmcHyQVm3sDwdYp9fsA== Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "McConnell, Stephen" Subject: Kernel panic in FreeBSD-8.3 from UFS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jun 2012 12:11:36 -0000 Hi, We have seen kernel panic while doing IO along with HBA reset. This looks t= o be very rare but not sure if someone can help me to understand what is a = issue here. To me it does not look any issue with underline Device Driver <= mps>=20 See below back trace. #0 doadump (textdump=3D1) at pcpu.h:244 244 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=3D1) at pcpu.h:244 #1 0xc0a1845a in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:442 #2 0xc0a186f1 in panic (fmt=3DVariable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:607 #3 0xc0c7ceda in kmem_malloc (map=3D0xc15c808c, size=3D32768, flags=3D2) at /usr/src/sys/vm/vm_kern.c:334 #4 0xc0c708e7 in page_alloc (zone=3D0x0, bytes=3D32768, pflag=3D0xf19839bf= "\002",=20 wait=3D2) at /usr/src/sys/vm/uma_core.c:994 #5 0xc0c72fe0 in uma_large_malloc (size=3D32768, wait=3D2) at /usr/src/sys/vm/uma_core.c:3067 #6 0xc0a04fac in malloc (size=3D32768, mtp=3D0xc102b808, flags=3D2) at /usr/src/sys/kern/kern_malloc.c:492 #7 0xc0c42e89 in softdep_disk_io_initiation (bp=3D0xdef881fc) at /usr/src/sys/ufs/ffs/ffs_softdep.c:10126 #8 0xc0c5208f in ffs_geom_strategy (bo=3D0xc5fc30ac, bp=3D0xdef881fc) at buf.h:411 #9 0xc0c65a43 in ufs_strategy (ap=3D0xf1983b00) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2317 #10 0xc0d6a6dd in VOP_STRATEGY_APV (vop=3D0xc102e4a0, a=3D0xf1983b00) at vnode_if.c:2171 #11 0xc0a8d19e in bufstrategy (bo=3D0xc6b901bc, bp=3D0xdef881fc) at vnode_i= f.h:940 #12 0xc0a9352e in bufwrite (bp=3D0xdef881fc) at buf.h:404 #13 0xc0a8db5c in vfs_bio_awrite (bp=3D0xdef881fc) at buf.h:392 #14 0xc0c584c5 in ffs_syncvnode (vp=3D0xc6b90110, waitfor=3D1) at /usr/src/sys/ufs/ffs/ffs_vnops.c:288 #15 0xc0c58739 in ffs_fsync (ap=3D0xf1983c4c) at /usr/src/sys/ufs/ffs/ffs_vnops.c:187 #16 0xc0d69712 in VOP_FSYNC_APV (vop=3D0xc102dfc0, a=3D0xf1983c4c) at vnode_if.c:1267 #17 0xc0ab5d49 in sys_fsync (td=3D0xc64ea8a0, uap=3D0xf1983cec) at vnode_if= .h:549 #18 0xc0d49315 in syscall (frame=3D0xf1983d28) at subr_syscall.c:131 #19 0xc0d32af1 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:266 #20 0x00000033 in ?? ( To me it looks like UFS is doing something to crash the kernel. ` Kashyap From owner-freebsd-fs@FreeBSD.ORG Fri Jun 1 12:43:47 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 217C2106566C; Fri, 1 Jun 2012 12:43:47 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id A2B308FC08; Fri, 1 Jun 2012 12:43:46 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q51Chd6X088710; Fri, 1 Jun 2012 15:43:39 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q51ChdVp054522; Fri, 1 Jun 2012 15:43:39 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q51ChcMg054521; Fri, 1 Jun 2012 15:43:38 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 1 Jun 2012 15:43:38 +0300 From: Konstantin Belousov To: "Desai, Kashyap" Message-ID: <20120601124338.GU2358@deviant.kiev.zoral.com.ua> References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="NIjw7+ppZq6Cg8/r" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: "freebsd-fs@freebsd.org" , "freebsd-scsi@freebsd.org" , "McConnell, Stephen" Subject: Re: Kernel panic in FreeBSD-8.3 from UFS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jun 2012 12:43:47 -0000 --NIjw7+ppZq6Cg8/r Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jun 01, 2012 at 05:30:39PM +0530, Desai, Kashyap wrote: > Hi, >=20 > We have seen kernel panic while doing IO along with HBA reset. > This looks to be very rare but not sure if someone can help me to > understand what is a issue here. To me it does not look any issue with > underline Device Driver > > See below back trace. You did not specified the panic message. Was it 'kmem_map too small' ? Unless HBA driver causes memory leak, this is probably indeed unrelated. >=20 >=20 > #0 doadump (textdump=3D1) at pcpu.h:244 > 244 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) #0 doadump (textdump=3D1) at pcpu.h:244 > #1 0xc0a1845a in kern_reboot (howto=3D260) > at /usr/src/sys/kern/kern_shutdown.c:442 > #2 0xc0a186f1 in panic (fmt=3DVariable "fmt" is not available. > ) at /usr/src/sys/kern/kern_shutdown.c:607 > #3 0xc0c7ceda in kmem_malloc (map=3D0xc15c808c, size=3D32768, flags=3D2) > at /usr/src/sys/vm/vm_kern.c:334 > #4 0xc0c708e7 in page_alloc (zone=3D0x0, bytes=3D32768, pflag=3D0xf19839= bf "\002",=20 > wait=3D2) at /usr/src/sys/vm/uma_core.c:994 > #5 0xc0c72fe0 in uma_large_malloc (size=3D32768, wait=3D2) > at /usr/src/sys/vm/uma_core.c:3067 > #6 0xc0a04fac in malloc (size=3D32768, mtp=3D0xc102b808, flags=3D2) > at /usr/src/sys/kern/kern_malloc.c:492 > #7 0xc0c42e89 in softdep_disk_io_initiation (bp=3D0xdef881fc) > at /usr/src/sys/ufs/ffs/ffs_softdep.c:10126 > #8 0xc0c5208f in ffs_geom_strategy (bo=3D0xc5fc30ac, bp=3D0xdef881fc) > at buf.h:411 > #9 0xc0c65a43 in ufs_strategy (ap=3D0xf1983b00) > at /usr/src/sys/ufs/ufs/ufs_vnops.c:2317 > #10 0xc0d6a6dd in VOP_STRATEGY_APV (vop=3D0xc102e4a0, a=3D0xf1983b00) > at vnode_if.c:2171 > #11 0xc0a8d19e in bufstrategy (bo=3D0xc6b901bc, bp=3D0xdef881fc) at vnode= _if.h:940 > #12 0xc0a9352e in bufwrite (bp=3D0xdef881fc) at buf.h:404 > #13 0xc0a8db5c in vfs_bio_awrite (bp=3D0xdef881fc) at buf.h:392 > #14 0xc0c584c5 in ffs_syncvnode (vp=3D0xc6b90110, waitfor=3D1) > at /usr/src/sys/ufs/ffs/ffs_vnops.c:288 > #15 0xc0c58739 in ffs_fsync (ap=3D0xf1983c4c) > at /usr/src/sys/ufs/ffs/ffs_vnops.c:187 > #16 0xc0d69712 in VOP_FSYNC_APV (vop=3D0xc102dfc0, a=3D0xf1983c4c) > at vnode_if.c:1267 > #17 0xc0ab5d49 in sys_fsync (td=3D0xc64ea8a0, uap=3D0xf1983cec) at vnode_= if.h:549 > #18 0xc0d49315 in syscall (frame=3D0xf1983d28) at subr_syscall.c:131 > #19 0xc0d32af1 in Xint0x80_syscall () > at /usr/src/sys/i386/i386/exception.s:266 > #20 0x00000033 in ?? ( >=20 >=20 > To me it looks like UFS is doing something to crash the kernel. You might try to use vmstat -z and vmstat -m on core to see what has used KVA. --NIjw7+ppZq6Cg8/r Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk/IuPoACgkQC3+MBN1Mb4jDTwCg3cDGolPvQ5drJZ/aWLWFnU9f xpsAoMYn44lshfjyxGQLhL/CGYg52JuC =L/q9 -----END PGP SIGNATURE----- --NIjw7+ppZq6Cg8/r-- From owner-freebsd-fs@FreeBSD.ORG Fri Jun 1 12:50:11 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A9F991065678; Fri, 1 Jun 2012 12:50:11 +0000 (UTC) (envelope-from Kashyap.Desai@lsi.com) Received: from na3sys009aog123.obsmtp.com (na3sys009aog123.obsmtp.com [74.125.149.149]) by mx1.freebsd.org (Postfix) with ESMTP id 494CC8FC25; Fri, 1 Jun 2012 12:50:09 +0000 (UTC) Received: from paledge01.lsi.com ([192.19.193.42]) (using TLSv1) by na3sys009aob123.postini.com ([74.125.148.12]) with SMTP ID DSNKT8i6ezGNwyCrE9UVStQIApRr3Ia6BriA@postini.com; Fri, 01 Jun 2012 05:50:09 PDT Received: from PALCAS01.lsi.com (128.94.213.117) by PALEDGE01.lsi.com (192.19.193.42) with Microsoft SMTP Server (TLS) id 8.3.213.0; Fri, 1 Jun 2012 08:56:13 -0400 Received: from inbexch01.lsi.com (135.36.98.37) by PALCAS01.lsi.com (128.94.213.117) with Microsoft SMTP Server (TLS) id 8.3.213.0; Fri, 1 Jun 2012 08:50:02 -0400 Received: from inbmail01.lsi.com ([135.36.98.64]) by inbexch01.lsi.com ([135.36.98.37]) with mapi; Fri, 1 Jun 2012 18:19:59 +0530 From: "Desai, Kashyap" To: Konstantin Belousov Date: Fri, 1 Jun 2012 18:19:56 +0530 Thread-Topic: Kernel panic in FreeBSD-8.3 from UFS Thread-Index: Ac0/9DQxCjvBzL4sTc2DlPByoowCKAAAKAOw Message-ID: References: <20120601124338.GU2358@deviant.kiev.zoral.com.ua> In-Reply-To: <20120601124338.GU2358@deviant.kiev.zoral.com.ua> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-fs@freebsd.org" , "freebsd-scsi@freebsd.org" , "McConnell, Stephen" Subject: RE: Kernel panic in FreeBSD-8.3 from UFS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jun 2012 12:50:11 -0000 Thanks for the information. *YES* to me also looks like memory leaks only..= but it is CAM XPT who is using "366195K " memory.. See below output of "vmstat -m" vmstat -m Type InUse MemUse HighUse Requests Size(s) feeder 7 1K - 7 16 acpiintr 1 1K - 1 32 isadev 9 1K - 9 64 acpica 3179 172K - 73127 16,32,64,128,256,512,1024,2048 cdev 7 1K - 7 128 acpitask 1 1K - 1 1024 sigio 1 1K - 1 32 filedesc 50 14K - 2926 16,256,512,1024 kenv 121 9K - 130 16,32,64,128,4096 kqueue 0 0K - 266 128,1024 CAM dev queue 8 1K - 8 128 proc-args 26 2K - 4746 16,32,64,128 hhook 2 1K - 2 128 ithread 128 11K - 128 16,64,128 CAM queue 43 9K - 595257 16,32,64,128,256,512,1024,2048= ,4096 KTRACE 100 13K - 100 128 acpisem 21 3K - 21 64,128 linker 157 6K - 166 16,32,256 lockf 1751 61K - 2311 32,64,128,256,512,1024 loginclass 2 1K - 96 64 ip6ndp 12 1K - 13 64,128 ip6opt 0 0K - 3 32 temp 56 233K - 11165 16,32,64,128,256,512,1024,2048= ,4096 devbuf 5248 4507K - 5360 16,32,64,128,256,512,1024,2048= ,4096 module 493 31K - 493 64,128 mtx_pool 2 8K - 2 4096 CAM SIM 8 1K - 8 128 subproc 216 219K - 3091 256,4096 proc 2 8K - 2 4096 session 18 2K - 109 64 pgrp 25 2K - 129 64 cred 62 6K - 13960 64,128 uidinfo 3 2K - 88 64,1024 plimit 18 5K - 1389 256 scsi_cd 0 0K - 4 16 CAM periph 22 3K - 84532 16,32,64,128 CAM XPT 183208 366195K - 722021 16,32,64,256,1024,2048 sysctltmp 0 0K - 453 16,32,64,128,4096 sysctloid 5010 158K - 5286 16,32,64,128 sysctl 0 0K - 763 16,32,64 tidhash 1 8K - 1 =20 callout 7 1792K - 7 =20 umtx 750 71K - 750 64,128 p1003.1b 1 1K - 1 16 SWAP 2 4373K - 2 64 bus-sc 97 417K - 6298 16,32,64,128,256,512,1024,2048= ,4096 bus 1382 64K - 9711 16,32,64,128,256,1024 devstat 16 33K - 16 16,4096 eventhandler 73 4K - 73 32,64,128 UART 6 3K - 6 16,256,1024 kobj 358 716K - 634 2048 Per-cpu 1 1K - 1 16 ata_pci 2 1K - 2 32 rman 226 13K - 424 16,32,64 sbuf 0 0K - 1636 16,32,64,128,256,512,1024,2048= ,4096 scsi_da 0 0K - 186 16 stack 0 0K - 2 128 taskqueue 15 1K - 15 16,64 Unitno 14 1K - 5912 16,64 iov 0 0K - 1847877 16,64,128,256 select 9 1K - 9 64 ioctlops 0 0K - 5848 16,32,64,128,256,512,1024,2048 msg 4 25K - 4 1024,4096 sem 4 101K - 4 1024,4096 shm 1 12K - 1 =20 tty 21 11K - 23 512,2048 mbuf_tag 0 0K - 549 32,64 shmfd 1 4K - 1 4096 pcb 18 79K - 567 16,32,64,512,1024,2048,4096 soname 3 1K - 16885 16,32,128 vfscache 1 512K - 1 =20 cl_savebuf 0 0K - 48 32 vfs_hash 1 256K - 1 =20 acpidev 50 2K - 50 32 vnodes 2 1K - 2 128 vnodemarker 0 0K - 6497 512 mount 94 4K - 197 16,32,64,128,256 BPF 10 1K - 10 64 ether_multi 21 1K - 24 16,32,64 ifaddr 90 17K - 90 16,32,64,128,256,512,2048 ifnet 11 11K - 11 64,1024 USBdev 35 9K - 35 32,128,1024 clone 6 24K - 6 4096 arpcom 2 1K - 2 16 lltable 23 6K - 23 256 USB 66 40K - 69 16,32,64,128,256,1024,4096 routetbl 29 4K - 245 16,64,128,256 igmp 10 2K - 10 128 in_multi 1 1K - 1 128 sctp_iter 0 0K - 3 256 sctp_ifn 2 1K - 2 128 sctp_ifa 4 1K - 4 128 sctp_vrf 1 1K - 1 64 sctp_a_it 0 0K - 3 16 hostcache 1 16K - 1 =20 syncache 1 72K - 1 =20 entropy 1024 64K - 1024 64 in6_multi 15 2K - 15 16,256 pci_link 16 2K - 16 64,128 mld 10 2K - 10 128 rpc 2 1K - 2 128 audit_evclass 179 3K - 218 16 jblocks 2 1K - 2 128 savedino 0 0K - 121 256 sbdep 0 0K - 464 32 jsegdep 1 1K - 6778 32 jseg 1 1K - 4558 128 jfreefrag 0 0K - 179 64 jnewblk 0 0K - 5965 64 jremref 0 0K - 317 64 jaddref 0 0K - 317 64 freedep 0 0K - 9 32 freework 1 1K - 268 32,128 newdirblk 0 0K - 6 32 dirrem 0 0K - 305 64 mkdir 0 0K - 12 64 diradd 0 0K - 305 64 freefile 0 0K - 72 32 freeblks 0 0K - 157 128 freefrag 0 0K - 179 64 indirdep 1 1K - 4235 64 newblk 2 65K - 5966 128 bmsafemap 2 5K - 4389 128,4096 inodedep 2 257K - 4997 256 pagedep 1 64K - 51 128 ufs_dirhash 8 4K - 24 16,32,64,512 ufs_mount 21 390K - 21 256,4096 vm_pgdata 2 65K - 2 64 UMAHash 1 1K - 1 256 acpi_perf 2 1K - 2 256 DEVFS1 127 32K - 187 256 atkbddev 2 1K - 2 32 DEVFS3 141 18K - 223 128,256 DEVFS 24 1K - 25 16,64 memdesc 1 4K - 1 4096 apmdev 1 1K - 1 64 io_apic 2 2K - 2 1024 pfs_nodes 21 3K - 21 128 msi 3 1K - 3 64 nexusdev 5 1K - 5 16 GEOM 117 19K - 2291 16,32,64,128,256,512,1024,2048 SCSI SES 2 4K - 2 2048 kbdmux 7 18K - 7 16,256,1024,2048 mps 22 280K - 24141 16,32,64,128,256,512,2048,4096 mps_user 0 0K - 662 32,64 ` Kashyap > -----Original Message----- > From: Konstantin Belousov [mailto:kostikbel@gmail.com] > Sent: Friday, June 01, 2012 6:14 PM > To: Desai, Kashyap > Cc: freebsd-scsi@freebsd.org; freebsd-fs@freebsd.org; McConnell, Stephen > Subject: Re: Kernel panic in FreeBSD-8.3 from UFS >=20 > On Fri, Jun 01, 2012 at 05:30:39PM +0530, Desai, Kashyap wrote: > > Hi, > > > > We have seen kernel panic while doing IO along with HBA reset. > > This looks to be very rare but not sure if someone can help me to > > understand what is a issue here. To me it does not look any issue with > > underline Device Driver > > > > See below back trace. > You did not specified the panic message. Was it 'kmem_map too small' ? >=20 > Unless HBA driver causes memory leak, this is probably indeed unrelated. > > > > > > #0 doadump (textdump=3D1) at pcpu.h:244 > > 244 pcpu.h: No such file or directory. > > in pcpu.h > > (kgdb) #0 doadump (textdump=3D1) at pcpu.h:244 > > #1 0xc0a1845a in kern_reboot (howto=3D260) > > at /usr/src/sys/kern/kern_shutdown.c:442 > > #2 0xc0a186f1 in panic (fmt=3DVariable "fmt" is not available. > > ) at /usr/src/sys/kern/kern_shutdown.c:607 > > #3 0xc0c7ceda in kmem_malloc (map=3D0xc15c808c, size=3D32768, flags=3D= 2) > > at /usr/src/sys/vm/vm_kern.c:334 > > #4 0xc0c708e7 in page_alloc (zone=3D0x0, bytes=3D32768, pflag=3D0xf198= 39bf > "\002", > > wait=3D2) at /usr/src/sys/vm/uma_core.c:994 > > #5 0xc0c72fe0 in uma_large_malloc (size=3D32768, wait=3D2) > > at /usr/src/sys/vm/uma_core.c:3067 > > #6 0xc0a04fac in malloc (size=3D32768, mtp=3D0xc102b808, flags=3D2) > > at /usr/src/sys/kern/kern_malloc.c:492 > > #7 0xc0c42e89 in softdep_disk_io_initiation (bp=3D0xdef881fc) > > at /usr/src/sys/ufs/ffs/ffs_softdep.c:10126 > > #8 0xc0c5208f in ffs_geom_strategy (bo=3D0xc5fc30ac, bp=3D0xdef881fc) > > at buf.h:411 > > #9 0xc0c65a43 in ufs_strategy (ap=3D0xf1983b00) > > at /usr/src/sys/ufs/ufs/ufs_vnops.c:2317 > > #10 0xc0d6a6dd in VOP_STRATEGY_APV (vop=3D0xc102e4a0, a=3D0xf1983b00) > > at vnode_if.c:2171 > > #11 0xc0a8d19e in bufstrategy (bo=3D0xc6b901bc, bp=3D0xdef881fc) at > > vnode_if.h:940 > > #12 0xc0a9352e in bufwrite (bp=3D0xdef881fc) at buf.h:404 > > #13 0xc0a8db5c in vfs_bio_awrite (bp=3D0xdef881fc) at buf.h:392 > > #14 0xc0c584c5 in ffs_syncvnode (vp=3D0xc6b90110, waitfor=3D1) > > at /usr/src/sys/ufs/ffs/ffs_vnops.c:288 > > #15 0xc0c58739 in ffs_fsync (ap=3D0xf1983c4c) > > at /usr/src/sys/ufs/ffs/ffs_vnops.c:187 > > #16 0xc0d69712 in VOP_FSYNC_APV (vop=3D0xc102dfc0, a=3D0xf1983c4c) > > at vnode_if.c:1267 > > #17 0xc0ab5d49 in sys_fsync (td=3D0xc64ea8a0, uap=3D0xf1983cec) at > > vnode_if.h:549 > > #18 0xc0d49315 in syscall (frame=3D0xf1983d28) at subr_syscall.c:131 > > #19 0xc0d32af1 in Xint0x80_syscall () > > at /usr/src/sys/i386/i386/exception.s:266 > > #20 0x00000033 in ?? ( > > > > > > To me it looks like UFS is doing something to crash the kernel. >=20 > You might try to use vmstat -z and vmstat -m on core to see what has > used KVA. From owner-freebsd-fs@FreeBSD.ORG Fri Jun 1 12:58:30 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D6B551065670; Fri, 1 Jun 2012 12:58:30 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 2FB038FC1D; Fri, 1 Jun 2012 12:58:29 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q51CwOXi093185; Fri, 1 Jun 2012 15:58:24 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q51CwOml054641; Fri, 1 Jun 2012 15:58:24 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q51CwOZn054640; Fri, 1 Jun 2012 15:58:24 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 1 Jun 2012 15:58:24 +0300 From: Konstantin Belousov To: "Desai, Kashyap" Message-ID: <20120601125824.GV2358@deviant.kiev.zoral.com.ua> References: <20120601124338.GU2358@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Azlbkg4pdDb7tRhV" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: "freebsd-fs@freebsd.org" , "freebsd-scsi@freebsd.org" , "McConnell, Stephen" Subject: Re: Kernel panic in FreeBSD-8.3 from UFS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jun 2012 12:58:31 -0000 --Azlbkg4pdDb7tRhV Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jun 01, 2012 at 06:19:56PM +0530, Desai, Kashyap wrote: >=20 > Thanks for the information. *YES* to me also looks like memory leaks only= .. but it is CAM XPT who is using "366195K " memory.. >=20 Yes, it seems that cam should be investigated next. It is indeed most likely not related to UFS at all, and just happens to trace through the UFS code since you might have run fs-intensive test and filesystem just called allocator most often. > See below output of "vmstat -m" >=20 > vmstat -m >=20 > Type InUse MemUse HighUse Requests Size(s) > feeder 7 1K - 7 16 > acpiintr 1 1K - 1 32 > isadev 9 1K - 9 64 > acpica 3179 172K - 73127 16,32,64,128,256,512,1024,20= 48 > cdev 7 1K - 7 128 > acpitask 1 1K - 1 1024 > sigio 1 1K - 1 32 > filedesc 50 14K - 2926 16,256,512,1024 > kenv 121 9K - 130 16,32,64,128,4096 > kqueue 0 0K - 266 128,1024 > CAM dev queue 8 1K - 8 128 > proc-args 26 2K - 4746 16,32,64,128 > hhook 2 1K - 2 128 > ithread 128 11K - 128 16,64,128 > CAM queue 43 9K - 595257 16,32,64,128,256,512,1024,20= 48,4096 > KTRACE 100 13K - 100 128 > acpisem 21 3K - 21 64,128 > linker 157 6K - 166 16,32,256 > lockf 1751 61K - 2311 32,64,128,256,512,1024 > loginclass 2 1K - 96 64 > ip6ndp 12 1K - 13 64,128 > ip6opt 0 0K - 3 32 > temp 56 233K - 11165 16,32,64,128,256,512,1024,20= 48,4096 > devbuf 5248 4507K - 5360 16,32,64,128,256,512,1024,20= 48,4096 > module 493 31K - 493 64,128 > mtx_pool 2 8K - 2 4096 > CAM SIM 8 1K - 8 128 > subproc 216 219K - 3091 256,4096 > proc 2 8K - 2 4096 > session 18 2K - 109 64 > pgrp 25 2K - 129 64 > cred 62 6K - 13960 64,128 > uidinfo 3 2K - 88 64,1024 > plimit 18 5K - 1389 256 > scsi_cd 0 0K - 4 16 > CAM periph 22 3K - 84532 16,32,64,128 > CAM XPT 183208 366195K - 722021 16,32,64,256,1024,2048 > sysctltmp 0 0K - 453 16,32,64,128,4096 > sysctloid 5010 158K - 5286 16,32,64,128 > sysctl 0 0K - 763 16,32,64 > tidhash 1 8K - 1 =20 > callout 7 1792K - 7 =20 > umtx 750 71K - 750 64,128 > p1003.1b 1 1K - 1 16 > SWAP 2 4373K - 2 64 > bus-sc 97 417K - 6298 16,32,64,128,256,512,1024,20= 48,4096 > bus 1382 64K - 9711 16,32,64,128,256,1024 > devstat 16 33K - 16 16,4096 > eventhandler 73 4K - 73 32,64,128 > UART 6 3K - 6 16,256,1024 > kobj 358 716K - 634 2048 > Per-cpu 1 1K - 1 16 > ata_pci 2 1K - 2 32 > rman 226 13K - 424 16,32,64 > sbuf 0 0K - 1636 16,32,64,128,256,512,1024,20= 48,4096 > scsi_da 0 0K - 186 16 > stack 0 0K - 2 128 > taskqueue 15 1K - 15 16,64 > Unitno 14 1K - 5912 16,64 > iov 0 0K - 1847877 16,64,128,256 > select 9 1K - 9 64 > ioctlops 0 0K - 5848 16,32,64,128,256,512,1024,20= 48 > msg 4 25K - 4 1024,4096 > sem 4 101K - 4 1024,4096 > shm 1 12K - 1 =20 > tty 21 11K - 23 512,2048 > mbuf_tag 0 0K - 549 32,64 > shmfd 1 4K - 1 4096 > pcb 18 79K - 567 16,32,64,512,1024,2048,4096 > soname 3 1K - 16885 16,32,128 > vfscache 1 512K - 1 =20 > cl_savebuf 0 0K - 48 32 > vfs_hash 1 256K - 1 =20 > acpidev 50 2K - 50 32 > vnodes 2 1K - 2 128 > vnodemarker 0 0K - 6497 512 > mount 94 4K - 197 16,32,64,128,256 > BPF 10 1K - 10 64 > ether_multi 21 1K - 24 16,32,64 > ifaddr 90 17K - 90 16,32,64,128,256,512,2048 > ifnet 11 11K - 11 64,1024 > USBdev 35 9K - 35 32,128,1024 > clone 6 24K - 6 4096 > arpcom 2 1K - 2 16 > lltable 23 6K - 23 256 > USB 66 40K - 69 16,32,64,128,256,1024,4096 > routetbl 29 4K - 245 16,64,128,256 > igmp 10 2K - 10 128 > in_multi 1 1K - 1 128 > sctp_iter 0 0K - 3 256 > sctp_ifn 2 1K - 2 128 > sctp_ifa 4 1K - 4 128 > sctp_vrf 1 1K - 1 64 > sctp_a_it 0 0K - 3 16 > hostcache 1 16K - 1 =20 > syncache 1 72K - 1 =20 > entropy 1024 64K - 1024 64 > in6_multi 15 2K - 15 16,256 > pci_link 16 2K - 16 64,128 > mld 10 2K - 10 128 > rpc 2 1K - 2 128 > audit_evclass 179 3K - 218 16 > jblocks 2 1K - 2 128 > savedino 0 0K - 121 256 > sbdep 0 0K - 464 32 > jsegdep 1 1K - 6778 32 > jseg 1 1K - 4558 128 > jfreefrag 0 0K - 179 64 > jnewblk 0 0K - 5965 64 > jremref 0 0K - 317 64 > jaddref 0 0K - 317 64 > freedep 0 0K - 9 32 > freework 1 1K - 268 32,128 > newdirblk 0 0K - 6 32 > dirrem 0 0K - 305 64 > mkdir 0 0K - 12 64 > diradd 0 0K - 305 64 > freefile 0 0K - 72 32 > freeblks 0 0K - 157 128 > freefrag 0 0K - 179 64 > indirdep 1 1K - 4235 64 > newblk 2 65K - 5966 128 > bmsafemap 2 5K - 4389 128,4096 > inodedep 2 257K - 4997 256 > pagedep 1 64K - 51 128 > ufs_dirhash 8 4K - 24 16,32,64,512 > ufs_mount 21 390K - 21 256,4096 > vm_pgdata 2 65K - 2 64 > UMAHash 1 1K - 1 256 > acpi_perf 2 1K - 2 256 > DEVFS1 127 32K - 187 256 > atkbddev 2 1K - 2 32 > DEVFS3 141 18K - 223 128,256 > DEVFS 24 1K - 25 16,64 > memdesc 1 4K - 1 4096 > apmdev 1 1K - 1 64 > io_apic 2 2K - 2 1024 > pfs_nodes 21 3K - 21 128 > msi 3 1K - 3 64 > nexusdev 5 1K - 5 16 > GEOM 117 19K - 2291 16,32,64,128,256,512,1024,20= 48 > SCSI SES 2 4K - 2 2048 > kbdmux 7 18K - 7 16,256,1024,2048 > mps 22 280K - 24141 16,32,64,128,256,512,2048,40= 96 > mps_user 0 0K - 662 32,64 >=20 >=20 > ` Kashyap >=20 > > -----Original Message----- > > From: Konstantin Belousov [mailto:kostikbel@gmail.com] > > Sent: Friday, June 01, 2012 6:14 PM > > To: Desai, Kashyap > > Cc: freebsd-scsi@freebsd.org; freebsd-fs@freebsd.org; McConnell, Stephen > > Subject: Re: Kernel panic in FreeBSD-8.3 from UFS > >=20 > > On Fri, Jun 01, 2012 at 05:30:39PM +0530, Desai, Kashyap wrote: > > > Hi, > > > > > > We have seen kernel panic while doing IO along with HBA reset. > > > This looks to be very rare but not sure if someone can help me to > > > understand what is a issue here. To me it does not look any issue with > > > underline Device Driver > > > > > > See below back trace. > > You did not specified the panic message. Was it 'kmem_map too small' ? > >=20 > > Unless HBA driver causes memory leak, this is probably indeed unrelated. > > > > > > > > > #0 doadump (textdump=3D1) at pcpu.h:244 > > > 244 pcpu.h: No such file or directory. > > > in pcpu.h > > > (kgdb) #0 doadump (textdump=3D1) at pcpu.h:244 > > > #1 0xc0a1845a in kern_reboot (howto=3D260) > > > at /usr/src/sys/kern/kern_shutdown.c:442 > > > #2 0xc0a186f1 in panic (fmt=3DVariable "fmt" is not available. > > > ) at /usr/src/sys/kern/kern_shutdown.c:607 > > > #3 0xc0c7ceda in kmem_malloc (map=3D0xc15c808c, size=3D32768, flags= =3D2) > > > at /usr/src/sys/vm/vm_kern.c:334 > > > #4 0xc0c708e7 in page_alloc (zone=3D0x0, bytes=3D32768, pflag=3D0xf1= 9839bf > > "\002", > > > wait=3D2) at /usr/src/sys/vm/uma_core.c:994 > > > #5 0xc0c72fe0 in uma_large_malloc (size=3D32768, wait=3D2) > > > at /usr/src/sys/vm/uma_core.c:3067 > > > #6 0xc0a04fac in malloc (size=3D32768, mtp=3D0xc102b808, flags=3D2) > > > at /usr/src/sys/kern/kern_malloc.c:492 > > > #7 0xc0c42e89 in softdep_disk_io_initiation (bp=3D0xdef881fc) > > > at /usr/src/sys/ufs/ffs/ffs_softdep.c:10126 > > > #8 0xc0c5208f in ffs_geom_strategy (bo=3D0xc5fc30ac, bp=3D0xdef881fc) > > > at buf.h:411 > > > #9 0xc0c65a43 in ufs_strategy (ap=3D0xf1983b00) > > > at /usr/src/sys/ufs/ufs/ufs_vnops.c:2317 > > > #10 0xc0d6a6dd in VOP_STRATEGY_APV (vop=3D0xc102e4a0, a=3D0xf1983b00) > > > at vnode_if.c:2171 > > > #11 0xc0a8d19e in bufstrategy (bo=3D0xc6b901bc, bp=3D0xdef881fc) at > > > vnode_if.h:940 > > > #12 0xc0a9352e in bufwrite (bp=3D0xdef881fc) at buf.h:404 > > > #13 0xc0a8db5c in vfs_bio_awrite (bp=3D0xdef881fc) at buf.h:392 > > > #14 0xc0c584c5 in ffs_syncvnode (vp=3D0xc6b90110, waitfor=3D1) > > > at /usr/src/sys/ufs/ffs/ffs_vnops.c:288 > > > #15 0xc0c58739 in ffs_fsync (ap=3D0xf1983c4c) > > > at /usr/src/sys/ufs/ffs/ffs_vnops.c:187 > > > #16 0xc0d69712 in VOP_FSYNC_APV (vop=3D0xc102dfc0, a=3D0xf1983c4c) > > > at vnode_if.c:1267 > > > #17 0xc0ab5d49 in sys_fsync (td=3D0xc64ea8a0, uap=3D0xf1983cec) at > > > vnode_if.h:549 > > > #18 0xc0d49315 in syscall (frame=3D0xf1983d28) at subr_syscall.c:131 > > > #19 0xc0d32af1 in Xint0x80_syscall () > > > at /usr/src/sys/i386/i386/exception.s:266 > > > #20 0x00000033 in ?? ( > > > > > > > > > To me it looks like UFS is doing something to crash the kernel. > >=20 > > You might try to use vmstat -z and vmstat -m on core to see what has > > used KVA. --Azlbkg4pdDb7tRhV Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk/IvG8ACgkQC3+MBN1Mb4je2gCgqc5sAMpSoKxbz+NPCLz1fAa7 4FMAnj3hA9UgE3oSu6KLkverbyhQ1Goz =0QcW -----END PGP SIGNATURE----- --Azlbkg4pdDb7tRhV-- From owner-freebsd-fs@FreeBSD.ORG Fri Jun 1 16:47:53 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3D402106566B; Fri, 1 Jun 2012 16:47:53 +0000 (UTC) (envelope-from alessio@interconnessioni.it) Received: from zimbra.interconnessioni.it (zimbra.interconnessioni.it [194.126.148.30]) by mx1.freebsd.org (Postfix) with ESMTP id E8FBD8FC14; Fri, 1 Jun 2012 16:47:52 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by zimbra.interconnessioni.it (Postfix) with ESMTP id 22F5964005; Fri, 1 Jun 2012 18:47:12 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.interconnessioni.it.interconnessioni.it Received: from zimbra.interconnessioni.it ([127.0.0.1]) by localhost (zimbra.interconnessioni.it [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4iVZIFRAkLZQ; Fri, 1 Jun 2012 18:47:11 +0200 (CEST) Received: from zimbra.interconnessioni.it (localhost.localdomain [127.0.0.1]) by zimbra.interconnessioni.it (Postfix) with ESMTP id EAB1064011; Fri, 1 Jun 2012 18:47:11 +0200 (CEST) Date: Fri, 1 Jun 2012 18:47:11 +0200 (CEST) From: Alessio Focardi To: Garance A Drosehn , freebsd-fs@freebsd.org Message-ID: <225885946.5069.1338569231830.JavaMail.root@zimbra.interconnessioni.it> In-Reply-To: <4FC802E6.2060202@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [83.149.137.176] X-Mailer: Zimbra 7.1.4_GA_2555 (ZimbraWebClient - FF3.0 (Win)/7.1.4_GA_2555) Cc: Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jun 2012 16:47:53 -0000 > Instead of creating a *FILE* at /AA/BB/CC/DD, create a symlink. This is one of the most valuable suggestions I have received so far!!! Going to test it in a few hours, I guess I will have to format the partition to be mostly composed of inodes, right? Is not that clear how it can be accomplished using the formula -i number of bytes per inode Specify the density of inodes in the file system. The default is to create an inode for every (4 * frag-size) bytes of data space. If fewer inodes are desired, a larger number should be used; to create more inodes a smaller number should be given. One inode is required for each distinct file, so this value effectively specifies the average file size on the file system. maybe should "number of bytes for inode" (make an inode every X number of bytes) be set to 1? really tnx!!! From owner-freebsd-fs@FreeBSD.ORG Sat Jun 2 19:00:35 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E350C1065673 for ; Sat, 2 Jun 2012 19:00:35 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id CCA208FC38 for ; Sat, 2 Jun 2012 19:00:35 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q52J0ZUG066217 for ; Sat, 2 Jun 2012 19:00:35 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q52J0Zm8066216; Sat, 2 Jun 2012 19:00:35 GMT (envelope-from gnats) Date: Sat, 2 Jun 2012 19:00:35 GMT Message-Id: <201206021900.q52J0Zm8066216@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: dfilter@FreeBSD.ORG (dfilter service) Cc: Subject: Re: bin/155104: commit references a PR X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dfilter service List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Jun 2012 19:00:36 -0000 The following reply was made to PR bin/155104; it has been noted by GNATS. From: dfilter@FreeBSD.ORG (dfilter service) To: bug-followup@FreeBSD.org Cc: Subject: Re: bin/155104: commit references a PR Date: Sat, 2 Jun 2012 18:57:01 +0000 (UTC) Author: avg Date: Sat Jun 2 18:56:41 2012 New Revision: 236467 URL: http://svn.freebsd.org/changeset/base/236467 Log: MFC r235479: zpool_find_import_impl: another /dev/dsk -> /dev fix PR: bin/155104 Modified: stable/9/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_import.c Directory Properties: stable/9/cddl/contrib/opensolaris/ (props changed) Modified: stable/9/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_import.c ============================================================================== --- stable/9/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_import.c Sat Jun 2 18:55:25 2012 (r236466) +++ stable/9/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_import.c Sat Jun 2 18:56:41 2012 (r236467) @@ -1145,7 +1145,7 @@ zpool_find_import_impl(libzfs_handle_t * char *end, **dir = iarg->path; size_t pathleft; nvlist_t *ret = NULL; - static char *default_dir = "/dev/dsk"; + static char *default_dir = "/dev"; pool_list_t pools = { 0 }; pool_entry_t *pe, *penext; vdev_entry_t *ve, *venext; _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Sat Jun 2 19:10:21 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 08F811065731 for ; Sat, 2 Jun 2012 19:10:21 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id E1AFD8FC1E for ; Sat, 2 Jun 2012 19:10:16 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q52JAGxC085848 for ; Sat, 2 Jun 2012 19:10:16 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q52JAGBf085847; Sat, 2 Jun 2012 19:10:16 GMT (envelope-from gnats) Date: Sat, 2 Jun 2012 19:10:16 GMT Message-Id: <201206021910.q52JAGBf085847@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: dfilter@FreeBSD.ORG (dfilter service) Cc: Subject: Re: bin/155104: commit references a PR X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dfilter service List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Jun 2012 19:10:21 -0000 The following reply was made to PR bin/155104; it has been noted by GNATS. From: dfilter@FreeBSD.ORG (dfilter service) To: bug-followup@FreeBSD.org Cc: Subject: Re: bin/155104: commit references a PR Date: Sat, 2 Jun 2012 19:02:07 +0000 (UTC) Author: avg Date: Sat Jun 2 19:01:47 2012 New Revision: 236471 URL: http://svn.freebsd.org/changeset/base/236471 Log: MFC r235479: zpool_find_import_impl: another /dev//dsk -> /dev fix PR: bin/155104 Modified: stable/8/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_import.c Directory Properties: stable/8/cddl/contrib/opensolaris/ (props changed) Modified: stable/8/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_import.c ============================================================================== --- stable/8/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_import.c Sat Jun 2 18:59:18 2012 (r236470) +++ stable/8/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_import.c Sat Jun 2 19:01:47 2012 (r236471) @@ -1145,7 +1145,7 @@ zpool_find_import_impl(libzfs_handle_t * char *end, **dir = iarg->path; size_t pathleft; nvlist_t *ret = NULL; - static char *default_dir = "/dev/dsk"; + static char *default_dir = "/dev"; pool_list_t pools = { 0 }; pool_entry_t *pe, *penext; vdev_entry_t *ve, *venext; _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"