From owner-freebsd-questions@freebsd.org Fri Apr 22 21:57:40 2016 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CB99AB1914C for ; Fri, 22 Apr 2016 21:57:40 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: from cosmo.uchicago.edu (cosmo.uchicago.edu [128.135.70.90]) by mx1.freebsd.org (Postfix) with ESMTP id 868A81DF3 for ; Fri, 22 Apr 2016 21:57:40 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: by cosmo.uchicago.edu (Postfix, from userid 48) id 02630CB8C9D; Fri, 22 Apr 2016 16:57:38 -0500 (CDT) Received: from 128.135.52.6 (SquirrelMail authenticated user valeri) by cosmo.uchicago.edu with HTTP; Fri, 22 Apr 2016 16:57:38 -0500 (CDT) Message-ID: <52604.128.135.52.6.1461362258.squirrel@cosmo.uchicago.edu> Date: Fri, 22 Apr 2016 16:57:38 -0500 (CDT) Subject: Re: Storage cluster advise, anybody? From: "Valeri Galtsev" To: "Mehmet Erol Sanliturk" Cc: "FreeBSD Questions Mailing List" Reply-To: galtsev@kicp.uchicago.edu User-Agent: SquirrelMail/1.4.8-5.el5.centos.7 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal References: <29462.128.135.52.6.1461352625.squirrel@cosmo.uchicago.edu> In-Reply-To: X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Apr 2016 21:57:40 -0000 On Fri, April 22, 2016 2:38 pm, Mehmet Erol Sanliturk wrote: > On Fri, Apr 22, 2016 at 12:17 PM, Valeri Galtsev > > wrote: > >> Dear Experts, >> I would like to ask everybody: what would you advise to use as a storage >> cluster, or as a distributed filesystem. >> >> I really didn't mean to write this, but I figure it probably will surface >> once I start getting your advices, so here it is. I did my research having >> my requirements in mind and came up with the solution: moosefs. It is not >> reviewed much, no reviews with criticism at all, and not much you can ("I >> could" I should say) find howtos about customizations, performance tuning >> etc. It installs without a hitch. It runs well, until you start stress writing a lot to it in parallel, then it started performing >> exponentially >> badly for me. Here is where extensive attempts to find performance tuning >> documentation faces lack of success. What made my decision to never ever >> use it in a future was the following. I started migrating data back from >> moosefs to local UFS (that is FreeBSD box) filesystem using rsync command. >> What I observed was: source files after they have been touched by rsync changed their timestamps. As if instead of creation timestamp it is an access timestamp on moosefs. This renders rsync from moosefs useless, as >> you can not re-run failed rsync, and you obliterate some of metadata of the source ("creation" timestamp). I wrote e-mail to sourceforge moosefs >> mail list, mentioning all this and the fact that I am using open source moosefs. Next day they replied asking whether I use version 3."this" or version 3."that", as they want to know in which of them they have a bug. >> Whereas latest open source version they have everywhere, including sourceforge is older version: 2.0.88. >> Basically, my decision was made. Sorry for venting it out here, but I figured, it will happen some moment when I will get your advises. >> Thanks a lot for all your advises! >> Valeri > In page : > http://moosefs.org/download.html > > there are the following lines with download links : > > Current release version 3.0.73-1 (GPLv2) 2016-03-04 > Current release version 3.0.69-1 (GPLv2) 2016-01-12 > Current release version 3.0.55-1 (GPLv2) 2015-10-21 > Current release version 3.0.37-1 (GPLv2) 2015-05-15 > Current release version 3.0.17-1 (GPLv2) 2015-04-21 > . > . > . > > > > Perhaps they assumed you are using one of them . Thanks for pointing that out. I see now that they are there under the name "current" as opposed to 2.0.88 "stable" - pretty much consistent with what FreeBSD terminology would be. So, 2.0.88 _is_ the latest "stable". No, I'm using "stable" on all production everything (my laptop doesn't count). And the weirdness I see is in production version (built on my machine from port I must admit). Thanks! Valeri > > > Mehmet Erol Sanliturk > ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++