From owner-freebsd-stable@FreeBSD.ORG Fri May 10 10:48:45 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 7E0DC901 for ; Fri, 10 May 2013 10:48:45 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.21.123]) by mx1.freebsd.org (Postfix) with ESMTP id EC4F5C7 for ; Fri, 10 May 2013 10:48:44 +0000 (UTC) Received: from [193.68.6.170] ([193.68.6.170]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.6/8.14.6) with ESMTP id r4AAfbin009148 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Fri, 10 May 2013 13:41:39 +0300 (EEST) (envelope-from daniel@digsys.bg) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: recommended memory for zfs From: Daniel Kalchev In-Reply-To: Date: Fri, 10 May 2013 13:41:37 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <60A0A396-2E56-488F-BB5A-6EDD51B8039D@digsys.bg> References: <518BA237.3030700@gmail.com> <518C450B.5070809@ShaneWare.Biz> <518C51AF.5050609@gmail.com> <20130510020628.GA98750@icarus.home.lan> <518C5902.5050909@gmail.com> To: "Ronald Klop" X-Mailer: Apple Mail (2.1503) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 May 2013 10:48:45 -0000 On May 10, 2013, at 1:08 PM, "Ronald Klop" = wrote: > On Fri, 10 May 2013 04:18:42 +0200, Benjamin Adams = wrote: >=20 >> On 05/09/2013 10:06 PM, Jeremy Chadwick wrote: >>> On Thu, May 09, 2013 at 09:47:27PM -0400, Benjamin Adams wrote: >>>> On 05/09/2013 08:53 PM, Shane Ambler wrote: >>>>> On 09/05/2013 22:48, Benjamin Adams wrote: >>>>>> Hello zfs question about memory. >>>>>> I heard zfs is very ram hungry. >>>>>> Service looking to run: >>>>>> - nginx >>>>>> - postgres >>>>>> - php-fpm >>>>>> - python >>>>>>=20 >>>>>> I have a machine with two quad core cpus but only 4 G Memory >>>>>>=20 >>>>>> I'm looking to buy more ram now. >>>>>> What would be the recommend amount of memory for zfs across 6 = drives on >>>>>> this setup? >>>>>>=20 >>>>> I believe I heard a calculation of 1GB cache per 1TB of disk. But >>>>> basically zfs will use all free ram available if you access that >>>>> much data from disk. You will want to set vfs.zfs.arc_max to allow >>>>> enough ram for your apps to work in. >>>>>=20 >>>>> If you consider the files for your website and the data you store >>>>> you may find that you would never fill more than 500MB of cache. >>>>>=20 >>>>> If you will be serving large media files that will easily use up >>>>> the cache you could give them their own filesystem that only >>>>> caches metadata - zfs set primarycache=3Dmetadata zroot/mediafiles >>>>>=20 >>>>>=20 >>>> Thanks for all the replies Size of DB and HD's are: >>>>=20 >>>> Current DB Size =3D 23 GB >>>> HD sizes =3D (6) 500 GB drives >>> Nobody is going to be able to give you a precise/accurate = recommendation >>> given the lack of detail provided, I'm sorry to say. What's the RES >>> size of nginx (all processes combined)? What's the RES size of >>> postgres (same)? Do you have PHP scripts that "run amok" for long >>> periods of time and take up lots of RAM? Same with python? How = many >>> concurrent visitors and what sort of content are you hosting? Do = you >>> maintain/write your own PHP/Python code or are you using some crap = like >>> Wordpress? >>>=20 >>> This is just a **small** list of questions -- and what may come as a >>> shock is that I do not expect you to provide answers to any of them. >>> They are questions that you should, for yourself, attempt to answer = and >>> work out what you need from there ("teach a man to fish" and all = that). >>>=20 >>> The advice of "1GB of RAM per 1TB of disk space" is absolute = nonsense on >>> numerous levels -- whoever gave this advice to Shane either has no >>> understanding of how filesystems/ZFS works, or does but chose to >>> simplify to the point where they're providing half-ass information. >>> There is no direct, or even indirect, correlation between disk = capacity >>> and ZFS ARC size -- what matter is your "working set" (to quote = Tom). >>> You need to have some idea of how much disk I/O you're doing, and = what >>> type of I/O (sequential or random). >>>=20 >>> If you want my general advice, Benjamin, it's this: get yourself a >>> system with *minimum* 8GB of RAM but has the physical possibility of >>> supporting more (and only add more RAM when/if you know you need = it); do >>> not bother with ZFS on a system with 4GB. Run amd64, not i386 (I = don't >>> recommend bothering with ZFS on i386 -- I am not going to get into a >>> discussion about this either). Run stable/9, not 9.1-RELEASE. = Avoid >>> compression and dedup. And test disk failures as well (don't get = caught >>> with your pants down later). >>>=20 >>> The above advice comes from someone who did hosting (web/ssh/etc.) = for >>> almost 20 years with KISS principle applied at all levels. YMMV = though, >>> depending on what all you're doing/what you truly need. >>>=20 >>> Good luck. >>>=20 >> Jeremy, >>=20 >> Was just see if I should just get raid controller and more ram down = the road. >> List of priorities. >>=20 >> Main thing is I move from BSD when 9.0 came out. Was looking to see = if zfs is included in the installer. Now. >>=20 >> Sum up: >> upgrade ram to 16GB (not 64 like plained) >> and raid controller that supports level 5. >>=20 >=20 > Let ZFS do the RAID stuff. Do not use a RAID controller, but give the = plain disks to ZFS. Some of the nice features come from ZFS doing the = RAID stuff. To paraphrase this. Get yourself a nice HBA. non-RAID! For example, something based on = LSI2008 with IT firmware. With few enough discs you can avoid using SAS = expanders as well. RAID controllers, in addition to causing all kinds of = troubles are unlikely to have sufficient bandwidth and might turn out to = be the bottleneck (unless you are prepared to spend an unholy amount of = money -- which are better spent for RAM and CPU). If you want performance and low latency, avoid using compression and = dedup in ZFS. Set your record size appropriately for postgresql (8k) = *before* you run initdb. It is best to create an separate filesystem for = the database and set that property only there. If your database is heavy = on updates, you might be interested to use an SSD for ZIL. In general, = if you can afford it, an cheap SSD for L2ARC might do wonders -- if your = data set can fit there. If you intend to use SSDs and want the best = performance, use different SSDs for ZIL and L2ARC. The first needs to be = fast at writing and optimised for his (for example, the OCZ Vector) -- = it does not need to be large, at all. The second has to provide high = read throughput and IOPS, not so much for writing. Lately, I am building more and more servers with SSDs only and ZFS and = performance has been incomparable with spinning drives. Also, check what the sector size on your drives is. Most these days are = 4k already and you need to tell ZFS that (because the drives lie). It is = safer to plan for 4k drives, as future replacements are likely to be of = that kind. The same about SSDs.=20 Daniel=