From owner-freebsd-fs@FreeBSD.ORG Wed Jan 14 17:28:35 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 05E60CC6 for ; Wed, 14 Jan 2015 17:28:35 +0000 (UTC) Received: from mail-oi0-x22e.google.com (mail-oi0-x22e.google.com [IPv6:2607:f8b0:4003:c06::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B9169F00 for ; Wed, 14 Jan 2015 17:28:34 +0000 (UTC) Received: by mail-oi0-f46.google.com with SMTP id a3so8385868oib.5 for ; Wed, 14 Jan 2015 09:28:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=DUo8jUq6PmonmKJ3RhJt3pbFTUXves40HOxE7AwEo2s=; b=e79VtMcjLxae2nHgCgPVsEeZgdFgQM5omUuSMEcSsV4xet1jTpiM7uvqhx/tWFbaKP bp20SS9XNLWS5RWAKbtWgtBw+ZvVVdYoOlTE1pzji0JFv1u60VkMx9fQpSMn5l3/NgFV FIksgn3uQKTNMChULZwmJZWlhhxmnK2bUR6r6ZGVkMg0zbzxv9Mha4MZ6jDJgQJ2YERa Pi5xFGyQ3CvzMlEGGNjGlYUiywMCmAbz6gGXOJqBONZVWHs8WN6k3+HH51Rhz8f62LJl c9f5OpLL1GeWi9vUtP1ddntfK6DDbuvvQHvRTgjHLM7vHGeJxuuCipCbsdabZu1b0ml2 UbBA== MIME-Version: 1.0 X-Received: by 10.202.54.86 with SMTP id d83mr2951186oia.55.1421256514203; Wed, 14 Jan 2015 09:28:34 -0800 (PST) Received: by 10.202.76.71 with HTTP; Wed, 14 Jan 2015 09:28:34 -0800 (PST) In-Reply-To: <20150114170051.GB97640@pcjas.obspm.fr> References: <20150113105240.GA33162@pcjas.obspm.fr> <54B528AC.9090901@kateley.com> <20150114163849.GA97640@pcjas.obspm.fr> <54B69E4A.9010402@kateley.com> <20150114170051.GB97640@pcjas.obspm.fr> Date: Wed, 14 Jan 2015 09:28:34 -0800 Message-ID: Subject: Re: How many ram... From: Freddie Cash To: Albert Shih Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2015 17:28:35 -0000 On Wed, Jan 14, 2015 at 9:00 AM, Albert Shih wrote: > Le 14/01/2015 =C3=A0 10:50:18-0600, Linda Kateley a =C3=A9crit > > I will say one more thing.. I also have a customer who uses zfs for > > security camera storage. The cameras deliver 100's of k bytes per > > minute... But they save the data for a very very very long time. That > > kind of system would need very little ram(maybe 8GB) but lots and lots > > of disk. > > Thanks you very much. > > May I ask you something (feel free to not answer of course ). I saw on yo= ur > > > >> 32GB ram. http://kateleyco.com/?p=3D815 > > you have install I quote > > =C2=ABhey will have 252 4TB drives in 6 45-drive chassis with multiple > controllers=C2=BB > > do you have any idea how many pool they have ? how many disk they put in > one raid ? how many raid they put in one pool ? > > Actually I've one server a very big pool (I known some tell me it's too > big) with 72 disks in 6 raidz2. > > =E2=80=8BWe have two storage systems with 90 harddrives each (2 TB drives= ). These are setup with 2 45-drive 4U JBOD chassis each, and a 2U head unit with SSDs for the OS and L2ARC/ZIL, and the SATA controllers. The way the hardware is configured, they can each handle another 2 JBOD chassis without daisy-chaining anything. =E2=80=8BUsing 6-disk raidz2 vdevs, for a total of 15 raidz2 vdevs per stor= age pool.=E2=80=8B =E2=80=8BThese are backup systems and an off-site replication system, so st= orage throughput and IOps wasn't super critical, while storage space and manageability were. They only have gigabit NICs, and can saturate those while running backups or "zfs send" / "zfs recv". They do have 128 GB of RAM, though, more as a "get it now while it's cheap instead of waiting until we need it" than anything. And they have dual 8-core AMD Opteron CPUs (again, more as a "they're inexpensive now, so get as much as we can afford" than any real need for it). One of them has dedupe enabled (yeah, yeah, we know, we're moving away from it, it's actually the last one with it enabled), and actually does use the RAM for DDT storage in the ARC. The other one doesn't have dedupe enable, and most of the RAM sits "idle". =E2=80=8B --=20 Freddie Cash fjwcash@gmail.com