From owner-freebsd-fs@freebsd.org Wed May 18 09:28:53 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D071EB3D72B for ; Wed, 18 May 2016 09:28:53 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from smtp.rlwinm.de (smtp.rlwinm.de [148.251.233.239]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8A2CD1BB4 for ; Wed, 18 May 2016 09:28:53 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from crest.local (unknown [87.253.189.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.rlwinm.de (Postfix) with ESMTPSA id AA7C86E14 for ; Wed, 18 May 2016 11:28:50 +0200 (CEST) Subject: Re: ZFS performance bottlenecks: CPU or RAM or anything else? To: freebsd-fs@freebsd.org References: <8441f4c0-f8d1-f540-b928-7ae60998ba8e@lexa.ru> From: Jan Bramkamp Message-ID: Date: Wed, 18 May 2016 11:28:49 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 In-Reply-To: <8441f4c0-f8d1-f540-b928-7ae60998ba8e@lexa.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 May 2016 09:28:53 -0000 On 17/05/16 14:00, Alex Tutubalin wrote: > Hi, > > I'm new to the list, sorry if the subject was discussed earlier (for > many times), just point to archives.... > > I'm building new storage server for 'linear read/linear write' > performance with limited number of parallel data streams (load like > read/write multi-gigabyte photoshop files, or read many large raw photo > files). > Target is to saturate 10G link using SMB or iSCSI. > > Several years ago I've tested small zpool (5x3Tb 7200rpm drives in > RAIDZ) with different CPU/memory combos and have got these results for > linear write speed by big chunks: > > 440 Mb/sec with Core i3-2120/DDR3-1600 ram (2 channel) > 360 Mb/sec with core i7-920/DDR3-1333 (3 channel RAM) > 280 Mb/sec with Core 2Q Q9300 /DDR2-800 (2 channel) > > Mixed thoughts: i7-920 is fastest of the three, RAM linear access also > fastest, but beaten by i3-2120 with lower latency memory. > > Also, I've found this link: > https://calomel.org/zfs_raid_speed_capacity.html > For 6x SSD and 10x SSD in RAIDZ2, there is very similar read speed > (1.7Gb/sec) and very close in write speed (721/806 Mb/sec for 6/10 drives). > > Assuming HBA/PCIe performance to be very same for read and write > operations, write speed is not limited by HBA/bus... so it is limited by > what? CPU or RAM or ...? > > So, my question is 'what CPU/memory is optimal for ZFS performance'? > > In particular: > - DDR3 or DDR4 (twice the bandwidth) ? > - limited number of cores and high clock rate (e.g. i3-6xxxx) or many > cores/slower clock ? > > No plans to use compression or deduplication, only raidz2 with 8-10 HDD > spindles and 3-4-5 SSDs for L2ARC. Don't forget that you're not just benchmarking CPUs. You're measuring whole systems with different disk controllers, memory controllers, interrupt routing etc. For example the Core 2 CPU is limited by its old design putting the memory controllers into the northbridge. Maybe you can reduce some of the differences by using the same PCI-e SAS HBA in each system.