From owner-freebsd-fs@freebsd.org Tue Dec 6 08:45:45 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A61BC68C84 for ; Tue, 6 Dec 2016 08:45:45 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E6417F9D for ; Tue, 6 Dec 2016 08:45:44 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id uB68jbPQ059071 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 6 Dec 2016 10:45:37 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua uB68jbPQ059071 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id uB68jbhW059070; Tue, 6 Dec 2016 10:45:37 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 6 Dec 2016 10:45:37 +0200 From: Konstantin Belousov To: Alex Tutubalin Cc: "freebsd-fs@freebsd.org" Subject: Re: 12-CURRENT: ZFS single stream/large files read performance degradation Message-ID: <20161206084537.GJ54029@kib.kiev.ua> References: <477264f9-9263-6df8-4566-2911e84f2ad8@lexa.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <477264f9-9263-6df8-4566-2911e84f2ad8@lexa.ru> User-Agent: Mutt/1.7.1 (2016-10-04) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Dec 2016 08:45:45 -0000 On Tue, Dec 06, 2016 at 10:52:12AM +0300, Alex Tutubalin wrote: > Hi, > > I'm using freebsd/zfs box to store/access media files (mostly RAW photos > /thousands/ and some video), so it is 'single user NAS' (connected via > 10G to my workstation). > > This box has upgraded to 12-current several weeks ago, it was also > updated with new disks and after that I see serious read speed > performance degradation. > > This hurts me so much, that I've run tests with separate set of HDDs > yesterday, using FreeBSD 10.1, 10.2, 10.3, 11.0, 12-20161130 booted from > memstick (so, default settings, no adjustments). > > In details: > > Box configuration: Intel i5-2400 (3.1Ghz), 16GB RAM, LSI-9211-8i > controller in dumb IT-mode. Disks: WD1003FBYX (6-drive and 7-drive > RAIDZ2 were tested) > > Testing methodology: > > boot from memstick > zpool create -O recordsize=nn ztest raidz2 /dev/gpt/DISK[0-5/6] (for 6 > and 7 disk RAIDZ2) > zfs create ztest/pool > dd if=/dev/zero of=/ztest/pool/100gfile bs=1m count=100k > dd if=/ztest/pool/100gfile of=/dev/null bs=1m > > And write numbers (first free digits :) into excel table. > > I've tested 5 versions of FreeBSD (all memstick images are from > ftp.freebsd.org), two recordsizes (128k and 1m) and two RAIDZ2 disc > count: 6 and 7. > > Results are here in table: > http://blog.lexa.ru/2016/12/06/freebsd_zfs_read_speed_oni_ubili_kenni.html > My blog post is in Russian, but the table has english captions, so > speaks for itself. > > In short, for 6-drive array and 11.0 against 12-20163011 > > 128k recordsize: > > 11.0 503Mb/s(read)/462 Mb/s(write) > > 12-Current: 380/189 > > 1m recordsize: > > 11.0: 507/521 > > 12-current: 515/203. > > It is possible to get single-stream read speed back in 12-Current by > setting vfs.zfs.zfetch.max_distance to very high number (1 gigabyte or > even more, there is significant difference between 800m and 1800m), but > this is not acceptable solution because any small read will become very > large. > > For now, I'm downgrading back to 11.0, but it looks like I need to > report this degradation to whom it may concern :) Are you using memstick image from the download site ? Note that all HEAD images produced by re@ have full debugging enabled, among which both kernel-side INVARIANTS and WITNESS, and usermode malloc debugging, have potentially huge impact on the system performance.