Date: Tue, 6 Dec 2016 10:45:37 +0200 From: Konstantin Belousov <kostikbel@gmail.com> To: Alex Tutubalin <lexa@lexa.ru> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: 12-CURRENT: ZFS single stream/large files read performance degradation Message-ID: <20161206084537.GJ54029@kib.kiev.ua> In-Reply-To: <477264f9-9263-6df8-4566-2911e84f2ad8@lexa.ru> References: <477264f9-9263-6df8-4566-2911e84f2ad8@lexa.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Dec 06, 2016 at 10:52:12AM +0300, Alex Tutubalin wrote: > Hi, > > I'm using freebsd/zfs box to store/access media files (mostly RAW photos > /thousands/ and some video), so it is 'single user NAS' (connected via > 10G to my workstation). > > This box has upgraded to 12-current several weeks ago, it was also > updated with new disks and after that I see serious read speed > performance degradation. > > This hurts me so much, that I've run tests with separate set of HDDs > yesterday, using FreeBSD 10.1, 10.2, 10.3, 11.0, 12-20161130 booted from > memstick (so, default settings, no adjustments). > > In details: > > Box configuration: Intel i5-2400 (3.1Ghz), 16GB RAM, LSI-9211-8i > controller in dumb IT-mode. Disks: WD1003FBYX (6-drive and 7-drive > RAIDZ2 were tested) > > Testing methodology: > > boot from memstick > zpool create -O recordsize=nn ztest raidz2 /dev/gpt/DISK[0-5/6] (for 6 > and 7 disk RAIDZ2) > zfs create ztest/pool > dd if=/dev/zero of=/ztest/pool/100gfile bs=1m count=100k > dd if=/ztest/pool/100gfile of=/dev/null bs=1m > > And write numbers (first free digits :) into excel table. > > I've tested 5 versions of FreeBSD (all memstick images are from > ftp.freebsd.org), two recordsizes (128k and 1m) and two RAIDZ2 disc > count: 6 and 7. > > Results are here in table: > http://blog.lexa.ru/2016/12/06/freebsd_zfs_read_speed_oni_ubili_kenni.html > My blog post is in Russian, but the table has english captions, so > speaks for itself. > > In short, for 6-drive array and 11.0 against 12-20163011 > > 128k recordsize: > > 11.0 503Mb/s(read)/462 Mb/s(write) > > 12-Current: 380/189 > > 1m recordsize: > > 11.0: 507/521 > > 12-current: 515/203. > > It is possible to get single-stream read speed back in 12-Current by > setting vfs.zfs.zfetch.max_distance to very high number (1 gigabyte or > even more, there is significant difference between 800m and 1800m), but > this is not acceptable solution because any small read will become very > large. > > For now, I'm downgrading back to 11.0, but it looks like I need to > report this degradation to whom it may concern :) Are you using memstick image from the download site ? Note that all HEAD images produced by re@ have full debugging enabled, among which both kernel-side INVARIANTS and WITNESS, and usermode malloc debugging, have potentially huge impact on the system performance.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20161206084537.GJ54029>