Date: Tue, 6 Dec 2016 13:22:18 +0300 From: Alex Tutubalin <lexa@lexa.ru> To: freebsd-fs@freebsd.org Subject: Re: 12-CURRENT AND 11-stable: ZFS single stream/large files read performance degradation Message-ID: <1051d1c4-8ade-7a10-1007-c86e6f18cfe7@lexa.ru> In-Reply-To: <477264f9-9263-6df8-4566-2911e84f2ad8@lexa.ru> References: <477264f9-9263-6df8-4566-2911e84f2ad8@lexa.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On 12/6/2016 10:52 AM, Alex Tutubalin wrote: > This box has upgraded to 12-current several weeks ago, it was also > updated with new disks and after that I see serious read speed > performance degradation. It is looks even worse, 11-stable is affected too: - Another box, different hardware: i3-6300T, 32Gb ram, C236 Chipset, 6 x 6Tb HGST drives (3x7k6000 and 3xHe8) connected to motherboard SATA ports. The only zfs related tunables are about ARC size: vm.kmem_size="30G" vfs.zfs.arc_max="29G" 11-STABLE: FreeBSD home-gw.lexa.ru 11.0-STABLE FreeBSD 11.0-STABLE #1 r308516: Fri Nov 11 13:15:21 MSK 2016 lexa@home-gw.lexa.ru:/usr/obj/usr/src/sys/GENERIC amd64 Read speed of 100Gb file: lexa@home-gw:~# dd if=/ztest/100g of=/dev/null bs=1m 102400+0 records in 102400+0 records out 107374182400 bytes transferred in 640.324508 secs (167687135 bytes/sec) (zfs recordsize is 1m for this dataset) (Note, this is about single disk speed, He8 drives are capable to get 175Mb/sec at faster tracks, 7k600 are about 195Mb/s). Downgraded to 11-RELENG: FreeBSD home-gw.lexa.ru 11.0-RELEASE-p3 FreeBSD 11.0-RELEASE-p3 #2 r309605: Tue Dec 6 12:31:16 MSK 2016 lexa@home-gw.lexa.ru:/usr/obj/usr/src/sys/GENERIC amd64 read same file: same shit, about 160Mb/s. Re-created the file using downgdaded ZFS code: lexa@home-gw:/home/lexa# dd if=/dev/zero of=/ztest/100gb bs=1m count=100k 102400+0 records in 102400+0 records out 107374182400 bytes transferred in 170.845743 secs (628486146 bytes/sec) Rebooted to get cold read and run read test: dd if=/ztest/100gb of=/dev/null bs=1m 102400+0 records in 102400+0 records out 107374182400 bytes transferred in 168.924032 secs (635635919 bytes/sec) It is still not perfect, because only 4 disks are used in read iostat -x 5: device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b ada1 0 0 0.0 0.0 0 0 0 0 0 0 ada2 1229 0 157362.7 0.0 1 0 0 1 0 44 ada3 1231 0 157373.1 0.0 0 0 0 0 0 38 ada4 1229 0 157337.1 0.0 4 0 0 4 5 98 ada5 1231 0 157372.3 0.0 1 0 0 1 0 45 ada6 0 0 0.8 0.0 27 0 0 27 0 1 ada1 and ada6 are idle, while it is possible to read some data from it. But this is MUCH MUCH better, than on 11-STABLE and 12-CURRENT -- Alex Tutubalin Web: http://blog.lexa.ru mailto:lexa@lexa.ru
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1051d1c4-8ade-7a10-1007-c86e6f18cfe7>