From owner-freebsd-fs@freebsd.org Tue Dec 6 10:24:09 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1AE14C6A15E for ; Tue, 6 Dec 2016 10:24:09 +0000 (UTC) (envelope-from lexa@lexa.ru) Received: from mx3.lexa.ru (mx3.lexa.ru [198.27.68.102]) by mx1.freebsd.org (Postfix) with ESMTP id F170BF78 for ; Tue, 6 Dec 2016 10:24:08 +0000 (UTC) (envelope-from lexa@lexa.ru) Received: by mx3.lexa.ru (Postfix, from userid 66) id 65120224A5D; Tue, 6 Dec 2016 05:24:07 -0500 (EST) Received: from [193.124.130.130] (unknown [193.124.130.130]) by home-gw.lexa.ru (Postfix) with ESMTP id 79137B0583 for ; Tue, 6 Dec 2016 13:22:18 +0300 (MSK) Subject: Re: 12-CURRENT AND 11-stable: ZFS single stream/large files read performance degradation To: freebsd-fs@freebsd.org References: <477264f9-9263-6df8-4566-2911e84f2ad8@lexa.ru> From: Alex Tutubalin Message-ID: <1051d1c4-8ade-7a10-1007-c86e6f18cfe7@lexa.ru> Date: Tue, 6 Dec 2016 13:22:18 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <477264f9-9263-6df8-4566-2911e84f2ad8@lexa.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Dec 2016 10:24:09 -0000 On 12/6/2016 10:52 AM, Alex Tutubalin wrote: > This box has upgraded to 12-current several weeks ago, it was also > updated with new disks and after that I see serious read speed > performance degradation. It is looks even worse, 11-stable is affected too: - Another box, different hardware: i3-6300T, 32Gb ram, C236 Chipset, 6 x 6Tb HGST drives (3x7k6000 and 3xHe8) connected to motherboard SATA ports. The only zfs related tunables are about ARC size: vm.kmem_size="30G" vfs.zfs.arc_max="29G" 11-STABLE: FreeBSD home-gw.lexa.ru 11.0-STABLE FreeBSD 11.0-STABLE #1 r308516: Fri Nov 11 13:15:21 MSK 2016 lexa@home-gw.lexa.ru:/usr/obj/usr/src/sys/GENERIC amd64 Read speed of 100Gb file: lexa@home-gw:~# dd if=/ztest/100g of=/dev/null bs=1m 102400+0 records in 102400+0 records out 107374182400 bytes transferred in 640.324508 secs (167687135 bytes/sec) (zfs recordsize is 1m for this dataset) (Note, this is about single disk speed, He8 drives are capable to get 175Mb/sec at faster tracks, 7k600 are about 195Mb/s). Downgraded to 11-RELENG: FreeBSD home-gw.lexa.ru 11.0-RELEASE-p3 FreeBSD 11.0-RELEASE-p3 #2 r309605: Tue Dec 6 12:31:16 MSK 2016 lexa@home-gw.lexa.ru:/usr/obj/usr/src/sys/GENERIC amd64 read same file: same shit, about 160Mb/s. Re-created the file using downgdaded ZFS code: lexa@home-gw:/home/lexa# dd if=/dev/zero of=/ztest/100gb bs=1m count=100k 102400+0 records in 102400+0 records out 107374182400 bytes transferred in 170.845743 secs (628486146 bytes/sec) Rebooted to get cold read and run read test: dd if=/ztest/100gb of=/dev/null bs=1m 102400+0 records in 102400+0 records out 107374182400 bytes transferred in 168.924032 secs (635635919 bytes/sec) It is still not perfect, because only 4 disks are used in read iostat -x 5: device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b ada1 0 0 0.0 0.0 0 0 0 0 0 0 ada2 1229 0 157362.7 0.0 1 0 0 1 0 44 ada3 1231 0 157373.1 0.0 0 0 0 0 0 38 ada4 1229 0 157337.1 0.0 4 0 0 4 5 98 ada5 1231 0 157372.3 0.0 1 0 0 1 0 45 ada6 0 0 0.8 0.0 27 0 0 27 0 1 ada1 and ada6 are idle, while it is possible to read some data from it. But this is MUCH MUCH better, than on 11-STABLE and 12-CURRENT -- Alex Tutubalin Web: http://blog.lexa.ru mailto:lexa@lexa.ru