Skip site navigation (1)Skip section navigation (2) 
Date:      Tue, 20 Jun 2017 14:23:20 -0500
From:      Karl Denninger <karl@denninger.net>
To:        "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>
Subject:   Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs
Message-ID:  <54684f63-81c4-053c-7253-8aa07433e8fa@denninger.net>
In-Reply-To: <b7350cca59624e91abee6697aaf9e1b6@DM2PR58MB013.032d.mgd.msft.net>
index | next in thread | previous in thread | raw e-mail 

[-- Attachment #1 --]
On 6/20/2017 13:50, Caza, Aaron wrote:
>
> I've observed this performance degradation on 6 different hardware systems using 4 differents SSDS (2x Intel 510 120GB, 2x Intel 520 120GB, 2x Intel 540 120GB, 2x Samsung 850 Pro SSDs) on FreeBSD10.3 RELEASE, FreeBSD 10.3 RELEASEp6, FreeBSD 10.3RELEASEp19, FreeBSD 10-Stable, FreeBSD11.0 RELEASE, FreeBSD 11-Stable and now FreeBSD11.1 Beta 2.  This latest testing I'm not doing much in the way of writing - only logging the output of the 'dd' command along with 'zfs-stats -a' and 'uptime' to go along with it once an hour.   Ran for ~20hrs before performance drop kicked in though why it happens is inexplicable as this server isn't doing anything other than running this test hourly.
>
> I have a FreeBSD9.0 system using 2x Intel 520 120GB SSDs that doesn't exhibit this performance degradation, maintaining ~400MB/s speeds even after many days of uptime.  This is using the GEOM ELI layer to provide 4k sector emulation for the mirrored zpool as I previously described.
>
> Interestingly, using the GEOM ELI layering, I was seeing the following
> - FreeBSD 10.3 RELEASE  :  performance ~750MB/s when dd'ing 16GB file
> - FreeBSD 10 Stable         :  performance ~850MB/s when dd'ing 16GB file
> - FreeBSD 11 Stable         :  performance ~950MB/s when dd'ing 16GB file
>
> During the above testing, which was all done after reboot, gstat would show %busy of 90-95%.  When performance degradation hits, %busy drops to ~15%.
>
> Switching to FreeBSD 11.1 Beta 2 with Auto(ZFS) ashift-based 4k emulation of ZFS mirrored pool:
> - FreeBSD 11.1 Beta 2     :  performance ~450MB/s when dd'ing 16GB file with gstat %busy of ~60%.  When performance degradation hits, %busy drops to ~15%.
>
> Now, I expected that removing the GEOM ELI layer and just using vfs.zfs.min_auto_ashift=12 to do the 4k sector emulation would provide even better performance.  It's seems strange to me that it doesn't.

On one of my production systems (albeit in "hot spare" mode) here with
~20 days of uptime (plenty to saturate whatever, and this system DOES
have my patch in it.)

[\u@NewFS /dbms]# ls -al
total 65580101
drwxr-xr-x   4 root   wheel            5 Jun 20 14:06 .
drwxr-xr-x  45 root   wheel           55 Jun  1 10:58 ..
-rw-r-----   1 root   wheel  33554432000 Jun 20 14:13 test
drwxr-xr-x   2 root   wheel            2 Feb  4  2016 ticker-9.5
drwx------  19 pgsql  wheel           29 Apr 29 16:51 ticker-9.6
[\u@NewFS /dbms]# dd if=test of=/dev/null bs=1m
32000+0 records in
32000+0 records out
33554432000 bytes transferred in 43.023505 secs (779909306 bytes/sec)
[\u@NewFS /dbms]# uname -v
FreeBSD 11.0-STABLE #15 r312669M: Mon Jan 23 14:01:03 CST 2017    
karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP

~780 Mbps, more or less; the test file size is ~3x RAM and was created
using a dd off /dev/random, so it should not benefit from compression
(which IS on.)

This is with 128kbps recordsize and lz4 on that particular zfs dataset. 
The physical pool is a mirrored pair of Intel 730s and while this system
is reasonably quiet right now it's not quiescent.  ARC target and fill
at present is 8.43Gb (out of ~12Gb physical)

-- 
Karl Denninger
karl@denninger.net <mailto:karl@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/

[-- Attachment #2 --]
0	*H
010
	`He0	*H
\0X0@=0
	*H
010	UUS10UFlorida10U	Niceville10U
Cuda Systems LLC10UCuda Systems LLC CA1"0 	*H
	Cuda Systems LLC CA0
161218194535Z
211217194535Z0W10	UUS10UFlorida10U
Cuda Systems LLC10Ukarl@denninger.net0"0
	*H
0
͍fd`1ie6";fSz`5¹/?{=Ӵowjħ_fnӴMG\ҢҖ4ib}>@mJo&mM;
Q9U cj]p퐆W.2E=
^¢tzĄ'5i7_`~#dY
`]R]N%R}EXzqV@[oN	T>5AwYˡA"\v&YG]+($p:M,T?=mJkMљg*ym
L!J[./d׷?W^LysD'1
+V'~{-SSX=q-f=%&V<m4BeSet|
l2m 6iO{wv
+aHXˈ5=~é*C!?uJr3tb'3`Oe)üLxt&3N526llU
.|Cp[l?007++0)0'+0http://cudasystems.net:88880	U00	`HB0U0,	`HB
OpenSSL Generated Certificate0U/Zi
0GhG0U#0$q}ݽʒm50U0karl@denninger.net0
	*H
b%X%gwq	
ɁэrK[DMJ35W6
sz8d|qB2Cyw2PbV}
â[!W{HD7oD.TZ'w6~g( -,]R8P{*[f<1=7jGj9铚~3f2AʺN	k~@vz^j(>ͺyh2y{/9}4.45#S|<fW!.,Bss*Q+h=}l@	"q "M&6J5*,G {hɫjbNgǠ.ЃXȶ4$O.5evHlZba!4eE!x|Za1򹿈nZ5TuPvW|#G+	DZpI7S'n0 haGa@vZ	e|]Cu+))vRyY100010	UUS10UFlorida10U	Niceville10U
Cuda Systems LLC10UCuda Systems LLC CA1"0 	*H
	Cuda Systems LLC CA=0
	`HeM0	*H
	1	*H
0	*H
	1
170620192320Z0O	*H
	1B@3sݐs:Nky=aTH<E}*NqU6uJ(=
eve0l	*H
	1_0]0	`He*0	`He0
*H
0*H
0
*H
@0+0
*H
(0	+710010	UUS10UFlorida10U	Niceville10U
Cuda Systems LLC10UCuda Systems LLC CA1"0 	*H
	Cuda Systems LLC CA=0*H
	1010	UUS10UFlorida10U	Niceville10U
Cuda Systems LLC10UCuda Systems LLC CA1"0 	*H
	Cuda Systems LLC CA=0
	*H

%.G9w[UZLNc֪
C@nɌ׳UR@bhӗ`}-lQ{`ӘoĈ&eR16f#?T[Q3._hT!s㴫uC>'u4@;zV<oK&W7A9LO:(<=-ö7o#p>Eb@=[X6U}A[2Y)P=2|?F)/݁&;/_U;C0Њ_M?,N#.Kؑ^ZTtm4MM	Q
bt8s751r#tv܂꽬Bke%l
",fH5rV{cl}<b#x"e.\#d6*#Aw'Gԙ1lVywavº%鏗pqt`p
home | help

Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54684f63-81c4-053c-7253-8aa07433e8fa>