Date: Wed, 21 Jun 2017 16:58:55 +0100 From: Steven Hartland <killing@multiplay.co.uk> To: "Caza, Aaron" <Aaron.Caza@ca.weatherford.com>, "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: [EXTERNAL] Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Message-ID: <febef542-377b-c672-206b-c4599b25fae9@multiplay.co.uk> In-Reply-To: <7dbee74c0071463db4fbb0a5de2e7244@DM2PR58MB013.032d.mgd.msft.net> References: <c4c6d9a56a6648189011a28ab1a90f59@DM2PR58MB013.032d.mgd.msft.net> <86bf6fad-977a-b096-46b9-e9099a57a1f4@multiplay.co.uk> <7dbee74c0071463db4fbb0a5de2e7244@DM2PR58MB013.032d.mgd.msft.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 21/06/2017 13:31, Caza, Aaron wrote: > > > > > > *From:*Steven Hartland [mailto:killing@multiplay.co.uk] > > *Sent:*Wednesday, June 21, 2017 2:01 AM > > *To:*Caza, Aaron; freebsd-fs@freebsd.org > > *Subject:*[EXTERNAL] Re: FreeBSD 11.1 Beta 2 ZFS performance > degradation on SSDs > > > > > > On 20/06/2017 21:26, Caza, Aaron wrote: > > On 20/06/2017 17:58, Caza, Aaron wrote: > > dT: 1.001s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > > 0 4318 4318 34865 0.0 0 0 0.0 0 0 0.0 14.2| ada0 > > 0 4402 4402 35213 0.0 0 0 0.0 0 0 0.0 14.4| ada1 > > dT: 1.002s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > > 1 4249 4249 34136 0.0 0 0 0.0 0 0 0.0 14.1| ada0 > > 0 4393 4393 35287 0.0 0 0 0.0 0 0 0.0 14.5| ada1 > > You %busy is very low, so sounds like the bottleneck isn't in raw disk performance but somewhere else. > > Might be interesting to see if anything stands out in top -Sz and then press h for threads. > > I rebooted the system to disable Trim so currently not degraded. > > dT: 1.001s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > > 3 3887 3887 426514 0.7 0 0 0.0 0 0 0.0 90.7| ada0 > > 3 3987 3987 434702 0.7 0 0 0.0 0 0 0.0 92.0| ada1 > > dT: 1.002s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > > 3 3958 3958 433563 0.7 0 0 0.0 0 0 0.0 91.6| ada0 > > 3 3989 3989 438417 0.7 0 0 0.0 0 0 0.0 93.0| ada1 > > test@f111beta2:~ # dd if=/testdb/test of=/dev/null bs=1m > > 16000+0 records in > > 16000+0 records out > > 16777216000 bytes transferred in 19.385855 secs (865435959 bytes/sec) > > > Now that is interesting, as your getting smaller number ops/s but > much higher throughput. > > > > In the normal case you're seeing ~108Kb per read where in the > degraded case you're seeing 8Kb per read. > > > > Given this and knowing the application level isn't effecting it, we > need to identify where in the IO stack the reads are getting limited > to 8Kb? > > > > With your additional information about ARC, it could be that the > limited memory is the cause. > > > > Regards > > Steve > > I’m glad to learn that the above info is of some use. The 50M limit I > previously used for the ZFS ACS served me well for the past several > years. And, in fact, I thought it was still working well and only > accidentally stumbled over the performance drop when testing some > Intel 540 SSDs which were working surprisingly snappily despite using > TLC NAND flash. Initially, I saw a simple SQL query in Postgres go > from ~35 seconds to ~635 seconds and suspected the Intel 540s were the > cause. Turns out it was me hamstringing the ARC that was to blame. > That said, it’s interesting that, using the GEOM ELI layer for 4k > sector emulation, it runs fine for several hours before performance > drops off when the ARC is set too small going from 850MB/s down to > 80MB/s. In the case of ashift=12, the initial performance impact is > immediate on bootup going from 850MB/s with default ARC settings down > to 450MB/s with ARC set to 50M. Then, some hours later, dropping down > to ~70MB/s. > > With regards to Trim, there were a number of suggestions to disable > it. My understanding is that Trim support is highly desirable to > maintain peak performance but it seems it’s the #1 suspect when > there’s a performance drop. Is it that problematic? I’m considering > switching from GEOM ELI to ashift=12 for my 4k sector emulation in > order to get Trim but if it’s of no benefit then there’s not much point. > The GEOM hack is just that, you can actually remove the device once you've created the pool and it will have no effect.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?febef542-377b-c672-206b-c4599b25fae9>