Date: Tue, 20 Jun 2017 20:52:45 +0100 From: Steven Hartland <killing@multiplay.co.uk> To: "Caza, Aaron" <Aaron.Caza@ca.weatherford.com>, "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Message-ID: <f60672a8-1e06-acf5-a789-3795867af9a0@multiplay.co.uk> In-Reply-To: <a5eea2354897429abd8c072d464c785f@DM2PR58MB013.032d.mgd.msft.net> References: <431d958c658a408d8bfd4c574a565439@DM2PR58MB013.032d.mgd.msft.net> <990886ae-b7c1-8630-1cef-f6678f0b5b63@multiplay.co.uk> <a5eea2354897429abd8c072d464c785f@DM2PR58MB013.032d.mgd.msft.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 20/06/2017 17:58, Caza, Aaron wrote: > >> Can you show the output from gstat -pd during this DD please. > dT: 1.001s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > 0 4318 4318 34865 0.0 0 0 0.0 0 0 0.0 14.2| ada0 > 0 4402 4402 35213 0.0 0 0 0.0 0 0 0.0 14.4| ada1 > > dT: 1.002s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > 1 4249 4249 34136 0.0 0 0 0.0 0 0 0.0 14.1| ada0 > 0 4393 4393 35287 0.0 0 0 0.0 0 0 0.0 14.5| ada1 You %busy is very low, so sounds like the bottleneck isn't in raw disk performance but somewhere else. Might be interesting to see if anything stands out in top -Sz and then press h for threads. You also mention ada, could you share: sysctl kern.cam And: sysctl vfs.zfs Finally when its performing well can you repeat the gstat -pd > Every now and again, I was seeing d/s hit, which I understand to be TRIM operations - it would briefly show 16 then go back to 0. > > test@f111beta2:~ # dd if=/testdb/test of=/dev/null bs=1m > 16000+0 records in > 16000+0 records out > 16777216000 bytes transferred in 43.447343 secs (386150561 bytes/sec) > test@f111beta2:~ # uptime > 9:54AM up 19:38, 2 users, load averages: 2.92, 1.01, 0.44 > root@f111beta2:~ # dd if=/testdb/test of=/dev/null bs=1m > 16000+0 records in > 16000+0 records out > 16777216000 bytes transferred in 236.097011 secs (71060688 bytes/sec) > test@f111beta2:~ # uptime > 10:36AM up 20:20, 2 users, load averages: 0.90, 0.62, 0.36 > > As can be seen in the above 'dd' test results, I'm back to seeing the original issue I reported on freebsd-hackers - performance degrading after < 24 hours of uptime going from ~386MB/sec to ~71MB/sec inexpicably - this server isn't doing anything other than running this test hourly. > > Please note in the gstat -pd output above, this was after the performance degradation hit. Prior to this, I was seeing %busy of ~60%. In this particular instance, the performance degradation hit ~20hrs into the test but I've see it hit as soon as ~5hrs. > > Previously, Allan Jude had advised zfs.vfs.trim.enabled=0 to see if this changed the behavior. I did this; however, it had no impact - but that was when I was using the GEOM ELI 4k sector emulation and not the ashift 4k sector emulation. The GEOM ELI 4k sector emulation does not appear to work with TRIM operations as gstat -d in that case always stayed at 0 ops/s. I can try disabling trim, but did not want to reboot the server to restart the test in case there was some additional info worth capturing. > > I have captured an hourly log that can be provided containing the initial dmsg, zpool status, zfs list, zfs get all along with an hourly capture of the results of running the above 'dd' test with associated zfs-stats -a and sysctl -a output though it's currently 2.8MB hence too large to post to this list. > > Also, there seems to be a problem with my freebsd-fs subscription as I'm not getting e-mail notifications despite having submitted a subscription request so apologies for my slow responses. >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?f60672a8-1e06-acf5-a789-3795867af9a0>