Date: Tue, 20 Jun 2017 14:23:20 -0500 From: Karl Denninger <karl@denninger.net> To: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Message-ID: <54684f63-81c4-053c-7253-8aa07433e8fa@denninger.net> In-Reply-To: <b7350cca59624e91abee6697aaf9e1b6@DM2PR58MB013.032d.mgd.msft.net> References: <b7350cca59624e91abee6697aaf9e1b6@DM2PR58MB013.032d.mgd.msft.net>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms050708040103040502030205 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 6/20/2017 13:50, Caza, Aaron wrote: > > I've observed this performance degradation on 6 different hardware syst= ems using 4 differents SSDS (2x Intel 510 120GB, 2x Intel 520 120GB, 2x I= ntel 540 120GB, 2x Samsung 850 Pro SSDs) on FreeBSD10.3 RELEASE, FreeBSD = 10.3 RELEASEp6, FreeBSD 10.3RELEASEp19, FreeBSD 10-Stable, FreeBSD11.0 RE= LEASE, FreeBSD 11-Stable and now FreeBSD11.1 Beta 2. This latest testing= I'm not doing much in the way of writing - only logging the output of th= e 'dd' command along with 'zfs-stats -a' and 'uptime' to go along with it= once an hour. Ran for ~20hrs before performance drop kicked in though = why it happens is inexplicable as this server isn't doing anything other = than running this test hourly. > > I have a FreeBSD9.0 system using 2x Intel 520 120GB SSDs that doesn't e= xhibit this performance degradation, maintaining ~400MB/s speeds even aft= er many days of uptime. This is using the GEOM ELI layer to provide 4k s= ector emulation for the mirrored zpool as I previously described. > > Interestingly, using the GEOM ELI layering, I was seeing the following > - FreeBSD 10.3 RELEASE : performance ~750MB/s when dd'ing 16GB file > - FreeBSD 10 Stable : performance ~850MB/s when dd'ing 16GB fi= le > - FreeBSD 11 Stable : performance ~950MB/s when dd'ing 16GB fi= le > > During the above testing, which was all done after reboot, gstat would = show %busy of 90-95%. When performance degradation hits, %busy drops to = ~15%. > > Switching to FreeBSD 11.1 Beta 2 with Auto(ZFS) ashift-based 4k emulati= on of ZFS mirrored pool: > - FreeBSD 11.1 Beta 2 : performance ~450MB/s when dd'ing 16GB file= with gstat %busy of ~60%. When performance degradation hits, %busy drop= s to ~15%. > > Now, I expected that removing the GEOM ELI layer and just using vfs.zfs= =2Emin_auto_ashift=3D12 to do the 4k sector emulation would provide even = better performance. It's seems strange to me that it doesn't. On one of my production systems (albeit in "hot spare" mode) here with ~20 days of uptime (plenty to saturate whatever, and this system DOES have my patch in it.) [\u@NewFS /dbms]# ls -al total 65580101 drwxr-xr-x 4 root wheel 5 Jun 20 14:06 . drwxr-xr-x 45 root wheel 55 Jun 1 10:58 .. -rw-r----- 1 root wheel 33554432000 Jun 20 14:13 test drwxr-xr-x 2 root wheel 2 Feb 4 2016 ticker-9.5 drwx------ 19 pgsql wheel 29 Apr 29 16:51 ticker-9.6 [\u@NewFS /dbms]# dd if=3Dtest of=3D/dev/null bs=3D1m 32000+0 records in 32000+0 records out 33554432000 bytes transferred in 43.023505 secs (779909306 bytes/sec) [\u@NewFS /dbms]# uname -v FreeBSD 11.0-STABLE #15 r312669M: Mon Jan 23 14:01:03 CST 2017 =20 karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP ~780 Mbps, more or less; the test file size is ~3x RAM and was created using a dd off /dev/random, so it should not benefit from compression (which IS on.) This is with 128kbps recordsize and lz4 on that particular zfs dataset.=20 The physical pool is a mirrored pair of Intel 730s and while this system is reasonably quiet right now it's not quiescent. ARC target and fill at present is 8.43Gb (out of ~12Gb physical) --=20 Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms050708040103040502030205 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC BlwwggZYMIIEQKADAgECAgE9MA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE2MTIxODE5NDUzNVoXDTIxMTIxNzE5NDUzNVowVzEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxGzAZBgNVBAMUEmthcmxAZGVubmluZ2VyLm5ldDCCAiIwDQYJKoZIhvcNAQEBBQADggIP ADCCAgoCggIBAM2N5maxs7NkoY9g5NMxFWll0TYiO7gXrGZTo3q25ZJgNdPMwrntLz/5ewE9 07TEbwJ3ah/Ep9BfZm7JF9vTtE1HkgKtXNKi0pawNGm1Yn26Dz5AbUr1byby6dFtDJr14E07 trzDCtRRvTkOVSBj6PQPal0fAnDtkIYQBVcuMkXkuMCtyfE95pjm8g4K9l7lAcKii3T1/3rE hCc1o2nBnb7EN1/XwBeCDGB+I2SN/ftZDbKQqGAF5q9dUn+iXU7Z/CVSfUWmhVh6cVZA4Ftv TglUqj410OuPx+cUQch3h1kFgsuhQR63HiJc3HbRJllHsV0rihvL1CjeARQkhnA6uY9NLFST p5I/PfzBzW2MSmtN/tGZvmfKKnmtbfUNgkzbIR1K3lsum+yEL71kB93Xtz/4f1demEx5c8TJ RBIniDHjDeLGK1aoBu8nfnvXAvgthFNTWBOEoR49AHEPjC3kZj0l8JQml1Y8bTQD5gtC5txl klO60WV0EufU7Hy9CmynMuFtjiA2v71pm097rXeCdrAKgisdYeEESB+SFrlY65rLiLv4n8o1 PX7DqRfqKkOYIakZ0ug/yHVKcq2EM3RiJxwzls5gT70CoOBlKbrC98O8TA6teON0Jq30M06t NTI2HhvNbJDLbBH+Awf4h1UKB+0ufENwjVvF5Jfz8Ww/FaSDAgMBAAGjgfQwgfEwNwYIKwYB BQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgwCQYD VR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIBDQQf Fh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUpfAI3y+751pp9A0w 6vJHx8RoR/MwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYwFIES a2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBiB6MlugxYJdccD8boZ/u8 d8VxmLkJCtbfyYHRjYdyoABLW5hE3k3xSpYCM9L7vzWyV/UWwDYKi4ZzxHo4g+jG/GQZfKhx v38BQjL2G9xD0Hn2d+cygOq3UPjVYlbbfQoew6JbyCFXrrZ7/0jvRMLAN2+bRC7ynaFUixPH Whnj9JSH7ieYdzak8KN+G2coIC2t2iyfXVKehzi5gdNQ0vJ7+ypbGsRm4gE8Mdo9N/WgFPvZ HPFqR9Dwas7Z+aHwOabpk5r/336SyjOaZsn3MqKJQZL6GqDKusVOCWt+9uFAD8kadg7FetZe atIoD9I+zbp59oVoMnkMDMx7Hi85faU03csusqMGsjSsAzWSI1N8PJytZlchLiykokLKc3OL G87QKlErotlou7cfPX2BbEAH5wmkj9oiqZhxIL/wwAUA+PkiTbEmksKBNompSjUq/6UsR8EA s74gnu17lmijv8mrg2qMlwRirE7qG8pnE8egLtCDxcjd0Of9WMi2NJskn0/ovC7P+J60Napl m3ZIgPJst1piYSE0Zc1FIat4fFphMfK5v4iLblo1tFSlkdx1UNDGdg/U+LaXkNVXlMp8fyPm R80V6cIrCAlEWnBJNxG1UyfbbsvNMCCZBM4faGGsR/hhQOiydlruxhjL6P8J2WV8p11DdeGx KymWoil2s1J5WTGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxv cmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRww GgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5c3Rl bXMgTExDIENBAgE9MA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcN AQcBMBwGCSqGSIb3DQEJBTEPFw0xNzA2MjAxOTIzMjBaME8GCSqGSIb3DQEJBDFCBEAIAPEz c4rdkKOywgcEczoaTsBr9qB5PWEbgFRI8uU8n0UGfSroTnG5h1XeATa4qnXCStQoPdzaDeOE Zcl2fxRlMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggq hkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZI hvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNVBAgT B0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExM QzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3VkYSBT eXN0ZW1zIExMQyBDQQIBPTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYTAlVT MRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEg U3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0B CQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECAT0wDQYJKoZIhvcNAQEBBQAEggIAHgol0gzU2C5H rjntgfZ3W/KW2VXwWkxO3GONl9aqhr7mCuNDCBu2uUDqzuhuyYzXs+/uVVJAwWK7mMVo05eq 0/1gfbSIuRAt2BBsUXvTG2AHqNOYb6nEiLnoJmXOUvcxBTauqBXjZiO6P7B/ytzP4VRbUegz p9LrLtj/Xx6kHt3MEmiW+1Qh3s4EhnMfAuO0q3X7Q6ui2D6FwBXKzcwndZmyNO1AlDt6VvQ8 4W9L3ybnAVf4N0HC7IYPOUzy/E8IuQs6Byg8p7DaFOmdPZAtw7Y3byNwPsTT4fRFt9BiQAE9 W1iWNq6U9VV9QRNbBPQyBPRZgSlQrT0yfD9G2xUpxvLZL92BJu0fs587Lx5fGFUMgTug6fxD MJv7F5HI0IpfoU2fPwcZ0uMs5U4jGS7dSwXYkabnXlpUBsX0FYvDzXRtnzTw+00D29fuqE25 h7XVCZVRqJ2J+baamtkNFt9izXQ4c4n3taUBhh7LE+KDCzeGgfn2sDWJMQf14OjOciN0dtyC utPqvaxCs+lrZRv5ESXPbA0iLGbsSDWncv39ilbhz3tjqRQObIGXfTxiI9l4h/ki4WW8nhUu XOqKI6lkzTYHwvcqn9IjQXcP8h4YsyeL8+eaR8zUmYsCMflsprQDVpwIeXcQYXbCuiXPAemP l/nwqHBxhvXAdPbi3sdgcAAAAAAAAAA= --------------ms050708040103040502030205--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54684f63-81c4-053c-7253-8aa07433e8fa>