From owner-freebsd-fs@freebsd.org Sun Apr 24 21:00:29 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65A30B1B194 for ; Sun, 24 Apr 2016 21:00:29 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 44B391462 for ; Sun, 24 Apr 2016 21:00:29 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u3OL01ap027824 for ; Sun, 24 Apr 2016 21:00:29 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201604242100.u3OL01ap027824@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention Date: Sun, 24 Apr 2016 21:00:29 +0000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Apr 2016 21:00:29 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f 4 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Apr 25 04:59:27 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59B06B1BE9F for ; Mon, 25 Apr 2016 04:59:27 +0000 (UTC) (envelope-from ike@michaeleichorn.com) Received: from mx1.eichornenterprises.com (mx1.eichornenterprises.com [104.236.13.122]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.eichornenterprises.com", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 0631611F6 for ; Mon, 25 Apr 2016 04:59:26 +0000 (UTC) (envelope-from ike@michaeleichorn.com) Received: from smtp.eichornenterprises.com (cpe-184-59-147-149.neo.res.rr.com [184.59.147.149]) by mx1.eichornenterprises.com (OpenSMTPD) with ESMTP id 561c2ebd for ; Mon, 25 Apr 2016 00:59:23 -0400 (EDT) Received: by smtp.eichornenterprises.com (OpenSMTPD) with ESMTPSA id aaaa6420 TLS version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO for ; Mon, 25 Apr 2016 00:59:22 -0400 (EDT) Message-ID: <1461560445.22294.53.camel@michaeleichorn.com> Subject: GELI + Zpool Scrub Results in GELI Device Destruction (and Later a Corrupt Pool) From: "Michael B. Eichorn" To: freebsd-fs@freebsd.org Date: Mon, 25 Apr 2016 01:00:45 -0400 Content-Type: multipart/signed; micalg="sha-256"; protocol="application/x-pkcs7-signature"; boundary="=-L6MSVWIZbTNx6ORYtI3v" X-Mailer: Evolution 3.18.5.2 Mime-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Apr 2016 04:59:27 -0000 --=-L6MSVWIZbTNx6ORYtI3v Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I just ran into something rather unexpected. I have a pool consisting of a mirrored pair of geli encrypted partitions on WD Red 3TB disks. The machine is running 10.3-RELEASE, the root zpool was setup with GELI encryption from the installer, the pool that is acting up was setup per the handbook. See the below timeline for what happened, tldr: zpool scrub destroyed the eli devices, my attempt to recreate the eli device earned me a ZFS-8000-8A critical error (corrupted data). All of the errors reported with zpool status -v are metadata and not regualar files, but as I now have permanent metadata errors I am looking for guidance as to: 1) Is it safe to keep running the pool as-is for a day or two or am I risking data corruption? 2) It would be much much faster to copy the data to another pool than recreate the pool and copy the data back, rather than restore from backups, am I looking at any potential data loss if I do this? 3) What infomation would be useful to generate for the PR, the error is reproducable so what should be tried before I nuke the pool? Thanks, Ike -- TIMELINE -- I had just noticed that I had failed to enable the zpool scrub periodic on this machine. So I began to run zpool scrub by hand. It succeeded for the root pool which is also geli encrypted, but when I ran it against my primary data pool I encountered: Apr 24 23:18:23 terra kernel: GEOM_ELI: Device ada3p1.eli destroyed. Apr 24 23:18:23 terra kernel: GEOM_ELI: Detached ada3p1.eli on last close. Apr 24 23:18:23 terra kernel: GEOM_ELI: Device ada2p1.eli destroyed. Apr 24 23:18:23 terra kernel: GEOM_ELI: Detached ada2p1.eli on last close. And the scrub failed to initialize (command never returned to the shell). I then performed a reboot, which suceeded and brought everything up as normal. I then attempted to scrub the pool again. This time I only lost one of the partitions: Apr 24 23:37:34 terra kernel: GEOM_ELI: Device ada2p1.eli destroyed. Apr 24 23:37:34 terra kernel: GEOM_ELI: Detached ada2p1.eli on last close. I then performed a geli attach and zpool online, which onlined the disk that was offline and offlined the disk that was online (EEEK!): Apr 24 23:38:28 terra kernel: GEOM_ELI: Device ada2p1.eli created. Apr 24 23:38:28 terra kernel: GEOM_ELI: Encryption: AES-XTS 256 Apr 24 23:38:28 terra kernel: GEOM_ELI:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Crypto= : hardware Apr 24 23:41:05 terra kernel: GEOM_ELI: Device ada3p1.eli destroyed. Apr 24 23:41:05 terra kernel: GEOM_ELI: Detached ada3p1.eli on last close. Apr 24 23:41:05 terra devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=3D5890893416839487107 vdev_guid=3D17504861086892353515'' Apr 24 23:41:05 terra ZFS: vdev state changed, pool_guid=3D5890893416839487107 vdev_guid=3D17504861086892353515 I immediately rebooted and both disks came back and resilvered, with permanent metadata errors -- END TIMELINE -- --=-L6MSVWIZbTNx6ORYtI3v Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Disposition: attachment; filename="smime.p7s" Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCEqAw ggYwMIIFGKADAgECAgMOXcYwDQYJKoZIhvcNAQELBQAwgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQK Ew1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWdu aW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFByaW1hcnkgSW50ZXJtZWRpYXRlIENsaWVu dCBDQTAeFw0xNTA2MTMyMDI0NDZaFw0xNjA2MTQwMDM1NTBaMEgxHzAdBgNVBAMMFmlrZUBtaWNo YWVsZWljaG9ybi5jb20xJTAjBgkqhkiG9w0BCQEWFmlrZUBtaWNoYWVsZWljaG9ybi5jb20wggEi MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDJVdWALPz5h2s5zUQGIJYl6Vp8FPtZNko8q/3s crCsxXJLprMaDdpnqTsmkbmEfKvsqPQE6HVOpGxVRTl/tCm+VvouW9eY9ITMigb1OnHdU13CKO0j drgeU1nHst0qxwsIofRD7nC4dakT6exnrVndlBmLrf/bLPh2qOM8YK5qKK6m33fE7AyYrwiYAWFT 3fERI7LakjaabrIoS/Y1rCdL5FaCTMOlRbZyduc8HkrgjT2JW+i4fVcKyGL5gExBJWfS3q1uGFaB ie6pYtl8lZPtvN0JSfibP003RBoLgzqHJKW91RL0qNeDjKZi/5nrlU398l9UoVvLLO3KxoPBXKCx AgMBAAGjggLcMIIC2DAJBgNVHRMEAjAAMAsGA1UdDwQEAwIEsDAdBgNVHSUEFjAUBggrBgEFBQcD AgYIKwYBBQUHAwQwHQYDVR0OBBYEFJZqarc6CcrOs6eAwOgrMznk5ZWWMB8GA1UdIwQYMBaAFFNy 7ZKc4NrLAVx8fpY1TvLUuFGCMCEGA1UdEQQaMBiBFmlrZUBtaWNoYWVsZWljaG9ybi5jb20wggFM BgNVHSAEggFDMIIBPzCCATsGCysGAQQBgbU3AQIDMIIBKjAuBggrBgEFBQcCARYiaHR0cDovL3d3 dy5zdGFydHNzbC5jb20vcG9saWN5LnBkZjCB9wYIKwYBBQUHAgIwgeowJxYgU3RhcnRDb20gQ2Vy dGlmaWNhdGlvbiBBdXRob3JpdHkwAwIBARqBvlRoaXMgY2VydGlmaWNhdGUgd2FzIGlzc3VlZCBh Y2NvcmRpbmcgdG8gdGhlIENsYXNzIDEgVmFsaWRhdGlvbiByZXF1aXJlbWVudHMgb2YgdGhlIFN0 YXJ0Q29tIENBIHBvbGljeSwgcmVsaWFuY2Ugb25seSBmb3IgdGhlIGludGVuZGVkIHB1cnBvc2Ug aW4gY29tcGxpYW5jZSBvZiB0aGUgcmVseWluZyBwYXJ0eSBvYmxpZ2F0aW9ucy4wNgYDVR0fBC8w LTAroCmgJ4YlaHR0cDovL2NybC5zdGFydHNzbC5jb20vY3J0dTEtY3JsLmNybDCBjgYIKwYBBQUH AQEEgYEwfzA5BggrBgEFBQcwAYYtaHR0cDovL29jc3Auc3RhcnRzc2wuY29tL3N1Yi9jbGFzczEv Y2xpZW50L2NhMEIGCCsGAQUFBzAChjZodHRwOi8vYWlhLnN0YXJ0c3NsLmNvbS9jZXJ0cy9zdWIu Y2xhc3MxLmNsaWVudC5jYS5jcnQwIwYDVR0SBBwwGoYYaHR0cDovL3d3dy5zdGFydHNzbC5jb20v MA0GCSqGSIb3DQEBCwUAA4IBAQB4K8iQw+0FRn3xEnB3vIIu2Vi4C3ZGnOMWP90FFXLrZ6uAu9AK xVCjXUVP6nAEsOopTMu769vVecdBvg0KO2i5aTDTdTLX4g9d020g4OLWW1NiynAkX8oKqJLqZ53q vHK4zP4KWPS3bSqDWVCosTMfI+H6tkg+6G3gS0HHoHTLKZhIT3z6PQZAfeofM7ed6NOdAcj0J2lP ODHzzz7Y9x4wMwYJdidorzUDVYkNIkim8ak7hK9F60NadA5w/BirFATSlzRyV0h1tl6oNisEaQcq tGvy6UoCTDhzaJ7pQValfDXJ/A47P0hNj/CX/PmkY1wQHsEJz2pbh5lqteP/fO0rMIIGMDCCBRig AwIBAgIDDl3GMA0GCSqGSIb3DQEBCwUAMIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRD b20gTHRkLjErMCkGA1UECxMiU2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYG A1UEAxMvU3RhcnRDb20gQ2xhc3MgMSBQcmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0EwHhcN MTUwNjEzMjAyNDQ2WhcNMTYwNjE0MDAzNTUwWjBIMR8wHQYDVQQDDBZpa2VAbWljaGFlbGVpY2hv cm4uY29tMSUwIwYJKoZIhvcNAQkBFhZpa2VAbWljaGFlbGVpY2hvcm4uY29tMIIBIjANBgkqhkiG 9w0BAQEFAAOCAQ8AMIIBCgKCAQEAyVXVgCz8+YdrOc1EBiCWJelafBT7WTZKPKv97HKwrMVyS6az Gg3aZ6k7JpG5hHyr7Kj0BOh1TqRsVUU5f7Qpvlb6LlvXmPSEzIoG9Tpx3VNdwijtI3a4HlNZx7Ld KscLCKH0Q+5wuHWpE+nsZ61Z3ZQZi63/2yz4dqjjPGCuaiiupt93xOwMmK8ImAFhU93xESOy2pI2 mm6yKEv2NawnS+RWgkzDpUW2cnbnPB5K4I09iVvouH1XCshi+YBMQSVn0t6tbhhWgYnuqWLZfJWT 7bzdCUn4mz9NN0QaC4M6hySlvdUS9KjXg4ymYv+Z65VN/fJfVKFbyyztysaDwVygsQIDAQABo4IC 3DCCAtgwCQYDVR0TBAIwADALBgNVHQ8EBAMCBLAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUF BwMEMB0GA1UdDgQWBBSWamq3OgnKzrOngMDoKzM55OWVljAfBgNVHSMEGDAWgBRTcu2SnODaywFc fH6WNU7y1LhRgjAhBgNVHREEGjAYgRZpa2VAbWljaGFlbGVpY2hvcm4uY29tMIIBTAYDVR0gBIIB QzCCAT8wggE7BgsrBgEEAYG1NwECAzCCASowLgYIKwYBBQUHAgEWImh0dHA6Ly93d3cuc3RhcnRz c2wuY29tL3BvbGljeS5wZGYwgfcGCCsGAQUFBwICMIHqMCcWIFN0YXJ0Q29tIENlcnRpZmljYXRp b24gQXV0aG9yaXR5MAMCAQEagb5UaGlzIGNlcnRpZmljYXRlIHdhcyBpc3N1ZWQgYWNjb3JkaW5n IHRvIHRoZSBDbGFzcyAxIFZhbGlkYXRpb24gcmVxdWlyZW1lbnRzIG9mIHRoZSBTdGFydENvbSBD QSBwb2xpY3ksIHJlbGlhbmNlIG9ubHkgZm9yIHRoZSBpbnRlbmRlZCBwdXJwb3NlIGluIGNvbXBs aWFuY2Ugb2YgdGhlIHJlbHlpbmcgcGFydHkgb2JsaWdhdGlvbnMuMDYGA1UdHwQvMC0wK6ApoCeG JWh0dHA6Ly9jcmwuc3RhcnRzc2wuY29tL2NydHUxLWNybC5jcmwwgY4GCCsGAQUFBwEBBIGBMH8w OQYIKwYBBQUHMAGGLWh0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbS9zdWIvY2xhc3MxL2NsaWVudC9j YTBCBggrBgEFBQcwAoY2aHR0cDovL2FpYS5zdGFydHNzbC5jb20vY2VydHMvc3ViLmNsYXNzMS5j bGllbnQuY2EuY3J0MCMGA1UdEgQcMBqGGGh0dHA6Ly93d3cuc3RhcnRzc2wuY29tLzANBgkqhkiG 9w0BAQsFAAOCAQEAeCvIkMPtBUZ98RJwd7yCLtlYuAt2RpzjFj/dBRVy62ergLvQCsVQo11FT+pw BLDqKUzLu+vb1XnHQb4NCjtouWkw03Uy1+IPXdNtIODi1ltTYspwJF/KCqiS6med6rxyuMz+Clj0 t20qg1lQqLEzHyPh+rZIPuht4EtBx6B0yymYSE98+j0GQH3qHzO3nejTnQHI9CdpTzgx888+2Pce MDMGCXYnaK81A1WJDSJIpvGpO4SvRetDWnQOcPwYqxQE0pc0cldIdbZeqDYrBGkHKrRr8ulKAkw4 c2ie6UFWpXw1yfwOOz9ITY/wl/z5pGNcEB7BCc9qW4eZarXj/3ztKzCCBjQwggQcoAMCAQICAR4w DQYJKoZIhvcNAQEFBQAwfTELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzAp BgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcxKTAnBgNVBAMTIFN0YXJ0 Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTA3MTAyNDIxMDE1NVoXDTE3MTAyNDIxMDE1 NVowgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1 cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAx IFByaW1hcnkgSW50ZXJtZWRpYXRlIENsaWVudCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC AQoCggEBAMcJg8zOLdgasSmkLhOrlr6KMoOMpohBllVHrdRvEg/q6r8jR+EK75xCGhR8ToREoqe7 zM9/UnC6TS2y9UKTpT1v7RSMzR0t6ndl0TWBuUr/UXBhPk+Kmy7bI4yW4urC+y7P3/1/X7U8ocb8 VpH/Clt+4iq7nirMcNh6qJR+xjOhV+VHzQMALuGYn5KZmc1NbJQYclsGkDxDz2UbFqE2+6vIZoL+ jb9x4Pa5gNf1TwSDkOkikZB1xtB4ZqtXThaABSONdfmv/Z1pua3FYxnCFmdr/+N2JLKutIxMYqQO Jebr/f/h5t95m4JgrM3Y/w7YX9d7YAL9jvN4SydHsU6n65cCAwEAAaOCAa0wggGpMA8GA1UdEwEB /wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBRTcu2SnODaywFcfH6WNU7y1LhRgjAf BgNVHSMEGDAWgBROC+8apEBbpRdphzDKNGhD0EGu8jBmBggrBgEFBQcBAQRaMFgwJwYIKwYBBQUH MAGGG2h0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbS9jYTAtBggrBgEFBQcwAoYhaHR0cDovL3d3dy5z dGFydHNzbC5jb20vc2ZzY2EuY3J0MFsGA1UdHwRUMFIwJ6AloCOGIWh0dHA6Ly93d3cuc3RhcnRz c2wuY29tL3Nmc2NhLmNybDAnoCWgI4YhaHR0cDovL2NybC5zdGFydHNzbC5jb20vc2ZzY2EuY3Js MIGABgNVHSAEeTB3MHUGCysGAQQBgbU3AQIBMGYwLgYIKwYBBQUHAgEWImh0dHA6Ly93d3cuc3Rh cnRzc2wuY29tL3BvbGljeS5wZGYwNAYIKwYBBQUHAgEWKGh0dHA6Ly93d3cuc3RhcnRzc2wuY29t L2ludGVybWVkaWF0ZS5wZGYwDQYJKoZIhvcNAQEFBQADggIBAAqDCH14qywGXLhjjF6uHLkjd02h cdh9hrw+VUsv+q1eeQWB21jWj3kJ96AUlPCoEGZ/ynJNScWy6QMVQjbbMXltUfO4n4bGGdKo3awP Wp61tjAFgraLJgDk+DsSvUD6EowjMTNx25GQgyYJ5RPIzKKR9tQW8gGK+2+RHxkUCTbYFnL6kl8C h507rUdPPipJ9CgJFws3kDS3gOS5WFMxcjO5DwKfKSETEPrHh7p5shuuNktvsv6hxHTLhiMKX893 gxdT3XLS9OKmCv87vkINQcNEcIIoFWbP9HORz9v3vQwR4e3ksLc2JZOAFK+ssS5XMEoznzpihEP0 PLc4dCBYjbvSD7kxgDwZ+Aj8Q9PkbvE9sIPP7ON0fz095HdThKjiVJe6vofq+n6b1NBc8XdrQvBm unwxD5nvtTW4vtN6VY7mUCmxsCieuoBJ9OlqmsVWQvifIYf40dJPZkk9YgGTzWLpXDSfLSplbY2L L9C9U0ptvjcDjefLTvqSFc7tw1sEhF0n/qpA2r0GpvkLRDmcSwVyPvmjFBGqUp/pNy8ZuPGQmHwF i2/14+xeSUDG2bwnsYJQG2EdJCB6luQ57GEnTA/yKZSTKI8dDQa8Sd3zfXb19mOgSF0bBdXbuKhE puP9wirslFe6fQ1t5j5R0xi72MZ8ikMu1RQZKCyDbMwazlHiMYIDfzCCA3sCAQEwgZQwgYwxCzAJ BgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRh bCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFByaW1hcnkg SW50ZXJtZWRpYXRlIENsaWVudCBDQQIDDl3GMA0GCWCGSAFlAwQCAQUAoIIBuzAYBgkqhkiG9w0B CQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjA0MjUwNTAwNDVaMC8GCSqGSIb3DQEJ BDEiBCDNkeTPOwqD0UdCFu2QKuxc8qTU1WVbbF52/gpsGWR8ljCBpQYJKwYBBAGCNxAEMYGXMIGU MIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMiU2VjdXJl IERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UEAxMvU3RhcnRDb20gQ2xhc3MgMSBQ cmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0ECAw5dxjCBpwYLKoZIhvcNAQkQAgsxgZeggZQw gYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUg RGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFBy aW1hcnkgSW50ZXJtZWRpYXRlIENsaWVudCBDQQIDDl3GMA0GCSqGSIb3DQEBAQUABIIBAGO9knHD MeDoF2cDumaEAKRGKsjTeMyu6CuT7xekkl9jvbkVMLy5+P2Y7dM57Wa9WScAgnYbqrz5CilxCIH/ cUHXPWcRUkDfk0+wicTt36Bilpm2EcRfX7Gx6rf+qr7QwkRo1HaFPx7xiaN26C8MaV6SGCUuXIjl /kjyV/kolggar25Sbe957ebtigGAGzlOdBi0fziQ2R022GxdFfjLq6N3y7gnCmY921XfMUy3uRic 6sp+2kDzuS4GWRbKqngcUX9IkH7z9MwNQquUuiwC4iTDiexcX1AgQhDAJRpN/IKfF41yBTDdL7m3 8YuK5FrWNn0AF9C50hXwsqUN+0ATVxgAAAAAAAA= --=-L6MSVWIZbTNx6ORYtI3v-- From owner-freebsd-fs@freebsd.org Mon Apr 25 08:08:05 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CE71BB11055 for ; Mon, 25 Apr 2016 08:08:05 +0000 (UTC) (envelope-from maciej@suszko.eu) Received: from archeo.suszko.eu (archeo.unixguru.pl [IPv6:2001:41d0:2:8316::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 81B721BA1 for ; Mon, 25 Apr 2016 08:08:05 +0000 (UTC) (envelope-from maciej@suszko.eu) Received: from archeo (localhost [127.0.0.1]) by archeo.suszko.eu (Postfix) with ESMTP id E1C73D877; Mon, 25 Apr 2016 10:08:02 +0200 (CEST) X-Virus-Scanned: amavisd-new at archeo.local Received: from archeo.suszko.eu ([127.0.0.1]) by archeo (archeo.local [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id MgPOy5eWXIVR; Mon, 25 Apr 2016 10:08:02 +0200 (CEST) Received: from helium (gate.grtech.pl [195.8.99.234]) by archeo.suszko.eu (Postfix) with ESMTPSA id 55AFFD86F; Mon, 25 Apr 2016 10:08:02 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=suszko.eu; s=dkim; t=1461571682; bh=OyYt0UoCh0yjfC7u1JbpS2uzbZ0tDerCPzATfsEw2Sw=; h=Date:From:To:Cc:Subject:In-Reply-To:References; b=VveY4szsB+03gjLX6WbV5JuM3erh6N7yVuz5wjHP7kVYVE3RWR0F01UWy+6We5SmT 0BdNA6HkEJwwcCUvw2dU6FRR4PrY3KMOKAycufAO4rZv+Iwb1oXkfYwLfuiRxZJjBa mZt0ThkO53Tbv5TCSnDvEfYuWwrt7rLxH1K/g6g0= Date: Mon, 25 Apr 2016 10:07:54 +0200 From: Maciej Suszko To: "Michael B. Eichorn" Cc: freebsd-fs@freebsd.org Subject: Re: GELI + Zpool Scrub Results in GELI Device Destruction (and Later a Corrupt Pool) Message-ID: <20160425100754.0db9cd2b@helium> In-Reply-To: <1461560445.22294.53.camel@michaeleichorn.com> References: <1461560445.22294.53.camel@michaeleichorn.com> X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.29; amd64-portbld-freebsd10.3) MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/mIKl+VjPI0/0u3AGJVrMF_f"; protocol="application/pgp-signature" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Apr 2016 08:08:05 -0000 --Sig_/mIKl+VjPI0/0u3AGJVrMF_f Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Mon, 25 Apr 2016 01:00:45 -0400 "Michael B. Eichorn" wrote: > I just ran into something rather unexpected. I have a pool consisting > of a mirrored pair of geli encrypted partitions on WD Red 3TB disks. >=20 > The machine is running 10.3-RELEASE, the root zpool was setup with > GELI encryption from the installer, the pool that is acting up was > setup per the handbook. >=20 > See the below timeline for what happened, tldr: zpool scrub destroyed > the eli devices, my attempt to recreate the eli device earned me a > ZFS-8000-8A critical error (corrupted data). >=20 > All of the errors reported with zpool status -v are metadata and not > regualar files, but as I now have permanent metadata errors I am > looking for guidance as to: >=20 > 1) Is it safe to keep running the pool as-is for a day or two or am I > risking data corruption? >=20 > 2) It would be much much faster to copy the data to another pool than > recreate the pool and copy the data back, rather than restore from > backups, am I looking at any potential data loss if I do this? >=20 > 3) What infomation would be useful to generate for the PR, the error > is reproducable so what should be tried before I nuke the pool? >=20 > Thanks, > Ike >=20 > -- TIMELINE -- >=20 > I had just noticed that I had failed to enable the zpool scrub > periodic on this machine. So I began to run zpool scrub by hand. It > succeeded for the root pool which is also geli encrypted, but when I > ran it against my primary data pool I encountered: >=20 > Apr 24 23:18:23 terra kernel: GEOM_ELI: Device ada3p1.eli destroyed. > Apr 24 23:18:23 terra kernel: GEOM_ELI: Detached ada3p1.eli on last > close. > Apr 24 23:18:23 terra kernel: GEOM_ELI: Device ada2p1.eli destroyed. > Apr 24 23:18:23 terra kernel: GEOM_ELI: Detached ada2p1.eli on last > close. >=20 > And the scrub failed to initialize (command never returned to the > shell). >=20 > I then performed a reboot, which suceeded and brought everything up as > normal. I then attempted to scrub the pool again. This time I only > lost one of the partitions: >=20 > Apr 24 23:37:34 terra kernel: GEOM_ELI: Device ada2p1.eli destroyed. > Apr 24 23:37:34 terra kernel: GEOM_ELI: Detached ada2p1.eli on last > close. >=20 > I then performed a geli attach and zpool online, which onlined the > disk that was offline and offlined the disk that was online (EEEK!): >=20 > Apr 24 23:38:28 terra kernel: GEOM_ELI: Device ada2p1.eli created. > Apr 24 23:38:28 terra kernel: GEOM_ELI: Encryption: AES-XTS 256 > Apr 24 23:38:28 terra kernel: GEOM_ELI:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Cryp= to: hardware > Apr 24 23:41:05 terra kernel: GEOM_ELI: Device ada3p1.eli destroyed. > Apr 24 23:41:05 terra kernel: GEOM_ELI: Detached ada3p1.eli on last > close. > Apr 24 23:41:05 terra devd: Executing 'logger -p kern.notice -t ZFS > 'vdev state changed, pool_guid=3D5890893416839487107 > vdev_guid=3D17504861086892353515'' > Apr 24 23:41:05 terra ZFS: vdev state changed, > pool_guid=3D5890893416839487107 vdev_guid=3D17504861086892353515 >=20 > I immediately rebooted and both disks came back and resilvered, with > permanent metadata errors >=20 > -- END TIMELINE -- Hi, Configure your geli devices not to autodetach on last close... something like this in your rc.conf should work: geli_ada2p1_autodetach=3D"NO" geli_ada3p1_autodetach=3D"NO" --=20 regards, Maciej Suszko. --Sig_/mIKl+VjPI0/0u3AGJVrMF_f Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlcd0FsACgkQCikUk0l7iGr1BQCfV8P0qAceydOm3TV6USj1JsJ3 Sx0Anja9gq+xCxgBwW/kfW89etbMPeAX =3q/L -----END PGP SIGNATURE----- --Sig_/mIKl+VjPI0/0u3AGJVrMF_f-- From owner-freebsd-fs@freebsd.org Mon Apr 25 09:02:02 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 49408B1240F for ; Mon, 25 Apr 2016 09:02:02 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay05.ispgateway.de (smtprelay05.ispgateway.de [80.67.31.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 108351720 for ; Mon, 25 Apr 2016 09:02:01 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from [78.35.175.160] (helo=fabiankeil.de) by smtprelay05.ispgateway.de with esmtpsa (TLSv1.2:AES128-GCM-SHA256:128) (Exim 4.84) (envelope-from ) id 1aubca-0001Fa-1h for freebsd-fs@freebsd.org; Mon, 25 Apr 2016 10:11:56 +0200 Date: Mon, 25 Apr 2016 10:11:24 +0200 From: Fabian Keil To: freebsd-fs@freebsd.org Subject: Re: GELI + Zpool Scrub Results in GELI Device Destruction (and Later a Corrupt Pool) Message-ID: <20160425101124.068c8333@fabiankeil.de> In-Reply-To: <1461560445.22294.53.camel@michaeleichorn.com> References: <1461560445.22294.53.camel@michaeleichorn.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/eNw78W8gnK47rzOSmFw=BpX"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Apr 2016 09:02:02 -0000 --Sig_/eNw78W8gnK47rzOSmFw=BpX Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable "Michael B. Eichorn" wrote: > I just ran into something rather unexpected. I have a pool consisting > of a mirrored pair of geli encrypted partitions on WD Red 3TB disks. >=20 > The machine is running 10.3-RELEASE, the root zpool was setup with GELI > encryption from the installer, the pool that is acting up was setup per > the handbook. [...] > I had just noticed that I had failed to enable the zpool scrub periodic > on this machine. So I began to run zpool scrub by hand. It succeeded > for the root pool which is also geli encrypted, but when I ran it > against my primary data pool I encountered: >=20 > Apr 24 23:18:23 terra kernel: GEOM_ELI: Device ada3p1.eli destroyed. > Apr 24 23:18:23 terra kernel: GEOM_ELI: Detached ada3p1.eli on last > close. > Apr 24 23:18:23 terra kernel: GEOM_ELI: Device ada2p1.eli destroyed. > Apr 24 23:18:23 terra kernel: GEOM_ELI: Detached ada2p1.eli on last > close. Did you attach the devices using geli's -d (auto-detach) flag? If yes, this is a known issue: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D117158 > And the scrub failed to initialize (command never returned to the > shell). This could be the result of another known bug: https://lists.freebsd.org/pipermail/freebsd-current/2015-October/057988.html > I immediately rebooted and both disks came back and resilvered, with > permanent metadata errors Did those errors appear while resilvering or could they have been already present before? Fabian --Sig_/eNw78W8gnK47rzOSmFw=BpX Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlcd0SwACgkQBYqIVf93VJ0uoQCggfPmGuYbp2v6Bxow312Kw44D EYgAniwBPf6pKCMayM20gGQCIGGBqEF2 =Q75r -----END PGP SIGNATURE----- --Sig_/eNw78W8gnK47rzOSmFw=BpX-- From owner-freebsd-fs@freebsd.org Mon Apr 25 10:42:57 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E658EB1B981 for ; Mon, 25 Apr 2016 10:42:57 +0000 (UTC) (envelope-from woodsb02@gmail.com) Received: from mail-wm0-x231.google.com (mail-wm0-x231.google.com [IPv6:2a00:1450:400c:c09::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 765491384 for ; Mon, 25 Apr 2016 10:42:57 +0000 (UTC) (envelope-from woodsb02@gmail.com) Received: by mail-wm0-x231.google.com with SMTP id v188so92588959wme.1 for ; Mon, 25 Apr 2016 03:42:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=mv6xipL+fx/sojmnQEZhJJrHHlWM6r/hNNLEI9a2a0c=; b=unxEiIlmc8MMPpzx5C41ix11FTuLzla5hrA25UmrjPJ5lEdGsnAn5EqKdf+iQC97TD VLTlo6y/q3FVRGNZJo7fC1lLwMbbrtreCHp1g4viOPiHv6/mtk1xhRkXu07ShbvfGzVR 24XgPwa2p2JbN7DDqccBbVMC6uZtljKmq/FEzfp7T9pekYT324E3QPkeENZHRqSoZaa1 udr/v7OERprnW0buyCR0S3/6pEk0AkIqM0ygeqM+e6KIKS+YJ+Won83knedGssucIXeC ePPJTIdF7Wc2CZ5J6qxeZBd7AtqqQgzY+M0jJFx4CnStBpcuj7nh2NwFbDvtFXdyna79 sZ7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=mv6xipL+fx/sojmnQEZhJJrHHlWM6r/hNNLEI9a2a0c=; b=hCDhiMZYgvrGTuOm56LM2ddzCbOTwi+oyJ6bg7a2Ax90KzbkCgrBuP4wybT8DkVS1i diV/CcD0GKmGvJhpIN/5iWtTbpCqdTax76kCsXonavTdvT32jk3bGWKzI3tJKJ6cGY95 qkEQYDi8CjWxOHwFFNO8TBlboNb4prGrFZsXKxcvUBQh+UDqoRQ260s2h0Hqb6YC4GgY hKORbhIrbNoEo/Qhny6RdQYV+7JAE4xHDl6fdOpugFw3Z05GH8HYOgxxlIFVN0n9pWVA SQSZ6HdhCYAkErBaXBOI5OW+9immxwiqXzLJsctDWgiBiSTNPf0ccThXu2XzLA8QqcIF 6wqA== X-Gm-Message-State: AOPr4FWTbRB4eyDpeeA1nlTTbOJFEHSxlpOWcb0wMDFbL/ORIL4s0FDgwHr54lg66rd5A1lTAV64Mwwy0Akl0g== MIME-Version: 1.0 X-Received: by 10.28.32.199 with SMTP id g190mr11984920wmg.62.1461580975182; Mon, 25 Apr 2016 03:42:55 -0700 (PDT) Received: by 10.194.42.41 with HTTP; Mon, 25 Apr 2016 03:42:55 -0700 (PDT) In-Reply-To: <20160425100754.0db9cd2b@helium> References: <1461560445.22294.53.camel@michaeleichorn.com> <20160425100754.0db9cd2b@helium> Date: Mon, 25 Apr 2016 12:42:55 +0200 Message-ID: Subject: Re: GELI + Zpool Scrub Results in GELI Device Destruction (and Later a Corrupt Pool) From: Ben Woods To: Maciej Suszko Cc: "Michael B. Eichorn" , freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Apr 2016 10:42:58 -0000 On 25 April 2016 at 10:07, Maciej Suszko wrote: > Hi, > > Configure your geli devices not to autodetach on last close... > something like this in your rc.conf should work: > > geli_ada2p1_autodetach="NO" > geli_ada3p1_autodetach="NO" > -- > regards, Maciej Suszko. > It is also possible to do this for all drives, rather than having to specify for each drive: geli_autodetach="NO" https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=117158 Regards, Ben From owner-freebsd-fs@freebsd.org Mon Apr 25 12:27:23 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5968AB1B7B3 for ; Mon, 25 Apr 2016 12:27:23 +0000 (UTC) (envelope-from ike@michaeleichorn.com) Received: from mx1.eichornenterprises.com (mx1.eichornenterprises.com [104.236.13.122]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.eichornenterprises.com", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 0501618EA for ; Mon, 25 Apr 2016 12:27:22 +0000 (UTC) (envelope-from ike@michaeleichorn.com) Received: from smtp.eichornenterprises.com (cpe-184-59-147-149.neo.res.rr.com [184.59.147.149]) by mx1.eichornenterprises.com (OpenSMTPD) with ESMTP id 6a9aac5f; Mon, 25 Apr 2016 08:27:18 -0400 (EDT) Received: by smtp.eichornenterprises.com (OpenSMTPD) with ESMTPSA id b9b8314d TLS version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO; Mon, 25 Apr 2016 08:27:18 -0400 (EDT) Message-ID: <1461587321.22294.85.camel@michaeleichorn.com> Subject: Re: GELI + Zpool Scrub Results in GELI Device Destruction (and Later a Corrupt Pool) From: "Michael B. Eichorn" To: Fabian Keil , freebsd-fs@freebsd.org Date: Mon, 25 Apr 2016 08:28:41 -0400 In-Reply-To: <20160425101124.068c8333@fabiankeil.de> References: <1461560445.22294.53.camel@michaeleichorn.com> <20160425101124.068c8333@fabiankeil.de> Content-Type: multipart/signed; micalg="sha-256"; protocol="application/x-pkcs7-signature"; boundary="=-wepH41RJb3WWeJYSiD0q" X-Mailer: Evolution 3.18.5.2 Mime-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Apr 2016 12:27:23 -0000 --=-wepH41RJb3WWeJYSiD0q Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, 2016-04-25 at 10:11 +0200, Fabian Keil wrote: > "Michael B. Eichorn" wrote: >=20 > >=20 > > I just ran into something rather unexpected. I have a pool > > consisting > > of a mirrored pair of geli encrypted partitions on WD Red 3TB > > disks. > >=20 > > The machine is running 10.3-RELEASE, the root zpool was setup with > > GELI > > encryption from the installer, the pool that is acting up was setup > > per > > the handbook. > [...] > >=20 > > I had just noticed that I had failed to enable the zpool scrub > > periodic > > on this machine. So I began to run zpool scrub by hand. It > > succeeded > > for the root pool which is also geli encrypted, but when I ran it > > against my primary data pool I encountered: > >=20 > > Apr 24 23:18:23 terra kernel: GEOM_ELI: Device ada3p1.eli > > destroyed. > > Apr 24 23:18:23 terra kernel: GEOM_ELI: Detached ada3p1.eli on last > > close. > > Apr 24 23:18:23 terra kernel: GEOM_ELI: Device ada2p1.eli > > destroyed. > > Apr 24 23:18:23 terra kernel: GEOM_ELI: Detached ada2p1.eli on last > > close. > Did you attach the devices using geli's -d (auto-detach) flag? >=20 >=20 I am using whatever the default setup comes out of the rc.d scripts. My rc.conf was: geli_devices=3D"ada2p1 ada3p1" geli_default_flags=3D"-k /root/encryption.key" zfs_enable=3D"YES" I will try adding geli_autodetach=3D"NO" and scubbing in about 9 hours. > If yes, this is a known issue: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D117158 >=20 Reading that bug in detail it appears to be *specifically* for the kernel panic and that zfs closing and reopening providers is expected behavior, and that if geli has autodetach configured that it would detach. It stikes me that even though this is expected behavior it should not be. Is there a way we could prevent the detach when zfs does closes and reopens providers? I cannnot think of a case where the desired behavior is for the pool to detach when zfs wants to reopen it. > >=20 > > And the scrub failed to initialize (command never returned to the > > shell). > This could be the result of another known bug: > https://lists.freebsd.org/pipermail/freebsd-current/2015-October/0579 > 88.html >=20 > >=20 > > I immediately rebooted and both disks came back and resilvered, > > with > > permanent metadata errors > Did those errors appear while resilvering or could they have been > already present before? I do not think they were presesent before the disks flip-flopped. There was no error before my attempt to resliver. I would expect metadata errors as I effectively had: disk 1 online disk 2 offline then immediately without a resliver: disk 1 offline disk 1 online > Fabian --=-wepH41RJb3WWeJYSiD0q Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Disposition: attachment; filename="smime.p7s" Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCEqAw ggYwMIIFGKADAgECAgMOXcYwDQYJKoZIhvcNAQELBQAwgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQK Ew1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWdu aW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFByaW1hcnkgSW50ZXJtZWRpYXRlIENsaWVu dCBDQTAeFw0xNTA2MTMyMDI0NDZaFw0xNjA2MTQwMDM1NTBaMEgxHzAdBgNVBAMMFmlrZUBtaWNo YWVsZWljaG9ybi5jb20xJTAjBgkqhkiG9w0BCQEWFmlrZUBtaWNoYWVsZWljaG9ybi5jb20wggEi MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDJVdWALPz5h2s5zUQGIJYl6Vp8FPtZNko8q/3s crCsxXJLprMaDdpnqTsmkbmEfKvsqPQE6HVOpGxVRTl/tCm+VvouW9eY9ITMigb1OnHdU13CKO0j drgeU1nHst0qxwsIofRD7nC4dakT6exnrVndlBmLrf/bLPh2qOM8YK5qKK6m33fE7AyYrwiYAWFT 3fERI7LakjaabrIoS/Y1rCdL5FaCTMOlRbZyduc8HkrgjT2JW+i4fVcKyGL5gExBJWfS3q1uGFaB ie6pYtl8lZPtvN0JSfibP003RBoLgzqHJKW91RL0qNeDjKZi/5nrlU398l9UoVvLLO3KxoPBXKCx AgMBAAGjggLcMIIC2DAJBgNVHRMEAjAAMAsGA1UdDwQEAwIEsDAdBgNVHSUEFjAUBggrBgEFBQcD AgYIKwYBBQUHAwQwHQYDVR0OBBYEFJZqarc6CcrOs6eAwOgrMznk5ZWWMB8GA1UdIwQYMBaAFFNy 7ZKc4NrLAVx8fpY1TvLUuFGCMCEGA1UdEQQaMBiBFmlrZUBtaWNoYWVsZWljaG9ybi5jb20wggFM BgNVHSAEggFDMIIBPzCCATsGCysGAQQBgbU3AQIDMIIBKjAuBggrBgEFBQcCARYiaHR0cDovL3d3 dy5zdGFydHNzbC5jb20vcG9saWN5LnBkZjCB9wYIKwYBBQUHAgIwgeowJxYgU3RhcnRDb20gQ2Vy dGlmaWNhdGlvbiBBdXRob3JpdHkwAwIBARqBvlRoaXMgY2VydGlmaWNhdGUgd2FzIGlzc3VlZCBh Y2NvcmRpbmcgdG8gdGhlIENsYXNzIDEgVmFsaWRhdGlvbiByZXF1aXJlbWVudHMgb2YgdGhlIFN0 YXJ0Q29tIENBIHBvbGljeSwgcmVsaWFuY2Ugb25seSBmb3IgdGhlIGludGVuZGVkIHB1cnBvc2Ug aW4gY29tcGxpYW5jZSBvZiB0aGUgcmVseWluZyBwYXJ0eSBvYmxpZ2F0aW9ucy4wNgYDVR0fBC8w LTAroCmgJ4YlaHR0cDovL2NybC5zdGFydHNzbC5jb20vY3J0dTEtY3JsLmNybDCBjgYIKwYBBQUH AQEEgYEwfzA5BggrBgEFBQcwAYYtaHR0cDovL29jc3Auc3RhcnRzc2wuY29tL3N1Yi9jbGFzczEv Y2xpZW50L2NhMEIGCCsGAQUFBzAChjZodHRwOi8vYWlhLnN0YXJ0c3NsLmNvbS9jZXJ0cy9zdWIu Y2xhc3MxLmNsaWVudC5jYS5jcnQwIwYDVR0SBBwwGoYYaHR0cDovL3d3dy5zdGFydHNzbC5jb20v MA0GCSqGSIb3DQEBCwUAA4IBAQB4K8iQw+0FRn3xEnB3vIIu2Vi4C3ZGnOMWP90FFXLrZ6uAu9AK xVCjXUVP6nAEsOopTMu769vVecdBvg0KO2i5aTDTdTLX4g9d020g4OLWW1NiynAkX8oKqJLqZ53q vHK4zP4KWPS3bSqDWVCosTMfI+H6tkg+6G3gS0HHoHTLKZhIT3z6PQZAfeofM7ed6NOdAcj0J2lP ODHzzz7Y9x4wMwYJdidorzUDVYkNIkim8ak7hK9F60NadA5w/BirFATSlzRyV0h1tl6oNisEaQcq tGvy6UoCTDhzaJ7pQValfDXJ/A47P0hNj/CX/PmkY1wQHsEJz2pbh5lqteP/fO0rMIIGMDCCBRig AwIBAgIDDl3GMA0GCSqGSIb3DQEBCwUAMIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRD b20gTHRkLjErMCkGA1UECxMiU2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYG A1UEAxMvU3RhcnRDb20gQ2xhc3MgMSBQcmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0EwHhcN MTUwNjEzMjAyNDQ2WhcNMTYwNjE0MDAzNTUwWjBIMR8wHQYDVQQDDBZpa2VAbWljaGFlbGVpY2hv cm4uY29tMSUwIwYJKoZIhvcNAQkBFhZpa2VAbWljaGFlbGVpY2hvcm4uY29tMIIBIjANBgkqhkiG 9w0BAQEFAAOCAQ8AMIIBCgKCAQEAyVXVgCz8+YdrOc1EBiCWJelafBT7WTZKPKv97HKwrMVyS6az Gg3aZ6k7JpG5hHyr7Kj0BOh1TqRsVUU5f7Qpvlb6LlvXmPSEzIoG9Tpx3VNdwijtI3a4HlNZx7Ld KscLCKH0Q+5wuHWpE+nsZ61Z3ZQZi63/2yz4dqjjPGCuaiiupt93xOwMmK8ImAFhU93xESOy2pI2 mm6yKEv2NawnS+RWgkzDpUW2cnbnPB5K4I09iVvouH1XCshi+YBMQSVn0t6tbhhWgYnuqWLZfJWT 7bzdCUn4mz9NN0QaC4M6hySlvdUS9KjXg4ymYv+Z65VN/fJfVKFbyyztysaDwVygsQIDAQABo4IC 3DCCAtgwCQYDVR0TBAIwADALBgNVHQ8EBAMCBLAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUF BwMEMB0GA1UdDgQWBBSWamq3OgnKzrOngMDoKzM55OWVljAfBgNVHSMEGDAWgBRTcu2SnODaywFc fH6WNU7y1LhRgjAhBgNVHREEGjAYgRZpa2VAbWljaGFlbGVpY2hvcm4uY29tMIIBTAYDVR0gBIIB QzCCAT8wggE7BgsrBgEEAYG1NwECAzCCASowLgYIKwYBBQUHAgEWImh0dHA6Ly93d3cuc3RhcnRz c2wuY29tL3BvbGljeS5wZGYwgfcGCCsGAQUFBwICMIHqMCcWIFN0YXJ0Q29tIENlcnRpZmljYXRp b24gQXV0aG9yaXR5MAMCAQEagb5UaGlzIGNlcnRpZmljYXRlIHdhcyBpc3N1ZWQgYWNjb3JkaW5n IHRvIHRoZSBDbGFzcyAxIFZhbGlkYXRpb24gcmVxdWlyZW1lbnRzIG9mIHRoZSBTdGFydENvbSBD QSBwb2xpY3ksIHJlbGlhbmNlIG9ubHkgZm9yIHRoZSBpbnRlbmRlZCBwdXJwb3NlIGluIGNvbXBs aWFuY2Ugb2YgdGhlIHJlbHlpbmcgcGFydHkgb2JsaWdhdGlvbnMuMDYGA1UdHwQvMC0wK6ApoCeG JWh0dHA6Ly9jcmwuc3RhcnRzc2wuY29tL2NydHUxLWNybC5jcmwwgY4GCCsGAQUFBwEBBIGBMH8w OQYIKwYBBQUHMAGGLWh0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbS9zdWIvY2xhc3MxL2NsaWVudC9j YTBCBggrBgEFBQcwAoY2aHR0cDovL2FpYS5zdGFydHNzbC5jb20vY2VydHMvc3ViLmNsYXNzMS5j bGllbnQuY2EuY3J0MCMGA1UdEgQcMBqGGGh0dHA6Ly93d3cuc3RhcnRzc2wuY29tLzANBgkqhkiG 9w0BAQsFAAOCAQEAeCvIkMPtBUZ98RJwd7yCLtlYuAt2RpzjFj/dBRVy62ergLvQCsVQo11FT+pw BLDqKUzLu+vb1XnHQb4NCjtouWkw03Uy1+IPXdNtIODi1ltTYspwJF/KCqiS6med6rxyuMz+Clj0 t20qg1lQqLEzHyPh+rZIPuht4EtBx6B0yymYSE98+j0GQH3qHzO3nejTnQHI9CdpTzgx888+2Pce MDMGCXYnaK81A1WJDSJIpvGpO4SvRetDWnQOcPwYqxQE0pc0cldIdbZeqDYrBGkHKrRr8ulKAkw4 c2ie6UFWpXw1yfwOOz9ITY/wl/z5pGNcEB7BCc9qW4eZarXj/3ztKzCCBjQwggQcoAMCAQICAR4w DQYJKoZIhvcNAQEFBQAwfTELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzAp BgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcxKTAnBgNVBAMTIFN0YXJ0 Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTA3MTAyNDIxMDE1NVoXDTE3MTAyNDIxMDE1 NVowgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1 cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAx IFByaW1hcnkgSW50ZXJtZWRpYXRlIENsaWVudCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC AQoCggEBAMcJg8zOLdgasSmkLhOrlr6KMoOMpohBllVHrdRvEg/q6r8jR+EK75xCGhR8ToREoqe7 zM9/UnC6TS2y9UKTpT1v7RSMzR0t6ndl0TWBuUr/UXBhPk+Kmy7bI4yW4urC+y7P3/1/X7U8ocb8 VpH/Clt+4iq7nirMcNh6qJR+xjOhV+VHzQMALuGYn5KZmc1NbJQYclsGkDxDz2UbFqE2+6vIZoL+ jb9x4Pa5gNf1TwSDkOkikZB1xtB4ZqtXThaABSONdfmv/Z1pua3FYxnCFmdr/+N2JLKutIxMYqQO Jebr/f/h5t95m4JgrM3Y/w7YX9d7YAL9jvN4SydHsU6n65cCAwEAAaOCAa0wggGpMA8GA1UdEwEB /wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBRTcu2SnODaywFcfH6WNU7y1LhRgjAf BgNVHSMEGDAWgBROC+8apEBbpRdphzDKNGhD0EGu8jBmBggrBgEFBQcBAQRaMFgwJwYIKwYBBQUH MAGGG2h0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbS9jYTAtBggrBgEFBQcwAoYhaHR0cDovL3d3dy5z dGFydHNzbC5jb20vc2ZzY2EuY3J0MFsGA1UdHwRUMFIwJ6AloCOGIWh0dHA6Ly93d3cuc3RhcnRz c2wuY29tL3Nmc2NhLmNybDAnoCWgI4YhaHR0cDovL2NybC5zdGFydHNzbC5jb20vc2ZzY2EuY3Js MIGABgNVHSAEeTB3MHUGCysGAQQBgbU3AQIBMGYwLgYIKwYBBQUHAgEWImh0dHA6Ly93d3cuc3Rh cnRzc2wuY29tL3BvbGljeS5wZGYwNAYIKwYBBQUHAgEWKGh0dHA6Ly93d3cuc3RhcnRzc2wuY29t L2ludGVybWVkaWF0ZS5wZGYwDQYJKoZIhvcNAQEFBQADggIBAAqDCH14qywGXLhjjF6uHLkjd02h cdh9hrw+VUsv+q1eeQWB21jWj3kJ96AUlPCoEGZ/ynJNScWy6QMVQjbbMXltUfO4n4bGGdKo3awP Wp61tjAFgraLJgDk+DsSvUD6EowjMTNx25GQgyYJ5RPIzKKR9tQW8gGK+2+RHxkUCTbYFnL6kl8C h507rUdPPipJ9CgJFws3kDS3gOS5WFMxcjO5DwKfKSETEPrHh7p5shuuNktvsv6hxHTLhiMKX893 gxdT3XLS9OKmCv87vkINQcNEcIIoFWbP9HORz9v3vQwR4e3ksLc2JZOAFK+ssS5XMEoznzpihEP0 PLc4dCBYjbvSD7kxgDwZ+Aj8Q9PkbvE9sIPP7ON0fz095HdThKjiVJe6vofq+n6b1NBc8XdrQvBm unwxD5nvtTW4vtN6VY7mUCmxsCieuoBJ9OlqmsVWQvifIYf40dJPZkk9YgGTzWLpXDSfLSplbY2L L9C9U0ptvjcDjefLTvqSFc7tw1sEhF0n/qpA2r0GpvkLRDmcSwVyPvmjFBGqUp/pNy8ZuPGQmHwF i2/14+xeSUDG2bwnsYJQG2EdJCB6luQ57GEnTA/yKZSTKI8dDQa8Sd3zfXb19mOgSF0bBdXbuKhE puP9wirslFe6fQ1t5j5R0xi72MZ8ikMu1RQZKCyDbMwazlHiMYIDfzCCA3sCAQEwgZQwgYwxCzAJ BgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRh bCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFByaW1hcnkg SW50ZXJtZWRpYXRlIENsaWVudCBDQQIDDl3GMA0GCWCGSAFlAwQCAQUAoIIBuzAYBgkqhkiG9w0B CQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjA0MjUxMjI4NDFaMC8GCSqGSIb3DQEJ BDEiBCBMfl3zK+qCk4+hNVVhVdwiBZRHuxHOO8rxMx0CGJ0MITCBpQYJKwYBBAGCNxAEMYGXMIGU MIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMiU2VjdXJl IERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UEAxMvU3RhcnRDb20gQ2xhc3MgMSBQ cmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0ECAw5dxjCBpwYLKoZIhvcNAQkQAgsxgZeggZQw gYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUg RGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFBy aW1hcnkgSW50ZXJtZWRpYXRlIENsaWVudCBDQQIDDl3GMA0GCSqGSIb3DQEBAQUABIIBAHj/Uj+n h8fkMpX6YSf5TXCAaC32YUnttDTjE8WGRdJz8/IM3qpk1pWM7RK+mD8ompClc7Ta0xq7wyCwaGdK Lgg28rbjfdfanVqCFBFIv3dFvuCvx8D+KTpVV7bphwfnvvuW0i1OnOYjYIT9yXtg5dZ0EZB/flMI wZ5xf8BXPOCmIvoX9HbNTJaM5unbkKGfizSf6dW3xWlwxx1Tsal6dzoWIMdoxg6S5NeooNAm4Syh Jk/E5DajE+16/RZ0TOfz76V+i/w9P9UPS7FlkbGgVnpkn08MPvJ26NQAOq2rSGfROiSjqZLpnKlU 81JkusfCj4b3CdYm0UV7n8CzdfuBDEIAAAAAAAA= --=-wepH41RJb3WWeJYSiD0q-- From owner-freebsd-fs@freebsd.org Mon Apr 25 14:51:33 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BC535B1BF35 for ; Mon, 25 Apr 2016 14:51:33 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay03.ispgateway.de (smtprelay03.ispgateway.de [80.67.31.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7B3D419A2 for ; Mon, 25 Apr 2016 14:51:32 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from [78.35.175.160] (helo=fabiankeil.de) by smtprelay03.ispgateway.de with esmtpsa (TLSv1.2:AES128-GCM-SHA256:128) (Exim 4.84) (envelope-from ) id 1auhMQ-000185-9i for freebsd-fs@freebsd.org; Mon, 25 Apr 2016 16:19:38 +0200 Date: Mon, 25 Apr 2016 16:06:25 +0200 From: Fabian Keil To: freebsd-fs@freebsd.org Subject: Re: GELI + Zpool Scrub Results in GELI Device Destruction (and Later a Corrupt Pool) Message-ID: <20160425160625.5b585c8d@fabiankeil.de> In-Reply-To: <1461587321.22294.85.camel@michaeleichorn.com> References: <1461560445.22294.53.camel@michaeleichorn.com> <20160425101124.068c8333@fabiankeil.de> <1461587321.22294.85.camel@michaeleichorn.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/Awxbp/X4B=qp6AnH7BO5GJy"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Apr 2016 14:51:33 -0000 --Sig_/Awxbp/X4B=qp6AnH7BO5GJy Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable "Michael B. Eichorn" wrote: > On Mon, 2016-04-25 at 10:11 +0200, Fabian Keil wrote: > > "Michael B. Eichorn" wrote: > > =20 > > >=20 > > > I just ran into something rather unexpected. I have a pool > > > consisting > > > of a mirrored pair of geli encrypted partitions on WD Red 3TB > > > disks. > > >=20 > > > The machine is running 10.3-RELEASE, the root zpool was setup with > > > GELI > > > encryption from the installer, the pool that is acting up was setup > > > per > > > the handbook. =20 > > [...] =20 > > >=20 > > > I had just noticed that I had failed to enable the zpool scrub > > > periodic > > > on this machine. So I began to run zpool scrub by hand. It > > > succeeded > > > for the root pool which is also geli encrypted, but when I ran it > > > against my primary data pool I encountered: > > >=20 > > > Apr 24 23:18:23 terra kernel: GEOM_ELI: Device ada3p1.eli > > > destroyed. > > > Apr 24 23:18:23 terra kernel: GEOM_ELI: Detached ada3p1.eli on last > > > close. > > > Apr 24 23:18:23 terra kernel: GEOM_ELI: Device ada2p1.eli > > > destroyed. > > > Apr 24 23:18:23 terra kernel: GEOM_ELI: Detached ada2p1.eli on last > > > close. =20 > > Did you attach the devices using geli's -d (auto-detach) flag? >=20 > I am using whatever the default setup comes out of the rc.d scripts. > My rc.conf was: >=20 > geli_devices=3D"ada2p1 ada3p1" > geli_default_flags=3D"-k /root/encryption.key" > zfs_enable=3D"YES" >=20 > I will try adding geli_autodetach=3D"NO" and scubbing in about 9 hours. On FreeBSD the default (set in /etc/defaults/rc.conf) is YES. > > If yes, this is a known issue: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D117158 > > =20 >=20 > Reading that bug in detail it appears to be *specifically* for the > kernel panic and that zfs closing and reopening providers is expected > behavior, and that if geli has autodetach configured that it would > detach. >=20 > It stikes me that even though this is expected behavior it should not > be. Is there a way we could prevent the detach when zfs does closes and > reopens providers? I cannnot think of a case where the desired behavior > is for the pool to detach when zfs wants to reopen it. I suppose geli could delay the detachment a bit to give the consumer a chance to reopen it. Fabian --Sig_/Awxbp/X4B=qp6AnH7BO5GJy Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEUEARECAAYFAlceJGEACgkQBYqIVf93VJ0dIACgto8ufhTsp8yx2efZjeN5a0ze 3u0AmPLxV2QspWG57OTC1ADtc9/YZ0U= =7XHt -----END PGP SIGNATURE----- --Sig_/Awxbp/X4B=qp6AnH7BO5GJy-- From owner-freebsd-fs@freebsd.org Mon Apr 25 16:46:11 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 629FDB1CDE5 for ; Mon, 25 Apr 2016 16:46:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 52DDE1387 for ; Mon, 25 Apr 2016 16:46:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u3PGkB2J099849 for ; Mon, 25 Apr 2016 16:46:11 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 208882] zfs root filesystem mount failure on startup in FreeBSD 10.3-RELEASE if USB hdd with zpool is attached to another port Date: Mon, 25 Apr 2016 16:46:11 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: fk@fabiankeil.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Apr 2016 16:46:11 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D208882 --- Comment #5 from Fabian Keil --- No worries, I did not expect you to test the NFS mount retries. The freebsd-fs@ list is already CC'd, so an additional request to get the patch reviewed and committed should not be necessary. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Apr 25 22:49:48 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 26216B1C71F for ; Mon, 25 Apr 2016 22:49:48 +0000 (UTC) (envelope-from ike@michaeleichorn.com) Received: from mx1.eichornenterprises.com (mx1.eichornenterprises.com [104.236.13.122]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.eichornenterprises.com", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C12391C88 for ; Mon, 25 Apr 2016 22:49:47 +0000 (UTC) (envelope-from ike@michaeleichorn.com) Received: from smtp.eichornenterprises.com (cpe-184-59-147-149.neo.res.rr.com [184.59.147.149]) by mx1.eichornenterprises.com (OpenSMTPD) with ESMTP id 3bd52496; Mon, 25 Apr 2016 18:49:43 -0400 (EDT) Received: by smtp.eichornenterprises.com (OpenSMTPD) with ESMTPSA id 970058e3 TLS version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO; Mon, 25 Apr 2016 18:49:42 -0400 (EDT) Message-ID: <1461624665.22294.103.camel@michaeleichorn.com> Subject: Re: GELI + Zpool Scrub Results in GELI Device Destruction (and Later a Corrupt Pool) From: "Michael B. Eichorn" To: Fabian Keil , freebsd-fs@freebsd.org Date: Mon, 25 Apr 2016 18:51:05 -0400 In-Reply-To: <20160425160625.5b585c8d@fabiankeil.de> References: <1461560445.22294.53.camel@michaeleichorn.com> <20160425101124.068c8333@fabiankeil.de> <1461587321.22294.85.camel@michaeleichorn.com> <20160425160625.5b585c8d@fabiankeil.de> Content-Type: multipart/signed; micalg="sha-256"; protocol="application/x-pkcs7-signature"; boundary="=-wD5QTYVlVzzZDasXjPwC" X-Mailer: Evolution 3.18.5.2 Mime-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Apr 2016 22:49:48 -0000 --=-wD5QTYVlVzzZDasXjPwC Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, 2016-04-25 at 16:06 +0200, Fabian Keil wrote: > "Michael B. Eichorn" wrote: >=20 > >=20 > > On Mon, 2016-04-25 at 10:11 +0200, Fabian Keil wrote: > > >=20 > > > "Michael B. Eichorn" wrote: > > > =C2=A0=C2=A0 > > > >=20 > > > >=20 > > > > I just ran into something rather unexpected. I have a pool > > > > consisting > > > > of a mirrored pair of geli encrypted partitions on WD Red 3TB > > > > disks. > > > >=20 > > > > The machine is running 10.3-RELEASE, the root zpool was setup > > > > with > > > > GELI > > > > encryption from the installer, the pool that is acting up was > > > > setup > > > > per > > > > the handbook.=C2=A0=C2=A0 > > > [...]=C2=A0=C2=A0 > > > >=20 > > > >=20 > > > > I had just noticed that I had failed to enable the zpool scrub > > > > periodic > > > > on this machine. So I began to run zpool scrub by hand. It > > > > succeeded > > > > for the root pool which is also geli encrypted, but when I ran > > > > it > > > > against my primary data pool I encountered: > > > >=20 > > > > Apr 24 23:18:23 terra kernel: GEOM_ELI: Device ada3p1.eli > > > > destroyed. > > > > Apr 24 23:18:23 terra kernel: GEOM_ELI: Detached ada3p1.eli on > > > > last > > > > close. > > > > Apr 24 23:18:23 terra kernel: GEOM_ELI: Device ada2p1.eli > > > > destroyed. > > > > Apr 24 23:18:23 terra kernel: GEOM_ELI: Detached ada2p1.eli on > > > > last > > > > close.=C2=A0=C2=A0 > > > Did you attach the devices using geli's -d (auto-detach) flag? > > I am using whatever the default setup comes out of the rc.d > > scripts. > > My rc.conf was: > >=20 > > geli_devices=3D"ada2p1 ada3p1" > > geli_default_flags=3D"-k /root/encryption.key" > > zfs_enable=3D"YES" > >=20 > > I will try adding geli_autodetach=3D"NO" and scubbing in about 9 > > hours. > On FreeBSD the default (set in /etc/defaults/rc.conf) is YES. Ah, I forgot about that file. For the record geli_autodetach=3D"NO" did work properly. Interestingly, when I rebooted to test that rc.d modification when the pool came up it resilvered for a second time and now reports no errors at all. > > >=20 > > > If yes, this is a known issue: > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D117158 > > > =C2=A0=C2=A0 > > Reading that bug in detail it appears to be *specifically* for the > > kernel panic and that zfs closing and reopening providers is > > expected > > behavior, and that if geli has autodetach configured that it would > > detach. > >=20 > > It stikes me that even though this is expected behavior it should > > not > > be. Is there a way we could prevent the detach when zfs does closes > > and > > reopens providers? I cannnot think of a case where the desired > > behavior > > is for the pool to detach when zfs wants to reopen it. > I suppose geli could delay the detachment a bit to give the consumer > a chance to reopen it. That would probably work, but my inner engineer is dissatisfied with the idea of slowing down all detachments to solve a single case. What about a new feature whereby zfs (or any other consumer) can inhibit the autodetach for a brief period? I don't really know if this is feasible, but I thought I would ask. Anything like this is probably too major to implement without some thought and probably consensus building. So in the meantime I will file my bug against the handbook to make sure geli_autodetach and zfs are mentioned. --=-wD5QTYVlVzzZDasXjPwC Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Disposition: attachment; filename="smime.p7s" Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCEqAw ggYwMIIFGKADAgECAgMOXcYwDQYJKoZIhvcNAQELBQAwgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQK Ew1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWdu aW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFByaW1hcnkgSW50ZXJtZWRpYXRlIENsaWVu dCBDQTAeFw0xNTA2MTMyMDI0NDZaFw0xNjA2MTQwMDM1NTBaMEgxHzAdBgNVBAMMFmlrZUBtaWNo YWVsZWljaG9ybi5jb20xJTAjBgkqhkiG9w0BCQEWFmlrZUBtaWNoYWVsZWljaG9ybi5jb20wggEi MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDJVdWALPz5h2s5zUQGIJYl6Vp8FPtZNko8q/3s crCsxXJLprMaDdpnqTsmkbmEfKvsqPQE6HVOpGxVRTl/tCm+VvouW9eY9ITMigb1OnHdU13CKO0j drgeU1nHst0qxwsIofRD7nC4dakT6exnrVndlBmLrf/bLPh2qOM8YK5qKK6m33fE7AyYrwiYAWFT 3fERI7LakjaabrIoS/Y1rCdL5FaCTMOlRbZyduc8HkrgjT2JW+i4fVcKyGL5gExBJWfS3q1uGFaB ie6pYtl8lZPtvN0JSfibP003RBoLgzqHJKW91RL0qNeDjKZi/5nrlU398l9UoVvLLO3KxoPBXKCx AgMBAAGjggLcMIIC2DAJBgNVHRMEAjAAMAsGA1UdDwQEAwIEsDAdBgNVHSUEFjAUBggrBgEFBQcD AgYIKwYBBQUHAwQwHQYDVR0OBBYEFJZqarc6CcrOs6eAwOgrMznk5ZWWMB8GA1UdIwQYMBaAFFNy 7ZKc4NrLAVx8fpY1TvLUuFGCMCEGA1UdEQQaMBiBFmlrZUBtaWNoYWVsZWljaG9ybi5jb20wggFM BgNVHSAEggFDMIIBPzCCATsGCysGAQQBgbU3AQIDMIIBKjAuBggrBgEFBQcCARYiaHR0cDovL3d3 dy5zdGFydHNzbC5jb20vcG9saWN5LnBkZjCB9wYIKwYBBQUHAgIwgeowJxYgU3RhcnRDb20gQ2Vy dGlmaWNhdGlvbiBBdXRob3JpdHkwAwIBARqBvlRoaXMgY2VydGlmaWNhdGUgd2FzIGlzc3VlZCBh Y2NvcmRpbmcgdG8gdGhlIENsYXNzIDEgVmFsaWRhdGlvbiByZXF1aXJlbWVudHMgb2YgdGhlIFN0 YXJ0Q29tIENBIHBvbGljeSwgcmVsaWFuY2Ugb25seSBmb3IgdGhlIGludGVuZGVkIHB1cnBvc2Ug aW4gY29tcGxpYW5jZSBvZiB0aGUgcmVseWluZyBwYXJ0eSBvYmxpZ2F0aW9ucy4wNgYDVR0fBC8w LTAroCmgJ4YlaHR0cDovL2NybC5zdGFydHNzbC5jb20vY3J0dTEtY3JsLmNybDCBjgYIKwYBBQUH AQEEgYEwfzA5BggrBgEFBQcwAYYtaHR0cDovL29jc3Auc3RhcnRzc2wuY29tL3N1Yi9jbGFzczEv Y2xpZW50L2NhMEIGCCsGAQUFBzAChjZodHRwOi8vYWlhLnN0YXJ0c3NsLmNvbS9jZXJ0cy9zdWIu Y2xhc3MxLmNsaWVudC5jYS5jcnQwIwYDVR0SBBwwGoYYaHR0cDovL3d3dy5zdGFydHNzbC5jb20v MA0GCSqGSIb3DQEBCwUAA4IBAQB4K8iQw+0FRn3xEnB3vIIu2Vi4C3ZGnOMWP90FFXLrZ6uAu9AK xVCjXUVP6nAEsOopTMu769vVecdBvg0KO2i5aTDTdTLX4g9d020g4OLWW1NiynAkX8oKqJLqZ53q vHK4zP4KWPS3bSqDWVCosTMfI+H6tkg+6G3gS0HHoHTLKZhIT3z6PQZAfeofM7ed6NOdAcj0J2lP ODHzzz7Y9x4wMwYJdidorzUDVYkNIkim8ak7hK9F60NadA5w/BirFATSlzRyV0h1tl6oNisEaQcq tGvy6UoCTDhzaJ7pQValfDXJ/A47P0hNj/CX/PmkY1wQHsEJz2pbh5lqteP/fO0rMIIGMDCCBRig AwIBAgIDDl3GMA0GCSqGSIb3DQEBCwUAMIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRD b20gTHRkLjErMCkGA1UECxMiU2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYG A1UEAxMvU3RhcnRDb20gQ2xhc3MgMSBQcmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0EwHhcN MTUwNjEzMjAyNDQ2WhcNMTYwNjE0MDAzNTUwWjBIMR8wHQYDVQQDDBZpa2VAbWljaGFlbGVpY2hv cm4uY29tMSUwIwYJKoZIhvcNAQkBFhZpa2VAbWljaGFlbGVpY2hvcm4uY29tMIIBIjANBgkqhkiG 9w0BAQEFAAOCAQ8AMIIBCgKCAQEAyVXVgCz8+YdrOc1EBiCWJelafBT7WTZKPKv97HKwrMVyS6az Gg3aZ6k7JpG5hHyr7Kj0BOh1TqRsVUU5f7Qpvlb6LlvXmPSEzIoG9Tpx3VNdwijtI3a4HlNZx7Ld KscLCKH0Q+5wuHWpE+nsZ61Z3ZQZi63/2yz4dqjjPGCuaiiupt93xOwMmK8ImAFhU93xESOy2pI2 mm6yKEv2NawnS+RWgkzDpUW2cnbnPB5K4I09iVvouH1XCshi+YBMQSVn0t6tbhhWgYnuqWLZfJWT 7bzdCUn4mz9NN0QaC4M6hySlvdUS9KjXg4ymYv+Z65VN/fJfVKFbyyztysaDwVygsQIDAQABo4IC 3DCCAtgwCQYDVR0TBAIwADALBgNVHQ8EBAMCBLAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUF BwMEMB0GA1UdDgQWBBSWamq3OgnKzrOngMDoKzM55OWVljAfBgNVHSMEGDAWgBRTcu2SnODaywFc fH6WNU7y1LhRgjAhBgNVHREEGjAYgRZpa2VAbWljaGFlbGVpY2hvcm4uY29tMIIBTAYDVR0gBIIB QzCCAT8wggE7BgsrBgEEAYG1NwECAzCCASowLgYIKwYBBQUHAgEWImh0dHA6Ly93d3cuc3RhcnRz c2wuY29tL3BvbGljeS5wZGYwgfcGCCsGAQUFBwICMIHqMCcWIFN0YXJ0Q29tIENlcnRpZmljYXRp b24gQXV0aG9yaXR5MAMCAQEagb5UaGlzIGNlcnRpZmljYXRlIHdhcyBpc3N1ZWQgYWNjb3JkaW5n IHRvIHRoZSBDbGFzcyAxIFZhbGlkYXRpb24gcmVxdWlyZW1lbnRzIG9mIHRoZSBTdGFydENvbSBD QSBwb2xpY3ksIHJlbGlhbmNlIG9ubHkgZm9yIHRoZSBpbnRlbmRlZCBwdXJwb3NlIGluIGNvbXBs aWFuY2Ugb2YgdGhlIHJlbHlpbmcgcGFydHkgb2JsaWdhdGlvbnMuMDYGA1UdHwQvMC0wK6ApoCeG JWh0dHA6Ly9jcmwuc3RhcnRzc2wuY29tL2NydHUxLWNybC5jcmwwgY4GCCsGAQUFBwEBBIGBMH8w OQYIKwYBBQUHMAGGLWh0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbS9zdWIvY2xhc3MxL2NsaWVudC9j YTBCBggrBgEFBQcwAoY2aHR0cDovL2FpYS5zdGFydHNzbC5jb20vY2VydHMvc3ViLmNsYXNzMS5j bGllbnQuY2EuY3J0MCMGA1UdEgQcMBqGGGh0dHA6Ly93d3cuc3RhcnRzc2wuY29tLzANBgkqhkiG 9w0BAQsFAAOCAQEAeCvIkMPtBUZ98RJwd7yCLtlYuAt2RpzjFj/dBRVy62ergLvQCsVQo11FT+pw BLDqKUzLu+vb1XnHQb4NCjtouWkw03Uy1+IPXdNtIODi1ltTYspwJF/KCqiS6med6rxyuMz+Clj0 t20qg1lQqLEzHyPh+rZIPuht4EtBx6B0yymYSE98+j0GQH3qHzO3nejTnQHI9CdpTzgx888+2Pce MDMGCXYnaK81A1WJDSJIpvGpO4SvRetDWnQOcPwYqxQE0pc0cldIdbZeqDYrBGkHKrRr8ulKAkw4 c2ie6UFWpXw1yfwOOz9ITY/wl/z5pGNcEB7BCc9qW4eZarXj/3ztKzCCBjQwggQcoAMCAQICAR4w DQYJKoZIhvcNAQEFBQAwfTELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzAp BgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcxKTAnBgNVBAMTIFN0YXJ0 Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTA3MTAyNDIxMDE1NVoXDTE3MTAyNDIxMDE1 NVowgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1 cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAx IFByaW1hcnkgSW50ZXJtZWRpYXRlIENsaWVudCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC AQoCggEBAMcJg8zOLdgasSmkLhOrlr6KMoOMpohBllVHrdRvEg/q6r8jR+EK75xCGhR8ToREoqe7 zM9/UnC6TS2y9UKTpT1v7RSMzR0t6ndl0TWBuUr/UXBhPk+Kmy7bI4yW4urC+y7P3/1/X7U8ocb8 VpH/Clt+4iq7nirMcNh6qJR+xjOhV+VHzQMALuGYn5KZmc1NbJQYclsGkDxDz2UbFqE2+6vIZoL+ jb9x4Pa5gNf1TwSDkOkikZB1xtB4ZqtXThaABSONdfmv/Z1pua3FYxnCFmdr/+N2JLKutIxMYqQO Jebr/f/h5t95m4JgrM3Y/w7YX9d7YAL9jvN4SydHsU6n65cCAwEAAaOCAa0wggGpMA8GA1UdEwEB /wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBRTcu2SnODaywFcfH6WNU7y1LhRgjAf BgNVHSMEGDAWgBROC+8apEBbpRdphzDKNGhD0EGu8jBmBggrBgEFBQcBAQRaMFgwJwYIKwYBBQUH MAGGG2h0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbS9jYTAtBggrBgEFBQcwAoYhaHR0cDovL3d3dy5z dGFydHNzbC5jb20vc2ZzY2EuY3J0MFsGA1UdHwRUMFIwJ6AloCOGIWh0dHA6Ly93d3cuc3RhcnRz c2wuY29tL3Nmc2NhLmNybDAnoCWgI4YhaHR0cDovL2NybC5zdGFydHNzbC5jb20vc2ZzY2EuY3Js MIGABgNVHSAEeTB3MHUGCysGAQQBgbU3AQIBMGYwLgYIKwYBBQUHAgEWImh0dHA6Ly93d3cuc3Rh cnRzc2wuY29tL3BvbGljeS5wZGYwNAYIKwYBBQUHAgEWKGh0dHA6Ly93d3cuc3RhcnRzc2wuY29t L2ludGVybWVkaWF0ZS5wZGYwDQYJKoZIhvcNAQEFBQADggIBAAqDCH14qywGXLhjjF6uHLkjd02h cdh9hrw+VUsv+q1eeQWB21jWj3kJ96AUlPCoEGZ/ynJNScWy6QMVQjbbMXltUfO4n4bGGdKo3awP Wp61tjAFgraLJgDk+DsSvUD6EowjMTNx25GQgyYJ5RPIzKKR9tQW8gGK+2+RHxkUCTbYFnL6kl8C h507rUdPPipJ9CgJFws3kDS3gOS5WFMxcjO5DwKfKSETEPrHh7p5shuuNktvsv6hxHTLhiMKX893 gxdT3XLS9OKmCv87vkINQcNEcIIoFWbP9HORz9v3vQwR4e3ksLc2JZOAFK+ssS5XMEoznzpihEP0 PLc4dCBYjbvSD7kxgDwZ+Aj8Q9PkbvE9sIPP7ON0fz095HdThKjiVJe6vofq+n6b1NBc8XdrQvBm unwxD5nvtTW4vtN6VY7mUCmxsCieuoBJ9OlqmsVWQvifIYf40dJPZkk9YgGTzWLpXDSfLSplbY2L L9C9U0ptvjcDjefLTvqSFc7tw1sEhF0n/qpA2r0GpvkLRDmcSwVyPvmjFBGqUp/pNy8ZuPGQmHwF i2/14+xeSUDG2bwnsYJQG2EdJCB6luQ57GEnTA/yKZSTKI8dDQa8Sd3zfXb19mOgSF0bBdXbuKhE puP9wirslFe6fQ1t5j5R0xi72MZ8ikMu1RQZKCyDbMwazlHiMYIDfzCCA3sCAQEwgZQwgYwxCzAJ BgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRh bCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFByaW1hcnkg SW50ZXJtZWRpYXRlIENsaWVudCBDQQIDDl3GMA0GCWCGSAFlAwQCAQUAoIIBuzAYBgkqhkiG9w0B CQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjA0MjUyMjUxMDVaMC8GCSqGSIb3DQEJ BDEiBCDxC81i9hTa8veTypl0/Q/xpyCeN4onJyavcVToel1pRzCBpQYJKwYBBAGCNxAEMYGXMIGU MIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMiU2VjdXJl IERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UEAxMvU3RhcnRDb20gQ2xhc3MgMSBQ cmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0ECAw5dxjCBpwYLKoZIhvcNAQkQAgsxgZeggZQw gYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUg RGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFBy aW1hcnkgSW50ZXJtZWRpYXRlIENsaWVudCBDQQIDDl3GMA0GCSqGSIb3DQEBAQUABIIBALry4w1w VeBV8SogW6sBq36+BOo9/YoWHZx6c+oA7QnrZKKDu3hsbDtw8CS4Tw2/MxPIYFkcAHDMdhaI7Pme ZwvyWqUHt9PGF7rLpBy10PHrYuD/gf0P1JQrq+jnEeMJAOMso10EbZjoYv0K9PBZl6azAdK9T9dk g6yCqQTZWWfpWttYlbeVFNWU6Mo23c492TBLH95ajwXuo7LWYPThcrarVZo+vmY/W7zcM0j2pc5C APADwc/+FWPEI9d+2jITRVE8RPuF5EBwiKPtivER9xprTPdh7TqADw8kh1lrNaaeI5Q/TJ0Qjn/X LYrvuPv63NNNH+jRze7yg/j1JyqsVswAAAAAAAA= --=-wD5QTYVlVzzZDasXjPwC-- From owner-freebsd-fs@freebsd.org Tue Apr 26 03:29:31 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9E75EB0DEE9 for ; Tue, 26 Apr 2016 03:29:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8EB4517CA for ; Tue, 26 Apr 2016 03:29:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u3Q3TVWe033307 for ; Tue, 26 Apr 2016 03:29:31 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 208882] zfs root filesystem mount failure on startup in FreeBSD 10.3-RELEASE if USB hdd with zpool is attached to another port Date: Tue, 26 Apr 2016 03:29:31 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ish@amail.plala.or.jp X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 03:29:31 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D208882 --- Comment #6 from Masachika ISHIZUKA --- (In reply to Fabian Keil from comment #5) Thank you for reply. > committed should not be necessary. It is very disappointing. By the way, I think that although the retry messages are shown in seconds, = it is not match real time. It may be shown in count. i.e. printf("Mounting from %s:%s failed with error %d. " "%d time(s) left. Retrying.\n", fs, dev, error, (timeout + delay - 1)/ delay); --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Apr 26 12:44:42 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 45CAEB1C6FF for ; Tue, 26 Apr 2016 12:44:42 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0AC451C67 for ; Tue, 26 Apr 2016 12:44:41 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 6B64F28426 for ; Tue, 26 Apr 2016 14:44:33 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 58EB028428 for ; Tue, 26 Apr 2016 14:44:29 +0200 (CEST) Message-ID: <571F62AD.6080005@quip.cz> Date: Tue, 26 Apr 2016 14:44:29 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: How to speed up slow zpool scrub? Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 12:44:42 -0000 Hi, is there any way to make zpool scrub faster? We have one older machine with CPU Pentium(R) Dual E2160 @1.80GHz, 5GB of RAM and 4x 4TB HDDs. It is just a storage for backups for about 20 machines. Scrub is scheduled from periodic each 30 days but it takes about 4 days to complete and everything during scrub is slow. Backups takes 8 hours instead of 5 (made by rsync), deleting of old files is even more slower. The backups are made every night from the midnight to morning, the machine is idle for the rest of the day. Is there any tuning to make scrub faster in this idle time? Or is it better to do it other way - slower scrub with even lower priority taking for about one week but not affecting time of normal operations? (is it dangerous to have scrub running this long or reboot machine during the scrub?) I have a performance graphs of this machine and CPU is about 70% idle during scrub, but hard drives are busy 75% (according to iostat) FreeBSD 10.3-RELEASE amd64 GENERIC Miroslav Lachman From owner-freebsd-fs@freebsd.org Tue Apr 26 13:17:18 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 80366B1CF81 for ; Tue, 26 Apr 2016 13:17:18 +0000 (UTC) (envelope-from jg@internetx.com) Received: from mx1.internetx.com (mx1.internetx.com [62.116.129.39]) by mx1.freebsd.org (Postfix) with ESMTP id 45BDB1F2D for ; Tue, 26 Apr 2016 13:17:18 +0000 (UTC) (envelope-from jg@internetx.com) Received: from localhost (localhost [127.0.0.1]) by mx1.internetx.com (Postfix) with ESMTP id BD18C4C4C820; Tue, 26 Apr 2016 15:09:31 +0200 (CEST) X-Virus-Scanned: InterNetX GmbH amavisd-new at ix-mailer.internetx.de Received: from mx1.internetx.com ([62.116.129.39]) by localhost (ix-mailer.internetx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qCbPEnV2sgFq; Tue, 26 Apr 2016 15:09:26 +0200 (CEST) Received: from [192.168.100.26] (pizza.internetx.de [62.116.129.3]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.internetx.com (Postfix) with ESMTPSA id EE0594C4C79A; Tue, 26 Apr 2016 15:09:25 +0200 (CEST) Subject: Re: How to speed up slow zpool scrub? References: <571F62AD.6080005@quip.cz> To: Miroslav Lachman <000.fbsd@quip.cz>, freebsd-fs@freebsd.org Reply-To: jg@internetx.com From: InterNetX - Juergen Gotteswinter Message-ID: <571F687D.8040103@internetx.com> Date: Tue, 26 Apr 2016 15:09:17 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <571F62AD.6080005@quip.cz> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 13:17:18 -0000 to speed up the scrub itself you can try sysctl -w vfs.zfs.scrub_delay = 4 (default, 0 means higher prio) but be careful as this can cause a serious performance impact, the value can be changed on the fly your pool is raidz, mirror ? dedup is hopefully disabled? Am 4/26/2016 um 2:44 PM schrieb Miroslav Lachman: > Hi, > > is there any way to make zpool scrub faster? > We have one older machine with CPU Pentium(R) Dual E2160 @1.80GHz, 5GB > of RAM and 4x 4TB HDDs. It is just a storage for backups for about 20 > machines. > Scrub is scheduled from periodic each 30 days but it takes about 4 days > to complete and everything during scrub is slow. Backups takes 8 hours > instead of 5 (made by rsync), deleting of old files is even more slower. > > The backups are made every night from the midnight to morning, the > machine is idle for the rest of the day. > > Is there any tuning to make scrub faster in this idle time? > Or is it better to do it other way - slower scrub with even lower > priority taking for about one week but not affecting time of normal > operations? (is it dangerous to have scrub running this long or reboot > machine during the scrub?) > > I have a performance graphs of this machine and CPU is about 70% idle > during scrub, but hard drives are busy 75% (according to iostat) > > FreeBSD 10.3-RELEASE amd64 GENERIC > > Miroslav Lachman > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Tue Apr 26 13:35:38 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9D43B1D4B0 for ; Tue, 26 Apr 2016 13:35:38 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 65D731BC7 for ; Tue, 26 Apr 2016 13:35:37 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id CB4E328430; Tue, 26 Apr 2016 15:35:34 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 114C428426; Tue, 26 Apr 2016 15:35:33 +0200 (CEST) Message-ID: <571F6EA4.90800@quip.cz> Date: Tue, 26 Apr 2016 15:35:32 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: jg@internetx.com, freebsd-fs@freebsd.org Subject: Re: How to speed up slow zpool scrub? References: <571F62AD.6080005@quip.cz> <571F687D.8040103@internetx.com> In-Reply-To: <571F687D.8040103@internetx.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 13:35:38 -0000 InterNetX - Juergen Gotteswinter wrote on 04/26/2016 15:09: > to speed up the scrub itself you can try > > sysctl -w vfs.zfs.scrub_delay = 4 (default, 0 means higher prio) I will try it in the idle times > but be careful as this can cause a serious performance impact, the value > can be changed on the fly > > your pool is raidz, mirror ? dedup is hopefully disabled? I forgot to mention it. Disks are partitioned to four partitions: # gpart show -l ada0 => 34 7814037101 ada0 GPT (3.6T) 34 6 - free - (3.0K) 40 1024 1 boot0 (512K) 1064 10485760 2 swap0 (5.0G) 10486824 31457280 3 disk0sys (15G) 41944104 7769948160 4 disk0tank0 (3.6T) 7811892264 2144871 - free - (1.0G) diskXsys partitions are used for base system pool which is 4-way mirror diskXtank0 partitions are used for data storage as RAIDZ # zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT sys 14.9G 11.0G 3.92G - 79% 73% 1.00x ONLINE - tank0 14.4T 10.8T 3.56T - 19% 75% 1.00x ONLINE - # zpool status -v pool: sys state: ONLINE scan: scrub repaired 0 in 1h2m with 0 errors on Sun Apr 24 04:03:54 2016 config: NAME STATE READ WRITE CKSUM sys ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/disk0sys ONLINE 0 0 0 gpt/disk1sys ONLINE 0 0 0 gpt/disk2sys ONLINE 0 0 0 gpt/disk3sys ONLINE 0 0 0 errors: No known data errors pool: tank0 state: ONLINE scan: scrub in progress since Sun Apr 24 03:01:35 2016 7.63T scanned out of 10.6T at 36.7M/s, 23h32m to go 0 repaired, 71.98% done config: NAME STATE READ WRITE CKSUM tank0 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gpt/disk0tank0 ONLINE 0 0 0 gpt/disk1tank0 ONLINE 0 0 0 gpt/disk2tank0 ONLINE 0 0 0 gpt/disk3tank0 ONLINE 0 0 0 errors: No known data errors # zdb | grep ashift ashift: 12 ashift: 12 Thank you for your informations. > Am 4/26/2016 um 2:44 PM schrieb Miroslav Lachman: >> Hi, >> >> is there any way to make zpool scrub faster? >> We have one older machine with CPU Pentium(R) Dual E2160 @1.80GHz, 5GB >> of RAM and 4x 4TB HDDs. It is just a storage for backups for about 20 >> machines. >> Scrub is scheduled from periodic each 30 days but it takes about 4 days >> to complete and everything during scrub is slow. Backups takes 8 hours >> instead of 5 (made by rsync), deleting of old files is even more slower. >> >> The backups are made every night from the midnight to morning, the >> machine is idle for the rest of the day. >> >> Is there any tuning to make scrub faster in this idle time? >> Or is it better to do it other way - slower scrub with even lower >> priority taking for about one week but not affecting time of normal >> operations? (is it dangerous to have scrub running this long or reboot >> machine during the scrub?) >> >> I have a performance graphs of this machine and CPU is about 70% idle >> during scrub, but hard drives are busy 75% (according to iostat) >> >> FreeBSD 10.3-RELEASE amd64 GENERIC >> >> Miroslav Lachman >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Tue Apr 26 14:02:40 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6C717B1D993 for ; Tue, 26 Apr 2016 14:02:40 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-wm0-x22d.google.com (mail-wm0-x22d.google.com [IPv6:2a00:1450:400c:c09::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 01E7D17B8 for ; Tue, 26 Apr 2016 14:02:39 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: by mail-wm0-x22d.google.com with SMTP id v188so130868729wme.1 for ; Tue, 26 Apr 2016 07:02:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=Rpw4vzDYYQQZ6RMZTDwF06nfQuakZcMOv6Z0eJZq2dw=; b=eVgdnY7ezhfoTbJLtTMGbui9AWjLGLNS5FynE54qbaxS4iZZXWtrvRo8fjaG7UE3k1 zSXXcs6H8fcMrJuYOL8gPKhMFQsgEgY0Ymb7tjWvo23L+XEqHF6AEs+dKsEDa9cunl8q tdjBihqpHKCBond1F8bqeIX82RxENWm7E8LyBR5rGseSOdzTar50ssmMdl2DWzLUtunD 08zYT88E08XRrSkxmbbxUB+JnIcoqDmU1eHVTv2VN3Rkw5keWpx0LYgwYHbHjDqYEOho iyHoMhpGHzkjvX5knf1j0IRDTMwDksKchd8aAxZH3zNGIOx7g9u7KEa4QzV/J2ZBktDs 1qMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=Rpw4vzDYYQQZ6RMZTDwF06nfQuakZcMOv6Z0eJZq2dw=; b=OgzvoIAzsCG5oTx7QgShgmHG349SZ2Z+ZY7YU/BCexZeWJtf5oOPfadiG/qdl3FBQA qmg+EgvfLLSvIehKnGVplLxRgunfeiNqfcohe37juBNgpJVF9+kOr00vJN7UZliBSq4c j+72AdSYFnUGWhaTWug75QTb6Qr8IrjfPpGUqU2TXi9i5UNeUxC+3CZ3IrPZzIwzq6v1 zitSWWGUsNiH7TI9i8jMmiTCf1N6PEztPPp6FnJzcixV692cY1Dw+pszOiyqKpW/gEBl flUmmA0xWsS+RdzrehaS8VoELolaRbrbFty6FWywzdhnqCJsBfND76hSnZCLaiMooa3/ EEtg== X-Gm-Message-State: AOPr4FUDfUNqqe/3GvOoinCN9EgVtvYX9ZTp1TT+jsVUFM9KvRVBKhgCXNW4Zq21jBmNlEY95j3wYkg5jThSgg== MIME-Version: 1.0 X-Received: by 10.194.175.168 with SMTP id cb8mr3470672wjc.56.1461679350026; Tue, 26 Apr 2016 07:02:30 -0700 (PDT) Received: by 10.28.46.67 with HTTP; Tue, 26 Apr 2016 07:02:29 -0700 (PDT) In-Reply-To: <571F6EA4.90800@quip.cz> References: <571F62AD.6080005@quip.cz> <571F687D.8040103@internetx.com> <571F6EA4.90800@quip.cz> Date: Tue, 26 Apr 2016 15:02:29 +0100 Message-ID: Subject: Re: How to speed up slow zpool scrub? From: krad To: Miroslav Lachman <000.fbsd@quip.cz> Cc: jg@internetx.com, FreeBSD FS Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 14:02:40 -0000 Erk, i would try to move your system off those data disks as you have two pools competing for the disk spindles. This is never ideal. You can by all means backup your os to those data pools but keep them on separate physical mediums. A couple of small SSD would do the trick nicely and could probably be added with no down time. You would probably want to find a suitable window though to make sure the box reboots nicely though. On 26 April 2016 at 14:35, Miroslav Lachman <000.fbsd@quip.cz> wrote: > InterNetX - Juergen Gotteswinter wrote on 04/26/2016 15:09: > >> to speed up the scrub itself you can try >> >> sysctl -w vfs.zfs.scrub_delay = 4 (default, 0 means higher prio) >> > > I will try it in the idle times > > but be careful as this can cause a serious performance impact, the value >> can be changed on the fly >> >> your pool is raidz, mirror ? dedup is hopefully disabled? >> > > I forgot to mention it. Disks are partitioned to four partitions: > > # gpart show -l ada0 > => 34 7814037101 ada0 GPT (3.6T) > 34 6 - free - (3.0K) > 40 1024 1 boot0 (512K) > 1064 10485760 2 swap0 (5.0G) > 10486824 31457280 3 disk0sys (15G) > 41944104 7769948160 4 disk0tank0 (3.6T) > 7811892264 2144871 - free - (1.0G) > > diskXsys partitions are used for base system pool which is 4-way mirror > > diskXtank0 partitions are used for data storage as RAIDZ > > # zpool list > NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > sys 14.9G 11.0G 3.92G - 79% 73% 1.00x ONLINE - > tank0 14.4T 10.8T 3.56T - 19% 75% 1.00x ONLINE - > > > # zpool status -v > pool: sys > state: ONLINE > scan: scrub repaired 0 in 1h2m with 0 errors on Sun Apr 24 04:03:54 2016 > config: > > NAME STATE READ WRITE CKSUM > sys ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > gpt/disk0sys ONLINE 0 0 0 > gpt/disk1sys ONLINE 0 0 0 > gpt/disk2sys ONLINE 0 0 0 > gpt/disk3sys ONLINE 0 0 0 > > errors: No known data errors > > pool: tank0 > state: ONLINE > scan: scrub in progress since Sun Apr 24 03:01:35 2016 > 7.63T scanned out of 10.6T at 36.7M/s, 23h32m to go > 0 repaired, 71.98% done > config: > > NAME STATE READ WRITE CKSUM > tank0 ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > gpt/disk0tank0 ONLINE 0 0 0 > gpt/disk1tank0 ONLINE 0 0 0 > gpt/disk2tank0 ONLINE 0 0 0 > gpt/disk3tank0 ONLINE 0 0 0 > > errors: No known data errors > > > # zdb | grep ashift > ashift: 12 > ashift: 12 > > > Thank you for your informations. > > > > Am 4/26/2016 um 2:44 PM schrieb Miroslav Lachman: >> >>> Hi, >>> >>> is there any way to make zpool scrub faster? >>> We have one older machine with CPU Pentium(R) Dual E2160 @1.80GHz, 5GB >>> of RAM and 4x 4TB HDDs. It is just a storage for backups for about 20 >>> machines. >>> Scrub is scheduled from periodic each 30 days but it takes about 4 days >>> to complete and everything during scrub is slow. Backups takes 8 hours >>> instead of 5 (made by rsync), deleting of old files is even more slower. >>> >>> The backups are made every night from the midnight to morning, the >>> machine is idle for the rest of the day. >>> >>> Is there any tuning to make scrub faster in this idle time? >>> Or is it better to do it other way - slower scrub with even lower >>> priority taking for about one week but not affecting time of normal >>> operations? (is it dangerous to have scrub running this long or reboot >>> machine during the scrub?) >>> >>> I have a performance graphs of this machine and CPU is about 70% idle >>> during scrub, but hard drives are busy 75% (according to iostat) >>> >>> FreeBSD 10.3-RELEASE amd64 GENERIC >>> >>> Miroslav Lachman >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>> >> >> > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Tue Apr 26 14:28:02 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D0B9CB1DF5B for ; Tue, 26 Apr 2016 14:28:02 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5B3F816F0 for ; Tue, 26 Apr 2016 14:28:01 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 172ED28426; Tue, 26 Apr 2016 16:27:59 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id E2CFF28423; Tue, 26 Apr 2016 16:27:57 +0200 (CEST) Message-ID: <571F7AED.2040500@quip.cz> Date: Tue, 26 Apr 2016 16:27:57 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: krad CC: jg@internetx.com, FreeBSD FS Subject: Re: How to speed up slow zpool scrub? References: <571F62AD.6080005@quip.cz> <571F687D.8040103@internetx.com> <571F6EA4.90800@quip.cz> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 14:28:02 -0000 krad wrote on 04/26/2016 16:02: > Erk, i would try to move your system off those data disks as you have > two pools competing for the disk spindles. This is never ideal. You can > by all means backup your os to those data pools but keep them on > separate physical mediums. A couple of small SSD would do the trick > nicely and could probably be added with no down time. You would probably > want to find a suitable window though to make sure the box reboots > nicely though. The system pool is really small - only 15GB and scrub is done relatively fast. This machine cannot handle additional disks so I cannot move system to other devices anyway. I tried system on USB flashdisk (read only) in the past but it was slow and USB disk broke early. > On 26 April 2016 at 14:35, Miroslav Lachman <000.fbsd@quip.cz > > wrote: > > InterNetX - Juergen Gotteswinter wrote on 04/26/2016 15:09: > > to speed up the scrub itself you can try > > sysctl -w vfs.zfs.scrub_delay = 4 (default, 0 means higher prio) > > > I will try it in the idle times > > but be careful as this can cause a serious performance impact, > the value > can be changed on the fly > > your pool is raidz, mirror ? dedup is hopefully disabled? > > > I forgot to mention it. Disks are partitioned to four partitions: > > # gpart show -l ada0 > => 34 7814037101 ada0 GPT (3.6T) > 34 6 - free - (3.0K) > 40 1024 1 boot0 (512K) > 1064 10485760 2 swap0 (5.0G) > 10486824 31457280 3 disk0sys (15G) > 41944104 7769948160 4 disk0tank0 (3.6T) > 7811892264 2144871 - free - (1.0G) > > diskXsys partitions are used for base system pool which is 4-way mirror > > diskXtank0 partitions are used for data storage as RAIDZ > > # zpool list > NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH > ALTROOT > sys 14.9G 11.0G 3.92G - 79% 73% 1.00x ONLINE - > tank0 14.4T 10.8T 3.56T - 19% 75% 1.00x ONLINE - > > > # zpool status -v > pool: sys > state: ONLINE > scan: scrub repaired 0 in 1h2m with 0 errors on Sun Apr 24 > 04:03:54 2016 > config: > > NAME STATE READ WRITE CKSUM > sys ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > gpt/disk0sys ONLINE 0 0 0 > gpt/disk1sys ONLINE 0 0 0 > gpt/disk2sys ONLINE 0 0 0 > gpt/disk3sys ONLINE 0 0 0 > > errors: No known data errors > > pool: tank0 > state: ONLINE > scan: scrub in progress since Sun Apr 24 03:01:35 2016 > 7.63T scanned out of 10.6T at 36.7M/s, 23h32m to go > 0 repaired, 71.98% done > config: > > NAME STATE READ WRITE CKSUM > tank0 ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > gpt/disk0tank0 ONLINE 0 0 0 > gpt/disk1tank0 ONLINE 0 0 0 > gpt/disk2tank0 ONLINE 0 0 0 > gpt/disk3tank0 ONLINE 0 0 0 > > errors: No known data errors > > > # zdb | grep ashift > ashift: 12 > ashift: 12 > > > Thank you for your informations. > > From owner-freebsd-fs@freebsd.org Tue Apr 26 15:01:19 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E7DE3B1CAAA for ; Tue, 26 Apr 2016 15:01:19 +0000 (UTC) (envelope-from gldisater@gmail.com) Received: from mail-io0-x234.google.com (mail-io0-x234.google.com [IPv6:2607:f8b0:4001:c06::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B1C49130B for ; Tue, 26 Apr 2016 15:01:19 +0000 (UTC) (envelope-from gldisater@gmail.com) Received: by mail-io0-x234.google.com with SMTP id 190so12627096iow.1 for ; Tue, 26 Apr 2016 08:01:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=CTZQKFAF60nIO1w+KxqLDplAmfMQ1x7NuF+AyTb8bDE=; b=L4wT40w1IHb0fEfJtsOXCIhIjPIuoIynKDyBc6dXVN50hyGs06coAsQy08UslyO7uv TyXoVO9cLITU5mYqf8cJASIuUVgey/dlTavvIA+zMLytuEPUfxFfKpVaGJ6YLWs8eC8H IGfZOPX3EYAYPI1+QQvE978nLqwSp9QmGh1HDDJUWg6lJuZR+3Yn3KhZYAlTo8mvDDcy lm6M6frP4/+HdsR/E2kRPB8cZq77DddvG/u4ncgdidoggca6/xU9s1rmQ9vruCNIZueL rj+C29/QoBcbnTKZSHx2FNIWQx+W8+BzyTnSbSZ9re+IaC1c3B/veBa1fbuiRBbGMNTj GXaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=CTZQKFAF60nIO1w+KxqLDplAmfMQ1x7NuF+AyTb8bDE=; b=Bb0GHWW3gdepPNCZoXg8cev7AcjZFvrWLAowV40Go4oXkDLUxnMwri+76iEEXIcb7f 3SDm/b16KZ8BfCrRITztyR4dKL/s6FyabQm7VpQlc1jYoMR91kleLahLkq/Gw7FCodyl +5yjIr+IcZ5I3/Y/jAG5M6OKZ0ZgBdJURwe/9VINYJ/soEMjXNcdxqllgXFp5SX/lUXl QG9NFcGLyrRDPeF4Q2rckrQizKOXkx1wAm9QjzY/3ixxjGGV9T46DtRyAkcsGzo1Grnz WEiCXm/aCZ38fdfQa0rgfqvSZZ43zQm6lIvtVC2KP0URtshMHodwbRFZiqjdwS2kZiB7 5+5w== X-Gm-Message-State: AOPr4FUmBQH3TxbsZ6ph8kcQZT1lnOW5Gn86yFgJNO3kLSIUm984rmU2tXRTKFx0OiLg7Q== X-Received: by 10.107.151.8 with SMTP id z8mr3925719iod.191.1461682878953; Tue, 26 Apr 2016 08:01:18 -0700 (PDT) Received: from [192.168.1.128] (dhcp-108-168-15-83.cable.user.start.ca. [108.168.15.83]) by smtp.gmail.com with ESMTPSA id y198sm1812837iof.10.2016.04.26.08.01.18 for (version=TLSv1/SSLv3 cipher=OTHER); Tue, 26 Apr 2016 08:01:18 -0700 (PDT) Subject: Re: How to speed up slow zpool scrub? To: freebsd-fs@freebsd.org References: <571F62AD.6080005@quip.cz> <571F687D.8040103@internetx.com> <571F6EA4.90800@quip.cz> From: Jeremy Faulkner Message-ID: <571F82B5.3010807@gmail.com> Date: Tue, 26 Apr 2016 11:01:09 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <571F6EA4.90800@quip.cz> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 15:01:20 -0000 zfs get all tank0 On 2016-04-26 9:35 AM, Miroslav Lachman wrote: > InterNetX - Juergen Gotteswinter wrote on 04/26/2016 15:09: >> to speed up the scrub itself you can try >> >> sysctl -w vfs.zfs.scrub_delay = 4 (default, 0 means higher prio) > > I will try it in the idle times > >> but be careful as this can cause a serious performance impact, the value >> can be changed on the fly >> >> your pool is raidz, mirror ? dedup is hopefully disabled? > > I forgot to mention it. Disks are partitioned to four partitions: > > # gpart show -l ada0 > => 34 7814037101 ada0 GPT (3.6T) > 34 6 - free - (3.0K) > 40 1024 1 boot0 (512K) > 1064 10485760 2 swap0 (5.0G) > 10486824 31457280 3 disk0sys (15G) > 41944104 7769948160 4 disk0tank0 (3.6T) > 7811892264 2144871 - free - (1.0G) > > diskXsys partitions are used for base system pool which is 4-way mirror > > diskXtank0 partitions are used for data storage as RAIDZ > > # zpool list > NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > sys 14.9G 11.0G 3.92G - 79% 73% 1.00x ONLINE - > tank0 14.4T 10.8T 3.56T - 19% 75% 1.00x ONLINE - > > > # zpool status -v > pool: sys > state: ONLINE > scan: scrub repaired 0 in 1h2m with 0 errors on Sun Apr 24 04:03:54 2016 > config: > > NAME STATE READ WRITE CKSUM > sys ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > gpt/disk0sys ONLINE 0 0 0 > gpt/disk1sys ONLINE 0 0 0 > gpt/disk2sys ONLINE 0 0 0 > gpt/disk3sys ONLINE 0 0 0 > > errors: No known data errors > > pool: tank0 > state: ONLINE > scan: scrub in progress since Sun Apr 24 03:01:35 2016 > 7.63T scanned out of 10.6T at 36.7M/s, 23h32m to go > 0 repaired, 71.98% done > config: > > NAME STATE READ WRITE CKSUM > tank0 ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > gpt/disk0tank0 ONLINE 0 0 0 > gpt/disk1tank0 ONLINE 0 0 0 > gpt/disk2tank0 ONLINE 0 0 0 > gpt/disk3tank0 ONLINE 0 0 0 > > errors: No known data errors > > > # zdb | grep ashift > ashift: 12 > ashift: 12 > > > Thank you for your informations. > > From owner-freebsd-fs@freebsd.org Tue Apr 26 15:08:17 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 96ED2B1CCFD for ; Tue, 26 Apr 2016 15:08:17 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 112321832 for ; Tue, 26 Apr 2016 15:08:16 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id C75182840C; Tue, 26 Apr 2016 17:08:13 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 60FB428417; Tue, 26 Apr 2016 17:08:12 +0200 (CEST) Message-ID: <571F845C.5060902@quip.cz> Date: Tue, 26 Apr 2016 17:08:12 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: Jeremy Faulkner , freebsd-fs@freebsd.org Subject: Re: How to speed up slow zpool scrub? References: <571F62AD.6080005@quip.cz> <571F687D.8040103@internetx.com> <571F6EA4.90800@quip.cz> <571F82B5.3010807@gmail.com> In-Reply-To: <571F82B5.3010807@gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 15:08:17 -0000 Jeremy Faulkner wrote on 04/26/2016 17:01: > zfs get all tank0 I set checksum=fletcher4 and compression=lz4 (+ atime & exec to Off), anything else is in default state. There are 19 filesystems on tank0 and each have about 5 snapshots. I don't know how long scrub runs on some others system. If it is limited by CPU, or disk IOps... but for me 3 - 4 days are really long. # zfs get all tank0 NAME PROPERTY VALUE SOURCE tank0 type filesystem - tank0 creation Thu Jul 23 1:37 2015 - tank0 used 7.85T - tank0 available 2.26T - tank0 referenced 140K - tank0 compressratio 1.86x - tank0 mounted no - tank0 quota none default tank0 reservation none default tank0 recordsize 128K default tank0 mountpoint none local tank0 sharenfs off default tank0 checksum fletcher4 local tank0 compression lz4 local tank0 atime off local tank0 devices on default tank0 exec off local tank0 setuid on default tank0 readonly off default tank0 jailed off default tank0 snapdir hidden default tank0 aclmode discard default tank0 aclinherit restricted default tank0 canmount on default tank0 xattr on default tank0 copies 1 default tank0 version 5 - tank0 utf8only off - tank0 normalization none - tank0 casesensitivity sensitive - tank0 vscan off default tank0 nbmand off default tank0 sharesmb off default tank0 refquota none default tank0 refreservation none default tank0 primarycache all default tank0 secondarycache all default tank0 usedbysnapshots 0 - tank0 usedbydataset 140K - tank0 usedbychildren 7.85T - tank0 usedbyrefreservation 0 - tank0 logbias latency default tank0 dedup off default tank0 mlslabel - tank0 sync standard default tank0 refcompressratio 1.00x - tank0 written 140K - tank0 logicalused 13.3T - tank0 logicalreferenced 9.50K - tank0 volmode default default tank0 filesystem_limit none default tank0 snapshot_limit none default tank0 filesystem_count none default tank0 snapshot_count none default tank0 redundant_metadata all default > On 2016-04-26 9:35 AM, Miroslav Lachman wrote: >> InterNetX - Juergen Gotteswinter wrote on 04/26/2016 15:09: >>> to speed up the scrub itself you can try >>> >>> sysctl -w vfs.zfs.scrub_delay = 4 (default, 0 means higher prio) >> >> I will try it in the idle times >> >>> but be careful as this can cause a serious performance impact, the value >>> can be changed on the fly >>> >>> your pool is raidz, mirror ? dedup is hopefully disabled? >> >> I forgot to mention it. Disks are partitioned to four partitions: >> >> # gpart show -l ada0 >> => 34 7814037101 ada0 GPT (3.6T) >> 34 6 - free - (3.0K) >> 40 1024 1 boot0 (512K) >> 1064 10485760 2 swap0 (5.0G) >> 10486824 31457280 3 disk0sys (15G) >> 41944104 7769948160 4 disk0tank0 (3.6T) >> 7811892264 2144871 - free - (1.0G) >> >> diskXsys partitions are used for base system pool which is 4-way mirror >> >> diskXtank0 partitions are used for data storage as RAIDZ >> >> # zpool list >> NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH >> ALTROOT >> sys 14.9G 11.0G 3.92G - 79% 73% 1.00x ONLINE - >> tank0 14.4T 10.8T 3.56T - 19% 75% 1.00x ONLINE - >> >> >> # zpool status -v >> pool: sys >> state: ONLINE >> scan: scrub repaired 0 in 1h2m with 0 errors on Sun Apr 24 04:03:54 >> 2016 >> config: >> >> NAME STATE READ WRITE CKSUM >> sys ONLINE 0 0 0 >> mirror-0 ONLINE 0 0 0 >> gpt/disk0sys ONLINE 0 0 0 >> gpt/disk1sys ONLINE 0 0 0 >> gpt/disk2sys ONLINE 0 0 0 >> gpt/disk3sys ONLINE 0 0 0 >> >> errors: No known data errors >> >> pool: tank0 >> state: ONLINE >> scan: scrub in progress since Sun Apr 24 03:01:35 2016 >> 7.63T scanned out of 10.6T at 36.7M/s, 23h32m to go >> 0 repaired, 71.98% done >> config: >> >> NAME STATE READ WRITE CKSUM >> tank0 ONLINE 0 0 0 >> raidz1-0 ONLINE 0 0 0 >> gpt/disk0tank0 ONLINE 0 0 0 >> gpt/disk1tank0 ONLINE 0 0 0 >> gpt/disk2tank0 ONLINE 0 0 0 >> gpt/disk3tank0 ONLINE 0 0 0 >> >> errors: No known data errors >> >> >> # zdb | grep ashift >> ashift: 12 >> ashift: 12 >> >> >> Thank you for your informations. From owner-freebsd-fs@freebsd.org Tue Apr 26 15:10:19 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5B43DB1CD71 for ; Tue, 26 Apr 2016 15:10:19 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay01.ispgateway.de (smtprelay01.ispgateway.de [80.67.31.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1F5AE18DA for ; Tue, 26 Apr 2016 15:10:18 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from [78.35.159.183] (helo=fabiankeil.de) by smtprelay01.ispgateway.de with esmtpsa (TLSv1.2:AES128-GCM-SHA256:128) (Exim 4.84) (envelope-from ) id 1av3WK-0001n9-TA for freebsd-fs@freebsd.org; Tue, 26 Apr 2016 15:59:21 +0200 Date: Tue, 26 Apr 2016 15:53:32 +0200 From: Fabian Keil To: freebsd-fs@freebsd.org Subject: Re: GELI + Zpool Scrub Results in GELI Device Destruction (and Later a Corrupt Pool) Message-ID: <20160426155332.3b7f277f@fabiankeil.de> In-Reply-To: <1461624665.22294.103.camel@michaeleichorn.com> References: <1461560445.22294.53.camel@michaeleichorn.com> <20160425101124.068c8333@fabiankeil.de> <1461587321.22294.85.camel@michaeleichorn.com> <20160425160625.5b585c8d@fabiankeil.de> <1461624665.22294.103.camel@michaeleichorn.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/b=DucN3HhhVmotmLv_a5z=X"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 15:10:19 -0000 --Sig_/b=DucN3HhhVmotmLv_a5z=X Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable "Michael B. Eichorn" wrote: > On Mon, 2016-04-25 at 16:06 +0200, Fabian Keil wrote: > > "Michael B. Eichorn" wrote: > > =20 > > >=20 > > > On Mon, 2016-04-25 at 10:11 +0200, Fabian Keil wrote: =20 > > > > If yes, this is a known issue: > > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D117158 > > > > =C2=A0=C2=A0 =20 > > > Reading that bug in detail it appears to be *specifically* for the > > > kernel panic and that zfs closing and reopening providers is > > > expected > > > behavior, and that if geli has autodetach configured that it would > > > detach. > > >=20 > > > It stikes me that even though this is expected behavior it should > > > not > > > be. Is there a way we could prevent the detach when zfs does closes > > > and > > > reopens providers? I cannnot think of a case where the desired > > > behavior > > > is for the pool to detach when zfs wants to reopen it. =20 > > I suppose geli could delay the detachment a bit to give the consumer > > a chance to reopen it. =20 >=20 > That would probably work, but my inner engineer is dissatisfied with > the idea of slowing down all detachments to solve a single case. >=20 > What about a new feature whereby zfs (or any other consumer) can > inhibit the autodetach for a brief period? I don't really know if this > is feasible, but I thought I would ask. IIRC ZFS simply reopens the devices so nobody had to write code to read the label from an already opened device and update the internal state (and deal with the various locking issues). Changing this is feasible but someone would have to care enough to spend time on this. One could also work around the problem by letting ZFS open the already opened device read-only before reopening it with write access. That too is likely to dissatisfy your inner engineer. > Anything like this is probably too major to implement without some > thought and probably consensus building. So in the meantime I will file > my bug against the handbook to make sure geli_autodetach and zfs are > mentioned. The consensus seems to be that nobody cares enough about bugs like this one to work on fixing them. Fabian --Sig_/b=DucN3HhhVmotmLv_a5z=X Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlcfctwACgkQBYqIVf93VJ29CgCgo0hKqALtZEAvl+igImh7BUH4 7/EAn2Lo5ZP1tkyRGl/jGPm+Ogwyb8jm =zZ5w -----END PGP SIGNATURE----- --Sig_/b=DucN3HhhVmotmLv_a5z=X-- From owner-freebsd-fs@freebsd.org Tue Apr 26 15:53:18 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 95012B1D8AA for ; Tue, 26 Apr 2016 15:53:18 +0000 (UTC) (envelope-from dhutch9999@yahoo.com) Received: from nm40-vm10.bullet.mail.gq1.yahoo.com (nm40-vm10.bullet.mail.gq1.yahoo.com [98.136.216.251]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6B75F167F for ; Tue, 26 Apr 2016 15:53:17 +0000 (UTC) (envelope-from dhutch9999@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1461685829; bh=P+xlaO3IXuYztiexrS0kkD5lAyte8c3kgWjab0Ek18A=; h=Date:From:Reply-To:To:Subject:References:From:Subject; b=VrMk7Pwqk7HVRRPQWTu9Y7ZlBV2x8UCYMgyhwoATf0aCDkhaOLCiUvB9jcSvmQ7cHfgczqNk075meBzJsDhrpoH/7X/5DgnmqJEeOlXCtc23OUp7zF8DmnzVM772TTr0WRKFxE2v8mgwAMjjJrrm1EL3/h4Lq+WBFWu/lBuawAWLAME65M7KMNTZ0sHs/OOzQaMyb4i11yAQQGqmGr5zTwAKTENAI6VY8PFoam0HnSPSX95BHfRgo85kMaROfLYlbk3n4mTLZeHj70orA0xcGyWv9GFSbCCMMdAbplf4VHjK2wTNm09qKmOcF9upv+P8vEhOI7OsLR0Wg2nOPoOvag== Received: from [127.0.0.1] by nm40.bullet.mail.gq1.yahoo.com with NNFMP; 26 Apr 2016 15:50:29 -0000 Received: from [98.137.12.174] by nm40.bullet.mail.gq1.yahoo.com with NNFMP; 26 Apr 2016 15:47:34 -0000 Received: from [98.139.170.181] by tm13.bullet.mail.gq1.yahoo.com with NNFMP; 26 Apr 2016 15:47:34 -0000 Received: from [98.139.212.222] by tm24.bullet.mail.bf1.yahoo.com with NNFMP; 26 Apr 2016 15:47:34 -0000 Received: from [127.0.0.1] by omp1031.mail.bf1.yahoo.com with NNFMP; 26 Apr 2016 15:47:34 -0000 X-Yahoo-Newman-Property: ymail-4 X-Yahoo-Newman-Id: 558150.29124.bm@omp1031.mail.bf1.yahoo.com X-YMail-OSG: kZ8gh2AVM1lbe03W2oRyUonEww7mnSHXpe1bFhW_bnrffIYV4GVrc_WoyR.qtQJ E1RGkmCTqjqsUnAYO_m7PfXvqpAWhRyvyoHaV2ZJM2T4XCF7YeedGt4OmkdjUuqzSVW_LjbbpxeB oOM9pGsbEiBbFUUeF_NTs19KBVZkA.oae.0xAJrfFSWYZXBQRfeus5vuRPBC5dw0gl6jpt6TFERo WLljL.9okUnEKvBwRh3GmivgWyoUp1dKnCBd68Ydad7T_SJ6jbc1S8_Ad7tgvH8CHVKNKpfrQ3Le da4Zbs7iYGlw5RyWowDsAIm_apVgwQ0DYSeL2EkcxUewMMs0pc9VFHZ9jlSodN3Ah_jh8t1_hUy8 GofnDeLpYoTV_yKd3M_2QUBvd.nPzaGTJUGqgPLzowIzveTyGJCbOzMpG_R03_FaRP.w9gCSrBtZ zwOjA2sXrYVNZSKdL4tVUNlDGxzbz7wB.5k6Uq38cNg.DCyL3PFDvIaNtHnxTnjuTshY92Efb71B YvcnQQi2DwUlLsXGtC1AeEA4- Received: by 76.13.26.138; Tue, 26 Apr 2016 15:47:34 +0000 Date: Tue, 26 Apr 2016 15:47:33 +0000 (UTC) From: DH Reply-To: DH To: , Miroslav Lachman <000.fbsd@quip.cz> Message-ID: <698816653.2698619.1461685653634.JavaMail.yahoo@mail.yahoo.com> Subject: Re: How to speed up slow zpool scrub? MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit References: <698816653.2698619.1461685653634.JavaMail.yahoo.ref@mail.yahoo.com> X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 15:53:18 -0000 >5GB of RAM That seems to be an insufficient amount of system ram when employing zfs. Take a look at this: http://doc.freenas.org/9.3/freenas_intro.html#ram David Hutchens III System Administrator -------------------------------------------- On Tue, 4/26/16, Miroslav Lachman <000.fbsd@quip.cz> wrote: Subject: How to speed up slow zpool scrub? To: freebsd-fs@freebsd.org Date: Tuesday, April 26, 2016, 5:44 AM Hi, is there any way to make zpool scrub faster? We have one older machine with CPU Pentium(R) Dual E2160 @1.80GHz, 5GB of RAM and 4x 4TB HDDs. It is just a storage for backups for about 20 machines. Scrub is scheduled from periodic each 30 days but it takes about 4 days to complete and everything during scrub is slow. Backups takes 8 hours instead of 5 (made by rsync), deleting of old files is even more slower. The backups are made every night from the midnight to morning, the machine is idle for the rest of the day. Is there any tuning to make scrub faster in this idle time? Or is it better to do it other way - slower scrub with even lower priority taking for about one week but not affecting time of normal operations? (is it dangerous to have scrub running this long or reboot machine during the scrub?) I have a performance graphs of this machine and CPU is about 70% idle during scrub, but hard drives are busy 75% (according to iostat) FreeBSD 10.3-RELEASE amd64 GENERIC Miroslav Lachman _______________________________________________ freebsd-fs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Tue Apr 26 15:59:42 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 89BD3B1D9B4 for ; Tue, 26 Apr 2016 15:59:42 +0000 (UTC) (envelope-from gldisater@gmail.com) Received: from mail-ig0-x230.google.com (mail-ig0-x230.google.com [IPv6:2607:f8b0:4001:c05::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 52F9E1A57 for ; Tue, 26 Apr 2016 15:59:42 +0000 (UTC) (envelope-from gldisater@gmail.com) Received: by mail-ig0-x230.google.com with SMTP id g8so19650765igr.0 for ; Tue, 26 Apr 2016 08:59:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=fonYLOm8a2lM6KGzGV2fRbRxslzqGra6M4C8hEXWl+8=; b=WvOrh5xXVglCWGOTvakxeWvajTi0eWQmOBMdNM25GMTPIpKj/vySTO7igVz9wJftOO 4c1JWJK++NEMBcvNCeTJu4nbx3D7i1+HGp0Qlfj00f/RsZwgPC7rvfMNN41Y7q5fa81Y 2hOvc7tL6QIIDgyvzAThzTksK8P1BSMj2ccAq/3iMnhfom9Yz+VgLCgL9D5uKezAq3Ms ZQCLwywpDrX95bNKto+C4kOjiQCf45fd26BJQQMVK7HcJA7PyWX/wdProxCY7AyBbP2z sAslobHAMFix49KpcZ0aecUitRCPtkaI5CCXjwhmauphNK20WG8wFtVmBfi9FeAyPQW6 77LQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=fonYLOm8a2lM6KGzGV2fRbRxslzqGra6M4C8hEXWl+8=; b=g5lJ57HvCzr6c7xlrYHthHHpjtUESfswjDafi2Jw76e2+MW34C565iYF2SIgJu6MUX FvMxFRoFBIC8mK4bjaTsK6iZUgG6APonjTi4nl4U0yQ1Z7ecoDj6zjGB01xnFCmY8BTV 8cDYZ/c/Ig1HiQmksGxlZTXC+VSWSnG/RAdetPKZxuOxWypU5q5dL/L38v1M/NjXGm5l ISarLiVZFcTalvy/IlxYmNLOYCvCjiiQgvfmnqQBXpGZ+ny1fAAN9WHxgeLIFRQ2aHTs jyfnjYqHg6RP8Rsmu55KntcVkU69fBLcxTPMLk39yNmaBvZ9nWNS2kvjz5ypycD4/B85 5szA== X-Gm-Message-State: AOPr4FUOAtdUOrlYIZqHYebbyhrkeO5h/R1My/N9z91x0H+F/IrIfZcASBtCBOm9ymbw4A== X-Received: by 10.50.8.97 with SMTP id q1mr4635187iga.26.1461686381720; Tue, 26 Apr 2016 08:59:41 -0700 (PDT) Received: from [192.168.1.128] (dhcp-108-168-15-83.cable.user.start.ca. [108.168.15.83]) by smtp.gmail.com with ESMTPSA id d6sm12021537igx.8.2016.04.26.08.59.40 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 26 Apr 2016 08:59:41 -0700 (PDT) Subject: Re: How to speed up slow zpool scrub? To: Miroslav Lachman <000.fbsd@quip.cz>, freebsd-fs@freebsd.org References: <571F62AD.6080005@quip.cz> <571F687D.8040103@internetx.com> <571F6EA4.90800@quip.cz> <571F82B5.3010807@gmail.com> <571F845C.5060902@quip.cz> From: Jeremy Faulkner Message-ID: <571F9064.2010602@gmail.com> Date: Tue, 26 Apr 2016 11:59:32 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <571F845C.5060902@quip.cz> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 15:59:42 -0000 On 2016-04-26 11:08 AM, Miroslav Lachman wrote: > Jeremy Faulkner wrote on 04/26/2016 17:01: >> zfs get all tank0 > > I set checksum=fletcher4 and compression=lz4 (+ atime & exec to Off), > anything else is in default state. > > There are 19 filesystems on tank0 and each have about 5 snapshots. > > I don't know how long scrub runs on some others system. If it is limited > by CPU, or disk IOps... but for me 3 - 4 days are really long. > > > # zfs get all tank0 > NAME PROPERTY VALUE SOURCE > tank0 type filesystem - > tank0 creation Thu Jul 23 1:37 2015 - > tank0 used 7.85T - > tank0 available 2.26T - > tank0 referenced 140K - > tank0 compressratio 1.86x - > tank0 mounted no - > tank0 quota none default > tank0 reservation none default > tank0 recordsize 128K default > tank0 mountpoint none local > tank0 sharenfs off default > tank0 checksum fletcher4 local > tank0 compression lz4 local > tank0 atime off local > tank0 devices on default > tank0 exec off local > tank0 setuid on default > tank0 readonly off default > tank0 jailed off default > tank0 snapdir hidden default > tank0 aclmode discard default > tank0 aclinherit restricted default > tank0 canmount on default > tank0 xattr on default > tank0 copies 1 default > tank0 version 5 - > tank0 utf8only off - > tank0 normalization none - > tank0 casesensitivity sensitive - > tank0 vscan off default > tank0 nbmand off default > tank0 sharesmb off default > tank0 refquota none default > tank0 refreservation none default > tank0 primarycache all default > tank0 secondarycache all default > tank0 usedbysnapshots 0 - > tank0 usedbydataset 140K - > tank0 usedbychildren 7.85T - > tank0 usedbyrefreservation 0 - > tank0 logbias latency default > tank0 dedup off default > tank0 mlslabel - > tank0 sync standard default > tank0 refcompressratio 1.00x - > tank0 written 140K - > tank0 logicalused 13.3T - > tank0 logicalreferenced 9.50K - > tank0 volmode default default > tank0 filesystem_limit none default > tank0 snapshot_limit none default > tank0 filesystem_count none default > tank0 snapshot_count none default > tank0 redundant_metadata all default > Check the drive health with smartctl (part of sysutils/smartmontools). Are the drives desktop drives or nas drives? In gstat is one drive busier than the rest? If so, replace that drive. From owner-freebsd-fs@freebsd.org Tue Apr 26 16:30:41 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9BD6B1DFD3 for ; Tue, 26 Apr 2016 16:30:41 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7E74D1ABE for ; Tue, 26 Apr 2016 16:30:41 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 4789128426; Tue, 26 Apr 2016 18:30:38 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 7A29E28423; Tue, 26 Apr 2016 18:30:37 +0200 (CEST) Message-ID: <571F97AD.9080107@quip.cz> Date: Tue, 26 Apr 2016 18:30:37 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: Jeremy Faulkner , freebsd-fs@freebsd.org Subject: Re: How to speed up slow zpool scrub? References: <571F62AD.6080005@quip.cz> <571F687D.8040103@internetx.com> <571F6EA4.90800@quip.cz> <571F82B5.3010807@gmail.com> <571F845C.5060902@quip.cz> <571F9064.2010602@gmail.com> In-Reply-To: <571F9064.2010602@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 16:30:41 -0000 Jeremy Faulkner wrote on 04/26/2016 17:59: > > > On 2016-04-26 11:08 AM, Miroslav Lachman wrote: >> Jeremy Faulkner wrote on 04/26/2016 17:01: >>> zfs get all tank0 >> >> I set checksum=fletcher4 and compression=lz4 (+ atime & exec to Off), >> anything else is in default state. >> >> There are 19 filesystems on tank0 and each have about 5 snapshots. >> >> I don't know how long scrub runs on some others system. If it is limited >> by CPU, or disk IOps... but for me 3 - 4 days are really long. [...] > Check the drive health with smartctl (part of sysutils/smartmontools). > Are the drives desktop drives or nas drives? In gstat is one drive > busier than the rest? If so, replace that drive. Drives are OK, smartd is running on the background, I have monitoring of some SMART values and all drives are loaded equally. Model Family: Seagate NAS HDD Device Model: ST4000VN000-1H4168 Miroslav Lachman From owner-freebsd-fs@freebsd.org Tue Apr 26 16:34:35 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5AED4B1C1EC for ; Tue, 26 Apr 2016 16:34:35 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2062C1FC5 for ; Tue, 26 Apr 2016 16:34:34 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 098EB28423; Tue, 26 Apr 2016 18:34:33 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 0468F28431; Tue, 26 Apr 2016 18:34:32 +0200 (CEST) Message-ID: <571F9897.2070008@quip.cz> Date: Tue, 26 Apr 2016 18:34:31 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: DH , freebsd-fs@freebsd.org Subject: Re: How to speed up slow zpool scrub? References: <698816653.2698619.1461685653634.JavaMail.yahoo.ref@mail.yahoo.com> <698816653.2698619.1461685653634.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: <698816653.2698619.1461685653634.JavaMail.yahoo@mail.yahoo.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 16:34:35 -0000 DH wrote on 04/26/2016 17:47: >> 5GB of RAM > > That seems to be an insufficient amount of system ram when employing zfs. > > Take a look at this: > > http://doc.freenas.org/9.3/freenas_intro.html#ram I know 5GB is not much these days but is memory used for scrubbing a lot? Because I am satisfied with working performance. The only concern is slow scrubbing and I am not sure that more memory helps in this case. Miroslav Lachman From owner-freebsd-fs@freebsd.org Tue Apr 26 16:50:13 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 934A4B1C6A6 for ; Tue, 26 Apr 2016 16:50:13 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1E74F1B49 for ; Tue, 26 Apr 2016 16:50:12 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 1377628431; Tue, 26 Apr 2016 18:50:11 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 0571928426; Tue, 26 Apr 2016 18:50:09 +0200 (CEST) Message-ID: <571F9C41.9020406@quip.cz> Date: Tue, 26 Apr 2016 18:50:09 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: Jeremy Faulkner , freebsd-fs@freebsd.org Subject: Re: How to speed up slow zpool scrub? References: <571F62AD.6080005@quip.cz> <571F687D.8040103@internetx.com> <571F6EA4.90800@quip.cz> <571F82B5.3010807@gmail.com> <571F845C.5060902@quip.cz> <571F9064.2010602@gmail.com> In-Reply-To: <571F9064.2010602@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 16:50:13 -0000 Jeremy Faulkner wrote on 04/26/2016 17:59: > Check the drive health with smartctl (part of sysutils/smartmontools). > Are the drives desktop drives or nas drives? In gstat is one drive > busier than the rest? If so, replace that drive. # iostat -x -w 15 ada0 ada1 ada2 ada3 .. .. extended device statistics device r/s w/s kr/s kw/s qlen svc_t %b ada0 215.3 15.7 5045.6 492.5 2 8.1 64 ada1 220.1 15.6 5063.2 492.0 2 8.0 65 ada2 221.7 15.1 4917.9 479.2 1 8.1 66 ada3 224.1 15.2 4910.4 482.9 0 8.1 67 extended device statistics device r/s w/s kr/s kw/s qlen svc_t %b ada0 220.8 13.5 4497.7 624.8 0 6.7 61 ada1 224.7 13.1 4503.6 625.3 0 6.8 61 ada2 206.5 12.3 4287.3 599.7 0 6.2 60 ada3 208.3 12.8 4289.2 619.2 0 6.4 60 extended device statistics device r/s w/s kr/s kw/s qlen svc_t %b ada0 236.5 8.1 5219.4 392.2 1 6.4 61 ada1 231.5 8.1 5199.4 390.6 1 6.4 61 ada2 242.5 7.5 5075.7 379.7 2 5.4 58 ada3 240.3 7.7 5048.2 383.7 2 5.9 58 Drives are 60% busy when scrubbing and deleting old files are both running so I think there is still something to tune. Miroslav Lachman # ~/bin/arc_summary.sh System Memory: Physical RAM: 4962 MB Free Memory : 108 MB ARC Size: Current Size: 3373 MB (arcsize) Target Size (Adaptive): 3372 MB (c) Min Size (Hard Limit): 512 MB (zfs_arc_min) Max Size (Hard Limit): 3584 MB (zfs_arc_max) ARC Size Breakdown: Most Recently Used Cache Size: 93% 3161 MB (p) Most Frequently Used Cache Size: 6% 210 MB (c-p) ARC Efficency: Cache Access Total: 2605059457 Cache Hit Ratio: 79% 2072273907 [Defined State for buffer] Cache Miss Ratio: 20% 532785550 [Undefined State for Buffer] REAL Hit Ratio: 72% 1898031795 [MRU/MFU Hits Only] Data Demand Efficiency: 91% Data Prefetch Efficiency: 1% CACHE HITS BY CACHE LIST: Anon: 7% 158713873 [ New Customer, First Cache Hit ] Most Recently Used: 55% 1154441329 (mru) [ Return Customer ] Most Frequently Used: 35% 743590466 (mfu) [ Frequent Customer ] Most Recently Used Ghost: 0% 11032454 (mru_ghost) [ Return Customer Evicted, Now Back ] Most Frequently Used Ghost: 0% 4495785 (mfu_ghost) [ Frequent Customer Evicted, Now Back ] CACHE HITS BY DATA TYPE: Demand Data: 10% 226614173 Prefetch Data: 0% 519707 Demand Metadata: 80% 1669964501 Prefetch Metadata: 8% 175175526 CACHE MISSES BY DATA TYPE: Demand Data: 3% 20563634 Prefetch Data: 9% 50227920 Demand Metadata: 79% 422689982 Prefetch Metadata: 7% 39304014 --------------------------------------------- From owner-freebsd-fs@freebsd.org Tue Apr 26 17:25:20 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8E0B1B1D303 for ; Tue, 26 Apr 2016 17:25:20 +0000 (UTC) (envelope-from brandon.wandersee@gmail.com) Received: from mail-io0-f196.google.com (mail-io0-f196.google.com [209.85.223.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 65D9F1026 for ; Tue, 26 Apr 2016 17:25:20 +0000 (UTC) (envelope-from brandon.wandersee@gmail.com) Received: by mail-io0-f196.google.com with SMTP id k129so3009650iof.3 for ; Tue, 26 Apr 2016 10:25:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:references:user-agent:from:to:cc:subject :in-reply-to:date:message-id:mime-version; bh=Pw1um7OqUWHx6jgNb1q2I2T9sb1HhvnRkQf5HNPGFWo=; b=Hy+SjD8qjLemvohMxMOoBRLblYSInlvPHge+vVhLsSfRPNY+58EFfjUQR/LFL8y95A QmO9oSAKTD1wOADZ9KsnMvpHbgTg6dLc8zxo58T4PJzTdOso1zbYoxL+BMm6YIoHTF9e Y7O4N+ARELOhsuRpwF+asyoC+LqkvjfjNgYRKwaxNZu/jZtTiuqprKs8KBleerXQqtyo +wkUAFV6flUlujHHuqZtrcN90eNA+u5UUiZFI7hEzrDdbuc4aC/eoBQDrH2Ws4mols4r AkJtMxaebLo+Wo+jm56ySaGkm3eM9cc6iVk59r/v3bp/mSwFsBFY68jQg29MPjYMUGrj UAIw== X-Gm-Message-State: AOPr4FVAobkHMLrsOLU2mL0rCIkFQEXHtS8t69jRptVpXbfy+1oAJAewWwjwnpgAZwBcaw== X-Received: by 10.107.15.15 with SMTP id x15mr5008595ioi.139.1461690720480; Tue, 26 Apr 2016 10:12:00 -0700 (PDT) Received: from WorkBox.Home.gmail.com (63-231-164-200.mpls.qwest.net. [63.231.164.200]) by smtp.gmail.com with ESMTPSA id cy7sm2045879igc.17.2016.04.26.10.11.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Apr 2016 10:11:59 -0700 (PDT) References: <571F62AD.6080005@quip.cz> <571F687D.8040103@internetx.com> <571F6EA4.90800@quip.cz> <571F7AED.2040500@quip.cz> User-agent: mu4e 0.9.16; emacs 24.5.1 From: Brandon J. Wandersee To: Miroslav Lachman <000.fbsd@quip.cz> Cc: krad , FreeBSD FS Subject: Re: How to speed up slow zpool scrub? In-reply-to: <571F7AED.2040500@quip.cz> Date: Tue, 26 Apr 2016 12:11:57 -0500 Message-ID: <86lh409twi.fsf@WorkBox.Home> MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 17:25:20 -0000 Miroslav Lachman writes: > krad wrote on 04/26/2016 16:02: >> Erk, i would try to move your system off those data disks as you have >> two pools competing for the disk spindles. This is never ideal. You can >> by all means backup your os to those data pools but keep them on >> separate physical mediums. A couple of small SSD would do the trick >> nicely and could probably be added with no down time. You would probably >> want to find a suitable window though to make sure the box reboots >> nicely though. > > The system pool is really small - only 15GB and scrub is done relatively > fast. This machine cannot handle additional disks so I cannot move > system to other devices anyway. I tried system on USB flashdisk (read > only) in the past but it was slow and USB disk broke early. > I see no sign that the partitions are encrypted, so I can't think of a reason that the system and data can't reside on the same pool. The physical effort of the read arms on the disks isn't the only resource to consider---each pool has its own ARC, for example, and if you have compression enabled each pool will compete for CPU cycles. Having two pools grabbing at such limited resources is going to noticeably hurt performance. All that said, running even a single relative large pool on a low-powered dual-core CPU and 5G of RAM will noticeably limit *every* ZFS operation. I would bet that while you might be able to shave some time off the scrub, at best it's still going to take at least an entire day on that hardware. As a side note, I'm not so sure it's a good idea to use two different redundancy methods on the same set of disks, though I can't say exactly what the downsides would be (if any). It just seems odd. -- :: Brandon J. Wandersee :: brandon.wandersee@gmail.com :: -------------------------------------------------- :: 'The best design is as little design as possible.' :: --- Dieter Rams ---------------------------------- From owner-freebsd-fs@freebsd.org Tue Apr 26 18:29:47 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 15458B1C3C3 for ; Tue, 26 Apr 2016 18:29:47 +0000 (UTC) (envelope-from dhutch9999@yahoo.com) Received: from nm39-vm3.bullet.mail.bf1.yahoo.com (nm39-vm3.bullet.mail.bf1.yahoo.com [72.30.239.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BC22D1EB9 for ; Tue, 26 Apr 2016 18:29:46 +0000 (UTC) (envelope-from dhutch9999@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1461695277; bh=cAECIz3sS7ZHfh76X2bEizxg14lHClo3udXIMDrjHEQ=; h=Date:From:Reply-To:To:Cc:Subject:References:From:Subject; b=fPtioqpSOSMm9RCRLQ6PiGS0cC87gYQIiCcKbbSVSjsfQ86wn72BRxeOZEgOP4j7JJ96VUTv6rq1cf5ME/9FAQe+zRrTJSmpMjp8PdU0V3ubw5pFVrDGoJn9PVOMalv8DyaIw9AtOWa3c4mjepx1rxmU06RGUCKqdfXqoQ0CU+WbRHe1FYgQQvumtvwy2sjgbOWmxhm2LlC3YOV8JYx8ZUyJtGbh/cvI+05hxNeAvCtu507mYw/4U/Z6QkRuUu5roQx9C+nyMQ4liJO4UEh5sMH43hqCo6RBCmGKDgULuVcnvnE8Vi22UImIIrvLSb4O1PnUdHzy6JEQZjvhxY5M7w== Received: from [66.196.81.170] by nm39.bullet.mail.bf1.yahoo.com with NNFMP; 26 Apr 2016 18:27:57 -0000 Received: from [98.139.212.214] by tm16.bullet.mail.bf1.yahoo.com with NNFMP; 26 Apr 2016 18:27:57 -0000 Received: from [127.0.0.1] by omp1023.mail.bf1.yahoo.com with NNFMP; 26 Apr 2016 18:27:57 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 956300.36693.bm@omp1023.mail.bf1.yahoo.com X-YMail-OSG: majWoTwVM1m28eFdd1psPijiTDCTAZlseFBBEianfaRa.TMOo16QFUJ0PZHobDY l1sQuHZOWDOi87yNE5.FjWRXuAX8gxWV6aCYNJsatqeuic.kQz0PTBWdJYu4l8Jcrk48Ve7XZSuj 36EyV5pr8B0UGSx92jH_MwVjCQ.r5oMd3z1c3BbSuQP16vphzziw2ItlL5IPtBOGwX9R9IFfPLMn pKHw4.VzCTr6CUjnDZfuAJOqC.7otfjqs6u5JPrIe6QgNwtUurX_ate5T3TuUxhHk2HrGzVgIfu0 uKEjf1GA5l.HUqdRzbIARLV.ofAVexEdCnavWVCrYKKUR.ejj1CINPjCn5zWryoEZvCD2j3NTbWG GbMStCG2hqgu8ePo0w.jK98Kl1ZvSCX2CA0NPW4kiTD2wr2jivyboK1RamDP8VtizdV7qT8tubdu ihyN79jKNuZdeTF5DwcpEqJ.8_i50OF0CQqTwkCDIFVCbfBsyFY8ZtuTwrdKXo1yKkIPKW8xY_QG U248fhK6cf_.f Received: by 76.13.27.132; Tue, 26 Apr 2016 18:27:57 +0000 Date: Tue, 26 Apr 2016 18:27:57 +0000 (UTC) From: DH Reply-To: DH To: Miroslav Lachman <000.fbsd@quip.cz> Cc: Message-ID: <381846248.2672053.1461695277122.JavaMail.yahoo@mail.yahoo.com> Subject: Re: How to speed up slow zpool scrub? MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit References: <381846248.2672053.1461695277122.JavaMail.yahoo.ref@mail.yahoo.com> X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 18:29:47 -0000 Take a long look at the FreeNAS documentation and discussion threads - if you're not using a sufficient quantity of ECC ram with ZFS you're asking for anomalous behavior. I'm not a ZFS expert, but the documentation leads me to think inadequate system ram is certainly a consideration. FreeNAS requires a minimum of 8 GB of ECC ram when used with ZFS. David Hutchens III System Administrator -------------------------------------------- On Tue, 4/26/16, Miroslav Lachman <000.fbsd@quip.cz> wrote: Subject: Re: How to speed up slow zpool scrub? To: "DH" , freebsd-fs@freebsd.org Date: Tuesday, April 26, 2016, 9:34 AM DH wrote on 04/26/2016 17:47: >> 5GB of RAM > > That seems to be an insufficient amount of system ram when employing zfs. > > Take a look at this: > > http://doc.freenas.org/9.3/freenas_intro.html#ram I know 5GB is not much these days but is memory used for scrubbing a lot? Because I am satisfied with working performance. The only concern is slow scrubbing and I am not sure that more memory helps in this case. Miroslav Lachman From owner-freebsd-fs@freebsd.org Tue Apr 26 20:36:51 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 918A8B1D990 for ; Tue, 26 Apr 2016 20:36:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8208016BF for ; Tue, 26 Apr 2016 20:36:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u3QKapE3066009 for ; Tue, 26 Apr 2016 20:36:51 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204643] [msdosfs] [panic] Crash while accessing files with large, non-english names Date: Tue, 26 Apr 2016 20:36:51 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.3-RELEASE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 20:36:51 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D204643 --- Comment #6 from commit-hook@freebsd.org --- A commit references this bug: Author: kp Date: Tue Apr 26 20:36:32 UTC 2016 New revision: 298664 URL: https://svnweb.freebsd.org/changeset/base/298664 Log: msdosfs: Prevent buffer overflow when expanding win95 names In win2unixfn() we expand Windows 95 style long names. In some cases that requires moving the data in the nbp->nb_buf buffer backwards to make room. That code failed to check for overflows, leading to a stack overflow in win2unixfn(). We now check for this event, and mark the entire conversion as failed in = that case. This means we present the 8 character, dos style, name instead. PR: 204643 Differential Revision: https://reviews.freebsd.org/D6015 Changes: head/sys/fs/msdosfs/direntry.h head/sys/fs/msdosfs/msdosfs_conv.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Apr 26 21:32:54 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0A1D8B1DCC8 for ; Tue, 26 Apr 2016 21:32:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EECFA143C for ; Tue, 26 Apr 2016 21:32:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u3QLWrYG080614 for ; Tue, 26 Apr 2016 21:32:53 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204643] [msdosfs] [panic] Crash while accessing files with large, non-english names Date: Tue, 26 Apr 2016 21:32:53 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.3-RELEASE X-Bugzilla-Keywords: patch, security X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 21:32:54 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D204643 Garrett Cooper,425-314-3911 changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |security CC| |ngie@FreeBSD.org, | |secteam@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Apr 26 22:26:55 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0DE61B1CF8A for ; Tue, 26 Apr 2016 22:26:55 +0000 (UTC) (envelope-from eborisch@gmail.com) Received: from mail-oi0-x234.google.com (mail-oi0-x234.google.com [IPv6:2607:f8b0:4003:c06::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CC01D1CB0 for ; Tue, 26 Apr 2016 22:26:54 +0000 (UTC) (envelope-from eborisch@gmail.com) Received: by mail-oi0-x234.google.com with SMTP id r78so30963259oie.0 for ; Tue, 26 Apr 2016 15:26:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=Sg/RTPirXqSwqy7NyiqRx00YtgpAf+3Or26LUwDpy5Y=; b=vOlJs7W7EocObdq1ohRsdhqcQoJLzF/P8lc1Jx+QpqhJLqv1rZ2FGi06pTc/JVax4y BcLbYFiN50bzBZHpUYglbLxE6iJVS/fdyaur+ixQ327pkjnmxd1anzSjg/f2DFO5abpR u8x+5It8BeECJtLeEz3MXwwaCaoXZEOyI9OepESPL02jDjtpOMkxoSbqivb/gsEqBXK8 Tyjcj1BLVWEzqCJ5/fawJOwcMJdyga9aMAwcLsDGn1GvvZ5TupPZeLJ7+Ko2/gSniDFL PsiQwtKnLp+r8qTpkmRcSGjRNQiLIbIwaqGlKkRySqchd+GHHkFG8EPkpv71lFcBX3D4 YB4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=Sg/RTPirXqSwqy7NyiqRx00YtgpAf+3Or26LUwDpy5Y=; b=c9G98q1lz7h7J1cwdVc/fRQy9nT2onyZGJkH3stbYuOp+thaOdFEF5ejje11ruWxvD Hu/PMXgYniCnwGdYpaAnYFUZz5Z4ZA/82lYA4kLX5nHtVv2ytjPvNOw0WCtL2Hp5ZODF gZAjfQc0RbYI8MUAEO4DHzZlw5BswspFCGF6E3WpC86fp0vIOHlvb5f8qJ/nEpUI19zl FVoNjWHlNdDC9pLG/LNYs7WCLnciraL4rfpIsnfcf7yRsugPdCLS7T3Ny9wE4bGy6Keb XVMwzvTdkhdLa/+rDJ96ES6Vr/GEYPGuAggiNV/+cLEbfHu6tG4ELX/QjqRvcDCmnEcl hdSA== X-Gm-Message-State: AOPr4FVlLx4d7XNrAvP83ry++NBJjZYypvWD+CE4MDGxGwxzzUhX6KEzVzNT0c/vBCwG1y2AC+MJalMv7mskEA== MIME-Version: 1.0 X-Received: by 10.202.189.10 with SMTP id n10mr1911273oif.101.1461709614078; Tue, 26 Apr 2016 15:26:54 -0700 (PDT) Received: by 10.60.52.140 with HTTP; Tue, 26 Apr 2016 15:26:53 -0700 (PDT) In-Reply-To: <571F62AD.6080005@quip.cz> References: <571F62AD.6080005@quip.cz> Date: Tue, 26 Apr 2016 17:26:53 -0500 Message-ID: Subject: Re: How to speed up slow zpool scrub? From: "Eric A. Borisch" To: Miroslav Lachman <000.fbsd@quip.cz> Cc: "freebsd-fs@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 22:26:55 -0000 On Tue, Apr 26, 2016 at 7:44 AM, Miroslav Lachman <000.fbsd@quip.cz> wrote: > Hi, > > is there any way to make zpool scrub faster? > We have one older machine with CPU Pentium(R) Dual E2160 @1.80GHz, 5GB of > RAM and 4x 4TB HDDs. It is just a storage for backups for about 20 machines. > Scrub is scheduled from periodic each 30 days but it takes about 4 days to > complete and everything during scrub is slow. Backups takes 8 hours instead > of 5 (made by rsync), deleting of old files is even more slower. > > The backups are made every night from the midnight to morning, the machine > is idle for the rest of the day. > > Is there any tuning to make scrub faster in this idle time? > Or is it better to do it other way - slower scrub with even lower priority > taking for about one week but not affecting time of normal operations? (is > it dangerous to have scrub running this long or reboot machine during the > scrub?) It's likely easier to make scrub slower (but rsync faster) when the system is not idle, but hopefully no slower otherwise. I would start with increasing the vfs.zfs.scrub_delay (in ticks; typically 1/1000 of a second) settings to reduce scrub's impact to other IO this without impacting resilvers. (Expect longer total time as it will scrub less during rsyncs.) You can also increase vfs.zfs.scan_idle if scrub_delay isn't enough, but do note that vfs.zfs.scan_idle impacts resilvers, too -- so make sure you don't also increase vfs.zfs.resilver_delay, or you may get very slow resilvers during IO. The logic for this is in dsl_scan.c [1] -- look there for the final word on what the different parameters do; there are a number of informative comments throughout the file if you are not into the code itself. See also vfs.zfs.vdev.*_max_active and vfs.zfs.vdev.*_min_active, if you are really want more knobs to try. It's might also be worth trying '/usr/local/bin/perl /usr/share/dtrace/toolkit/hotkernel' to see what the kernel is up to... - Eric [1] https://svnweb.freebsd.org/base/releng/10.3/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c?view=markup or https://github.com/freebsd/freebsd/blob/releng/10.3/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c From owner-freebsd-fs@freebsd.org Tue Apr 26 22:27:19 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1F227B1CFEF for ; Tue, 26 Apr 2016 22:27:19 +0000 (UTC) (envelope-from andyf@andyit.com.au) Received: from alpine.spintel.net.au (alpine.spintel.net.au [IPv6:2407:e400:1::b]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CEC081D41 for ; Tue, 26 Apr 2016 22:27:18 +0000 (UTC) (envelope-from andyf@andyit.com.au) Received: from hummer.af.speednet.com.au (115-69-4-237.dyn.comcen.net.au [115.69.4.237]) by alpine.spintel.net.au (Postfix) with ESMTPS id 406AE4C052D for ; Wed, 27 Apr 2016 08:27:06 +1000 (AEST) Received: from snuggles.af.speednet.com.au (snuggles.af.speednet.com.au [172.22.2.2]) by hummer.af.speednet.com.au (8.14.5/8.14.5) with ESMTP id u3QMR0L0061507 for ; Wed, 27 Apr 2016 08:27:01 +1000 (EST) (envelope-from andyf@andyit.com.au) Message-ID: <571FEB34.7040305@andyit.com.au> Date: Wed, 27 Apr 2016 08:27:00 +1000 From: Andy Farkas User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:13.0) Gecko/20120614 Thunderbird/13.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: How to speed up slow zpool scrub? References: <698816653.2698619.1461685653634.JavaMail.yahoo.ref@mail.yahoo.com> <698816653.2698619.1461685653634.JavaMail.yahoo@mail.yahoo.com> <571F9897.2070008@quip.cz> In-Reply-To: <571F9897.2070008@quip.cz> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 22:27:19 -0000 On 27/04/2016 02:34, Miroslav Lachman wrote: > DH wrote on 04/26/2016 17:47: >>> 5GB of RAM >> >> That seems to be an insufficient amount of system ram when employing >> zfs. >> >> Take a look at this: >> >> http://doc.freenas.org/9.3/freenas_intro.html#ram > > I know 5GB is not much these days but is memory used for scrubbing a > lot? Because I am satisfied with working performance. The only concern > is slow scrubbing and I am not sure that more memory helps in this case. > CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz (3000.02-MHz 686-class CPU) real memory = 4294967296 (4096 MB) avail memory = 3404075008 (3246 MB) ACPI APIC Table: ad4: 3815447MB at ata2-master UDMA100 SATA ad5: 3815447MB at ata2-slave UDMA100 SATA ad6: 3815447MB at ata3-master UDMA100 SATA ad7: 3815447MB at ata3-slave UDMA100 SATA NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT z 14.6T 10.1T 4.50T 69% 1.00x ONLINE - scan: scrub repaired 43K in 25h15m with 0 errors <-- NAME STATE READ WRITE CKSUM z ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ad6 ONLINE 0 0 0 ad4 ONLINE 0 0 0 ad5 ONLINE 0 0 0 ad7 ONLINE 0 0 0 -andyf From owner-freebsd-fs@freebsd.org Wed Apr 27 03:24:45 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0670DB1DCFF for ; Wed, 27 Apr 2016 03:24:45 +0000 (UTC) (envelope-from paul@pk1048.com) Received: from cpanel61.fastdnsservers.com (server61.fastdnsservers.com [216.51.232.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CA8821DBA for ; Wed, 27 Apr 2016 03:24:44 +0000 (UTC) (envelope-from paul@pk1048.com) Received: from pool-100-4-209-221.albyny.fios.verizon.net ([100.4.209.221]:64521 helo=[192.168.2.133]) by cpanel61.fastdnsservers.com with esmtpsa (TLSv1:ECDHE-RSA-AES256-SHA:256) (Exim 4.86_1) (envelope-from ) id 1avFIR-002r4P-7t; Tue, 26 Apr 2016 21:33:47 -0500 Subject: Re: How to speed up slow zpool scrub? Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Content-Type: text/plain; charset=utf-8 From: PK1048 In-Reply-To: <571FEB34.7040305@andyit.com.au> Date: Tue, 26 Apr 2016 22:33:44 -0400 Cc: freebsd-fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <56C0A956-F134-4A8D-A8B6-B93DCA045BE4@pk1048.com> References: <698816653.2698619.1461685653634.JavaMail.yahoo.ref@mail.yahoo.com> <698816653.2698619.1461685653634.JavaMail.yahoo@mail.yahoo.com> <571F9897.2070008@quip.cz> <571FEB34.7040305@andyit.com.au> To: Andy Farkas X-Mailer: Apple Mail (2.3124) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cpanel61.fastdnsservers.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - pk1048.com X-Get-Message-Sender-Via: cpanel61.fastdnsservers.com: authenticated_id: info@pk1048.com X-Authenticated-Sender: cpanel61.fastdnsservers.com: info@pk1048.com X-Source: X-Source-Args: X-Source-Dir: X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 03:24:45 -0000 > On Apr 26, 2016, at 18:27, Andy Farkas wrote: >=20 > On 27/04/2016 02:34, Miroslav Lachman wrote: >> DH wrote on 04/26/2016 17:47: >>>> 5GB of RAM >>>=20 >>> That seems to be an insufficient amount of system ram when employing = zfs. >>>=20 >>> Take a look at this: >>>=20 >>> http://doc.freenas.org/9.3/freenas_intro.html#ram >>=20 >> I know 5GB is not much these days but is memory used for scrubbing a = lot? Because I am satisfied with working performance. The only concern = is slow scrubbing and I am not sure that more memory helps in this case. I don=E2=80=99t expect memory to make a big difference in scrub or = resilver performance. Rememeber the way ZFS uses memory is as both write = buffer and read cache (all in the ARC). So insufficient memory will hurt = real performance but it should not have any real effect on a scrub or = resilver (which are both operations that read all the data that has been = written to the zpool and check it against the metadata for consistency). Scrubs (and resilver) operations are essentially all random I/O. Those = drives are low end, low performance, desktop drives. The fact that the scrub _repaired_ anything means that there was damage = to data. If all of the data on the drives is good, then a scrub has = nothing to repair. What does an `iostat -x 1` show during the scrub ? How about a `zpool = iostat -v 1` ? How hard are you hitting those drives and are they all = really good ? I have seen svc_t values differ by a factor of two among = drives of all the same make and model. Is one drive slower than the rest = ? Perhaps that drive is on the way out. From owner-freebsd-fs@freebsd.org Wed Apr 27 04:46:27 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B272B1CB9C for ; Wed, 27 Apr 2016 04:46:27 +0000 (UTC) (envelope-from andyf@andyit.com.au) Received: from bmrg.org.au (11.173.70.115.static.exetel.com.au [115.70.173.11]) by mx1.freebsd.org (Postfix) with SMTP id 52ACD19E5 for ; Wed, 27 Apr 2016 04:46:24 +0000 (UTC) (envelope-from andyf@andyit.com.au) Received: (qmail 20084 invoked by uid 453); 27 Apr 2016 04:39:40 -0000 Received: from autodiscover.bmrg.org.au (HELO bmrg.org.au) (192.168.16.8) by bmrg.org.au (qpsmtpd/0.84) with (AES128-SHA encrypted) ESMTPS; Wed, 27 Apr 2016 14:39:40 +1000 Received: from Workstation117 (192.168.16.86) by BMRGSERVER.BurnettMary.local (192.168.16.8) with Microsoft SMTP Server id 8.3.348.2; Wed, 27 Apr 2016 14:39:44 +1000 From: Andy Farkas To: "'PK1048'" CC: References: <698816653.2698619.1461685653634.JavaMail.yahoo.ref@mail.yahoo.com> <698816653.2698619.1461685653634.JavaMail.yahoo@mail.yahoo.com> <571F9897.2070008@quip.cz> <571FEB34.7040305@andyit.com.au> <56C0A956-F134-4A8D-A8B6-B93DCA045BE4@pk1048.com> In-Reply-To: <56C0A956-F134-4A8D-A8B6-B93DCA045BE4@pk1048.com> Subject: RE: How to speed up slow zpool scrub? Date: Wed, 27 Apr 2016 14:39:43 +1000 Message-ID: <084201d1a03e$d2158fe0$7640afa0$@andyit.com.au> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Outlook 15.0 Thread-Index: AQHf0Y2E12vvbMASO5SJnzpSfT6WegBQMveTAsPburICJcZ0lQHSTNxjn0hzDvA= Content-Language: en-au X-Virus-Checked: Checked by ClamAV on bmrg.org.au X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 04:46:27 -0000 > -----Original Message----- > From: PK1048 [mailto:paul@pk1048.com] > Sent: Wednesday, 27 April 2016 12:34 PM > To: Andy Farkas > Cc: freebsd-fs@freebsd.org > Subject: Re: How to speed up slow zpool scrub? > > ... > Scrubs (and resilver) operations are essentially all random I/O. Those > drives are low end, low performance, desktop drives. Yes, the system is an old low end, low performance desktop. That was my point, that it took 25 hours to scrub 7.52T and not 4 days like the OP is saying. -andyf This message was scanned using Kaspersky Enpoint Security. From owner-freebsd-fs@freebsd.org Wed Apr 27 05:48:57 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5D30B1D9CC for ; Wed, 27 Apr 2016 05:48:57 +0000 (UTC) (envelope-from ike@michaeleichorn.com) Received: from mx1.eichornenterprises.com (mx1.eichornenterprises.com [104.236.13.122]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.eichornenterprises.com", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 602711A2C for ; Wed, 27 Apr 2016 05:48:56 +0000 (UTC) (envelope-from ike@michaeleichorn.com) Received: from smtp.eichornenterprises.com (cpe-184-59-147-149.neo.res.rr.com [184.59.147.149]) by mx1.eichornenterprises.com (OpenSMTPD) with ESMTP id 55f15f7d; Wed, 27 Apr 2016 01:48:48 -0400 (EDT) Received: by smtp.eichornenterprises.com (OpenSMTPD) with ESMTPSA id 67b44cde TLS version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO; Wed, 27 Apr 2016 01:48:47 -0400 (EDT) Message-ID: <1461736217.1121.17.camel@michaeleichorn.com> Subject: Re: How to speed up slow zpool scrub? From: "Michael B. Eichorn" To: DH , Miroslav Lachman <000.fbsd@quip.cz> Cc: freebsd-fs@freebsd.org Date: Wed, 27 Apr 2016 01:50:17 -0400 In-Reply-To: <381846248.2672053.1461695277122.JavaMail.yahoo@mail.yahoo.com> References: <381846248.2672053.1461695277122.JavaMail.yahoo.ref@mail.yahoo.com> <381846248.2672053.1461695277122.JavaMail.yahoo@mail.yahoo.com> Content-Type: multipart/signed; micalg="sha-256"; protocol="application/x-pkcs7-signature"; boundary="=-jOW/Jn9dWs3bk6QDmUct" X-Mailer: Evolution 3.20.1 Mime-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 05:48:57 -0000 --=-jOW/Jn9dWs3bk6QDmUct Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, 2016-04-26 at 18:27 +0000, DH via freebsd-fs wrote: > Take a long look at the FreeNAS documentation and discussion threads > - if you're not using a sufficient quantity of ECC ram with ZFS > you're asking for anomalous behavior.=C2=A0=C2=A0I'm not a ZFS expert, bu= t the > documentation leads me to think inadequate system ram is certainly a > consideration.=C2=A0=C2=A0 >=20 > FreeNAS requires a minimum of 8 GB of ECC ram when used with ZFS. It does not *need* to be ECC ram, ECC is just *highly recommended*. As one of the key features of zfs is bitrot prevention, it makes sense to protect against bitrot everywhere. Zfs (and thus freenas) is just fine with non-ecc ram. Just, like for any filesystem if the bit is flipped in ram it will be recorded as such on disk. And 8 GB is probably a safe minumum without tuning, but there are sufficent tunnables to do ~4 GB easily. This is not saying that you are incorrect about best practice, I am just picking nits since ECC and minium RAM are can be one of the more sensitive topics for ZFS-enthusiasts. --=-jOW/Jn9dWs3bk6QDmUct Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Disposition: attachment; filename="smime.p7s" Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCEqAw ggYwMIIFGKADAgECAgMOXcYwDQYJKoZIhvcNAQELBQAwgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQK Ew1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWdu aW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFByaW1hcnkgSW50ZXJtZWRpYXRlIENsaWVu dCBDQTAeFw0xNTA2MTMyMDI0NDZaFw0xNjA2MTQwMDM1NTBaMEgxHzAdBgNVBAMMFmlrZUBtaWNo YWVsZWljaG9ybi5jb20xJTAjBgkqhkiG9w0BCQEWFmlrZUBtaWNoYWVsZWljaG9ybi5jb20wggEi MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDJVdWALPz5h2s5zUQGIJYl6Vp8FPtZNko8q/3s crCsxXJLprMaDdpnqTsmkbmEfKvsqPQE6HVOpGxVRTl/tCm+VvouW9eY9ITMigb1OnHdU13CKO0j drgeU1nHst0qxwsIofRD7nC4dakT6exnrVndlBmLrf/bLPh2qOM8YK5qKK6m33fE7AyYrwiYAWFT 3fERI7LakjaabrIoS/Y1rCdL5FaCTMOlRbZyduc8HkrgjT2JW+i4fVcKyGL5gExBJWfS3q1uGFaB ie6pYtl8lZPtvN0JSfibP003RBoLgzqHJKW91RL0qNeDjKZi/5nrlU398l9UoVvLLO3KxoPBXKCx AgMBAAGjggLcMIIC2DAJBgNVHRMEAjAAMAsGA1UdDwQEAwIEsDAdBgNVHSUEFjAUBggrBgEFBQcD AgYIKwYBBQUHAwQwHQYDVR0OBBYEFJZqarc6CcrOs6eAwOgrMznk5ZWWMB8GA1UdIwQYMBaAFFNy 7ZKc4NrLAVx8fpY1TvLUuFGCMCEGA1UdEQQaMBiBFmlrZUBtaWNoYWVsZWljaG9ybi5jb20wggFM BgNVHSAEggFDMIIBPzCCATsGCysGAQQBgbU3AQIDMIIBKjAuBggrBgEFBQcCARYiaHR0cDovL3d3 dy5zdGFydHNzbC5jb20vcG9saWN5LnBkZjCB9wYIKwYBBQUHAgIwgeowJxYgU3RhcnRDb20gQ2Vy dGlmaWNhdGlvbiBBdXRob3JpdHkwAwIBARqBvlRoaXMgY2VydGlmaWNhdGUgd2FzIGlzc3VlZCBh Y2NvcmRpbmcgdG8gdGhlIENsYXNzIDEgVmFsaWRhdGlvbiByZXF1aXJlbWVudHMgb2YgdGhlIFN0 YXJ0Q29tIENBIHBvbGljeSwgcmVsaWFuY2Ugb25seSBmb3IgdGhlIGludGVuZGVkIHB1cnBvc2Ug aW4gY29tcGxpYW5jZSBvZiB0aGUgcmVseWluZyBwYXJ0eSBvYmxpZ2F0aW9ucy4wNgYDVR0fBC8w LTAroCmgJ4YlaHR0cDovL2NybC5zdGFydHNzbC5jb20vY3J0dTEtY3JsLmNybDCBjgYIKwYBBQUH AQEEgYEwfzA5BggrBgEFBQcwAYYtaHR0cDovL29jc3Auc3RhcnRzc2wuY29tL3N1Yi9jbGFzczEv Y2xpZW50L2NhMEIGCCsGAQUFBzAChjZodHRwOi8vYWlhLnN0YXJ0c3NsLmNvbS9jZXJ0cy9zdWIu Y2xhc3MxLmNsaWVudC5jYS5jcnQwIwYDVR0SBBwwGoYYaHR0cDovL3d3dy5zdGFydHNzbC5jb20v MA0GCSqGSIb3DQEBCwUAA4IBAQB4K8iQw+0FRn3xEnB3vIIu2Vi4C3ZGnOMWP90FFXLrZ6uAu9AK xVCjXUVP6nAEsOopTMu769vVecdBvg0KO2i5aTDTdTLX4g9d020g4OLWW1NiynAkX8oKqJLqZ53q vHK4zP4KWPS3bSqDWVCosTMfI+H6tkg+6G3gS0HHoHTLKZhIT3z6PQZAfeofM7ed6NOdAcj0J2lP ODHzzz7Y9x4wMwYJdidorzUDVYkNIkim8ak7hK9F60NadA5w/BirFATSlzRyV0h1tl6oNisEaQcq tGvy6UoCTDhzaJ7pQValfDXJ/A47P0hNj/CX/PmkY1wQHsEJz2pbh5lqteP/fO0rMIIGMDCCBRig AwIBAgIDDl3GMA0GCSqGSIb3DQEBCwUAMIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRD b20gTHRkLjErMCkGA1UECxMiU2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYG A1UEAxMvU3RhcnRDb20gQ2xhc3MgMSBQcmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0EwHhcN MTUwNjEzMjAyNDQ2WhcNMTYwNjE0MDAzNTUwWjBIMR8wHQYDVQQDDBZpa2VAbWljaGFlbGVpY2hv cm4uY29tMSUwIwYJKoZIhvcNAQkBFhZpa2VAbWljaGFlbGVpY2hvcm4uY29tMIIBIjANBgkqhkiG 9w0BAQEFAAOCAQ8AMIIBCgKCAQEAyVXVgCz8+YdrOc1EBiCWJelafBT7WTZKPKv97HKwrMVyS6az Gg3aZ6k7JpG5hHyr7Kj0BOh1TqRsVUU5f7Qpvlb6LlvXmPSEzIoG9Tpx3VNdwijtI3a4HlNZx7Ld KscLCKH0Q+5wuHWpE+nsZ61Z3ZQZi63/2yz4dqjjPGCuaiiupt93xOwMmK8ImAFhU93xESOy2pI2 mm6yKEv2NawnS+RWgkzDpUW2cnbnPB5K4I09iVvouH1XCshi+YBMQSVn0t6tbhhWgYnuqWLZfJWT 7bzdCUn4mz9NN0QaC4M6hySlvdUS9KjXg4ymYv+Z65VN/fJfVKFbyyztysaDwVygsQIDAQABo4IC 3DCCAtgwCQYDVR0TBAIwADALBgNVHQ8EBAMCBLAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUF BwMEMB0GA1UdDgQWBBSWamq3OgnKzrOngMDoKzM55OWVljAfBgNVHSMEGDAWgBRTcu2SnODaywFc fH6WNU7y1LhRgjAhBgNVHREEGjAYgRZpa2VAbWljaGFlbGVpY2hvcm4uY29tMIIBTAYDVR0gBIIB QzCCAT8wggE7BgsrBgEEAYG1NwECAzCCASowLgYIKwYBBQUHAgEWImh0dHA6Ly93d3cuc3RhcnRz c2wuY29tL3BvbGljeS5wZGYwgfcGCCsGAQUFBwICMIHqMCcWIFN0YXJ0Q29tIENlcnRpZmljYXRp b24gQXV0aG9yaXR5MAMCAQEagb5UaGlzIGNlcnRpZmljYXRlIHdhcyBpc3N1ZWQgYWNjb3JkaW5n IHRvIHRoZSBDbGFzcyAxIFZhbGlkYXRpb24gcmVxdWlyZW1lbnRzIG9mIHRoZSBTdGFydENvbSBD QSBwb2xpY3ksIHJlbGlhbmNlIG9ubHkgZm9yIHRoZSBpbnRlbmRlZCBwdXJwb3NlIGluIGNvbXBs aWFuY2Ugb2YgdGhlIHJlbHlpbmcgcGFydHkgb2JsaWdhdGlvbnMuMDYGA1UdHwQvMC0wK6ApoCeG JWh0dHA6Ly9jcmwuc3RhcnRzc2wuY29tL2NydHUxLWNybC5jcmwwgY4GCCsGAQUFBwEBBIGBMH8w OQYIKwYBBQUHMAGGLWh0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbS9zdWIvY2xhc3MxL2NsaWVudC9j YTBCBggrBgEFBQcwAoY2aHR0cDovL2FpYS5zdGFydHNzbC5jb20vY2VydHMvc3ViLmNsYXNzMS5j bGllbnQuY2EuY3J0MCMGA1UdEgQcMBqGGGh0dHA6Ly93d3cuc3RhcnRzc2wuY29tLzANBgkqhkiG 9w0BAQsFAAOCAQEAeCvIkMPtBUZ98RJwd7yCLtlYuAt2RpzjFj/dBRVy62ergLvQCsVQo11FT+pw BLDqKUzLu+vb1XnHQb4NCjtouWkw03Uy1+IPXdNtIODi1ltTYspwJF/KCqiS6med6rxyuMz+Clj0 t20qg1lQqLEzHyPh+rZIPuht4EtBx6B0yymYSE98+j0GQH3qHzO3nejTnQHI9CdpTzgx888+2Pce MDMGCXYnaK81A1WJDSJIpvGpO4SvRetDWnQOcPwYqxQE0pc0cldIdbZeqDYrBGkHKrRr8ulKAkw4 c2ie6UFWpXw1yfwOOz9ITY/wl/z5pGNcEB7BCc9qW4eZarXj/3ztKzCCBjQwggQcoAMCAQICAR4w DQYJKoZIhvcNAQEFBQAwfTELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzAp BgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcxKTAnBgNVBAMTIFN0YXJ0 Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTA3MTAyNDIxMDE1NVoXDTE3MTAyNDIxMDE1 NVowgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1 cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAx IFByaW1hcnkgSW50ZXJtZWRpYXRlIENsaWVudCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC AQoCggEBAMcJg8zOLdgasSmkLhOrlr6KMoOMpohBllVHrdRvEg/q6r8jR+EK75xCGhR8ToREoqe7 zM9/UnC6TS2y9UKTpT1v7RSMzR0t6ndl0TWBuUr/UXBhPk+Kmy7bI4yW4urC+y7P3/1/X7U8ocb8 VpH/Clt+4iq7nirMcNh6qJR+xjOhV+VHzQMALuGYn5KZmc1NbJQYclsGkDxDz2UbFqE2+6vIZoL+ jb9x4Pa5gNf1TwSDkOkikZB1xtB4ZqtXThaABSONdfmv/Z1pua3FYxnCFmdr/+N2JLKutIxMYqQO Jebr/f/h5t95m4JgrM3Y/w7YX9d7YAL9jvN4SydHsU6n65cCAwEAAaOCAa0wggGpMA8GA1UdEwEB /wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBRTcu2SnODaywFcfH6WNU7y1LhRgjAf BgNVHSMEGDAWgBROC+8apEBbpRdphzDKNGhD0EGu8jBmBggrBgEFBQcBAQRaMFgwJwYIKwYBBQUH MAGGG2h0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbS9jYTAtBggrBgEFBQcwAoYhaHR0cDovL3d3dy5z dGFydHNzbC5jb20vc2ZzY2EuY3J0MFsGA1UdHwRUMFIwJ6AloCOGIWh0dHA6Ly93d3cuc3RhcnRz c2wuY29tL3Nmc2NhLmNybDAnoCWgI4YhaHR0cDovL2NybC5zdGFydHNzbC5jb20vc2ZzY2EuY3Js MIGABgNVHSAEeTB3MHUGCysGAQQBgbU3AQIBMGYwLgYIKwYBBQUHAgEWImh0dHA6Ly93d3cuc3Rh cnRzc2wuY29tL3BvbGljeS5wZGYwNAYIKwYBBQUHAgEWKGh0dHA6Ly93d3cuc3RhcnRzc2wuY29t L2ludGVybWVkaWF0ZS5wZGYwDQYJKoZIhvcNAQEFBQADggIBAAqDCH14qywGXLhjjF6uHLkjd02h cdh9hrw+VUsv+q1eeQWB21jWj3kJ96AUlPCoEGZ/ynJNScWy6QMVQjbbMXltUfO4n4bGGdKo3awP Wp61tjAFgraLJgDk+DsSvUD6EowjMTNx25GQgyYJ5RPIzKKR9tQW8gGK+2+RHxkUCTbYFnL6kl8C h507rUdPPipJ9CgJFws3kDS3gOS5WFMxcjO5DwKfKSETEPrHh7p5shuuNktvsv6hxHTLhiMKX893 gxdT3XLS9OKmCv87vkINQcNEcIIoFWbP9HORz9v3vQwR4e3ksLc2JZOAFK+ssS5XMEoznzpihEP0 PLc4dCBYjbvSD7kxgDwZ+Aj8Q9PkbvE9sIPP7ON0fz095HdThKjiVJe6vofq+n6b1NBc8XdrQvBm unwxD5nvtTW4vtN6VY7mUCmxsCieuoBJ9OlqmsVWQvifIYf40dJPZkk9YgGTzWLpXDSfLSplbY2L L9C9U0ptvjcDjefLTvqSFc7tw1sEhF0n/qpA2r0GpvkLRDmcSwVyPvmjFBGqUp/pNy8ZuPGQmHwF i2/14+xeSUDG2bwnsYJQG2EdJCB6luQ57GEnTA/yKZSTKI8dDQa8Sd3zfXb19mOgSF0bBdXbuKhE puP9wirslFe6fQ1t5j5R0xi72MZ8ikMu1RQZKCyDbMwazlHiMYIDfzCCA3sCAQEwgZQwgYwxCzAJ BgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRh bCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFByaW1hcnkg SW50ZXJtZWRpYXRlIENsaWVudCBDQQIDDl3GMA0GCWCGSAFlAwQCAQUAoIIBuzAYBgkqhkiG9w0B CQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjA0MjcwNTUwMTdaMC8GCSqGSIb3DQEJ BDEiBCBxtMpPZIquZKGaK7slgNIRXMXIXLwxj1dNMooXkV/3jDCBpQYJKwYBBAGCNxAEMYGXMIGU MIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMiU2VjdXJl IERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UEAxMvU3RhcnRDb20gQ2xhc3MgMSBQ cmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0ECAw5dxjCBpwYLKoZIhvcNAQkQAgsxgZeggZQw gYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUg RGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFBy aW1hcnkgSW50ZXJtZWRpYXRlIENsaWVudCBDQQIDDl3GMA0GCSqGSIb3DQEBAQUABIIBAJso4YC6 Gswg0/QM3HnF3wMlhzikjozvz775H/0sDah+yygr+VDOLrMWbETGs/yOQM25IULNa1R9neS21T7M JbctW4WcFPZtGnHXJ0cQfYOxSUuHIHHQZD1pN0wHjqouAC8f3g8MvE1O0XMNs73BNZlVYYactQN8 GqElzckKa8LjRKPEDSz73XDdLlFD3SkZmmoYEpOWC9FmYH9eN+sJnATDdVPv1OBJ+NlpBGADVIXm 6lkcFooDgvk+PvOFEskqFx+T394ibxT/iEcbZorirz/PGF6OUDadtzItqqpPiJtQg44PhPCA2/d2 5uADeqTNG/kVbScVw+oIWTzOFSbyTzwAAAAAAAA= --=-jOW/Jn9dWs3bk6QDmUct-- From owner-freebsd-fs@freebsd.org Wed Apr 27 06:26:34 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0600B1E097 for ; Wed, 27 Apr 2016 06:26:34 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-wm0-x230.google.com (mail-wm0-x230.google.com [IPv6:2a00:1450:400c:c09::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7EA651815 for ; Wed, 27 Apr 2016 06:26:34 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: by mail-wm0-x230.google.com with SMTP id n129so12165590wmn.1 for ; Tue, 26 Apr 2016 23:26:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=dRsilN6Lfg2CQnCJKacxf/2Uhc8cA/xER/DplSxNsDY=; b=KEzbLaFKUlcDCrTyVO1zxMqXMl7cC7EOK8mJjGK5xYbx7/bZKB8Mcfpewj9EyPIK1F cxjJhzU4V3vuo5JpQPCI+7dZD56dKPTdkh+m+9HmhpROPRQZH27Bg5HEtExcnCWbvqQQ Vw39bs1Re02EsvbxYA1VRTyfNPkwe/rH851odAeB7IZEI/aNXxNVTKKji6uF1zqymesI 5ioUET0d6Q0Oc2qenKVC3Uomj+KJAxyxzqzE1ErFVqpGCBNQhNeTt2MFNzDYFq+nWu7+ 5kix6Ehw7TuC9Ut4xcuxoyeolmdCFZbrmJGyLqisWCKXmu3gB4J9BgroBUBz+2hMetUs scUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=dRsilN6Lfg2CQnCJKacxf/2Uhc8cA/xER/DplSxNsDY=; b=g6juE/3WfDIyCDAJ7Z8nV399lmHNV2sNZYQFD2dj1jDeOzWbdvfPxd6roVF+RCvYic lM6+9m2o123zzJtvxTIJm9K9ahscXXiH7b9cpjEaFoyN2UOt51nkDyin4RjPfAIr3CT3 JU/id79h6HwbzCd0EYMACTKmkk2p44AkfNBWUNz8JdEg+HGfjFn7q62WfjlZZoH5jdki iAEoU4IkTgw7XBSA3IvRXY/fhb8PhOLgGe4JBv5/f9KBa/1xVmCgzP16FxBoNUukb7Ux AuEGDbCSky8YwAT1/GYOWTV4B9pT9vHtLapUvs9M9kKAXisfKRG8/krJVBTK3K52LpsI aaIw== X-Gm-Message-State: AOPr4FVWAByfRSEOhdM4cLuDlaf+t/Uok6cUe9ilPGEwbrtEvdaSRzaXv6AgqgJgor1s5jNFUb158YIGn6Tb8A== MIME-Version: 1.0 X-Received: by 10.194.48.7 with SMTP id h7mr7776896wjn.81.1461738393186; Tue, 26 Apr 2016 23:26:33 -0700 (PDT) Received: by 10.28.26.17 with HTTP; Tue, 26 Apr 2016 23:26:33 -0700 (PDT) In-Reply-To: <571F7AED.2040500@quip.cz> References: <571F62AD.6080005@quip.cz> <571F687D.8040103@internetx.com> <571F6EA4.90800@quip.cz> <571F7AED.2040500@quip.cz> Date: Wed, 27 Apr 2016 07:26:33 +0100 Message-ID: Subject: Re: How to speed up slow zpool scrub? From: krad To: Miroslav Lachman <000.fbsd@quip.cz> Cc: jg@internetx.com, FreeBSD FS Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 06:26:35 -0000 thats a shame. I have used an ssd on usb before with success, as their quality is usually better than pen drives. On 26 April 2016 at 15:27, Miroslav Lachman <000.fbsd@quip.cz> wrote: > krad wrote on 04/26/2016 16:02: > >> Erk, i would try to move your system off those data disks as you have >> two pools competing for the disk spindles. This is never ideal. You can >> by all means backup your os to those data pools but keep them on >> separate physical mediums. A couple of small SSD would do the trick >> nicely and could probably be added with no down time. You would probably >> want to find a suitable window though to make sure the box reboots >> nicely though. >> > > The system pool is really small - only 15GB and scrub is done relatively > fast. This machine cannot handle additional disks so I cannot move system > to other devices anyway. I tried system on USB flashdisk (read only) in the > past but it was slow and USB disk broke early. > > On 26 April 2016 at 14:35, Miroslav Lachman <000.fbsd@quip.cz >> > wrote: >> >> InterNetX - Juergen Gotteswinter wrote on 04/26/2016 15:09: >> >> to speed up the scrub itself you can try >> >> sysctl -w vfs.zfs.scrub_delay = 4 (default, 0 means higher prio) >> >> >> I will try it in the idle times >> >> but be careful as this can cause a serious performance impact, >> the value >> can be changed on the fly >> >> your pool is raidz, mirror ? dedup is hopefully disabled? >> >> >> I forgot to mention it. Disks are partitioned to four partitions: >> >> # gpart show -l ada0 >> => 34 7814037101 ada0 GPT (3.6T) >> 34 6 - free - (3.0K) >> 40 1024 1 boot0 (512K) >> 1064 10485760 2 swap0 (5.0G) >> 10486824 31457280 3 disk0sys (15G) >> 41944104 7769948160 4 disk0tank0 (3.6T) >> 7811892264 2144871 - free - (1.0G) >> >> diskXsys partitions are used for base system pool which is 4-way >> mirror >> >> diskXtank0 partitions are used for data storage as RAIDZ >> >> # zpool list >> NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH >> ALTROOT >> sys 14.9G 11.0G 3.92G - 79% 73% 1.00x ONLINE - >> tank0 14.4T 10.8T 3.56T - 19% 75% 1.00x ONLINE - >> >> >> # zpool status -v >> pool: sys >> state: ONLINE >> scan: scrub repaired 0 in 1h2m with 0 errors on Sun Apr 24 >> 04:03:54 2016 >> config: >> >> NAME STATE READ WRITE CKSUM >> sys ONLINE 0 0 0 >> mirror-0 ONLINE 0 0 0 >> gpt/disk0sys ONLINE 0 0 0 >> gpt/disk1sys ONLINE 0 0 0 >> gpt/disk2sys ONLINE 0 0 0 >> gpt/disk3sys ONLINE 0 0 0 >> >> errors: No known data errors >> >> pool: tank0 >> state: ONLINE >> scan: scrub in progress since Sun Apr 24 03:01:35 2016 >> 7.63T scanned out of 10.6T at 36.7M/s, 23h32m to go >> 0 repaired, 71.98% done >> config: >> >> NAME STATE READ WRITE CKSUM >> tank0 ONLINE 0 0 0 >> raidz1-0 ONLINE 0 0 0 >> gpt/disk0tank0 ONLINE 0 0 0 >> gpt/disk1tank0 ONLINE 0 0 0 >> gpt/disk2tank0 ONLINE 0 0 0 >> gpt/disk3tank0 ONLINE 0 0 0 >> >> errors: No known data errors >> >> >> # zdb | grep ashift >> ashift: 12 >> ashift: 12 >> >> >> Thank you for your informations. >> >> >> From owner-freebsd-fs@freebsd.org Wed Apr 27 06:32:02 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 221B5B1E30D for ; Wed, 27 Apr 2016 06:32:02 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-wm0-x230.google.com (mail-wm0-x230.google.com [IPv6:2a00:1450:400c:c09::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AF3941B83 for ; Wed, 27 Apr 2016 06:32:01 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: by mail-wm0-x230.google.com with SMTP id n129so12315992wmn.1 for ; Tue, 26 Apr 2016 23:32:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=RYZS/ZOC+ICKe3IO0TbLgXS1kAawpx84CpsFxehSRI8=; b=qGjsGQenuTiEtueMdCbfQs5iM8XwFZTtYFCKOiGepRW0c4liNFVaM0cO2+FQbO9Mxp zpYxG+cjQRzCWh7Yw6RgYQXvDLfYkQUhspIHPQS8R+dMOGcZI4X2x9GpP4hPFmEiyfe2 Gl84IvlpT1yYsZhrvAwNyp0rsEFo9DD4vwfPStqUQg0hDkEgpCZE/ph8WvmewKZJdRYe SiFLSlvC2yIbgs3L5bysRaIpuMVHYhzB/ucmgDfsbYZFTzsy4HJ4l/c1oVCtBDZXwzqk nQ7FB2JL08syIYqDrWP91avim0vIa0XBTm23el3VLwwVgJwaC9eh2/g1XqB+4Z/WfICh xdKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=RYZS/ZOC+ICKe3IO0TbLgXS1kAawpx84CpsFxehSRI8=; b=Ge1ddHusARsS1WON5PpSJsDxuul6TZp0CJg3+1fxDoh/ZnoCN9d2M69oj2owkpU3HE twdkEDKU9dJU6MUZPqLIMC0D/jIssrgdMcloz3X0apGeAQMzVxteJ1ZH0Es0pQHj+irb nTAg3UgsRxMMs2HwCXeWbfesQ5mMY4ZgVN5eRkeE9KABrN+6LbgiMbfXOPY4AYQlMgsR UlhekBbL68jcdoyZ5SiX3hdPMfh/ToG8up9zKRw4s0zzHQ7M+5NSe6JAo4+opPGl+/bm TS94zDVwVTqO2kn2gcgmrB19bCjTuShfddHe/jGCtfoQwy+FS7KjgghXognBN1VQol1e 3Q2g== X-Gm-Message-State: AOPr4FUrqD71ILr1J+XETTtBDUqFYbNnY7ZgkS53xE/B+TYX7zgQqAF+jmoh1tiy8mDlLKrhLRG620tXIWOcjQ== MIME-Version: 1.0 X-Received: by 10.194.107.6 with SMTP id gy6mr7225256wjb.20.1461738720401; Tue, 26 Apr 2016 23:32:00 -0700 (PDT) Received: by 10.28.26.17 with HTTP; Tue, 26 Apr 2016 23:32:00 -0700 (PDT) In-Reply-To: <381846248.2672053.1461695277122.JavaMail.yahoo@mail.yahoo.com> References: <381846248.2672053.1461695277122.JavaMail.yahoo.ref@mail.yahoo.com> <381846248.2672053.1461695277122.JavaMail.yahoo@mail.yahoo.com> Date: Wed, 27 Apr 2016 07:32:00 +0100 Message-ID: Subject: Re: How to speed up slow zpool scrub? From: krad To: DH Cc: Miroslav Lachman <000.fbsd@quip.cz>, FreeBSD FS Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 06:32:02 -0000 That memory requirement really depends on your work load. I have run small boxes with 2gb and zfs and they ran fine. They weren't doing filer or desktop roles though I could get away with fairly easily. Having said that I wouldn't build a machine now with less than 16gb unless it was very special circumstances. On 26 April 2016 at 19:27, DH via freebsd-fs wrote: > Take a long look at the FreeNAS documentation and discussion threads - if > you're not using a sufficient quantity of ECC ram with ZFS you're asking > for anomalous behavior. I'm not a ZFS expert, but the documentation leads > me to think inadequate system ram is certainly a consideration. > > FreeNAS requires a minimum of 8 GB of ECC ram when used with ZFS. > > David Hutchens III > System Administrator > > -------------------------------------------- > On Tue, 4/26/16, Miroslav Lachman <000.fbsd@quip.cz> wrote: > > Subject: Re: How to speed up slow zpool scrub? > To: "DH" , freebsd-fs@freebsd.org > Date: Tuesday, April 26, 2016, 9:34 AM > > DH wrote on 04/26/2016 > 17:47: > >> 5GB of RAM > > > > That seems to be an > insufficient amount of system ram when employing zfs. > > > > Take a look at > this: > > > > http://doc.freenas.org/9.3/freenas_intro.html#ram > > I know 5GB is not much these > days but is memory used for scrubbing a > lot? Because I am satisfied with working > performance. The only concern > is slow > scrubbing and I am not sure that more memory helps in this > case. > > Miroslav Lachman > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Wed Apr 27 10:53:33 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B47AB1D459 for ; Wed, 27 Apr 2016 10:53:33 +0000 (UTC) (envelope-from girgen@FreeBSD.org) Received: from mail.pingpong.net (mail.pingpong.net [79.136.116.202]) by mx1.freebsd.org (Postfix) with ESMTP id DC9F214A2 for ; Wed, 27 Apr 2016 10:53:32 +0000 (UTC) (envelope-from girgen@FreeBSD.org) Received: from nt9.bahnhof.se (nt9.bahnhof.se [213.136.34.39]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.pingpong.net (Postfix) with ESMTPSA id 530E115FAC for ; Wed, 27 Apr 2016 12:53:25 +0200 (CEST) From: Palle Girgensohn Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Moved disks to new machine. Zfs won't boot Message-Id: <0E6A1067-B066-4C73-8D9C-BFA95C6FA338@FreeBSD.org> Date: Wed, 27 Apr 2016 12:53:24 +0200 To: freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 10:53:33 -0000 Hi, =20 We had a faulty machine and thought the simple solution to get our = systems online would be to move the disks to another identical box. But now the ZFS will not boot. I booted as USB stick and I can see the zfs pool after importing it with zpool import -o altroot=3D/mnt -f tank after this, I can see the pool and the file systems, but a reboot will = not see the disk in BIOS, and I have my root there... :-/ gpart bootcode is installed (again, should really have been there = already) The machines are HP DL380 G7 woth P410i. any ideas? zpool.cache?= From owner-freebsd-fs@freebsd.org Wed Apr 27 11:12:27 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0AD65B1DDDE for ; Wed, 27 Apr 2016 11:12:27 +0000 (UTC) (envelope-from matthew@FreeBSD.org) Received: from smtp.infracaninophile.co.uk (smtp.infracaninophile.co.uk [81.2.117.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8EF6610A5 for ; Wed, 27 Apr 2016 11:12:26 +0000 (UTC) (envelope-from matthew@FreeBSD.org) Received: from ox-dell39.ox.adestra.com (unknown [85.199.232.226]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: m.seaman@infracaninophile.co.uk) by smtp.infracaninophile.co.uk (Postfix) with ESMTPSA id 5B7C8A292 for ; Wed, 27 Apr 2016 11:12:17 +0000 (UTC) Authentication-Results: smtp.infracaninophile.co.uk; dmarc=none header.from=FreeBSD.org Authentication-Results: smtp.infracaninophile.co.uk/5B7C8A292; dkim=none; dkim-atps=neutral Subject: Re: Moved disks to new machine. Zfs won't boot To: freebsd-fs@freebsd.org References: <0E6A1067-B066-4C73-8D9C-BFA95C6FA338@FreeBSD.org> From: Matthew Seaman Message-ID: <0fd0dcc3-ea1d-97a4-ae35-990fff376c47@freebsd.org> Date: Wed, 27 Apr 2016 12:12:01 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 In-Reply-To: <0E6A1067-B066-4C73-8D9C-BFA95C6FA338@FreeBSD.org> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="hu2cx2JChP2OphxwGAubNsexJX9BwX9ap" X-Virus-Scanned: clamav-milter 0.99.1 at smtp.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-0.4 required=5.0 tests=BAYES_00,RDNS_NONE, SPF_SOFTFAIL autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on smtp.infracaninophile.co.uk X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 11:12:27 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --hu2cx2JChP2OphxwGAubNsexJX9BwX9ap Content-Type: multipart/mixed; boundary="xnTm0vSs1JEoHPIrBNgGUP69fGQa7EEIm" From: Matthew Seaman To: freebsd-fs@freebsd.org Message-ID: <0fd0dcc3-ea1d-97a4-ae35-990fff376c47@freebsd.org> Subject: Re: Moved disks to new machine. Zfs won't boot References: <0E6A1067-B066-4C73-8D9C-BFA95C6FA338@FreeBSD.org> In-Reply-To: <0E6A1067-B066-4C73-8D9C-BFA95C6FA338@FreeBSD.org> --xnTm0vSs1JEoHPIrBNgGUP69fGQa7EEIm Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 04/27/16 11:53, Palle Girgensohn wrote: > We had a faulty machine and thought the simple solution to get our syst= ems online would be to move the disks to another identical box. >=20 > But now the ZFS will not boot. >=20 > I booted as USB stick and I can see the zfs pool after importing it wit= h >=20 > zpool import -o altroot=3D/mnt -f tank >=20 > after this, I can see the pool and the file systems, but a reboot will = not see the disk in BIOS, and I have my root there... :-/ >=20 > gpart bootcode is installed (again, should really have been there alrea= dy) >=20 >=20 >=20 > The machines are HP DL380 G7 woth P410i. >=20 > any ideas? zpool.cache? Usually, to fix this, you just need to import the pool when booted from some alternative media -- which you've already done. Importing the pool will rebuild the zpool.cache for you. How is the on-board RAID controller configured? You may have to get into the RAID setup and tell it to scan the drives for a foreign config (err... 'foreign config' is what Dell calls it; not sure what the HP equivalent is.) A lot of RAID controllers will recognise that the disks have been moved from a different machine and won't automatically enable and mount them in case that results in multiple filesystems being mounted in the same place or other chaotic effects. Needless to say, fiddling around with the HW RAID config has plenty of potential for your disks to get trashed... Cheers, Matthew --xnTm0vSs1JEoHPIrBNgGUP69fGQa7EEIm-- --hu2cx2JChP2OphxwGAubNsexJX9BwX9ap Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJXIJ6IAAoJEABRPxDgqeTnJpgP/iAWgTnWUe6OeEWAEH4mD4sX /mnZoLkxmePqSwu4X7v8AaLwhyzuqn6FiSktf8ntdSS+QztCEeZDGiNZ+NNVfQQ2 CSGU5NpX2qFtpYD91EoP+k0ObpwbiIw1J3BXmhqTf0zEdj8xQBGzxOv3+Fla6v0r gWJUmmZGRedO3ZMR5ex9ktK1p8isFz4/XCVfPJ/02XuHtltkKu2JV661+7iaeqnQ HQ+UlX4S4IqbYZvj2dGmQms0eODipn8qniVilkWJO0/GEaIpaZypXdyljQNVYkQN fVjAEbJBy8wAvklM7RhfGYJHPo4Fy6el0STYphBXHgO0fP7T9dsWZtKDRkYl6SzJ fowY6Y9iL3TMXe40l3qZM+mp57FW7m/esMegTrXVg21kOearmVHm/NaczYAN94AY 9Fg58MDdPekeZv6Qg5bSJTYGjwtSHMsSECcFjJqVC2O1hH1tMp6Xi6VkI3eJrK+T 7j5YV8E843unQh7QqJJtSPZOuubkhwQty9s2D0AFED6b5cQbcXPabw9mGaI3xBXt jNbpo0ASxyoYomddIB4kjA/b0HemwSb2ohcObpFI1J+o0KqptnZSmSGqF4mDAvAK eq/PRwBe4UfCx1dN0S4vFJeD1D0pYUfzRur9GCei4kP/BSry0DdT9xYqo1cB/cF6 ovHmnUPinAZIOT+Jovv/ =dfna -----END PGP SIGNATURE----- --hu2cx2JChP2OphxwGAubNsexJX9BwX9ap-- From owner-freebsd-fs@freebsd.org Wed Apr 27 11:35:43 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BA91BB1E633 for ; Wed, 27 Apr 2016 11:35:43 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 805B11EDD; Wed, 27 Apr 2016 11:35:43 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 3268F28417; Wed, 27 Apr 2016 13:35:40 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 67B7E28412; Wed, 27 Apr 2016 13:35:39 +0200 (CEST) Message-ID: <5720A40B.2070305@quip.cz> Date: Wed, 27 Apr 2016 13:35:39 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: Palle Girgensohn , freebsd-fs@freebsd.org Subject: Re: Moved disks to new machine. Zfs won't boot References: <0E6A1067-B066-4C73-8D9C-BFA95C6FA338@FreeBSD.org> In-Reply-To: <0E6A1067-B066-4C73-8D9C-BFA95C6FA338@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 11:35:43 -0000 Palle Girgensohn wrote on 04/27/2016 12:53: > Hi, > > We had a faulty machine and thought the simple solution to get our systems online would be to move the disks to another identical box. > > But now the ZFS will not boot. > > I booted as USB stick and I can see the zfs pool after importing it with > > zpool import -o altroot=/mnt -f tank > > after this, I can see the pool and the file systems, but a reboot will not see the disk in BIOS, and I have my root there... :-/ > > gpart bootcode is installed (again, should really have been there already) > > > > The machines are HP DL380 G7 woth P410i. > > any ideas? zpool.cache? I think altroot disabled refreshing of zpool.cache. So you need to export and import pool without altroot option or you can manually copy this file. (I did something like this few years ago so I don't remember it well) The next problem can be the RAID controller. Was this same controller on an old machine too? There are some controllers that does not show all attached drives at boot time so RAIDZ configurations cannot boot on it. Miroslav Lachman From owner-freebsd-fs@freebsd.org Wed Apr 27 11:55:03 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 43BEDB1EB18 for ; Wed, 27 Apr 2016 11:55:03 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BE0A51A83 for ; Wed, 27 Apr 2016 11:55:02 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 5988C28416; Wed, 27 Apr 2016 13:55:00 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id D3A8328412; Wed, 27 Apr 2016 13:54:58 +0200 (CEST) Message-ID: <5720A892.5070807@quip.cz> Date: Wed, 27 Apr 2016 13:54:58 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: "Eric A. Borisch" CC: "freebsd-fs@freebsd.org" Subject: Re: How to speed up slow zpool scrub? References: <571F62AD.6080005@quip.cz> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 11:55:03 -0000 Eric A. Borisch wrote on 04/27/2016 00:26: >> Is there any tuning to make scrub faster in this idle time? >> Or is it better to do it other way - slower scrub with even lower priority >> taking for about one week but not affecting time of normal operations? (is >> it dangerous to have scrub running this long or reboot machine during the >> scrub?) > > It's likely easier to make scrub slower (but rsync faster) when the > system is not idle, but hopefully no slower otherwise. Yes, I've been thinking about this approach yesterday evening. > I would start with increasing the vfs.zfs.scrub_delay (in ticks; > typically 1/1000 of a second) settings to reduce scrub's impact to > other IO this without impacting resilvers. (Expect longer total time > as it will scrub less during rsyncs.) You can also increase > vfs.zfs.scan_idle if scrub_delay isn't enough, but do note that > vfs.zfs.scan_idle impacts resilvers, too -- so make sure you don't > also increase vfs.zfs.resilver_delay, or you may get very slow > resilvers during IO. > > The logic for this is in dsl_scan.c [1] -- look there for the final > word on what the different parameters do; there are a number of > informative comments throughout the file if you are not into the code > itself. > > See also vfs.zfs.vdev.*_max_active and vfs.zfs.vdev.*_min_active, if > you are really want more knobs to try. Than you for this pointers, I will look at it and hope find some way to make things better for this machine. > It's might also be worth trying '/usr/local/bin/perl > /usr/share/dtrace/toolkit/hotkernel' to see what the kernel is up > to... Showing results with 0.1% or more. kernel`atomic_add_long 647 0.1% kernel`uma_zfree_arg 666 0.1% kernel`ata_intel_sata_sidpr_read 695 0.1% kernel`uma_zalloc_arg 754 0.1% kernel`free 895 0.1% kernel`ata_tf_write 1011 0.1% kernel`bzero 1453 0.1% kernel`bcopy 1551 0.1% kernel`acpi_timer_get_timecount 2550 0.2% kernel`_sx_xunlock 2894 0.2% kernel`spinlock_exit 3347 0.3% kernel`_sx_xlock 3379 0.3% kernel`sched_idletd 4053 0.3% zfs.ko`lz4_decompress 5293 0.4% kernel`cpu_idle 65358 5.4% kernel`acpi_cpu_c1 1096734 89.8% zpool scrub and rsync backup is running at this time pool: tank0 state: ONLINE scan: scrub in progress since Sun Apr 24 03:01:35 2016 8.44T scanned out of 10.6T at 29.7M/s, 21h8m to go 0 repaired, 79.64% done > - Eric > > [1] https://svnweb.freebsd.org/base/releng/10.3/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c?view=markup > or > https://github.com/freebsd/freebsd/blob/releng/10.3/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c Thank you for your very useful reply! Miroslav Lachman From owner-freebsd-fs@freebsd.org Wed Apr 27 12:05:16 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8D03EB1D45E for ; Wed, 27 Apr 2016 12:05:16 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3170311D7 for ; Wed, 27 Apr 2016 12:05:15 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 2740F28416; Wed, 27 Apr 2016 14:05:14 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 40E6028412; Wed, 27 Apr 2016 14:05:12 +0200 (CEST) Message-ID: <5720AAF8.4090900@quip.cz> Date: Wed, 27 Apr 2016 14:05:12 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: Andy Farkas , 'PK1048' CC: freebsd-fs@freebsd.org Subject: Re: How to speed up slow zpool scrub? References: <698816653.2698619.1461685653634.JavaMail.yahoo.ref@mail.yahoo.com> <698816653.2698619.1461685653634.JavaMail.yahoo@mail.yahoo.com> <571F9897.2070008@quip.cz> <571FEB34.7040305@andyit.com.au> <56C0A956-F134-4A8D-A8B6-B93DCA045BE4@pk1048.com> <084201d1a03e$d2158fe0$7640afa0$@andyit.com.au> In-Reply-To: <084201d1a03e$d2158fe0$7640afa0$@andyit.com.au> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 12:05:16 -0000 Andy Farkas wrote on 04/27/2016 06:39: >> -----Original Message----- >> From: PK1048 [mailto:paul@pk1048.com] >> Sent: Wednesday, 27 April 2016 12:34 PM >> To: Andy Farkas >> Cc: freebsd-fs@freebsd.org >> Subject: Re: How to speed up slow zpool scrub? >> >> ... >> Scrubs (and resilver) operations are essentially all random I/O. Those >> drives are low end, low performance, desktop drives. > > Yes, the system is an old low end, low performance desktop. That was > my point, that it took 25 hours to scrub 7.52T and not 4 days like the > OP is saying. Thank you for output of your zpool scrub. It is definitely faster than mine. To: Paul pk1048 Mine scrub does not repair anything. Drives are OK (in SMART). CPU is about 70%-90% idle during scrub + rsync backup and drives are about 60%-70% busy according to iostat: root@kiwi ~/# iostat -x -w 10 ada0 ada1 ada2 ada3 extended device statistics device r/s w/s kr/s kw/s qlen svc_t %b ada0 70.1 17.9 1747.0 802.4 0 7.0 24 ada1 70.1 17.9 1747.1 802.4 0 7.0 25 ada2 66.9 17.1 1686.4 791.3 4 6.5 23 ada3 66.9 16.9 1686.3 790.1 2 6.6 23 extended device statistics device r/s w/s kr/s kw/s qlen svc_t %b ada0 93.6 13.0 576.4 244.0 0 21.6 70 ada1 98.6 12.9 587.2 246.4 2 20.5 71 ada2 87.9 15.3 566.0 242.4 3 20.5 67 ada3 84.9 14.5 549.2 237.2 3 20.4 66 extended device statistics device r/s w/s kr/s kw/s qlen svc_t %b ada0 98.7 42.5 1924.7 2536.3 1 26.3 86 ada1 99.1 45.5 1931.5 2671.5 1 23.8 87 ada2 94.2 44.9 1840.7 2720.3 0 20.1 76 ada3 93.6 42.7 1807.9 2607.1 0 18.7 75 extended device statistics device r/s w/s kr/s kw/s qlen svc_t %b ada0 108.2 28.2 1092.6 1316.6 2 17.3 68 ada1 101.6 26.3 1053.8 1183.4 3 15.5 67 ada2 98.6 26.0 1000.2 1126.2 2 12.2 57 ada3 104.0 24.0 1015.8 1080.6 3 14.1 60 extended device statistics device r/s w/s kr/s kw/s qlen svc_t %b ada0 116.0 18.5 821.8 807.8 0 12.9 62 ada1 117.2 18.5 822.2 807.0 0 13.5 63 ada2 110.8 20.9 743.0 803.8 0 11.1 58 ada3 108.2 20.0 688.2 755.0 2 11.3 55 extended device statistics device r/s w/s kr/s kw/s qlen svc_t %b ada0 121.8 16.6 602.1 526.9 3 9.2 52 ada1 122.2 16.5 606.9 528.5 4 9.8 54 ada2 117.0 14.6 601.7 524.9 2 11.3 60 ada3 120.6 13.5 610.1 491.3 0 11.4 61 I really don't know why it cannot go faster if nothing is loaded for 100%. Miroslav Lachman From owner-freebsd-fs@freebsd.org Wed Apr 27 12:14:29 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0A06EB1D6D4 for ; Wed, 27 Apr 2016 12:14:29 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.81]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C9D23178C for ; Wed, 27 Apr 2016 12:14:28 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.72) (envelope-from ) id 1avO7U-0005Ia-2X for freebsd-fs@freebsd.org; Wed, 27 Apr 2016 13:59:04 +0200 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: freebsd-fs@freebsd.org Subject: Re: Moved disks to new machine. Zfs won't boot References: <0E6A1067-B066-4C73-8D9C-BFA95C6FA338@FreeBSD.org> Date: Wed, 27 Apr 2016 13:59:02 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Ronald Klop" Message-ID: In-Reply-To: <0E6A1067-B066-4C73-8D9C-BFA95C6FA338@FreeBSD.org> User-Agent: Opera Mail/12.16 (FreeBSD) X-Authenticated-As-Hash: 398f5522cb258ce43cb679602f8cfe8b62a256d1 X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: -0.2 X-Spam-Status: No, score=-0.2 required=5.0 tests=ALL_TRUSTED, BAYES_50 autolearn=disabled version=3.4.0 X-Scan-Signature: a2d32f98be707cbcda8602d5fffa976a X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 12:14:29 -0000 If I read this correctly your problem has nothing to do with ZFS. It is plain recognizing the disk in the BIOS/RAID-card. And if it recognizes the disk it must recognize the boot-code. Please send more information about your setup. Output of /var/run/dmesg.boot. Is your BIOS set to MBR, GPT, (U)EFI? What is installed in your boot blocks (MBR, GPT or (U)EFI)? What version of FreeBSD are you running? Regards, Ronald. On Wed, 27 Apr 2016 12:53:24 +0200, Palle Girgensohn wrote: > Hi, > > We had a faulty machine and thought the simple solution to get our > systems online would be to move the disks to another identical box. > > But now the ZFS will not boot. > > I booted as USB stick and I can see the zfs pool after importing it with > > zpool import -o altroot=/mnt -f tank > > after this, I can see the pool and the file systems, but a reboot will > not see the disk in BIOS, and I have my root there... :-/ > > gpart bootcode is installed (again, should really have been there > already) > > > > The machines are HP DL380 G7 woth P410i. > > any ideas? zpool.cache? > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Wed Apr 27 12:19:59 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 71E39B1D7A9 for ; Wed, 27 Apr 2016 12:19:59 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 36A921898 for ; Wed, 27 Apr 2016 12:19:58 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id D8E6728412; Wed, 27 Apr 2016 14:19:56 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id B1DBA2840C; Wed, 27 Apr 2016 14:19:55 +0200 (CEST) Message-ID: <5720AE6B.6090109@quip.cz> Date: Wed, 27 Apr 2016 14:19:55 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: "Brandon J. Wandersee" CC: krad , FreeBSD FS Subject: Re: How to speed up slow zpool scrub? References: <571F62AD.6080005@quip.cz> <571F687D.8040103@internetx.com> <571F6EA4.90800@quip.cz> <571F7AED.2040500@quip.cz> <86lh409twi.fsf@WorkBox.Home> In-Reply-To: <86lh409twi.fsf@WorkBox.Home> Content-Type: text/plain; charset=ISO-8859-2; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 12:19:59 -0000 Brandon J. Wandersee wrote on 04/26/2016 19:11: > > Miroslav Lachman writes: > >> krad wrote on 04/26/2016 16:02: >>> Erk, i would try to move your system off those data disks as you have >>> two pools competing for the disk spindles. This is never ideal. You can >>> by all means backup your os to those data pools but keep them on >>> separate physical mediums. A couple of small SSD would do the trick >>> nicely and could probably be added with no down time. You would probably >>> want to find a suitable window though to make sure the box reboots >>> nicely though. >> >> The system pool is really small - only 15GB and scrub is done relatively >> fast. This machine cannot handle additional disks so I cannot move >> system to other devices anyway. I tried system on USB flashdisk (read >> only) in the past but it was slow and USB disk broke early. >> > > I see no sign that the partitions are encrypted, so I can't think of a > reason that the system and data can't reside on the same pool. Believe it or not there are systems which can't boot from RAIDZ but can boot from mirrored drives. Some controllers does not expose all drives at boot time, only the first one. > The > physical effort of the read arms on the disks isn't the only resource > to consider---each pool has its own ARC, for example, and if you have > compression enabled each pool will compete for CPU cycles. Having two > pools grabbing at such limited resources is going to noticeably hurt > performance. Read arms moving cannot be an issue. They must move the same way in one or ten pools if the need to read 100 files on different places of the disks. Sharing ARC anc CPU cycles can be an issue but as I said before the CPU is about 70% idle and I believe ARC is not used for scrub and resilver. > All that said, running even a single relative large pool on a > low-powered dual-core CPU and 5G of RAM will noticeably limit *every* > ZFS operation. I would bet that while you might be able to shave some > time off the scrub, at best it's still going to take at least an entire > day on that hardware. I will be fine with entire day. It is expected time for this pool size. But I don't know why it takes 3-4 days if CPU nor disks are utilized to 100% > As a side note, I'm not so sure it's a good idea to use two different > redundancy methods on the same set of disks, though I can't say exactly > what the downsides would be (if any). It just seems odd. Explained above. I don't think there can be any issue with two different redundancy methods. Thank you for your effort. Miroslav Lachman From owner-freebsd-fs@freebsd.org Wed Apr 27 12:31:11 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F140CB1DC98 for ; Wed, 27 Apr 2016 12:31:11 +0000 (UTC) (envelope-from girgen@FreeBSD.org) Received: from mail.pingpong.net (mail.pingpong.net [79.136.116.202]) by mx1.freebsd.org (Postfix) with ESMTP id B1F2E106A for ; Wed, 27 Apr 2016 12:31:11 +0000 (UTC) (envelope-from girgen@FreeBSD.org) Received: from nt9.bahnhof.se (nt9.bahnhof.se [213.136.34.39]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.pingpong.net (Postfix) with ESMTPSA id 0FE9215C8B; Wed, 27 Apr 2016 14:31:10 +0200 (CEST) Content-Type: text/plain; charset=iso-8859-1 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Moved disks to new machine. Zfs won't boot From: Palle Girgensohn In-Reply-To: <5720A40B.2070305@quip.cz> Date: Wed, 27 Apr 2016 14:31:09 +0200 Cc: freebsd-fs@freebsd.org, Julian Akehurst Content-Transfer-Encoding: quoted-printable Message-Id: <8F08B233-8E7F-4287-BB67-FED75A1F617A@FreeBSD.org> References: <0E6A1067-B066-4C73-8D9C-BFA95C6FA338@FreeBSD.org> <5720A40B.2070305@quip.cz> To: Miroslav Lachman <000.fbsd@quip.cz> X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 12:31:12 -0000 > 27 apr. 2016 kl. 13:35 skrev Miroslav Lachman <000.fbsd@quip.cz>: >=20 > Palle Girgensohn wrote on 04/27/2016 12:53: >> Hi, >>=20 >> We had a faulty machine and thought the simple solution to get our = systems online would be to move the disks to another identical box. >>=20 >> But now the ZFS will not boot. >>=20 >> I booted as USB stick and I can see the zfs pool after importing it = with >>=20 >> zpool import -o altroot=3D/mnt -f tank >>=20 >> after this, I can see the pool and the file systems, but a reboot = will not see the disk in BIOS, and I have my root there... :-/ >>=20 >> gpart bootcode is installed (again, should really have been there = already) >>=20 >>=20 >>=20 >> The machines are HP DL380 G7 woth P410i. >>=20 >> any ideas? zpool.cache? >=20 > I think altroot disabled refreshing of zpool.cache. So you need to = export and import pool without altroot option or you can manually copy = this file. (I did something like this few years ago so I don't remember = it well) >=20 > The next problem can be the RAID controller. Was this same controller = on an old machine too? There are some controllers that does not show all = attached drives at boot time so RAIDZ configurations cannot boot on it. >=20 > Miroslav Lachman Hi, Thanks for your reply. Yes, it was the cache file.=20 zpool import refused to import unless I added a -f. so booting from a memory stick and running zpool import -o altroot=3D/mnt -f tank rm /mnt/boot/zfs/zpool.cache zpool export zpool import -o altroot/mnt tank # this time it didn't require -f reboot did the trick. fresh cache file with the new machine's IDs. Simple once you know it... :-/ Palle From owner-freebsd-fs@freebsd.org Wed Apr 27 13:22:54 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A94A9B1E7CF for ; Wed, 27 Apr 2016 13:22:54 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from umail.aei.mpg.de (umail.aei.mpg.de [194.94.224.6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2EBC81F04 for ; Wed, 27 Apr 2016 13:22:53 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from mailgate.aei.mpg.de (mailgate.aei.mpg.de [194.94.224.5]) by umail.aei.mpg.de (Postfix) with ESMTP id 9E5F520028D for ; Wed, 27 Apr 2016 15:22:44 +0200 (CEST) Received: from mailgate.aei.mpg.de (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 8EDD9406ADE for ; Wed, 27 Apr 2016 15:22:44 +0200 (CEST) Received: from intranet.aei.uni-hannover.de (ahin1.aei.uni-hannover.de [130.75.117.40]) by mailgate.aei.mpg.de (Postfix) with ESMTP id 61ADB406ADB for ; Wed, 27 Apr 2016 15:22:44 +0200 (CEST) Received: from arc.aei.uni-hannover.de ([130.75.117.1]) by intranet.aei.uni-hannover.de (IBM Domino Release 9.0.1FP5) with ESMTP id 2016042715224342-55002 ; Wed, 27 Apr 2016 15:22:43 +0200 Date: Wed, 27 Apr 2016 15:22:44 +0200 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: freebsd-fs@freebsd.org Subject: zfs on nvme: gnop breaks pool, zfs gets stuck Message-Id: <20160427152244.ff36ff74ae64c1f86fdc960a@aei.mpg.de> Organization: Max Planck Gesellschaft X-Mailer: Sylpheed 3.4.2 (GTK+ 2.24.22; amd64-portbld-freebsd10.0) Mime-Version: 1.0 X-MIMETrack: Itemize by SMTP Server on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 27/04/2016 15:22:43, Serialize by Router on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 27/04/2016 15:22:43, Serialize complete at 27/04/2016 15:22:43 X-TNEFEvaluated: 1 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-PMX-Version: 6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2016.4.27.131516 X-PerlMx-Spam: Gauge=IIIIIIII, Probability=8%, Report=' HTML_00_01 0.05, HTML_00_10 0.05, MIME_LOWER_CASE 0.05, BODY_SIZE_3000_3999 0, BODY_SIZE_5000_LESS 0, BODY_SIZE_7000_LESS 0, NO_URI_HTTPS 0, SINGLE_URI_IN_BODY 0, __ANY_URI 0, __C230066_P5 0, __CP_URI_IN_BODY 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __FRAUD_MONEY_BIG_COIN 0, __FRAUD_MONEY_BIG_COIN_DIG 0, __HAS_FROM 0, __HAS_MSGID 0, __HAS_X_MAILER 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0, __SINGLE_URI_TEXT 0, __STOCK_PHRASE_24 0, __SUBJ_ALPHA_END 0, __SUBJ_ALPHA_START 0, __SUBJ_ALPHA_START_END 0, __TO_MALFORMED_2 0, __TO_NO_NAME 0, __URI_IN_BODY 0, __URI_NO_MAILTO 0, __URI_NO_WWW 0, __URI_NS , __URI_WITH_PATH 0' X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 13:22:54 -0000 Hello all, I have a set of three NVME-ssds on PCIe-converters: --- root@storage:~ # nvmecontrol devlist nvme0: SAMSUNG MZVPV512HDGL-00000 nvme0ns1 (488386MB) nvme1: SAMSUNG MZVPV512HDGL-00000 nvme1ns1 (488386MB) nvme2: SAMSUNG MZVPV512HDGL-00000 nvme2ns1 (488386MB) --- I want to use a z1 raid on these and created 1m-aligned partitions: --- root@storage:~ # gpart show => 34 1000215149 nvd0 GPT (477G) 34 2014 - free - (1.0M) 2048 1000212480 1 freebsd-zfs (477G) 1000214528 655 - free - (328K) => 34 1000215149 nvd1 GPT (477G) 34 2014 - free - (1.0M) 2048 1000212480 1 freebsd-zfs (477G) 1000214528 655 - free - (328K) => 34 1000215149 nvd2 GPT (477G) 34 2014 - free - (1.0M) 2048 1000212480 1 freebsd-zfs (477G) 1000214528 655 - free - (328K) --- After creating a zpool I recognized that it was using ashift=9. I vaguely remembered that SSDs usually have 4k (or even larger) sectors, so I destroyed the pool and set up gnop-providers with -S 4k to get ashift=12. This worked as expected: --- pool: flash state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM flash ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gpt/flash0.nop ONLINE 0 0 0 gpt/flash1.nop ONLINE 0 0 0 gpt/flash2.nop ONLINE 0 0 0 errors: No known data errors --- This pool can be used, exported and imported just fine as far as I can tell. Then I exported the pool and destroyed the gnop-providers. When starting with "advanced format" hdds some years ago, this was the way to make zfs recognize the disks with ashift=12. However, destroying the gnop-devices appears to have crashed the pool in this case: --- root@storage:~ # zpool import pool: flash id: 4978839938025863522 state: ONLINE status: One or more devices contains corrupted data. action: The pool can be imported using its name or numeric identifier. see: http://illumos.org/msg/ZFS-8000-4J config: flash ONLINE raidz1-0 ONLINE 11456367280316708003 UNAVAIL corrupted data gptid/55ae71aa-eb84-11e5-9298-0cc47a6c7484 ONLINE 6761786983139564172 UNAVAIL corrupted data --- How can the pool be online, when two of three devices are unavailable? I tried to import the pool nevertheless, but the zpool command got stuck in state tx-tx. "soft" reboot got stuck, too. I had to push the reset button to get my system back (still with a corrupt pool). I cleared the labels and re-did everything: the issue is perfectly reproducible. Am I doing something utterly wrong? Why is removing the gnop-nodes tampering with the devices (I think I did exactly this dozens of times on normal hdds during that previous years, and it always worked just fine)? And finally, why does the zpool import fail without any error message and requires me to reset the system? The system is 10.2-RELEASE-p9, update is scheduled for later this week (just in case it would make sense to try this again with 10.3). Any other hints are most welcome. cu Gerrit From owner-freebsd-fs@freebsd.org Wed Apr 27 14:14:39 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B376B1D839 for ; Wed, 27 Apr 2016 14:14:39 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from mail.in-addr.com (mail.in-addr.com [IPv6:2a01:4f8:191:61e8::2525:2525]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 34D9E1AA1 for ; Wed, 27 Apr 2016 14:14:39 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from gjp by mail.in-addr.com with local (Exim 4.86_2 (FreeBSD)) (envelope-from ) id 1avQEe-0000wi-R1; Wed, 27 Apr 2016 15:14:37 +0100 Date: Wed, 27 Apr 2016 15:14:36 +0100 From: Gary Palmer To: Gerrit K?hn Cc: freebsd-fs@freebsd.org Subject: Re: zfs on nvme: gnop breaks pool, zfs gets stuck Message-ID: <20160427141436.GA60370@in-addr.com> References: <20160427152244.ff36ff74ae64c1f86fdc960a@aei.mpg.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160427152244.ff36ff74ae64c1f86fdc960a@aei.mpg.de> X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: gpalmer@freebsd.org X-SA-Exim-Scanned: No (on mail.in-addr.com); SAEximRunCond expanded to false X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 14:14:39 -0000 On Wed, Apr 27, 2016 at 03:22:44PM +0200, Gerrit K?hn wrote: > Hello all, > > I have a set of three NVME-ssds on PCIe-converters: > > --- > root@storage:~ # nvmecontrol devlist > nvme0: SAMSUNG MZVPV512HDGL-00000 > nvme0ns1 (488386MB) > nvme1: SAMSUNG MZVPV512HDGL-00000 > nvme1ns1 (488386MB) > nvme2: SAMSUNG MZVPV512HDGL-00000 > nvme2ns1 (488386MB) > --- > > > I want to use a z1 raid on these and created 1m-aligned partitions: > > --- > root@storage:~ # gpart show > => 34 1000215149 nvd0 GPT (477G) > 34 2014 - free - (1.0M) > 2048 1000212480 1 freebsd-zfs (477G) > 1000214528 655 - free - (328K) > > => 34 1000215149 nvd1 GPT (477G) > 34 2014 - free - (1.0M) > 2048 1000212480 1 freebsd-zfs (477G) > 1000214528 655 - free - (328K) > > => 34 1000215149 nvd2 GPT (477G) > 34 2014 - free - (1.0M) > 2048 1000212480 1 freebsd-zfs (477G) > 1000214528 655 - free - (328K) > --- > > > After creating a zpool I recognized that it was using ashift=9. I vaguely > remembered that SSDs usually have 4k (or even larger) sectors, so I > destroyed the pool and set up gnop-providers with -S 4k to get ashift=12. > This worked as expected: > > --- > pool: flash > state: ONLINE > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > flash ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > gpt/flash0.nop ONLINE 0 0 0 > gpt/flash1.nop ONLINE 0 0 0 > gpt/flash2.nop ONLINE 0 0 0 > > errors: No known data errors > --- > > > This pool can be used, exported and imported just fine as far as I can > tell. Then I exported the pool and destroyed the gnop-providers. When > starting with "advanced format" hdds some years ago, this was the way to > make zfs recognize the disks with ashift=12. However, destroying the > gnop-devices appears to have crashed the pool in this case: > > --- > root@storage:~ # zpool import > pool: flash > id: 4978839938025863522 > state: ONLINE > status: One or more devices contains corrupted data. > action: The pool can be imported using its name or numeric identifier. > see: http://illumos.org/msg/ZFS-8000-4J > config: > > flash ONLINE > raidz1-0 ONLINE > 11456367280316708003 UNAVAIL corrupted > data gptid/55ae71aa-eb84-11e5-9298-0cc47a6c7484 ONLINE > 6761786983139564172 UNAVAIL corrupted > data > --- > > > How can the pool be online, when two of three devices are unavailable? I > tried to import the pool nevertheless, but the zpool command got stuck in > state tx-tx. "soft" reboot got stuck, too. I had to push the reset button > to get my system back (still with a corrupt pool). I cleared the labels > and re-did everything: the issue is perfectly reproducible. > > Am I doing something utterly wrong? Why is removing the gnop-nodes > tampering with the devices (I think I did exactly this dozens of times on > normal hdds during that previous years, and it always worked just fine)? > And finally, why does the zpool import fail without any error message and > requires me to reset the system? > The system is 10.2-RELEASE-p9, update is scheduled for later this week > (just in case it would make sense to try this again with 10.3). Any other > hints are most welcome. Did you destroy the gnop devices with the pool online? In the procedure I remember you export the pool, destroy the gnop devices, and then reimport the pool. Also, you only need to do the gnop trick for a single device in the pool for the entire pool's ashift to be changed AFAIK. There is a sysctl now too vfs.zfs.min_auto_ashift which lets you manage the ashift on a new pool without having to try the gnop trick Regards, Gary From owner-freebsd-fs@freebsd.org Wed Apr 27 16:42:10 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 00198B1E634 for ; Wed, 27 Apr 2016 16:42:09 +0000 (UTC) (envelope-from karli.sjoberg@slu.se) Received: from EXCH2-1.slu.se (webmail.slu.se [77.235.224.121]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "webmail.slu.se", Issuer "TERENA SSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 627261030 for ; Wed, 27 Apr 2016 16:42:08 +0000 (UTC) (envelope-from karli.sjoberg@slu.se) Received: from exch2-4.slu.se (77.235.224.124) by EXCH2-1.slu.se (77.235.224.121) with Microsoft SMTP Server (TLS) id 15.0.1156.6; Wed, 27 Apr 2016 18:26:54 +0200 Received: from exch2-4.slu.se ([fe80::8041:d235:9900:f9cc]) by exch2-4.slu.se ([fe80::8041:d235:9900:f9cc%22]) with mapi id 15.00.1156.000; Wed, 27 Apr 2016 18:26:54 +0200 From: =?utf-8?B?S2FybGkgU2rDtmJlcmc=?= To: =?utf-8?B?R2Vycml0IEvDvGhu?= CC: "freebsd-fs@freebsd.org" Subject: Re: zfs on nvme: gnop breaks pool, zfs gets stuck Thread-Topic: zfs on nvme: gnop breaks pool, zfs gets stuck Thread-Index: AQHRoKGc5DIMuw2bUEWMyK+GHJC6Xg== Date: Wed, 27 Apr 2016 16:26:53 +0000 Message-ID: <5d82e5664bea489fa9dd1f91a03806ca@exch2-4.slu.se> Accept-Language: sv-SE, en-US Content-Language: sv-SE X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 16:42:10 -0000 DQpEZW4gMjcgYXByIDIwMTYgMTU6MjMgc2tyZXYgR2Vycml0IEvDvGhuIDxnZXJyaXQua3VlaG5A YWVpLm1wZy5kZT46DQo+DQo+IEhlbGxvIGFsbCwNCj4NCj4gSSBoYXZlIGEgc2V0IG9mIHRocmVl IE5WTUUtc3NkcyBvbiBQQ0llLWNvbnZlcnRlcnM6DQo+DQo+IC0tLQ0KPiByb290QHN0b3JhZ2U6 fiAjIG52bWVjb250cm9sIGRldmxpc3QNCj4gIG52bWUwOiBTQU1TVU5HIE1aVlBWNTEySERHTC0w MDAwMA0KPiAgICAgbnZtZTBuczEgKDQ4ODM4Nk1CKQ0KPiAgbnZtZTE6IFNBTVNVTkcgTVpWUFY1 MTJIREdMLTAwMDAwDQo+ICAgICBudm1lMW5zMSAoNDg4Mzg2TUIpDQo+ICBudm1lMjogU0FNU1VO RyBNWlZQVjUxMkhER0wtMDAwMDANCj4gICAgIG52bWUybnMxICg0ODgzODZNQikNCj4gLS0tDQo+ DQo+DQo+IEkgd2FudCB0byB1c2UgYSB6MSByYWlkIG9uIHRoZXNlIGFuZCBjcmVhdGVkIDFtLWFs aWduZWQgcGFydGl0aW9uczoNCj4NCj4gLS0tDQo+IHJvb3RAc3RvcmFnZTp+ICMgZ3BhcnQgc2hv dw0KPiA9PiAgICAgICAgMzQgIDEwMDAyMTUxNDkgIG52ZDAgIEdQVCAgKDQ3N0cpDQo+ICAgICAg ICAgICAzNCAgICAgICAgMjAxNCAgICAgICAgLSBmcmVlIC0gICgxLjBNKQ0KPiAgICAgICAgIDIw NDggIDEwMDAyMTI0ODAgICAgIDEgIGZyZWVic2QtemZzICAoNDc3RykNCj4gICAxMDAwMjE0NTI4 ICAgICAgICAgNjU1ICAgICAgICAtIGZyZWUgLSAgKDMyOEspDQo+DQo+ID0+ICAgICAgICAzNCAg MTAwMDIxNTE0OSAgbnZkMSAgR1BUICAoNDc3RykNCj4gICAgICAgICAgIDM0ICAgICAgICAyMDE0 ICAgICAgICAtIGZyZWUgLSAgKDEuME0pDQo+ICAgICAgICAgMjA0OCAgMTAwMDIxMjQ4MCAgICAg MSAgZnJlZWJzZC16ZnMgICg0NzdHKQ0KPiAgIDEwMDAyMTQ1MjggICAgICAgICA2NTUgICAgICAg IC0gZnJlZSAtICAoMzI4SykNCj4NCj4gPT4gICAgICAgIDM0ICAxMDAwMjE1MTQ5ICBudmQyICBH UFQgICg0NzdHKQ0KPiAgICAgICAgICAgMzQgICAgICAgIDIwMTQgICAgICAgIC0gZnJlZSAtICAo MS4wTSkNCj4gICAgICAgICAyMDQ4ICAxMDAwMjEyNDgwICAgICAxICBmcmVlYnNkLXpmcyAgKDQ3 N0cpDQo+ICAgMTAwMDIxNDUyOCAgICAgICAgIDY1NSAgICAgICAgLSBmcmVlIC0gICgzMjhLKQ0K PiAtLS0NCj4NCj4NCj4gQWZ0ZXIgY3JlYXRpbmcgYSB6cG9vbCBJIHJlY29nbml6ZWQgdGhhdCBp dCB3YXMgdXNpbmcgYXNoaWZ0PTkuIEkgdmFndWVseQ0KPiByZW1lbWJlcmVkIHRoYXQgU1NEcyB1 c3VhbGx5IGhhdmUgNGsgKG9yIGV2ZW4gbGFyZ2VyKSBzZWN0b3JzLCBzbyBJDQo+IGRlc3Ryb3ll ZCB0aGUgcG9vbCBhbmQgc2V0IHVwIGdub3AtcHJvdmlkZXJzIHdpdGggLVMgNGsgdG8gZ2V0IGFz aGlmdD0xMi4NCj4gVGhpcyB3b3JrZWQgYXMgZXhwZWN0ZWQ6DQo+DQo+IC0tLQ0KPiAgIHBvb2w6 IGZsYXNoDQo+ICBzdGF0ZTogT05MSU5FDQo+ICAgc2Nhbjogbm9uZSByZXF1ZXN0ZWQNCj4gY29u ZmlnOg0KPg0KPiAgICAgICAgIE5BTUUgICAgICAgICAgICAgICAgU1RBVEUgICAgIFJFQUQgV1JJ VEUgQ0tTVU0NCj4gICAgICAgICBmbGFzaCAgICAgICAgICAgICAgIE9OTElORSAgICAgICAwICAg ICAwICAgICAwDQo+ICAgICAgICAgICByYWlkejEtMCAgICAgICAgICBPTkxJTkUgICAgICAgMCAg ICAgMCAgICAgMA0KPiAgICAgICAgICAgICBncHQvZmxhc2gwLm5vcCAgT05MSU5FICAgICAgIDAg ICAgIDAgICAgIDANCj4gICAgICAgICAgICAgZ3B0L2ZsYXNoMS5ub3AgIE9OTElORSAgICAgICAw ICAgICAwICAgICAwDQo+ICAgICAgICAgICAgIGdwdC9mbGFzaDIubm9wICBPTkxJTkUgICAgICAg MCAgICAgMCAgICAgMA0KPg0KPiBlcnJvcnM6IE5vIGtub3duIGRhdGEgZXJyb3JzDQo+IC0tLQ0K Pg0KPg0KPiBUaGlzIHBvb2wgY2FuIGJlIHVzZWQsIGV4cG9ydGVkIGFuZCBpbXBvcnRlZCBqdXN0 IGZpbmUgYXMgZmFyIGFzIEkgY2FuDQo+IHRlbGwuIFRoZW4gSSBleHBvcnRlZCB0aGUgcG9vbCBh bmQgZGVzdHJveWVkIHRoZSBnbm9wLXByb3ZpZGVycy4gV2hlbg0KPiBzdGFydGluZyB3aXRoICJh ZHZhbmNlZCBmb3JtYXQiIGhkZHMgc29tZSB5ZWFycyBhZ28sIHRoaXMgd2FzIHRoZSB3YXkgdG8N Cj4gbWFrZSB6ZnMgcmVjb2duaXplIHRoZSBkaXNrcyB3aXRoIGFzaGlmdD0xMi4gSG93ZXZlciwg ZGVzdHJveWluZyB0aGUNCj4gZ25vcC1kZXZpY2VzIGFwcGVhcnMgdG8gaGF2ZSBjcmFzaGVkIHRo ZSBwb29sIGluIHRoaXMgY2FzZToNCj4NCj4gLS0tDQo+IHJvb3RAc3RvcmFnZTp+ICMgenBvb2wg aW1wb3J0DQo+ICAgIHBvb2w6IGZsYXNoDQo+ICAgICAgaWQ6IDQ5Nzg4Mzk5MzgwMjU4NjM1MjIN Cj4gICBzdGF0ZTogT05MSU5FDQo+ICBzdGF0dXM6IE9uZSBvciBtb3JlIGRldmljZXMgY29udGFp bnMgY29ycnVwdGVkIGRhdGEuDQo+ICBhY3Rpb246IFRoZSBwb29sIGNhbiBiZSBpbXBvcnRlZCB1 c2luZyBpdHMgbmFtZSBvciBudW1lcmljIGlkZW50aWZpZXIuDQo+ICAgIHNlZTogaHR0cDovL2ls bHVtb3Mub3JnL21zZy9aRlMtODAwMC00Sg0KPiAgY29uZmlnOg0KPg0KPiAgICAgICAgIGZsYXNo ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIE9OTElORQ0KPiAgICAg ICAgICAgcmFpZHoxLTAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIE9OTElO RQ0KPiAgICAgICAgICAgICAxMTQ1NjM2NzI4MDMxNjcwODAwMyAgICAgICAgICAgICAgICAgICAg ICAgIFVOQVZBSUwgIGNvcnJ1cHRlZA0KPiBkYXRhIGdwdGlkLzU1YWU3MWFhLWViODQtMTFlNS05 Mjk4LTBjYzQ3YTZjNzQ4NCAgT05MSU5FDQo+ICAgICAgICAgICAgIDY3NjE3ODY5ODMxMzk1NjQx NzIgICAgICAgICAgICAgICAgICAgICAgICAgVU5BVkFJTCAgY29ycnVwdGVkDQo+IGRhdGENCj4g LS0tDQo+DQo+DQo+IEhvdyBjYW4gdGhlIHBvb2wgYmUgb25saW5lLCB3aGVuIHR3byBvZiB0aHJl ZSBkZXZpY2VzIGFyZSB1bmF2YWlsYWJsZT8gSQ0KPiB0cmllZCB0byBpbXBvcnQgdGhlIHBvb2wg bmV2ZXJ0aGVsZXNzLCBidXQgdGhlIHpwb29sIGNvbW1hbmQgZ290IHN0dWNrIGluDQo+IHN0YXRl IHR4LXR4LiAic29mdCIgcmVib290IGdvdCBzdHVjaywgdG9vLiBJIGhhZCB0byBwdXNoIHRoZSBy ZXNldCBidXR0b24NCj4gdG8gZ2V0IG15IHN5c3RlbSBiYWNrIChzdGlsbCB3aXRoIGEgY29ycnVw dCBwb29sKS4gSSBjbGVhcmVkIHRoZSBsYWJlbHMNCj4gYW5kIHJlLWRpZCBldmVyeXRoaW5nOiB0 aGUgaXNzdWUgaXMgcGVyZmVjdGx5IHJlcHJvZHVjaWJsZS4NCj4NCj4gQW0gSSBkb2luZyBzb21l dGhpbmcgdXR0ZXJseSB3cm9uZz8gV2h5IGlzIHJlbW92aW5nIHRoZSBnbm9wLW5vZGVzDQo+IHRh bXBlcmluZyB3aXRoIHRoZSBkZXZpY2VzIChJIHRoaW5rIEkgZGlkIGV4YWN0bHkgdGhpcyBkb3pl bnMgb2YgdGltZXMgb24NCj4gbm9ybWFsIGhkZHMgZHVyaW5nIHRoYXQgcHJldmlvdXMgeWVhcnMs IGFuZCBpdCBhbHdheXMgd29ya2VkIGp1c3QgZmluZSk/DQo+IEFuZCBmaW5hbGx5LCB3aHkgZG9l cyB0aGUgenBvb2wgaW1wb3J0IGZhaWwgd2l0aG91dCBhbnkgZXJyb3IgbWVzc2FnZSBhbmQNCj4g cmVxdWlyZXMgbWUgdG8gcmVzZXQgdGhlIHN5c3RlbT8NCj4gVGhlIHN5c3RlbSBpcyAxMC4yLVJF TEVBU0UtcDksIHVwZGF0ZSBpcyBzY2hlZHVsZWQgZm9yIGxhdGVyIHRoaXMgd2Vlaw0KPiAoanVz dCBpbiBjYXNlIGl0IHdvdWxkIG1ha2Ugc2Vuc2UgdG8gdHJ5IHRoaXMgYWdhaW4gd2l0aCAxMC4z KS4gQW55IG90aGVyDQo+IGhpbnRzIGFyZSBtb3N0IHdlbGNvbWUuDQo+DQo+DQo+IGN1DQo+ICAg R2Vycml0DQo+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f DQo+IGZyZWVic2QtZnNAZnJlZWJzZC5vcmcgbWFpbGluZyBsaXN0DQo+IGh0dHBzOi8vbGlzdHMu ZnJlZWJzZC5vcmcvbWFpbG1hbi9saXN0aW5mby9mcmVlYnNkLWZzDQo+IFRvIHVuc3Vic2NyaWJl LCBzZW5kIGFueSBtYWlsIHRvICJmcmVlYnNkLWZzLXVuc3Vic2NyaWJlQGZyZWVic2Qub3JnIg0K DQpZb3UgcHJvYmFibHkgbmVlZCB0byB0ZWxsIHpmcyAid2hlcmUiIHRoZSBwcm92aWRlcnMgYXJl Og0KIyB6cG9vbCBpbXBvcnQgLWQgL2Rldi9ncHQgZmxhc2gNCg0KL0sNCg== From owner-freebsd-fs@freebsd.org Wed Apr 27 16:59:28 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9C186B1EA62 for ; Wed, 27 Apr 2016 16:59:28 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x232.google.com (mail-wm0-x232.google.com [IPv6:2a00:1450:400c:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 29B3A1CD1 for ; Wed, 27 Apr 2016 16:59:28 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x232.google.com with SMTP id a17so24480911wme.0 for ; Wed, 27 Apr 2016 09:59:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=fHQ36Ku2CL6XQycu3paeLzw6jb8zJs59tv1KjXLN220=; b=frj/m9xzi+3jM4QpIMuIR5n4LLevCbi85d5rLIDsIAsByRUP6bhcTGA1V7jNWpXONn nzMgwL5WcpnNT3UxXIryHFFIoIr74MrKvSr6+Sl+FDL2LSY2xCYzGUCpHxzRQqWO4vvp JF5u2lwkrNL9FWMDB+4u9UI1VdYdsQdCXlWJP5zD1a0p1rS9IIMtFPIcARMKGJcWJXsR mlLuUVQ8vXyMqW2I2k3FpaCcPi6NiOgWZminLaEVOgqXzB1M9eTCDnjnYkRR6/Vzbe23 ZS/+a1u818IRdDORivOBJIINkO1nn1zQMXYS0QHZgUsptn28QA3hQkCwZ528GgEZxMNo tQQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=fHQ36Ku2CL6XQycu3paeLzw6jb8zJs59tv1KjXLN220=; b=WimhDhaLaYkNMN3WNRnhLb35MT1ndQGV1FJCEM3dX+d3JXjAw0UJ4YeYmIkUl1pJiv omYGjdJu7CUkX6fVsW9gkdIn4mLgsiVaMG+mIcvr5vdRlykUwv2CRQW8qMUvPZzyhFvy LCKeA2YDlYfb9EMhodsrC/V9TTILxLTAhfYYpa7MUHL/DL+sTmUwjemWYnJGAuad1tKd R7Q7Pr4adh74vpUnylAa8/6RyE+/2LfGf4pWqaQZqsNXewdSSgOaRSVZL9AD01pD79s3 aa8EQs6h+PtqEs9TKP2Oei3RfoSyskIvliEI3RJmw5DCbUHm34Gw/0lCeVHMV2RLFh1D ziUQ== X-Gm-Message-State: AOPr4FUPeOZJzWV1aoZUwvuKe+/BmsodNrjT/rsJkE87sXEdICE6DkLg2Fa6IHU4MAB31m2K X-Received: by 10.28.39.5 with SMTP id n5mr25693499wmn.13.1461776366569; Wed, 27 Apr 2016 09:59:26 -0700 (PDT) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id o4sm4970481wjx.45.2016.04.27.09.59.24 for (version=TLSv1/SSLv3 cipher=OTHER); Wed, 27 Apr 2016 09:59:24 -0700 (PDT) Subject: Re: zfs on nvme: gnop breaks pool, zfs gets stuck To: freebsd-fs@freebsd.org References: <20160427152244.ff36ff74ae64c1f86fdc960a@aei.mpg.de> <20160427141436.GA60370@in-addr.com> From: Steven Hartland Message-ID: <5720EFD8.60900@multiplay.co.uk> Date: Wed, 27 Apr 2016 17:59:04 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <20160427141436.GA60370@in-addr.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 16:59:28 -0000 On 27/04/2016 15:14, Gary Palmer wrote: > On Wed, Apr 27, 2016 at 03:22:44PM +0200, Gerrit K?hn wrote: >> Hello all, >> >> I have a set of three NVME-ssds on PCIe-converters: >> >> --- >> root@storage:~ # nvmecontrol devlist >> nvme0: SAMSUNG MZVPV512HDGL-00000 >> nvme0ns1 (488386MB) >> nvme1: SAMSUNG MZVPV512HDGL-00000 >> nvme1ns1 (488386MB) >> nvme2: SAMSUNG MZVPV512HDGL-00000 >> nvme2ns1 (488386MB) >> --- >> >> >> I want to use a z1 raid on these and created 1m-aligned partitions: >> >> --- >> root@storage:~ # gpart show >> => 34 1000215149 nvd0 GPT (477G) >> 34 2014 - free - (1.0M) >> 2048 1000212480 1 freebsd-zfs (477G) >> 1000214528 655 - free - (328K) >> >> => 34 1000215149 nvd1 GPT (477G) >> 34 2014 - free - (1.0M) >> 2048 1000212480 1 freebsd-zfs (477G) >> 1000214528 655 - free - (328K) >> >> => 34 1000215149 nvd2 GPT (477G) >> 34 2014 - free - (1.0M) >> 2048 1000212480 1 freebsd-zfs (477G) >> 1000214528 655 - free - (328K) >> --- >> >> >> After creating a zpool I recognized that it was using ashift=9. I vaguely >> remembered that SSDs usually have 4k (or even larger) sectors, so I >> destroyed the pool and set up gnop-providers with -S 4k to get ashift=12. >> This worked as expected: >> >> --- >> pool: flash >> state: ONLINE >> scan: none requested >> config: >> >> NAME STATE READ WRITE CKSUM >> flash ONLINE 0 0 0 >> raidz1-0 ONLINE 0 0 0 >> gpt/flash0.nop ONLINE 0 0 0 >> gpt/flash1.nop ONLINE 0 0 0 >> gpt/flash2.nop ONLINE 0 0 0 >> >> errors: No known data errors >> --- >> >> >> This pool can be used, exported and imported just fine as far as I can >> tell. Then I exported the pool and destroyed the gnop-providers. When >> starting with "advanced format" hdds some years ago, this was the way to >> make zfs recognize the disks with ashift=12. However, destroying the >> gnop-devices appears to have crashed the pool in this case: >> >> --- >> root@storage:~ # zpool import >> pool: flash >> id: 4978839938025863522 >> state: ONLINE >> status: One or more devices contains corrupted data. >> action: The pool can be imported using its name or numeric identifier. >> see: http://illumos.org/msg/ZFS-8000-4J >> config: >> >> flash ONLINE >> raidz1-0 ONLINE >> 11456367280316708003 UNAVAIL corrupted >> data gptid/55ae71aa-eb84-11e5-9298-0cc47a6c7484 ONLINE >> 6761786983139564172 UNAVAIL corrupted >> data >> --- >> >> >> How can the pool be online, when two of three devices are unavailable? I >> tried to import the pool nevertheless, but the zpool command got stuck in >> state tx-tx. "soft" reboot got stuck, too. I had to push the reset button >> to get my system back (still with a corrupt pool). I cleared the labels >> and re-did everything: the issue is perfectly reproducible. >> >> Am I doing something utterly wrong? Why is removing the gnop-nodes >> tampering with the devices (I think I did exactly this dozens of times on >> normal hdds during that previous years, and it always worked just fine)? >> And finally, why does the zpool import fail without any error message and >> requires me to reset the system? >> The system is 10.2-RELEASE-p9, update is scheduled for later this week >> (just in case it would make sense to try this again with 10.3). Any other >> hints are most welcome. > Did you destroy the gnop devices with the pool online? In the procedure > I remember you export the pool, destroy the gnop devices, and then > reimport the pool. > > Also, you only need to do the gnop trick for a single device in the pool > for the entire pool's ashift to be changed AFAIK. There is a sysctl > now too > > vfs.zfs.min_auto_ashift > > which lets you manage the ashift on a new pool without having to try > the gnop trick > This applies to each top level vdev that makes up a pool, so its not limited to just new pool creation, so there should be never a reason to use the gnop hack to set ashift. Regards Steve From owner-freebsd-fs@freebsd.org Wed Apr 27 17:46:09 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 253FCB1ECBD for ; Wed, 27 Apr 2016 17:46:09 +0000 (UTC) (envelope-from nowakpl@platinum.linux.pl) Received: from platinum.linux.pl (platinum.edu.pl [51.255.84.233]) by mx1.freebsd.org (Postfix) with ESMTP id E792114E6 for ; Wed, 27 Apr 2016 17:46:08 +0000 (UTC) (envelope-from nowakpl@platinum.linux.pl) Received: by platinum.linux.pl (Postfix, from userid 87) id 51B29A001A4; Wed, 27 Apr 2016 19:36:43 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on platinum.edu.pl X-Spam-Level: X-Spam-Status: No, score=-1.3 required=3.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.4.1 Received: from [10.255.1.11] (unknown [83.151.38.73]) by platinum.linux.pl (Postfix) with ESMTPA id D0618A001A0 for ; Wed, 27 Apr 2016 19:36:35 +0200 (CEST) Subject: Re: How to speed up slow zpool scrub? To: freebsd-fs@freebsd.org References: <698816653.2698619.1461685653634.JavaMail.yahoo.ref@mail.yahoo.com> <698816653.2698619.1461685653634.JavaMail.yahoo@mail.yahoo.com> <571F9897.2070008@quip.cz> <571FEB34.7040305@andyit.com.au> <56C0A956-F134-4A8D-A8B6-B93DCA045BE4@pk1048.com> <084201d1a03e$d2158fe0$7640afa0$@andyit.com.au> <5720AAF8.4090900@quip.cz> From: Adam Nowacki Message-ID: <5720F890.3040600@platinum.linux.pl> Date: Wed, 27 Apr 2016 19:36:16 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <5720AAF8.4090900@quip.cz> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 17:46:09 -0000 On 2016-04-27 14:05, Miroslav Lachman wrote: > Andy Farkas wrote on 04/27/2016 06:39: >>> -----Original Message----- >>> From: PK1048 [mailto:paul@pk1048.com] >>> Sent: Wednesday, 27 April 2016 12:34 PM >>> To: Andy Farkas >>> Cc: freebsd-fs@freebsd.org >>> Subject: Re: How to speed up slow zpool scrub? >>> >>> ... >>> Scrubs (and resilver) operations are essentially all random I/O. Those >>> drives are low end, low performance, desktop drives. >> >> Yes, the system is an old low end, low performance desktop. That was >> my point, that it took 25 hours to scrub 7.52T and not 4 days like the >> OP is saying. > > Thank you for output of your zpool scrub. It is definitely faster than > mine. > > To: Paul pk1048 > Mine scrub does not repair anything. Drives are OK (in SMART). > CPU is about 70%-90% idle during scrub + rsync backup and drives are > about 60%-70% busy according to iostat: > > root@kiwi ~/# iostat -x -w 10 ada0 ada1 ada2 ada3 > extended device statistics > device r/s w/s kr/s kw/s qlen svc_t %b > ada0 70.1 17.9 1747.0 802.4 0 7.0 24 > ada1 70.1 17.9 1747.1 802.4 0 7.0 25 > ada2 66.9 17.1 1686.4 791.3 4 6.5 23 > ada3 66.9 16.9 1686.3 790.1 2 6.6 23 > extended device statistics > device r/s w/s kr/s kw/s qlen svc_t %b > ada0 93.6 13.0 576.4 244.0 0 21.6 70 > ada1 98.6 12.9 587.2 246.4 2 20.5 71 > ada2 87.9 15.3 566.0 242.4 3 20.5 67 > ada3 84.9 14.5 549.2 237.2 3 20.4 66 > extended device statistics > device r/s w/s kr/s kw/s qlen svc_t %b > ada0 98.7 42.5 1924.7 2536.3 1 26.3 86 > ada1 99.1 45.5 1931.5 2671.5 1 23.8 87 > ada2 94.2 44.9 1840.7 2720.3 0 20.1 76 > ada3 93.6 42.7 1807.9 2607.1 0 18.7 75 > extended device statistics > device r/s w/s kr/s kw/s qlen svc_t %b > ada0 108.2 28.2 1092.6 1316.6 2 17.3 68 > ada1 101.6 26.3 1053.8 1183.4 3 15.5 67 > ada2 98.6 26.0 1000.2 1126.2 2 12.2 57 > ada3 104.0 24.0 1015.8 1080.6 3 14.1 60 > extended device statistics > device r/s w/s kr/s kw/s qlen svc_t %b > ada0 116.0 18.5 821.8 807.8 0 12.9 62 > ada1 117.2 18.5 822.2 807.0 0 13.5 63 > ada2 110.8 20.9 743.0 803.8 0 11.1 58 > ada3 108.2 20.0 688.2 755.0 2 11.3 55 > extended device statistics > device r/s w/s kr/s kw/s qlen svc_t %b > ada0 121.8 16.6 602.1 526.9 3 9.2 52 > ada1 122.2 16.5 606.9 528.5 4 9.8 54 > ada2 117.0 14.6 601.7 524.9 2 11.3 60 > ada3 120.6 13.5 610.1 491.3 0 11.4 61 > > I really don't know why it cannot go faster if nothing is loaded for 100%. 1) zpool scrub is single threaded with prefetch, 2) some data blocks do not span all disks (metadata, small files, compression) End result is that zfs can't always read from all disks during scrub so disk utilization is going to be less than 100% even when going at full speed. From owner-freebsd-fs@freebsd.org Wed Apr 27 18:00:54 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8855AB1E2EC for ; Wed, 27 Apr 2016 18:00:54 +0000 (UTC) (envelope-from bsdunix44@gmail.com) Received: from mail-oi0-x233.google.com (mail-oi0-x233.google.com [IPv6:2607:f8b0:4003:c06::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 55C38124B for ; Wed, 27 Apr 2016 18:00:54 +0000 (UTC) (envelope-from bsdunix44@gmail.com) Received: by mail-oi0-x233.google.com with SMTP id x19so57612458oix.2 for ; Wed, 27 Apr 2016 11:00:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:mime-version:in-reply-to:content-transfer-encoding :message-id:cc:from:subject:date:to; bh=HQbqbywN/cMbr1u5TVIej631mkJiGlJag+BcnttVJgg=; b=lA8fqdc02fROV4PZrvpgUF6+nifBZbzBLv2tBt6ui0mBKAFkhLs65nVx85hWmJgRst YXhDf268vXgYcSKRM4RWNt6Ok9Nacf6bbera7bhxq4zzCwg/elqrp/v/CTUn2k8fGj6X uTDJ6x6BrV4eLps+yfDfzly+Ee8RtdAMi2BttNBKvN91KxOP8BL4NRTxfFz07JabMhY1 RfONMxFYyNUlgW43SkEAJNyM+lxAYOPvs2QixdIKNYVhNUHOG1Nn14/EQOVSmHSTf8XR 65sQZw2xqX8UjoJPh3+Xz6Y4ioANJaDxGkLmpmx7Yh3ScwWn2z5Dkyy2GS2fuiiPpg1Z rIag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:references:mime-version:in-reply-to :content-transfer-encoding:message-id:cc:from:subject:date:to; bh=HQbqbywN/cMbr1u5TVIej631mkJiGlJag+BcnttVJgg=; b=I7+BBgsVeGQVnVEylT4hEpdAEoZOooShuB3UvqyqBT+Hk/mXSL+ylluTkVqwEJGDzk 51dUirkpnJ1rccaVntrR5DWyuI/czlnG7pw1WjJAql963WXhfq0RgOPQTseWDLxXPR2P n2rt/voODNhThje1LLV+55/t0DlEmpbgt/4ZLd4WP4PM+Y4SNYm16NL0Qyjb9YEShpV7 XT4L0KGLX9yYkt/2AkJorOZbcrPCtO5NcdNW1H3hPTVQr40kAuswbaHuNT1geoC0O4qK 4mLQRCm9HaYBHd+9PAlIANSlyx46fAJ+GDnctCP28FY94TySg4erEbNpiVkJ95MSWWfT tbMQ== X-Gm-Message-State: AOPr4FVrvEzqZwmxvn26zzbg+HsqVf2YZSE/xrheICH2nSIhBDG4U68arHrNMKG8g/qPsA== X-Received: by 10.202.105.78 with SMTP id e75mr4064395oic.132.1461780053634; Wed, 27 Apr 2016 11:00:53 -0700 (PDT) Received: from [192.168.0.10] (cpe-70-118-225-173.kc.res.rr.com. [70.118.225.173]) by smtp.gmail.com with ESMTPSA id u41sm721684otd.37.2016.04.27.11.00.51 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 27 Apr 2016 11:00:52 -0700 (PDT) References: <20160427152244.ff36ff74ae64c1f86fdc960a@aei.mpg.de> <20160427141436.GA60370@in-addr.com> <5720EFD8.60900@multiplay.co.uk> Mime-Version: 1.0 (1.0) In-Reply-To: <5720EFD8.60900@multiplay.co.uk> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Message-Id: Cc: freebsd-fs@freebsd.org X-Mailer: iPhone Mail (13E238) From: Chris Watson Subject: Re: zfs on nvme: gnop breaks pool, zfs gets stuck Date: Wed, 27 Apr 2016 13:00:50 -0500 To: Steven Hartland X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 18:00:54 -0000 I think for most people, the gnop hack is what is documented on the web. Hen= ce why people are using it versus the ashift sysctl. If the sysctl for ashif= t is not documented in the ZFS section of the handbook, it probably should b= e.=20 Chris Sent from my iPhone 5 > On Apr 27, 2016, at 11:59 AM, Steven Hartland wr= ote: >=20 >=20 >=20 >> On 27/04/2016 15:14, Gary Palmer wrote: >>> On Wed, Apr 27, 2016 at 03:22:44PM +0200, Gerrit K?hn wrote: >>> Hello all, >>>=20 >>> I have a set of three NVME-ssds on PCIe-converters: >>>=20 >>> --- >>> root@storage:~ # nvmecontrol devlist >>> nvme0: SAMSUNG MZVPV512HDGL-00000 >>> nvme0ns1 (488386MB) >>> nvme1: SAMSUNG MZVPV512HDGL-00000 >>> nvme1ns1 (488386MB) >>> nvme2: SAMSUNG MZVPV512HDGL-00000 >>> nvme2ns1 (488386MB) >>> --- >>>=20 >>>=20 >>> I want to use a z1 raid on these and created 1m-aligned partitions: >>>=20 >>> --- >>> root@storage:~ # gpart show >>> =3D> 34 1000215149 nvd0 GPT (477G) >>> 34 2014 - free - (1.0M) >>> 2048 1000212480 1 freebsd-zfs (477G) >>> 1000214528 655 - free - (328K) >>>=20 >>> =3D> 34 1000215149 nvd1 GPT (477G) >>> 34 2014 - free - (1.0M) >>> 2048 1000212480 1 freebsd-zfs (477G) >>> 1000214528 655 - free - (328K) >>>=20 >>> =3D> 34 1000215149 nvd2 GPT (477G) >>> 34 2014 - free - (1.0M) >>> 2048 1000212480 1 freebsd-zfs (477G) >>> 1000214528 655 - free - (328K) >>> --- >>>=20 >>>=20 >>> After creating a zpool I recognized that it was using ashift=3D9. I vagu= ely >>> remembered that SSDs usually have 4k (or even larger) sectors, so I >>> destroyed the pool and set up gnop-providers with -S 4k to get ashift=3D= 12. >>> This worked as expected: >>>=20 >>> --- >>> pool: flash >>> state: ONLINE >>> scan: none requested >>> config: >>>=20 >>> NAME STATE READ WRITE CKSUM >>> flash ONLINE 0 0 0 >>> raidz1-0 ONLINE 0 0 0 >>> gpt/flash0.nop ONLINE 0 0 0 >>> gpt/flash1.nop ONLINE 0 0 0 >>> gpt/flash2.nop ONLINE 0 0 0 >>>=20 >>> errors: No known data errors >>> --- >>>=20 >>>=20 >>> This pool can be used, exported and imported just fine as far as I can >>> tell. Then I exported the pool and destroyed the gnop-providers. When >>> starting with "advanced format" hdds some years ago, this was the way to= >>> make zfs recognize the disks with ashift=3D12. However, destroying the >>> gnop-devices appears to have crashed the pool in this case: >>>=20 >>> --- >>> root@storage:~ # zpool import >>> pool: flash >>> id: 4978839938025863522 >>> state: ONLINE >>> status: One or more devices contains corrupted data. >>> action: The pool can be imported using its name or numeric identifier. >>> see: http://illumos.org/msg/ZFS-8000-4J >>> config: >>>=20 >>> flash ONLINE >>> raidz1-0 ONLINE >>> 11456367280316708003 UNAVAIL corrupted >>> data gptid/55ae71aa-eb84-11e5-9298-0cc47a6c7484 ONLINE >>> 6761786983139564172 UNAVAIL corrupted >>> data >>> --- >>>=20 >>>=20 >>> How can the pool be online, when two of three devices are unavailable? I= >>> tried to import the pool nevertheless, but the zpool command got stuck i= n >>> state tx-tx. "soft" reboot got stuck, too. I had to push the reset butto= n >>> to get my system back (still with a corrupt pool). I cleared the labels >>> and re-did everything: the issue is perfectly reproducible. >>>=20 >>> Am I doing something utterly wrong? Why is removing the gnop-nodes >>> tampering with the devices (I think I did exactly this dozens of times o= n >>> normal hdds during that previous years, and it always worked just fine)?= >>> And finally, why does the zpool import fail without any error message an= d >>> requires me to reset the system? >>> The system is 10.2-RELEASE-p9, update is scheduled for later this week >>> (just in case it would make sense to try this again with 10.3). Any othe= r >>> hints are most welcome. >> Did you destroy the gnop devices with the pool online? In the procedure >> I remember you export the pool, destroy the gnop devices, and then >> reimport the pool. >>=20 >> Also, you only need to do the gnop trick for a single device in the pool >> for the entire pool's ashift to be changed AFAIK. There is a sysctl >> now too >>=20 >> vfs.zfs.min_auto_ashift >>=20 >> which lets you manage the ashift on a new pool without having to try >> the gnop trick > This applies to each top level vdev that makes up a pool, so its not limit= ed to just new pool creation, so there should be never a reason to use the g= nop hack to set ashift. >=20 > Regards > Steve > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Wed Apr 27 18:11:31 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61FCFB1E7FB for ; Wed, 27 Apr 2016 18:11:31 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 257681D41 for ; Wed, 27 Apr 2016 18:11:30 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 48FEE28416; Wed, 27 Apr 2016 20:11:22 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 6BCEB2840C; Wed, 27 Apr 2016 20:11:21 +0200 (CEST) Message-ID: <572100C9.8010606@quip.cz> Date: Wed, 27 Apr 2016 20:11:21 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: Adam Nowacki , freebsd-fs@freebsd.org Subject: Re: How to speed up slow zpool scrub? References: <698816653.2698619.1461685653634.JavaMail.yahoo.ref@mail.yahoo.com> <698816653.2698619.1461685653634.JavaMail.yahoo@mail.yahoo.com> <571F9897.2070008@quip.cz> <571FEB34.7040305@andyit.com.au> <56C0A956-F134-4A8D-A8B6-B93DCA045BE4@pk1048.com> <084201d1a03e$d2158fe0$7640afa0$@andyit.com.au> <5720AAF8.4090900@quip.cz> <5720F890.3040600@platinum.linux.pl> In-Reply-To: <5720F890.3040600@platinum.linux.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 18:11:31 -0000 Adam Nowacki wrote on 04/27/2016 19:36: > On 2016-04-27 14:05, Miroslav Lachman wrote: >> Thank you for output of your zpool scrub. It is definitely faster than >> mine. >> >> To: Paul pk1048 >> Mine scrub does not repair anything. Drives are OK (in SMART). >> CPU is about 70%-90% idle during scrub + rsync backup and drives are >> about 60%-70% busy according to iostat: >> >> root@kiwi ~/# iostat -x -w 10 ada0 ada1 ada2 ada3 >> device r/s w/s kr/s kw/s qlen svc_t %b >> ada0 121.8 16.6 602.1 526.9 3 9.2 52 >> ada1 122.2 16.5 606.9 528.5 4 9.8 54 >> ada2 117.0 14.6 601.7 524.9 2 11.3 60 >> ada3 120.6 13.5 610.1 491.3 0 11.4 61 >> >> I really don't know why it cannot go faster if nothing is loaded for 100%. > > 1) zpool scrub is single threaded with prefetch, Hmm, this can be the cause. Does it mean that ZFS is faster on CPU with higher "per core" power and number of cores (threads) is not so important? (in this case of scrub) > 2) some data blocks do not span all disks (metadata, small files, > compression) > End result is that zfs can't always read from all disks during scrub so > disk utilization is going to be less than 100% even when going at full > speed. Thank you for the explanation. Miroslav Lachman From owner-freebsd-fs@freebsd.org Wed Apr 27 19:59:26 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 28A92B1E03E for ; Wed, 27 Apr 2016 19:59:26 +0000 (UTC) (envelope-from brandon.wandersee@gmail.com) Received: from mail-io0-f196.google.com (mail-io0-f196.google.com [209.85.223.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EA1731053 for ; Wed, 27 Apr 2016 19:59:25 +0000 (UTC) (envelope-from brandon.wandersee@gmail.com) Received: by mail-io0-f196.google.com with SMTP id u185so9011744iod.2 for ; Wed, 27 Apr 2016 12:59:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:references:user-agent:from:to:cc:subject :in-reply-to:date:message-id:mime-version; bh=2Bttn2jyi2SN2tOwPommxSC2OUpJLQhw2y8XTtcjErc=; b=h9OOXfdsQI4SwDnv6fjsODHR8osjnMxtccm1qk4JN+u9YK13bKf1wBLUbpfGxZoYCQ YMe34RMd8OGOQ3LVkRB8z6eU08E7jCSS/ylCSC7Lz5gxJDJ8ZjTJrIW7MhYIfdjUVgn0 h8wKdkwME6yb1d6NON1MCaqvU/238SaFl5/pL2sqqhI0jdmMxWeZ2yx7FbjpIa7vQF2I K7M24tXuO1wH6G2slX3lNx2S1LVovVZCH/46WQgSziD0fJrcvu6EG1a/1EzKQXGPrH/8 VzKqiotxA1cxkfEfYCfj65Id7hinJAP1aIYEOIihOPywUMRoU8sHNuUcAh2cbwnHiQ+s LIuA== X-Gm-Message-State: AOPr4FVXUJZSDmeqIAGUvPmps+Wezh4Q3QDjQRJjeZ/D4Wz0Rjh5G7NNbd8gIE2OhjH43g== X-Received: by 10.107.136.208 with SMTP id s77mr14287823ioi.0.1461786346632; Wed, 27 Apr 2016 12:45:46 -0700 (PDT) Received: from WorkBox.Home.gmail.com (63-231-164-200.mpls.qwest.net. [63.231.164.200]) by smtp.gmail.com with ESMTPSA id i9sm5225824ioo.38.2016.04.27.12.45.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Apr 2016 12:45:45 -0700 (PDT) References: <381846248.2672053.1461695277122.JavaMail.yahoo.ref@mail.yahoo.com> <381846248.2672053.1461695277122.JavaMail.yahoo@mail.yahoo.com> User-agent: mu4e 0.9.16; emacs 24.5.1 From: Brandon J. Wandersee To: krad Cc: DH , FreeBSD FS Subject: Re: How to speed up slow zpool scrub? In-reply-to: Date: Wed, 27 Apr 2016 14:45:43 -0500 Message-ID: <86d1pax2c8.fsf@WorkBox.Home> MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 19:59:26 -0000 krad writes: > That memory requirement really depends on your work load. I have run small > boxes with 2gb and zfs and they ran fine. They weren't doing filer or > desktop roles though I could get away with fairly easily. Having said that > I wouldn't build a machine now with less than 16gb unless it was very > special circumstances. > Both the FreeNAS documentation and community make some pretty strong recommendations, but last time I checked their docs they admitted their recommendations were aimed more at 100%-uptime professional use cases than casual, personal use cases. Their ECC RAM recommendation also came with a qualification: basically, "If you can't afford it, and downtime to restore a backup is a minor inconvenience, don't worry about it." At this point time, if you're buying all new parts for a DIY storage project, 16Gb of decent non-ECC RAM is likely to be your smallest expense. But if you just want to get some light use out of a spare motherboard at home, less RAM is probably fine. Not necessarily a great idea, but not a bad one either. -- :: Brandon J. Wandersee :: brandon.wandersee@gmail.com :: -------------------------------------------------- :: 'The best design is as little design as possible.' :: --- Dieter Rams ---------------------------------- From owner-freebsd-fs@freebsd.org Wed Apr 27 20:20:42 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6FFEAB1E5A8 for ; Wed, 27 Apr 2016 20:20:42 +0000 (UTC) (envelope-from nowakpl@platinum.linux.pl) Received: from platinum.linux.pl (platinum.edu.pl [51.255.84.233]) by mx1.freebsd.org (Postfix) with ESMTP id 3D9E21F6E for ; Wed, 27 Apr 2016 20:20:42 +0000 (UTC) (envelope-from nowakpl@platinum.linux.pl) Received: by platinum.linux.pl (Postfix, from userid 87) id 057BBA001A5; Wed, 27 Apr 2016 22:20:39 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on platinum.edu.pl X-Spam-Level: X-Spam-Status: No, score=-1.3 required=3.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.4.1 Received: from [10.255.1.11] (unknown [83.151.38.73]) by platinum.linux.pl (Postfix) with ESMTPA id AF2F3A001A0; Wed, 27 Apr 2016 22:20:34 +0200 (CEST) Subject: Re: How to speed up slow zpool scrub? To: Miroslav Lachman <000.fbsd@quip.cz>, freebsd-fs@freebsd.org References: <698816653.2698619.1461685653634.JavaMail.yahoo.ref@mail.yahoo.com> <698816653.2698619.1461685653634.JavaMail.yahoo@mail.yahoo.com> <571F9897.2070008@quip.cz> <571FEB34.7040305@andyit.com.au> <56C0A956-F134-4A8D-A8B6-B93DCA045BE4@pk1048.com> <084201d1a03e$d2158fe0$7640afa0$@andyit.com.au> <5720AAF8.4090900@quip.cz> <5720F890.3040600@platinum.linux.pl> <572100C9.8010606@quip.cz> From: Adam Nowacki Message-ID: <57211EFF.4000000@platinum.linux.pl> Date: Wed, 27 Apr 2016 22:20:15 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <572100C9.8010606@quip.cz> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 20:20:42 -0000 On 2016-04-27 20:11, Miroslav Lachman wrote: > Adam Nowacki wrote on 04/27/2016 19:36: >> On 2016-04-27 14:05, Miroslav Lachman wrote: > >>> Thank you for output of your zpool scrub. It is definitely faster than >>> mine. >>> >>> To: Paul pk1048 >>> Mine scrub does not repair anything. Drives are OK (in SMART). >>> CPU is about 70%-90% idle during scrub + rsync backup and drives are >>> about 60%-70% busy according to iostat: >>> >>> root@kiwi ~/# iostat -x -w 10 ada0 ada1 ada2 ada3 > >>> device r/s w/s kr/s kw/s qlen svc_t %b >>> ada0 121.8 16.6 602.1 526.9 3 9.2 52 >>> ada1 122.2 16.5 606.9 528.5 4 9.8 54 >>> ada2 117.0 14.6 601.7 524.9 2 11.3 60 >>> ada3 120.6 13.5 610.1 491.3 0 11.4 61 >>> >>> I really don't know why it cannot go faster if nothing is loaded for >>> 100%. >> >> 1) zpool scrub is single threaded with prefetch, > > Hmm, this can be the cause. Does it mean that ZFS is faster on CPU with > higher "per core" power and number of cores (threads) is not so > important? (in this case of scrub) No. Zpool scrub thread doesn't need much CPU time as disk I/O handling (including decompression and checkums) happens in other threads. >> 2) some data blocks do not span all disks (metadata, small files, >> compression) >> End result is that zfs can't always read from all disks during scrub so >> disk utilization is going to be less than 100% even when going at full >> speed. > > Thank you for the explanation. Try increasing vfs.zfs.top_maxinflight to 100 or more. From owner-freebsd-fs@freebsd.org Wed Apr 27 20:45:02 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 11483B1EDB8 for ; Wed, 27 Apr 2016 20:45:02 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x234.google.com (mail-wm0-x234.google.com [IPv6:2a00:1450:400c:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A471312B0 for ; Wed, 27 Apr 2016 20:45:01 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x234.google.com with SMTP id g17so9425097wme.1 for ; Wed, 27 Apr 2016 13:45:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=84IL+hH2Bn1ajkNcuqNMclwwB8Qs665DsTJGgHa2OGE=; b=o3vw9+exErOisJeIy1L5CckI5sau2QWhpS6cB5JkIX0RIqR9SrWtMNz27llSP5eqAE tr+1ST9hP6j5+Soyvm13Q5L8dmz/OvPymA1T3uVYERFKFN9bZE1cGpNRdJqkCYhY8dfT X7m60+1JUdWecQXgsjYa2TrZARC8wvBcYb/oES0mcHPp/ZSyE4B3Xz+Ku79S6ZVk5taw FUi9xoy+EEUFwWmfDXL+Lo8VlGj5Qsh3JgvJqss4asP+d881vw9QaKJ5T6PoT0sR85WG URnwSQuvyis4Ddu35P80h2g4/1WWpoXNOvOlJVhffycJQsmv27QYe020gWWUTV2cQy7T JwMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=84IL+hH2Bn1ajkNcuqNMclwwB8Qs665DsTJGgHa2OGE=; b=FgAiiqnrjLBsPneh1I/Z3ut9N15FoBLUDjXQIP4zU/CNUSTE1gOet+67brKbwl03IF xYON74cSTQiS0JXrWljWti+ujl2NgvzMOS/FEMjHXcYL1LNmBeOP7kiN75afZlA6W74a qAvy2dYH0MNoIWv+ADuo7o1VRsn0YpqNH5FKAbc+oZoJzfkrVU9vKWmHLNQYj2SrcGDB VzJ5LXuO1p/HOxMIWov6YLv4uANfc/mNdaINhrgkHF5P1qoc03I1YMu/r6Dgq3mRyAI4 ORer8gD2O/mKy0nc/5go4VFFUld+ApdvPd9hSegAvGMYtTKzZpmTMg2BFjvvDgTuYk8L pNGw== X-Gm-Message-State: AOPr4FVxW0hJ91crkRjSm1C5R8W+B6gfmowr/7fbJtX85O6R5KejnViZm5++/s3iPTprYGGC X-Received: by 10.194.107.74 with SMTP id ha10mr11422779wjb.24.1461789899936; Wed, 27 Apr 2016 13:44:59 -0700 (PDT) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id ju3sm5910831wjb.11.2016.04.27.13.44.58 for (version=TLSv1/SSLv3 cipher=OTHER); Wed, 27 Apr 2016 13:44:58 -0700 (PDT) Subject: Re: How to speed up slow zpool scrub? To: freebsd-fs@freebsd.org References: <698816653.2698619.1461685653634.JavaMail.yahoo.ref@mail.yahoo.com> <698816653.2698619.1461685653634.JavaMail.yahoo@mail.yahoo.com> <571F9897.2070008@quip.cz> <571FEB34.7040305@andyit.com.au> <56C0A956-F134-4A8D-A8B6-B93DCA045BE4@pk1048.com> <084201d1a03e$d2158fe0$7640afa0$@andyit.com.au> <5720AAF8.4090900@quip.cz> <5720F890.3040600@platinum.linux.pl> <572100C9.8010606@quip.cz> <57211EFF.4000000@platinum.linux.pl> From: Steven Hartland Message-ID: <572124CC.6050808@multiplay.co.uk> Date: Wed, 27 Apr 2016 21:45:00 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <57211EFF.4000000@platinum.linux.pl> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 20:45:02 -0000 On 27/04/2016 21:20, Adam Nowacki wrote: > On 2016-04-27 20:11, Miroslav Lachman wrote: >> Adam Nowacki wrote on 04/27/2016 19:36: >>> On 2016-04-27 14:05, Miroslav Lachman wrote: >>>> Thank you for output of your zpool scrub. It is definitely faster than >>>> mine. >>>> >>>> To: Paul pk1048 >>>> Mine scrub does not repair anything. Drives are OK (in SMART). >>>> CPU is about 70%-90% idle during scrub + rsync backup and drives are >>>> about 60%-70% busy according to iostat: >>>> >>>> root@kiwi ~/# iostat -x -w 10 ada0 ada1 ada2 ada3 >>>> device r/s w/s kr/s kw/s qlen svc_t %b >>>> ada0 121.8 16.6 602.1 526.9 3 9.2 52 >>>> ada1 122.2 16.5 606.9 528.5 4 9.8 54 >>>> ada2 117.0 14.6 601.7 524.9 2 11.3 60 >>>> ada3 120.6 13.5 610.1 491.3 0 11.4 61 >>>> >>>> I really don't know why it cannot go faster if nothing is loaded for >>>> 100%. >>> 1) zpool scrub is single threaded with prefetch, >> Hmm, this can be the cause. Does it mean that ZFS is faster on CPU with >> higher "per core" power and number of cores (threads) is not so >> important? (in this case of scrub) > No. Zpool scrub thread doesn't need much CPU time as disk I/O handling > (including decompression and checkums) happens in other threads. > >>> 2) some data blocks do not span all disks (metadata, small files, >>> compression) >>> End result is that zfs can't always read from all disks during scrub so >>> disk utilization is going to be less than 100% even when going at full >>> speed. >> Thank you for the explanation. > Try increasing vfs.zfs.top_maxinflight to 100 or more. > If your on a recent version vfs.zfs.vdev.scrub_max_active is usually the most pertinent change to make. Regards Steve From owner-freebsd-fs@freebsd.org Thu Apr 28 05:27:27 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31F42B1F38D for ; Thu, 28 Apr 2016 05:27:27 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from umail.aei.mpg.de (umail.aei.mpg.de [194.94.224.6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D6E9D1FD9; Thu, 28 Apr 2016 05:27:26 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from mailgate.aei.mpg.de (mailgate.aei.mpg.de [194.94.224.5]) by umail.aei.mpg.de (Postfix) with ESMTP id BCFB420028C; Thu, 28 Apr 2016 07:27:23 +0200 (CEST) Received: from mailgate.aei.mpg.de (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id AD963406ADE; Thu, 28 Apr 2016 07:27:23 +0200 (CEST) Received: from intranet.aei.uni-hannover.de (ahin1.aei.uni-hannover.de [130.75.117.40]) by mailgate.aei.mpg.de (Postfix) with ESMTP id 8F4AD406ADB; Thu, 28 Apr 2016 07:27:23 +0200 (CEST) Received: from arc.aei.uni-hannover.de ([130.75.117.1]) by intranet.aei.uni-hannover.de (IBM Domino Release 9.0.1FP5) with ESMTP id 2016042807272308-55741 ; Thu, 28 Apr 2016 07:27:23 +0200 Date: Thu, 28 Apr 2016 07:27:23 +0200 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: Gary Palmer Cc: freebsd-fs@freebsd.org Subject: Re: zfs on nvme: gnop breaks pool, zfs gets stuck Message-Id: <20160428072723.9871ac9ede56ebbbc17ee734@aei.mpg.de> In-Reply-To: <20160427141436.GA60370@in-addr.com> References: <20160427152244.ff36ff74ae64c1f86fdc960a@aei.mpg.de> <20160427141436.GA60370@in-addr.com> Organization: Max Planck Gesellschaft X-Mailer: Sylpheed 3.4.2 (GTK+ 2.24.22; amd64-portbld-freebsd10.0) Mime-Version: 1.0 X-MIMETrack: Itemize by SMTP Server on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 28/04/2016 07:27:23, Serialize by Router on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 28/04/2016 07:27:23, Serialize complete at 28/04/2016 07:27:23 X-TNEFEvaluated: 1 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-PMX-Version: 6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2016.4.28.51518 X-PerlMx-Spam: Gauge=IIIIIIII, Probability=8%, Report=' HTML_00_01 0.05, HTML_00_10 0.05, MIME_LOWER_CASE 0.05, BODYTEXTP_SIZE_3000_LESS 0, BODY_SIZE_1000_LESS 0, BODY_SIZE_2000_LESS 0, BODY_SIZE_5000_LESS 0, BODY_SIZE_600_699 0, BODY_SIZE_7000_LESS 0, NO_URI_HTTPS 0, REFERENCES 0, __ANY_URI 0, __BOUNCE_CHALLENGE_SUBJ 0, __BOUNCE_NDR_SUBJ_EXEMPT 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_FROM 0, __HAS_MSGID 0, __HAS_X_MAILER 0, __IN_REP_TO 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __REFERENCES 0, __SANE_MSGID 0, __SUBJ_ALPHA_END 0, __SUBJ_ALPHA_NEGATE 0, __TO_MALFORMED_2 0, __URI_NO_WWW 0, __URI_NS ' X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2016 05:27:27 -0000 On Wed, 27 Apr 2016 15:14:36 +0100 Gary Palmer wrote about Re: zfs on nvme: gnop breaks pool, zfs gets stuck: GP> Did you destroy the gnop devices with the pool online? In the GP> procedure I remember you export the pool, destroy the gnop devices, GP> and then reimport the pool. Yes, I exported the pool before destroying the gnop devices. That's why I was wodering about what is happening. GP> vfs.zfs.min_auto_ashift GP> GP> which lets you manage the ashift on a new pool without having to try GP> the gnop trick That's a great hint I didn't know about. I'll try this, thanks! cu Gerrit From owner-freebsd-fs@freebsd.org Thu Apr 28 05:31:14 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7912BB1F435 for ; Thu, 28 Apr 2016 05:31:14 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from umail.aei.mpg.de (umail.aei.mpg.de [194.94.224.6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 32E26111B for ; Thu, 28 Apr 2016 05:31:13 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from mailgate.aei.mpg.de (mailgate.aei.mpg.de [194.94.224.5]) by umail.aei.mpg.de (Postfix) with ESMTP id B320620028C; Thu, 28 Apr 2016 07:31:11 +0200 (CEST) Received: from mailgate.aei.mpg.de (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id A3800406ADE; Thu, 28 Apr 2016 07:31:11 +0200 (CEST) Received: from intranet.aei.uni-hannover.de (ahin1.aei.uni-hannover.de [130.75.117.40]) by mailgate.aei.mpg.de (Postfix) with ESMTP id 85634406ADB; Thu, 28 Apr 2016 07:31:11 +0200 (CEST) Received: from arc.aei.uni-hannover.de ([130.75.117.1]) by intranet.aei.uni-hannover.de (IBM Domino Release 9.0.1FP5) with ESMTP id 2016042807311099-55745 ; Thu, 28 Apr 2016 07:31:10 +0200 Date: Thu, 28 Apr 2016 07:31:11 +0200 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: Chris Watson Cc: Steven Hartland , freebsd-fs@freebsd.org Subject: Re: zfs on nvme: gnop breaks pool, zfs gets stuck Message-Id: <20160428073111.83029b9955c60622796c0202@aei.mpg.de> In-Reply-To: References: <20160427152244.ff36ff74ae64c1f86fdc960a@aei.mpg.de> <20160427141436.GA60370@in-addr.com> <5720EFD8.60900@multiplay.co.uk> Organization: Max Planck Gesellschaft X-Mailer: Sylpheed 3.4.2 (GTK+ 2.24.22; amd64-portbld-freebsd10.0) Mime-Version: 1.0 X-MIMETrack: Itemize by SMTP Server on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 28/04/2016 07:31:11, Serialize by Router on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 28/04/2016 07:31:11, Serialize complete at 28/04/2016 07:31:11 X-TNEFEvaluated: 1 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-PMX-Version: 6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2016.4.28.52417 X-PerlMx-Spam: Gauge=IIIIIIIII, Probability=9%, Report=' MULTIPLE_RCPTS 0.1, HTML_00_01 0.05, HTML_00_10 0.05, MIME_LOWER_CASE 0.05, BODYTEXTP_SIZE_3000_LESS 0, BODY_SIZE_1000_LESS 0, BODY_SIZE_2000_LESS 0, BODY_SIZE_5000_LESS 0, BODY_SIZE_600_699 0, BODY_SIZE_7000_LESS 0, NO_URI_HTTPS 0, REFERENCES 0, __ANY_URI 0, __BOUNCE_CHALLENGE_SUBJ 0, __BOUNCE_NDR_SUBJ_EXEMPT 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __FRAUD_BODY_WEBMAIL 0, __FRAUD_WEBMAIL 0, __HAS_FROM 0, __HAS_MSGID 0, __HAS_X_MAILER 0, __IN_REP_TO 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __MULTIPLE_RCPTS_CC_X2 0, __REFERENCES 0, __SANE_MSGID 0, __SUBJ_ALPHA_END 0, __SUBJ_ALPHA_NEGATE 0, __TO_MALFORMED_2 0, __URI_NO_WWW 0, __URI_NS ' X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2016 05:31:14 -0000 On Wed, 27 Apr 2016 13:00:50 -0500 Chris Watson wrote about Re: zfs on nvme: gnop breaks pool, zfs gets stuck: CW> I think for most people, the gnop hack is what is documented on the CW> web. Hence why people are using it versus the ashift sysctl. If the CW> sysctl for ashift is not documented in the ZFS section of the CW> handbook, it probably should be. Yes, please. When you search for this, you find the gnop hack documented everywhere. This is the first time I read about the sysctl. Maybe another good idea would be to mention it in the manpage. cu Gerrit From owner-freebsd-fs@freebsd.org Thu Apr 28 05:48:50 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2B05EB1F6F9 for ; Thu, 28 Apr 2016 05:48:50 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from umail.aei.mpg.de (umail.aei.mpg.de [194.94.224.6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C74B617B3; Thu, 28 Apr 2016 05:48:49 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from mailgate.aei.mpg.de (mailgate.aei.mpg.de [194.94.224.5]) by umail.aei.mpg.de (Postfix) with ESMTP id 8637A20028C; Thu, 28 Apr 2016 07:48:47 +0200 (CEST) Received: from mailgate.aei.mpg.de (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 761BB406ADE; Thu, 28 Apr 2016 07:48:47 +0200 (CEST) Received: from intranet.aei.uni-hannover.de (ahin1.aei.uni-hannover.de [130.75.117.40]) by mailgate.aei.mpg.de (Postfix) with ESMTP id E2765406ADB; Thu, 28 Apr 2016 07:48:46 +0200 (CEST) Received: from arc.aei.uni-hannover.de ([130.75.117.1]) by intranet.aei.uni-hannover.de (IBM Domino Release 9.0.1FP5) with ESMTP id 2016042807484630-55750 ; Thu, 28 Apr 2016 07:48:46 +0200 Date: Thu, 28 Apr 2016 07:48:46 +0200 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: Gary Palmer Cc: freebsd-fs@freebsd.org Subject: Re: zfs on nvme: gnop breaks pool, zfs gets stuck Message-Id: <20160428074846.a28da3113294f1edc6ed9ce6@aei.mpg.de> In-Reply-To: <20160427141436.GA60370@in-addr.com> References: <20160427152244.ff36ff74ae64c1f86fdc960a@aei.mpg.de> <20160427141436.GA60370@in-addr.com> Organization: Max Planck Gesellschaft X-Mailer: Sylpheed 3.4.2 (GTK+ 2.24.22; amd64-portbld-freebsd10.0) Mime-Version: 1.0 X-MIMETrack: Itemize by SMTP Server on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 28/04/2016 07:48:46, Serialize by Router on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 28/04/2016 07:48:46, Serialize complete at 28/04/2016 07:48:46 X-TNEFEvaluated: 1 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-PMX-Version: 6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2016.4.28.53616 X-PerlMx-Spam: Gauge=IIIIIIII, Probability=8%, Report=' HTML_00_01 0.05, HTML_00_10 0.05, MIME_LOWER_CASE 0.05, BODYTEXTP_SIZE_3000_LESS 0, BODY_SIZE_2000_2999 0, BODY_SIZE_5000_LESS 0, BODY_SIZE_7000_LESS 0, NO_URI_HTTPS 0, REFERENCES 0, __ANY_URI 0, __BOUNCE_CHALLENGE_SUBJ 0, __BOUNCE_NDR_SUBJ_EXEMPT 0, __C230066_P5 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __FRAUD_MONEY_BIG_COIN 0, __FRAUD_MONEY_BIG_COIN_DIG 0, __HAS_FROM 0, __HAS_MSGID 0, __HAS_X_MAILER 0, __IN_REP_TO 0, __LINES_OF_YELLING 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __REFERENCES 0, __SANE_MSGID 0, __STOCK_PHRASE_7 0, __SUBJ_ALPHA_END 0, __SUBJ_ALPHA_NEGATE 0, __TO_MALFORMED_2 0, __URI_NO_WWW 0, __URI_NS ' X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2016 05:48:50 -0000 On Wed, 27 Apr 2016 15:14:36 +0100 Gary Palmer wrote about Re: zfs on nvme: gnop breaks pool, zfs gets stuck: GP> vfs.zfs.min_auto_ashift GP> GP> which lets you manage the ashift on a new pool without having to try GP> the gnop trick I just tried this, and it appears to work fine: --- root@storage:~ # sysctl vfs.zfs.min_auto_ashift vfs.zfs.min_auto_ashift: 12 root@storage:~ # zpool status pool: data state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/disk0 ONLINE 0 0 0 gpt/disk1 ONLINE 0 0 0 gpt/disk2 ONLINE 0 0 0 gpt/disk3 ONLINE 0 0 0 gpt/disk4 ONLINE 0 0 0 gpt/disk5 ONLINE 0 0 0 gpt/disk6 ONLINE 0 0 0 gpt/disk7 ONLINE 0 0 0 gpt/disk8 ONLINE 0 0 0 gpt/disk9 ONLINE 0 0 0 gpt/disk10 ONLINE 0 0 0 gpt/disk11 ONLINE 0 0 0 errors: No known data errors pool: flash state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM flash ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gpt/flash0 ONLINE 0 0 0 gpt/flash1 ONLINE 0 0 0 gpt/flash2 ONLINE 0 0 0 errors: No known data errors root@storage:~ # zdb | grep ashift ashift: 12 ashift: 12 root@storage:~ # zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data 65T 1.88M 65.0T - 0% 0% 1.00x ONLINE - flash 1.39T 800K 1.39T - 0% 0% 1.00x ONLINE - --- I still wonder why the gnop workaround went so terribly wrong. Anyway, thanks again for pointing out this new sysctl to me! And for the record: this is what I get with a simple linear write test: --- root@storage:~ # dd if=/dev/zero of=/flash/Z bs=1024k count=10000 10000+0 records in 10000+0 records out 10485760000 bytes transferred in 3.912829 secs (2679840997 bytes/sec) --- I guess I won't complain... cu Gerrit From owner-freebsd-fs@freebsd.org Thu Apr 28 05:52:53 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B8663B1F81F for ; Thu, 28 Apr 2016 05:52:53 +0000 (UTC) (envelope-from karli.sjoberg@slu.se) Received: from EXCH2-1.slu.se (pop.slu.se [77.235.224.121]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "webmail.slu.se", Issuer "TERENA SSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DDDA1A8D; Thu, 28 Apr 2016 05:52:52 +0000 (UTC) (envelope-from karli.sjoberg@slu.se) Received: from exch2-4.slu.se (77.235.224.124) by EXCH2-1.slu.se (77.235.224.121) with Microsoft SMTP Server (TLS) id 15.0.1156.6; Thu, 28 Apr 2016 07:52:49 +0200 Received: from exch2-4.slu.se ([fe80::8041:d235:9900:f9cc]) by exch2-4.slu.se ([fe80::8041:d235:9900:f9cc%22]) with mapi id 15.00.1156.000; Thu, 28 Apr 2016 07:52:49 +0200 From: =?utf-8?B?S2FybGkgU2rDtmJlcmc=?= To: "gerrit.kuehn@aei.mpg.de" , "gpalmer@freebsd.org" CC: "freebsd-fs@freebsd.org" Subject: Re: zfs on nvme: gnop breaks pool, zfs gets stuck Thread-Topic: zfs on nvme: gnop breaks pool, zfs gets stuck Thread-Index: AQHRoIfy5DIMuw2bUEWMyK+GHJC6Xp+du6oAgAEFAQCAAAEgAA== Date: Thu, 28 Apr 2016 05:52:48 +0000 Message-ID: <1461822768.16231.6.camel@slu.se> References: <20160427152244.ff36ff74ae64c1f86fdc960a@aei.mpg.de> <20160427141436.GA60370@in-addr.com> <20160428074846.a28da3113294f1edc6ed9ce6@aei.mpg.de> In-Reply-To: <20160428074846.a28da3113294f1edc6ed9ce6@aei.mpg.de> Reply-To: =?utf-8?B?S2FybGkgU2rDtmJlcmc=?= Accept-Language: sv-SE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [77.235.228.117] Content-Type: text/plain; charset="utf-8" Content-ID: <8B3EBB5A5DF4A64ABA7FC614E5553D68@ad.slu.se> Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2016 05:52:53 -0000 T24gVGh1LCAyMDE2LTA0LTI4IGF0IDA3OjQ4ICswMjAwLCBHZXJyaXQgS8O8aG4gd3JvdGU6DQo+ IE9uIFdlZCwgMjcgQXByIDIwMTYgMTU6MTQ6MzYgKzAxMDAgR2FyeSBQYWxtZXIgPGdwYWxtZXJA ZnJlZWJzZC5vcmc+DQo+IHdyb3RlDQo+IGFib3V0IFJlOiB6ZnMgb24gbnZtZTogZ25vcCBicmVh a3MgcG9vbCwgemZzIGdldHMgc3R1Y2s6DQo+IA0KPiBHUD4gdmZzLnpmcy5taW5fYXV0b19hc2hp ZnQNCj4gR1A+wqANCj4gR1A+IHdoaWNoIGxldHMgeW91IG1hbmFnZSB0aGUgYXNoaWZ0IG9uIGEg bmV3IHBvb2wgd2l0aG91dCBoYXZpbmcgdG8NCj4gdHJ5DQo+IEdQPiB0aGUgZ25vcCB0cmljaw0K PiANCj4gSSBqdXN0IHRyaWVkIHRoaXMsIGFuZCBpdCBhcHBlYXJzIHRvIHdvcmsgZmluZToNCj4g DQo+IC0tLQ0KPiByb290QHN0b3JhZ2U6fiAjIHN5c2N0bCB2ZnMuemZzLm1pbl9hdXRvX2FzaGlm dA0KPiB2ZnMuemZzLm1pbl9hdXRvX2FzaGlmdDogMTINCj4gDQo+IHJvb3RAc3RvcmFnZTp+ICMg enBvb2wgc3RhdHVzDQo+IMKgIHBvb2w6IGRhdGENCj4gwqBzdGF0ZTogT05MSU5FDQo+IMKgIHNj YW46IG5vbmUgcmVxdWVzdGVkDQo+IGNvbmZpZzoNCj4gDQo+IAlOQU1FwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgU1RBVEXCoMKgwqDCoMKgUkVBRCBXUklURSBDS1NVTQ0KPiAJZGF0YcKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoE9OTElORcKgwqDCoMKgwqDCoMKgMMKgwqDCoMKgwqAwwqDCoMKgwqDC oDANCj4gCcKgwqByYWlkejItMMKgwqDCoMKgwqDCoE9OTElORcKgwqDCoMKgwqDCoMKgMMKgwqDC oMKgwqAwwqDCoMKgwqDCoDANCj4gCcKgwqDCoMKgZ3B0L2Rpc2swwqDCoMKgT05MSU5FwqDCoMKg wqDCoMKgwqAwwqDCoMKgwqDCoDDCoMKgwqDCoMKgMA0KPiAJwqDCoMKgwqBncHQvZGlzazHCoMKg wqBPTkxJTkXCoMKgwqDCoMKgwqDCoDDCoMKgwqDCoMKgMMKgwqDCoMKgwqAwDQo+IAnCoMKgwqDC oGdwdC9kaXNrMsKgwqDCoE9OTElORcKgwqDCoMKgwqDCoMKgMMKgwqDCoMKgwqAwwqDCoMKgwqDC oDANCj4gCcKgwqDCoMKgZ3B0L2Rpc2szwqDCoMKgT05MSU5FwqDCoMKgwqDCoMKgwqAwwqDCoMKg wqDCoDDCoMKgwqDCoMKgMA0KPiAJwqDCoMKgwqBncHQvZGlzazTCoMKgwqBPTkxJTkXCoMKgwqDC oMKgwqDCoDDCoMKgwqDCoMKgMMKgwqDCoMKgwqAwDQo+IAnCoMKgwqDCoGdwdC9kaXNrNcKgwqDC oE9OTElORcKgwqDCoMKgwqDCoMKgMMKgwqDCoMKgwqAwwqDCoMKgwqDCoDANCj4gCcKgwqDCoMKg Z3B0L2Rpc2s2wqDCoMKgT05MSU5FwqDCoMKgwqDCoMKgwqAwwqDCoMKgwqDCoDDCoMKgwqDCoMKg MA0KPiAJwqDCoMKgwqBncHQvZGlzazfCoMKgwqBPTkxJTkXCoMKgwqDCoMKgwqDCoDDCoMKgwqDC oMKgMMKgwqDCoMKgwqAwDQo+IAnCoMKgwqDCoGdwdC9kaXNrOMKgwqDCoE9OTElORcKgwqDCoMKg wqDCoMKgMMKgwqDCoMKgwqAwwqDCoMKgwqDCoDANCj4gCcKgwqDCoMKgZ3B0L2Rpc2s5wqDCoMKg T05MSU5FwqDCoMKgwqDCoMKgwqAwwqDCoMKgwqDCoDDCoMKgwqDCoMKgMA0KPiAJwqDCoMKgwqBn cHQvZGlzazEwwqDCoE9OTElORcKgwqDCoMKgwqDCoMKgMMKgwqDCoMKgwqAwwqDCoMKgwqDCoDAN Cj4gCcKgwqDCoMKgZ3B0L2Rpc2sxMcKgwqBPTkxJTkXCoMKgwqDCoMKgwqDCoDDCoMKgwqDCoMKg MMKgwqDCoMKgwqAwDQo+IA0KPiBlcnJvcnM6IE5vIGtub3duIGRhdGEgZXJyb3JzDQo+IA0KPiDC oCBwb29sOiBmbGFzaA0KPiDCoHN0YXRlOiBPTkxJTkUNCj4gwqAgc2Nhbjogbm9uZSByZXF1ZXN0 ZWQNCj4gY29uZmlnOg0KPiANCj4gCU5BTUXCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqBTVEFURcKg wqDCoMKgwqBSRUFEIFdSSVRFIENLU1VNDQo+IAlmbGFzaMKgwqDCoMKgwqDCoMKgwqDCoMKgwqBP TkxJTkXCoMKgwqDCoMKgwqDCoDDCoMKgwqDCoMKgMMKgwqDCoMKgwqAwDQo+IAnCoMKgcmFpZHox LTDCoMKgwqDCoMKgwqBPTkxJTkXCoMKgwqDCoMKgwqDCoDDCoMKgwqDCoMKgMMKgwqDCoMKgwqAw DQo+IAnCoMKgwqDCoGdwdC9mbGFzaDDCoMKgT05MSU5FwqDCoMKgwqDCoMKgwqAwwqDCoMKgwqDC oDDCoMKgwqDCoMKgMA0KPiAJwqDCoMKgwqBncHQvZmxhc2gxwqDCoE9OTElORcKgwqDCoMKgwqDC oMKgMMKgwqDCoMKgwqAwwqDCoMKgwqDCoDANCj4gCcKgwqDCoMKgZ3B0L2ZsYXNoMsKgwqBPTkxJ TkXCoMKgwqDCoMKgwqDCoDDCoMKgwqDCoMKgMMKgwqDCoMKgwqAwDQo+IA0KPiBlcnJvcnM6IE5v IGtub3duIGRhdGEgZXJyb3JzDQo+IA0KPiByb290QHN0b3JhZ2U6fiAjIHpkYiB8IGdyZXAgYXNo aWZ0DQo+IMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoGFzaGlmdDogMTINCj4gwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgYXNoaWZ0OiAxMg0KPiANCj4gcm9vdEBzdG9yYWdlOn4gIyB6cG9vbCBsaXN0 DQo+IE5BTUXCoMKgwqDCoFNJWkXCoMKgQUxMT0PCoMKgwqBGUkVFwqDCoEVYUEFORFNawqDCoMKg RlJBR8KgwqDCoMKgQ0FQwqDCoERFRFVQwqDCoEhFQUxUSMKgwqBBTA0KPiBUUk9PVA0KPiBkYXRh wqDCoMKgwqDCoDY1VMKgwqAxLjg4TcKgwqA2NS4wVMKgwqDCoMKgwqDCoMKgwqDCoC3CoMKgwqDC oMKgMCXCoMKgwqDCoMKgMCXCoMKgMS4wMHjCoMKgT05MSU5FwqDCoC0NCj4gZmxhc2jCoMKgMS4z OVTCoMKgwqA4MDBLwqDCoDEuMzlUwqDCoMKgwqDCoMKgwqDCoMKgLcKgwqDCoMKgwqAwJcKgwqDC oMKgwqAwJcKgwqAxLjAweMKgwqBPTkxJTkXCoMKgLQ0KPiANCj4gLS0tDQo+IA0KPiANCj4gSSBz dGlsbCB3b25kZXIgd2h5IHRoZSBnbm9wIHdvcmthcm91bmQgd2VudCBzbyB0ZXJyaWJseSB3cm9u Zy4gDQoNCkFnYWluLCBiZWNhdXNlIHlvdSBuZWVkIHRvIHRlbGwgemZzIHdoZXJlIHRoZSBwcm92 aWRlcnMgYXJlOg0KDQojIHpwb29sIGltcG9ydCAtZCAvZGV2L2dwdCBmbGFzaA0KDQovSw0KDQo+ IEFueXdheSwNCj4gdGhhbmtzIGFnYWluIGZvciBwb2ludGluZyBvdXQgdGhpcyBuZXcgc3lzY3Rs IHRvIG1lIQ0KPiANCj4gQW5kIGZvciB0aGUgcmVjb3JkOiB0aGlzIGlzIHdoYXQgSSBnZXQgd2l0 aCBhIHNpbXBsZSBsaW5lYXIgd3JpdGUNCj4gdGVzdDoNCj4gDQo+IC0tLQ0KPiByb290QHN0b3Jh Z2U6fiAjIGRkIGlmPS9kZXYvemVybyBvZj0vZmxhc2gvWiBicz0xMDI0ayBjb3VudD0xMDAwMA0K PiAxMDAwMCswIHJlY29yZHMgaW4NCj4gMTAwMDArMCByZWNvcmRzIG91dA0KPiAxMDQ4NTc2MDAw MCBieXRlcyB0cmFuc2ZlcnJlZCBpbiAzLjkxMjgyOSBzZWNzICgyNjc5ODQwOTk3IGJ5dGVzL3Nl YykNCj4gLS0tDQo+IA0KPiANCj4gSSBndWVzcyBJIHdvbid0IGNvbXBsYWluLi4uDQo+IA0KPiAN Cj4gY3UNCj4gwqAgR2Vycml0DQo+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fDQo+IGZyZWVic2QtZnNAZnJlZWJzZC5vcmcgbWFpbGluZyBsaXN0DQo+IGh0 dHBzOi8vbGlzdHMuZnJlZWJzZC5vcmcvbWFpbG1hbi9saXN0aW5mby9mcmVlYnNkLWZzDQo+IFRv IHVuc3Vic2NyaWJlLCBzZW5kIGFueSBtYWlsIHRvICJmcmVlYnNkLWZzLXVuc3Vic2NyaWJlQGZy ZWVic2Qub3JnIg== From owner-freebsd-fs@freebsd.org Thu Apr 28 07:52:49 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BEBF8B1FF05 for ; Thu, 28 Apr 2016 07:52:49 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 803341E52 for ; Thu, 28 Apr 2016 07:52:48 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 60AA82840C; Thu, 28 Apr 2016 09:52:46 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 98FFA28417; Thu, 28 Apr 2016 09:52:45 +0200 (CEST) Message-ID: <5721C14D.8@quip.cz> Date: Thu, 28 Apr 2016 09:52:45 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: Steven Hartland , freebsd-fs@freebsd.org, Adam Nowacki Subject: Re: How to speed up slow zpool scrub? References: <698816653.2698619.1461685653634.JavaMail.yahoo.ref@mail.yahoo.com> <698816653.2698619.1461685653634.JavaMail.yahoo@mail.yahoo.com> <571F9897.2070008@quip.cz> <571FEB34.7040305@andyit.com.au> <56C0A956-F134-4A8D-A8B6-B93DCA045BE4@pk1048.com> <084201d1a03e$d2158fe0$7640afa0$@andyit.com.au> <5720AAF8.4090900@quip.cz> <5720F890.3040600@platinum.linux.pl> <572100C9.8010606@quip.cz> <57211EFF.4000000@platinum.linux.pl> <572124CC.6050808@multiplay.co.uk> In-Reply-To: <572124CC.6050808@multiplay.co.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2016 07:52:49 -0000 Steven Hartland wrote on 04/27/2016 22:45: > > > On 27/04/2016 21:20, Adam Nowacki wrote: [...] >>>> 2) some data blocks do not span all disks (metadata, small files, >>>> compression) >>>> End result is that zfs can't always read from all disks during scrub so >>>> disk utilization is going to be less than 100% even when going at full >>>> speed. >>> Thank you for the explanation. >> Try increasing vfs.zfs.top_maxinflight to 100 or more. >> > If your on a recent version vfs.zfs.vdev.scrub_max_active is usually the > most pertinent change to make. Thank you to both of you. It seems that vfs.zfs.top_maxinflight=128 did higher disk utilization (about 90%). Scrubbing is over after 4 days so I will play with these sysctls next time # zpool status -v tank0 pool: tank0 state: ONLINE scan: scrub repaired 0 in 96h27m with 0 errors on Thu Apr 28 03:29:21 2016 config: NAME STATE READ WRITE CKSUM tank0 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gpt/disk0tank0 ONLINE 0 0 0 gpt/disk1tank0 ONLINE 0 0 0 gpt/disk2tank0 ONLINE 0 0 0 gpt/disk3tank0 ONLINE 0 0 0 errors: No known data errors Thank you! Miroslav Lachman From owner-freebsd-fs@freebsd.org Thu Apr 28 11:58:06 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E7357B1E7D8 for ; Thu, 28 Apr 2016 11:58:06 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from umail.aei.mpg.de (umail.aei.mpg.de [194.94.224.6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A0CE1149A for ; Thu, 28 Apr 2016 11:58:05 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from mailgate.aei.mpg.de (mailgate.aei.mpg.de [194.94.224.5]) by umail.aei.mpg.de (Postfix) with ESMTP id 690F29AE016; Thu, 28 Apr 2016 13:58:02 +0200 (CEST) Received: from mailgate.aei.mpg.de (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 5B51E406ADE; Thu, 28 Apr 2016 13:58:02 +0200 (CEST) Received: from intranet.aei.uni-hannover.de (ahin1.aei.uni-hannover.de [130.75.117.40]) by mailgate.aei.mpg.de (Postfix) with ESMTP id 3D7A3406ADB; Thu, 28 Apr 2016 13:58:02 +0200 (CEST) Received: from arc.aei.uni-hannover.de ([130.75.117.1]) by intranet.aei.uni-hannover.de (IBM Domino Release 9.0.1FP5) with ESMTP id 2016042813580151-57427 ; Thu, 28 Apr 2016 13:58:01 +0200 Date: Thu, 28 Apr 2016 13:58:01 +0200 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: Karli =?ISO-8859-1?Q?Sj=F6berg?= Cc: "freebsd-fs@freebsd.org" Subject: Re: zfs on nvme: gnop breaks pool, zfs gets stuck Message-Id: <20160428135801.7eed38d20cd975a7d67c9283@aei.mpg.de> In-Reply-To: <1461822768.16231.6.camel@slu.se> References: <20160427152244.ff36ff74ae64c1f86fdc960a@aei.mpg.de> <20160427141436.GA60370@in-addr.com> <20160428074846.a28da3113294f1edc6ed9ce6@aei.mpg.de> <1461822768.16231.6.camel@slu.se> Organization: Max Planck Gesellschaft X-Mailer: Sylpheed 3.4.2 (GTK+ 2.24.22; amd64-portbld-freebsd10.0) Mime-Version: 1.0 X-MIMETrack: Itemize by SMTP Server on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 28/04/2016 13:58:01, Serialize by Router on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 28/04/2016 13:58:01, Serialize complete at 28/04/2016 13:58:01 X-TNEFEvaluated: 1 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=ISO-8859-1 X-PMX-Version: 6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2016.4.28.115117 X-PerlMx-Spam: Gauge=IIIIIIII, Probability=8%, Report=' HTML_00_01 0.05, HTML_00_10 0.05, MIME_LOWER_CASE 0.05, BODYTEXTP_SIZE_3000_LESS 0, BODY_SIZE_1000_LESS 0, BODY_SIZE_2000_LESS 0, BODY_SIZE_5000_LESS 0, BODY_SIZE_500_599 0, BODY_SIZE_7000_LESS 0, NO_URI_HTTPS 0, REFERENCES 0, __ANY_URI 0, __BOUNCE_CHALLENGE_SUBJ 0, __BOUNCE_NDR_SUBJ_EXEMPT 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_FROM 0, __HAS_MSGID 0, __HAS_X_MAILER 0, __IN_REP_TO 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __REFERENCES 0, __SANE_MSGID 0, __SUBJ_ALPHA_END 0, __SUBJ_ALPHA_NEGATE 0, __TO_MALFORMED_2 0, __URI_NO_WWW 0, __URI_NS ' X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2016 11:58:07 -0000 On Thu, 28 Apr 2016 05:52:48 +0000 Karli Sj=F6berg wrote about Re: zfs on nvme: gnop breaks pool, zfs gets stuck: > > I still wonder why the gnop workaround went so terribly wrong.=20 KS> Again, because you need to tell zfs where the providers are: KS> # zpool import -d /dev/gpt flash ? Never needed to do that before, and why would it miss only 2 out of 3 device nodes that are all located in gpt? And why would it hang instead of giving an error message? cu Gerrit From owner-freebsd-fs@freebsd.org Thu Apr 28 23:26:30 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 23756B1F216 for ; Thu, 28 Apr 2016 23:26:30 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id B0EB919F6 for ; Thu, 28 Apr 2016 23:26:29 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:dOHbvRDJBkYMLrunToRoUyQJP3N1i/DPJgcQr6AfoPdwSP7+o8bcNUDSrc9gkEXOFd2CrakU26yG7uu6ByQp2tWojjMrSNR0TRgLiMEbzUQLIfWuLgnFFsPsdDEwB89YVVVorDmROElRH9viNRWJ+iXhpQAbFhi3DwdpPOO9QteU1JTnkbDvsMeNKyxzxxODIppKZC2sqgvQssREyaBDEY0WjiXzn31TZu5NznlpL1/A1zz158O34YIxu38I46FppIZ8VvD0Zak1R6dUSTo9ezQ7/sDmvwLPCAWUznUGX2gciRYOBBLKukLURJD05xH7vek1/SCRPsn7SPhgQzGr5KRvRRrAlSAIKjM96GGRgcUm3/ETmw6ouxEqm92cW4qSLvcrJq4= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2B7AgAsmyJX/61jaINehAt9BrliDoF2HoVxAoFeFAEBAQEBAQEBZCeCLYIbIwRkATEZAgRVAgSIPaJJj2KRKQEBCAEBAQEBEwiGIYF9gk6EPIMBglYFmBCDKIJUigKHdoU0jy4CHgFDhAcgMIZpfwEBAQ X-IronPort-AV: E=Sophos;i="5.24,549,1454994000"; d="scan'208";a="280785685" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 28 Apr 2016 19:26:28 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 2A98015F55D for ; Thu, 28 Apr 2016 19:26:28 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 2IFLIx2LAfiJ for ; Thu, 28 Apr 2016 19:26:27 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 67AE915F565 for ; Thu, 28 Apr 2016 19:26:27 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id eZT1mpqwA3zD for ; Thu, 28 Apr 2016 19:26:27 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 410FB15F55D for ; Thu, 28 Apr 2016 19:26:27 -0400 (EDT) Date: Thu, 28 Apr 2016 19:26:27 -0400 (EDT) From: Rick Macklem To: freebsd-fs Message-ID: <738276500.82484810.1461885987214.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <683515500.82484581.1461885974792.JavaMail.zimbra@uoguelph.ca> Subject: review of 2 fuse patches MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_82484807_1596909650.1461885987209" X-Originating-IP: [172.17.95.10] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF18 (Linux)/8.0.9_GA_6191) Thread-Topic: review of 2 fuse patches Thread-Index: admh6ODqSEQShD6CPP5tAOmxheMVPg== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2016 23:26:30 -0000 ------=_Part_82484807_1596909650.1461885987209 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, I've attached two small patches for fuse. (They both are in PR#206238.) Is anyone willing to review these? fuse-wronly.patch - Fixes a problem where the fuse interface has a file opened write-only and a write of a partial buffer cache block is done. At that point, the buffer cache code tries to read the block in and this fails, since the file isn't opened for reading. (The thread basically gets hung in the buffer cache code retrying the read over and over...) I could see two fixes for this: 1 - Always open files read-write when write-only opens are specified. (The problem with this is that a user might have write, but not read access to the file.) 2 - When files are opened write-only, force DirectIO, so that the buffer cache isn't used. *** This patch implements #2. fuse-createio.patch - This simple patch fixes the code so that it obeys the fuse file system's request that an open/create uses DirectIO. (Without this patch, fuse ignores the flag in the file system's reply and does buffered I/O.) This is apparently needed by MooseFS. (This patch assumes that fuse-wronly.patch has already been applied.) Thanks in advance for any review, rick ps: If you want these in phabricator, let me know. However they seem pretty straightforward, except for "how to fix the first problem?". ------=_Part_82484807_1596909650.1461885987209 Content-Type: text/x-patch; name=fuse-createio.patch Content-Disposition: attachment; filename=fuse-createio.patch Content-Transfer-Encoding: base64 LS0tIGZzL2Z1c2UvZnVzZV92bm9wcy5jLnNhdjUJMjAxNi0wMS0xNSAxNzoxNDo0NC4xMDUxOTQw MDAgLTA1MDAKKysrIGZzL2Z1c2UvZnVzZV92bm9wcy5jCTIwMTYtMDEtMTUgMTc6MTQ6NDkuNzIz Nzk5MDAwIC0wNTAwCkBAIC0xMTU4LDcgKzExNTgsOCBAQCBmdXNlX3Zub3Bfb3BlbihzdHJ1Y3Qg dm9wX29wZW5fYXJncyAqYXApCiAJCWZ1ZmhfdHlwZSA9IEZVRkhfUkRPTkxZOwogCX0gZWxzZSB7 CiAJCWZ1ZmhfdHlwZSA9IGZ1c2VfZmlsZWhhbmRsZV94bGF0ZV9mcm9tX2ZmbGFncyhtb2RlKTsK LQkJaWYgKGZ1ZmhfdHlwZSA9PSBGVUZIX1dST05MWSkKKwkJaWYgKGZ1ZmhfdHlwZSA9PSBGVUZI X1dST05MWSB8fAorCQkgICAgKGZ2ZGF0LT5mbGFnICYgRk5fRElSRUNUSU8pICE9IDApCiAJCQlm dXNlX29wZW5fZmxhZ3MgPSBGT1BFTl9ESVJFQ1RfSU87CiAJfQogCg== ------=_Part_82484807_1596909650.1461885987209 Content-Type: text/x-patch; name=fuse-wronly.patch Content-Disposition: attachment; filename=fuse-wronly.patch Content-Transfer-Encoding: base64 LS0tIGZzL2Z1c2UvZnVzZV9maWxlLmMueHh4CTIwMTUtMTItMjggMTA6NDI6MjYuOTE3ODU1MDAw IC0wNTAwCisrKyBmcy9mdXNlL2Z1c2VfZmlsZS5jCTIwMTUtMTItMjggMTE6NDY6NTYuOTA5NDU0 MDAwIC0wNTAwCkBAIC0xNDEsNyArMTQxLDE3IEBAIGZ1c2VfZmlsZWhhbmRsZV9vcGVuKHN0cnVj dCB2bm9kZSAqdnAsCiAJZm9vID0gZmRpLmFuc3c7CiAKIAlmdXNlX2ZpbGVoYW5kbGVfaW5pdCh2 cCwgZnVmaF90eXBlLCBmdWZocCwgZm9vLT5maCk7Ci0JZnVzZV92bm9kZV9vcGVuKHZwLCBmb28t Pm9wZW5fZmxhZ3MsIHRkKTsKKworCS8qCisJICogRm9yIFdST05MWSBvcGVucywgZm9yY2UgRElS RUNUX0lPLiAgVGhpcyBpcyBuZWNlc3NhcnkKKwkgKiBzaW5jZSB3cml0aW5nIGEgcGFydGlhbCBi bG9jayB0aHJvdWdoIHRoZSBidWZmZXIgY2FjaGUKKwkgKiB3aWxsIHJlc3VsdCBpbiBhIHJlYWQg b2YgdGhlIGJsb2NrIGFuZCB0aGF0IHJlYWQgd29uJ3QKKwkgKiBiZSBhbGxvd2VkIGJ5IHRoZSBX Uk9OTFkgb3Blbi4KKwkgKi8KKwlpZiAoZnVmaF90eXBlID09IEZVRkhfV1JPTkxZKQorCQlmdXNl X3Zub2RlX29wZW4odnAsIGZvby0+b3Blbl9mbGFncyB8IEZPUEVOX0RJUkVDVF9JTywgdGQpOwor CWVsc2UKKwkJZnVzZV92bm9kZV9vcGVuKHZwLCBmb28tPm9wZW5fZmxhZ3MsIHRkKTsKIAogb3V0 OgogCWZkaXNwX2Rlc3Ryb3koJmZkaSk7Ci0tLSBmcy9mdXNlL2Z1c2Vfdm5vcHMuYy54eHgJMjAx NS0xMi0yOCAxMDozNjoyMy4yODkyOTcwMDAgLTA1MDAKKysrIGZzL2Z1c2UvZnVzZV92bm9wcy5j CTIwMTUtMTItMjggMTA6NDI6MDYuMjY0NzM3MDAwIC0wNTAwCkBAIC0xMTM5LDYgKzExMzksNyBA QCBmdXNlX3Zub3Bfb3BlbihzdHJ1Y3Qgdm9wX29wZW5fYXJncyAqYXApCiAJc3RydWN0IGZ1c2Vf dm5vZGVfZGF0YSAqZnZkYXQ7CiAKIAlpbnQgZXJyb3IsIGlzZGlyID0gMDsKKwlpbnQzMl90IGZ1 c2Vfb3Blbl9mbGFnczsKIAogCUZTX0RFQlVHMkcoImlub2RlPSVqdSBtb2RlPTB4JXhcbiIsICh1 aW50bWF4X3QpVlRPSSh2cCksIG1vZGUpOwogCkBAIC0xMTUwLDE0ICsxMTUxLDIzIEBAIGZ1c2Vf dm5vcF9vcGVuKHN0cnVjdCB2b3Bfb3Blbl9hcmdzICphcCkKIAlpZiAodm5vZGVfaXNkaXIodnAp KSB7CiAJCWlzZGlyID0gMTsKIAl9CisJZnVzZV9vcGVuX2ZsYWdzID0gMDsKIAlpZiAoaXNkaXIp IHsKIAkJZnVmaF90eXBlID0gRlVGSF9SRE9OTFk7CiAJfSBlbHNlIHsKIAkJZnVmaF90eXBlID0g ZnVzZV9maWxlaGFuZGxlX3hsYXRlX2Zyb21fZmZsYWdzKG1vZGUpOworCQkvKgorCQkgKiBGb3Ig V1JPTkxZIG9wZW5zLCBmb3JjZSBESVJFQ1RfSU8uICBUaGlzIGlzIG5lY2Vzc2FyeQorCQkgKiBz aW5jZSB3cml0aW5nIGEgcGFydGlhbCBibG9jayB0aHJvdWdoIHRoZSBidWZmZXIgY2FjaGUKKwkJ ICogd2lsbCByZXN1bHQgaW4gYSByZWFkIG9mIHRoZSBibG9jayBhbmQgdGhhdCByZWFkIHdvbid0 CisJCSAqIGJlIGFsbG93ZWQgYnkgdGhlIFdST05MWSBvcGVuLgorCQkgKi8KKwkJaWYgKGZ1Zmhf dHlwZSA9PSBGVUZIX1dST05MWSkKKwkJCWZ1c2Vfb3Blbl9mbGFncyA9IEZPUEVOX0RJUkVDVF9J TzsKIAl9CiAKIAlpZiAoZnVzZV9maWxlaGFuZGxlX3ZhbGlkKHZwLCBmdWZoX3R5cGUpKSB7Ci0J CWZ1c2Vfdm5vZGVfb3Blbih2cCwgMCwgdGQpOworCQlmdXNlX3Zub2RlX29wZW4odnAsIGZ1c2Vf b3Blbl9mbGFncywgdGQpOwogCQlyZXR1cm4gMDsKIAl9CiAJZXJyb3IgPSBmdXNlX2ZpbGVoYW5k bGVfb3Blbih2cCwgZnVmaF90eXBlLCBOVUxMLCB0ZCwgY3JlZCk7Cg== ------=_Part_82484807_1596909650.1461885987209-- From owner-freebsd-fs@freebsd.org Fri Apr 29 13:32:58 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8E479B21A4F for ; Fri, 29 Apr 2016 13:32:58 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from smtp.simplesystems.org (smtp.simplesystems.org [65.66.246.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 56542194F for ; Fri, 29 Apr 2016 13:32:57 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by smtp.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id u3TDVMSJ005146; Fri, 29 Apr 2016 08:31:23 -0500 (CDT) Date: Fri, 29 Apr 2016 08:31:22 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: "Michael B. Eichorn" cc: freebsd-fs@freebsd.org Subject: Re: How to speed up slow zpool scrub? In-Reply-To: <1461736217.1121.17.camel@michaeleichorn.com> Message-ID: References: <381846248.2672053.1461695277122.JavaMail.yahoo.ref@mail.yahoo.com> <381846248.2672053.1461695277122.JavaMail.yahoo@mail.yahoo.com> <1461736217.1121.17.camel@michaeleichorn.com> User-Agent: Alpine 2.20 (GSO 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (smtp.simplesystems.org [65.66.246.90]); Fri, 29 Apr 2016 08:31:23 -0500 (CDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2016 13:32:58 -0000 On Wed, 27 Apr 2016, Michael B. Eichorn wrote: > > It does not *need* to be ECC ram, ECC is just *highly recommended*. As > one of the key features of zfs is bitrot prevention, it makes sense to > protect against bitrot everywhere. Zfs (and thus freenas) is just fine > with non-ecc ram. Just, like for any filesystem if the bit is flipped > in ram it will be recorded as such on disk. This is not necessarily the case. Zfs does not offer additional protections for data in RAM. It assumes that data in RAM is protected in other ways. The on-disk checksum only verifies that the data was not modified since it was checksummed, but it may already be corrupt. The risk factor is pretty high if RAM becomes corrupted since zfs uses so much RAM. It is possible to lose data and even the whole pool due to memory corruption. There are well known cases where users encountered continual/periodic pool corruptions due to flaky RAM. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@freebsd.org Fri Apr 29 13:47:50 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0DCB6B21C8B for ; Fri, 29 Apr 2016 13:47:50 +0000 (UTC) (envelope-from karl@denninger.net) Received: from mail.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A1E8F1F41 for ; Fri, 29 Apr 2016 13:47:49 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (Karl-Desktop.Denninger.net [192.168.1.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.denninger.net (Postfix) with ESMTPSA id A780C235817 for ; Fri, 29 Apr 2016 08:47:41 -0500 (CDT) Subject: Re: How to speed up slow zpool scrub? To: freebsd-fs@freebsd.org References: <381846248.2672053.1461695277122.JavaMail.yahoo.ref@mail.yahoo.com> <381846248.2672053.1461695277122.JavaMail.yahoo@mail.yahoo.com> <1461736217.1121.17.camel@michaeleichorn.com> From: Karl Denninger Message-ID: <08d59afe-c835-fa8d-0e52-78afcb1cc030@denninger.net> Date: Fri, 29 Apr 2016 08:47:22 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms070706040406030308010403" X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2016 13:47:50 -0000 This is a cryptographically signed message in MIME format. --------------ms070706040406030308010403 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 4/29/2016 08:31, Bob Friesenhahn wrote: > On Wed, 27 Apr 2016, Michael B. Eichorn wrote: >> >> It does not *need* to be ECC ram, ECC is just *highly recommended*. As= >> one of the key features of zfs is bitrot prevention, it makes sense to= >> protect against bitrot everywhere. Zfs (and thus freenas) is just fine= >> with non-ecc ram. Just, like for any filesystem if the bit is flipped >> in ram it will be recorded as such on disk. > > This is not necessarily the case. Zfs does not offer additional > protections for data in RAM. It assumes that data in RAM is protected > in other ways. The on-disk checksum only verifies that the data was > not modified since it was checksummed, but it may already be corrupt. > The risk factor is pretty high if RAM becomes corrupted since zfs uses > so much RAM. > > It is possible to lose data and even the whole pool due to memory > corruption. > > There are well known cases where users encountered continual/periodic > pool corruptions due to flaky RAM. > > Bob To amplify what Bob said using ZFS on a system without ECC RAM is just begging to lose the entire pool at some point due to a random bit-error in system memory and the fact that it happened may be completely concealed from you for quite a while until at a random later point in time you discover the pool is hopelessly corrupt. ZFS makes the *assumption*, fair or not, that everything in its RAM-based caches is correct. If that assumption is violated you will eventually be a very sad Panda. Use ECC memory or don't use ZFS. --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms070706040406030308010403 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjA0MjkxMzQ3MjJaME8GCSqGSIb3DQEJBDFCBEBk QPpKcj6iM6XPwVxyYGq73+Vn3pqio8qTo2c4pxTHz6wBwZxOkDwbeadmt4V8RogBKJisXdhR XIA2NPc3hwg/MGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIAp7hEk/Z7 +Eo8f39yLrz2L2PINDsJTxgRh6xRxKQeUGPMqU4mh8+Fb5SfH0i2ZEZsS+91QeY88vCre/ZK 9HEdn0QWhskft3ejHnpcQg93VyH0DdM2vlxdOIDnRHsN9m7zqxmPyjOpVyPXl/BzZXfxtxBS YVf8JAyGwwuSRtv6u6MQYpCs/sf5shbqmgqyi3EgRfgdkAzpbVz2RPTa5vKazES0LWU18UnY O5i1jwmvBCTvnu648k74GImv6CtckIcYPt7BXSpvaps1t63uHzgRnvxSeHhfbwVIAdogjPVi z/SG0hKQhHe7uojvaR4im4E+qqmpIp8oGjy/DuinCZYa7+pTDhm7SWHenP6bm5dGBnioIiSd Z8Ndt1+1Y+vxZF7diw9n4+BwJzeMvBDcdq0B7jpcX8sOvjzd8yhB+8JGPEXDPxEbo/B/2495 08k7aDFcnxCst5TdruE+qHiGogCZfsThbngH1ItX/jQJwIEe3rMhiCbO2UsWaYs5zY0J6tZD 0E5hxommg+5YU9LLUPGLKFhOhYQNWh0CVPOGYhu9/AaRoo5cnl/A2No6GCSFtmfkdNlqoG+m gO5FS5wmPBy+oCxZbWiabivtULpFfK7/thOGDBl5E4m9RRvxK5WlB3DMhfucpQQicAOFWVV7 11fP8HYPjjuVNmPoxv7IHQ/jQ+IAAAAAAAA= --------------ms070706040406030308010403-- From owner-freebsd-fs@freebsd.org Fri Apr 29 14:28:27 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CE23BB1FA0D for ; Fri, 29 Apr 2016 14:28:27 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.81]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 99D9C1499 for ; Fri, 29 Apr 2016 14:28:27 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.72) (envelope-from ) id 1aw9P0-0006lI-WB; Fri, 29 Apr 2016 16:28:19 +0200 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: freebsd-fs@freebsd.org, "Karl Denninger" Subject: Re: How to speed up slow zpool scrub? References: <381846248.2672053.1461695277122.JavaMail.yahoo.ref@mail.yahoo.com> <381846248.2672053.1461695277122.JavaMail.yahoo@mail.yahoo.com> <1461736217.1121.17.camel@michaeleichorn.com> <08d59afe-c835-fa8d-0e52-78afcb1cc030@denninger.net> Date: Fri, 29 Apr 2016 16:28:18 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Ronald Klop" Message-ID: In-Reply-To: <08d59afe-c835-fa8d-0e52-78afcb1cc030@denninger.net> User-Agent: Opera Mail/1.0 (Win32) X-Authenticated-As-Hash: 398f5522cb258ce43cb679602f8cfe8b62a256d1 X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: -0.2 X-Spam-Status: No, score=-0.2 required=5.0 tests=ALL_TRUSTED, BAYES_50 autolearn=disabled version=3.4.0 X-Scan-Signature: 66f4fda096222dd2b2010deb1ce817c5 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2016 14:28:27 -0000 On Fri, 29 Apr 2016 15:47:22 +0200, Karl Denninger wrote: > On 4/29/2016 08:31, Bob Friesenhahn wrote: >> On Wed, 27 Apr 2016, Michael B. Eichorn wrote: >>> >>> It does not *need* to be ECC ram, ECC is just *highly recommended*. As >>> one of the key features of zfs is bitrot prevention, it makes sense to >>> protect against bitrot everywhere. Zfs (and thus freenas) is just fine >>> with non-ecc ram. Just, like for any filesystem if the bit is flipped >>> in ram it will be recorded as such on disk. >> >> This is not necessarily the case. Zfs does not offer additional >> protections for data in RAM. It assumes that data in RAM is protected >> in other ways. The on-disk checksum only verifies that the data was >> not modified since it was checksummed, but it may already be corrupt. >> The risk factor is pretty high if RAM becomes corrupted since zfs uses >> so much RAM. >> >> It is possible to lose data and even the whole pool due to memory >> corruption. >> >> There are well known cases where users encountered continual/periodic >> pool corruptions due to flaky RAM. >> >> Bob > > To amplify what Bob said using ZFS on a system without ECC RAM is just > begging to lose the entire pool at some point due to a random bit-error > in system memory and the fact that it happened may be completely > concealed from you for quite a while until at a random later point in > time you discover the pool is hopelessly corrupt. > > ZFS makes the *assumption*, fair or not, that everything in its > RAM-based caches is correct. If that assumption is violated you will > eventually be a very sad Panda. Use ECC memory or don't use ZFS. > Just like UFS makes an assumption about correct memory and correct disks? ECC helps ZFS as much as ECC helps UFS. And without ECC ZFS provides more failsafes than UFS. But nothing is perfect. You guys make it sound like ZFS has no added benefits if you don't use ECC, which is not true. UFS < ZFS < ZFS+ECC And UFS+ECC is somewhere in between probably. As long as people understand the risks/benefits things are ok. Ronald. From owner-freebsd-fs@freebsd.org Fri Apr 29 14:36:50 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C481DB1FD39 for ; Fri, 29 Apr 2016 14:36:50 +0000 (UTC) (envelope-from aberg010@my.HennepinTech.edu) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bon0140.outbound.protection.outlook.com [157.56.111.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 409A719E0 for ; Fri, 29 Apr 2016 14:36:49 +0000 (UTC) (envelope-from aberg010@my.HennepinTech.edu) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=myhennepintech.onmicrosoft.com; s=selector1-my-hennepintech-edu; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=jVEkWrwUSHGaiydZ5MS6FApKhHcXh6DtvYuEXkrbMfo=; b=QnrI0hu8T89afEvpC6/gm/FZDaichY0g9MgGb30Gkz8mqLWNLqcSqTJyvXwxR0evabN5mgjHd+iLZ/enwPwoXwtV+SU6K2dQjGCptRohnp+BYrb/YTTXCa/u1/TSVcVl9+8yqb/pJmmhZZP6QwxMtLPLoB9qo7yryMBCB1aWav0= Authentication-Results: freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=none action=none header.from=my.hennepintech.edu; Received: from [IPv6:2601:440:c000:982::30] (2601:440:c000:982::30) by BN3PR03MB1496.namprd03.prod.outlook.com (10.163.35.147) with Microsoft SMTP Server (TLS) id 15.1.477.8; Fri, 29 Apr 2016 14:03:09 +0000 Subject: Re: How to speed up slow zpool scrub? To: References: <381846248.2672053.1461695277122.JavaMail.yahoo.ref@mail.yahoo.com> <381846248.2672053.1461695277122.JavaMail.yahoo@mail.yahoo.com> <1461736217.1121.17.camel@michaeleichorn.com> <08d59afe-c835-fa8d-0e52-78afcb1cc030@denninger.net> From: Andrew Berg Message-ID: <57236998.5090908@my.hennepintech.edu> Date: Fri, 29 Apr 2016 09:03:04 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <08d59afe-c835-fa8d-0e52-78afcb1cc030@denninger.net> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [2601:440:c000:982::30] X-ClientProxiedBy: CY1PR10CA0021.namprd10.prod.outlook.com (10.162.208.31) To BN3PR03MB1496.namprd03.prod.outlook.com (10.163.35.147) X-MS-Office365-Filtering-Correlation-Id: 50171079-675d-4c6a-5f0b-08d37036ff27 X-Microsoft-Exchange-Diagnostics: 1; BN3PR03MB1496; 2:BiAjNr9t5ATtI8/QujS6n0RCDJpK+zIrWPrqzRxgbzhSi0eqVkwXH3mbqtHrSe93kO7IiVxSCe3ui4y7CRnXFYvtD/9cCbI3M9x6G43h6ZnifkNvX8svAMRkibbYNbULKKQ0sV42EIc7gjJtxLFosU46hkUsNCv+vLCzEZPOwvKrOz0zCtQxiULqBRwukJGZ; 3:XgTEk0QgGyw+cOKvAl3gme/FU3p6dD2AGXhaGqRfkOUHFL9OQZLIHPb2DYbQmbQYMk7SHBaZYp+h4rV3HPa5CEWhDOFlKepyxJbtxzmwEh9XkRZh9GFXaNj8nwYROFg+ X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BN3PR03MB1496; X-Microsoft-Exchange-Diagnostics: 1; BN3PR03MB1496; 25:1iWw2m23Px9JMmMvosCnaTWmmN3R263B5m6gZdWNcOZaNX50z5AtlXGHM+B4cI8B2C5RDjFZo4A8ESadh/XlvWWu4vow0LRL0uXyjcGtHd7o1SwHduWmIpxE7elb3Zv5BNaF3tCfEcKYvqypMHSvG/+2C82PbeM1fN2TQOnRivnxmiRrx+NZwQzGXUePsd/5DQb9Ba7RO2QuAOyFlFWTUkbrGL/nmUaDvmq8+NUwJLcsomHtciyGoz7ulw1eJ76Lyczzr3AtO3IrbYPS/BP6WJ3AK7FHO7yRwkzK19rnpvDlEMc8V71xyt8v+mGgRiVYvkStu776TBrj/dgdLU4qEKOjr3iFeyhc1Ytagz7n+Mkw4tIk5aWJJ5SHxRmsFea5inQWosAd1uAcryJ8FfaKvdbFvLAI9dmvSviqRSD1JA75xsP/W1DuxNKD5TixbVu7J5OInvEVfiETJ99J26eUJORhl8Awgr2WVvB2kFmsWF9coI/GlaK8GkGt+WeTEjWUtSPgyjzIJ2L3hwrOdYqULkhB9RYqydeTbEkKowSP0Rhhl+4RPC4mZVDdVFv3opZ0++3VeCBtIbcP5xC7aKZkVCrdFIYb64rV5990edH8smZRe48OQ3wWBr+i+6tp8NKbBOBTQuIHDSpV7S1YrZCa12jNiWTnhGudcJwtJNa683WnCCStgcgoIm1jonJGV8qC X-Microsoft-Exchange-Diagnostics: 1; BN3PR03MB1496; 20:pYp9YnhuVJWvCuV4YSh8K+pv9C5PDPmjwosxL0LFpKsWRi9Xfxt/kpulL8dE5J36PBjDnyESfIP1g/RbBkejl64S8Tz2nGT5kh+Da5XAHPkWPWqwgVsVLi7T6IpkKDVhxJVo8oShgkfZoUuTIDWATS3nodPcnmfhPW1rjkLCbm35JFeYCMeqShnYVcWXT09f/0W+28whrlUlJV6IuttQcbf/GXEds9JIpyrBUNmpi+WKuUfLkOc2e0Y7xHZ4wR5JgYw6BMgWMUuWHMQ5URFnUISDNkvzT0yInaWj1S6SwMfDgQTZ9GU/zJNPM5sS8zCDeUhyo59A/6L278PX51f7Y/ToqoX/j9FsGmZC/OfOQOvpMJaiQLXC3/2AUkKi2etlkMGiwJVejb79Cv5GDACj7QsQI85d3Auu5zGW/AhgxfkKKpdPvC7Smai4h/OsGd/Rm6Q0cuv2X0DjhbVGCnWbBbpnW+OdMYZOsWDpn+gfN7+O1FIvvm24THZU6JLzNE4e; 4:hBIh4dU9m6lQovmCfMtSl6HwdlfKV2L7UBoRcxq4v5Ps5YDc94CnyRRMDuIizJ55drl6pE0OatGtUUcn+upc3gfY3CoeC5wXoTUHDoGP4Eh0eAFYkhc6EhispbzOo6Bk3kcqM+EenBpZrUs7caZ3i7NqS6LaVZkRjK7Ib9QCDO24EjtCGVAOktjfMgt3BxCjRVP+UffLw92e/5/bayESxHQQfMKNJNEUTg2Ca+VnQjz9FRvt37WYGuxMr+D7J3TB0aFE+OtjDjyLwfzG035GwL/6neR7UyTv7FBp8IittXqcxqPZ0wiKanvjifWrHUY4Pxy8nmG2tPLkJwJ3k2fBxSwxBSVqDMwLhQGpOP0nu946W2A4txOxLePC9hcKR+8fe3joRk68sydgLvIokd4sFw== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(9101521072)(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046); SRVR:BN3PR03MB1496; BCL:0; PCL:0; RULEID:; SRVR:BN3PR03MB1496; X-Forefront-PRVS: 0927AA37C7 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10019020)(4630300001)(6009001)(24454002)(92566002)(77096005)(450100001)(86362001)(189998001)(122286003)(99136001)(93886004)(50986999)(76176999)(230700001)(2351001)(65816999)(83506001)(54356999)(4001350100001)(89122001)(65956001)(65806001)(107886002)(110136002)(47776003)(2950100001)(42186005)(50466002)(87266999)(1096002)(117636001)(81166005)(6116002)(586003)(75432002)(59896002)(23676002)(5008740100001)(5004730100002)(88552002)(64126003)(2906002)(89472002)(3826002); DIR:OUT; SFP:1102; SCL:1; SRVR:BN3PR03MB1496; H:[IPv6:2601:440:c000:982::30]; FPR:; SPF:None; MLV:sfv; LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtCTjNQUjAzTUIxNDk2OzIzOis2N3JQc3IxRmV1b3pkSFZkM2hVWlcyemwx?= =?utf-8?B?SU11V0U4eExVUFR0c2M4VC9QL0Y4WjJWQnVNUENTQ1E3UEIzVm95ZElMOXNW?= =?utf-8?B?N2ZqUFlFWDIwaU1Bd253UFpUL0NhcEcxUUU5MUFjdG1WSmI4bkEycFA0TlpG?= =?utf-8?B?eHVCVnRWSFArSy9lUFYwazlwaGpSbEtrTTlqejdpQzV5dExBMTFRNU9vSHhC?= =?utf-8?B?K0lSSjhCR05mY1U1MVNQbngwcnkwSTlMWGR6SFdudTVHb285K1hyRC9oMTZj?= =?utf-8?B?YlFibk5ReDNWVDZaS3ZTYlRVYkpTUkVZd3VUR3FId3dHSWh3K3RibHFoWGpW?= =?utf-8?B?ZVRYUlBkNlZZb0tpb2UycEJWS3FLcnNCdkdwTjhmNmFuS2NQOTc3UGU0S0k3?= =?utf-8?B?RE56ZlhyM09EbDZ6UkpKeHRjMEFaUTN4K3dyb3ZaTDZWelZCZ2Zjc0xYa2dK?= =?utf-8?B?SnhEM2p2bEV5MlEwaklJVXVVSW1CNnlSeXpXRHM5MldSMTQ5ZTUwZ2hZNnor?= =?utf-8?B?L0pERjJ2OXlBM1hZOUNETUR0ZnpnY2M4bHV0K3BmVjR6RExiV0EzZGlUbWgw?= =?utf-8?B?UENkTElFdVE2WHk4N3prQ1Riblo2dmlIV0NqUUFpOU84SVcxZDNaOFdiWG5T?= =?utf-8?B?Smc5ZHE0T3RzczhnaUs1bVdFdGozTk1BcFBDZ0l1VlRIbVErZGdCZDJxK0Ix?= =?utf-8?B?WDkyYS9kc2lpLzlyR2hFb1RGVDczenJrZmRRTG9SdFY1Tm9YVUJGUjl4djlD?= =?utf-8?B?QTVOV2ZSYlhNeURrek5QU2pNb3hMT3ZtZUMrMWNJSWxLQllmekQ5amc2VE5U?= =?utf-8?B?djE5V2N2QzhJN2IxSGEwdGhpTUVrVjl1YlZWTzJKT2lQS2FpUnZ6czd3Qjhn?= =?utf-8?B?dVBmSFdGejJHcW1kamdyYU5udkwvdlV4emptK0t1aTFkSnBNQzNtM3lKMWtM?= =?utf-8?B?c0FGbjVPUEVqYlFUY3ZXcVo2NVZNYkJ5dTBCKzU1QWFXVC91OE1yaU1EMjV0?= =?utf-8?B?SDNFNVZVVFRoakhSRmZoZTFjV0xqUXVhWjlFTU5LWGoyZFJMY3Z5OHA0c3RS?= =?utf-8?B?eFVNUDNSS1Rad3FNRE8zRUtrdWFFTlVTTVRwV3J1cmFRZXI3K2l6MDZWYXVD?= =?utf-8?B?bWk0T1BPWDcvUnVQNWFxRTdpZC8rK3ZJcVdDNmEyTHlJdjh3eHBrT0pjRFAy?= =?utf-8?B?SnNqT3pMa0pZWDZHR2pma243NHQrNU5IVVlYSHNqZHdBZ3ZyTS9sblo5Si9y?= =?utf-8?B?S1NXcGNBWVE5cW5nT09Qa1AxdjVaYXhsNnZxa3A2Wk1jOGZaQWQvT28vRDlw?= =?utf-8?B?ek1zUGZ2Q0doVlEzcjRIakhpNTBzOUlodEhOVnc3MzZwRDRydmtMZFBUN0Nz?= =?utf-8?B?eG53di9TR283eVFLcFJ2dHdzN0x0YmhId0gyMjRTdjhwYkdJZHhDbnFRaSsv?= =?utf-8?B?WEdNZjl5Y2F3TzRvSmVHc0F3cDRmbFdTNG1vc2NPT0dpSHl3cHdEbGVybnd0?= =?utf-8?Q?FudP1UP2qWlK90FQa3s2PMezhN8gfUIbIbCJMUpVeIjmh/?= X-Microsoft-Exchange-Diagnostics: 1; BN3PR03MB1496; 5:xtU10cOTdd+6Loo7cARBDUnNLojpQr1LV3az/fjKFVRQAT0zqsovefGua0cfLfRPXjS1mv7OqKNBDVIFxyWZxlfYbgQp9zuT6w/HJ4AckJxzDcKEsa/Q6dAD+HXTARUWWhioiPZXIq9qyEmrm/0gYA==; 24:YPn1TxsfeM07OQOCT05hNZdXBJ1SGW4thoxtcmd1ATsGPlX1IEDjJ83qrcy8PNVtPenC4shTpX4knaJX7iuB/gSOpXygNESIWfm3GF8bYDk=; 7:bSEXyARV53aPluFqHk7zxrXYThxt4buUo+aQjp4G9J5T246UoWigrIwN7nztUqJHTZyhi8b6Xp1RI9kRDyCCpF3u14+cU3Sg/7BhJnOpfOGKp8zz2h2sKO6gwrFLJVuvME4ACydWkdMEVqvXFU8MUxs8tZtLw4EaSMLo+E1lyFElEpBDPXmTFnaLMBwV2/yW SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: my.hennepintech.edu X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Apr 2016 14:03:09.7874 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR03MB1496 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2016 14:36:50 -0000 On 2016.04.29 08:47, Karl Denninger wrote: > On 4/29/2016 08:31, Bob Friesenhahn wrote: >> On Wed, 27 Apr 2016, Michael B. Eichorn wrote: >>> >>> It does not *need* to be ECC ram, ECC is just *highly recommended*. As >>> one of the key features of zfs is bitrot prevention, it makes sense to >>> protect against bitrot everywhere. Zfs (and thus freenas) is just fine >>> with non-ecc ram. Just, like for any filesystem if the bit is flipped >>> in ram it will be recorded as such on disk. >> >> This is not necessarily the case. Zfs does not offer additional >> protections for data in RAM. It assumes that data in RAM is protected >> in other ways. The on-disk checksum only verifies that the data was >> not modified since it was checksummed, but it may already be corrupt. >> The risk factor is pretty high if RAM becomes corrupted since zfs uses >> so much RAM. >> >> It is possible to lose data and even the whole pool due to memory >> corruption. >> >> There are well known cases where users encountered continual/periodic >> pool corruptions due to flaky RAM. >> >> Bob > > To amplify what Bob said using ZFS on a system without ECC RAM is just > begging to lose the entire pool at some point due to a random bit-error > in system memory and the fact that it happened may be completely > concealed from you for quite a while until at a random later point in > time you discover the pool is hopelessly corrupt. > > ZFS makes the *assumption*, fair or not, that everything in its > RAM-based caches is correct. If that assumption is violated you will > eventually be a very sad Panda. Use ECC memory or don't use ZFS. > ZFS assumes a lot less than other filesystems. ZFS will complain if things aren't right. It is *less* likely to fall apart than another filesystem under this condition since it keeps redundant checksummed copies of metadata. There is zero reason to think than any other filesystem will protect you from bad RAM. If a bit flip messes up the metadata, you are going to have a bad day no matter what. At least with ZFS, you will know about it right away. Your argument really boils down to "Use ECC memory or don't use computers". To put it another way: if you are experiencing corruption from bad RAM, what filesystem would you go with to protect your data? On a side note, ZFS uses lots of RAM for *caching*. Any corruption there might affect your applications and/or system, but not your on-disk data. It's also highly unlikely that you will have bad data written, but then never have any bit flips on reads later. You *will* know something is wrong. I have personally had bad RAM, and ZFS was the only reason I was able to figure out why I had issues. If your bit flips aren't in kernel memory, then you won't have crashes and other obvious problems, but you will have weird data in your applications, and then you'll see some checksum errors in a 'zpool status'. From owner-freebsd-fs@freebsd.org Fri Apr 29 14:38:41 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B877B1FDAB for ; Fri, 29 Apr 2016 14:38:41 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 635751A74 for ; Fri, 29 Apr 2016 14:38:39 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id RAA10562; Fri, 29 Apr 2016 17:38:29 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1aw9Yr-000IUv-6U; Fri, 29 Apr 2016 17:38:29 +0300 Subject: Re: How to speed up slow zpool scrub? To: Ronald Klop , freebsd-fs@FreeBSD.org References: <381846248.2672053.1461695277122.JavaMail.yahoo.ref@mail.yahoo.com> <381846248.2672053.1461695277122.JavaMail.yahoo@mail.yahoo.com> <1461736217.1121.17.camel@michaeleichorn.com> <08d59afe-c835-fa8d-0e52-78afcb1cc030@denninger.net> From: Andriy Gapon Message-ID: <169a2eb9-621f-4fc8-982d-9550ee9581cb@FreeBSD.org> Date: Fri, 29 Apr 2016 17:37:32 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2016 14:38:41 -0000 On 29/04/2016 17:28, Ronald Klop wrote: > Just like UFS makes an assumption about correct memory and correct disks? > > ECC helps ZFS as much as ECC helps UFS. > And without ECC ZFS provides more failsafes than UFS. But nothing is perfect. > You guys make it sound like ZFS has no added benefits if you don't use ECC, > which is not true. > > UFS < ZFS < ZFS+ECC > And UFS+ECC is somewhere in between probably. > As long as people understand the risks/benefits things are ok. One thing to keep in mind is that ZFS on-disk structures are much more complex than those of UFS. It is much harder to recover from some ZFS failure modes. (And there is no zfsck...) -- Andriy Gapon From owner-freebsd-fs@freebsd.org Fri Apr 29 15:03:48 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65680B20647 for ; Fri, 29 Apr 2016 15:03:48 +0000 (UTC) (envelope-from eborisch@gmail.com) Received: from mail-oi0-x233.google.com (mail-oi0-x233.google.com [IPv6:2607:f8b0:4003:c06::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2B7321A17 for ; Fri, 29 Apr 2016 15:03:48 +0000 (UTC) (envelope-from eborisch@gmail.com) Received: by mail-oi0-x233.google.com with SMTP id k142so121418091oib.1 for ; Fri, 29 Apr 2016 08:03:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=tLhF5UbD5AOG7Sj7Ve7AkmtyYirMvvTBciH2WpLuw3c=; b=vrNqvolcUrrBgbdpMHCLdFbosvA0hmkk0lWAWA1JKAOQcIdRz/qPBuofm+RjcnfD+r umHs7z+ID20QuPV/y5RHZ5wcwey8n855XXq5GLfViL0EEdlCMU3lU2ohcNtogr8FOb6k rxwDUjLGiAFTYC/FPibBBfcsBSpbGZZ9pfguy7m3QmDJU6JWiRihXeIESEByd0hymj0k eUxmZiikVqkjN8zmQHxXV0Y6Iq8aR9YApwje+/vDQljh3+2pnrUELNYe/uQdVrns4PPj oz4Wi+O2k9xY0F0UB1C2XiEgUfAto6qmApoe4HaCIaKYuUWocf9RSVbBPj/v89JELWH6 vkOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=tLhF5UbD5AOG7Sj7Ve7AkmtyYirMvvTBciH2WpLuw3c=; b=fWOhJ6BHbuDH5xoCAdqhVg+SWnqEmX5YxGywrJTZxuZMceBaXJjjD9q+RatFR1qrcc VmLdvW3BndZcOl08ii788GSt3kbYwsf4m+n1su6Wi2Qz3OIoYn6lt5g9CsX8unBuQKAL LYzXM1b2+Wgu7dEz2iO4JgDteVzTwuXBt9GHjJwiPkXj8eW4IkEJFCj+GJbDtRdFT3NN 89a2ehBtWDrnC4XOdgFEp+PdzL2fNoLfrvtRFLEUX4di61agbTSB9MTTJfw9WXQ2/9r6 4mKfV8mXACYWSzOlpkIOqVfWQZvKScJlnNkuGuFnp62kRkidrVUbKdtmEOd0OV2iSMf5 keQg== X-Gm-Message-State: AOPr4FWfh/0/G9Qr19vOmEsKW3UfDrDQFgwZ0hYadTru+e4Cyuqo7Vch1wbK1JlMOzQ/g47btgWYMrG0zTzOjw== MIME-Version: 1.0 X-Received: by 10.202.189.10 with SMTP id n10mr8897147oif.101.1461942227278; Fri, 29 Apr 2016 08:03:47 -0700 (PDT) Received: by 10.60.52.145 with HTTP; Fri, 29 Apr 2016 08:03:47 -0700 (PDT) In-Reply-To: References: <381846248.2672053.1461695277122.JavaMail.yahoo.ref@mail.yahoo.com> <381846248.2672053.1461695277122.JavaMail.yahoo@mail.yahoo.com> <1461736217.1121.17.camel@michaeleichorn.com> <08d59afe-c835-fa8d-0e52-78afcb1cc030@denninger.net> Date: Fri, 29 Apr 2016 10:03:47 -0500 Message-ID: Subject: Re: How to speed up slow zpool scrub? From: "Eric A. Borisch" To: Ronald Klop Cc: "freebsd-fs@freebsd.org" , Karl Denninger Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2016 15:03:48 -0000 On Fri, Apr 29, 2016 at 9:28 AM, Ronald Klop wrote: > On Fri, 29 Apr 2016 15:47:22 +0200, Karl Denninger >> ZFS makes the *assumption*, fair or not, that everything in its >> RAM-based caches is correct. If that assumption is violated you will >> eventually be a very sad Panda. Use ECC memory or don't use ZFS. >> > > Just like UFS makes an assumption about correct memory and correct disks? > > ECC helps ZFS as much as ECC helps UFS. > And without ECC ZFS provides more failsafes than UFS. But nothing is > perfect. > You guys make it sound like ZFS has no added benefits if you don't use ECC, > which is not true. > > UFS < ZFS < ZFS+ECC > And UFS+ECC is somewhere in between probably. > As long as people understand the risks/benefits things are ok. I a key distinction is that UFS has fsck for attempting to repair inconsistencies on the drive, where ZFS does not have a similar tool, because "[t]he only way for inconsistent data to exist on disk in a ZFS configuration is through hardware failure [...] or when a bug exists in the ZFS software." [1] So if ZFS fails, it is more likely to fail hard; enter the ECC (avoid hardware failure) "requirement". I personally have one system running without ECC, but it is a tiny system at home that serves as the firewall the cable modem runs into. It is backed up and stores nothing of real value on the media, but I love having ZFS on it because I can do things like beadm for upgrades, or diffs of /etc files with previous (automated) snapshots. (It's also running with less than 4G of RAM, tsk-tsk...) If you are storing data you care about* on a ZFS system without ECC, you are doing it wrong. If that system *is* the backup, you are in a gray area depending on your risk profile. (Is it OK if the backups fail, because I still have the source and am willing to risk having only one copy while I rebuild the backup?) So I'd temper Karl's statement to "Use ECC memory -- or really understand the risks -- or don't use ZFS." - Eric [1] http://docs.oracle.com/cd/E19253-01/819-5461/6n7ht6r6p/index.html * can't afford to lose / can't recreate From owner-freebsd-fs@freebsd.org Fri Apr 29 15:05:13 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E7655B206C5 for ; Fri, 29 Apr 2016 15:05:13 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from smtp.simplesystems.org (smtp.simplesystems.org [65.66.246.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B75CA1ABF for ; Fri, 29 Apr 2016 15:05:13 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by smtp.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id u3TF5BaL010587; Fri, 29 Apr 2016 10:05:11 -0500 (CDT) Date: Fri, 29 Apr 2016 10:05:11 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Andrew Berg cc: freebsd-fs@freebsd.org Subject: Re: How to speed up slow zpool scrub? In-Reply-To: <57236998.5090908@my.hennepintech.edu> Message-ID: References: <381846248.2672053.1461695277122.JavaMail.yahoo.ref@mail.yahoo.com> <381846248.2672053.1461695277122.JavaMail.yahoo@mail.yahoo.com> <1461736217.1121.17.camel@michaeleichorn.com> <08d59afe-c835-fa8d-0e52-78afcb1cc030@denninger.net> <57236998.5090908@my.hennepintech.edu> User-Agent: Alpine 2.20 (GSO 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (smtp.simplesystems.org [65.66.246.90]); Fri, 29 Apr 2016 10:05:11 -0500 (CDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2016 15:05:14 -0000 On Fri, 29 Apr 2016, Andrew Berg wrote: > > On a side note, ZFS uses lots of RAM for *caching*. Any corruption there > might affect your applications and/or system, but not your on-disk data. It's > also highly unlikely that you will have bad data written, but then never have > any bit flips on reads later. You *will* know something is wrong. ZFS uses RAM for caching the pool and filesystem structures, and metadata. If this data becomes correct, then innocent data may be overwritten with corrupted data because zfs made the wrong decisions. It is true that you are likely to soon know that something is wrong, but by then it may be too late. For proper operation of ZFS one needs: o Reliable RAM o Reliable disk cache sync request Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@freebsd.org Fri Apr 29 20:20:41 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0208FB2141D for ; Fri, 29 Apr 2016 20:20:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E69631D4C for ; Fri, 29 Apr 2016 20:20:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u3TKKeUg036674 for ; Fri, 29 Apr 2016 20:20:40 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204643] [msdosfs] [panic] Crash while accessing files with large, non-english names Date: Fri, 29 Apr 2016 20:20:41 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.3-RELEASE X-Bugzilla-Keywords: patch, security X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2016 20:20:41 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D204643 --- Comment #7 from commit-hook@freebsd.org --- A commit references this bug: Author: kp Date: Fri Apr 29 20:19:41 UTC 2016 New revision: 298799 URL: https://svnweb.freebsd.org/changeset/base/298799 Log: MFC r298664 msdosfs: Prevent buffer overflow when expanding win95 names In win2unixfn() we expand Windows 95 style long names. In some cases that requires moving the data in the nbp->nb_buf buffer backwards to make room. That code failed to check for overflows, leading to a stack overflow in win2unixfn(). We now check for this event, and mark the entire conversion as failed in = that case. This means we present the 8 character, dos style, name instead. PR: 204643 Differential Revision: https://reviews.freebsd.org/D6015 Changes: _U stable/10/ stable/10/sys/fs/msdosfs/direntry.h stable/10/sys/fs/msdosfs/msdosfs_conv.c --=20 You are receiving this mail because: You are the assignee for the bug.=