From owner-freebsd-fs@FreeBSD.ORG Sun Apr 6 05:21:22 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E9603390 for ; Sun, 6 Apr 2014 05:21:22 +0000 (UTC) Received: from fs.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "NewFS.denninger.net", Issuer "NewFS.denninger.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id AFE7D784 for ; Sun, 6 Apr 2014 05:21:22 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by fs.denninger.net (8.14.8/8.14.8) with ESMTP id s3656xoB080037 for ; Sun, 6 Apr 2014 00:06:59 -0500 (CDT) (envelope-from karl@denninger.net) Received: from [127.0.0.1] (TLS/SSL) [192.168.1.40] by Spamblock-sys (LOCAL/AUTH); Sun Apr 6 00:06:59 2014 Message-ID: <5340E0EE.8010905@denninger.net> Date: Sun, 06 Apr 2014 00:06:54 -0500 From: Karl Denninger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: Device Removed by Administrator in ZPOOL? References: <53408FAB.8080202@gmail.com> <512A7865-CEFD-4BDA-A060-AE911BEDD5B7@tuxsystems.co.za> <53409BF1.6050001@gmail.com> <20140406002849.GA14765@neutralgood.org> <5340B1C5.4000700@gmail.com> In-Reply-To: <5340B1C5.4000700@gmail.com> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms060107070107090608010508" X-Antivirus: avast! (VPS 140405-4, 04/05/2014), Outbound message X-Antivirus-Status: Clean X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Apr 2014 05:21:23 -0000 This is a cryptographically signed message in MIME format. --------------ms060107070107090608010508 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable On 4/5/2014 8:45 PM, Kaya Saman wrote: > On 04/06/2014 01:28 AM, kpneal@pobox.com wrote: >> On Sun, Apr 06, 2014 at 01:12:33AM +0100, Kaya Saman wrote: >>> Many thanks for the response! >>> >>> The server doesn't show any lights for "drive error" however, the blu= e >>> read LED isn't coming on, on the drive in question (as removed from=20 >>> ZPOOL). >>> >>> I will have a look for LSI tools in @Ports and also see if the BIOS L= SI >>> hook comes up with anything. >> Have you seen any other errors in your logs? Seems like if a drive fai= ls >> there should be some other error message reporting the errors that=20 >> resulted >> in ZFS marking the drive removed. What does 'dmesg' have to say? >> >> Once ZFS has stopped using the drive (for whatever reason) I wouldn't >> expect you to see anything else happening on the drive. So the light n= ot >> coming on doesn't really tell us anything new. >> >> Also, aren't 'green' drives the kind that spin down and then have to=20 >> spin >> back up when a request comes in? I don't know what happens if a drive = >> takes >> "too long" to respond because it has spun down. I have no idea how=20 >> FreeBSD >> handles that, and I also don't know if ZFS adds anything to the=20 >> equation. >> Hopefully someone else here will clue me/us in. > > Ok this is really weird.... just did a reboot and now: > > $ zpool status > pool: ZPOOL_2 > state: ONLINE > status: One or more devices is currently being resilvered. The pool wi= ll > continue to function, possibly in a degraded state. > action: Wait for the resilver to complete. > scan: resilver in progress since Sun Apr 6 02:43:03 2014 > 1.13G scanned out of 7.77T at 22.2M/s, 101h57m to go > 227M resilvered, 0.01% done > config: > > NAME STATE READ WRITE CKSUM > ZPOOL_2 ONLINE 0 0 0 > raidz2-0 ONLINE 0 0 0 > da0 ONLINE 0 0 0 > da1 ONLINE 0 0 0 (resilvering) > da2 ONLINE 0 0 0 > da3 ONLINE 0 0 0 > da4 ONLINE 0 0 0 > > > ???? Looks like the drive might have fallen off the controller? > > Am just looking at the tools for it on the LSI website but there=20 > doesn't seem to be anything FreeBSD related.... Linux and Solaris yes=20 > but no FBSD? > > Model is LSI SAS 9207-4i4e > It looks like the drive detached itself. I've seen those "Green" drives = do this before; they go to "sleep" if quiescent, and sometimes fail to=20 wake up properly. The controller then detaches them thinking they're=20 dead, but they're not... I'd get those things off your system. They work ok for desktop PCs but=20 I don't like them in servers. --=20 -- Karl karl@denninger.net --------------ms060107070107090608010508 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFTzCC BUswggQzoAMCAQICAQgwDQYJKoZIhvcNAQEFBQAwgZ0xCzAJBgNVBAYTAlVTMRAwDgYDVQQI EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkqhkiG9w0BCQEWIGN1c3Rv bWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0MB4XDTEzMDgyNDE5MDM0NFoXDTE4MDgyMzE5 MDM0NFowWzELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExFzAVBgNVBAMTDkthcmwg RGVubmluZ2VyMSEwHwYJKoZIhvcNAQkBFhJrYXJsQGRlbm5pbmdlci5uZXQwggIiMA0GCSqG SIb3DQEBAQUAA4ICDwAwggIKAoICAQC5n2KBrBmG22nVntVdvgKCB9UcnapNThrW1L+dq6th d9l4mj+qYMUpJ+8I0rTbY1dn21IXQBoBQmy8t1doKwmTdQ59F0FwZEPt/fGbRgBKVt3Quf6W 6n7kRk9MG6gdD7V9vPpFV41e+5MWYtqGWY3ScDP8SyYLjL/Xgr+5KFKkDfuubK8DeNqdLniV jHo/vqmIgO+6NgzPGPgmbutzFQXlxUqjiNAAKzF2+Tkddi+WKABrcc/EqnBb0X8GdqcIamO5 SyVmuM+7Zdns7D9pcV16zMMQ8LfNFQCDvbCuuQKMDg2F22x5ekYXpwjqTyfjcHBkWC8vFNoY 5aFMdyiN/Kkz0/kduP2ekYOgkRqcShfLEcG9SQ4LQZgqjMpTjSOGzBr3tOvVn5LkSJSHW2Z8 Q0dxSkvFG2/lsOWFbwQeeZSaBi5vRZCYCOf5tRd1+E93FyQfpt4vsrXshIAk7IK7f0qXvxP4 GDli5PKIEubD2Bn+gp3vB/DkfKySh5NBHVB+OPCoXRUWBkQxme65wBO02OZZt0k8Iq0i4Rci WV6z+lQHqDKtaVGgMsHn6PoeYhjf5Al5SP+U3imTjF2aCca1iDB5JOccX04MNljvifXgcbJN nkMgrzmm1ZgJ1PLur/ADWPlnz45quOhHg1TfUCLfI/DzgG7Z6u+oy4siQuFr9QT0MQIDAQAB o4HWMIHTMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgWgMAsGA1UdDwQEAwIF4DAsBglg hkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFHw4 +LnuALyLA5Cgy7T5ZAX1WzKPMB8GA1UdIwQYMBaAFF3U3hpBZq40HB5VM7B44/gmXiI0MDgG CWCGSAGG+EIBAwQrFilodHRwczovL2N1ZGFzeXN0ZW1zLm5ldDoxMTQ0My9yZXZva2VkLmNy bDANBgkqhkiG9w0BAQUFAAOCAQEAZ0L4tQbBd0hd4wuw/YVqEBDDXJ54q2AoqQAmsOlnoxLO 31ehM/LvrTIP4yK2u1VmXtUumQ4Ao15JFM+xmwqtEGsh70RRrfVBAGd7KOZ3GB39FP2TgN/c L5fJKVxOqvEnW6cL9QtvUlcM3hXg8kDv60OB+LIcSE/P3/s+0tEpWPjxm3LHVE7JmPbZIcJ1 YMoZvHh0NSjY5D0HZlwtbDO7pDz9sZf1QEOgjH828fhtborkaHaUI46pmrMjiBnY6ujXMcWD pxtikki0zY22nrxfTs5xDWGxyrc/cmucjxClJF6+OYVUSaZhiiHfa9Pr+41okLgsRB0AmNwE f6ItY3TI8DGCBQowggUGAgEBMIGjMIGdMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxvcmlk YTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRwwGgYD VQQDExNDdWRhIFN5c3RlbXMgTExDIENBMS8wLQYJKoZIhvcNAQkBFiBjdXN0b21lci1zZXJ2 aWNlQGN1ZGFzeXN0ZW1zLm5ldAIBCDAJBgUrDgMCGgUAoIICOzAYBgkqhkiG9w0BCQMxCwYJ KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDA0MDYwNTA2NTRaMCMGCSqGSIb3DQEJBDEW BBTPTqLFYunS7jZu+ZEJV8xwB13sYTBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjAL BglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFA MAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIG0BgkrBgEEAYI3EAQxgaYwgaMwgZ0xCzAJBgNV BAYTAlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoT EEN1ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkq hkiG9w0BCQEWIGN1c3RvbWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0AgEIMIG2BgsqhkiG 9w0BCRACCzGBpqCBozCBnTELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNV BAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3Vk YSBTeXN0ZW1zIExMQyBDQTEvMC0GCSqGSIb3DQEJARYgY3VzdG9tZXItc2VydmljZUBjdWRh c3lzdGVtcy5uZXQCAQgwDQYJKoZIhvcNAQEBBQAEggIAtS1T63NeFF1nwDELi7aEHOI5g7rv vmiFTY1IIax2/klcKmlin0/VcDsgfasSwt01puAKVv3rs0bIt+BanwjUjtaGW/Be34YDvXL4 TmGh0YpCPmiE8yCxRqNL3IGkQdpJnvbqushkdaVcrSFWZix/BQyZ+NPjQl6S5cPyFgkywcOf ktIn67rZ+82O4LPS/X8doFpp03Djn48bzbuk/Tdg2jotI3UqxcyCDpFgKMPEAFdGS4KtqF/f BlH8vMaf8YqoZFJ9lAzmAfgqcguPJOF7ktQZNjcEAYSvovRU6tWGRigpscN0PhAgHbTd7gG/ Z+iOBXTlFiiUB0hVKjvZKGTt+7gdkIA7S+Si0DT9njFxz2GvGMsGpGUNgWF6oxdbs3qHGP79 Hcz1Q8nWwKPNAR39Hoqm2/bK0NIbQANMJOZtbl7yIIAd2sDUqe40ErmxdxkvWrrZMI9QrdBX XPCvsN9JJ61ElyRLf2pX8tVo7z0C5vdK5FLr5ZScf7a4S8ZachXLA5eZ31Vpd1ps1Mllc7s/ K0alcW6u/oxIbSomTxfap//olOySTOmQ7SsPr4O2PsndnSfuO0GZJK8+EVHrx5ornFTBwEQ9 IrGtyBz0uvSKjiJl8abuuWMekoHscGoNgZ4sAmQF11LAP3tsvdPkBNS7rDzybcQ3b1oFJhbg cajy9tAAAAAAAAA= --------------ms060107070107090608010508--