From owner-freebsd-current@freebsd.org Tue Dec 12 22:58:39 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 84DD3E87D57 for ; Tue, 12 Dec 2017 22:58:39 +0000 (UTC) (envelope-from cy.schubert@komquats.com) Received: from smtp-out-no.shaw.ca (smtp-out-no.shaw.ca [64.59.134.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 42F5C77117; Tue, 12 Dec 2017 22:58:38 +0000 (UTC) (envelope-from cy.schubert@komquats.com) Received: from spqr.komquats.com ([96.50.22.10]) by shaw.ca with SMTP id OtVNesatfp2osOtVOeJiDv; Tue, 12 Dec 2017 15:58:32 -0700 X-Authority-Analysis: v=2.2 cv=KLEqNBNo c=1 sm=1 tr=0 a=jvE2nwUzI0ECrNeyr98KWA==:117 a=jvE2nwUzI0ECrNeyr98KWA==:17 a=ocR9PWop10UA:10 a=YxBL1-UpAAAA:8 a=6I5d2MoRAAAA:8 a=iKhvJSA4AAAA:8 a=QaFPDjgRRQkh4GSrB6QA:9 a=6pCtFBtut8bTVypo:21 a=3ge-KubrqmnXQUf8:21 a=QEXdDO2ut3YA:10 a=jkd8HGfWraxzm8Cd1awA:9 a=8gwv1ZuDhNe8EdX7:21 a=JIcB6NhWvQbpWVuT:21 a=PIwEOpKiG8dlj1kq:21 a=_W_S_7VecoQA:10 a=Ia-lj3WSrqcvXOmTRaiG:22 a=IjZwj45LgO3ly-622nXo:22 a=odh9cflL3HIXMm4fY7Wr:22 Received: from [25.81.38.66] (S0106d4ca6d8943b0.gv.shawcable.net [24.68.134.59]) by spqr.komquats.com (Postfix) with ESMTPSA id CBCF2266; Tue, 12 Dec 2017 14:58:27 -0800 (PST) MIME-Version: 1.0 From: Cy Schubert Subject: RE: SMART: disk problems on RAIDZ1 pool: (ada6:ahcich6:0:0:0): CAMstatus: ATA Status Error Date: Tue, 12 Dec 2017 14:58:28 -0800 To: "O. Hartmann" , "Rodney W. Grimes" CC: FreeBSD CURRENT , Freddie Cash , Alan Somers Message-Id: <20171212225827.CBCF2266@spqr.komquats.com> X-CMAE-Envelope: MS4wfMQ/99Ijn4dNclK4c5aRGAeJ4QabyKPCe7xwxg22MiZJ3/OuyKL7f2vFvTj7fkUv3kX/DA5ktphEkRVDepFc+UmVZgZPD8iGy+7c7KQz4jKxKTd7/VBv hBl0hSGk7oWk61jPS/HT3O6cgnYt1ke2jUjbfvPPCwvMj8ScjDqtVL4jbtdqugQLHtQpE5Ppg+AVtsYdYu7/KSMmluLEBwbq9XWqVpQ68391zUrzHQfTLO2P bGhc1OvI2GBXGGbAF2JoR2JUmYLXtoLK8AfnR2XaC/ytHaiEWLnaQthiCnNoJg/IbU9ob54XTM2fLgr/A5Rp0ZcG4KAZ3d1c1k9G6X2KmiY= Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Dec 2017 22:58:39 -0000 There are a couple of ways you can address this. You'll need to offline the= vdev first. If you've done a smartcrl -t long and if the test failed, smar= tcrl -a will tell you which block it had an issue with. You can use dd, ddr= escue or dd_rescue to dd the block over itself. The drive may rewrite the (= weak) block or if it fails to it will remap it (subsequently showing as rea= llocated). Of course there is a risk. If the sector is any of the boot blocks there is= a good chance the server will hang. You have to be *absolutely* sure which the bad sector is. And, there may be= more. There is a risk of data loss. I've used this technique many times. Most times it works perfectly. Other t= imes the affected file is lost but the rest of the file system is recovered= . And again there is always the risk. Replace the disk immediately if you experience a growing succession of pend= ing sectors. Otherwise replace the disk at your earliest convenience. If using a zfs mirror (not in your case) detatch and attach will rewrite an= y weakly written sectors and reallocate pending sectors. --- Sent using a tiny phone keyboard. Apologies for any typos and autocorrect. Also, this old phone only supports top post. Apologies. Cy Schubert or The need of the many outweighs the greed of the few. --- -----Original Message----- From: O. Hartmann Sent: 12/12/2017 14:19 To: Rodney W. Grimes Cc: O. Hartmann; FreeBSD CURRENT; Freddie Cash; Alan Somers Subject: Re: SMART: disk problems on RAIDZ1 pool: (ada6:ahcich6:0:0:0): CAM= status: ATA Status Error Am Tue, 12 Dec 2017 10:52:27 -0800 (PST) "Rodney W. Grimes" schrieb: Thank you for answering that fast! > > Hello, > >=20 > > running CURRENT (recent r326769), I realised that smartmond sends out s= ome console > > messages when booting the box: > >=20 > > [...] > > Dec 12 14:14:33 <3.2> box1 smartd[68426]: Device: /dev/ada6, 1 Currentl= y unreadable > > (pending) sectors Dec 12 14:14:33 <3.2> box1 smartd[68426]: Device: /de= v/ada6, 1 > > Offline uncorrectable sectors > > [...] > >=20 > > Checking the drive's SMART log with smartctl (it is one of four 3TB dis= k drives), I > > gather these informations: > >=20 > > [... smartctl -x /dev/ada6 ...] > > Error 42 [17] occurred at disk power-on lifetime: 25335 hours (1055 day= s + 15 hours) > > When the command that caused the error occurred, the device was activ= e or idle. > >=20 > > After command completion occurred, registers were: > > ER -- ST COUNT LBA_48 LH LM LL DV DC > > -- -- -- =3D=3D -- =3D=3D =3D=3D =3D=3D -- -- -- -- -- > > 40 -- 51 00 00 00 00 c2 7a 72 98 40 00 Error: UNC at LBA =3D 0xc27a7= 298 =3D 3262804632 > >=20 > > Commands leading to the command that caused the error were: > > CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feat= ure_Name > > -- =3D=3D -- =3D=3D -- =3D=3D =3D=3D =3D=3D -- -- -- -- -- ---------= ------ -------------------- > > 60 00 b0 00 88 00 00 c2 7a 73 20 40 08 23:38:12.195 READ FPDMA Q= UEUED > > 60 00 b0 00 80 00 00 c2 7a 72 70 40 08 23:38:12.195 READ FPDMA Q= UEUED > > 2f 00 00 00 01 00 00 00 00 00 10 40 08 23:38:12.195 READ LOG EXT > > 60 00 b0 00 70 00 00 c2 7a 73 20 40 08 23:38:09.343 READ FPDMA Q= UEUED > > 60 00 b0 00 68 00 00 c2 7a 72 70 40 08 23:38:09.343 READ FPDMA Q= UEUED > > [...] > >=20 > > and > >=20 > > [...] > > SMART Attributes Data Structure revision number: 16 > > Vendor Specific SMART Attributes with Thresholds: > > ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE > > 1 Raw_Read_Error_Rate POSR-K 200 200 051 - 64 > > 3 Spin_Up_Time POS--K 178 170 021 - 6075 > > 4 Start_Stop_Count -O--CK 098 098 000 - 2406 > > 5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0 > > 7 Seek_Error_Rate -OSR-K 200 200 000 - 0 > > 9 Power_On_Hours -O--CK 066 066 000 - 25339 > > 10 Spin_Retry_Count -O--CK 100 100 000 - 0 > > 11 Calibration_Retry_Count -O--CK 100 100 000 - 0 > > 12 Power_Cycle_Count -O--CK 098 098 000 - 2404 > > 192 Power-Off_Retract_Count -O--CK 200 200 000 - 154 > > 193 Load_Cycle_Count -O--CK 001 001 000 - 2055746 > > 194 Temperature_Celsius -O---K 122 109 000 - 28 > > 196 Reallocated_Event_Count -O--CK 200 200 000 - 0 > > 197 Current_Pending_Sector -O--CK 200 200 000 - 1 > > 198 Offline_Uncorrectable ----CK 200 200 000 - 1 > > 199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0 > > 200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 5 > > ||||||_ K auto-keep > > |||||__ C event count > > ||||___ R error rate > > |||____ S speed/performance > > ||_____ O updated online > > |______ P prefailure warning > >=20 > > [...] =20 >=20 > The data up to this point informs us that you have 1 bad sector > on a 3TB drive, that is actually an expected event given the data > error rate on this stuff is such that your gona have these now > and again. >=20 > Given you have 1 single event I would not suspect that this drive > is dying, but it would be prudent to prepare for that possibility. Hello. Well, I copied simply "one single event" that has been logged so far. As you (and I) can see, it is error #42. After I posted here, a reboot has = taken place because the "repair" process on the Pool suddenly increased time and now I'= m with error #47, but interestingly, it is a new block that is damaged, but the SMART at= tribute fields show this for now: [...] SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 1 Raw_Read_Error_Rate POSR-K 200 200 051 - 69 3 Spin_Up_Time POS--K 178 170 021 - 6075 4 Start_Stop_Count -O--CK 098 098 000 - 2406 5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0 7 Seek_Error_Rate -OSR-K 200 200 000 - 0 9 Power_On_Hours -O--CK 066 066 000 - 25343 10 Spin_Retry_Count -O--CK 100 100 000 - 0 11 Calibration_Retry_Count -O--CK 100 100 000 - 0 12 Power_Cycle_Count -O--CK 098 098 000 - 2404 192 Power-Off_Retract_Count -O--CK 200 200 000 - 154 193 Load_Cycle_Count -O--CK 001 001 000 - 2055746 194 Temperature_Celsius -O---K 122 109 000 - 28 196 Reallocated_Event_Count -O--CK 200 200 000 - 0 197 Current_Pending_Sector -O--CK 200 200 000 - 0 198 Offline_Uncorrectable ----CK 200 200 000 - 1 199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0 200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 5 ||||||_ K auto-keep |||||__ C event count ||||___ R error rate |||____ S speed/performance ||_____ O updated online |______ P prefailure warning [...] 197 Current_Pending_Sector decreased to zero so far, but with every reboot,= the error count seems to increase: [...] Error 47 [22] occurred at disk power-on lifetime: 25343 hours (1055 days + = 23 hours) When the command that caused the error occurred, the device was active or= idle. After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- =3D=3D -- =3D=3D =3D=3D =3D=3D -- -- -- -- -- 40 -- 51 00 00 00 00 c2 19 d9 88 40 00 Error: UNC at LBA =3D 0xc219d988 = =3D 3256473992 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_= Name -- =3D=3D -- =3D=3D -- =3D=3D =3D=3D =3D=3D -- -- -- -- -- -------------= -- -------------------- 60 00 b0 00 d0 00 00 c2 19 da 28 40 08 1d+07:12:34.336 READ FPDMA QUEUE= D 60 00 b0 00 c8 00 00 c2 19 d9 78 40 08 1d+07:12:34.336 READ FPDMA QUEUE= D 2f 00 00 00 01 00 00 00 00 00 10 40 08 1d+07:12:34.336 READ LOG EXT 60 00 b0 00 b8 00 00 c2 19 da 28 40 08 1d+07:12:31.484 READ FPDMA QUEUE= D 60 00 b0 00 b0 00 00 c2 19 d9 78 40 08 1d+07:12:31.483 READ FPDMA QUEUE= D I think this is watching a HDD dying, isn't it? I'd say, a broken cabling would produce different errors, wouldn't it? The Western Digital Green series HDD is a useful fellow when the HDD is use= d as a single drive. I think there might be an issue with paring 4 HDDs, 3 of them "GREEN= ", in a RAIDZ and physically sitting next to each other. Maybe it is time to replace them= one by one ... >=20 >=20 > >=20 > > The ZFS pool is RAIDZ1, comprised of 3 WD Green 3TB HDD and one WD RED = 3 TB HDD. The > > failure occured is on one of the WD Green 3 TB HDD. =20 > Ok, so the data is redundantly protected. This helps a lot. >=20 > > The pool is marked as "resilvered" - I do scrubbing on a regular basis = and the > > "resilvering" message has now aapeared the second time in row. Searchin= g the net > > recommend on SMART attribute 197 errors, in my case it is one, and in c= ombination with > > the problems occured that I should replace the disk. =20 >=20 > It is probably putting the RAIDZ in that state as the scrub is finding a = block > it can not read. >=20 > >=20 > > Well, here comes the problem. The box is comprised from "electronical w= aste" made by > > ASRock - it is a Socket 1150/IvyBridge board, which has its last Firmwa= re/BIOS update > > got in 2013 and since then UEFI booting FreeBSD from a HDD isn't possib= le (just to > > indicate that I'm aware of having issues with crap, but that is some ot= her issue > > right now). The board's SATA connectors are all populated. > >=20 > > So: Due to the lack of adequate backup space I can only selectively bac= kup portions, > > most of the space is occupied by scientific modelling data, which I had= worked on. So > > backup exists! In one way or the other. My concern is how to replace th= e faulty HDD! > > Most HowTo's indicate a replacement disk being prepared and then "repla= ced" via ZFS's > > replace command. This isn't applicable here. > >=20 > > Question: is it possible to simply pull the faulty disk (implies I know= exactly which > > one to pull!) and then prepare and add the replacement HDD and let the = system do its > > job resilvering the pool? =20 >=20 > That may work, but I think I have a simpler solution. >=20 > >=20 > > Next question is: I'm about to replace the 3 TB HDD with a more recent = and modern 4 TB > > HDD (WD RED 4TB). I'm aware of the fact that I can only use 3 TB as the= other disks > > are 3 TB, but I'd like to know whether FreeBSD's ZFS is capable of hand= ling it? =20 >=20 > Someone else? >=20 > >=20 > > This is the first time I have issues with ZFS and a faulty drive, so if= some of my > > questions sound naive, please forgive me. =20 >=20 > One thing to try is to see if we can get the drive to fix itself, first o= rder > of business is can you take this server out of service? If so I would > simply try to do a > repeat 100 dd if=3D/dev/whicheverhdisbad of=3D/dev/null conv=3Dnoerror, s= ync iseek=3D3262804632 >=20 > That is trying to read that block 100 times, if it successful even 1 time > smart should remap the block and you are all done. Given the fact, that this errorneous block is like a moving target, it this= solution still the favorite one? I'll try, but I already have the replacement 4 TB H= DD at hand. >=20 > If that fails we can try to zero the block, there is a risk here, but rai= dz should just > handle this as a data corruption of a block. This could possibly lead to= data loss, > so USE AT YOUR OWN RISK ASSESMENT. > dd if=3D/dev/zero of=3D/dev/whateverdrivehasissues bs=3D512 count=3D1 ose= ek=3D3262804632 I would then be oseek=3D3256473992, too. >=20 > That should forceable overwrite the bad block with 0's, the smart firmwar= e > well see this in the pending list, write the data, read it back, if succe= ssful > remove it from the pending list, if failed reallocate the block and write > the 0's to the reallocation and add 1 to the remapped block count. >=20 > You might google for "how to fix a pending reallocation" >=20 > > Thanks in advance, > > Oliver > > --=20 > > O. Hartmann =20 >=20 Kind regards, Oliver --=20 O. Hartmann Ich widerspreche der Nutzung oder =C3=9Cbermittlung meiner Daten f=C3=BCr Werbezwecke oder f=C3=BCr die Markt- oder Meinungsforschung (=C2=A7 28 Abs.= 4 BDSG).