From owner-freebsd-hackers@freebsd.org Thu Jul 11 07:39:45 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3527C15EEB54 for ; Thu, 11 Jul 2019 07:39:45 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9B4B26A572; Thu, 11 Jul 2019 07:39:42 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cs.huji.ac.il; s=57791128; h=To:References:Message-Id:Content-Transfer-Encoding:Cc:Date:In-Reply-To:From:Subject:Mime-Version:Content-Type; bh=VsvUJmje+zRMyj8ne5WJr98VWP+SaKYV5+45HFk2HFQ=; b=vcfHsUbGS0m6scJ5CRLwK6BJPS5Ri7e1t7rCeH4heRX6BYLK/gbAMfcTNCZj9MKDWQ4hAvQFDfXzCEyMH/3+E+3xjCuNwrkl/oRaHYft4gd5EZ7LB9zqPiM8lG1hu/qhp2P0mZCGsTwLYMK06lMcV8ujrEz6VI9CLvwfYf9/toTBaBFXJfIJi10F+NZQ1aHKiVTG1fT/tVsw+4QBs3Xu+ZMAnmjfs9CUqstX62LEVOIG7xgIA3CelrfB29IPsahaBRSgefNwx1lo/VlS12iOFef4ZbKlCICHYhcUPGbRyjApSFXYTOzG8WplUQA948ORo72pRHHKBvDvpIU3qG4Fzw==; Received: from bach.cs.huji.ac.il ([132.65.80.20]) by kabab.cs.huji.ac.il with esmtp id 1hlTfz-000Fql-1C; Thu, 11 Jul 2019 10:39:35 +0300 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: zpool errors From: Daniel Braniss In-Reply-To: <70f1be10-e37a-de20-e188-6155fda2d06a@freebsd.org> Date: Thu, 11 Jul 2019 10:39:34 +0300 Cc: freebsd-hackers@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <05D8BD75-78B4-4336-8A8A-C84A901CB3D4@cs.huji.ac.il> References: <52CE32B1-7E01-4C35-A2AB-84D3D5BD4E2F@cs.huji.ac.il> <27c3e59a-07ea-5df3-9de2-302d5290a477@freebsd.org> <831204B6-3F3B-4736-89FA-1207C4C46A7E@cs.huji.ac.il> <70f1be10-e37a-de20-e188-6155fda2d06a@freebsd.org> To: Allan Jude X-Mailer: Apple Mail (2.3445.9.1) X-Rspamd-Queue-Id: 9B4B26A572 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cs.huji.ac.il header.s=57791128 header.b=vcfHsUbG X-Spamd-Result: default: False [-2.38 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.88)[-0.884,0]; R_DKIM_ALLOW(-0.20)[cs.huji.ac.il:s=57791128]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[huji.ac.il]; NEURAL_HAM_LONG(-1.00)[-0.996,0]; IP_SCORE(-0.41)[ipnet: 132.64.0.0/13(-1.18), asn: 378(-0.94), country: IL(0.05)]; MX_GOOD(-0.01)[cached: kabab.cs.huji.ac.il]; DKIM_TRACE(0.00)[cs.huji.ac.il:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[210.116.65.132.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; NEURAL_HAM_SHORT(-0.28)[-0.280,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:378, ipnet:132.64.0.0/13, country:IL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Jul 2019 07:39:45 -0000 > On 10 Jul 2019, at 20:23, Allan Jude wrote: >=20 > On 2019-07-10 11:37, Daniel Braniss wrote: >>=20 >>=20 >>> On 10 Jul 2019, at 18:24, Allan Jude wrote: >>>=20 >>> On 2019-07-10 10:48, Daniel Braniss wrote: >>>> hi, >>>> i got a degraded pool, but can=E2=80=99t make sense of the file = name: >>>>=20 >>>> protonew-2# zpool status -vx >>>> pool: h >>>> state: ONLINE >>>> status: One or more devices has experienced an error resulting in = data >>>> corruption. Applications may be affected. >>>> action: Restore the file in question if possible. Otherwise = restore the >>>> entire pool from backup. >>>> see: http://illumos.org/msg/ZFS-8000-8A = >>>> scan: scrub repaired 6.50K in 17h30m with 0 errors on Wed Jul 10 = 12:06:14 2019 >>>> config: >>>>=20 >>>> NAME STATE READ WRITE CKSUM >>>> h ONLINE 0 0 14.4M >>>> gpt/r5/zfs ONLINE 0 0 57.5M >>>>=20 >>>> errors: Permanent errors have been detected in the following files: >>>>=20 >>>> <0x102>:<0x30723> >>>> <0x102>:<0x30726> >>>> <0x102>:<0x3062a> >>>> =E2=80=A6 >>>> <0x281>:<0x0> >>>> <0x6aa>:<0x305cd> >>>> <0xffffffffffffffff>:<0x305cd> >>>>=20 >>>>=20 >>>> any hints as how I can identify third files? >>>>=20 >>>> thanks, >>>> danny >>>>=20 >>>> _______________________________________________ >>>> freebsd-hackers@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >>>> To unsubscribe, send any mail to = "freebsd-hackers-unsubscribe@freebsd.org" >>>>=20 >>>=20 >>> Once a file has been deleted, ZFS can have a hard time determining = its >>> filename. >>>=20 >>> It is inode 198186 (0x3062a) on dataset 0x102. The file has been >>> deleted, but still exists in at least one snapshot. >>>=20 >>> Although, 57 million checksum errors seems like there may be some = other >>> problem. You might look for and resolve the problem with what = appears to >>> be a raid5 you have built your ZFS pool on top of it? Then do 'zpool >>> clear' to reset the counters to zero, and 'zpool scrub' to try to = read >>> everything again. >>>=20 >>> --=20 >>> Allan Jude >>>=20 >> I don=E2=80=99t know when the first error was detected, and this host = has been up for 367 days! >> I did a scrub but no change. >> i will remove old snapshots and see if it helps. >>=20 >> is it possible to know at least which volume? >>=20 >> thanks, >> danny >>=20 >>=20 >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to = "freebsd-hackers-unsubscribe@freebsd.org" >>=20 >=20 > zdb -ddddd h 0x102 >=20 > Should tell you about which dataset that is >=20 > --=20 > Allan Jude >=20 firstly, thanks for your help! now, after doing a zpool clear, I notice that the CHKSUM is growing, the pool is on a raid controller raid5 (PERC from dell) which is showing it=E2=80=99s correcting the errors (=E2=80=98Corrected medium error = during recovery on PD =E2=80=A6). so what can be the cause? btw, the FreeBSD is 10.3-stable.