From owner-freebsd-hackers@freebsd.org Thu Jul 11 13:18:11 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7AFEF15D242F for ; Thu, 11 Jul 2019 13:18:11 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E35477688B; Thu, 11 Jul 2019 13:18:09 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cs.huji.ac.il; s=57791128; h=To:References:Message-Id:Content-Transfer-Encoding:Cc:Date:In-Reply-To:From:Subject:Mime-Version:Content-Type; bh=mpgsTtLaBjyWeYtdbOac0I4mspLt4bOw8de7noS3tfk=; b=f3QtQRdCeEUXwEQpCBA4a4QE+wKazAhmlPtWJdiSr93nPt/WuHlFGBT1ErhelgO5bQFERwz/J/rX8/Q8yF6nzu6DQDqKi9yiQlIYRTsM6nzxR9V5/HQ1W71YjuwwHAQnfpfmMSE31ylc6qZTbpAfAD1+DRup16bgwp+4VlE7FBdZr2mGkaAYmjPmCrL/eba3H1+KOpOS+qMn11CZLMadNEAQhY8lNtNK+XXMq2swkwftJiPKDqjqHbkQLzIKI13Y+dXUPCvHxUMeCfOFkb4T33OXXugyWTQJzT46Erx4alowtuRRRdonjwhvgf3g8dkRwG1ExfvoRVAtK47BrLwt1w==; Received: from macmini.bk.cs.huji.ac.il ([132.65.179.19]) by kabab.cs.huji.ac.il with esmtp id 1hlYxV-0006qX-3Z; Thu, 11 Jul 2019 16:18:01 +0300 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: zpool errors From: Daniel Braniss In-Reply-To: <05D8BD75-78B4-4336-8A8A-C84A901CB3D4@cs.huji.ac.il> Date: Thu, 11 Jul 2019 16:18:00 +0300 Cc: freebsd-hackers@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <88CFC175-8275-4C4E-B7BE-110E07C0A31C@cs.huji.ac.il> References: <52CE32B1-7E01-4C35-A2AB-84D3D5BD4E2F@cs.huji.ac.il> <27c3e59a-07ea-5df3-9de2-302d5290a477@freebsd.org> <831204B6-3F3B-4736-89FA-1207C4C46A7E@cs.huji.ac.il> <70f1be10-e37a-de20-e188-6155fda2d06a@freebsd.org> <05D8BD75-78B4-4336-8A8A-C84A901CB3D4@cs.huji.ac.il> To: Allan Jude X-Mailer: Apple Mail (2.3445.104.11) X-Rspamd-Queue-Id: E35477688B X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cs.huji.ac.il header.s=57791128 header.b=f3QtQRdC X-Spamd-Result: default: False [-2.13 / 15.00]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; MX_GOOD(-0.01)[cached: kabab.cs.huji.ac.il]; DKIM_TRACE(0.00)[cs.huji.ac.il:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.05)[-0.049,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:378, ipnet:132.64.0.0/13, country:IL]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.88)[-0.879,0]; R_DKIM_ALLOW(-0.20)[cs.huji.ac.il:s=57791128]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.99)[-0.994,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[huji.ac.il]; IP_SCORE(-0.39)[ipnet: 132.64.0.0/13(-1.12), asn: 378(-0.90), country: IL(0.05)]; RCVD_IN_DNSWL_NONE(0.00)[210.116.65.132.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Jul 2019 13:18:11 -0000 > On 11 Jul 2019, at 10:39, Daniel Braniss wrote: >=20 >=20 >=20 >> On 10 Jul 2019, at 20:23, Allan Jude wrote: >>=20 >> On 2019-07-10 11:37, Daniel Braniss wrote: >>>=20 >>>=20 >>>> On 10 Jul 2019, at 18:24, Allan Jude wrote: >>>>=20 >>>> On 2019-07-10 10:48, Daniel Braniss wrote: >>>>> hi, >>>>> i got a degraded pool, but can=E2=80=99t make sense of the file = name: >>>>>=20 >>>>> protonew-2# zpool status -vx >>>>> pool: h >>>>> state: ONLINE >>>>> status: One or more devices has experienced an error resulting in = data >>>>> corruption. Applications may be affected. >>>>> action: Restore the file in question if possible. Otherwise = restore the >>>>> entire pool from backup. >>>>> see: http://illumos.org/msg/ZFS-8000-8A = >>>>> scan: scrub repaired 6.50K in 17h30m with 0 errors on Wed Jul 10 = 12:06:14 2019 >>>>> config: >>>>>=20 >>>>> NAME STATE READ WRITE CKSUM >>>>> h ONLINE 0 0 14.4M >>>>> gpt/r5/zfs ONLINE 0 0 57.5M >>>>>=20 >>>>> errors: Permanent errors have been detected in the following = files: >>>>>=20 >>>>> <0x102>:<0x30723> >>>>> <0x102>:<0x30726> >>>>> <0x102>:<0x3062a> >>>>> =E2=80=A6 >>>>> <0x281>:<0x0> >>>>> <0x6aa>:<0x305cd> >>>>> <0xffffffffffffffff>:<0x305cd> >>>>>=20 >>>>>=20 >>>>> any hints as how I can identify third files? >>>>>=20 >>>>> thanks, >>>>> danny >>>>>=20 >>>>> _______________________________________________ >>>>> freebsd-hackers@freebsd.org mailing list >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >>>>> To unsubscribe, send any mail to = "freebsd-hackers-unsubscribe@freebsd.org" >>>>>=20 >>>>=20 >>>> Once a file has been deleted, ZFS can have a hard time determining = its >>>> filename. >>>>=20 >>>> It is inode 198186 (0x3062a) on dataset 0x102. The file has been >>>> deleted, but still exists in at least one snapshot. >>>>=20 >>>> Although, 57 million checksum errors seems like there may be some = other >>>> problem. You might look for and resolve the problem with what = appears to >>>> be a raid5 you have built your ZFS pool on top of it? Then do = 'zpool >>>> clear' to reset the counters to zero, and 'zpool scrub' to try to = read >>>> everything again. >>>>=20 >>>> --=20 >>>> Allan Jude >>>>=20 >>> I don=E2=80=99t know when the first error was detected, and this = host has been up for 367 days! >>> I did a scrub but no change. >>> i will remove old snapshots and see if it helps. >>>=20 >>> is it possible to know at least which volume? >>>=20 >>> thanks, >>> danny >>>=20 >>>=20 >>> _______________________________________________ >>> freebsd-hackers@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >>> To unsubscribe, send any mail to = "freebsd-hackers-unsubscribe@freebsd.org" >>>=20 >>=20 >> zdb -ddddd h 0x102 >>=20 >> Should tell you about which dataset that is >>=20 >> --=20 >> Allan Jude >>=20 >=20 the above did=E2=80=99t work for me, but, after removing old snapshots I reduced the problematic files to 1! <0xffffffffffffffff>:<0x305cd> which seems very odd -1? so now I removed more old snapshots, and started a a new zpool scrub. what still worries me is the fast growing checksum count, thanks, danny > firstly, thanks for your help! > now, after doing a zpool clear, I notice that the CHKSUM is growing, > the pool is on a raid controller raid5 (PERC from dell) which is = showing > it=E2=80=99s correcting the errors (=E2=80=98Corrected medium error = during recovery on PD =E2=80=A6). >=20 > so what can be the cause? btw, the FreeBSD is 10.3-stable. >=20 > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to = "freebsd-hackers-unsubscribe@freebsd.org"