Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 3 Dec 2018 20:45:41 -0800 (PST)
From:      Don Lewis <truckman@FreeBSD.org>
To:        freebsd-fs@FreeBSD.org
Subject:   stuck with ufs CHECK-HASH errors
Message-ID:  <tkrat.98eb81fc50b0c146@FreeBSD.org>

next in thread | raw e-mail | index | archive | help
I just ran into an issue with a vm guest running 13.0-CURRENT r341355
GENERIC amd64.

The problem started when the vm guest paniced and when I rebooted it
encountered an unexpected SU inconsitency and dropped into single user
mode. When I ran fsck manually, I *think* it asked if I wanted to enable
CHECK-HASH and I said yes.  Now when I run fsck, it reports a bunch of
CHECK-HASH errors, but doesn't fix them.

I've copied the filesystem to a local file on the host that I'm
accessing via /dev/md0 so that I can easily capture the results.

# fsck /dev/md0
** /dev/md0
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
CG 0: BAD CHECK-HASH 0 vs 0x767f70f1
SUMMARY INFORMATION BAD
SALVAGE? [yn] y

ALLOCATED FILE 273 MARKED FREE
ALLOCATED FRAG 37855 MARKED FREE
BLK(S) MISSING IN BIT MAPS
SALVAGE? [yn] y

CG 1: BAD CHECK-HASH 0 vs 0x2a1c3669
CG 2: BAD CHECK-HASH 0 vs 0x4271e253
CG 3: BAD CHECK-HASH 0 vs 0x2b2ec911
CG 4: BAD CHECK-HASH 0 vs 0x650a2356
CG 5: BAD CHECK-HASH 0 vs 0x78a8ef2b
CG 6: BAD CHECK-HASH 0 vs 0xb9879f89
CG 7: BAD CHECK-HASH 0 vs 0x5e6cffed
CG 8: BAD CHECK-HASH 0 vs 0xacad5919
CG 9: BAD CHECK-HASH 0 vs 0x54f7e667
CG 10: BAD CHECK-HASH 0 vs 0x9b819e39
CG 11: BAD CHECK-HASH 0 vs 0xf4f4ab1a
CG 12: BAD CHECK-HASH 0 vs 0x2c9be1ec
CG 13: BAD CHECK-HASH 0 vs 0x52af104
CG 14: BAD CHECK-HASH 0 vs 0xcfc9d02d
CG 15: BAD CHECK-HASH 0 vs 0xaf439644
CG 16: BAD CHECK-HASH 0 vs 0x8ff167a7
CG 17: BAD CHECK-HASH 0 vs 0x24cf0d5f
CG 18: BAD CHECK-HASH 0 vs 0x7c9b00c5
CG 19: BAD CHECK-HASH 0 vs 0x8e58a45d
CG 20: BAD CHECK-HASH 0 vs 0x427a8a1
CG 21: BAD CHECK-HASH 0 vs 0x7b624c0c
CG 22: BAD CHECK-HASH 0 vs 0x4016d045
CG 23: BAD CHECK-HASH 0 vs 0xbeb868d1
CG 24: BAD CHECK-HASH 0 vs 0xbf3550a7
CG 25: BAD CHECK-HASH 0 vs 0xe1748785
CG 26: BAD CHECK-HASH 0 vs 0xacc5837a
CG 27: BAD CHECK-HASH 0 vs 0x397b8e79
CG 28: BAD CHECK-HASH 0 vs 0xef868063
CG 29: BAD CHECK-HASH 0 vs 0x28ab03a9
CG 30: BAD CHECK-HASH 0 vs 0x8c1c2f40
CG 31: BAD CHECK-HASH 0 vs 0x2825b898
CG 32: BAD CHECK-HASH 0 vs 0x57200d54
CG 33: BAD CHECK-HASH 0 vs 0x8ddd9f4e
CG 34: BAD CHECK-HASH 0 vs 0xa25fc59b
CG 35: BAD CHECK-HASH 0 vs 0xbaf45843
CG 36: BAD CHECK-HASH 0 vs 0xb82def37
CG 37: BAD CHECK-HASH 0 vs 0x320cb852
CG 38: BAD CHECK-HASH 0 vs 0xa8d433d9
CG 39: BAD CHECK-HASH 0 vs 0x4fb023bb
CG 40: BAD CHECK-HASH 0 vs 0x7606a0ac
CG 41: BAD CHECK-HASH 0 vs 0xb25d3f34
CG 42: BAD CHECK-HASH 0 vs 0xb43acd7f
CG 43: BAD CHECK-HASH 0 vs 0xd3c1ed7a
CG 44: BAD CHECK-HASH 0 vs 0xf5df5b0d
CG 45: BAD CHECK-HASH 0 vs 0x5f070643
CG 46: BAD CHECK-HASH 0 vs 0x1038a213
CG 47: BAD CHECK-HASH 0 vs 0x5e038a7b
CG 48: BAD CHECK-HASH 0 vs 0xe30f28bb
CG 49: BAD CHECK-HASH 0 vs 0x853827ef
720309 files, 2926458 used, 4686237 free (279357 frags, 550860 blocks, 3.7% fragmentation)

***** FILE SYSTEM IS CLEAN *****

***** FILE SYSTEM WAS MODIFIED *****

If I run fsck again, I get the same result:

# fsck /dev/md0
** /dev/md0
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
CG 0: BAD CHECK-HASH 0 vs 0x767f70f1
SUMMARY INFORMATION BAD
SALVAGE? [yn] y

ALLOCATED FILE 273 MARKED FREE
ALLOCATED FRAG 37855 MARKED FREE
BLK(S) MISSING IN BIT MAPS
SALVAGE? [yn] y

CG 1: BAD CHECK-HASH 0 vs 0x2a1c3669
CG 2: BAD CHECK-HASH 0 vs 0x4271e253
CG 3: BAD CHECK-HASH 0 vs 0x2b2ec911
CG 4: BAD CHECK-HASH 0 vs 0x650a2356
CG 5: BAD CHECK-HASH 0 vs 0x78a8ef2b
CG 6: BAD CHECK-HASH 0 vs 0xb9879f89
CG 7: BAD CHECK-HASH 0 vs 0x5e6cffed
CG 8: BAD CHECK-HASH 0 vs 0xacad5919
CG 9: BAD CHECK-HASH 0 vs 0x54f7e667
CG 10: BAD CHECK-HASH 0 vs 0x9b819e39
CG 11: BAD CHECK-HASH 0 vs 0xf4f4ab1a
CG 12: BAD CHECK-HASH 0 vs 0x2c9be1ec
CG 13: BAD CHECK-HASH 0 vs 0x52af104
CG 14: BAD CHECK-HASH 0 vs 0xcfc9d02d
CG 15: BAD CHECK-HASH 0 vs 0xaf439644
CG 16: BAD CHECK-HASH 0 vs 0x8ff167a7
CG 17: BAD CHECK-HASH 0 vs 0x24cf0d5f
CG 18: BAD CHECK-HASH 0 vs 0x7c9b00c5
CG 19: BAD CHECK-HASH 0 vs 0x8e58a45d
CG 20: BAD CHECK-HASH 0 vs 0x427a8a1
CG 21: BAD CHECK-HASH 0 vs 0x7b624c0c
CG 22: BAD CHECK-HASH 0 vs 0x4016d045
CG 23: BAD CHECK-HASH 0 vs 0xbeb868d1
CG 24: BAD CHECK-HASH 0 vs 0xbf3550a7
CG 25: BAD CHECK-HASH 0 vs 0xe1748785
CG 26: BAD CHECK-HASH 0 vs 0xacc5837a
CG 27: BAD CHECK-HASH 0 vs 0x397b8e79
CG 28: BAD CHECK-HASH 0 vs 0xef868063
CG 29: BAD CHECK-HASH 0 vs 0x28ab03a9
CG 30: BAD CHECK-HASH 0 vs 0x8c1c2f40
CG 31: BAD CHECK-HASH 0 vs 0x2825b898
CG 32: BAD CHECK-HASH 0 vs 0x57200d54
CG 33: BAD CHECK-HASH 0 vs 0x8ddd9f4e
CG 34: BAD CHECK-HASH 0 vs 0xa25fc59b
CG 35: BAD CHECK-HASH 0 vs 0xbaf45843
CG 36: BAD CHECK-HASH 0 vs 0xb82def37
CG 37: BAD CHECK-HASH 0 vs 0x320cb852
CG 38: BAD CHECK-HASH 0 vs 0xa8d433d9
CG 39: BAD CHECK-HASH 0 vs 0x4fb023bb
CG 40: BAD CHECK-HASH 0 vs 0x7606a0ac
CG 41: BAD CHECK-HASH 0 vs 0xb25d3f34
CG 42: BAD CHECK-HASH 0 vs 0xb43acd7f
CG 43: BAD CHECK-HASH 0 vs 0xd3c1ed7a
CG 44: BAD CHECK-HASH 0 vs 0xf5df5b0d
CG 45: BAD CHECK-HASH 0 vs 0x5f070643
CG 46: BAD CHECK-HASH 0 vs 0x1038a213
CG 47: BAD CHECK-HASH 0 vs 0x5e038a7b
CG 48: BAD CHECK-HASH 0 vs 0xe30f28bb
CG 49: BAD CHECK-HASH 0 vs 0x853827ef
720309 files, 2926458 used, 4686237 free (279357 frags, 550860 blocks, 3.7% fragmentation)

***** FILE SYSTEM IS CLEAN *****

***** FILE SYSTEM WAS MODIFIED *****

Note that the check hashes flags are set below:

# dumpfs /dev/md0 | head -20
dumpfs: /dev/md0: cylinder group checks failed
magic	19540119 (UFS2)	time	Mon Dec  3 20:36:43 2018
superblock location	65536	id	[ 55dce11e c2eee767 ]
ncg	50	size	7864320	blocks	7612695
bsize	32768	shift	15	mask	0xffff8000
fsize	4096	shift	12	mask	0xfffff000
frag	8	shift	3	fsbtodb	3
minfree	8%	optim	time	symlinklen 120
maxbsize 32768	maxbpg	4096	maxcontig 4	contigsumsize 4
nbfree	550860	ndir	11826	nifree	3292489	nffree	279357
bpg	20035	fpg	160280	ipg	80256	unrefs	0
nindir	4096	inopb	128	maxfilesize	2252349704110079
sbsize	4096	cgsize	32768	csaddr	5056	cssize	4096
sblkno	24	cblkno	32	iblkno	40	dblkno	5056
cgrotor	25	fmod	0	ronly	0	clean	1
metaspace 6408	avgfpdir 64	avgfilesize 16384
flags	soft-updates 
check hashes	superblock cylinder-groups 
fsmnt	/
volname		swuid	0	providersize	7864320


Digging into fsck_ffs pass5.c seems to show that the cylinder group
blocks are getting fixed and marked dirty, but I haven't been able to
figure out what actually is supposed to cause them to get written back
to disk.

In the meantime, my vm guest is stuck because the back cylinder group
hashes cause the cylinder groups to be treated as read-only, meaning
that I can't create any new files.




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?tkrat.98eb81fc50b0c146>