Date: Mon, 3 Dec 2018 20:45:41 -0800 (PST) From: Don Lewis <truckman@FreeBSD.org> To: freebsd-fs@FreeBSD.org Subject: stuck with ufs CHECK-HASH errors Message-ID: <tkrat.98eb81fc50b0c146@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
I just ran into an issue with a vm guest running 13.0-CURRENT r341355 GENERIC amd64. The problem started when the vm guest paniced and when I rebooted it encountered an unexpected SU inconsitency and dropped into single user mode. When I ran fsck manually, I *think* it asked if I wanted to enable CHECK-HASH and I said yes. Now when I run fsck, it reports a bunch of CHECK-HASH errors, but doesn't fix them. I've copied the filesystem to a local file on the host that I'm accessing via /dev/md0 so that I can easily capture the results. # fsck /dev/md0 ** /dev/md0 ** Last Mounted on / ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups CG 0: BAD CHECK-HASH 0 vs 0x767f70f1 SUMMARY INFORMATION BAD SALVAGE? [yn] y ALLOCATED FILE 273 MARKED FREE ALLOCATED FRAG 37855 MARKED FREE BLK(S) MISSING IN BIT MAPS SALVAGE? [yn] y CG 1: BAD CHECK-HASH 0 vs 0x2a1c3669 CG 2: BAD CHECK-HASH 0 vs 0x4271e253 CG 3: BAD CHECK-HASH 0 vs 0x2b2ec911 CG 4: BAD CHECK-HASH 0 vs 0x650a2356 CG 5: BAD CHECK-HASH 0 vs 0x78a8ef2b CG 6: BAD CHECK-HASH 0 vs 0xb9879f89 CG 7: BAD CHECK-HASH 0 vs 0x5e6cffed CG 8: BAD CHECK-HASH 0 vs 0xacad5919 CG 9: BAD CHECK-HASH 0 vs 0x54f7e667 CG 10: BAD CHECK-HASH 0 vs 0x9b819e39 CG 11: BAD CHECK-HASH 0 vs 0xf4f4ab1a CG 12: BAD CHECK-HASH 0 vs 0x2c9be1ec CG 13: BAD CHECK-HASH 0 vs 0x52af104 CG 14: BAD CHECK-HASH 0 vs 0xcfc9d02d CG 15: BAD CHECK-HASH 0 vs 0xaf439644 CG 16: BAD CHECK-HASH 0 vs 0x8ff167a7 CG 17: BAD CHECK-HASH 0 vs 0x24cf0d5f CG 18: BAD CHECK-HASH 0 vs 0x7c9b00c5 CG 19: BAD CHECK-HASH 0 vs 0x8e58a45d CG 20: BAD CHECK-HASH 0 vs 0x427a8a1 CG 21: BAD CHECK-HASH 0 vs 0x7b624c0c CG 22: BAD CHECK-HASH 0 vs 0x4016d045 CG 23: BAD CHECK-HASH 0 vs 0xbeb868d1 CG 24: BAD CHECK-HASH 0 vs 0xbf3550a7 CG 25: BAD CHECK-HASH 0 vs 0xe1748785 CG 26: BAD CHECK-HASH 0 vs 0xacc5837a CG 27: BAD CHECK-HASH 0 vs 0x397b8e79 CG 28: BAD CHECK-HASH 0 vs 0xef868063 CG 29: BAD CHECK-HASH 0 vs 0x28ab03a9 CG 30: BAD CHECK-HASH 0 vs 0x8c1c2f40 CG 31: BAD CHECK-HASH 0 vs 0x2825b898 CG 32: BAD CHECK-HASH 0 vs 0x57200d54 CG 33: BAD CHECK-HASH 0 vs 0x8ddd9f4e CG 34: BAD CHECK-HASH 0 vs 0xa25fc59b CG 35: BAD CHECK-HASH 0 vs 0xbaf45843 CG 36: BAD CHECK-HASH 0 vs 0xb82def37 CG 37: BAD CHECK-HASH 0 vs 0x320cb852 CG 38: BAD CHECK-HASH 0 vs 0xa8d433d9 CG 39: BAD CHECK-HASH 0 vs 0x4fb023bb CG 40: BAD CHECK-HASH 0 vs 0x7606a0ac CG 41: BAD CHECK-HASH 0 vs 0xb25d3f34 CG 42: BAD CHECK-HASH 0 vs 0xb43acd7f CG 43: BAD CHECK-HASH 0 vs 0xd3c1ed7a CG 44: BAD CHECK-HASH 0 vs 0xf5df5b0d CG 45: BAD CHECK-HASH 0 vs 0x5f070643 CG 46: BAD CHECK-HASH 0 vs 0x1038a213 CG 47: BAD CHECK-HASH 0 vs 0x5e038a7b CG 48: BAD CHECK-HASH 0 vs 0xe30f28bb CG 49: BAD CHECK-HASH 0 vs 0x853827ef 720309 files, 2926458 used, 4686237 free (279357 frags, 550860 blocks, 3.7% fragmentation) ***** FILE SYSTEM IS CLEAN ***** ***** FILE SYSTEM WAS MODIFIED ***** If I run fsck again, I get the same result: # fsck /dev/md0 ** /dev/md0 ** Last Mounted on / ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups CG 0: BAD CHECK-HASH 0 vs 0x767f70f1 SUMMARY INFORMATION BAD SALVAGE? [yn] y ALLOCATED FILE 273 MARKED FREE ALLOCATED FRAG 37855 MARKED FREE BLK(S) MISSING IN BIT MAPS SALVAGE? [yn] y CG 1: BAD CHECK-HASH 0 vs 0x2a1c3669 CG 2: BAD CHECK-HASH 0 vs 0x4271e253 CG 3: BAD CHECK-HASH 0 vs 0x2b2ec911 CG 4: BAD CHECK-HASH 0 vs 0x650a2356 CG 5: BAD CHECK-HASH 0 vs 0x78a8ef2b CG 6: BAD CHECK-HASH 0 vs 0xb9879f89 CG 7: BAD CHECK-HASH 0 vs 0x5e6cffed CG 8: BAD CHECK-HASH 0 vs 0xacad5919 CG 9: BAD CHECK-HASH 0 vs 0x54f7e667 CG 10: BAD CHECK-HASH 0 vs 0x9b819e39 CG 11: BAD CHECK-HASH 0 vs 0xf4f4ab1a CG 12: BAD CHECK-HASH 0 vs 0x2c9be1ec CG 13: BAD CHECK-HASH 0 vs 0x52af104 CG 14: BAD CHECK-HASH 0 vs 0xcfc9d02d CG 15: BAD CHECK-HASH 0 vs 0xaf439644 CG 16: BAD CHECK-HASH 0 vs 0x8ff167a7 CG 17: BAD CHECK-HASH 0 vs 0x24cf0d5f CG 18: BAD CHECK-HASH 0 vs 0x7c9b00c5 CG 19: BAD CHECK-HASH 0 vs 0x8e58a45d CG 20: BAD CHECK-HASH 0 vs 0x427a8a1 CG 21: BAD CHECK-HASH 0 vs 0x7b624c0c CG 22: BAD CHECK-HASH 0 vs 0x4016d045 CG 23: BAD CHECK-HASH 0 vs 0xbeb868d1 CG 24: BAD CHECK-HASH 0 vs 0xbf3550a7 CG 25: BAD CHECK-HASH 0 vs 0xe1748785 CG 26: BAD CHECK-HASH 0 vs 0xacc5837a CG 27: BAD CHECK-HASH 0 vs 0x397b8e79 CG 28: BAD CHECK-HASH 0 vs 0xef868063 CG 29: BAD CHECK-HASH 0 vs 0x28ab03a9 CG 30: BAD CHECK-HASH 0 vs 0x8c1c2f40 CG 31: BAD CHECK-HASH 0 vs 0x2825b898 CG 32: BAD CHECK-HASH 0 vs 0x57200d54 CG 33: BAD CHECK-HASH 0 vs 0x8ddd9f4e CG 34: BAD CHECK-HASH 0 vs 0xa25fc59b CG 35: BAD CHECK-HASH 0 vs 0xbaf45843 CG 36: BAD CHECK-HASH 0 vs 0xb82def37 CG 37: BAD CHECK-HASH 0 vs 0x320cb852 CG 38: BAD CHECK-HASH 0 vs 0xa8d433d9 CG 39: BAD CHECK-HASH 0 vs 0x4fb023bb CG 40: BAD CHECK-HASH 0 vs 0x7606a0ac CG 41: BAD CHECK-HASH 0 vs 0xb25d3f34 CG 42: BAD CHECK-HASH 0 vs 0xb43acd7f CG 43: BAD CHECK-HASH 0 vs 0xd3c1ed7a CG 44: BAD CHECK-HASH 0 vs 0xf5df5b0d CG 45: BAD CHECK-HASH 0 vs 0x5f070643 CG 46: BAD CHECK-HASH 0 vs 0x1038a213 CG 47: BAD CHECK-HASH 0 vs 0x5e038a7b CG 48: BAD CHECK-HASH 0 vs 0xe30f28bb CG 49: BAD CHECK-HASH 0 vs 0x853827ef 720309 files, 2926458 used, 4686237 free (279357 frags, 550860 blocks, 3.7% fragmentation) ***** FILE SYSTEM IS CLEAN ***** ***** FILE SYSTEM WAS MODIFIED ***** Note that the check hashes flags are set below: # dumpfs /dev/md0 | head -20 dumpfs: /dev/md0: cylinder group checks failed magic 19540119 (UFS2) time Mon Dec 3 20:36:43 2018 superblock location 65536 id [ 55dce11e c2eee767 ] ncg 50 size 7864320 blocks 7612695 bsize 32768 shift 15 mask 0xffff8000 fsize 4096 shift 12 mask 0xfffff000 frag 8 shift 3 fsbtodb 3 minfree 8% optim time symlinklen 120 maxbsize 32768 maxbpg 4096 maxcontig 4 contigsumsize 4 nbfree 550860 ndir 11826 nifree 3292489 nffree 279357 bpg 20035 fpg 160280 ipg 80256 unrefs 0 nindir 4096 inopb 128 maxfilesize 2252349704110079 sbsize 4096 cgsize 32768 csaddr 5056 cssize 4096 sblkno 24 cblkno 32 iblkno 40 dblkno 5056 cgrotor 25 fmod 0 ronly 0 clean 1 metaspace 6408 avgfpdir 64 avgfilesize 16384 flags soft-updates check hashes superblock cylinder-groups fsmnt / volname swuid 0 providersize 7864320 Digging into fsck_ffs pass5.c seems to show that the cylinder group blocks are getting fixed and marked dirty, but I haven't been able to figure out what actually is supposed to cause them to get written back to disk. In the meantime, my vm guest is stuck because the back cylinder group hashes cause the cylinder groups to be treated as read-only, meaning that I can't create any new files.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?tkrat.98eb81fc50b0c146>