Date: Sat, 19 Sep 2020 03:00:50 +0000 From: Colin Percival <cperciva@tarsnap.com> To: ericr <erobison@gmail.com>, freebsd-cloud@freebsd.org, Kirk McKusick <mckusick@FreeBSD.org> Subject: Re: filesystem checksum problems on AWS EC2 instances Message-ID: <01000174a44ea02e-d511a094-ae7b-4c0c-b017-df25edaddd42-000000@email.amazonses.com> In-Reply-To: <CALLYBo7gAfxCPUjh=bDMe748xTwvVB66%2Bq4-X8JRCq6c-mXrfA@mail.gmail.com> References: <CALLYBo5iifB8P5Tf1MTpawj=NRYfjbyT1QHbS=NeeFfVYg0vAg@mail.gmail.com> <01000174944cbe04-bc3ea16f-b855-4aac-9bcf-1b9c4e30a241-000000@email.amazonses.com> <CALLYBo7gAfxCPUjh=bDMe748xTwvVB66%2Bq4-X8JRCq6c-mXrfA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[Adding Kirk since this seems like a UFS issue...] On 2020-09-16 15:15, ericr wrote: > On Tue, Sep 15, 2020 at 6:24 PM Colin Percival <cperciva@tarsnap.com> wrote: >> On 2020-09-15 14:30, ericr wrote: >>> Sep 1 20:50:15 <kern.crit> freebsd kernel: UFS /dev/gpt/rootfs (/) >>> cylinder checksum failed: cg 0, cgp: 0x9c14700e != bp: 0x27bfa3d0 >>> Sep 1 20:50:15 <kern.crit> freebsd syslogd: last message repeated 1 >> times >>> Sep 1 20:50:15 <kern.crit> freebsd kernel: UFS /dev/gpt/rootfs (/) >>> cylinder checksum failed: cg 7, cgp: 0x43ed3fa1 != bp: 0xe9b0182e >>> >>> and from there on, I get cylinder checksum errors pretty often. >> >> Do you get this if you launch from the non-Marketplace AMIs listed in the >> release announcement? >> https://www.freebsd.org/releases/12.1R/announce.html > > > Yes. I just tried both of these AMI's from the release notes: > us-east-1 region: ami-0de268ac2498ba33d > us-east-2 region: ami-0a44f10b2c6deb365 > > I got the same errors. I've managed to reproduce this, with a filesystem which I've verified is clean (at least, which passes fsck) before resizing up to ~ 200 GB: > root@freebsd:/usr/home/ec2-user # fsck_ufs /dev/nvd1p2 > ** /dev/nvd1p2 > ** Last Mounted on /releng/12-amd64-GENERIC-release/usr/obj/usr/src/amd64.amd64/release/cw-ec2/new > ** Phase 1 - Check Blocks and Sizes > ** Phase 2 - Check Pathnames > ** Phase 3 - Check Connectivity > ** Phase 4 - Check Reference Counts > ** Phase 5 - Check Cyl groups > 25701 files, 758977 used, 229774 free (9654 frags, 27515 blocks, 1.0% fragmentation) > > ***** FILE SYSTEM IS CLEAN ***** > root@freebsd:/usr/home/ec2-user # gpart recover /dev/nvd1 > nvd1 recovered > root@freebsd:/usr/home/ec2-user # gpart resize -i 2 /dev/nvd1 > nvd1p2 resized > root@freebsd:/usr/home/ec2-user # growfs -y /dev/nvd1p2 > super-block backups (for fsck_ffs -b #) at: > [snip] > root@freebsd:/usr/home/ec2-user # fsck_ufs /dev/nvd1p2 > ** /dev/nvd1p2 > ** Last Mounted on > ** Phase 1 - Check Blocks and Sizes > ** Phase 2 - Check Pathnames > ** Phase 3 - Check Connectivity > ** Phase 4 - Check Reference Counts > ** Phase 5 - Check Cyl groups > CG 0: BAD CHECK-HASH 0x9c14700e vs 0xc9441f74 > SUMMARY INFORMATION BAD > SALVAGE? [yn] n > > CG 7: BAD CHECK-HASH 0xad168305 vs 0x74ba48a > 25701 files, 758977 used, 50019285 free (9661 frags, 6251203 blocks, 0.0% fragmentation) > > ***** FILE SYSTEM MARKED DIRTY ***** > > ***** PLEASE RERUN FSCK ***** This seems like a bug in UFS and/or growfs, but I'm not familiar enough with either to say any more. Kirk, are you aware of any issues on FreeBSD 12.1-RELEASE which can cause cylinder checksum errors after growfs? (On amd64 if it matters.) If it would help I can provide you with SSH access to an affected EC2 instance. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?01000174a44ea02e-d511a094-ae7b-4c0c-b017-df25edaddd42-000000>