Date: Tue, 15 Sep 2020 15:30:57 -0600 From: ericr <erobison@gmail.com> To: freebsd-cloud@freebsd.org Subject: filesystem checksum problems on AWS EC2 instances Message-ID: <CALLYBo5iifB8P5Tf1MTpawj=NRYfjbyT1QHbS=NeeFfVYg0vAg@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hi, I'm having disk/filesystem problems creating EC2 instances using 12.1-RELEASE (AMI ami-034de978f5d9c9b1b in us-east-1 and AMI ami-06d9854b88cba8200 in us-east-2, both from the Marketplace). I'm using t2.large instances. As a part of the instance creation, the OS gets updated from 12.1-RELEASE to 12.1-RELEASE-P9. THe AMIs use UFS. I created instances with a 200GB disk, in both us-east-1 and us-east-2, to rule out flakey hardware. Once the OS starts at first boot, I get cylinder checksum errors: Sep 1 20:50:15 <kern.crit> freebsd kernel: xbd0: attaching as ada0 Sep 1 20:50:15 <kern.crit> freebsd kernel: GEOM: ada0: the secondary GPT header is not in the last LBA. Sep 1 20:50:15 <kern.crit> freebsd kernel: Trying to mount root from ufs:/dev/gpt/rootfs [rw]... Sep 1 20:50:15 <kern.crit> freebsd kernel: random: unblocking device. Sep 1 20:50:15 <kern.crit> freebsd kernel: UFS /dev/gpt/rootfs (/) cylinder checksum failed: cg 0, cgp: 0x9c14700e != bp: 0x27bfa3d0 ... Sep 1 20:50:15 <kern.crit> freebsd kernel: device_attach: intsmb0 attach returned 6 Sep 1 20:50:15 <kern.crit> freebsd kernel: UFS /dev/gpt/rootfs (/) cylinder checksum failed: cg 0, cgp: 0x9c14700e != bp: 0x27bfa3d0 Sep 1 20:50:15 <kern.crit> freebsd syslogd: last message repeated 1 times Sep 1 20:50:15 <kern.crit> freebsd kernel: UFS /dev/gpt/rootfs (/) cylinder checksum failed: cg 7, cgp: 0x43ed3fa1 != bp: 0xe9b0182e and from there on, I get cylinder checksum errors pretty often. Forcing fsck to run at the next boot produces messages about the cylinder groups: ** Phase 5 - Check Cyl groups CG 0: BAD CHECK-HASH 0x9c14700e vs 0x27bfa3d0 FREE BLK COUNT(S) WRONG IN SUPERBLK SALVAGE? no SUMMARY INFORMATION BAD SALVAGE? no BLK(S) MISSING IN BIT MAPS SALVAGE? no CG 7: BAD CHECK-HASH 0x43ed3fa1 vs 0xe9b0182e 43494 files, 1012718 used, 49736354 free (8570 frags, 6215973 blocks, 0.0% fragmentation) Running fsck multiple times at boot doesn't seem to fix the problems. (I edited /etc/rc.d/fsck so that anytime fsck runs, it uses "fsck -f -d -T ffs:-R -T ufs:-R" ) I've also tried creating instances with 100GB disks, I get similar issues. tunefs didn't show any obvious issues: tunefs: POSIX.1e ACLs: (-a) disabled tunefs: NFSv4 ACLs: (-N) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) enabled tunefs: soft update journaling: (-j) disabled tunefs: gjournal: (-J) disabled tunefs: trim: (-t) disabled tunefs: maximum blocks per file in a cylinder group: (-e) 4096 tunefs: average file size: (-f) 16384 tunefs: average number of files in a directory: (-s) 64 tunefs: minimum percentage of free space: (-m) 8% tunefs: space to hold for metadata blocks: (-k) 6408 tunefs: optimization preference: (-o) time tunefs: volume label: (-L) rootfs I can replicate the problem pretty easily, I've created instances half a dozen times now, they all have the same issue. So I'm stumped. I looked in the bug database and the list archives and didn't find anything like this. I can't figure out if it's a problem with the way AWS creates the disk, or if it's an installer issue with FreeBSD. (or if I'm just screwing up the creation myself) I know the default disk size is 10GB, and growfs doesn't seem to complain when it increases the size to 100 or 200GB. Is there some size limit on EBS disks? Also, does anyone have docs on how to create a custom AMI that'll create a ZFS root filesystem and install the OS to it? I can't seem to find AMIs for 12.1-RELEASE that use it. Thanks!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALLYBo5iifB8P5Tf1MTpawj=NRYfjbyT1QHbS=NeeFfVYg0vAg>