From owner-freebsd-cloud@freebsd.org Tue Sep 15 21:31:37 2020 Return-Path: Delivered-To: freebsd-cloud@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0EA313E085E for ; Tue, 15 Sep 2020 21:31:37 +0000 (UTC) (envelope-from erobison@gmail.com) Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Brbyb6W87z3S6m for ; Tue, 15 Sep 2020 21:31:35 +0000 (UTC) (envelope-from erobison@gmail.com) Received: by mail-wr1-x436.google.com with SMTP id w5so4774841wrp.8 for ; Tue, 15 Sep 2020 14:31:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=lWkzxdSUkUjYFZNUHHDx+8cNSrQYp+3AP22qqItK1Og=; b=dWTATV4n07IsZQg2C1TR1ASMseSQInn82YDrzj1xNtwGELZz/Ew4sGNa3or6ifJjFn 42yNZtY9Tg8vK9+4cKn2fwd/tTtsZcJrGJ3UCcAEvDOv7nxsP7tN5Lqrz4g0zZCX6uPl ruvrSu6Tki66achf0UZPSuHfUJ9LM26EZvOecVZKAjd1NCyLJJGdgM+1Zzkp7FQR3Nlz rtG633IulOf4RIy4YJqd48QjUUECf1zf+YcbvevzOGJrc7/8dKPvG0mFJNOFXQiblGX+ V3vXjV6m7EUiX40ClYpKFSUlvKuwxWEZnK9iUlHxj0/MyvF30URTKARj6t7jeX298RL/ faXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=lWkzxdSUkUjYFZNUHHDx+8cNSrQYp+3AP22qqItK1Og=; b=nI0ScTyQwTLhwfJ+Njz2UFSVwg767AepYQiEw0Qu+4v7IFehOJBzJA1H0Hcy1hkjCx S8rqZtv8k0/vXo664/XeDvYxIbLWgxyBcnIaZWR96zfMlmc4AjVHe01/aXSB2AUFq7bO OqxcskZMYWP22uKI9NYW2ruEbY/2HJ6QxxRMc2IIOxSr2ytHdHskf0lIhD4RSFuprucU GraX6pq1CcqteQEcn0h8wg6DX7t11V8hzmxcr9M2yuOCx+I6fZ44he+21YmNb3NZBr/Z eFAhbef8ib7wNBXIqRq7RJ10MjotObR+1gsG8E3DEBA/tjPIB8ECbUS4qDCMnvRpcPJA j24Q== X-Gm-Message-State: AOAM5318MIFztZjEnWtnhIP6R9AvM71zuUx/i/JmDObxoR6ud7OqzRZ7 ifyudWH1T3IZRWRZaXAE0cpP04XuZ2+ihN2A0DGk+wuocQU= X-Google-Smtp-Source: ABdhPJyeehrnGx5QjBOTnHitY08W3CPsF54sxV6x4ggv4ilQk99GyOQpcoMwc1IDKx/e0XUgDga7BFmC9o/WO/h9bCE= X-Received: by 2002:a5d:6547:: with SMTP id z7mr23176372wrv.322.1600205493910; Tue, 15 Sep 2020 14:31:33 -0700 (PDT) MIME-Version: 1.0 From: ericr Date: Tue, 15 Sep 2020 15:30:57 -0600 Message-ID: Subject: filesystem checksum problems on AWS EC2 instances To: freebsd-cloud@freebsd.org X-Rspamd-Queue-Id: 4Brbyb6W87z3S6m X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=dWTATV4n; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of erobison@gmail.com designates 2a00:1450:4864:20::436 as permitted sender) smtp.mailfrom=erobison@gmail.com X-Spamd-Result: default: False [-2.64 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.87)[-0.865]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-cloud@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.98)[-0.984]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::436:from]; NEURAL_SPAM_SHORT(0.21)[0.206]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-cloud]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Sep 2020 21:31:37 -0000 Hi, I'm having disk/filesystem problems creating EC2 instances using 12.1-RELEASE (AMI ami-034de978f5d9c9b1b in us-east-1 and AMI ami-06d9854b88cba8200 in us-east-2, both from the Marketplace). I'm using t2.large instances. As a part of the instance creation, the OS gets updated from 12.1-RELEASE to 12.1-RELEASE-P9. THe AMIs use UFS. I created instances with a 200GB disk, in both us-east-1 and us-east-2, to rule out flakey hardware. Once the OS starts at first boot, I get cylinder checksum errors: Sep 1 20:50:15 freebsd kernel: xbd0: attaching as ada0 Sep 1 20:50:15 freebsd kernel: GEOM: ada0: the secondary GPT header is not in the last LBA. Sep 1 20:50:15 freebsd kernel: Trying to mount root from ufs:/dev/gpt/rootfs [rw]... Sep 1 20:50:15 freebsd kernel: random: unblocking device. Sep 1 20:50:15 freebsd kernel: UFS /dev/gpt/rootfs (/) cylinder checksum failed: cg 0, cgp: 0x9c14700e != bp: 0x27bfa3d0 ... Sep 1 20:50:15 freebsd kernel: device_attach: intsmb0 attach returned 6 Sep 1 20:50:15 freebsd kernel: UFS /dev/gpt/rootfs (/) cylinder checksum failed: cg 0, cgp: 0x9c14700e != bp: 0x27bfa3d0 Sep 1 20:50:15 freebsd syslogd: last message repeated 1 times Sep 1 20:50:15 freebsd kernel: UFS /dev/gpt/rootfs (/) cylinder checksum failed: cg 7, cgp: 0x43ed3fa1 != bp: 0xe9b0182e and from there on, I get cylinder checksum errors pretty often. Forcing fsck to run at the next boot produces messages about the cylinder groups: ** Phase 5 - Check Cyl groups CG 0: BAD CHECK-HASH 0x9c14700e vs 0x27bfa3d0 FREE BLK COUNT(S) WRONG IN SUPERBLK SALVAGE? no SUMMARY INFORMATION BAD SALVAGE? no BLK(S) MISSING IN BIT MAPS SALVAGE? no CG 7: BAD CHECK-HASH 0x43ed3fa1 vs 0xe9b0182e 43494 files, 1012718 used, 49736354 free (8570 frags, 6215973 blocks, 0.0% fragmentation) Running fsck multiple times at boot doesn't seem to fix the problems. (I edited /etc/rc.d/fsck so that anytime fsck runs, it uses "fsck -f -d -T ffs:-R -T ufs:-R" ) I've also tried creating instances with 100GB disks, I get similar issues. tunefs didn't show any obvious issues: tunefs: POSIX.1e ACLs: (-a) disabled tunefs: NFSv4 ACLs: (-N) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) enabled tunefs: soft update journaling: (-j) disabled tunefs: gjournal: (-J) disabled tunefs: trim: (-t) disabled tunefs: maximum blocks per file in a cylinder group: (-e) 4096 tunefs: average file size: (-f) 16384 tunefs: average number of files in a directory: (-s) 64 tunefs: minimum percentage of free space: (-m) 8% tunefs: space to hold for metadata blocks: (-k) 6408 tunefs: optimization preference: (-o) time tunefs: volume label: (-L) rootfs I can replicate the problem pretty easily, I've created instances half a dozen times now, they all have the same issue. So I'm stumped. I looked in the bug database and the list archives and didn't find anything like this. I can't figure out if it's a problem with the way AWS creates the disk, or if it's an installer issue with FreeBSD. (or if I'm just screwing up the creation myself) I know the default disk size is 10GB, and growfs doesn't seem to complain when it increases the size to 100 or 200GB. Is there some size limit on EBS disks? Also, does anyone have docs on how to create a custom AMI that'll create a ZFS root filesystem and install the OS to it? I can't seem to find AMIs for 12.1-RELEASE that use it. Thanks!