From owner-freebsd-cloud@freebsd.org Tue Sep 22 00:01:14 2020 Return-Path: Delivered-To: freebsd-cloud@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 57A24422781 for ; Tue, 22 Sep 2020 00:01:14 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BwM0T2ctLz48BK for ; Tue, 22 Sep 2020 00:01:13 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id 08M02YQ3054819; Mon, 21 Sep 2020 17:02:34 -0700 (PDT) (envelope-from mckusick@mckusick.com) Message-Id: <202009220002.08M02YQ3054819@chez.mckusick.com> From: Kirk McKusick To: Colin Percival Subject: Re: Fwd: filesystem checksum problems on AWS EC2 instances cc: ericr , freebsd-cloud@freebsd.org X-URL: http://WWW.McKusick.COM/ Reply-To: Kirk McKusick In-reply-to: <01000174b15f88a5-8de101fb-8f1b-4adb-bd7e-3c752bf86d61-000000@email.amazonses.com> Comments: In-reply-to Colin Percival message dated "Mon, 21 Sep 2020 15:54:22 -0000." MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <54817.1600732954.1@chez.mckusick.com> Content-Transfer-Encoding: quoted-printable Date: Mon, 21 Sep 2020 17:02:34 -0700 X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,MISSING_MID, UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on chez.mckusick.com X-Rspamd-Queue-Id: 4BwM0T2ctLz48BK X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of mckusick@mckusick.com has no SPF policy when checking 70.36.157.235) smtp.mailfrom=mckusick@mckusick.com X-Spamd-Result: default: False [-0.27 / 15.00]; HAS_REPLYTO(0.00)[mckusick@mckusick.com]; ARC_NA(0.00)[]; REPLYTO_EQ_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEFALL_USER(0.00)[mckusick]; NEURAL_HAM_LONG(-0.97)[-0.966]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[mckusick.com]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.40)[0.404]; AUTH_NA(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.61)[-0.606]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:46375, ipnet:70.36.128.0/19, country:US]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; MAILMAN_DEST(0.00)[freebsd-cloud]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Sep 2020 00:01:14 -0000 > Date: Fri, 18 Sep 2020 20:00:49 -0700 > From: Colin Percival > Subject: Re: filesystem checksum problems on AWS EC2 instances > To: ericr , freebsd-cloud@freebsd.org, > Kirk McKusick > = > [Adding Kirk since this seems like a UFS issue...] > = > On 2020-09-16 15:15, ericr wrote: >> On Tue, Sep 15, 2020 at 6:24 PM Colin Percival w= rote: >>> On 2020-09-15 14:30, ericr wrote: >>>> Sep 1 20:50:15 freebsd kernel: UFS /dev/gpt/rootfs (/) >>>> cylinder checksum failed: cg 0, cgp: 0x9c14700e !=3D bp: 0x27bfa3d0 >>>> Sep 1 20:50:15 freebsd syslogd: last message repeated 1 >>> times >>>> Sep 1 20:50:15 freebsd kernel: UFS /dev/gpt/rootfs (/) >>>> cylinder checksum failed: cg 7, cgp: 0x43ed3fa1 !=3D bp: 0xe9b0182e >>>> >>>> and from there on, I get cylinder checksum errors pretty often. >>> >>> Do you get this if you launch from the non-Marketplace AMIs listed in = the >>> release announcement? >>> https://www.freebsd.org/releases/12.1R/announce.html >> = >> = >> Yes. I just tried both of these AMI's from the release notes: >> us-east-1 region: ami-0de268ac2498ba33d >> us-east-2 region: ami-0a44f10b2c6deb365 >> = >> I got the same errors. > = > I've managed to reproduce this, with a filesystem which I've > verified is clean (at least, which passes fsck) before resizing > up to ~ 200 GB: > = >> root@freebsd:/usr/home/ec2-user # fsck_ufs /dev/nvd1p2 = >> ** /dev/nvd1p2 >> ** Last Mounted on /releng/12-amd64-GENERIC-release/usr/obj/usr/src/amd= 64.amd64/release/cw-ec2/new >> ** Phase 1 - Check Blocks and Sizes >> ** Phase 2 - Check Pathnames >> ** Phase 3 - Check Connectivity >> ** Phase 4 - Check Reference Counts >> ** Phase 5 - Check Cyl groups >> 25701 files, 758977 used, 229774 free (9654 frags, 27515 blocks, 1.0% f= ragmentation) >> = >> ***** FILE SYSTEM IS CLEAN ***** >> root@freebsd:/usr/home/ec2-user # gpart recover /dev/nvd1 >> nvd1 recovered >> root@freebsd:/usr/home/ec2-user # gpart resize -i 2 /dev/nvd1 >> nvd1p2 resized >> root@freebsd:/usr/home/ec2-user # growfs -y /dev/nvd1p2 >> super-block backups (for fsck_ffs -b #) at: >> [snip] >> root@freebsd:/usr/home/ec2-user # fsck_ufs /dev/nvd1p2 >> ** /dev/nvd1p2 >> ** Last Mounted on = >> ** Phase 1 - Check Blocks and Sizes >> ** Phase 2 - Check Pathnames >> ** Phase 3 - Check Connectivity >> ** Phase 4 - Check Reference Counts >> ** Phase 5 - Check Cyl groups >> CG 0: BAD CHECK-HASH 0x9c14700e vs 0xc9441f74 >> SUMMARY INFORMATION BAD >> SALVAGE? [yn] n >> = >> CG 7: BAD CHECK-HASH 0xad168305 vs 0x74ba48a >> 25701 files, 758977 used, 50019285 free (9661 frags, 6251203 blocks, 0.= 0% fragmentation) >> = >> ***** FILE SYSTEM MARKED DIRTY ***** >> = >> ***** PLEASE RERUN FSCK ***** > = > This seems like a bug in UFS and/or growfs, but I'm not familiar enough > with either to say any more. > = > Kirk, are you aware of any issues on FreeBSD 12.1-RELEASE which can caus= e > cylinder checksum errors after growfs? (On amd64 if it matters.) If it > would help I can provide you with SSH access to an affected EC2 instance= . > = > -- = > Colin Percival > Security Officer Emeritus, FreeBSD | The power to serve > Founder, Tarsnap | www.tarsnap.com | Online backups for the truly parano= id I have managed to reproduce a similar problem in one of my rather ancient 12.0 bhyve images that I have lying around: FreeBSD 12.0-STABLE (GENERIC) #5 r350458M: Sat Oct 26 21:18:51 UTC 2019 The follow patch fixes it in that instance. Could you please try this in the EC2 instance and see if it also resolves your problem. Kirk McKusick =3D-=3D-=3D Index: sbin/growfs/growfs.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sbin/growfs/growfs.c (revision 365971) +++ sbin/growfs/growfs.c (working copy) @@ -572,6 +572,7 @@ updjcg(int cylno, time_t modtime, int fsi, int fso if (sblock.fs_magic =3D=3D FS_UFS1_MAGIC) acg.cg_old_ncyl =3D sblock.fs_old_cpg; = + cgckhash(&acg); wtfs(fsbtodb(&sblock, cgtod(&sblock, cylno)), (size_t)sblock.fs_cgsize, (void *)&acg, fso, Nflag); DBG_PRINT0("jcg written\n"); @@ -947,6 +948,7 @@ updcsloc(time_t modtime, int fsi, int fso, unsigne * Now write the former cylinder group containing the cylinder * summary back to disk. */ + cgckhash(&acg); wtfs(fsbtodb(&sblock, cgtod(&sblock, ocscg)), (size_t)sblock.fs_cgsize, (void *)&acg, fso, Nflag); DBG_PRINT0("oscg written\n"); @@ -1039,6 +1041,7 @@ updcsloc(time_t modtime, int fsi, int fso, unsigne * Write the new cylinder group containing the cylinder summary * back to disk. */ + cgckhash(&acg); wtfs(fsbtodb(&sblock, cgtod(&sblock, ncscg)), (size_t)sblock.fs_cgsize, (void *)&acg, fso, Nflag); DBG_PRINT0("nscg written\n"); From owner-freebsd-cloud@freebsd.org Tue Sep 22 00:46:37 2020 Return-Path: Delivered-To: freebsd-cloud@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 231C8423C13 for ; Tue, 22 Sep 2020 00:46:37 +0000 (UTC) (envelope-from 01000174b346c7ab-3f67a9cc-d92f-4092-8537-5434250138c4-000000@amazonses.com) Received: from a8-56.smtp-out.amazonses.com (a8-56.smtp-out.amazonses.com [54.240.8.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BwN0q6bNNz4DW2 for ; Tue, 22 Sep 2020 00:46:35 +0000 (UTC) (envelope-from 01000174b346c7ab-3f67a9cc-d92f-4092-8537-5434250138c4-000000@amazonses.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ae7m2yrxjw65l2cqdpjxuucyrvy564tn; d=tarsnap.com; t=1600735594; h=Subject:To:Cc:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding; bh=4XKm3+anLvFOiFl9MzDZYrej/GbdOunM5dAFqZxipB8=; b=E/HuBCI85iYlmQNkCQp1MEx+aMy3nUBjRERatpGMMwwCZkl1cdW0QfC1Ddfm3z2h zVsikKvbC85q+ztauD09tmCvfuuzWEWw/vff9l/vbD7/mJlu4dY6FpN4McxjhCbQ75G N0THEvxbI6hUKbj8E+NiV+y2kOwNqoFAg1onxTxA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=224i4yxa5dv7c2xz3womw6peuasteono; d=amazonses.com; t=1600735594; h=Subject:To:Cc:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding:Feedback-ID; bh=4XKm3+anLvFOiFl9MzDZYrej/GbdOunM5dAFqZxipB8=; b=IfAx4y/QJa8cdcpCF2G8couMHE5zZj+TWf4y+mPiHcK96I1o8sb2AAU/KvIa3Anl yX0NWSZ/UlzgIauVmpjt5zVDtLGBmO9bJvgyyLvrNnFyf1r5m3vzGuevtHf7gM0l/II JNRov8CiGRJFbJsY57sNDrrE+3blXr1TbPLKyAOU= Subject: Re: Fwd: filesystem checksum problems on AWS EC2 instances To: Kirk McKusick Cc: ericr , freebsd-cloud@freebsd.org References: <202009220002.08M02YQ3054819@chez.mckusick.com> From: Colin Percival Autocrypt: addr=cperciva@tarsnap.com; prefer-encrypt=mutual; keydata= mQGhBElrAAcRBACDfDys4ZtK+ErCJ1HAzYeteKpm3OEsvT/49AjUTLihkF79HhIKrCQU+1KC zv7BwHCMLb6hq30As9L7iFKG7n5QFLFC4Te/VcITUnWHMG/c3ViLOfJGvi+9/nOEHaM1dVJY D6tEp5yM1nHmVQpo9932j4KGuGFR0LhOK5IHXOSfGwCgxSFDPdgxe2OEjWxjGgY+oV3EafcD +JROXCTjlcQiG/OguQH4Vks3mhHfFnEppLxTkDuYgHZQiUtpcT9ssH5khgqoTyMar05OUdAj ZIhNbWDh4LgTj+7ZmvLhXT5Zxw8LX9d7T36aTB8XDQSenDqEtinMWOb0TCBBLbsB8EFG1WTT ESbZci9jJS5yhtktuZoY/eM8uXMD/3k4FWFO80VRRkELSp+XSy/VlSQjyi/rhl2nQq/oOA9F oJbDaB0yq9VNhxP+uFBzBWSqeIX0t1ZWLtNfVFr4TRP5hihI5ICrg/0OpqgisKsU2NFe9xyO hyJLYmfD8ebpDJ/9k30C7Iju9pVrwLm1QgS4S2fqJRcR+U4WbjvP7CgStCVDb2xpbiBQZXJj aXZhbCA8Y3BlcmNpdmFAdGFyc25hcC5jb20+iGEEExECACEFAklrALYCGwMHCwkIBwMCAQQV AggDBBYCAwECHgECF4AACgkQOM7KaQxqam6/igCgn+z2k3V5ggNppmWrZstt1U2lugsAoL7L wS9V9yLtil3oWmHtwpUqYruEuQINBElrAAcQCAD3ZLMIsP4CIDoJORg+YY0lqLVBgcnF7pFb 4Uy2+KvdWofN+DKH61rZLjgXXkNE9M4EQC1B4lGttBP8IY2gs41y3AUogGdyFbidq99rCBz7 LTsgARHwFxZoaHmXyiZLEU1QZuMqwPZV1mCviRhN5E3rRqYNXVcrnXAAuhBpvNyj/ntHvcDN 2/m+ochiuBYueU4kX3lHya7sOj+mTsndcWmQ9soOUyr8O0r/BG088bMn4qqtUw4dl5/pglXk jbl7uOOPinKf0WVd2r6M0wLPJCD4NPHrCWRLLLAjwfjrtoSRvXxDbXhCdgGBa72+K8eYLzVs hgq7tJOoBWzjVK6XRxR7AAMGB/9Mo3iJ2DxqDecd02KCB5BsFDICbJGhPltU7FwrtbC7djSb XUrwsEVLHi4st4cbdGNCWCrp0BRezXZKohKnNAPFOTK++ZfgeKxrV2sJod+Q9RILF86tQ4XF 7A7Yme5hy92t/WgiU4vc/fWbgP8gV/19f8nunaT2E9NSa70mZFjZNu4iuwThoUUO5CV3Wo0Y UISsnRK8XD1+LR3A2qVyLiFRwh/miC1hgLFCTGCQ3GLxZeZzIpYSlGdQJ0L5lixW5ZQD9r1I 8i/8zhE6qRFAM0upUMI3Gt1Oq2w03DiXrZU0Fu/R8Rm8rlnkQKA+95mRTUq1xL5P5NZIi4gJ Z569OPMFiEkEGBECAAkFAklrAAcCGwwACgkQOM7KaQxqam41igCfbaldnFTu5uAdrnrghESv EI3CAo8AoLkNMks1pThl2BJNRm4CtTK9xZeH Message-ID: <01000174b346c7ab-3f67a9cc-d92f-4092-8537-5434250138c4-000000@email.amazonses.com> Date: Tue, 22 Sep 2020 00:46:34 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: <202009220002.08M02YQ3054819@chez.mckusick.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-SES-Outgoing: 2020.09.22-54.240.8.56 Feedback-ID: 1.us-east-1.Lv9FVjaNvvR5llaqfLoOVbo2VxOELl7cjN0AOyXnPlk=:AmazonSES X-Rspamd-Queue-Id: 4BwN0q6bNNz4DW2 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tarsnap.com header.s=ae7m2yrxjw65l2cqdpjxuucyrvy564tn header.b=E/HuBCI8; dkim=pass header.d=amazonses.com header.s=224i4yxa5dv7c2xz3womw6peuasteono header.b=IfAx4y/Q; dmarc=pass (policy=none) header.from=tarsnap.com; spf=pass (mx1.freebsd.org: domain of 01000174b346c7ab-3f67a9cc-d92f-4092-8537-5434250138c4-000000@amazonses.com designates 54.240.8.56 as permitted sender) smtp.mailfrom=01000174b346c7ab-3f67a9cc-d92f-4092-8537-5434250138c4-000000@amazonses.com X-Spamd-Result: default: False [-1.08 / 15.00]; FORGED_MUA_THUNDERBIRD_MSGID_UNKNOWN(2.50)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[tarsnap.com:s=ae7m2yrxjw65l2cqdpjxuucyrvy564tn,amazonses.com:s=224i4yxa5dv7c2xz3womw6peuasteono]; NEURAL_HAM_MEDIUM(-1.01)[-1.006]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:54.240.0.0/18]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.03)[-1.032]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[tarsnap.com:+,amazonses.com:+]; DMARC_POLICY_ALLOW(-0.50)[tarsnap.com,none]; RCVD_IN_DNSWL_NONE(0.00)[54.240.8.56:from]; NEURAL_HAM_SHORT(-0.84)[-0.843]; FORGED_SENDER(0.30)[cperciva@tarsnap.com,01000174b346c7ab-3f67a9cc-d92f-4092-8537-5434250138c4-000000@amazonses.com]; RCVD_COUNT_ZERO(0.00)[0]; RWL_MAILSPIKE_POSSIBLE(0.00)[54.240.8.56:from]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:14618, ipnet:54.240.8.0/21, country:US]; FROM_NEQ_ENVFROM(0.00)[cperciva@tarsnap.com,01000174b346c7ab-3f67a9cc-d92f-4092-8537-5434250138c4-000000@amazonses.com]; MAILMAN_DEST(0.00)[freebsd-cloud]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org] X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Sep 2020 00:46:37 -0000 On 2020-09-21 17:02, Kirk McKusick wrote: > I have managed to reproduce a similar problem in one of my rather > ancient 12.0 bhyve images that I have lying around: > > FreeBSD 12.0-STABLE (GENERIC) #5 r350458M: Sat Oct 26 21:18:51 UTC 2019 > > The follow patch fixes it in that instance. Could you please try this > in the EC2 instance and see if it also resolves your problem. This fixes growfs in my tests on 12.2-BETA2. Please commit (and MFC before the release)! -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid