From owner-freebsd-fs@freebsd.org Wed Jan 27 18:15:38 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AE974525259 for ; Wed, 27 Jan 2021 18:15:38 +0000 (UTC) (envelope-from stevenschlansker@gmail.com) Received: from mail-io1-xd34.google.com (mail-io1-xd34.google.com [IPv6:2607:f8b0:4864:20::d34]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DQsGd5Kxzz4r8C for ; Wed, 27 Jan 2021 18:15:37 +0000 (UTC) (envelope-from stevenschlansker@gmail.com) Received: by mail-io1-xd34.google.com with SMTP id y19so2850831iov.2 for ; Wed, 27 Jan 2021 10:15:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=AVLbdv6Q/tlFqKEWoDW2zRDxO04tRPHGjw5ZapdNiZ0=; b=t2BIyFY3g32XqbueJC3pGMI90yhp3TYgBJtDETtHKMbOmKGYFNh3ER7hPw11xXf5nO DYYkAESIWOU42N+B9ruGCcpAKBBCkHlx3F0EFvDcJYc0f23Q4gMNZcKdLlo1i18KZBDI 5b9zXKZG/WIZHiRGJJcaY4bMZ7CQkHBoasoXh5O2KOZD+V/pkr5sr9kkW38ZaSmVm4ql +VURJLnEPqH/g1s/aBvWbe/fTpvna6qBgW2TSC8f/aUPS4ATEDoQxH9zaB6rt0kmWf8o 0KGudyvUakL5YjFR5hS0Hu/wkCbU4UHlbSLvW4Zu8VHPS9t/NZ0O749wYEteOWnknOwW 0+VQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=AVLbdv6Q/tlFqKEWoDW2zRDxO04tRPHGjw5ZapdNiZ0=; b=VE1QhdzW+4y7vofK4oQG5FbNSm9dkos1gPZXuwC1PUPCLzQEtSMovzP3B7q4Pptlml HM/SpuTk4m+izUcF290PWSqVp9FcTveJURKnyhLtc9lQ6V65kOoxKA9BZtArHJcPu3W9 V/rYdcBAPkS8SFkwuJvpo7T8h2M8SUK+Aci1q5oo82AGOX/v0HuJS1M1c9ZzN0Jt5/ZT EVrWjrSCJinr/SJ+dp4yxIfgTDMiF/NAZufPL+p7snpfNRZOalIiN2z+JbhiNRryibPy A0I1Sx9jhhdh5f3a/PEga9vKBg4Rv6xkdl1AhdeDWwr3aOjbxQUOUgSZ5djVXZUkPThd s4xg== X-Gm-Message-State: AOAM531ZSTFwmQ9t5hQTxbGHzY0eFstBGWY65PDR0BuUYyGPdkOhYfU1 zJad8KKd/w8L8zd6HCRCjipjVvOu9KO76yPiiVAQ6MFAzEE= X-Google-Smtp-Source: ABdhPJzSHbDIVvOJvpPhkw9fFaVZUbBhCz3Pxuwm4jMAapbF2cz8/1M5u1yxstgKOUKIN94nxMgo7dQBH930KrqA3nE= X-Received: by 2002:a6b:6a0e:: with SMTP id x14mr8370281iog.57.1611771336179; Wed, 27 Jan 2021 10:15:36 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Steven Schlansker Date: Wed, 27 Jan 2021 10:15:24 -0800 Message-ID: Subject: Re: persistent integer divide fault panic in zfs_rmnode To: freebsd-fs@freebsd.org X-Rspamd-Queue-Id: 4DQsGd5Kxzz4r8C X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=t2BIyFY3; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of stevenschlansker@gmail.com designates 2607:f8b0:4864:20::d34 as permitted sender) smtp.mailfrom=stevenschlansker@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-0.999]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::d34:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::d34:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::d34:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2021 18:15:38 -0000 Does anybody have any suggestions as to what I can try next regarding this panic? At this point the only path forward I see is to declare the zpool corrupt and attempt to move all the data off, destroy, and migrate back, and hope the recreated pool does not tickle this bug. That would be a pretty disappointing end to a long fatal-problem-free run with ZFS. Thanks, Steven On Fri, Jan 8, 2021 at 3:41 PM Steven Schlansker wrote: > Hi freebsd-fs, > > I have a 8-way raidz2 system running FreeBSD 12.2-RELEASE-p1 GENERIC > Approximately since upgrading to FreeBSD 12.2-RELEASE, I receive a nasty > panic when trying to unlink any of a large number of files. > > Fatal trap 18: integer divide fault while in kernel mode > > > The pool reports as healthy: > > pool: universe > state: ONLINE > status: One or more devices are configured to use a non-native block size. > Expect reduced performance. > action: Replace affected devices with devices that support the > configured block size, or migrate data to a properly configured > pool. > scan: resilvered 416M in 0 days 00:08:35 with 0 errors on Thu Jan 7 > 02:16:03 2021 > When some files are unlinked, the system panics with a partial backtrace > of: > > #6 0xffffffff82a148ce at zfs_rmnode+0x5e > #7 0xffffffff82a35612 at zfs_freebsd_reclaim+0x42 > #8 0xffffffff812482db at VOP_RECLAIM_APV+0x7b > #9 0xffffffff80c8e376 at vgonel+0x216 > #10 0xffffffff80c8e9c5 at vrecycle+0x45 > > I captured a dump, and using kgdb extracted a full backtrace, and filed it > as https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=250784 > > #8 0xffffffff82963725 in get_next_chunk (dn=0xfffff804325045c0, > start=, minimum=0, l1blks=) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:721 > warning: Source file is more recent than executable. > 721 (roundup(*start, iblkrange) - (minimum / iblkrange * > iblkrange)) / > (kgdb) list > 716 * L1 blocks in this range have data. If we can, we use > this > 717 * worst case value as an estimate so we can avoid having > to look > 718 * at the object's actual data. > 719 */ > 720 uint64_t total_l1blks = > 721 (roundup(*start, iblkrange) - (minimum / iblkrange * > iblkrange)) / > 722 iblkrange; > 723 if (total_l1blks <= maxblks) { > 724 *l1blks = total_l1blks; > 725 *start = minimum; > (kgdb) print iblkrange > $1 = 0 > (kgdb) print minimum > $2 = 0 > > It looks like it is attempting to compute 0 / 0, causing the panic. > > How can I restore my zpool to a working state? Thank you for any > assistance. >