From nobody Fri Nov 5 15:59:26 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id C24ED184A792 for ; Fri, 5 Nov 2021 15:59:32 +0000 (UTC) (envelope-from wfc@mintsol.com) Received: from scully.mintsol.com (scully.mintsol.com [199.182.77.206]) by mx1.freebsd.org (Postfix) with ESMTP id 4Hm4vS49d6z3HdZ for ; Fri, 5 Nov 2021 15:59:32 +0000 (UTC) (envelope-from wfc@mintsol.com) Received: from mintsol.com (officecc.mintsol.com [96.85.114.33]) by scully.mintsol.com with esmtp; Fri, 05 Nov 2021 11:59:26 -0400 id 00BA88D1.00000000618554DE.0000149F Received: from localhost (localhost [127.0.0.1]) (IDENT: uid 1002) by mintsol.com with esmtp; Fri, 05 Nov 2021 11:59:26 -0400 id 000001FC.618554DE.000177CA Date: Fri, 5 Nov 2021 11:59:26 -0400 (EDT) From: Walter Cramer To: Chris Ross cc: freebsd-fs Subject: Re: ZFS operations hanging, but no visible errors? In-Reply-To: Message-ID: <20211105115234.H95319@mulder.mintsol.com> References: List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="0-2138392286-1636127895=:95319" Content-ID: <20211105115818.U95319@mulder.mintsol.com> X-Rspamd-Queue-Id: 4Hm4vS49d6z3HdZ X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; TAGGED_RCPT(0.00)[freebsd]; REPLY(-4.00)[] X-Spam: Yes X-ThisMailContainsUnwantedMimeParts: N This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --0-2138392286-1636127895=:95319 Content-Type: TEXT/PLAIN; CHARSET=X-UNKNOWN; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE Content-ID: <20211105115818.E95319@mulder.mintsol.com> On Fri, 5 Nov 2021, Chris Ross wrote: > > > On Nov 5, 2021, at 11:40, Chris Ross wrote: > > Hey there. I have a server running FreeBSD 13.0-RELEASE, with a large ZF= S zpool. I have a UFS mirror on hardware raid, then a bunch of JBOD disks = in a pool. I recently added a new vdev to this pool, which may or may not = be related. > > Today, I started an rsync of a large (100GB) file from the pool to anothe= r host. After a while (7%), it seemed no progres was being made. I tried = to kill the rsync, which didn=E2=80=99t exit, or suspend. Now anything tha= t touches the pool seems to hang. But, the system is otherwise functional,= console shows no issues, the controller (via out-of-band management interf= ace) shows all disks as having no errors or issues. > > Any idea what I should be looking for, and if there=E2=80=99s any way to = recover it without reboot? Another operation I had running was in the midd= le of writing data I=E2=80=99d rather not lose progress on, which is why I = haven=E2=80=99t rebooted yet. > > - Chris > > Ps, apologies for the rapid secondary email, but I forgot to mention some= thing important. A =E2=80=9Czpool status tank=E2=80=9D is also hanging. S= o, it=E2=80=99s not just FS operations that are stuck. > Any chance that you have output from an earlier `zpool status`, or even=20 `zpool list`, to give us a better sense of your situation? Might `dmesg -a` show any relevant errors? (FWIW, I usually have a daily cron job log a few things like `zpool=20 status` to /var/backups/zfs, to help with situations like this.) --0-2138392286-1636127895=:95319--