From nobody Fri Nov 5 16:59:47 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 076E9183CE24 for ; Fri, 5 Nov 2021 16:59:52 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hm6F35MHwz3vYY for ; Fri, 5 Nov 2021 16:59:51 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail.distal.com (pool-108-48-165-176.washdc.fios.verizon.net [108.48.165.176]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 5 Nov 2021 16:59:50 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1005::15]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 351d76a9 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO) for ; Fri, 5 Nov 2021 12:59:48 -0400 (EDT) From: Chris Ross Content-Type: multipart/alternative; boundary="Apple-Mail=_9D4F7EDC-2107-491F-8DD4-205632C97D87" List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.20.0.1.32\)) Subject: Re: ZFS operations hanging, but no visible errors? Date: Fri, 5 Nov 2021 12:59:47 -0400 References: <20211105173935.7aa53269@fabiankeil.de> To: freebsd-fs In-Reply-To: <20211105173935.7aa53269@fabiankeil.de> Message-Id: <86999084-7007-4F08-A4C4-4A835A7E1C78@distal.com> X-Mailer: Apple Mail (2.3693.20.0.1.32) X-Rspamd-Queue-Id: 4Hm6F35MHwz3vYY X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[freebsd] X-ThisMailContainsUnwantedMimeParts: Y --Apple-Mail=_9D4F7EDC-2107-491F-8DD4-205632C97D87 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Nov 5, 2021, at 12:39, Fabian Keil = wrote: > You can check with "procstat -kk" where the hanging > processes are stuck. Running prostate -kk on the rsync that was hung, then killed, then = SIGKILL=E2=80=99d shows: procstat -kk 35220 PID TID COMM TDNAME KSTACK 35220 102499 rsync - mi_switch+0xc1 = _sleep+0x1cb vm_wait_doms+0xe2 vm_wait_domain+0x51 = vm_domain_alloc_fail+0x86 vm_page_alloc_domain_after+0x7e = uma_small_alloc+0x58 keg_alloc_slab+0xba zone_import+0xee = zone_alloc_item+0x6f abd_alloc_chunks+0x61 abd_alloc+0x102 = arc_hdr_alloc_abd+0xb0 arc_hdr_alloc+0x11e arc_read+0x4f4 = dbuf_issue_final_prefetch+0x108 dbuf_prefetch_impl+0x3d0 = dmu_zfetch+0x558 >=20 > The output of: > dtrace -qn 'zfs-dbgmsg {printf("%Y: %s\n", walltimestamp, = stringof(arg0))}' > could be useful as well. I=E2=80=99ve run=20 % sudo dtrace -qn 'zfs-dbgmsg {printf("%Y: %s\n", walltimestamp, = stringof(arg0))}=E2=80=99 But, it=E2=80=99s yielding no output. Is that normal because it outputs = information about operations (which aren=E2=80=99t happening), or should = it=E2=80=99s output have been immediate? - Chris= --Apple-Mail=_9D4F7EDC-2107-491F-8DD4-205632C97D87--