From nobody Fri Nov 5 17:03:35 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 00AFE183FAEF for ; Fri, 5 Nov 2021 17:03:41 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: from mail-vk1-xa30.google.com (mail-vk1-xa30.google.com [IPv6:2607:f8b0:4864:20::a30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hm6KS4zgkz4Rcg for ; Fri, 5 Nov 2021 17:03:40 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: by mail-vk1-xa30.google.com with SMTP id bc10so4935559vkb.1 for ; Fri, 05 Nov 2021 10:03:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=YElTuuUyZQ9lhs/B6o8hdLBMTagP6HXunYVrb6L1aZc=; b=ZPKCiVBevoLD8psDCk1krobLlqsWHqAM2IFJP/M+u1xiQzuclvpnDKv78N/Gns9cP6 LNLlYnl32rdPSXH2VDc5INHaADL7HAb+m4vJGj6GrhATDIo5aW197xkzjaL/u4dgHnkU ninYR9/O+gLORddd1pzjFo3tiLYnchvlCmm9r2fCA8POVDTyICdZbSqbGcbrg38oU6W6 4gdBCZdo9dvo0RnZx6O5Bw7V3oxENm57lDOMeaZ8/fEXW9MVA8Rv2pPBWucAJFugxiHj 4hXH/Nkx/uaCqSxS0Ispxa8uPOfLYqjQsZ62M8tl2+duFurTn7nOjG5BrJDhujEwXTn3 15+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=YElTuuUyZQ9lhs/B6o8hdLBMTagP6HXunYVrb6L1aZc=; b=qoLtJsk15NuVszlq2yQM30QjTY3+q4r16Fl6I0t0hoOProGYlwgifirW7fplnEYreS iNkWHi3W2jWNR3C6KCS6NMTiEYjl6jQazk6aa/YXTLtlTJlZiOhRJ13mNW0bKI95FKXT b4rOlgTyYcDgcyepfyQU9HY43wUgIKI+FjsSAN5hTrjqCf9S768mldTixNy3kvRMghpR OY6jOuJk7tlubR2KCtfc9uDnnmrPxgozOcytQ4toUS2qaa6vZ91UMxMidfCh6IQ4PlqA KPAfwdcPDM8qAwK6ZRFGCmBcgEGqcXA9+wsHO1Ze073C4RmoIW3EHgR4ioQDJn3E2Xpi 3TGw== X-Gm-Message-State: AOAM533A+LNNoYRf4Uus82ZBPpQIp6btbXOBuOQyAI8Mfwvin808NB+m geMX10LguwhkK2PxiTuuNHWu73pXzuAojvT5BlnIJMGfLko= X-Google-Smtp-Source: ABdhPJz0Gj6ImZAf6E5uTQtbKJWzqcGT+l7gulaOFJzhm3gIGDx9zWgMfVI6LqmvCtlGoAFqnDILZUcVrsPCSCw+xPs= X-Received: by 2002:a1f:cf47:: with SMTP id f68mr45968837vkg.10.1636131820205; Fri, 05 Nov 2021 10:03:40 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <20211105173935.7aa53269@fabiankeil.de> <86999084-7007-4F08-A4C4-4A835A7E1C78@distal.com> In-Reply-To: <86999084-7007-4F08-A4C4-4A835A7E1C78@distal.com> From: Rich Date: Fri, 5 Nov 2021 13:03:35 -0400 Message-ID: Subject: Re: ZFS operations hanging, but no visible errors? To: Chris Ross Cc: freebsd-fs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4Hm6KS4zgkz4Rcg X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; TAGGED_RCPT(0.00)[freebsd]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N That would, indeed, print things as they happen, so if nothing's happening,= ... I believe "sysctl kstat.zfs.misc.dbgmsg" would give you all currently logged ones. - Rich On Fri, Nov 5, 2021 at 1:00 PM Chris Ross wrote: > > > > > On Nov 5, 2021, at 12:39, Fabian Keil wr= ote: > > You can check with "procstat -kk" where the hanging > > processes are stuck. > > Running prostate -kk on the rsync that was hung, then killed, then SIGKIL= L=E2=80=99d shows: > > procstat -kk 35220 > PID TID COMM TDNAME KSTACK > 35220 102499 rsync - mi_switch+0xc1 _slee= p+0x1cb vm_wait_doms+0xe2 vm_wait_domain+0x51 vm_domain_alloc_fail+0x86 vm_= page_alloc_domain_after+0x7e uma_small_alloc+0x58 keg_alloc_slab+0xba zone_= import+0xee zone_alloc_item+0x6f abd_alloc_chunks+0x61 abd_alloc+0x102 arc_= hdr_alloc_abd+0xb0 arc_hdr_alloc+0x11e arc_read+0x4f4 dbuf_issue_final_pref= etch+0x108 dbuf_prefetch_impl+0x3d0 dmu_zfetch+0x558 > > > > > > The output of: > > dtrace -qn 'zfs-dbgmsg {printf("%Y: %s\n", walltimestamp, stringof(arg0= ))}' > > could be useful as well. > > I=E2=80=99ve run > > % sudo dtrace -qn 'zfs-dbgmsg {printf("%Y: %s\n", walltimestamp, stringof= (arg0))}=E2=80=99 > > But, it=E2=80=99s yielding no output. Is that normal because it outputs = information about operations (which aren=E2=80=99t happening), or should it= =E2=80=99s output have been immediate? > > - Chris