From nobody Thu Nov 11 21:49:21 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A0A6718482DB for ; Thu, 11 Nov 2021 21:49:27 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4HqwNR2Gc9z3Kv3 for ; Thu, 11 Nov 2021 21:49:27 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail.distal.com (pool-108-48-165-176.washdc.fios.verizon.net [108.48.165.176]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Thu, 11 Nov 2021 21:49:24 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1004::ab]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id fe44d9e4 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Thu, 11 Nov 2021 16:49:23 -0500 (EST) Content-Type: multipart/alternative; boundary="Apple-Mail=_A7A226AD-D288-4FA6-8B73-D4E088DDCBCF" List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.20.0.1.32\)) Subject: Re: swap_pager: cannot allocate bio From: Chris Ross In-Reply-To: Date: Thu, 11 Nov 2021 16:49:21 -0500 Cc: freebsd-fs Message-Id: <4E5511DF-B163-4928-9CC3-22755683999E@distal.com> References: <9FE99EEF-37C5-43D1-AC9D-17F3EDA19606@distal.com> <09989390-FED9-45A6-A866-4605D3766DFE@distal.com> To: ronald-lists@klop.ws X-Mailer: Apple Mail (2.3693.20.0.1.32) X-Rspamd-Queue-Id: 4HqwNR2Gc9z3Kv3 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[freebsd] X-ThisMailContainsUnwantedMimeParts: Y --Apple-Mail=_A7A226AD-D288-4FA6-8B73-D4E088DDCBCF Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Nov 11, 2021, at 13:50, Ronald Klop via freebsd-fs = wrote: >=20 > Can you press ctrl-t on the hanging process? That should print the = stacktrace indicating where it is waiting on. So, I rebooted the machine this morning, but now have [tried to] log = into it to check on it and find that an ssh connection doesn=E2=80=99t = result in a shell. I logged into the console, tried to start a = =E2=80=9Cscreen=E2=80=9D to get more prompts, and it hung. Ctrl-T on = that shows (after running a console screen-capture through OCR, and hand = correction, so may not be 100%): root@host:~ # screen load: 0.07 cmd: csh 56116 [vmwait] 35.00r 0.00u 0.01s 0% 3984k mi_switch+0xc1 _sleep+0x1cb vm_wait_doms+0xe2 vm_wait_domain+0x51 = vm_domain_alloc_fail+0x86 vm_page_alloc_domain_after+0x7e = uma_small_alloc+0x58 keg_alloc_slab+0xba zone_import+0xee = zone_alloc_item+0x6f malloc+0x5d sigacts_alloc+0x1c fork1+0x9fb = sys_fork+0x54 amd64_syscall+0x10c fast_syscall_common+0xf8=20 As before, ps and even mount and df work here on console. But, a = =E2=80=9Czpool status tank=E2=80=9D will hang as before. A Ctrl+D on it root@host:~ # screen load: 0.00 cmd: zpool 62829 [aw.aew_cv] 37.89r 0.00u 0.00s 0% 6976k mi_switch+0xc1 _cv_wait+0xf2 arc_wait_for_eviction+0x14a = arc_get_data_impl+0xdb arc_hdr_alloc_abd+0xa6 arc_hdr_alloc+0x11e = arc_read+0x4f4 dbuf_read+0xc08 dmu_buf_hold+0x46 zap_lookup_norm+0x35 = zap_contains+0x26 vdev_rebuild_get_stats+0xac vdev_config_generate+0x3e9 = vdev_config_generate+0x74f spa_config_generate+0x2a2 = spa_open_common+0x25c spa_get_stats+0x4e zfs_ioc_pool_stats+0x22 > On Nov 11, 2021, at 14:10, Dave Cottlehuber <> wrote: >=20 > Grab output of =E2=80=98procstat-kk=E2=80=99 and see if this is = similar to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258208 a = few more prods might get this one addressed! procstat -kk 62829 yields the same as above. Which I presume is = expected, I=E2=80=99d just never used procstate -kk before. Unfortunately, I can=E2=80=99t tell if this is sufficiently similar to = bug 258208. A different ZFS operation is happening here, so the calls = behind my zpool status are different. The other non-zfs stat above = (screen in my case) doesn=E2=80=99t seem to be hitting zfs at all, but I = may be missing something. Andriy, Mark J, let me know if you think this = is relevant, I can build a 13-STABLE with D32931 if you think it will be = of use. Thanks. Let me know any thoughts you have. - Chris --Apple-Mail=_A7A226AD-D288-4FA6-8B73-D4E088DDCBCF--