Date: Fri, 12 Nov 2021 10:35:57 +0100 (CET) From: Ronald Klop via freebsd-fs <freebsd-fs@freebsd.org> To: freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: swap_pager: cannot allocate bio Message-ID: <42006135.15.1636709757975@mailrelay> In-Reply-To: <4E5511DF-B163-4928-9CC3-22755683999E@distal.com> References: <9FE99EEF-37C5-43D1-AC9D-17F3EDA19606@distal.com> <09989390-FED9-45A6-A866-4605D3766DFE@distal.com> <op.1cpimpsmkndu52@joepie> <4E5511DF-B163-4928-9CC3-22755683999E@distal.com>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_14_1187132451.1636709757927 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable =20 Van: Chris Ross <cross+freebsd@distal.com> Datum: donderdag, 11 november 2021 22:49 Aan: ronald-lists@klop.ws CC: freebsd-fs <freebsd-fs@freebsd.org> Onderwerp: Re: swap_pager: cannot allocate bio >=20 > =20 > >>=20 >> On Nov 11, 2021, at 13:50, Ronald Klop via freebsd-fs <freebsd-fs@freebs= d.org> wrote: >>=20 >> Can you press ctrl-t on the hanging process? That should print the stack= trace indicating where it is waiting on. >=20 > =20 > So, I rebooted the machine this morning, but now have [tried to] log into= it to check on it and find that an ssh connection doesn=E2=80=99t result i= n a shell. I logged into the console, tried to start a =E2=80=9Cscreen=E2= =80=9D to get more prompts, and it hung. Ctrl-T on that shows (after runni= ng a console screen-capture through OCR, and hand correction, so may not be= 100%): > =20 > root@host:~ # screen > load: 0.07 cmd: csh 56116 [vmwait] 35.00r 0.00u 0.01s 0% 3984k > mi_switch+0xc1 _sleep+0x1cb vm_wait_doms+0xe2 vm_wait_domain+0x51 vm_doma= in_alloc_fail+0x86 vm_page_alloc_domain_after+0x7e uma_small_alloc+0x58 keg= _alloc_slab+0xba zone_import+0xee zone_alloc_item+0x6f malloc+0x5d sigacts_= alloc+0x1c fork1+0x9fb sys_fork+0x54 amd64_syscall+0x10c fast_syscall_commo= n+0xf8=20 > =20 > As before, ps and even mount and df work here on console. But, a =E2=80= =9Czpool status tank=E2=80=9D will hang as before. A Ctrl+D on it > =20 > root@host:~ # screen > load: 0.00 cmd: zpool 62829 [aw.aew_cv] 37.89r 0.00u 0.00s 0% 6976k > mi_switch+0xc1 _cv_wait+0xf2 arc_wait_for_eviction+0x14a arc_get_data_imp= l+0xdb arc_hdr_alloc_abd+0xa6 arc_hdr_alloc+0x11e arc_read+0x4f4 dbuf_read+= 0xc08 dmu_buf_hold+0x46 zap_lookup_norm+0x35 zap_contains+0x26 vdev_rebuild= _get_stats+0xac vdev_config_generate+0x3e9 vdev_config_generate+0x74f spa_c= onfig_generate+0x2a2 spa_open_common+0x25c spa_get_stats+0x4e zfs_ioc_pool_= stats+0x22 > =20 > =20 > =20 >>=20 >> On Nov 11, 2021, at 14:10, Dave Cottlehuber <> wrote: >> =20 >> Grab output of =E2=80=98procstat-kk=E2=80=99 and see if this is similar = to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258208 a few more p= rods might get this one addressed! >=20 > procstat -kk 62829 yields the same as above. Which I presume is expected= , I=E2=80=99d just never used procstate -kk before. > =20 > Unfortunately, I can=E2=80=99t tell if this is sufficiently similar to bu= g 258208. A different ZFS operation is happening here, so the calls behind= my zpool status are different. The other non-zfs stat above (screen in my= case) doesn=E2=80=99t seem to be hitting zfs at all, but I may be missing = something. Andriy, Mark J, let me know if you think this is relevant, I ca= n build a 13-STABLE with D32931 if you think it will be of use. > =20 > Thanks. Let me know any thoughts you have. > =20 > - Chris > =20 Hi, Interesting. The details of these stacktraces are unknown to me. But it loo= ks like it is waiting for available memory in both cases. What is the memor= y usage of the system while all this is happening. Is it swapping a lot? And what is the real setup of the disks? Are things like GELI used (not tha= t the stack shows that) or swap-on-zfs? And is there something else interesting in the logs than "swap_pager: canno= t allocate bio"? Maybe a reason why it can't allocate the bio. I would not know a pointer on how to debug this except for checking tools l= ike iostat, vmstat, etc.. Of course running 13-STABLE can give an interesti= ng data point. Regards, Ronald. =20 ------=_Part_14_1187132451.1636709757927--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?42006135.15.1636709757975>