Date: Wed, 17 Mar 2021 16:48:21 +0100 From: Yamagi <lists@yamagi.org> To: mjguzik@gmail.com Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: Re: 13.0-RC2 / 14-CURRENT: Processes getting stuck in vlruwk state Message-ID: <20210317164821.a65559ba0df6645085466484@yamagi.org> In-Reply-To: <CAGudoHG5emBBEMS_aZUH7jqfnEXWELsAhu-iCXhV=NkQ1g4QMQ@mail.gmail.com> References: <20210317143307.20beb5fca0814246f2a91e9a@yamagi.org> <CAGudoHG5emBBEMS_aZUH7jqfnEXWELsAhu-iCXhV=NkQ1g4QMQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Signature=_Wed__17_Mar_2021_16_48_21_+0100_DEdMUc8dIpFmFhem Content-Type: multipart/mixed; boundary="Multipart=_Wed__17_Mar_2021_16_48_21_+0100_/VOKigFwl+jSLIdi" --Multipart=_Wed__17_Mar_2021_16_48_21_+0100_/VOKigFwl+jSLIdi Content-Type: text/plain; charset=US-ASCII Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi Mateusz, the sysctl output after about 10 minutes into the problem is attached. In case that its stripped by Mailman a copy can be found here: https://deponie.yamagi.org/temp/sysctl_vlruwk.txt.xz Regards, Yamagi On Wed, 17 Mar 2021 15:57:59 +0100 Mateusz Guzik <mjguzik@gmail.com> wrote: > Can you reproduce the problem and run obtain "sysctl -a"? >=20 > In general, there is a vnode limit which is probably too small. The > reclamation mechanism is deficient in that it will eventually inject > an arbitrary pause. >=20 > On 3/17/21, Yamagi <lists@yamagi.org> wrote: > > Hi, > > me and some other users in the ##bsdforen.de IRC channel have the > > problem that during Poudriere runs processes getting stuck in the > > 'vlruwk' state. > > > > For me it's fairly reproduceable. The problems begin about 20 to 25 > > minutes after I've started poudriere. At first only some ccache > > processes hang in the 'vlruwk' state, after another 2 to 3 minutes > > nearly everything hangs and the total CPU load drops to about 5%. > > When I stop poudriere with ctrl-c it takes another 3 to 5 minutes > > until the system recovers. > > > > First the setup: > > * poudriere runs in a bhyve vm on zvol. The host is a 12.2-RELEASE-p2. > > The zvol has a 8k blocksize, the guests partition are aligned to 8k. > > The guest has only zpool, the pool was created with ashift=3D13. The > > vm has 16 E5-2620 and 16 gigabytes RAM assigned to it. > > * poudriere is configured with ccache and ALLOW_MAKE_JOBS=3Dyes. Removi= ng > > either of these options lowers the probability of the problem to show > > up significantly. > > > > I've tried several git revisions starting with 14-CURRENT at > > 54ac6f721efccdba5a09aa9f38be0a1c4ef6cf14 in the hope that I can find at > > least one known to be good revision. No chance, even a kernel build > > from 0932ee9fa0d82b2998993b649f9fa4cc95ba77d6 (Wed Sep 2 19:18:27 2020 > > +0000) has the problem. The problem isn't reproduceable with > > 12.2-RELEASE. > > > > The kernel stack ('procstat -kk') of a hanging process is: > > mi_switch+0x155 sleepq_switch+0x109 sleepq_catch_signals+0x3f1 > > sleepq_wait_sig+0x9 _sleep+0x2aa kern_wait6+0x482 sys_wait4+0x7d > > amd64_syscall+0x140 fast_syscall_common+0xf8 > > > > The kernel stack of vnlru is changing, even while the processes are > > hanging: > > * mi_switch+0x155 sleepq_switch+0x109 sleepq_timedwait+0x4b > > _sleep+0x29b vnlru_proc+0xa05 fork_exit+0x80 fork_trampoline+0xe > > * fork_exit+0x80 fork_trampoline+0xe > > > > Since vnlru is accumulating CPU time it looks like it's doing at least > > something. As an educated guess I would say that vn_alloc_hard() is > > waiting a long time or even forever to allocate new vnodes. > > > > I can provide more information, I just need to know what. > > > > > > Regards, > > Yamagi > > > > -- > > Homepage: https://www.yamagi.org > > Github: https://github.com/yamagi > > GPG: 0x1D502515 > > >=20 >=20 > --=20 > Mateusz Guzik <mjguzik gmail.com> --=20 Homepage: https://www.yamagi.org Github: https://github.com/yamagi GPG: 0x1D502515 --Multipart=_Wed__17_Mar_2021_16_48_21_+0100_/VOKigFwl+jSLIdi-- --Signature=_Wed__17_Mar_2021_16_48_21_+0100_DEdMUc8dIpFmFhem Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEOXu/lxyufwz0gC5x6xRy5x1QJRUFAmBSJMUACgkQ6xRy5x1Q JRX8CBAAtm3zBXsxt+dcfDOXr2xMq2pwEM2mvmNkUYwwPv7puP44Awx/CzaZ7e02 uLqxfdujyP5w7oVTekTDbGx6KvbJipd4gGacbkaKOGhL3p66eYH/rECztkfA0xMA y6YacNIlYYHuwyTKvG9BjbhDNDWz7xmDz4WZxsnPIdWShMFtri0LHkiOztCcN8pc Hb5n78MplG+Q5qlvE0zDykhS1PrYJy+TZwk/x+4FUISEilzfZ2hyAOoFBYhuCFIM decjPEPGFzGTIiR9j8Sa0/shdVGGn4b6ZBFBBCMRxYFUn6Ar+Z9Q5+/ez+loiKR2 Po/i/eCY2MgcaKn8UyiwR7Ip4HvFgfvHH0WNzZM8CV8xaR6Hfu3zw7vnAsyd82A5 jTZoLKxbknXvQ4A+qq5hmVzJ/Nli+hnf8BUk3/T3EkPCTsqh5hfAv6sewAHV+rtU hXckf7qbzcuqLhflEkC9SQJxT9i0U8lxSE1h5vSYT4TgqTzomIQM1pzJiBLNl76l Zo0lqEcLkT6oGQEjTTBtFY2PW38q2pgY3hJi4yzMxEaJ6FizREXFEND/u0n4+L92 RSrfXOg/AdjFb4e/nWNt/xFxxw6lF4vPq+B55NWD223hCPFBDIFEc01qiS6weWQK 0d1c92lOT2CUX/JNzCD5N2/p6Hn+Wx7GUnfBxWWoEH6LVYO+JFc= =P6Ld -----END PGP SIGNATURE----- --Signature=_Wed__17_Mar_2021_16_48_21_+0100_DEdMUc8dIpFmFhem--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20210317164821.a65559ba0df6645085466484>