saF3+0aUPR9YdphMw6QQRylpPPBzfNv6Bt7ObFISPjJlDFSIeGVm e2XCh//7wjKTCdt4Utb0c64aZsQnvafELNEcWuwON3n53q7K7S8MViIvu6OAQe7jT2SbzZBkM xd5Hj0+AToDaew/C49WqN/FejeSDfJSx80tXo1daHhK7tSmTKC7uRZ5/lP0MitOFMyykAYsQu V00XHKDce47iyH48JBjir0gPpAByBaRr8SO7Jg2nhdeTI4NoDUAyJcQKcB98Xkp8UFj9eJaVb TkNmbbfrrHZOcH9H8XBQAUlIbqwzaSF73dfQQlkVEisxDshl11qG50TPV/afZm2fQrVJjxVCJ wVaWZqoc9xZlonNd+tEUGRXzCB/BIUbjdbptO+vD168tbHnv/UqZFlXtbr9XrVBMNFEsmJPp2 JdLyYKN6kJkXMOcykveULCKarhSrnSJylRxKyL1YLjeJ2pMRpQTGSp+kO2DXlizV/yIqHd05i zrWW98fWbdoWVPWfRk+Wu9l2AXI1frCe1MGT4cMp+nRBj9tXydodrn4/Khfi+qmqjF674W3gn bgXyHrcuKbQTKwwWXp8/0bo/es8HP+j9JpJ7yvtmk8XlN1w4uOFKg8HX0HdUMJEmaJ9xtu3wQ lxk3gqCzxOkIyxugP036JTFNrBsdTriD84hCpNdI0ppeRgWXtAWOcDd9QgpZnrAmzizxYh24m 7jRKGIV9Q8WsLFZrXJ5TSQWxqKbPk7QavVwmMcOOsxKZp2pSclkZ5UrvidkgHIFfMKUgVyJMj 6LRvLnITY4AFNB9JRNhBE9y7VTTHRfCltj5JOGb27EDYDjuC89rE4YLjGcIbIHDfgSbpGpaQp d9j6UOlSn60KyYXj4Ptd9JRNYaonlRseFOIQK12zombC8gSl7KO4R66dX8NL19i9z0nyOQWJJ M+891xOy5+QPmNJcS1UuQlIv5KG2v5N1O6UEax5AeKXa2HnAeiEefFvFmAsaeMJrAx7yobv1E 7e5kD/ZRwVCTympp6a4EVpNxgS5A5pzX+Tb6VnTQTph8esw00MyQB7qz7P/TUyNelu9GKkZQl Pfbdslc1rRe+BTfz0ygcQ5gCa7Gcl39d9x6+gZa0oSZv1PHCJBGapeBAxuim5xWvlfsM9uMZx DqyOKuAHQYjffvvTS0FRejBxmeJEzPvNYd9ltIx6lOxPbTBf+xcYOvvy2IiFuMPEL6X9SXxMB gClGhaGUm/ERhp2dHGrhqP7sabG3K/7Kgx3fT9Ax0UGAGoe62kgHQu3NQuSyRO3nYliwJdlLY llyRdh4NoU3N4JQs8Ol0AnJm7f8YzFcHug90bGdHZsqfXqK3HLIcVsWC4KhLFM+zEnBpZgqxb gavplynz+PPI9U0OmDhKsjAL5Artjj61D/Mx7pBTQog6cW1lLbDAdk1nkFnsvl3p5QJtHZqr5 ATj8I9qUwa5aZezZtNU0k4OH25YZw+5+f9MolPF8q9INofrwAiGqPO+7VcdnHQwBJvWeqh3oq CJz3Hx/EZ6jcTPnLRf5D4fG2/U68TKXNR2YF8+ecB4x8+YKGrTOlKf3aOIBoSI9wDfeGrM2b6 tRo9EjM++tHQOD4+tl/xKxv2An+xrs0Ap21QcTAVMcA2DsbWrVgNletKjKPi1SrhefHcNnc/9 lF1vw4XGB1ThYkedVlSEkNY3gk2yRt0Qi5ipN2ENBuk/likjw+q1WrOfczS+SEiZ5QMhu2wwS FWNrtNaOzQg9t33TUNkcXeS9491J20tjBqSsb6qn8c15Q/S5/UvDZ2/caZMYUNDwvOJTEI8tB Av8G2G9z825w4W0Sjh7g2ngVIJfkYLP+Ah7LIVAhQzeZ02CmRkZdeOLUegF4XMmLBeA0So8yB NVwEbLvZ+gqEvPzxby+paI9MhaqnkdR0JtXfMoke8caDcIM5KMK1jPHSwTOEaGFIiIN4gvy5y 62OeCCsH7S6PDNYpjTtH4y+WSWML6Yz6KtxS9+0T/j+PCJ9Nh6dw07DTXF/npoqLVlQybYpf5 5JbaCJ5+XPY+HRup9+KJ9Tpp0tpW3X0S1D3UaGWZTK5Im3t7OF319EYAocEZcpe6cheZBRa6m oD+1diUNVizhpfc34gQfyWdobk4Qaq3XhmGKCT49xwva7Q1cnBpgY6AmmnckSSmN9ginU9XjO 4LgeCiCV48ATyQMnhcquaqGeu38nRWcDOcTWtqMGV2Vn+wLL7JOSFnoQ7ss+ChFyw5gndCC/b TesQ4+UbuV30wsktwMux9wpbBZX+GPbKKzyYLq2ZHfrqaXVs5mZD1u4XONynvLUu2mCW+3eBS PFBdZOBxBKcQnQ== X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.54 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[gmx.net:dkim]; NEURAL_HAM_MEDIUM(-0.97)[-0.971]; DMARC_POLICY_ALLOW(-0.50)[gmx.net,quarantine]; NEURAL_HAM_SHORT(-0.47)[-0.466]; RWL_MAILSPIKE_VERYGOOD(-0.20)[212.227.15.19:from]; R_DKIM_ALLOW(-0.20)[gmx.net:s=s31663417]; ONCE_RECEIVED(0.20)[]; R_SPF_ALLOW(-0.20)[+a:mout.gmx.net]; RCVD_IN_DNSWL_LOW(-0.10)[212.227.15.19:from]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[gmx.net:+]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:8560, ipnet:212.227.0.0/16, country:DE]; FREEMAIL_FROM(0.00)[gmx.net]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmx.net] X-Rspamd-Queue-Id: 4cxjBC06clz3MmJ After updating from 14.3 to 15.0-BETA I've noticed that chromium builds=20 in poudriere slow down to a crawl after some time. top shows > 95%=20 system usage. dtrace/hotkernel shows > 90% spent in kernel`lock_delay. dtrace -n 'fbt::lock_delay:entry { @[stack()] =3D count(); }' has lots of= =20 traces similar to this one: kernel`__mtx_lock_sleep+0xe8 kernel`vdbatch_process+0x4fb kernel`vdropl+0x20e kernel`vput_final+0xa3 kernel`vn_close1+0x186 kernel`vn_closefile+0x3d kernel`_fdrop+0x11 kernel`closef+0x24a kernel`closefp_impl+0x58 kernel`amd64_syscall+0x126 kernel`0xffffffff809f8a0b In vdbatch_process() there is the following comment above the condition=20 that is controlled by vfs.vnode.param.can_skip_requeue: /* * Attempt to requeue the passed batch, but give up easily. * * Despite batching the mechanism is prone to transient *significant* * lock contention, where vnode_list_mtx becomes the primary bottleneck * if multiple CPUs get here (one real-world example is highly parallel * do-nothing make , which will stat *tons* of vnodes). Since it is * quasi-LRU (read: not that great even if fully honoured) provide an * option to just dodge the problem. Parties which don't like it are * welcome to implement something better. */ if (vnode_can_skip_requeue) { ... Setting "sysctl vfs.vnode.param.can_skip_requeue=3D1" remedies the=20 situation immediately and system usage returns to ~15%. I cannot recall such problems in 14.3, is this a regression in 15.x?