From owner-freebsd-stable@freebsd.org Wed Mar 17 17:55:46 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5601B57366A; Wed, 17 Mar 2021 17:55:46 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-lf1-x135.google.com (mail-lf1-x135.google.com [IPv6:2a00:1450:4864:20::135]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0yW55BF7z4pjH; Wed, 17 Mar 2021 17:55:45 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-lf1-x135.google.com with SMTP id b83so240477lfd.11; Wed, 17 Mar 2021 10:55:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=oJHhs2pYPd1wT4t+P0dWfdjDLXJX7od6HXr5i5wj058=; b=ezNYL5XcUNCCu+U+Bq1h/LyzEgs3j5NHeyR88KFlEUrUWfS5T4SZ/ZMt9Z7E41Dld7 8X99Tm5y6SuhfCOfrBR2JfiLdbfHK5esuXoBXi3zryJpqvP3hJt6gUDERu2FsQ7ucy1m Nnf4ZcxYBGaRLbsQKaA6hTNXe4DPP/jwOi3iUealkR9hFuuofcZsy+KFMRJCoupt+I7C I9iMBVPsUm60PRVs7Bnk5YiykR/6WoZI5RqH6Q3AeZiEb1UDmnhVm8ZfO4X38j70SevB wxoyBPTqRRLm58wDAncoqKGE/GHMpSHOAnqVDSZn74WJMgziLv4pw6PyYAdu5eIuzlRU EVHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=oJHhs2pYPd1wT4t+P0dWfdjDLXJX7od6HXr5i5wj058=; b=rgQgfTXe+PeRR9aTc/Vu3cYROGnE46GiBxO0Au3LDAtnM8zWtO1Nm4k93BB4qRZPRc Fiw3uZwVjq9PYJgjfdmXx4tGuyuNlghJQwnkRhcbEUjxe80U3M/7/41UzrEIdFRhM14X gRwvYoP5ULAy93cpTmkDvLkAG9K7nRFxU1+o8hQ1rVJ/5OFqt/yIJnoTKFF9296jE4RA /Jnlw9tO3+9RidIvd2uljZpu9Bac2WbbEQH5TLKhomaAS4Y8VIjjhVptBLvWBdOLgUK2 jiai31Ho6igvpnacMw+1sL/UiH5w5rq52lr0hmyo8clN3lt/mU3wi6QVNJ8kvc9LYyRL U+kA== X-Gm-Message-State: AOAM530iInvislLDYcrUIFWVSKI5PobmVkIf1LI4L9VHZRxc4nedIieh HQ+HvvPTMHCB5HNhPVrEa204sOPdiucBas1/zi6a71tpOsE= X-Google-Smtp-Source: ABdhPJw2Bo68AWjDH8qvaWgflvTgz5V87ey0wWm6suU5gO/1UWzKQBPMd1HST0RITG+QS7rCeYRh/Ts3heR16JRxG+U= X-Received: by 2002:a19:e0d:: with SMTP id 13mr2932646lfo.549.1616003744179; Wed, 17 Mar 2021 10:55:44 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a2e:b54e:0:0:0:0:0 with HTTP; Wed, 17 Mar 2021 10:55:43 -0700 (PDT) In-Reply-To: <20210317164821.a65559ba0df6645085466484@yamagi.org> References: <20210317143307.20beb5fca0814246f2a91e9a@yamagi.org> <20210317164821.a65559ba0df6645085466484@yamagi.org> From: Mateusz Guzik Date: Wed, 17 Mar 2021 18:55:43 +0100 Message-ID: Subject: Re: 13.0-RC2 / 14-CURRENT: Processes getting stuck in vlruwk state To: Yamagi Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4F0yW55BF7z4pjH X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=ezNYL5Xc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2a00:1450:4864:20::135 as permitted sender) smtp.mailfrom=mjguzik@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::135:from]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_DN_SOME(0.00)[]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::135:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::135:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-current,freebsd-stable]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Mailman-Approved-At: Thu, 18 Mar 2021 09:10:46 +0000 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Mar 2021 17:55:46 -0000 Thanks, I'm going to have to ponder a little bit. In the meantime can you apply this: https://people.freebsd.org/~mjg/maxvnodes.diff Once you boot, tweak maxvnodes: sysctl kern.maxvnodes=1049226 Run poudriere. Once it finishes, inspect sysctl vfs.highest_numvnodes On 3/17/21, Yamagi wrote: > Hi Mateusz, > the sysctl output after about 10 minutes into the problem is attached. > In case that its stripped by Mailman a copy can be found here: > https://deponie.yamagi.org/temp/sysctl_vlruwk.txt.xz > > Regards, > Yamagi > > On Wed, 17 Mar 2021 15:57:59 +0100 > Mateusz Guzik wrote: > >> Can you reproduce the problem and run obtain "sysctl -a"? >> >> In general, there is a vnode limit which is probably too small. The >> reclamation mechanism is deficient in that it will eventually inject >> an arbitrary pause. >> >> On 3/17/21, Yamagi wrote: >> > Hi, >> > me and some other users in the ##bsdforen.de IRC channel have the >> > problem that during Poudriere runs processes getting stuck in the >> > 'vlruwk' state. >> > >> > For me it's fairly reproduceable. The problems begin about 20 to 25 >> > minutes after I've started poudriere. At first only some ccache >> > processes hang in the 'vlruwk' state, after another 2 to 3 minutes >> > nearly everything hangs and the total CPU load drops to about 5%. >> > When I stop poudriere with ctrl-c it takes another 3 to 5 minutes >> > until the system recovers. >> > >> > First the setup: >> > * poudriere runs in a bhyve vm on zvol. The host is a 12.2-RELEASE-p2. >> > The zvol has a 8k blocksize, the guests partition are aligned to 8k. >> > The guest has only zpool, the pool was created with ashift=13. The >> > vm has 16 E5-2620 and 16 gigabytes RAM assigned to it. >> > * poudriere is configured with ccache and ALLOW_MAKE_JOBS=yes. Removing >> > either of these options lowers the probability of the problem to show >> > up significantly. >> > >> > I've tried several git revisions starting with 14-CURRENT at >> > 54ac6f721efccdba5a09aa9f38be0a1c4ef6cf14 in the hope that I can find at >> > least one known to be good revision. No chance, even a kernel build >> > from 0932ee9fa0d82b2998993b649f9fa4cc95ba77d6 (Wed Sep 2 19:18:27 2020 >> > +0000) has the problem. The problem isn't reproduceable with >> > 12.2-RELEASE. >> > >> > The kernel stack ('procstat -kk') of a hanging process is: >> > mi_switch+0x155 sleepq_switch+0x109 sleepq_catch_signals+0x3f1 >> > sleepq_wait_sig+0x9 _sleep+0x2aa kern_wait6+0x482 sys_wait4+0x7d >> > amd64_syscall+0x140 fast_syscall_common+0xf8 >> > >> > The kernel stack of vnlru is changing, even while the processes are >> > hanging: >> > * mi_switch+0x155 sleepq_switch+0x109 sleepq_timedwait+0x4b >> > _sleep+0x29b vnlru_proc+0xa05 fork_exit+0x80 fork_trampoline+0xe >> > * fork_exit+0x80 fork_trampoline+0xe >> > >> > Since vnlru is accumulating CPU time it looks like it's doing at least >> > something. As an educated guess I would say that vn_alloc_hard() is >> > waiting a long time or even forever to allocate new vnodes. >> > >> > I can provide more information, I just need to know what. >> > >> > >> > Regards, >> > Yamagi >> > >> > -- >> > Homepage: https://www.yamagi.org >> > Github: https://github.com/yamagi >> > GPG: 0x1D502515 >> > >> >> >> -- >> Mateusz Guzik > > > -- > Homepage: https://www.yamagi.org > Github: https://github.com/yamagi > GPG: 0x1D502515 > -- Mateusz Guzik